로딩 RDF는 거장 오픈 소스

Question

내가 this dump에서 LinkedGeoData.org의 로컬 미러를 만들려고 해요로 세배.

약 6,1000,000 개의 트리플이 있습니다. 버추 오소 (Virtuoso)는 그보다 훨씬 더 쉽게 처리 할 수 ​​있지만, 매번 약 4,000,000 개의 트리플을 실행하면로드가 중단됩니다. 저는 Amazon EC2의 두 배의 초대형 인스턴스를 사용하고 있습니다.이 인스턴스에는 30GB의 RAM이 있으며 충분한 저장 공간이 남아 있습니다. 구성 파일에 이상이 있습니까? 나는 우분투 서버 12.04를 사용하고, 나는 Jörn Hees' instructions 다음 (버전 6.1.6) apt-get (버전 6.1.5)를 통해 거장을 설치하고 GitHub의에서 최신 안정 소스에서 컴파일 시도했습니다.

는 또한 작은 조각으로 덤프 파일을 분할하고 하나씩로드 시도했습니다. 이것은 약 4,000,000 개의 트리플이 삽입 된 후에도 무너집니다.

로그 파일에 이상한 내용이 표시되지 않습니다. virtuoso-t은 실제로 충돌하지 않고 작동을 멈추고 top은 0 %의 CPU를 사용하는 프로세스를 보여줍니다. 처음 30 분 정도 지나면 아무런 진전없이 며칠 동안 프로세스를 진행했습니다. 어떤 도움을 크게

[Database] 
DatabaseFile   = /var/lib/virtuoso/db/virtuoso.db 
ErrorLogFile   = /var/lib/virtuoso/db/virtuoso.log 
LockFile   = /var/lib/virtuoso/db/virtuoso.lck 
TransactionFile   = /var/lib/virtuoso/db/virtuoso.trx 
xa_persistent_file  = /var/lib/virtuoso/db/virtuoso.pxa 
ErrorLogLevel   = 7 
FileExtend   = 200 
MaxCheckpointRemap  = 625000 
Striping   = 0 
TempStorage   = TempDatabase 


[TempDatabase] 
DatabaseFile   = /var/lib/virtuoso/db/virtuoso-temp.db 
TransactionFile   = /var/lib/virtuoso/db/virtuoso-temp.trx 
MaxCheckpointRemap  = 2000 
Striping   = 0 


; 
; Server parameters 
; 
[Parameters] 
ServerPort   = 1111 
LiteMode   = 0 
DisableUnixSocket  = 1 
DisableTcpSocket  = 0 
;SSLServerPort   = 2111 
;SSLCertificate   = cert.pem 
;SSLPrivateKey   = pk.pem 
;X509ClientVerify  = 0 
;X509ClientVerifyDepth  = 0 
;X509ClientVerifyCAFile  = ca.pem 
ServerThreads   = 20 
CheckpointInterval  = 60 
O_DIRECT   = 0 
CaseMode   = 2 
MaxStaticCursorRows  = 5000 
CheckpointAuditTrail  = 0 
AllowOSCalls   = 0 
SchedulerInterval  = 10 
DirsAllowed   = ., /usr/share/virtuoso/vad, /home/ubuntu/lgd 
ThreadCleanupInterval  = 0 
ThreadThreshold   = 10 
ResourcesCleanupInterval = 0 
FreeTextBatchSize  = 100000 
SingleCPU   = 0 
VADInstallDir   = /usr/share/virtuoso/vad/ 
PrefixResultNames    = 0 
RdfFreeTextRulesSize  = 100 
IndexTreeMaps   = 256 
MaxMemPoolSize     = 200000000 
PrefixResultNames    = 0 
MacSpotlight     = 0 
IndexTreeMaps     = 64 
;; 
;; When running with large data sets, one should configure the Virtuoso 
;; process to use between 2/3 to 3/5 of free system memory and to stripe 
;; storage on all available disks. 
;; 
;; Uncomment next two lines if there is 2 GB system memory free 
;  NumberOfBuffers   = 170000 
;  MaxDirtyBuffers   = 130000 
;; Uncomment next two lines if there is 4 GB system memory free 
;  NumberOfBuffers   = 340000 
;  MaxDirtyBuffers   = 250000 
;; Uncomment next two lines if there is 8 GB system memory free 
;  NumberOfBuffers   = 680000 
;  MaxDirtyBuffers   = 500000 
;; Uncomment next two lines if there is 16 GB system memory free 
;  NumberOfBuffers   = 1360000 
;  MaxDirtyBuffers   = 1000000 
;; Uncomment next two lines if there is 32 GB system memory free 
     NumberOfBuffers   = 2720000 
     MaxDirtyBuffers   = 2000000 
;; Uncomment next two lines if there is 48 GB system memory free 
;  NumberOfBuffers   = 4000000 
;  MaxDirtyBuffers   = 3000000 
;; Uncomment next two lines if there is 64 GB system memory free 
;  NumberOfBuffers   = 5450000 
;  MaxDirtyBuffers   = 4000000 
;; 
;; Note the default settings will take very little memory 
;; but will not result in very good performance 
;; 


[HTTPServer] 
ServerPort   = 8890 
ServerRoot   = /var/lib/virtuoso/vsp 
ServerThreads   = 20 
DavRoot    = DAV 
EnabledDavVSP   = 0 
HTTPProxyEnabled  = 0 
TempASPXDir   = 0 
DefaultMailServer  = localhost:25 
ServerThreads   = 10 
MaxKeepAlives   = 10 
KeepAliveTimeout  = 10 
MaxCachedProxyConnections = 10 
ProxyConnectionCacheTimeout = 15 
HTTPThreadSize   = 280000 
HttpPrintWarningsInOutput = 0 
Charset    = UTF-8 
;HTTPLogFile    = logs/http.log 

[AutoRepair] 
BadParentLinks   = 0 

[Client] 
SQL_PREFETCH_ROWS  = 100 
SQL_PREFETCH_BYTES  = 16000 
SQL_QUERY_TIMEOUT  = 0 
SQL_TXN_TIMEOUT   = 0 
;SQL_NO_CHAR_C_ESCAPE  = 1 
;SQL_UTF8_EXECS   = 0 
;SQL_NO_SYSTEM_TABLES  = 0 
;SQL_BINARY_TIMESTAMP  = 1 
;SQL_ENCRYPTION_ON_PASSWORD = -1 

[VDB] 
ArrayOptimization  = 0 
NumArrayParameters  = 10 
VDBDisconnectTimeout  = 1000 
KeepConnectionOnFixedThread = 0 

[Replication] 
ServerName   = db-IP-10-252-61-61 
ServerEnable   = 1 
QueueMax   = 50000 


; 
; Striping setup 
; 
; These parameters have only effect when Striping is set to 1 in the 
; [Database] section, in which case the DatabaseFile parameter is ignored. 
; 
; With striping, the database is spawned across multiple segments 
; where each segment can have multiple stripes. 
; 
; Format of the lines below: 
; Segment<number> = <size>, <stripe file name> [, <stripe file name> .. ] 
; 
; <number> must be ordered from 1 up. 
; 
; The <size> is the total size of the segment which is equally divided 
; across all stripes forming the segment. Its specification can be in 
; gigabytes (g), megabytes (m), kilobytes (k) or in database blocks 
; (b, the default) 
; 
; Note that the segment size must be a multiple of the database page size 
; which is currently 8k. Also, the segment size must be divisible by the 
; number of stripe files forming the segment. 
; 
; The example below creates a 200 meg database striped on two segments 
; with two stripes of 50 meg and one of 100 meg. 
; 
; You can always add more segments to the configuration, but once 
; added, do not change the setup. 
; 
[Striping] 
Segment1   = 100M, db-seg1-1.db, db-seg1-2.db 
Segment2   = 100M, db-seg2-1.db 
;... 

;[TempStriping] 
;Segment1   = 100M, db-seg1-1.db, db-seg1-2.db 
;Segment2   = 100M, db-seg2-1.db 
;... 

;[Ucms] 
;UcmPath   = <path> 
;Ucm1    = <file> 
;Ucm2    = <file> 
;... 


[Zero Config] 
ServerName   = virtuoso (IP-10-252-61-61) 
;ServerDSN   = ZDSN 
;SSLServerName   = 
;SSLServerDSN   = 


[Mono] 
;MONO_TRACE   = Off 
;MONO_PATH   = <path_here> 
;MONO_ROOT   = <path_here> 
;MONO_CFG_DIR   = <path_here> 
;virtclr.dll   = 


[URIQA] 
DynamicLocal   = 0 
DefaultHost   = localhost:8890 


[SPARQL] 
;ExternalQuerySource  = 1 
;ExternalXsltSource   = 1 
;DefaultGraph   = http://localhost:8890/dataspace 
;ImmutableGraphs   = http://localhost:8890/dataspace 
ResultSetMaxRows   = 10000 
MaxQueryCostEstimationTime = 4000 ; in seconds 
MaxQueryExecutionTime  = 600 ; in seconds 
DefaultQuery    = select distinct ?Concept where {[] a ?Concept} LIMIT 100 
DeferInferenceRulesInit  = 0 ; controls inference rules loading 
;PingService   = http://rpc.pingthesemanticweb.com/ 
ShortenLongURIs = 1 

[Plugins] 
LoadPath   = /usr/lib/virtuoso/hosting 
Load1    = plain, wikiv 
Load2    = plain, mediawiki 
Load3    = plain, creolewiki 
Load4   = plain, im 

를 감사 :

여기 내 virtuoso.ini 파일입니다.

Answer 1

내 질문에 답해주세요. 문제는 그 삭제 라인

NumberOfBuffers   = 2720000 
    MaxDirtyBuffers   = 2000000 

의 선행 공백이었다 거장 실제로 사용 가능한 메모리 대신 기본 16메가바이트을 사용했다.