R 병렬 코드의 parSapply 문제

Question

parSapply로 내 R 코드를 최적화하려고합니다. 나는 전체 변수로 xmlfile과 x을 가지고 있습니다.

clusterExport (cl, "X") 및 clusterExport (cl, "xmlfile")를 사용하지 않았을 때 "xmlfile 객체를 찾을 수 없습니다"라는 메시지가 나타납니다.

내가이 두 가지를 사용했을 때 clusterExport " 'externalptr'유형의 객체가 하위 집합이 아닙니다. '라는 오류가 발생했습니다.

일반 sapply에서는 정상적으로 작동합니다.

누군가가 문제를 볼 수 있습니까? 내가 parSapply 문제합니까 무엇

require("XML") 
library(parallel) 


setwd("C:/PcapParser") 
# A helper function that enables the dynamic additon of new rows and unseen variables to a data.frame 
# field is an R XML leaf-node (capturing a field of a protocol) 
# X is the current data.frame to which the feature in field should be added 
# rowNum is the row (packet) to which the feature should be added. [must be that rowNum <= dim(X)[1]+1] 
addFeature <- function(field, X, rowNum) 
{ 
    # extract xml name and value 
    featureName = xmlAttrs(field)['name'] 

    if (featureName == "") 
    featureName = xmlAttrs(field)['show'] 

    value = xmlAttrs(field)['value'] 
    if (is.na(value) | value=="") 
    value = xmlAttrs(field)['show'] 

    # attempt to add feature (add rows/cols if neccessary) 
    if (!(featureName %in% colnames(X))) #we are adding a new feature 
    { 
    #Special cases 
    #Bad column names: anything that has the prefix... 
    badCols = list("<","Content-encoded entity body"," ","\\?") 
    for(prefix in badCols) 
     if(grepl(paste("^",prefix,sep=""),featureName)) 
     return(X) #don't include this new feature 

    X[[featureName]]=array(dim=dim(X)[1]) #add this new feature column with NAs 
    } 

    if (rowNum > dim(X)[1]) #we are trying to add a new row 
    {X = rbind(X,array(dim=dim(X)[2]))} #add row of NA 

    X[[featureName]][rowNum] = value 
    return(X) 
} 

firstLoop<-function(x) 
{ 


    packet = xmlfile[[x]] 

    # Iterate over all protocols in this packet 
    for (prot in 1:xmlSize(packet)) 
    { 
    protocol = packet[[prot]] 
    numFields = xmlSize(protocol) 

    # Iterate over all fields in this protocol (recursion is not used since the passed dataset is large) 
    if(numFields>0) 
     for (f in 1:numFields) 
     { 
     field = protocol[[f]] 

     if (xmlSize(field) == 0) # leaf 
      X<<-addFeature(field,X,x) 
     else #not leaf xml element (assumption: there are at most three more steps down) 
     { 
      # Iterate over all sub-fields in this field 
      for (ff in 1:xmlSize(field)) 
      { #extract sub-field data for this packet 
      subField = field[[ff]] 

      if (xmlSize(subField) == 0) # leaf 
       X<<-addFeature(subField,X,x) 
      else #not leaf xml element (assumption: there are at most two more steps down) 
      { 
       # Iterate over all subsub-fields in this field 
       for (fff in 1:xmlSize(subField)) 
       { #extract sub-field data for this packet 
       subsubField = subField[[fff]] 

       if (xmlSize(subsubField) == 0) # leaf 
        X<<-addFeature(subsubField,X,x) 
       else #not leaf xml element (assumption: there is at most one more step down) 
       { 
        # Iterate over all subsubsub-fields in this field 
        for (ffff in 1:xmlSize(subsubField)) 
        { #extract sub-field data for this packet 
        subsubsubField = subsubField[[ffff]] 
        X<<-addFeature(subsubsubField,X,x) #must be leaf 
        } 
       } 
       } 
      } 
      } 
     } 
     } 
    } 
} 
# Given the path to a pcap file, this function returns a dataframe 'X' 
# with m rows that contain data fields extractable from each of the m packets in XMLcap. 
# Wireshark must be intalled to work 
raw_feature_extractor <- function(pcapPath){ 
    ## Step 1: convert pcap into PDML XML file with wireshark 
    #to run this line, wireshark must be installed in the location referenced in the pdmlconv.bat file 
    print("Converting pcap file with Wireshark.") 
    system(paste("pdmlconv",pcapPath,"tmp.xml")) 

    ## Step 2: load XML file into R 
    print("Parsing XML.") 
    xmlfile<<-xmlRoot(xmlParse("tmp.xml")) 

    ## Step 3: Extract all feature into data.frame 
    print("Extracting raw features.") 
    X <<- data.frame(num=NA) #first feature is packet number 


    # Iterate over all packets 
    # Calculate the number of cores 
    no_cores <- detectCores() - 1 

    # Initiate cluster 
    cl <- makeCluster(3) 

    parSapply (cl,seq(from=1,to=xmlSize(xmlfile),by=1),firstLoop) 




    print("Done.") 
return(X) 
} 

:

나는이 R 코드가? (아마 전역 변수를 고려) 그래서이 코드에 명백한 문제의 몇 가지를 볼

당신에게

Answer 1

감사드립니다. 전역 변수와 함수는 명시 적으로 강제로 호출하거나 호출하지 않는 한 병렬 환경에서 이 아닌이 될 수 있습니다. addFunction 및 raw_feature_extractor 함수를 firstLoop 안에 정의해야합니다. 기존 패키지에서 함수를 호출 할 때 패키지를 firstLoop (부적합한 코딩!)의 일부로로드하거나 package::function 표기법 (좋은 코딩!)을 사용하여 패키지를 명시 적으로 호출해야합니다. StackOverflow에서 R 문서를 살펴보면 적절히 병렬화 된 함수를 만드는 데 도움이 될 것입니다.