GAE 데이터 스토어 로더 클래스 생성에 대한 도움이 필요합니까?

도움이 필요하십니까? appcfg.py를 사용하여 데이터를 업로드하기위한 GAE Datastore Loader 클래스를 만드시겠습니까? 다른 방법으로는이 프로세스를 단순화 할 수 있습니까? 는이 모든 자세한 예보다 나은 here GAE 데이터 스토어 로더 클래스 생성에 대한 도움이 필요합니까?

시도하여 bulkloader.yaml 경우 : 과정에서

Uploading data records. 
[INFO ] Logging to bulkloader-log-20100701.041515 
[INFO ] Throttling transfers: 
[INFO ] Bandwidth: 250000 bytes/second 
[INFO ] HTTP connections: 8/second 
[INFO ] Entities inserted/fetched/modified: 20/second 
[INFO ] Batch Size: 10 
[INFO ] Opening database: bulkloader-progress-20100701.041515.sql3 
[INFO ] Connecting to livelihoodproducer.appspot.com/remote_api 
[INFO ] Starting import; maximum 10 entities per post 
[ERROR ] [Thread-1] WorkerThread: 
Traceback (most recent call last): 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 150, in WorkOnItems 
    status, instruction = item.PerformWork(self.__thread_pool) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 693, in PerformWork 
    transfer_time = self._TransferItem(thread_pool) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 848, in _TransferItem 
    self.content = self.request_manager.EncodeContent(self.rows) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1269, in EncodeContent 
    entity = loader.create_entity(values, key_name=key, parent=parent) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 385, in create_entity 
    return self.dict_to_entity(input_dict, self.bulkload_state) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 133, in dict_to_entity 
    self.__run_import_transforms(input_dict, instance, bulkload_state_copy) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 233, in __run_import_transforms 
    value = self.__dict_to_prop(transform, input_dict, bulkload_state) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 188, in __dict_to_prop 
    value = transform.import_transform(value) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_parser.py", line 93, in __call__ 
    return self.method(*args, **kwargs) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/transform.py", line 143, in generate_foreign_key_lambda 
    return datastore.Key.from_path(kind, value) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_types.py", line 387, in from_path 
    'received %r (a %s).' % (i + 2, id_or_name, typename(id_or_name))) 
BadArgumentError: Expected an integer id or string name as argument 2; received None (a NoneType). 
[INFO ] [Thread-3] Backing off due to errors: 1.0 seconds 
[INFO ] Unexpected thread death: Thread-1 
[INFO ] An error occurred. Shutting down... 
[ERROR ] Error in Thread-1: Expected an integer id or string name as argument 2; received None (a NoneType). 

[INFO ] 30 entites total, 0 previously transferred 
[INFO ] 0 entities (733 bytes) transferred in 2.8 seconds 
[INFO ] Some entities not successfully transferred

, 난 수동 appspot.com에 삽입 CSV 데이터를 downloadeded했습니다. 내 CSV 데이터를 업로드하려고 시도하는 동안 열 순서는 appspot.com에서 다운로드 한 CSV와 정확히 일치해야합니다. 빈 값은 어떨까요?

출처

2010-06-30 Ivan Slaughter

[없음] 값이있는 참조 속성이있는 것처럼 보입니다. 이러한 값은 대량로드 프로그램의 도우미가 잘못 처리합니다.

출처

2010-08-25 17:35:16

아니 확신

도우미는 (파일 helpers.py 그냥 당신이 config.yaml을 배치 같은 디렉토리에 배치, 매우 간단합니다) 이 도우미 함수, bulkloader 도우미를 만드는 데 필요한 참조는 무엇입니까? 나는이 참조를 찾으려고 할 때 아무렇지도 않게 얻는다. 세르게이 감사합니다. –

해결책으로 다른 답을 쓸 것입니다. –

bulkloader config를 사용하여 config.yaml을 만들었으며 None-references를 처리하는 간단한 도우미 함수를 작성했습니다. 원래 도우미가 아닌 이유는 모르겠다.

from google.appengine.api import datastore 
def create_foreign_key(kind, key_is_id=False): 
    def generate_foreign_key_lambda(value): 
    if value is None: 
     return None 

    if key_is_id: 
     value = int(value) 
    return datastore.Key.from_path(kind, value) 

    return generate_foreign_key_lambda

그리고 이것은 내 config.yaml에서 절단 :

python_preamble: 
- import: helpers # this will import our helper 
[other imports] 
... 
- kind: ArticleComment 
    connector: simplexml 
    connector_options: 
    xpath_to_nodes: "/blog/Comments/Comment" 
    style: element_centric 

    property_map: 
    - property: __key__ 
     external_name: key 
     export_transform: transform.key_id_or_name_as_string 

    - property: parent_comment 
     external_name: parent-comment 
     export_transform: transform.key_id_or_name_as_string 
     import_transform: helpers.create_foreign_key('ArticleComment') 
     #     ^^^^^^^ here it is 
     #     use this instead of transform.create_foreign_key

출처

2010-08-28 11:41:06

GAE 데이터 스토어 로더 클래스 생성에 대한 도움이 필요합니까?

답변

관련 문제