Scrapy : 여러 페이지에 걸쳐 항목 로더로 항목 채우기

Question

여러 개의 URL이있는 경우 여러 페이지를 크롤링하고 다 써야합니다. 나는 Wikipedia로 테스트 중이며 더 쉽게 각 페이지에 대해 동일한 Xpath 선택기를 사용했지만 각 페이지마다 고유 한 많은 다른 Xpath 선택기를 사용하므로 각 페이지마다 별도의 parsePage 메서드가 있습니다.

이 코드는 항목 로더를 사용하지 않고 항목을 직접 채울 때 완벽하게 작동합니다. 항목 로더를 사용할 때 항목이 이상하게 채워지고 parse 메서드에 할당 된 콜백을 완전히 무시하고 parsePage 메서드에 대한 start_urls 만 사용하는 것처럼 보입니다. 항목 로더와 함께 여기

{'title1': [u'2016 Rugby Championship'], 
'title': [u'SANZAAR'], 
'title3': [u'2016 Super Rugby season']} 

(가) 로그의 약간의 :

{'title2': u'SANZAAR'} 
2016-09-24 14:30:43 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/2016_Rugby_Championship> (referer: https://en.wikipedia.org/wiki/SANZAAR) 
2016-09-24 14:30:43 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/2016_Rugby_Championship> (referer: https://en.wikipedia.org/wiki/2016_Rugby_Championship) 
2016-09-24 14:30:43 [scrapy] DEBUG: Scraped from <200 https://en.wikipedia.org/wiki/2016_Super_Rugby_season> 
{'title2': u'SANZAAR', 'title3': u'SANZAAR'} 
2016-09-24 14:30:43 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/SANZAAR> (referer: https://en.wikipedia.org/wiki/2016_Rugby_Championship) 
2016-09-24 14:30:43 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/2016_Rugby_Championship> (referer: https://en.wikipedia.org/wiki/2016_Super_Rugby_season) 
2016-09-24 14:30:43 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/2016_Super_Rugby_season> (referer: https://en.wikipedia.org/wiki/2016_Rugby_Championship) 
2016-09-24 14:30:43 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/2016_Super_Rugby_season> (referer: https://en.wikipedia.org/wiki/2016_Super_Rugby_season) 
2016-09-24 14:30:43 [scrapy] DEBUG: Scraped from <200 https://en.wikipedia.org/wiki/2016_Rugby_Championship> 
{'title1': u'SANZAAR', 'title2': u'SANZAAR', 'title3': u'SANZAAR'} 
2016-09-24 14:30:43 [scrapy] DEBUG: Scraped from <200 https://en.wikipedia.org/wiki/2016_Rugby_Championship> 
{'title1': u'2016 Rugby Championship'} 
2016-09-24 14:30:43 [scrapy] DEBUG: Scraped from <200 https://en.wikipedia.org/wiki/SANZAAR> 
{'title1': u'2016 Rugby Championship', 'title2': u'2016 Rugby Championship'} 
2016-09-24 14:30:43 [scrapy] DEBUG: Scraped from <200 https://en.wikipedia.org/wiki/2016_Rugby_Championship> 
{'title1': u'2016 Super Rugby season'} 
2016-09-24 14:30:43 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/SANZAAR> (referer: https://en.wikipedia.org/wiki/2016_Super_Rugby_season) 
2016-09-24 14:30:43 [scrapy] DEBUG: Scraped from <200 https://en.wikipedia.org/wiki/2016_Super_Rugby_season> 
{'title1': u'2016 Rugby Championship', 
'title2': u'2016 Rugby Championship', 
'title3': u'2016 Rugby Championship'} 
2016-09-24 14:30:43 [scrapy] DEBUG: Scraped from <200 https://en.wikipedia.org/wiki/2016_Super_Rugby_season> 
{'title1': u'2016 Super Rugby season', 'title3': u'2016 Super Rugby season'} 
2016-09-24 14:30:43 [scrapy] DEBUG: Scraped from <200 https://en.wikipedia.org/wiki/SANZAAR> 
{'title1': u'2016 Super Rugby season', 
'title2': u'2016 Super Rugby season', 
'title3': u'2016 Super Rugby season'} 
2016-09-24 14:30:43 [scrapy] INFO: Clos 

을 정확히 잘못된 것입니다 내가 항목 로더를 사용하지 않는 경우

import scrapy 
from scrapy.http import Request 
from scrapy import Spider, Request, Selector 
from testanother.items import TestItems, TheLoader 

class tester(scrapy.Spider): 
name = 'vs' 
handle_httpstatus_list = [404, 200, 300] 
#Usually, I only get data from the first start url 
start_urls = ['https://en.wikipedia.org/wiki/SANZAAR','https://en.wikipedia.org/wiki/2016_Rugby_Championship','https://en.wikipedia.org/wiki/2016_Super_Rugby_season'] 
def parse(self, response): 
    #item = TestItems() 
    l = TheLoader(item=TestItems(), response=response) 
    #when I use an item loader, the url in the request is completely ignored. without the item loader, it works properly. 
    request = Request("https://en.wikipedia.org/wiki/2016_Rugby_Championship", callback=self.parsePage1, meta={'loadernext':l}, dont_filter=True) 
    yield request 

    request = Request("https://en.wikipedia.org/wiki/SANZAAR", callback=self.parsePage2, meta={'loadernext1': l}, dont_filter=True) 
    yield request 

    yield Request("https://en.wikipedia.org/wiki/2016_Super_Rugby_season", callback=self.parsePage3, meta={'loadernext2': l}, dont_filter=True) 

def parsePage1(self,response): 
    loadernext = response.meta['loadernext'] 
    loadernext.add_xpath('title1', '//*[@id="firstHeading"]/text()') 
    return loadernext.load_item() 
#I'm not sure if this return and load_item is the problem, because I've tried yielding/returning to another method that does the item loading instead and the first start url is still the only url scraped. 
def parsePage2(self,response): 
    loadernext1 = response.meta['loadernext1'] 
    loadernext1.add_xpath('title2', '//*[@id="firstHeading"]/text()') 
    return loadernext1.load_item() 

def parsePage3(self,response): 
    loadernext2 = response.meta['loadernext2'] 
    loadernext2.add_xpath('title3', '//*[@id="firstHeading"]/text()') 
    return loadernext2.load_item() 

다음은 결과입니까? 감사!

Answer 1

하나의 문제는 개의 동일한 참조 로더 인스턴스의 여러 참조를 여러 콜백으로 전달한다는 것입니다. parse에는 두 개의 yield request 명령어가 있습니다.

또한 다음 콜백에서 로더는 여전히 이전 response 개체 (예 : parsePage1에서 항목 로더는 response에서 여전히 parse에서 작동합니다.

대부분의 경우 항목 로더를 다른 콜백에 전달하지 않는 것이 좋습니다. 또는 항목 개체를 직접 전달하는 것이 더 나을 수도 있습니다.

여기에 코드를 편집하여, 짧은 (불완전한) 예제 :

def parse(self, response): 
    l = TheLoader(item=TestItems(), response=response) 
    request = Request(
     "https://en.wikipedia.org/wiki/2016_Rugby_Championship", 
     callback=self.parsePage1, 
     meta={'item': l.load_item()}, 
     dont_filter=True 
    ) 
    yield request 

def parsePage1(self,response): 
    loadernext = TheLoader(item=response.meta['item'], response=response) 
    loadernext.add_xpath('title1', '//*[@id="firstHeading"]/text()') 
    return loadernext.load_item()