Scrapy를 사용하여 크롤링 할 때 아무런 오류도 나타나지 않지만 아무 것도 크롤링하지 않습니다.

Question

저는 Python을 처음 접해서 Scrapy를 배우기 시작합니다. 다음과 같이 거미 파일

내 코드는 다음과 같습니다

from openbl.items import OpenblItem 
import scrapy 
import time 

class OpenblSpider(scrapy.Spider): 
    name='openbl' 
    start_url=['http://www.openbl.org/lists/base_1days.txt'] 
    def parse(self, response): 
     #get the content within 'pre', select the 1st element to get the content string. 
     #split the space of the content 
     content=response.xpath('/pre/text').extract()[0].split() 
     # This for loop is used to get the num of element in list content 
     # after which the elements of the list are the IPs we desire. 
     for i in range(0,len(content)): 
      if content[i]=='ip': 
       i+=1 
       break 
      else: 
       pass 
     # construct a new list content_data for putting IPs in. 
     content_data=[] 
     # This for loop put useful data(IPs) into the new list above. 
     for x in range(i,len(content)): 
      content_data.append(content(i)) 

     for cont in content_data: 
      item=OpenblItem() 
      item['name']=cont 
      item['date']=time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())) 
      item['type']='other' 
      yield item 

내가 크롤링하고있어 웹 사이트는 다음과 같습니다 http://www.openbl.org/lists/base_1days.txt 내가 항목 [ '이름']

I로이 웹 사이트 IP를 얻으려면 누군가가 내 질문에 대답 할 수 있다면 고맙겠습니다.

---

일부 오류가

가

V:\work\openbl>scrapy crawl openbl -o openbl_data.json 
2017-01-05 10:46:22 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: openbl) 
2017-01-05 10:46:22 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'openbl.spiders', 'FEED_URI': 'openbl_data.json', 'SPIDER_MODULES': ['openbl.spiders'], 'BOT_NAME': 'openbl', 'ROBOTSTXT_OBEY': True, 'FEED_FORMAT': 'json'} 
2017-01-05 10:46:22 [scrapy.middleware] INFO: Enabled extensions: 
['scrapy.extensions.feedexport.FeedExporter', 
'scrapy.extensions.logstats.LogStats', 
'scrapy.extensions.telnet.TelnetConsole', 
'scrapy.extensions.corestats.CoreStats'] 
2017-01-05 10:46:22 [scrapy.middleware] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2017-01-05 10:46:22 [scrapy.middleware] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
'scrapy.spidermiddlewares.referer.RefererMiddleware', 
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2017-01-05 10:46:22 [scrapy.middleware] INFO: Enabled item pipelines: 
[] 
2017-01-05 10:46:22 [scrapy.core.engine] INFO: Spider opened 
2017-01-05 10:46:22 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2017-01-05 10:46:22 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 
2017-01-05 10:46:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.openbl.org/robots.txt> (referer: None) 
2017-01-05 10:46:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.openbl.org/lists/base_1days.txt> (referer: None) 
2017-01-05 10:46:23 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.openbl.org/lists/base_1days.txt> (referer: None) 
Traceback (most recent call last): 
    File "c:\python27\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback 
    yield next(it) 
    File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output 
    for x in result: 
    File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 22, in <genexpr> 
    return (_set_referer(r) for r in result or()) 
    File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr> 
    return (r for r in result or() if _filter(r)) 
    File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr> 
    return (r for r in result or() if _filter(r)) 
    File "V:\work\openbl\openbl\spiders\openbl_spider.py", line 11, in parse 
    content=response.xpath('/pre/text').extract()[0].split() 
IndexError: list index out of range 
2017-01-05 10:46:23 [scrapy.core.engine] INFO: Closing spider (finished) 
2017-01-05 10:46:23 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 454, 
'downloader/request_count': 2, 
'downloader/request_method_count/GET': 2, 
'downloader/response_bytes': 4907, 
'downloader/response_count': 2, 
'downloader/response_status_count/200': 2, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2017, 1, 5, 2, 46, 23, 515000), 
'log_count/DEBUG': 3, 
'log_count/ERROR': 1, 
'log_count/INFO': 7, 
'response_received_count': 2, 
'scheduler/dequeued': 1, 
'scheduler/dequeued/memory': 1, 
'scheduler/enqueued': 1, 
'scheduler/enqueued/memory': 1, 
'spider_exceptions/IndexError': 1, 
'start_time': datetime.datetime(2017, 1, 5, 2, 46, 22, 383000)} 
2017-01-05 10:46:23 [scrapy.core.engine] INFO: Spider closed (finished) 

Answer 1

그렇지 않으면 나는 그것의 거 크롤링 아무것도 생각하지 않는 모든

start_urls의 첫 번째 코드와 몇 가지 문제가 있습니다 .. 지금 온다

일반 텍스트 파일이므로 response.body은 문자 그대로 일반 텍스트이기 때문에 오류가 발생합니다. 태그가 없습니다. 그래서 당신은 바운드 예외로부터 인덱스를 얻고 있습니다. 일반 텍스트로 처리하고 정규 표현식으로 정보를 추출하고 \n 등으로 나눌 수 있습니다. 이렇게하는 방법은 너무 많습니다.

또한 루프 변수를 사용하지 마십시오. 이것은 단지 옳다고 생각하지 않습니다. 목록에있는 항목 중 첫 번째 항목의 색인을 찾으려면 index() 기능이 있습니다. 또는 What is the best way to get the first item from an iterable matching a condition?