치료법, 도메인 당 시간을 제한하는 방법은 무엇입니까?

Question

나는 몇 가지 질문을했지만 답변을 찾고 있으며이 포럼에 대한 답변이 없습니다. 한 가지 대답은 특정 시간 이후에 거미를 멈추는 것이 가능하다는 것이지만, 거미 당 10 개의 웹 사이트를 시작하기 때문에 나에게 적합하지 않습니다. 그래서 나의 도전은 내가 10 websited 거미가 있고 일부 webshop에 갇혀지고 피하기 위해 도메인 당 20 초 시간을 제한하고 싶습니다. 그것을하는 방법?

일반적으로 나는 2000 개의 회사 웹 사이트를 크롤링한다고 말하면서 언젠가는이 웹 사이트를 200 개의 10 개의 웹 사이트 그룹으로 나눠서 200 개의 스파이더를 병렬로 실행합니다. 그것은 아마추어일지도 모르지만 나는 내가 아는 최고입니다. 스파이더가 전체 CPU와 메모리를 소비하기 때문에 컴퓨터가 거의 정지되지만 다음 날에는 결과가 나옵니다. 내가 찾고있는 것은 회사 웹 사이트의 고용 웹 페이지입니다. 2000 개 웹 사이트를 크롤링하는 방법에 대해 더 잘 알고있는 사람이 있습니까? 웹 사이트간에 웹샵이있는 경우 크롤링에 며칠이 걸릴 수 있으므로 도메인 당 시간을 제한하고 싶습니다.

미리 감사드립니다.

마르코

내 코드 :

는

#!/usr/bin/python 
# -*- coding: utf-8 -*- 
# encoding=UTF-8 
import scrapy, urlparse, time, sys 
from scrapy.http import Request 
from scrapy.utils.response import get_base_url 
from urlparse import urlparse, urljoin 
from vacancies.items import JobItem 

#We need that in order to force Slovenian pages instead of English pages. It happened at "http://www.g-gmi.si/gmiweb/" that only English pages were found and no Slovenian. 
from scrapy.conf import settings 
settings.overrides['DEFAULT_REQUEST_HEADERS'] = {'Accept':'text/html,application/xhtml+xml;q=0.9,*/*;q=0.8','Accept-Language':'sl',} 
#Settings.set(name, value, priority='cmdline') 
#settings.overrides['DEFAULT_REQUEST_HEADERS'] = {'Accept':'text/html,application/xhtml+xml;q=0.9,*/*;q=0.8','Accept-Language':'sl','en':q=0.8,} 




#start_time = time.time() 
# We run the programme in the command line with this command: 

#  scrapy crawl jobs -o urls.csv -t csv --logfile log.txt 


# We get two output files 
# 1) urls.csv 
# 2) log.txt 

# Url whitelist. 
with open("Q:/Big_Data/Spletne_strani_podjetij/strganje/kljucne_besede/url_whitelist.txt", "r+") as kw: 
    url_whitelist = kw.read().replace('\n', '').split(",") 
url_whitelist = map(str.strip, url_whitelist) 

# Tab whitelist. 
# We need to replace character the same way as in detector. 
with open("Q:/Big_Data/Spletne_strani_podjetij/strganje/kljucne_besede/tab_whitelist.txt", "r+") as kw: 
    tab_whitelist = kw.read().decode(sys.stdin.encoding).encode('utf-8') 
tab_whitelist = tab_whitelist.replace('Ŕ', 'č') 
tab_whitelist = tab_whitelist.replace('L', 'č') 
tab_whitelist = tab_whitelist.replace('Ő', 'š') 
tab_whitelist = tab_whitelist.replace('Ü', 'š') 
tab_whitelist = tab_whitelist.replace('Ä', 'ž') 
tab_whitelist = tab_whitelist.replace('×', 'ž') 
tab_whitelist = tab_whitelist.replace('\n', '').split(",") 
tab_whitelist = map(str.strip, tab_whitelist) 



# Look for occupations in url. 
with open("Q:/Big_Data/Spletne_strani_podjetij/strganje/kljucne_besede/occupations_url.txt", "r+") as occ_url: 
    occupations_url = occ_url.read().replace('\n', '').split(",") 
occupations_url = map(str.strip, occupations_url) 

# Look for occupations in tab. 
# We need to replace character the same way as in detector. 
with open("Q:/Big_Data/Spletne_strani_podjetij/strganje/kljucne_besede/occupations_tab.txt", "r+") as occ_tab: 
    occupations_tab = occ_tab.read().decode(sys.stdin.encoding).encode('utf-8') 
occupations_tab = occupations_tab.replace('Ŕ', 'č') 
occupations_tab = occupations_tab.replace('L', 'č') 
occupations_tab = occupations_tab.replace('Ő', 'š') 
occupations_tab = occupations_tab.replace('Ü', 'š') 
occupations_tab = occupations_tab.replace('Ä', 'ž') 
occupations_tab = occupations_tab.replace('×', 'ž') 
occupations_tab = occupations_tab.replace('\n', '').split(",") 
occupations_tab = map(str.strip, occupations_tab) 

#Join url whitelist and occupations. 
url_whitelist_occupations = url_whitelist + occupations_url 

#Join tab whitelist and occupations. 
tab_whitelist_occupations = tab_whitelist + occupations_tab 


#base = open("G:/myVE/vacancies/bazni.txt", "w") 
#non_base = open("G:/myVE/vacancies/ne_bazni.txt", "w") 


class JobSpider(scrapy.Spider): 

    #Name of spider 
    name = "jobs" 

    #start_urls = open("Q:\Big_Data\Utrip\spletne_strani.txt", "r+").readlines()[0] 
    #print urls 
    #start_urls = map(str.strip, urls) 
    #Start urls 
    start_urls = ["http://www.alius.si"] 
    print "\nSpletna stran   ", start_urls, "\n" 

    #Result of the programme is this list of job vacancies webpages. 
    jobs_urls = [] 


    def parse(self, response): 

     #Theoretically I could save the HTML of webpage to be able to check later and see how it looked like 
     # at the time of downloading. That is important for validation, because it is easier to look at nice HTML webpage instead of naked text. 
     # but I have to write a pipeline http://doc.scrapy.org/en/0.20/topics/item-pipeline.html 

     response.selector.remove_namespaces() 
     #print "response url" , str(response.url) 

     #Take url of response, because we would like to stay on the same domain. 
     parsed = urlparse(response.url) 

     #Base url.   
     #base_url = get_base_url(response).strip() 
     base_url = parsed.scheme+'://'+parsed.netloc 
     #print "base url" , str(base_url) 
     #If the urls grows from seeds, it's ok, otherwise not. 
     if base_url in self.start_urls: 
      #print "base url je v start" 
      #base.write(response.url+"\n") 



      #net1 = parsed.netloc 

      #Take all urls, they are marked by "href" or "data-link". These are either webpages on our website either new websites. 
      urls_href = response.xpath('//@href').extract()  
      urls_datalink = response.xpath('//@data-link').extract() 
      urls = urls_href + urls_datalink 
      #print "povezave na tej strani ", urls 




      #Loop through all urls on the webpage. 
      for url in urls: 

       #Test all new urls. NE DELA 

       #print "url ", str(url) 

       #If url doesn't start with "http", it is relative url, and we add base url to get absolute url.  
       if not (url.startswith("http")): 

        #Povežem delni url z baznim url. 
        url = urljoin(base_url,url).strip() 

       #print "new url ", str(url) 

       new_parsed = urlparse(url) 
       new_base_url = new_parsed.scheme+'://'+new_parsed.netloc 
       #print "new base url ", str(new_base_url) 

       if new_base_url in self.start_urls: 
        #print "yes" 

        url = url.replace("\r", "") 
        url = url.replace("\n", "") 
        url = url.replace("\t", "") 
        url = url.strip() 

        #Remove anchors '#', that point to a section on the same webpage, because this is the same webpage. 
        #But we keep question marks '?', which mean, that different content is pulled from database. 
        if '#' in url: 
         index = url.find('#') 
         url = url[:index] 
         if url in self.jobs_urls: 
          continue 




        #Ignore ftp and sftp. 
        if url.startswith("ftp") or url.startswith("sftp"): 

         continue 





        #Compare each url on the webpage with original url, so that spider doesn't wander away on the net. 
        #net2 = urlparse(url).netloc 
        #test.write("lokacija novega url "+ str(net2)+"\n") 

        #if net2 != net1: 
        # continue 
         #test.write("ni ista lokacija, nadaljujemo\n") 

        #If the last character is slash /, I remove it to avoid duplicates. 
        if url[len(url)-1] == '/':   
         url = url[:(len(url)-1)] 


        #If url includes characters like %, ~ ... it is LIKELY NOT to be the one I are looking for and I ignore it. 
        #However in this case I exclude good urls like http://www.mdm.si/company#employment 
        if any(x in url for x in ['%', '~', 

         #slike 
         '.jpg', '.jpeg', '.png', '.gif', '.eps', '.ico', '.svg', '.tif', '.tiff', 
         '.JPG', '.JPEG', '.PNG', '.GIF', '.EPS', '.ICO', '.SVG', '.TIF', '.TIFF', 

         #dokumenti 
         '.xls', '.ppt', '.doc', '.xlsx', '.pptx', '.docx', '.txt', '.csv', '.pdf', '.pd', 
         '.XLS', '.PPT', '.DOC', '.XLSX', '.PPTX', '.DOCX', '.TXT', '.CSV', '.PDF', '.PD', 

         #glasba in video 
         '.mp3', '.mp4', '.mpg', '.ai', '.avi', '.swf', 
         '.MP3', '.MP4', '.MPG', '.AI', '.AVI', '.SWF', 

         #stiskanje in drugo 
         '.zip', '.rar', '.css', '.flv', '.xml' 
         '.ZIP', '.RAR', '.CSS', '.FLV', '.XML' 

         #Twitter, Facebook, Youtube 
         '://twitter.com', '://mobile.twitter.com', 'www.twitter.com', 
         'www.facebook.com', 'www.youtube.com' 

         #Feeds, RSS, arhiv 
         '/feed', '=feed', '&feed', 'rss.xml', 'arhiv' 


           ]): 

         continue 


        #We need to save original url for xpath, in case we change it later (join it with base_url) 
        #url_xpath = url      


        #We don't want to go to other websites. We want to stay on our website, so we keep only urls with domain (netloc) of the company we are investigating.   
        #if (urlparse(url).netloc == urlparse(base_url).netloc): 



        #The main part. We look for webpages, whose urls include one of the employment words as strings. 
        #We will check the tab of the url as well. This is additional filter, suggested by Dan Wu, to improve accuracy. 
        #tabs = response.xpath('//a[@href="%s"]/text()' % url_xpath).extract() 
        tabs = response.xpath('//a[@href="%s"]/text()' % url).extract() 

        # Sometimes tabs can be just empty spaces like '\t' and '\n' so in this case we replace it with []. 
        # That was the case when the spider didn't find this employment url: http://www.terme-krka.com/si/sl/o-termah-krka/o-podjetju-in-skupini-krka/zaposlitev/ 
        tabs = [tab.encode('utf-8') for tab in tabs] 
        tabs = [tab.replace('\t', '') for tab in tabs] 
        tabs = [tab.replace('\n', '') for tab in tabs] 
        tab_empty = True 
        for tab in tabs: 
         if tab != '': 
          tab_empty = False 
        if tab_empty == True: 
         tabs = [] 


        # -- Instruction. 
        # -- Users in other languages, please insert employment words in your own language, like jobs, vacancies, career, employment ... -- 
        # Starting keyword_url is zero, then we add keywords as we find them in url. 
        keyword_url = '' 
        #for keyword in url_whitelist: 
        for keyword in url_whitelist_occupations: 

         if keyword in url: 
          keyword_url = keyword_url + keyword + ' ' 
        # a) If we find at least one keyword in url, we continue. 
        if keyword_url != '':     

         #1. Tabs are empty. 
         if tabs == []: 



          #We found url that includes one of the magic words and also the text includes a magic word. 
          #We check url, if we have found it before. If it is new, we add it to the list "jobs_urls". 
          if url not in self.jobs_urls : 


           self.jobs_urls.append(url) 
           item = JobItem() 
           item["url"] = url 
           #item["keyword_url"] = keyword_url 
           #item["keyword_url_tab"] = ' ' 
           #item["keyword_tab"] = ' ' 
           print "Zaposlitvena podstran ", url 

           #We return the item. 
           yield item 



         #2. There are texts in tabs, one or more. 
         else: 

          #For the same partial url several texts are possible. 
          for tab in tabs:        

           #We search for keywords in tabs. 
           keyword_url_tab = '' 
           #for key in tab_whitelist: 
           for key in tab_whitelist_occupations: 

            if key in tab: 
             keyword_url_tab = keyword_url_tab + key + ' ' 

           # If we find some keywords in tabs, then we have found keywords in both url and tab and we can save the url. 
           if keyword_url_tab != '': 

            # keyword_url_tab starts with keyword_url from before, because we want to remember keywords from both url and tab. So we add initial keyword_url. 
            keyword_url_tab = 'URL ' + keyword_url + ' TAB ' + keyword_url_tab 

            #We found url that includes one of the magic words and also the tab includes a magic word. 
            #We check url, if we have found it before. If it is new, we add it to the list "jobs_urls". 
            if url not in self.jobs_urls:        

             self.jobs_urls.append(url) 
             item = JobItem() 
             item["url"] = url 
             #item["keyword_url"] = ' ' 
             #item["keyword_url_tab"] = keyword_url_tab 
             #item["keyword_tab"] = ' ' 
             print "Zaposlitvena podstran ", url 

             #We return the item. 
             yield item 

           #We haven't found any keywords in tabs, but url is still good, because it contains some keywords, so we save it. 
           else: 

            if url not in self.jobs_urls:        

             self.jobs_urls.append(url) 
             item = JobItem() 
             item["url"] = url 
             #item["keyword_url"] = keyword_url 
             #item["keyword_url_tab"] = ' ' 
             #item["keyword_tab"] = ' ' 
             print "Zaposlitvena podstran ", url 

             #We return the item. 
             yield item        

        # b) If keyword_url = empty, there are no keywords in url, but perhaps there are keywords in tabs. So we check tabs. 
        else: 
         for tab in tabs: 


          keyword_tab = '' 
          #for key in tab_whitelist: 
          for key in tab_whitelist_occupations: 


           if key in tab: 
            keyword_tab = keyword_tab + key + ' ' 
          if keyword_tab != '':       

           if url not in self.jobs_urls:        

            self.jobs_urls.append(url) 
            item = JobItem() 
            item["url"] = url 
            #item["keyword_url"] = ' ' 
            #item["keyword_url_tab"] = ' ' 
            #item["keyword_tab"] = keyword_tab 
            print "Zaposlitvena podstran ", url 

            #We return the item. 
            yield item     

        #We don't put "else" sentence because we want to further explore the employment webpage to find possible new employment webpages. 
        #We keep looking for employment webpages, until we reach the DEPTH set in settings.py. 
        yield Request(url, callback = self.parse) 

      #else: 
       #non_base.write(response.url+"\n") 

Answer 1

만 2000 단일 웹 사이트 크롤링 일정을 scrapyd를 사용합니다. max_proc = 10 [1]을 설정하면 10 개의 스파이더를 병렬로 실행할 수 있습니다. 거미의 CLOSESPIDER_TIMEOUT [2]을 모든 거미를 20 초 동안 20 번 돌린다. 그것이 기본적으로 Windows 사용을 중단하십시오. 나는 Scrap와 scrapyd가 Windows에서 네이티브가 아닌 VM에서 더 빠르게 실행되는 것을 관찰했습니다. 내가 틀릴 수도 있습니다 - 직접 확인해보십시오. 그러나 Windows에서 Ubuntu 14.04 virtualbox image을 사용하면 더 빠를 것입니다. 귀하의 크롤링은 정확히 2000 * 20/10 = 17 분이 소요됩니다.