0
크롤링을 시도하고있는 페이지에서 크롤링을 시도하고 있습니다. 예를 들어 beautifulsoup 문서를 참조하면 페이지를 크롤링하고 해당 페이지에서 정보를 얻는 방법 만 알면됩니다. 어떤 제안?아름다운 수프가 구문 분석하지 못했습니다.
여기에 코드
에게
#!/usr/bin/env python # -*- coding: utf-8 -*- import requests import urllib from bs4 import BeautifulSoup glimit = 100 def my_spider(max_pages): page = 2 while page <= max_pages: url = 'http://www.bbb.org/search/?type=name&input=constrution&location=Austin%2c+TX&filter=combined&accredited=&radius=5000&country=USA&language=en&codeType=YPPA' url_2 = url + '&page='+ str(page) +'&source=bbbse' source_code = requests.get(url_2) plain_text = source_code.text soup = BeautifulSoup(plain_text, "html5lib") limit = glimit li = soup.find('h4', {'class': 'hcolor'}) children = li.find_all("a") for result in children: href = "http://www.bbb.org" + result.get('href') owl = (result.string) print owl get_single_item_data(href) page += 1 def get_single_item_data(item_url): source_code = requests.get(item_url) plain_text = source_code.text soup = BeautifulSoup(plain_text, "html5lib") limit = glimit mysoup = soup.findAll('h3',{'class': 'address__heading' })[:limit] mysoup2 = mysoup.find_all("a") for item in mysoup2: href = "http://www.bbb.org" + item.get('href') print (item.string) my_spider(2)
Main.py
이며, 여기에 오류Traceback (most recent call last):
File "main.py", line 44, in <module>
my_spider(2)
File "main.py", line 27, in my_spider
get_single_item_data(href)
File "main.py", line 33, in get_single_item_data
source_code = requests.get(item_url)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
prep = self.prepare_request(req)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
self.prepare_url(url, params)
File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
scheme, auth, host, port, path, query, fragment = parse_url(url)
File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
raise LocationParseError("Failed to parse: %s" % url)
urllib3.exceptions.LocationParseError: Failed to parse: Failed to parse: www.bbb.orghttp:
이 작업에 도움이되었습니다. 시간을내어 도와 주셔서 감사합니다. 감사합니다. – BARNOWL
아무 문제없이 재미있게 코딩하십시오. – Zroq