중복 된 URL의 파이썬을 제거하십시오

Question

URL 목록을 가진 파일에서 중복 된 URL을 제거하고 싶습니다. 내 bugun_url_given.txt는 "http://www.bugun.com.tr/ara/Ak%20Parti/1"을 가지고 있으며 모든 URL을 가져 와서 그들은 모두 고유 한 URL의에 저장 "bugun_url_collection.tx" 여기 내 코드입니다 .. 을 반복됩니다

from cookielib import CookieJar 
import urllib2 
import json 
from bs4 import BeautifulSoup 
cj = CookieJar() 
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) 
try: 
    text_file = open('bugun_url_given.txt', 'r') 
    for line in text_file: 
     print line 
     soup = BeautifulSoup(opener.open(line)) 
     links = soup.select('div.nwslist a') 
     for link in links: 
      print link 
      #unique_url = set(map(lambda url : url.strip("/ "), links)) 
      with open('bugun_url_collection.txt', 'a') as f: 
       for link in links: 
        f.write(link.get('href') + '\n') 
except ValueError: 
    pass    

Answer 1

  for link in links: 
       f.write(link.get('href') + '\n') 

수 (정확) 주석에 대한 응답으로

  for link in set(link.get('href') for link in links): 
       f.write(link + '\n') 

될,의 제대로이 문제를 다시 보자

from cookielib import CookieJar 
import urllib2 
import json 
from bs4 import BeautifulSoup 
cj = CookieJar() 
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) 


def write_links_to_file(links): 
    with open('bugun_url_collection.txt', 'a') as f: 
     f.writeline(link) 


def get_links_from_file(text_file): 
    for line in text_file: 
     print line 
     soup = BeautifulSoup(opener.open(line)) 
     links = soup.select('div.nwslist a') 
     for link in links: 
      yield link.get('href') 


with open('bugun_url_given.txt', 'r') as text_file: 
    links = get_links_from_file(text_file) 

unique_links = set(link for link in links) 
write_links_to_file(unique_links) 

Answer 2

당신은 할 수

hrefs = [] 
for link in links: 
    print link 
    hrefs.append(link.get('href')) 
hrefs = list(set(hrefs)) 
with open('bugun_url_collection.txt', 'a') as f: 
    f.write('\n'.join(hrefs)) 

Answer 3

당신이 그들을 저장하는 코드에서 링크를 생성하는 코드를 분리해야

:

당신이 링크를 len(links) 번 통과되도록 당신은, 당신의 for 루프를 중첩 한

def generate_urls(filename, urlopen): 
    with open(filename) as file: 
     for line in file: 
      soup = BeautifulSoup(urlopen(line.strip())) 
      for link in soup.select('div.nwslist a[href^="http"]'): 
       yield link['href'] 

links = set(generate_urls('bugun_url_given.txt', opener.open)) 
with open('bugun_url_collection.txt', 'w') as file: 
    file.write("\n".join(links)) 

Answer 4

.

links = soup.select('div.nwslist a') 
    for link in links: 
     ... 
     with open('bugun_url_collection.txt', 'a') as f: 
      for link in links: 
       f.write(link.get('href') + '\n') 

당신이 정말로 원하는 것입니다 :

이 가 가

with open('bugun_url_given.txt', 'r') as text_file, text_file = open('bugun_url_given.txt', 'r'): 
    for line in text_file: 
     print line 
     soup = BeautifulSoup(opener.open(line)) 
     links = set(link for link in soup.select('div.nwslist a')) 
     for link in links: 
      print link 
      #unique_url = set(map(lambda url : url.strip("/ "), links)) 
      f.write(link.get('href') + '\n') 

가