1
나는 웹 사이트에서 일부 데이터를 긁어 postgree의 데이터베이스에 저장하려면 원하는,하지만 난 다음 오류 받고 있어요 :Scrapy ImportError를 : 이름 없음 모듈 없다 '설정'
ImportError: No module named 'settings'
내 트리 뷰를 폴더는 다음과 같습니다
Coder
├── coder_app
│ ├── __init__.py
│ ├── items.py
│ ├── models.py
│ ├── pipelines.py
│ ├── settings.py
│ └── spiders
│ ├── __init__.py
│ └── livingsocial.py
├── output.json
└── scrapy.cfg
스파이더 코드는 다음과 같습니다 :
# -*- coding: utf-8 -*-
import scrapy
from coder_app.items import CoderItem
# from scrapy.loader import ItemLoader
class LivingsocialSpider(scrapy.Spider):
name = "livingsocial"
allowed_domains = ["livingsocial.com"]
start_urls = (
'http://www.livingsocial.com/cities/15-san-francisco',
)
def parse(self, response):
# deals = response.xpath('//li')
for deal in response.xpath('//li[a//h2/text()]'):
item = CoderItem()
item['title'] = deal.xpath('a//h2/text()').extract_first()
item['link'] = deal.xpath('a/@href').extract_first()
item['location'] = deal.xpath('a//p[@class="location"]/text()').extract_first()
item['price'] = deal.xpath('a//div[@class="deal-price"]/text()').extract_first()
item['end_date'] = deal.xpath('a//p[@class="dates"]/text()').extract_first()
yield item
Pipeline.py은 다음과 같습니다
from sqlalchemy import create_engine, Column, Integer, String, DateTime
from sqlalchemy.engine.url import URL
from sqlalchemy.ext.declarative import declarative_base
import settings
DeclarativeBase = declarative_base()
def db_connect():
"""
Performs database connection using database settings from settings.py.
Returns sqlalchemy engine instance
"""
return create_engine(URL(**settings.DATABASE))
def create_deals_table(engine):
"""
Performs table creation
"""
DeclarativeBase.metadata.create_all(engine)
class Deals(DeclarativeBase):
"""Sqlalchemy deals model"""
__tablename__ = "deals"
id = Column(Integer, primary_key=True)
title = Column('title', String)
link = Column('link', String, nullable=True)
location = Column('location', String, nullable=True)
#original_price = Column('original_price', String, nullable=True)
price = Column('price', String, nullable=True)
end_date = Column('end_date', DateTime, nullable=True)
및 settings.py은 다음과 같습니다 :
from sqlalchemy.orm import sessionmaker
from coder_app.models import Deals, db_connect, create_deals_table
class CoderPipeline(object):
def process_item(self, item, spider):
return item
class LivingSocialPipeline(object):
"""Livingsocial pipeline for storing scraped items in the database"""
def __init__(self):
"""
Initializes database connection and sessionmaker.
Creates deals table.
"""
engine = db_connect()
create_deals_table(engine)
self.Session = sessionmaker(bind=engine)
def process_item(self, item, spider):
"""Save deals in the database.
This method is called for every item pipeline component.
"""
session = self.Session()
deal = Deals(**item)
try:
session.add(deal)
session.commit()
except:
session.rollback()
raise
finally:
session.close()
return item
Model.py 코드는
BOT_NAME = 'coder_app'
SPIDER_MODULES = ['coder_app.spiders']
NEWSPIDER_MODULE = 'coder_app.spiders'
DATABASE = {
'drivername': 'postgres',
'host': 'localhost',
'port': '5432',
'username': 'mohib',
'password': '100200',
'database': 'scrape'
}
ITEM_PIPELINES = {
'coder_app.pipelines.LivingSocialPipeline': 300,
}
왜 내가 또한 내가 시도,이 오류를 얻고있다
code_app.import settings
model.py하지만 다음 오류가 발생했습니다 :
NameError: name 'settings' is not defined
저는 정말로 여기에 갇혀 있습니다. 누구든지 나를 도울 수 있습니까? 대신
import settings
쓰기
from scrapy.utils.project import get_project_settings
다음 함수 또는 메소드 내부의
'code_app 가져 오기 설정에서 가져 왔습니까? – elethan
은 매력처럼 작동합니다! 고마워 :) – Mohib