V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
tbag781623489
V2EX  ›  Python

请教 scrapy 的 cookie 导入方式?

  •  
  •   tbag781623489 · 2018-06-02 14:20:40 +08:00 · 2542 次点击
    这是一个创建于 2355 天前的主题,其中的信息可能已经有所发展或是发生改变。

    本人刚接触scrapy,主要看中了能暂停,继续,去查这几点,以前一直用requests或者requests-html,所以都是这样处理cookie的:

    headers = {
        'Accept-Language':'zh-CN',
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36',
        # dummy:
        "cookie":"__cfduid=d2ea661e4cddec1eebedc2c54d78e6156698409;__utma=50351329.433346.1527221979.1527221979.1527221979.1;__utmc=501329;__utmz=501329.15271979.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);__51cke__=;__51laig__=1;__tins__3878067=%7B%22sid%22%3A%201527221980323%2C%20%22vd%22%3A%201%2C%20%22expires%22%3A%201527223780323%7D;91username=regf3qc;CLIPSHARE=n9u906378onmf990kdlpq3;DUID=15fdcQq0vvU0U10%2FdVsdg5smxiBM18RO%2FQ5O3uuV%2F3pe;EMAILVERIFIED=no;language=cn_CN;level=1;remainclosed=1;user_level=1;USERNAME=1a9dkmjCzR6TueiJdjZzM6gIUE8AXAHzxIuxb4x6vEk;",
    
    }
    r = requests.get('http://example.com',headers=headers)
    

    但是看scrapy文档的话是把cookieheader分开输入的,而且cookie还只能用 dict 输入。上面的cookie直接用EditThisCookie导出就可以了。cookie不是包含在了header里面了吗?在scrapy里这样写似乎也没用:

    class PornSpider(scrapy.Spider):
        name = 'pron'
    
        def start_requests(self):
            urls = ['http://pron.com/page={}'.format(i) for i in range(1,3)]
            for url in urls:
                yield Request(url,callback=self.parse,headers=headers) #这样似乎是没有用的, 返回的应该是不带 cookie 的页面
    

    scrapy看起来比自己写方便很多,那想请教下能不能用requests库取代scrapy自带的Request呢?或者有没有更优雅的cookie导入方法呢?

    谢谢大家

    第 1 条附言  ·  2018-06-02 22:27:15 +08:00

    Resigned to fate.

    cookie_raw = "__cfduid=d2ea661e4cddec1eebedc2c54d78e6156698409;__utma=50351329.433346.1527221979.1527221979.1527221979.1;__utmc=501329;__utmz=501329.15271979.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);__51cke__=;__51laig__=1;__tins__3878067=%7B%22sid%22%3A%201527221980323%2C%20%22vd%22%3A%201%2C%20%22expires%22%3A%201527223780323%7D;91username=regf3qc;CLIPSHARE=n9u906378onmf990kdlpq3;DUID=15fdcQq0vvU0U10%2FdVsdg5smxiBM18RO%2FQ5O3uuV%2F3pe;EMAILVERIFIED=no;language=cn_CN;level=1;remainclosed=1;user_level=1;USERNAME=1a9dkmjCzR6TueiJdjZzM6gIUE8AXAHzxIuxb4x6vEk;"
    cookie = {}
    for i in cookie_raw.split(';')[:-1]:
        cookie[i.split('=')[0]] = i.split('=')[1]
    
    2 条回复    2018-06-02 22:24:34 +08:00
    Leigg
        1
    Leigg  
       2018-06-02 22:23:05 +08:00
    def start_requests(self):
    cookies = settings.weibo_cookies (字典)
    #cookie 登录
    return [FormRequest(self.start_urls[1], cookies=cookies, formdata={}:这个参数是请求的 body,callback=self.parse)]
    Leigg
        2
    Leigg  
       2018-06-02 22:24:34 +08:00
    注意缩进。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   3556 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 22ms · UTC 00:09 · PVG 08:09 · LAX 16:09 · JFK 19:09
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.