V2EX = way to explore
V2EX 是一个关于分享和探索的地方
Sign Up Now
For Existing Member Sign In
推荐学习书目
› Learn Python the Hard Way
Python Sites
› PyPI - Python Package Index
› http://diveintopython.org/toc/index.html
› Pocoo
值得关注的项目
› PyPy
› Celery
› Jinja2
› Read the Docs
› gevent
› pyenv
› virtualenv
› Stackless Python
› Beautiful Soup
› 结巴中文分词
› Green Unicorn
› Sentry
› Shovel
› Pyflakes
› pytest
Python 编程
› pep8 Checker
Styles
› PEP 8
› Google Python Style Guide
› Code Style from The Hitchhiker's Guide
This topic created in 2581 days ago, the information mentioned may be changed or developed.
最近用 scrapy 框架做爬虫，前几天的数据一直都没问题，这几天爬取的数据明显减少了；应该不是代码的原因，看了下日志，有些 URL 没有抓下来，怎么回事？
2019-04-01 00:00:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yellowpages.com/search?search_terms=Tires&geo_location_terms=Dundalk%2C+MD&page=4> (referer: https://www.yellowpages.com/search?search_terms=Tires&geo_location_terms=Dundalk%2C+MD)
2019-04-01 00:00:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yellowpages.com/search?search_terms=Tires&geo_location_terms=Dundalk%2C+MD&page=3> (referer: https://www.yellowpages.com/search?search_terms=Tires&geo_location_terms=Dundalk%2C+MD)
2019-04-01 00:00:06 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.yellowpages.com/search?search_terms=Tires&geo_location_terms=Dundalk%2C+MD&page=3>
如上，page=3 的就 scraped 下来了，而 page=4 的这个只是 crawled 了并没有 scraped，这是为什么，存在好多这样的情况。
2 replies • 2019-04-07 22:50:53 +08:00
 |
1
dylanhu OP 重点是前几天没什么这种情况，这两天开始数据少了很多
|
Select Language