请问如何爬取 crunchbase 的站点。
目前用的是python的requests，但不知道要在headers中加上什么内容。
推荐学习书目
› Learn Python the Hard Way
Python Sites
› PyPI - Python Package Index
› http://diveintopython.org/toc/index.html
› Pocoo
值得关注的项目
› PyPy
› Celery
› Jinja2
› Read the Docs
› gevent
› pyenv
› virtualenv
› Stackless Python
› Beautiful Soup
› 结巴中文分词
› Green Unicorn
› Sentry
› Shovel
› Pyflakes
› pytest
Python 编程
› pep8 Checker
Styles
› PEP 8
› Google Python Style Guide
› Code Style from The Hitchhiker's Guide
This topic created in 3707 days ago, the information mentioned may be changed or developed.
Supplement 1 · Sep 16, 2016
我的测试代码是：
# encoding=utf-8
import requests
headers = {
'Host':'www.crunchbase.com',
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0",
"Upgrade-Insecure-Requests":"1",
"Accept-Encoding":"gzip, deflate, br",
"Accept-Language":"en-US,en;q=0.5",
"Connection":"keep-alive",
}
url="https://www.crunchbase.com/sitemap/organizations/4a5eb6a619f10f89ea9cccd15c6fa38f-eb176350a92046f2425523ca58f58ce6"
try:
page1 = requests.get(
url=url,headers=headers,
)
print page1.content
print "--------------------"
print page1.headers
print "--------------------"
print page1.request.headers
print "--------------------"
print page1.status_code
except Exception, e:
print e
# encoding=utf-8
import requests
headers = {
'Host':'www.crunchbase.com',
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0",
"Upgrade-Insecure-Requests":"1",
"Accept-Encoding":"gzip, deflate, br",
"Accept-Language":"en-US,en;q=0.5",
"Connection":"keep-alive",
}
url="https://www.crunchbase.com/sitemap/organizations/4a5eb6a619f10f89ea9cccd15c6fa38f-eb176350a92046f2425523ca58f58ce6"
try:
page1 = requests.get(
url=url,headers=headers,
)
print page1.content
print "--------------------"
print page1.headers
print "--------------------"
print page1.request.headers
print "--------------------"
print page1.status_code
except Exception, e:
print e
6 replies • 2016-09-24 14:37:40 +08:00
 |
1
看一下瀏覽器的 header 唄
|
1 |
2
page1.headers 是 Response headers ， page1.request.headers 是 Request headers 。
 Copy Request headers 就可以了。 |
 |
3
谢谢各位指点，我试了很多 headers ，我一直怀疑是 headers 的问题，最后发现是阿里云的 ip 的问题。我换 ip 后就好了。
|
 |
googlefansSep 24, 2016
6
爬后的网站是什么
|
Select Language