Python 下载今日头条的文章（不是爬虫），无法解码，问题到底出在哪儿？

python requests 能成功获取到文章数据，用 charles 抓包也可以看到 requests.get 是成本获取到了文章数据，但是在本地解码的时候，无论是用 response.text,还是 response.content.decode，还是直接把 respons 接收到的数据以二进制写入文本，要么是乱码，要么是报错。（ charles 抓包里查看是正常的数据）

respose 解码时试过 gbk,utf-8,iso ； decode 参数试过 ignore, replace 也没用。

谷歌百度也搜不到可用的解决方法。

Supplement 1 · Feb 20, 2021

import requests

verify=False

headers = '''
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: zh-CN,zh;q=0.9,zh-TW;q=0.8,en;q=0.7
cache-control: no-cache
cookie: tt_webid=6930433366274721287; csrftoken=50dc3846aa3e01cf8e7cc3cacd3cb664; tt_webid=6930433366274721287; _tea_utm_cache_2256={%22utm_source%22:%22copy_link%22%2C%22utm_medium%22:%22toutiao_ios%22%2C%22utm_campaign%22:%22client_share%22}; s_v_web_id=verify_kld3dq66_Wvi5ATKw_nvWF_4CHH_9aBi_wzhdhfONq7HB; _ga=GA1.2.1782351744.1613793452; _gid=GA1.2.504008286.1613793452; csrftoken=50dc3846aa3e01cf8e7cc3cacd3cb664; __ac_nonce=06030995f005059b3fa30; __ac_signature=_02B4Z6wo00f01bNanKAAAIDB2pGskn.jLh2zfpgAAAznlJs4jAp.SxXUbLBYvi0aqQmu7OBPOm6M5vvKPJGKjxqNVGt0Cc2hHsiQ0bkQbsLQv.C4ZZyAHsoKG9tZtyfcvupCp.4rC56vmyGcb8; MONITOR_WEB_ID=72226c84-9c57-4b0e-931c-6a99c225be9d; tt_scid=w3cIoL.syHtp3f6lhdBFt08cmgEdko8ObFMxYA7uMNdRp-aVs1w0x852q6pFS7bybc1a
dnt: 1
pragma: no-cache
sec-ch-ua: "Chromium";v="88", "Google Chrome";v="88", ";Not A Brand";v="99"
sec-ch-ua-mobile: ?0
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240
'''

def todict(headers):
    return dict([ l.split(': ') for l in headers.split('\n') if l])


def downloadImages(url,headers):

    headers = todict(headers)
    res = requests.get(url=url,headers=headers,verify=False).content.decode('utf-8','ignore')

    print(res)



if __name__ == '__main__':


    url = 'https://www.toutiao.com/a6931116005670371853/'

    downloadImages(url,headers)

Charles

Python

文章

解码

12 replies • 2021-02-20 21:39:56 +08:00