推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
thekoc
V2EX  ›  Python

Python 多线程爬虫如何优雅的退出?

  •  
  •   thekoc · Nov 14, 2016 · 4679 views
    This topic created in 3495 days ago, the information mentioned may be changed or developed.

    爬虫如果不退出就会一直爬的……但是每次按 ctrl+c 退出的时候似乎都有点问题。爬虫的退出如何正确的实现?

    8 replies    2016-11-24 15:36:16 +08:00
    cdwyd
        1
    cdwyd  
       Nov 14, 2016 via Android
    标记下看看有什么好的解决方法没
    wellsc
        2
    wellsc  
       Nov 14, 2016 via iPhone
    sudo pkill -9 python
    loalj
        3
    loalj  
       Nov 14, 2016
    t = threading.Thread()
    t.daemon = True
    em70
        4
    em70  
       Nov 14, 2016 via iPhone
    每爬一个数据检测指定目录下指定文件里的值, 1 就忽略, 0 就退出,开另一个窗口用 vi 控制这个文件的值
    Thoxvi
        5
    Thoxvi  
       Nov 14, 2016 via Android
    看需求吧…生产者爬到阈值丢一个毒丸到队列消费者就没了
    stamaimer
        6
    stamaimer  
       Nov 14, 2016 via iPhone
    不知道为啥不能优雅的退出
    ryd994
        7
    ryd994  
       Nov 15, 2016
    用 signal 模块,保证信号发到主线程
    设置子线程 daemon=True
    同时,其他线程捕捉 keyboardinterrput , 然后 interrupt_main https://docs.python.org/2/library/thread.html#thread.interrupt_main
    Powered
        8
    Powered  
       Nov 24, 2016 via iPhone
    调度系统…
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   907 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 40ms · UTC 21:45 · PVG 05:45 · LAX 14:45 · JFK 17:45
    ♥ Do have faith in what you're doing.