asyncio 不是用协程吗?我理解协程就应该是一个进程只创建一个线程,由这个线程自行调度协程,而使用 asyncio+aiohttp 时创建了 25 个线程, gevent+requests 创建了 15 个线程,这又是怎么理解?协程不就是为了节省线程切换带来的性能损耗吗?
import random
import asyncio
from aiohttp import ClientSession
async def fetch(url, session):
async with session.get(url) as response:
delay = response.headers.get("DELAY")
date = response.headers.get("DATE")
print("{}:{} with delay {}".format(date, response.url, delay))
return await response.read()
async def bound_fetch(sem, url, session):
# Getter function with semaphore.
async with sem:
await fetch(url, session)
我在Windows上跑,在资源管理器看到创建了25个线程 async def run(r): url = "http://localhost:8080/{}" tasks = [] # create instance of Semaphore sem = asyncio.Semaphore(1000)
# Create client session that will ensure we dont open new connection
# per each request.
async with ClientSession() as session:
for i in range(r):
# pass Semaphore and session to every GET request
task = asyncio.ensure_future(bound_fetch(sem, url.format(i), session))
tasks.append(task)
responses = asyncio.gather(*tasks)
await responses
number = 10000
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(number))
loop.run_until_complete(future)
import random
import asyncio
from aiohttp import ClientSession
async def fetch(url, session):
async with session.get(url) as response:
delay = response.headers.get("DELAY")
date = response.headers.get("DATE")
print("{}:{} with delay {}".format(date, response.url, delay))
return await response.read()
async def bound_fetch(sem, url, session):
# Getter function with semaphore.
async with sem:
await fetch(url, session)
async def run(r):
url = "http://localhost:8080/{}"
tasks = []
# create instance of Semaphore
sem = asyncio.Semaphore(1000)
# Create client session that will ensure we dont open new connection
# per each request.
async with ClientSession() as session:
for i in range(r):
# pass Semaphore and session to every GET request
task = asyncio.ensure_future(bound_fetch(sem, url.format(i), session))
tasks.append(task)
responses = asyncio.gather(*tasks)
await responses
number = 10000
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(number))
loop.run_until_complete(future)
我在Windows的资源管理器看的线程数,刚才中间怎么打断了。。。
gevent:
import gevent.monkey
gevent.monkey.patch_socket()
import gevent
import urllib.request as req
import json
def fetch(pid):
response = req.urlopen('http://localhost:8080/')
result = response.read()
#json_result = json.loads(result)
#datetime = json_result['datetime']
print('Process %s' % (pid))
return result
def asynchronous():
threads = []
for i in range(1,800):
threads.append(gevent.spawn(fetch, i))
gevent.joinall(threads)
print('Asynchronous:')
asynchronous()
gevent打了monkey patch也是创建15线程呀。。
自己结帖,测试数据为:Windows下用asyncio
跑一个run_forever()
要开5个线程,用aiohttp
发起请求最大创建25个线程,使用基于线程池的DNS resolver的gevent发起请求创建15个线程。而Linux(Ubuntu)下用asyncio
跑空的事件循环只用1个线程,用aiohttp
发起请求也只用1个线程,使用基于线程池DNS resolver的gevent创建10个线程,而在设置export GEVENT_RESOLVER=ares
后也是只开一个线程,在Linux下的行为符合我对基于协程的异步IO的理解,而Windows下ayncio
库为什么要创建这么多线程还是不理解,google也搜不到什么信息,望有大神告知。
不对,刚再测了一下aiohttp
发起请求还是要创建5个线程的,但也比Windows下创建少的多。。
1
chy373180 2017-02-21 11:27:52 +08:00
你从哪里看到有 25 个线程?代码贴下?
|
2
wwqgtxx 2017-02-21 12:16:50 +08:00
别的不说,“ gevent+requests 创建了 15 个线程”那只能说明你根本没用对 gevent , monkey patch 肯定没有在代码的第一行就打上
|
5
zl2003cn OP |
6
wwqgtxx 2017-02-21 14:01:31 +08:00
@zl2003cn 不管别的类库打没打,你应该在你程序的第一行(!!!不是在你的 main 函数中,是程序的第一行,第一行,除了注释的第一行!!!)打上 monkey patch
|
7
Zzzzzzzzz 2017-02-21 14:09:21 +08:00
gevent 那个正常, 它的 dns resolver 默认是线程池实现, 可以通过设置 GEVENT_RESOLVER=ares 改成 c-ares 的实现
|
9
wwqgtxx 2017-02-21 14:39:50 +08:00
7 楼的解释是正确的
DNS queries performed through threadpool (default) or through c-ares (enabled via GEVENT_RESOLVER=ares env var). |
11
dant 2017-02-21 19:30:52 +08:00 via Android
线程池招你惹你了(
另外,如果你用的是基于 IOCP 的 ProactorEventLoop ,那么线程数量可能会更多。 最后,线程要用 Process Explorer 看。 |
13
pynix 2017-02-21 22:02:34 +08:00
你使用的可能是个假 aio
|
14
zungmou 2017-02-27 11:12:25 +08:00
不开多线程怎么异步呀?
|