aiohttp 支持異步的網絡請求模塊

　　一般在進行網絡數據採集時候咱們會用到requests,urllib等模塊,可是這些模塊在使用中並不支持異步,因此今天咱們介紹一個支持異步網絡請求的模塊aiohttp.

首先咱們使用flask簡單的搭一個服務器:

from flask import Flask

app = Flask(__name__)

@app.route('/xiaohua')
def xiaohua():
    return 'i am xiaohua'

@app.route('/xiaohuang')
def xiaohuang():
    return 'i am xiaohuang'

@app.route('/xiaoming')
def xiaoming():
    return 'i am xiaoming'


if __name__ == '__main__':
    app.run()　　

下面是經過aiohttp進行的異步訪問 :

import aiohttp
import time
import asyncio

async def get_page(url):               #加async關鍵字,使函數返回一個協程對象
    async with aiohttp.ClientSession() as session:  #建立ClientSession對象
        async with await session.get(url=url) as response:  #將url傳入get方法並賦值到response
            page_text = await response.text()    #遇IO手動掛起
            print(page_text)
start = time.time()
urls = [
    'http://127.0.0.1:5000/xiaohua',
    'http://127.0.0.1:5000/xiaoming',
    'http://127.0.0.1:5000/xiaohuang',
    'http://127.0.0.1:5000/xiaohua',
    'http://127.0.0.1:5000/xiaoming',
    'http://127.0.0.1:5000/xiaohuang',
    'http://127.0.0.1:5000/xiaohua',
    'http://127.0.0.1:5000/xiaoming',
    'http://127.0.0.1:5000/xiaohuang',
    
]
tasks = []
loop = asyncio.get_event_loop()    #建立事件循環
for url in urls:
    c = get_page(url)     #接收協程對象
    task = asyncio.ensure_future(c)   #對協程對象進一步封裝成future
    tasks.append(task)
loop.run_until_complete(asyncio.wait(tasks))   #將多個任務對象對應的列表註冊到事件循環中,需使用asyncio.wait進行逐一取值
print('總耗時：',time.time()-start)

結果 :

i am xiaoming
i am xiaoming
i am xiaohuang
i am xiaoming
i am xiaohua
i am xiaohuang
i am xiaohuang
i am xiaohua
i am xiaohua
總耗時： 0.018949031829833984

View Code

　　從結果能夠看出,經過使用aiohttp模塊,訪問實現了異步訪問,大大提升了效率.