使用asyncio模塊處理文件報錯

錯誤:ValueError: too many file descriptors in select()

解決方案:

  1. 加併發限制semaphore = asyncio.Semaphore(200),可是隻能針對爬蟲調接口
    semaphore = asyncio.Semaphore(100)
    async with semaphore:
        async with aiohttp.ClientSession() as session:
            async with session.post(url, data=data, headers=header) as res:
                if int(res.status) == 200:
                    text = await res.json(encoding="utf-8")
                    print(text)
                    return text
  2. 出現這個問題是由於千級文件操做,使用ProactorEventLoop,適用於windows io操做過多的狀況,問題解決
    from asyncio import ProactorEventLoop
    asyncio.set_event_loop(ProactorEventLoop())
    loop = asyncio.get_event_loop()
    # 源碼
    if sys.platform == 'win32':
        ProactorEventLoop: Type[AbstractEventLoop]
相關文章
相關標籤/搜索