3種方式實現python多線程併發處理

標籤： python奇淫技巧

---

最優線程數

• Ncpu=CPU的數量
• Ucpu=目標CPU使用率
• W/C=等待時間與計算時間的比率

爲保持處理器達到指望的使用率，最優的線程池的大小等於
$$Nthreads=Ncpu*Ucpu*（1+W/C$$

• cpu密集型任務，即$W<<C$，則$W/C≈0$，則$Nthreads=Ncpu*Ucpu$

若是但願CPU利用率爲100%，則$Nthreads=Ncpu$

• IO密集型任務，即系統大部分時間在跟I/O交互，而這個時間線程不會佔用CPU來處理，即在這個時間範圍內，能夠由其餘線程來使用CPU，於是能夠多配置一些線程。
• 混合型任務，兩者都佔有必定的時間

線城池

對於任務數量不斷增長的程序，每有一個任務就生成一個線程，最終會致使線程數量的失控。對於任務數量不端增長的程序，固定線程數量的線程池是必要的。

方法一：使用threadpool模塊

threadpool是一個比較老的模塊了，支持py2 和 py3 。

import threadpool
import time

def sayhello (a):
    print("hello: "+a)
    time.sleep(2)

def main():
    global result
    seed=["a","b","c"]
    start=time.time()
    task_pool=threadpool.ThreadPool(5)
    requests=threadpool.makeRequests(sayhello,seed)
    for req in requests:
        task_pool.putRequest(req)
    task_pool.wait()
    end=time.time()
    time_m = end-start
    print("time: "+str(time_m))
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))

if __name__ == '__main__':
    main(

方法二：使用concurrent.futures模塊

from concurrent.futures import ThreadPoolExecutor
import time

import time
from concurrent.futures import ThreadPoolExecutor, wait, as_completed

ll = []
def sayhello(a):
    print("hello: "+a)
    ll.append(a)
    time.sleep(0.8)

def main():
    seed=["a","b","c","e","f","g","h"]
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))
    start2=time.time()
    with ThreadPoolExecutor(2) as executor:
        for each in seed:
            executor.submit(sayhello,each)
    end2=time.time()
    print("time2: "+str(end2-start2))

def main2():
    seed = ["a", "b", "c", "e", "f", "g", "h"]
    executor = ThreadPoolExecutor(max_workers=10)
    f_list = []
    for each in seed:
        future = executor.submit(sayhello, each)
        f_list.append(future)
    wait(f_list)
    print(ll)
    print('主線程結束')


def main3():
    seed = ["a", "b", "c", "e", "f", "g", "h"]
    with ThreadPoolExecutor(max_workers=2) as executor:
        f_list = []
        for each in seed:
            future = executor.submit(sayhello, each)
            f_list.append(future)
        wait(f_list,return_when='ALL_COMPLETED')
        print(ll)
        print('主線程結束')

if __name__ == '__main__':
    main3()

方法三：使用vthread模塊

參考：https://pypi.org/project/vthr...

demo1

import vthread
 
@vthread.pool(6)
def some(a,b,c):
    import time;time.sleep(1)
    print(a+b+c)
 
for i in range(10):
    some(i,i,i)

demo2:分組線程池

import vthread
pool_1 = vthread.pool(5,gqueue=1) # open a threadpool with 5 threads named 1
pool_2 = vthread.pool(2,gqueue=2) # open a threadpool with 2 threads named 2

@pool_1
def foolfunc1(num):
    time.sleep(1)
    print(f"foolstring1, test3 foolnumb1:{num}")

@pool_2
def foolfunc2(num):
    time.sleep(1)
    print(f"foolstring2, test3 foolnumb2:{num}")

@pool_2
def foolfunc3(num):
    time.sleep(1)
    print(f"foolstring3, test3 foolnumb3:{num}")

for i in range(10): foolfunc1(i)
for i in range(4): foolfunc2(i)
for i in range(2): foolfunc3(i)