python多進程詳解和協程

一、因爲python多線程適合於多IO操做，但不適合於cpu計算型工做，這時候能夠經過多進程實現。python多進程簡單實用

# 多進程，能夠cpu保持一致，python多線程適合多io.對於高cpu的能夠經過多進程實現。
import  multiprocessing
import time

def run(name):
    print(" %s process is running "%(name))
    time.sleep(2)



if  __name__ == '__main__':
    for i in range(1, 10):
        process = multiprocessing.Process(target=run, args=(i,))
        process.start()

二、python多進程。在操做系統中全部的進程都是有根進程（進程號0來建立的）。python獲取主進程號和子進程號

# 多進程，能夠cpu保持一致，python多線程適合多io.對於高cpu的能夠經過多進程實現。
import  multiprocessing
import time
import os

def run(name):
    print(" %s process is running "%(name))
    time.sleep(2)

def info(name):
    print("%s"%(name))
    print("當前進程號：%s"%(os.getpid()))
    print("父進程號：%s"%(os.getppid()))





if  __name__ == '__main__':
    info("parent process")
    process = multiprocessing.Process(target=info,args=("child process",))
    process.start()

E:\Users\xiajinqi\PycharmProjects\Atm\venv\Scripts\python.exe E:/Users/xiajinqi/PycharmProjects/Atm/333.py
parent process
當前進程號：23420
父進程號：2372
child process
當前進程號：23468
父進程號：23420

Process finished with exit code 0

 

三、queue 實現進程通信，進程Q要實現通信。必須指明爲進程Q。原理：在建立子進程時候，傳遞一個Q，實際至關於複製了一個新的Q給子進程。子進程向新的Q輸入數據時候。進程Q經過一個反系列化實現將新的數據同步到舊的Q(即父進程Q)。兩個程序之間內存是隔離的。Q也是相互獨立的。

# 多進程，能夠cpu保持一致，python多線程適合多io.對於高cpu的能夠經過多進程實現。
from   multiprocessing import Process,Queue # 只名爲 進程Q
import time
import os

qu = Queue()


def test(qu):
    qu.put(["克隆Q"])




if  __name__ == '__main__':

    process = Process(target=test,args=(qu,))
    process.start()
    print(qu.get())

E:\Users\xiajinqi\PycharmProjects\Atm\venv\Scripts\python.exe E:/Users/xiajinqi/PycharmProjects/Atm/333.py
['克隆Q']

Process finished with exit code 0

 四、Pipe()實現管實現進程通信。

# Pipe()
from   multiprocessing import Process,Queue,Pipe # 只名爲 進程Q
import time
import os


def test(child_conn):
    child_conn.send(['hello world'])
    child_conn.send(['hello world'])
    child_conn.close()



if  __name__ == '__main__':
    parent_conn,child_conn = Pipe()
    process = Process(target=test,args=(child_conn,))
    process.start()
    message1 = parent_conn.recv()
    message2 = parent_conn.recv()
    print(message1)
    print(message2)

E:\Users\xiajinqi\PycharmProjects\Atm\venv\Scripts\python.exe E:/Users/xiajinqi/PycharmProjects/Atm/333.py
['hello world']
['hello world']

Process finished with exit code 0

　五、Manager 實現進程通信，實現修改同一個數據。不須要鎖

# Mangager
from   multiprocessing import Process,Queue,Pipe,Manager # 只名爲 進程Q
import time
import os


def test(d,l):
    d[os.getpid()] = os.getpid()
    l.append( os.getpid())
    print(d)
    print(l)



if  __name__ == '__main__':
    with Manager()  as  manager:
        d = manager.dict()
        l = manager.list(range(10))
        p_list = []
        for i in range(0, 10):
            process = Process(target=test, args=(d, l))
            process.start()
            p_list.append(process)

        for process in p_list:
            process.join()



E:\Users\xiajinqi\PycharmProjects\Atm\venv\Scripts\python.exe E:/Users/xiajinqi/PycharmProjects/Atm/333.py
{17876: 17876}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876]
{11504: 11504, 17876: 17876}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504]
{11504: 11504, 26212: 26212, 17876: 17876}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504, 26212]
{11504: 11504, 26212: 26212, 24244: 24244, 17876: 17876, 19900: 19900}
{11504: 11504, 26212: 26212, 24244: 24244, 17876: 17876, 19900: 19900}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504, 26212, 24244, 19900]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504, 26212, 24244, 19900]
{11504: 11504, 11824: 11824, 26212: 26212, 24244: 24244, 19900: 19900, 17876: 17876}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504, 26212, 24244, 19900, 11824]
{11504: 11504, 11824: 11824, 26212: 26212, 27592: 27592, 24244: 24244, 19900: 19900, 17876: 17876}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504, 26212, 24244, 19900, 11824, 27592]
{11504: 11504, 11824: 11824, 26212: 26212, 27592: 27592, 24244: 24244, 19900: 19900, 17876: 17876, 3248: 3248}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504, 26212, 24244, 19900, 11824, 27592, 3248]
{11504: 11504, 11824: 11824, 26212: 26212, 26312: 26312, 27592: 27592, 24244: 24244, 19900: 19900, 17876: 17876, 3248: 3248}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504, 26212, 24244, 19900, 11824, 27592, 3248, 26312]
{11504: 11504, 11824: 11824, 26212: 26212, 14320: 14320, 26312: 26312, 27592: 27592, 24244: 24244, 19900: 19900, 17876: 17876, 3248: 3248}
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 17876, 11504, 26212, 24244, 19900, 11824, 27592, 3248, 26312, 14320]

Process finished with exit code 0

六、進程池pool的使用，每啓動一個進程就會消耗必定資源。使用pool能夠避免機器崩潰發生。

pool = Pool(5) # 進程池5個，啓動100個進程，只有在進程池中的5個纔會執行。而不是同時執行100個，其餘掛起狀態。下面每次會打印5個

# Mangager
from   multiprocessing import Process,Queue,Pipe,Manager,Pool,freeze_support # 只名爲 進程Q
import time
import os


def test(test):
    time.sleep(1)
    print("test")

def bar(arg):
    print("子進程執行完之後，回調後開始執行",arg)


if  __name__ == '__main__':
    freeze_support()
    pool = Pool(5) # 進程池5個，啓動100個進程，只有在進程池中的5個纔會執行。而不是同時執行100個，其餘掛起狀態，避免機器崩潰
    for i in range(0, 10):
        # pool.apply()串行
        pool.apply_async(func=test,args=(test,),callback=bar)  #並行

    pool.close()
    pool.join()





E:\Users\xiajinqi\PycharmProjects\Atm\venv\Scripts\python.exe E:/Users/xiajinqi/PycharmProjects/Atm/333.py
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None
test
子進程執行完之後，回調後開始執行 None

Process finished with exit code 0

 

七、__name__ = "__main__" :手工執行時候。模塊名爲main。其餘調用時候則是模塊名。

八、協程本質上是單線程，是一種輕量級線程，協程有本身的寄存器上下文和堆棧。切換時候，將寄存器上下文和堆棧保存到其餘地方。切換回來時候。恢復以前保存的寄存器上下文內容。協程能保留上一次執行時候的狀態。

協程的好處：

  無需線程上下文切換的開銷（cpu）

  無需原子鎖定以及同步的開銷

 方便切換控制流，簡化編程模型

 高併發+高拓展性+低成本。一個CPU支持上萬的攜程不是問題。

缺點：

   沒法利用多核資源。由於協程本質就是一個單線程。須要和進程配合才能運行在多CPU上。

  進行阻塞操做時候會阻塞整個程序。。

 

#yiled實現的協程
# Mangager
from   multiprocessing import Process,Queue,Pipe,Manager,Pool,freeze_support # 只名爲 進程Q
import time
import os

def  consumer(name) :
    print("test")
    while True:
        new_baozi =yield
        print("%s is eating %s"%(name,new_baozi))


def producer():
    # 函數中若是有yiled，系統斷定爲生成器，須要用next纔會執行
    r = con1.__next__()
    r = con2.__next__()

    n= 0
    while n < 5:
        n+1
        time.sleep(2)
        #send 喚醒生成器。而且給他傳遞一個值
        con1.send(n)
        con2.send(n)


if __name__ =="__main__":
    con1 = consumer("test1")
    con2 = consumer("test1")
    p = producer()

八、手動切換協greenlet

# Mangager
from   multiprocessing import Process,Queue,Pipe,Manager,Pool,freeze_support # 只名爲 進程Q
from greenlet import greenlet
import time
import os

def test1():
    print("test1")
    t2.switch()
    time.sleep(1) #爲iO佔用時間
    print("test11")
    t2.switch()



def test2():
    print("tets2")
    t1.switch()
    time.sleep(1)
    print("test22")



t1 = greenlet(test1)
t2 = greenlet(test2)
t1.switch()

九、gevent 自動切換（前面的封裝）gevent 每一個1秒切換

# Mangager
from   multiprocessing import Process,Queue,Pipe,Manager,Pool,freeze_support # 只名爲 進程Q
import gevent
import time
import os

def test1():
    print("test1")
    gevent.sleep(1)
    print("test11")




def test2():
    print("tets2")
    gevent.sleep(1)
    print("test22")

#啓動攜程加入隊列
gevent.joinall([gevent.spawn(test1),agevent.spawn(test2)])

# Mangager
from   multiprocessing import Process,Queue,Pipe,Manager,Pool,freeze_support # 只名爲 進程Q
import gevent
import time
import os

def test1():
    print("test1")
    gevent.sleep(3)
    print("test11")




def test2():
    print("tets2")
    gevent.sleep(1)
    print("test22")

#啓動攜程加入隊列
gevent.joinall([gevent.spawn(test1),gevent.spawn(test2)])

E:\Users\xiajinqi\PycharmProjects\Atm\venv\Scripts\python.exe E:/Users/xiajinqi/PycharmProjects/Atm/333.py
test1
tets2
test22
test11

Process finished with exit code 0

10 、  gevent 實際應用時間測試。爬多個網頁

# Mangager
from   multiprocessing import Process,Queue,Pipe,Manager,Pool,freeze_support # 只名爲 進程Q
import gevent
import time
import os
from urllib  import request #
from gevent  import monkey # 這個補丁纔可讓gevent捕獲IO

def f(url):
    res = request.urlopen(url)

url = [
    "https://www.baidu.com",
    "https://www.yylending.com",
    "https://www.yyfax.com"
]

start_time = time.time()
gevent.joinall([gevent.spawn(f,"https://www.baidu.com"),gevent.spawn(f,"https://www.yylending.com"),gevent.spawn(f,"https://www.yyfax.com")])
print("並行",(time.time()-start_time))


start_time = time.time()
for i in url:
    f(i)
print("串行",(time.time()-start_time))

E:\Users\xiajinqi\PycharmProjects\Atm\venv\Scripts\python.exe E:/Users/xiajinqi/PycharmProjects/Atm/333.py
並行 1.3772048950195312
串行 0.9304625988006592

Process finished with exit code 0