多進程進程間通信multiprocessing

#!/usr/bin/env python
# Author:Zhangmingda
import queue,threading
from multiprocessing import Process,Queue
def f(q):
    q.put([234,None,'Hello'])
if __name__ == '__main__':
    q = queue.Queue() #使用線程隊列運行報錯TypeError: can't pickle _thread.lock objects
    p = Process(target=f,args=(q,)) #啓動一個子進程
    # p = threading.Thread(target=f,) #啓動線程直接共享主進程內存
    p.start()
    print(q.get())
    p.join()

線程報錯

C:\Users\Administrator\Desktop\Python3_study\venv\Scripts\python.exe C:/Users/Administrator/Desktop/Python3_study/day10/進程間通訊？.py
Traceback (most recent call last):
  File "C:/Users/Administrator/Desktop/Python3_study/day10/進程間通訊？.py", line 13, in <module>
    p.start()
  File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python36\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python36\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Python36\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

運行報錯效果

#!/usr/bin/env python
# Author:Zhangmingda
import queue,threading
from multiprocessing import Process,Queue
def f(q):
    q.put([234,None,'Hello'])
if __name__ == '__main__':
    # q = queue.Queue() #使用線程隊列運行報錯TypeError: can't pickle _thread.lock objects
    q = Queue() #進程隊列
    p = Process(target=f,args=(q,)) #啓動一個子進程
    # p = threading.Thread(target=f,) #啓動線程直接共享主進程內存
    p.start()
    print(q.get())
    p.join()

進程間通信

進程間通信原理：本例中等於將父進程中的q克隆了一份給子進程，子進程進行了序列化，在子進程中執行後又反序列化給父進程。因此在父進程中可以獲得子進程對父進程中數據的修改結果。

C:\Users\Administrator\Desktop\Python3_study\venv\Scripts\python.exe C:/Users/Administrator/Desktop/Python3_study/day10/進程間通訊？.py
[234, None, 'Hello']

Process finished with exit code 0

運行效果

 

#!/usr/bin/env python
# Author:Zhangmingda

from multiprocessing import Process,Manager
import os

def f(d,l):
    d[os.getpid()] = os.getpid() #對傳入的字典進行賦值
    l.append(os.getpid()) #向列表中增長數據

if __name__ == '__main__':
    with Manager() as manager:
        d = manager.dict()
        l = manager.list(range(5))
        print('初始化一個進程間能夠共享的字典和列表：',d,l)
        p_list = [] #準備存儲進程實例
        for i in range(10): #準備啓動10個進程
            p = Process(target=f,args=(d,l))
            p.start()
            print('啓動第%d個進程'%(i+1))
            p_list.append(p)
        for res in p_list:
            res.join() #等待全部進程運行結束
        print(d,l)

多進程之間共享數據(效果相似一個進程之間多線程)

C:\Users\Administrator\Desktop\Python3_study\venv\Scripts\python.exe C:/Users/Administrator/Desktop/Python3_study/day10/進程間數據共享manager.py
初始化一個進程間能夠共享的字典和列表： {} [0, 1, 2, 3, 4]
啓動第1個進程
啓動第2個進程
啓動第3個進程
啓動第4個進程
啓動第5個進程
啓動第6個進程
啓動第7個進程
啓動第8個進程
啓動第9個進程
啓動第10個進程
{6632: 6632, 5300: 5300, 6044: 6044, 2212: 2212, 3828: 3828, 5184: 5184, 5412: 5412, 7000: 7000, 5768: 5768, 5784: 5784} [0, 1, 2, 3, 4, 6632, 5300, 6044, 2212, 3828, 5184, 5412, 7000, 5768, 5784]

Process finished with exit code 0

運行效果

 (實際進程間仍是copy了一樣的數據最後彙總，Manager()本身內部有鎖)

#!/usr/bin/env python
# Author:Zhangmingda
from multiprocessing import Pool
import time,os

def Foo(i):
    time.sleep(1)
    print('in process',os.getpid())
    return i + 100 #return 的值能夠被當作參數傳給回調函數利用

def Bar(arg):
    print('-->exec done:',arg,os.getpid())

if __name__ == '__main__':
    pool = Pool(processes=3) #限制能夠同時運行的進程數量
    print('主進程ID：',os.getpid())
    for i in range(10):
        # pool.apply(func=Foo,args=(i,)) #串行
        # pool.apply_async(func=Foo,args=(i,)) #並行
        pool.apply_async(func=Foo,args=(i,),callback=Bar) #並行+回調主進程運行
    print('END。。。。。')
    pool.close()
    pool.join() #等待全部進程運行結束，不然會直接結束運行。

進程池 Pool+回調函數

C:\Users\Administrator\Desktop\Python3_study\venv\Scripts\python.exe C:/Users/Administrator/Desktop/Python3_study/day10/進程池(限制同時運行的進程數量).py
主進程ID： 5508
END。。。。。
in process 6856
-->exec done: 100 5508
in process 6620
-->exec done: 101 5508
in process 7988
-->exec done: 102 5508
in process 6856
-->exec done: 103 5508
in process 6620
-->exec done: 104 5508
in process 7988
-->exec done: 105 5508
in process 6856
-->exec done: 106 5508
in process 6620
-->exec done: 107 5508
in process 7988
-->exec done: 108 5508
in process 6856
-->exec done: 109 5508

Process finished with exit code 0

運行效果