Inter-process communication with queues

Let's now see how to communicate between processes using a queue. This example is very very similar to the one for threads:

# comm_queue_proc.py
import multiprocessing

SENTINEL = 'STOP'

def producer(q, n):
a, b = 0, 1
while a <= n:
q.put(a)
a, b = b, a + b
q.put(SENTINEL)

def consumer(q):
while True:
num = q.get()
if num == SENTINEL:
break
print(f'Got number {num}')

q = multiprocessing.Queue()
cns = multiprocessing.Process(target=consumer, args=(q, ))
prd = multiprocessing.Process(target=producer, args=(q, 35))
cns.start()
prd.start()

As you can see, in this case, we have to use a queue that is an instance of multiprocessing.Queue, which doesn't expose a task_done method. However, because of the way this queue is designed, it automatically joins the main thread, therefore we only need to start the two processes and all will work. The output of this example is the same as the one before.

When it comes to IPC, be careful. Objects are pickled when they enter the queue, so IDs get lost, and there are a few other subtle things to take care of. This is why in this example I can no longer use an object as a sentinel, and compare using is, like I did in the multi-threaded version. That sentinel object would be pickled in the queue (because this time the Queue comes from multiprocessing and not from queue like before), and would assume a new ID after unpickling, failing to compare correctly. The string "STOP" in this case does the trick, and it will be up to you to find a suitable value for a sentinel, which needs to be something that could never clash with any of the items that could be in the same queue. I leave it up to you to refer to the documentation, and learn as much as you can on this topic.

Queues aren't the only way to communicate between processes. You can also use pipes (multiprocessing.Pipe), which provide a connection (as in, a pipe, clearly) from one process to another, and vice versa. You can find plenty of examples in the documentation; they aren't that different from what we've seen here.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.137.161.222