Python multiprocessing without blocking parent process

Question

I am attempting to create a simple application which continuously monitors an inbox, then calls various functions as child processes, after categorising incoming mail.

I would like the parent process to continue it's while loop without waiting for the child process to complete. For example:

def main():
    while 1:
        checkForMail()
        if mail:
            if mail['type'] = 'Type1':
                process1() # 
                '''
                spawn process1, as long as no other process1 process running,
                however it's fine for a process2 to be currently running
                '''
            elif mail['type'] = 'Type2':
                process2()
                '''
                spawn process2, as long as no other process2 process running,
                however it's fine for a process1 to be currently running
                '''

        # Wait a bit, then continue loop regardless of whether child processes have finished or not
        time.sleep(10)
if __name__ == '__main__':
    main()

As commented above, there should never be more than once concurrent child process instance for a function, however processes can run concurrently if they are running different functions.

Is this possible to do with the multiprocessing package?

question from:https://stackoverflow.com/questions/65920110/python-multiprocessing-without-blocking-parent-process

Answer 1

You could use two Queues, one for mails of Type1 and one for mails of Type2 and two Processes again one for mails of Type1 and one for mails of Type2.

Start by creating these Queues. Then create the Processes and give the first Queue to the first Process and the second Queue to the second Process. Both Process objects need a parameter target which is the function that the Process executes. Depending on the logic you probably will need two functions (again one for each type). Inside the function you want something like an infinite loop which takes items from the Queue (i.e. the mails) and then act on them according to your logic. The main function would also consist of an infinite loop where the mails are retrieved and depending on their type they get placed on the correct Queue.

So start the two Processes before the main loop, then start the main loop and the mails should get put on the Queues where they get picked up in the subprocesses.