6. Design Notes¶

• Must be a simple solution to circumvent to the GIL limitation : i.e. distribute and run tasks in parallel on local CPU cores
• Must be a lightweight solution to Celery : no broker
• Should be portable on main OS like Unix and Windows
• Scheduler is implemented using an asyncio event loop and asyncio queue
• Scheduler queue can be limited on size so as to prevent a memory overshoot at startup
• Workers are spawned using asyncio subprocess and communicate through a pipe (stdin, stdout)
• Pass any pickable object as request and response
• Scheduler monitors workers and restart them when they was killed or they exceeded a memory or timeout threshold
• Provide basic metrics on workers

Technical details in brief:

• scheduler run a producer and N consumers coroutines in an asyncio event loop
• the producer coroutine awaits on queue.put(task) ( if a size limit is set )
• each consumer coroutine wraps a worker subprocess and await data on the stdout pipe