Hey Bogdan, first of all thanks a lot for creating this, I really hope it takes off, because the quality looks really nice compared to anything else from the Py ecosystem.
I browsed your website and the documentation for quite a while these past two days, and I must say I like what I see. The documentation is also of very good quality, with concrete examples. If there’s anything to improve, I could only think of the “client” side (sending / creating messages), for instance: best practices, “client” only actors and the likes.
When it comes to making it easier for new comers to decide on using it, it would be really cool to have some sort of use cases / success stories, maybe your own?
To help with this, I’d like to present what my first intended use case is, and maybe you could provide some feedback and thoughts as to how best deal with it. This projects consists mainly of two sort of tasks: the ones that do a rough analysis of existing data and sniff out some potential leads, and the others look very deep into those potential leads. All of the processing is distributed, running right now on a cluster of custom built AMD Ryzen boxes (each box with one CPU, 8x cores, 16x SMT units).
The tasks sniffing out leads do the exact same work on various data partitions. The throughput is roughly 10k jobs per 24h, every task takes several minutes to process. These tasks are created all at once for a given set of data. In case a lead is found, the data about that lead is persisted in a DB and another (auxiliary) task is created for the purpose of creating the tasks for analyzing that lead. Depending on the lead, 10’s to 1000’s of tasks are submitted. Each of them takes up to 10 minutes to process (average ~3 minutes). The jobs are not limited by IO, RAM or other factors, it’s all about CPU processing power.
- all tasks must be processed, this is paramount
- use all available systems at their fullest, maximum processing power efficiency. The customer plans to add AWS nodes into the mix
- be able to run all types of tasks (sniffing tasks, lead analyzes tasks and auxiliary tasks) on the same node, with sniffing jobs having the highest priority. Three priorities would do it right now
How would you deal with this using dramatiq? How many worker processes would you start on every box? Would you use threads at all, or just processes given that the tasks take minutes? I thought about 16 processes per box (1 for every SMT).
Obviously, we will have to experiment with these things on our own, but I’m really looking forward to your thought and suggestions.
Edit: using Ubuntu 18.04 LTS, Python 3.6, RabbitMQ