Scaling workers independently based on queues


#1

The way we use celery (with django) now is having multiple queues with different priorities and a bunch of instances running workers that pull out messages only from certain queues. This allow us to scale independently workers for certain messages easily when needed, targeting the queue where there is need of more power. I see that dramatiq can prioritize messages but I don’t see anything that could mimic such a behavior. Is it possible to spawn workers that will pull out messages only from queue X and not from queue Y?


#2

Yup! You can provide the -Q flag to dramatiq.

From dramatiq --help:

# Listen only to the "foo" and "bar" queues.
$ dramatiq some_module -Q foo bar