Broadcasting message/task to all workers


#1

Is it possible to send a message/task to all the workers ?

My case use : I have a pool of long running workers.
My tasks code base is changing over time (adding new tasks, fixing existing tasks).
I would like to tell the workers to reload themselves with the new code base.

Doing a git pull or a docker pull is easy. A simple exit on each dramatiq-cli should be enough.
But how can I talk to all my dramatiq instances (or all my dramatiq process or threads/workers) ?


#2

I already have an answer but I don’t like it :

Let’s say I have 10 dramatiq instances, running 4 workers each (processes/threads).
I could flood the broker with 2 messages per worker for a total of 80 messages (2104).

Each message will :

  • look for the worker version tag
  • if the version need to be upgraded :
    • the task will fetch the new code base (with git or docker)
    • update the version tag
    • gracefully quit the dramatiq instance
  • if the version is already upgraded :
    • wait to avoid all the messages to be consumed by the same worker (twice the duration of my longest task)

Seems overkill :confused: Anything smarter ?


#3

This way seems better :
I will have a queue per dramatiq instance handling the upgrade process.
Each dramatiq instance will have a special actor listening to its own queue (based on a unique id like hostname or IP or MAC address).
Using get_declared_queues() I should be able to trigger an upgrade on all my instances.


#4

I may be missing something, but why do you have to do this through Dramatiq itself? IMO, better approaches would be to either:

a) use something like ansible to manage your inventory of servers and run a command to ssh into your servers, upgrade the code and send a SIGHUP to the master dramatiq process, or
b) use immutable infrastructure and simply spin up new vms/isntances on each release.

A third option might be to have a cron job on the worker boxes that checks if a new tag has been released and does the pull+SIGHUP by itself. I don’t think I would recommend this approach though.


#5

I chose b) generating a new docker image on each new version.
Thanks.