Keeping worker code separate from main application


#1

Hi,

we had a short discussion on a Github issue regarding keeping worker code separate from the main application code because my worker code involves some funky stuff and has heavy dependencies (CUDA, etc.)

If I may refine the question:
(1) Is it possible to keep worker code completely separate, i.e. in a separate repository and only ever run the worker code on remote machines?

(2) as a consequence, is it possible to ask the broker for the names of the workers that joined?

Thanks!
Dave


#2

Regarding the first issue, I’d go with @bogdans solution as proposed on github.

Regarding nr. 2, I’d suggest you build a middleware that implements after_worker_boot and reports the workers status and capabilities to redis / some other database you might be using. You might also want to hijack dramatiqs heartbeat system, although I don’t think there’s an API in dramatiq for that anywhere.

…but I see @bogdan is already typing, I’m sure he’s got a smarter solution :stuck_out_tongue:


#3

You beat me to it! I was going to suggest the exact same things.


#4

Thanks @MoorsTech and @bogdan,

I deduct from your answers is that declaring a worker is a local procedure, i.e. local to the current broker. I’ll have a look at making the middleware solution (actually, I’ll look at the heartbeat mechanism first, maybe that’s less work than manually interacting with Redis).


#5

I should’ve mentioned this in my other post, but heartbeats are an implementation detail (so they’re open to change) and, like @MoorsTech mentioned, there’s really no API for taking advantage of them.

I would just spawn a background thread from your middleware and have that periodically set a key in redis with an expiration (2x the period or something) for the current worker and either keep those keys in a set or iterate over them when necessary.


#6

Is there a link to the original Github issue in question? I’d like to see more information about running the worker on a remote machine.


#7

It initially started with a different question, so it might be difficult to find. Here’s the link: