Passing config.ini files to Dramatiq’s workers


#1

Hi!

I’m running Dramatiq as part of a Pyramid appserver, where tasks are scheduled from within a view function and executed by workers (actors).

Messages and results are exchanged via Redis.

Now the Redis configuration is stored in different .ini files, one for development, one for beta, one for production. When the Pyramid appserver starts up, the .ini file is passed along and used:

./bin/gunicorn --paste appsrv-localhost.ini

The appserver reads the configuration from that .ini file and connects to Redis accordingly.

The Dramatiq actors are implemented in srv/dramatiq/actors.py, and view functions can call them just fine. It all works until here.

Next I want to start the broker to handle the messages and execute the tasks. Thus, here too I need to read the .ini file and set up Dramatiq’s Redis access. To do that, I added another file srv/dramatiq/broker.py

_cp = configparser.ConfigParser()                                                                             
_cp.read(os.path.join(os.path.split(__file__)[0], "../../appsrv-localhost.ini"))                                 
_redis_url = _cp.get("app:main", "redis.url")                                                                 
_redis_namespace = _cp.get("app:main", "redis.namespace")                                                     
_redis_broker = RedisBroker(url=_redis_url, namespace=_redis_namespace)                                       
dramatiq.set_broker(_redis_broker)                                                                            

from .actors import *                                                                                         

and then call it like so:

./bin/dramatiq --processes 1 --threads 2 --watch ./srv/dramatiq/ --verbose srv.dramatiq.broker

The problem is the configuration file: how can I pass it to Dramatiq so that options become available in this context? Or what is the recommended approach here?

Thanks!


#2

I’m not sure I 100% follow. Are you trying to dynamically figure out which config file to load?

If so, then you should use environment variables for that.

_cp.read(os.getenv("CONFIG_FILENAME"))
CONFIG_FILENAME=appsrv-prod.ini ./bin/dramatiq --processes 1 --threads 2 --watch ./srv/dramatiq/ --verbose srv.dramatiq.broker

#3

Yes, something along those lines. Is there a way to pass the config file as a command-line argument? For example, alembic uses an -x parameter that allows to pass command-line arguments through to the user code.

Other than that, is the above described approach the recommended one to parameterize demonizing Dramatiq workers?


#4

Environment variables are the way to go here. I don’t think we’d gain much by the added complexity of an -x flag like alembic has.

I personally wouldn’t turn on verbose logging unless debugging something. It’s not going to hurt anything, but it is going to be quite verbose. :smiley:


#5

Thanks, I’ll stick with the env variable then. As for the --verbose — I find it a good way of understanding what happens inside on Dramatiq, helps me to understand it better and maybe dig around the code sometime :wink: