Windows support using multiprocessing


#1

Hi Bogdan,

First: thanks for creating and maintaining dramatiq! I was really pleased to come across the project.

Regarding issues #75 and #97: would you be open to adding support for Windows in the main command line using Python’s native multiprocessing?

I was able to get the code in dramatiq.main to run on Windows by changing the os-related calls (fork, waitpid, pipe, etc.) to their multiprocessing equivalents. Coupled with Oisins’s suggestion to use portalocker from #75, I have a main executable that works in Windows and Linux and passes the test suite in Linux.

Per your guidelines, I wanted to see if this was a direction you wanted to go before offering a PR.


#2

I think the first step in adding proper Windows support should be to get the project to a point where it builds in CI (i.e. this issue). Without something like that, I would have no confidence that any changes I make or pull in don’t break things for Windows users.

I was able to get the code in dramatiq.__main__ to run on Windows by changing the os-related calls (fork, waitpid, pipe, etc.) to their multiprocessing equivalents.

That’s great! I’d definitely consider merging those changes as long as we can get a Windows CI build going first.

Coupled with Oisins’s suggestion to use portalocker from #75, I have a main executable that works in Windows and Linux and passes the test suite in Linux.

Rather than introducing a new dependency, I would prefer to turn off Prometheus for Windows users by default. Prometheus is going to be turned off by default with version 2 (a long ways off) and the automatic exposition server stuff will be removed (i.e. there won’t be a need for the file locking stuff) as it’s overly complex.

Would you be willing to take a stab at #78 first?


#3

Sure. I was able to build libmemcached and pylibmc and have a build that executes pytest up until the missing fcntl failure mentioned in #75. I’ll send a PR shortly.

EDIT: I saw that you dropped this already; disregard.
To be able to execute pytest, I had to remove the pinning for coverage (<4.4 ) because pytest-cov (now 2.6.0) requires coverage >=4.4. To get a working dependency set, pytest-cov will need to be pinned to an older version or coverage will need to be newer (possibly coupled with this workaround?). I opted for the latter because I don’t know what version of pytest-cov you were using.