Laurens Oostwegel (5d12c7b3) at 12 Jan 16:14
Somehow, when one or multiple tasks are sent to the message queue, where the message is also sent to.
The 'regular' message queue has a consumer (celery) that distributes the messages to workers. The second (unwanted) queue does not have a consumer, which means that the number of items in the queue just adds up and doesn't go down. We would need to purge the queue once in a few days, to keep memory low, if we don't change the code.
Laurens Oostwegel (5d12c7b3) at 12 Jan 16:14
Replace Broadcast queue for @task_prerun
decorator and set queue ...
Closes #55
In this MR two (linked) changes are made
rabotnik-obm
instance will result in two queues: rabotnik-obm
, that is used by the any user sending messages to the queue with RabbitMQ. This initial queue sends it onwards to the Celery worker, that uses the rabotnik-obm-queue
to distribute the tasks onto child workers.task_prerun
function is called, that (1) checks if it has been more than 60 seconds since the last check on the database and (2) if so, checks if there are new rules in the database.Finally, this means that the initialize_queues
function in the Rabotnik
class is deprecated & the CLI command -i
is not used anymore.
Closes #55
In this MR two (linked) changes are made
rabotnik-obm
instance will result in two queues: rabotnik-obm
, that is used by the any user sending messages to the queue with RabbitMQ. This initial queue sends it onwards to the Celery worker, that uses the rabotnik-obm-queue
to distribute the tasks onto child workers.task_prerun
function is called, that (1) checks if it has been more than 60 seconds since the last check on the database and (2) if so, checks if there are new rules in the database.Finally, this means that the initialize_queues
function in the Rabotnik
class is deprecated & the CLI command -i
is not used anymore.
Laurens Oostwegel (5d12c7b3) at 12 Jan 16:07
Replace Broadcast queue for @task_prerun
decorator and set queue ...
Closes #55
In this MR two (linked) changes are made
rabotnik-obm
instance will result in two queues: rabotnik-obm
, that is used by the any user sending messages to the queue with RabbitMQ. This initial queue sends it onwards to the Celery worker, that uses the rabotnik-obm-queue
to distribute the tasks onto child workers.task_prerun
function is called, that (1) checks if it has been more than 60 seconds since the last check on the database and (2) if so, checks if there are new rules in the database.Finally, this means that the initialize_queues
function in the Rabotnik
class is deprecated & the CLI command -i
is not used anymore.
It looks fine to me
Laurens Oostwegel (27db6eda) at 12 Jan 15:51
Apply 1 suggestion(s) to 1 file(s)
Laurens Oostwegel (1b426909) at 12 Jan 15:51
Change docstring
There is a unregister
task and there is a register
task. unregister
won't remove the rule from the PostGIS database, but will set its status to DELETED
, therefore we can detect which ones have been registered/unregistered since last check
Laurens Oostwegel (6c609cb7) at 12 Jan 15:47
Timestamps
Reads the Rabotnik Celery environment variables and creates the Celery broker URL
You still don't describe what you do or don't do if the rule exists. Also, if only new rules are added, how can one remove a rule?
Laurens Oostwegel (6bd302da) at 12 Jan 15:32
import
Laurens Oostwegel (1adb44da) at 12 Jan 15:31
Review
Laurens Oostwegel (6b376974) at 12 Jan 15:30
Add check_rule_frequency_
Laurens Oostwegel (e2e15ff7) at 12 Jan 15:27
Review
If the rule exists, it doesn't do anything. If the rule does not exist yet it is added to the rules defined in the worker and if it is deleted in the database, it is also deleted here
We can make it an optional environment variable? I hard-coded it because using CLI is not possible anymore... At least I can't find out how