rabotnik merge requestshttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests2024-01-12T16:14:19+01:00https://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/92Resolve "[Bug] Sending a task opens a new queue, where the messages are also ...2024-01-12T16:14:19+01:00Laurens OostwegelResolve "[Bug] Sending a task opens a new queue, where the messages are also sent to"Closes #55
In this MR two (linked) changes are made
- Each Rabotnik instance is reduced to two default queues. For instance, adding a `rabotnik-obm` instance will result in two queues: `rabotnik-obm`, that is used by the any user send...Closes #55
In this MR two (linked) changes are made
- Each Rabotnik instance is reduced to two default queues. For instance, adding a `rabotnik-obm` instance will result in two queues: `rabotnik-obm`, that is used by the any user sending messages to the queue with RabbitMQ. This initial queue sends it onwards to the Celery worker, that uses the `rabotnik-obm-queue` to distribute the tasks onto child workers.
- The Broadcasting queue is therefore removed. This queue was used to send a message to each worker each X seconds, to check if there are any new rules in the rule databsae. This didn't work correctly because only one of the workers received the message. Instead now, before each task, a `task_prerun` function is called, that (1) checks if it has been more than 60 seconds since the last check on the database and (2) if so, checks if there are new rules in the database.
Finally, this means that the `initialize_queues` function in the `Rabotnik` class is deprecated & the CLI command `-i` is not used anymore.
\approve @tara @dsLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/91Resolve "[Bug] Sending a task opens a new queue, where the messages are also ...2023-10-12T16:56:38+02:00Laurens OostwegelResolve "[Bug] Sending a task opens a new queue, where the messages are also sent to"Closes #55
\\approve @tara @ds
This is the proposed system:
![Untitled_Diagram.drawio](/uploads/59d960cf10a6e5be0fb929f09a27c598/Untitled_Diagram.drawio.png)Closes #55
\\approve @tara @ds
This is the proposed system:
![Untitled_Diagram.drawio](/uploads/59d960cf10a6e5be0fb929f09a27c598/Untitled_Diagram.drawio.png)Laurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/90Resolve "The connection to databases should be hashed"2023-09-25T15:49:43+02:00Laurens OostwegelResolve "The connection to databases should be hashed"Closes #52
\approve @ds @taraCloses #52
\approve @ds @taraLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/89Resolve "[Hotfix] 'rule_definition' should be only 'definition'"2023-09-22T11:17:45+02:00Laurens OostwegelResolve "[Hotfix] 'rule_definition' should be only 'definition'"Closes #54
\\approve @tara @dsCloses #54
\\approve @tara @dsLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/88Resolve "Create CLI"2023-09-22T12:34:14+02:00Laurens OostwegelResolve "Create CLI"Closes #53
\\approve @ds @taraCloses #53
\\approve @ds @taraLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/87Resolve "Update Rabotnik to work with new rule-lib"2023-09-20T14:21:52+02:00Laurens OostwegelResolve "Update Rabotnik to work with new rule-lib"Closes #51
This MR removes a whole lot of code from the 'old' rabotnik. The `storages` we do not need anymore, as we can replace this with our `database-lib`.
The new `Rabotnik` has three main rule types:
* Rules to get input data (fr...Closes #51
This MR removes a whole lot of code from the 'old' rabotnik. The `storages` we do not need anymore, as we can replace this with our `database-lib`.
The new `Rabotnik` has three main rule types:
* Rules to get input data (from different data sources for example)
* Rules to process input data
* Rules to upsert data (after processing)
Each of these should have a register and unregister function. Also should there be a `run` function, that runs all rules in order.
\\approve @tara @dsLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/86Draft: Resolve "Implement VHOSTS"2023-02-14T16:56:21+01:00Laurens OostwegelDraft: Resolve "Implement VHOSTS"Closes #50
\approve @ds @gislarsCloses #50
\approve @ds @gislarsLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/85Draft: Resolve "Create message chain in RabbitMQ"2022-12-22T13:18:34+01:00Laurens OostwegelDraft: Resolve "Create message chain in RabbitMQ"Closes #49Closes #49Laurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/84Resolve "Add priority queues"2022-09-14T16:39:54+02:00Laurens OostwegelResolve "Add priority queues"Closes #47
Also fixes a bug in the acknowledging.
\approve @ds @shinde @taraCloses #47
Also fixes a bug in the acknowledging.
\approve @ds @shinde @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/83Resolve "Acknowledge messages on queue"2022-09-14T12:19:42+02:00Laurens OostwegelResolve "Acknowledge messages on queue"Closes #34
Tasks are not acknowledged after they are finished (previous statement: `no_ack=True`). This means that whether or not a task if finished, is not given as feedback to the queue. If a task fails, it will not be re-added to th...Closes #34
Tasks are not acknowledged after they are finished (previous statement: `no_ack=True`). This means that whether or not a task if finished, is not given as feedback to the queue. If a task fails, it will not be re-added to the queue. Adding acknowledgements will make our system more fail-proof.
\approve @ds @tara @shindeLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/82Draft: Resolve "Rabotnik always needs a building_id to run"2023-06-07T08:20:37+02:00Laurens OostwegelDraft: Resolve "Rabotnik always needs a building_id to run"Closes #45Closes #45Laurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/81Resolve "Rabotnik type error in base.py"2022-06-29T11:59:02+02:00Laurens OostwegelResolve "Rabotnik type error in base.py"Implements backwards compatibility with Python<3.9 where type hinting with list() and dict() is not yet implemented (so List() and Dict() from typing module are needed)
Closes #44
\approve @ds @gislarsImplements backwards compatibility with Python<3.9 where type hinting with list() and dict() is not yet implemented (so List() and Dict() from typing module are needed)
Closes #44
\approve @ds @gislarsLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/80Switched to late ack=False (default)2022-05-06T15:24:42+02:00Marius Kriegerowskimarius@gfz-potsdam.deSwitched to late ack=False (default)https://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/79Added name method to task2022-04-29T16:13:54+02:00Marius Kriegerowskimarius@gfz-potsdam.deAdded name method to taskhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/78Implemented `run` on the Assembly and added docs2022-04-29T12:45:59+02:00Marius Kriegerowskimarius@gfz-potsdam.deImplemented `run` on the Assembly and added docshttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/77Introduce env var to ignore other missing environment variables2022-05-31T17:12:40+02:00Marius Kriegerowskimarius@gfz-potsdam.deIntroduce env var to ignore other missing environment variablesMarius Kriegerowskimarius@gfz-potsdam.deMarius Kriegerowskimarius@gfz-potsdam.dehttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/76Added try-catch to savely load environment variables2022-03-30T10:47:52+02:00Marius Kriegerowskimarius@gfz-potsdam.deAdded try-catch to savely load environment variableshttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/75Changed to environment variables for configuration2022-03-28T10:33:06+02:00Marius Kriegerowskimarius@gfz-potsdam.deChanged to environment variables for configurationhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/74Switch to environment variables for configuration2022-03-25T11:26:41+01:00Marius Kriegerowskimarius@gfz-potsdam.deSwitch to environment variables for configurationhttps://git.gfz-potsdam.de/globaldynamicexposure/rabotnik/rabotnik/-/merge_requests/73Use global scoped storages2022-02-11T13:55:29+01:00Marius Kriegerowskimarius@gfz-potsdam.deUse global scoped storagesStorages and the pool are now in global scope. This avoids accidental re-initialization.Storages and the pool are now in global scope. This avoids accidental re-initialization.