GitLab

Fix consumer hang on connection errors

See merge request !2

The consumer tended to get hung when connection errors occurred.
This commit re-implements the consumer based on the asynchronous
consumer example.

* commit '8c85d776':
  Log the tasks UUID when enqueueing

This allows us to correlate enqueueing and processing of tasks.

* commit '40213b2c':
  Fix bug in consumer_queues

consumer queues set in config was not used when initializing a Consumer

Merge pull request #42 in INT/tofh from add-separate-config-variable-for-queue-consumption to master

* commit 'c1874671':
  Generalise a sentence in README.md
  Add separate config variable for queue consumption

This allows tofh to consume queues that have bindings which are not
under the consumers control.

* commit 'a94ff89d':
  Support indefinite task retries

This allows setting max_retries to None, which will result in TOFH
retrying a task indefinitely.

This can typically be used when your APIs tend to be down over longer
time periods.

* commit '56e9099a':
  KUR-47: Add env and release to sentry

* commit 'ad454a48':
  tofh.plugins.mq: make use of get_consumer() in entry-point script
  tofh.plugins.mq: configure logging earlier
  tofh.plugins.mq: fix reading username and password from stdin
  tofh.config: add AppConfig.to_dict()
  tofh.config: simplify AppConfig.get()
  tofh.config: make Celery config introspectable
  tofh.plugins.mq.consumer: add new entry-point for getting Consumer
  tests: structure tests according to modules
  tofh.plugins.mq: move Consumer to own module
  vcs: ignore a usual venv path

* commit 'e8850889':
  KUR-42: Add versioning to kur

Now that the ConsumerConfig is not mutated and the username/password
is correctly read into the original AppConfig, we remove some
boilerplate for configuring the consumer by relying on the new
tofh.plugins.mq.get_consumer() API.

Fixes: TOFH-3

Logging was being initialised relatively late causing the loss of
(at least) two previous log entries.  This patch ensures the logger
is initialised right at the beginning, before we take care of any
other business.

Fixes: TOFH-3

Reading passwords from stdin was broken as it interacted with the
wrong configuration.  This fixes the problem and adds support for
reading a username.

This is a prerequisite for making use of the new get_consumer() API
introduced earlier in this changeset.

Fixes: TOFH-3

To alter an AppConfig that has already been read in from file or
YAML string, we must have a way to dump it to a mutable dictionary
that can later be read in again.

This will come in useful when the entry-point script of tofh.plugins.mq
asks for the username and password if they are not present in the
configuration.

The approach used here is delicately handcrafted and a future
improvement would be to formally define AppConfig's format in some
way.  See TOFH-4.

Fixes: TOFH-3

AppConfig.get() can rely on self.extra.get() for the exact same
result.

Fixes: TOFH-3

create_celery_config() crafts a celeryconfig object with the
conf_dict's items as its attributes.  Because it is an object we
cannot safely introspect its members without filtering its __dict__.

To make the Celery configuration introspectable, i.e. for testing
purposes, this patch replaces it with a functionally equivalent
class that inherits from a dict.  This enables us to enumerate it
in the usual ways.

Fixes: TOFH-3

Adds a new entry-point tofh.plugins.mq.get_consumer() for retrieving
the MQ consumer based on an AppConfig.

This makes it easier to use tofh.plugins.mq programmatically, i.e.
when not invoking 'TOFH_CONFIG=config.yml python -m tofh.plugins.mq'
from the shell.

Fixes: TOFH-3

Introduces a structure to the tests/ directory based on the module
hierarchy under tofh/.

Fixes: TOFH-3

__init__.py should primarily be used for what its name infers:
initialisation code.

This moves tofh.plugins.mq.Consumer to tofh.plugins.mq.consumer,
but maintains an import to not break API compatibility.

Fixes: TOFH-3

Fixes: TOFH-3

* commit '87b0e04c':
  Clarify priority number range
  Add a priority attribute to tasks. Document how to prioritize tasks.
  Allow setting arguments on exchanges, queues and bindings
  Format tofh.tasks with black
  Format tofh.plugins.mq with black

Issue: KAL-241

* commit '8c75056e':
  BOTTINT-28: Use filesystem broker

* commit '5b938e71':
  Fixing pull request comments
  Documentation of use
  Show example of plugin setup and use
  Added schedule type and frequency to task
  Added celery import to config
  Added scheduler plugin