Staging instance, all changes can be removed at any time

Skip to content

Refactor the pytest_plugin

David Douard requested to merge generated-differential-D3130-source into generated-differential-D3130-target

This diff consists in a stack of revisions that refactor the pytest_plugin so that:

  • the kafka_server fixture ensures topics exist on the broker beforehand,
  • the kafka_prefix is used as is everywhere to prevent some naughty code coupling (user of the plugin must also modify the kafka_prefix the same way it was done here).

Included revisions are:

  • pytest_plugin: improve error message in consume_messages() helper function

  • pytest_plugin: do not crash if the list of consumed messages is empty

    if assert_all_objects_consumed() helper function.

  • pytest_plugin: modify the kafka_server fixture so topics are created beforehand

    Since we now check that topics exist on the kafka broker when instanciating a JournalClient, topics but be created when the mock kafka broker is started for tests to keep working (e.g. in swh.storage).

    This needs a small adaptation in the JournalClient code itself to ensure "empty" messages used to create topics will be silently ignored.

    The kafka_server() fixture is split in 2 fixtures:

    • kafka_server_base() that only creates the mock kafka server,
    • kafka_server that() uses this later and creates the topics resulting from the conjunction of the kafka_prefix() and the new object_types() fixture.

    This new object_types() fixture is thus used to define the list of topics (in conjunction with the kafka_prefix fixture) a (mock) kafka_server() will create at startup time.

  • test: replace usage of a strategy .example() by a hardcoded value

    to prevent a warning from hypothesis.

  • pytest_plugin: remove '.swh.journal.objects' from test_config fixture

    This induces a tigh coupling with code from other packages that use this fixture (e.g. swh.storage currently have to modify the kafka_prefix resulting from the fixture so the consumer and the writer are in sync).

    If someone wants a customized version of a kafka_prefix, it must be done by overriding the kafka_prefix fixture, not by modifying its returned value.

    Note that this commit will break swh.storage.tests.test_kafka_writer until it is updated.

  • pytest_plugin: use the object_types fixture in test_config

    instead of hardcoding a specific list there that may become out of sync with the default list of topics used elsewhere in the pytest_plugin.

Depends on !168 (closed).

Migrated from D3130 (view on Phabricator)

Merge request reports