Staging instance, all changes can be removed at any time

Skip to content

Handle BufferErrors in writer.kafka.Producer

When the local rdkafka producer queue is full, the producer would crash with a BufferError. While this condition is rare, it can happen when running a backfill operation.

When such a condition arises, retry sending the message a few times with an increasing backoff. When giving up, use the common KafkaDeliveryError mechanism rather than the plain BufferError.

Related to swh-scheduler#3000 (closed)

Test Plan

new tests added for retry and DeliveryError


Migrated from D5144 (view on Phabricator)

Merge request reports

Loading