Handle BufferErrors in writer.kafka.Producer
When the local rdkafka producer queue is full, the producer would crash with a BufferError. While this condition is rare, it can happen when running a backfill operation.
When such a condition arises, retry sending the message a few times with an increasing backoff. When giving up, use the common KafkaDeliveryError mechanism rather than the plain BufferError.
Related to swh-scheduler#3000 (closed)
Test Plan
new tests added for retry and DeliveryError
Migrated from D5144 (view on Phabricator)