Unify retry/error handling for content replay
This uses a custom wrapper exception and tenacity callbacks to log exceptions when the copy of a given content fails several times. This makes the consumer more robust (fewer crashes), which in turns allows fewer consumer rebalances, which finally drastically reduces the consumer bandwidth consumption.
At this point, the retry of "definitely" failed content replays needs to be handled separately.
Depends on !117 (closed)
Test Plan
one new tox test added for the retry behavior
Migrated from D2782 (view on Phabricator)