Staging instance, all changes can be removed at any time

Skip to content

Add support for the static consumer group feature to journal client

Nicolas Dandrimont requested to merge generated-differential-D2751-source into generated-differential-D2751-target

The new KAFKA_GROUP_INSTANCE_ID env variable can be set on journal clients to set the group.instance.id enable support for Kafka Static Consumer Groups.

https://github.com/edenhill/librdkafka/pull/2525 https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

Combined with larger values for the session.timeout.ms and max.poll.interval.ms settings, this setting informs the Group Coordinator broker that the consumer group has static membership, and that the disappearance of a given member of the consumer group should not immediately trigger a rebalance; This allows crashing consumers to re-join the consumer group and start consuming from their assigned partitions immediately.

This setting is implemented as an environment variable so that several consumers in the same group can share a configuration file, and still override the value (e.g. by setting Environment=KAFKA_GROUP_INSTANCE_ID=groupname-%i in a systemd template unit).

Test Plan

this now has a new unit test 0:-)

Migrated from D2751 (view on Phabricator)

Merge request reports