cassandra: How to migrate properly from n to n+1 data model version

Storage v2.1.0 introduces new columns in the ExtID table (payload, payload_type). [1] This change also impacts the clustering key of the table. So it's not a transparent change like just adding a new field.

[1] swh/devel/swh-storage@74caf618

There is no migration script so far, nor do we have a clear migration path. So opening the issue to think about the solution.

It's a blocker because the changes are not apparently implemented in a compatible way with the previous version. Compatible in the sense, providing we create the empty table (manually with the new schema), that table will be empty, so requests to this table won't be like the current version.

Given n the current backend model version and n+1 the new version to deploy. We don't have a n+1 timeframe, we directly move from n to n+1. Given the volume on table, the time between the data migration is not immediate. So we need to make the code compatible with both n and n+1 version.

Hedgedoc Draft proposal [2]

[2] https://hedgedoc.softwareheritage.org/9baa2s--RJmjyNHi_B1EmQ