Design the production deployment of winery

From: https://wiki.softwareheritage.org/wiki/A_practical_approach_to_efficiently_store_100_billions_small_objects_in_Ceph

We have two frontend servers, which are expected to be redundant with one another.

One server is the primary and receives all writes; the other server is secondary and gets all writes replicated to it (via PostgreSQL physical replication).

Deployment common to both servers:

• Winery deployment
  • Winery Writer (active or standby)
  • Winery Packer (could be load-balanced depending on the actual load created)
  • Winery Reader (can be active on both servers)
• Load-balancer for the objstorage RPC
  • Routes writes to the active writer
  • Routes reads to the secondary server (or load-balances?)
  • Health-check?
• reverse proxy in front of Gunicorn / Winery
  • needed if load-balancer doesn't support HTTP connection buffering

Primary server:

• Read-write PostgreSQL deployment
  • Replication setup towards the secondary server
    • uses physical replication
  • User management Will be done later if necessary (#5201)
    • winery-writer user
      • supports writes to main winery database (to record shard creation and locking, as well as signature2shard mapping)
      • can read all data
      • database creation for shards
    • winery-packer user
      • supports writes to main winery database (to record packed shards)
      • can read all data
      • database deletion for (packed) shards
    • winery-reader user
      • read-only access to all databases

Secondary server:

• Read-only PostgreSQL "standby"
• ~~PostgreSQL backups (written to Ceph via pgbackrest + radosgw?) (#5202)

To be defined:

• objstorage replayers (Deployed in production kubernetes cluster)