staging/next-version: Migrate staging next version service to use next storage rpc independent from staging's
[better reviewed one commit at a time]
This adapts the storage deployment template so we can initialize an empty postgresql database through init-container.
This then adapts some minikube.yaml so we can actually create on one side (cluster-components) an empty db, and use it on the other side (swh) through a storage rpc service (which would initialize the empty db).
The end goal being for now is to be able to initialize the next-version postgresql db. So we can deploy a specific rpc storage service that loaders would use in the next-version environment (and stop writing into the staging environment).
helm diff [swh]
[swh] Comparing changes between branches production and initialize-pg-storage (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in initialize-pg-storage branch for environment staging...
[swh] Generate config in initialize-pg-storage branch for environment staging...
[swh] Generate config in initialize-pg-storage branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in initialize-pg-storage branch for environment production...
[swh] Generate config in initialize-pg-storage branch for environment production...
[swh] Generate config in initialize-pg-storage branch for environment production...
------------- diff for environment staging namespace swh -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.2vTOWZ23/staging-swh.before, 134 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.2vTOWZ23/staging-swh.after, 134 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned 16 differences
|___/
data (v1/ConfigMap/swh/database-utils)
+ two map entries added:
migrate-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim it
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
echo "The database should be initialized..."
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
swh db --config-file="${TEMP_CONFIG}" upgrade "${MODULE_NAME}"
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
initialize-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
set -x
CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
# This must be run as "postgres" user (for pg extensions installation)
uri=postgresql://postgres:$SWH_PGPASSWORD@$SWH_PGHOST:5432/$SWH_PGDATABASE
swh db init-admin --db-name=$uri "${MODULE}"
# This must be run with the owner of the db
swh db --config-file=$TEMP_CONFIG init "${MODULE}"
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
echo "Code and DB versions are different."
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
data.check-storage-db-version.sh (v1/ConfigMap/swh/database-utils)
± value change in multiline text (three inserts, three deletions)
- #!/bin/bash
+ #!/usr/bin/env bash
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
+ # trim
+ CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
+
+ DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
+ # trim it
+ DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
- if [ "$DB_VERSION" -ne "$CODE_VERSION" ]; then
+ if [ "$DB_VERSION" != "$CODE_VERSION" ]; then
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-content)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-directory)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-extid)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-metadata)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-origin)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-origin-visit)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-origin-visit-status)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-raw-extrinsic-metadata)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-release)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-revision)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-skipped-content)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh/storage-replayer-snapshot)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh/storage-postgresql-read-only)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh/storage-postgresql-read-write)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
------------- diff for environment staging namespace swh-cassandra -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.2vTOWZ23/staging-swh-cassandra.before, 421 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.2vTOWZ23/staging-swh-cassandra.after, 421 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned 17 differences
|___/
data (v1/ConfigMap/swh-cassandra/database-utils)
+ two map entries added:
migrate-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim it
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
echo "The database should be initialized..."
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
swh db --config-file="${TEMP_CONFIG}" upgrade "${MODULE_NAME}"
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
initialize-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
set -x
CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
# This must be run as "postgres" user (for pg extensions installation)
uri=postgresql://postgres:$SWH_PGPASSWORD@$SWH_PGHOST:5432/$SWH_PGDATABASE
swh db init-admin --db-name=$uri "${MODULE}"
# This must be run with the owner of the db
swh db --config-file=$TEMP_CONFIG init "${MODULE}"
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
echo "Code and DB versions are different."
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
data.check-storage-db-version.sh (v1/ConfigMap/swh-cassandra/database-utils)
± value change in multiline text (three inserts, three deletions)
- #!/bin/bash
+ #!/usr/bin/env bash
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
+ # trim
+ CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
+
+ DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
+ # trim it
+ DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
- if [ "$DB_VERSION" -ne "$CODE_VERSION" ]; then
+ if [ "$DB_VERSION" != "$CODE_VERSION" ]; then
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh-cassandra/indexer-storage-rpc)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-content)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-directory)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-extid)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-metadata)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-origin)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-origin-visit)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-origin-visit-status)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-raw-extrinsic-metadata)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-release)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-revision)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-skipped-content)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-snapshot)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh-cassandra/storage-cassandra)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh-cassandra/storage-cassandra-read-only)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
------------- diff for environment staging namespace swh-cassandra-next-version -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.2vTOWZ23/staging-swh-cassandra-next-version.before, 191 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.2vTOWZ23/staging-swh-cassandra-next-version.after, 189 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned 15 differences
|___/
(file level)
- five documents removed:
---
# Source: swh/templates/storage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: storage-cassandra-configuration-template
namespace: swh-cassandra-next-version
data:
config.yml.template: |
storage:
cls: pipeline
steps:
- blocking_db: host=db1.internal.staging.swh.network port=5432 user=swh-blocking
dbname=swh-blocking password=${BLOCKING_ADMIN_PGPASSWORD}
cls: blocking
- auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
cls: cassandra
consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
hosts:
- cassandra1.internal.staging.swh.network
- cassandra2.internal.staging.swh.network
- cassandra3.internal.staging.swh.network
journal_writer:
anonymize: true
brokers:
- journal2.internal.staging.swh.network
client_id: swh.storage-cassandra.journal_writer.${HOSTNAME}
cls: kafka
prefix: swh.journal.objects
producer_config:
message.max.bytes: 1000000000
keyspace: swh
objstorage:
cls: remote
url: http://objstorage-multiplexer-read-write-rpc-ingress
# Source: swh/templates/storage/service.yaml
apiVersion: v1
kind: Service
metadata:
name: storage-cassandra
namespace: swh-cassandra-next-version
labels:
app: storage-cassandra
spec:
type: ClusterIP
selector:
app: storage-cassandra
ports:
- port: 5002
targetPort: 5002
name: rpc
# Source: swh/templates/storage/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
name: storage-cassandra
labels:
app: storage-cassandra
spec:
revisionHistoryLimit: 2
selector:
matchLabels:
app: storage-cassandra
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-cassandra
annotations:
checksum/config: 6dc8642a7b0cacddd94fc99a10f099cf17ad520880f9e03d4adf5cca28845365
checksum/database-utils: 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
operator: In
values:
- "true"
priorityClassName: swh-cassandra-next-version-frontend-rpc
initContainers:
- name: prepare-configuration
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/utils:20231211.1"
imagePullPolicy: IfNotPresent
command:
- /entrypoints/prepare-configuration.sh
env:
- name: BLOCKING_ADMIN_PGPASSWORD
valueFrom:
secretKeyRef:
key: postgres-swh-blocking-password
name: swh-postgresql-blocking-secret
optional: false
- name: CASSANDRA_PASSWORD
valueFrom:
secretKeyRef:
key: cassandra-swh-rw-password
name: common-secrets
optional: false
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-template
mountPath: /etc/swh/configuration-template
- name: config-utils
mountPath: /entrypoints
readOnly: true
- name: init-database
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/storage:20240613.1"
imagePullPolicy: IfNotPresent
command:
- /usr/local/bin/python3
args:
- /entrypoints/init-keyspace.py
volumeMounts:
- name: configuration
mountPath: /etc/swh
readOnly: true
- name: database-utils
mountPath: /entrypoints
readOnly: true
containers:
- name: storage-cassandra
resources:
requests:
memory: 1500Mi
cpu: 500m
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/storage:20240613.1"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5002
name: rpc
readinessProbe:
httpGet:
path: /
port: rpc
initialDelaySeconds: 15
failureThreshold: 30
periodSeconds: 5
livenessProbe:
tcpSocket:
port: rpc
initialDelaySeconds: 10
periodSeconds: 5
command:
- /bin/bash
args:
- "-c"
- /opt/swh/entrypoint.sh
env:
- name: THREADS
value: 4
- name: WORKERS
value: 8
- name: TIMEOUT
value: 60
- name: STATSD_HOST
value: prometheus-statsd-exporter
- name: STATSD_PORT
value: 9125
- name: STATSD_TAGS
value: "deployment:storage-cassandra"
- name: SWH_LOG_LEVEL
value: INFO
- name: SWH_SENTRY_ENVIRONMENT
value: staging
- name: SWH_MAIN_PACKAGE
value: swh.storage
- name: SWH_SENTRY_DSN
valueFrom:
secretKeyRef:
name: common-secrets
key: storage-sentry-dsn
# 'name' secret should exist & include key
# if the setting doesn't exist, sentry pushes will be disabled
optional: true
- name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
value: "true"
volumeMounts:
- name: configuration
mountPath: /etc/swh
volumes:
- name: configuration
emptyDir: {}
- name: configuration-template
configMap:
name: storage-cassandra-configuration-template
items:
- key: config.yml.template
path: config.yml.template
- name: database-utils
configMap:
name: database-utils
defaultMode: 0555
- name: config-utils
configMap:
name: config-utils
defaultMode: 0555
# Source: swh/templates/storage/autoscaling.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
namespace: swh-cassandra-next-version
name: storage-cassandra
labels:
app: storage-cassandra
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: storage-cassandra
minReplicas: 1
maxReplicas: 2
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 150
# Source: swh/templates/storage/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: swh-cassandra-next-version
name: storage-cassandra-ingress-default
labels:
app: storage-cassandra
endpoint-definition: default
annotations:
nginx.ingress.kubernetes.io/client-body-buffer-size: 128K
nginx.ingress.kubernetes.io/proxy-body-size: 4G
nginx.ingress.kubernetes.io/proxy-buffering: on
nginx.ingress.kubernetes.io/service-upstream: "true"
nginx.ingress.kubernetes.io/whitelist-source-range: "10.42.0.0/16,10.43.0.0/16"
spec:
rules:
- host: storage-cassandra-next-version.internal.staging.swh.network
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: storage-cassandra
port:
number: 5002
---
# Source: swh/templates/storage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: storage-postgresql-read-write-configuration-template
namespace: swh-cassandra-next-version
data:
config.yml.template: |
storage:
cls: pipeline
steps:
- blocking_db: host=db1.internal.staging.swh.network port=5432 user=swh-blocking
dbname=swh-blocking password=${BLOCKING_ADMIN_PGPASSWORD}
cls: blocking
- cls: postgresql
db: host=next-version-swh-rw port=5432 user=${POSTGRESQL_USERNAME} dbname=swh
password=${POSTGRESQL_PASSWORD}
objstorage:
cls: noop
# Source: swh/templates/storage/service.yaml
apiVersion: v1
kind: Service
metadata:
name: storage-postgresql-read-write
namespace: swh-cassandra-next-version
labels:
app: storage-postgresql-read-write
spec:
type: ClusterIP
selector:
app: storage-postgresql-read-write
ports:
- port: 5002
targetPort: 5002
name: rpc
# Source: swh/templates/storage/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
name: storage-postgresql-read-write
labels:
app: storage-postgresql-read-write
spec:
revisionHistoryLimit: 2
selector:
matchLabels:
app: storage-postgresql-read-write
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-postgresql-read-write
annotations:
checksum/config: 83235b4dcd78d48daa5c973b8f7d405516054b67f7365b8ec310567944691e48
checksum/database-utils: bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
operator: In
values:
- "true"
priorityClassName: swh-cassandra-next-version-frontend-rpc
initContainers:
- name: prepare-configuration
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/utils:20231211.1"
imagePullPolicy: IfNotPresent
command:
- /entrypoints/prepare-configuration.sh
env:
- name: BLOCKING_ADMIN_PGPASSWORD
valueFrom:
secretKeyRef:
key: postgres-swh-blocking-password
name: swh-postgresql-blocking-secret
optional: false
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: swh-postgresql-swh-swh-secret
optional: false
- name: POSTGRESQL_USERNAME
valueFrom:
secretKeyRef:
key: username
name: swh-postgresql-swh-swh-secret
optional: false
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-template
mountPath: /etc/swh/configuration-template
- name: config-utils
mountPath: /entrypoints
readOnly: true
- name: initialize-backend
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/toolbox:20240614.1"
command:
- /entrypoints/initialize-storage-db-version.sh
env:
- name: MODULE
value: storage
- name: SWH_PGDATABASE
value: swh
- name: SWH_PGPASSWORD
valueFrom:
secretKeyRef:
name: next-version-swh-superuser
key: password
- name: SWH_PGHOST
valueFrom:
secretKeyRef:
name: next-version-swh-superuser
key: host
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: database-utils
mountPath: /entrypoints
containers:
- name: storage-postgresql-read-write
resources:
requests:
memory: 512Mi
cpu: 500m
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/storage:20240613.1"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5002
name: rpc
readinessProbe:
httpGet:
path: /
port: rpc
initialDelaySeconds: 15
failureThreshold: 30
periodSeconds: 5
livenessProbe:
tcpSocket:
port: rpc
initialDelaySeconds: 10
periodSeconds: 5
command:
- /bin/bash
args:
- "-c"
- /opt/swh/entrypoint.sh
env:
- name: STATSD_HOST
value: prometheus-statsd-exporter
- name: STATSD_PORT
value: 9125
- name: STATSD_TAGS
value: "deployment:storage-postgresql-read-write"
- name: SWH_LOG_LEVEL
value: INFO
- name: SWH_SENTRY_ENVIRONMENT
value: staging
- name: SWH_MAIN_PACKAGE
value: swh.storage
- name: SWH_SENTRY_DSN
valueFrom:
secretKeyRef:
name: common-secrets
key: storage-sentry-dsn
# 'name' secret should exist & include key
# if the setting doesn't exist, sentry pushes will be disabled
optional: true
- name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
value: "true"
volumeMounts:
- name: configuration
mountPath: /etc/swh
volumes:
- name: configuration
emptyDir: {}
- name: configuration-template
configMap:
name: storage-postgresql-read-write-configuration-template
items:
- key: config.yml.template
path: config.yml.template
- name: database-utils
configMap:
name: database-utils
defaultMode: 0555
- name: config-utils
configMap:
name: config-utils
defaultMode: 0555
data (v1/ConfigMap/swh-cassandra-next-version/database-utils)
+ two map entries added:
migrate-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim it
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
echo "The database should be initialized..."
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
swh db --config-file="${TEMP_CONFIG}" upgrade "${MODULE_NAME}"
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
initialize-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
set -x
CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
# This must be run as "postgres" user (for pg extensions installation)
uri=postgresql://postgres:$SWH_PGPASSWORD@$SWH_PGHOST:5432/$SWH_PGDATABASE
swh db init-admin --db-name=$uri "${MODULE}"
# This must be run with the owner of the db
swh db --config-file=$TEMP_CONFIG init "${MODULE}"
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
echo "Code and DB versions are different."
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
data.check-storage-db-version.sh (v1/ConfigMap/swh-cassandra-next-version/database-utils)
± value change in multiline text (three inserts, three deletions)
- #!/bin/bash
+ #!/usr/bin/env bash
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
+ # trim
+ CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
+
+ DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
+ # trim it
+ DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
- if [ "$DB_VERSION" -ne "$CODE_VERSION" ]; then
+ if [ "$DB_VERSION" != "$CODE_VERSION" ]; then
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-content)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-directory)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-extid)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-metadata)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-origin)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-origin-visit)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-origin-visit-status)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-raw-extrinsic-metadata)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-release)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-revision)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-skipped-content)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra-next-version/storage-replayer-snapshot)
± value change
- 949a1256aa77d984ff1ba75ff778094e41395762ae0492e6403c09cdddaf4b09
+ bd53dfa905f293a4a6de34cf8a621acb6e2bb0e95ac5b90f4a68a40430b14033
------------- diff for environment production namespace swh -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.2vTOWZ23/production-swh.before, 447 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.2vTOWZ23/production-swh.after, 447 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned six differences
|___/
data (v1/ConfigMap/swh/database-utils)
+ two map entries added:
migrate-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim it
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
echo "The database should be initialized..."
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
swh db --config-file="${TEMP_CONFIG}" upgrade "${MODULE_NAME}"
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
initialize-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
set -x
CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
# This must be run as "postgres" user (for pg extensions installation)
uri=postgresql://postgres:$SWH_PGPASSWORD@$SWH_PGHOST:5432/$SWH_PGDATABASE
swh db init-admin --db-name=$uri "${MODULE}"
# This must be run with the owner of the db
swh db --config-file=$TEMP_CONFIG init "${MODULE}"
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
echo "Code and DB versions are different."
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
data.check-storage-db-version.sh (v1/ConfigMap/swh/database-utils)
± value change in multiline text (three inserts, three deletions)
- #!/bin/bash
+ #!/usr/bin/env bash
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
+ # trim
+ CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
+
+ DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
+ # trim it
+ DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
- if [ "$DB_VERSION" -ne "$CODE_VERSION" ]; then
+ if [ "$DB_VERSION" != "$CODE_VERSION" ]; then
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh/indexer-storage-read-only)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh/indexer-storage-read-write)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh/storage-postgresql-azure-readonly)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh/storage-postgresql-winery)
± value change
- 1cc40c53ffcd5d3a71357e55336d394b97a8e3c6fe8cb7aedc0b595cea7f92b7
+ 018f43d7967dfc2cdeab2e40bb800d4c4109700cd7ee6d60e9d0d2b191f17b3f
------------- diff for environment production namespace swh-cassandra -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.2vTOWZ23/production-swh-cassandra.before, 117 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.2vTOWZ23/production-swh-cassandra.after, 117 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned 16 differences
|___/
data (v1/ConfigMap/swh-cassandra/database-utils)
+ two map entries added:
migrate-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim it
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
echo "The database should be initialized..."
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
swh db --config-file="${TEMP_CONFIG}" upgrade "${MODULE_NAME}"
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
initialize-storage-db-version.sh: |
#!/usr/bin/env bash
set -eu
TEMP_FILE=/tmp/db-version.txt
TEMP_CONFIG=/tmp/config.yml
if [ -z ${MODULE} ]; then
echo The env variable must be defined with the module to check
echo for example "storage"
exit 1
fi
# extracting the postgresql configuration from a full configuration
# possibly with a pipeline
set +e
python /entrypoints/extract-storage-postgresql-config-py || exit 0
set -e
# checking the database status
swh db --config-file=$TEMP_CONFIG version "${MODULE}" | tee "${TEMP_FILE}"
set -x
CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
# trim
CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
# trim
DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
if [ "${DB_VERSION}" = "None" ]; then
# This must be run as "postgres" user (for pg extensions installation)
uri=postgresql://postgres:$SWH_PGPASSWORD@$SWH_PGHOST:5432/$SWH_PGDATABASE
swh db init-admin --db-name=$uri "${MODULE}"
# This must be run with the owner of the db
swh db --config-file=$TEMP_CONFIG init "${MODULE}"
elif [ "$DB_VERSION" != "$CODE_VERSION" ]; then
echo "Code and DB versions are different."
else
echo "The database is initialized and up-to-date, nothing to do!"
echo "Continue with the deployment."
fi
data.check-storage-db-version.sh (v1/ConfigMap/swh-cassandra/database-utils)
± value change in multiline text (three inserts, three deletions)
- #!/bin/bash
+ #!/usr/bin/env bash
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ CODE_VERSION=$(awk -F':' '/code / {print $2}' ${TEMP_FILE})
+ # trim
+ CODE_VERSION=${CODE_VERSION#"${CODE_VERSION%%[![:space:]]*}"}
+
+ DB_VERSION=$(awk -F':' '/^version: / {print $2}' ${TEMP_FILE})
+ # trim it
+ DB_VERSION=${DB_VERSION#"${DB_VERSION%%[![:space:]]*}"}
- if [ "$DB_VERSION" -ne "$CODE_VERSION" ]; then
+ if [ "$DB_VERSION" != "$CODE_VERSION" ]; then
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-content)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-directory)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-extid)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-metadata)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-origin)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-origin-visit)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-origin-visit-status)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-raw-extrinsic-metadata)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-release)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-revision)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-skipped-content)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/config_utils (apps/v1/Deployment/swh-cassandra/storage-replayer-snapshot)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh-cassandra/storage-cassandra-readonly)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
spec.template.metadata.annotations.checksum/database-utils (apps/v1/Deployment/swh-cassandra/storage-cassandra-readonly-internal)
± value change
- 39e707ef793d465f866524b576ba11aff78e035fd84bacdf27d4782a6aac8628
+ 367643f221cc260d1b99fec1590d62cf5a290bf1e97ae86e56e640ed2225c2b7
Edited by Antoine R. Dumont