toolbox: Adapt db script for specific backend services
When it's needed (e.g. indexer, scrubber, ...), it allows to specify the module name (e.g. indexer, scrubber, ...) and the configuration key to introspect the db access (e.g. indexer_storage, scrubber, ...) in the configuration.
This is especially useful when the configuration key does not match the (top-level) service type (e.g. indexer). Or when multiple instances of a service exists (e.g. scrubber, storage, ...).
Implementation wise, this adapts the wrapper cli script to allow extra parameters from the main yaml to be passed along down to the clis.
Tested on various services with success [1]
[1] toolbox tests
swh@swh-toolbox-665d47b677-lmvhn:~$ ./check-db-version.sh scheduler /etc/swh/config-scheduler.yml
module: scheduler
current code version: 35
version: 35
Database already configured at the latest version.
swh@swh-toolbox-665d47b677-lmvhn:~$ ./check-db-version.sh scrubber /etc/swh/config-scrubber-storage-secondary.yml scrubber
module: scrubber
current code version: 7
version: 7
Database already configured at the latest version.
swh@swh-toolbox-665d47b677-lmvhn:~$ ./check-db-version.sh scrubber /etc/swh/config-scrubber-storage-primary.yml scrubber
module: scrubber
current code version: 7
version: 7
Database already configured at the latest version.
swh@swh-toolbox-665d47b677-lmvhn:~$ ./check-db-version.sh indexer /etc/swh/config-indexer-storage.yml indexer_storage
module: indexer
current code version: 137
version: 137
Database already configured at the latest version.
swh@swh-toolbox-665d47b677-lmvhn:~$ ./check-db-version.sh storage /etc/swh/config-storage-secondary.yml storage
module: storage
flavor: read_replica
current code version: 191
version: 191
Database already configured at the latest version.
swh@swh-toolbox-665d47b677-lmvhn:~$ ./check-db-version.sh
usage: ./check-db-version.sh MODULE_NAME [CONFIG_FILE [CONFIG_KEY]]
Check whether MODULE_NAME, configured in the CONFIG_KEY key of CONFIG_FILE, needs a migration.
CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
CONFIG_KEY sets which key is used in the config to find the database config. It defaults to <MODULE_NAME>
This script exits with code 1 if the settings are wrong, and with code 2 if a migration is needed.
[2] helm diff
[swh] Comparing changes between branches production and improve-toolbox (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in improve-toolbox branch for environment staging...
[swh] Generate config in improve-toolbox branch for environment staging...
[swh] Generate config in improve-toolbox branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in improve-toolbox branch for environment production...
[swh] Generate config in improve-toolbox branch for environment production...
[swh] Generate config in improve-toolbox branch for environment production...
------------- diff for environment staging namespace swh -------------
--- /tmp/swh-chart.swh.0rLQdha5/staging-swh.before 2024-02-21 17:40:31.612568975 +0100
+++ /tmp/swh-chart.swh.0rLQdha5/staging-swh.after 2024-02-21 17:40:32.328568072 +0100
@@ -1099,121 +1099,151 @@
set -eux
export SWH_CONFIG_FILENAME=/etc/swh/config-scheduler.yml
swh scheduler -C $SWH_CONFIG_FILENAME task-type register
check-db-version.sh: |
#!/bin/bash
set -eu
+ set -o pipefail
- TEMP_FILE=/tmp/db-version.txt
+ if [ "$#" -lt 1 ]; then
+ echo >&2 "\
+ usage: $0 MODULE_NAME [CONFIG_FILE [CONFIG_KEY]]
+
+ Check whether MODULE_NAME, configured in the CONFIG_KEY key of
+ CONFIG_FILE, needs a migration.
+
+ CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
+ CONFIG_KEY sets which key is used in the config to find the database
+ config. It defaults to <MODULE_NAME>
+
+ This script exits with code 1 if the settings are wrong, and with code
+ 2 if a migration is needed.
+ "
+ exit 127
+ fi
- MODULE=$1
- CODE_VERSION=${2-""}
- DB_VERSION=${3-""}
+ MODULE_NAME="$1"
- if [ -z ${MODULE} ]; then
- echo The environment variable must be defined with the module to check
- echo for example "storage"
- exit 1
- fi
+ CONFIG_FILE="${2:-/etc/swh/config-${MODULE_NAME}.yml}"
+ CONFIG_KEY="${3:-$MODULE_NAME}"
- # checking the database status
- swh db --config-file=/etc/swh/config-${MODULE}.yml version "${MODULE}" | \
- tee "${TEMP_FILE}"
+ # Create a temporary file, open it for writing as fd 3, for reading as fd 4 and 5
+ TEMP_FILE="$(mktemp)"
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ exec 3>"${TEMP_FILE}"
+ exec 4<"${TEMP_FILE}"
+ exec 5<"${TEMP_FILE}"
- if [ -e "${CODE_VERSION}" ]; then
- echo "Unable to find the code version"
+ # and remove the tempfile
+ rm -f "${TEMP_FILE}"
+
+ # check the database status into the tempfile
+ swh db --config-file="${CONFIG_FILE}" version "${MODULE_NAME}" \
+ --module-config-key="${CONFIG_KEY}" | tee >&2 /dev/fd/3
+
+ # Read the code and db version from the tempfile
+ CODE_VERSION="$(awk -F': ' '/code/ {print $2}' <&4)"
+ DB_VERSION="$(awk -F': ' '/^version/ {print $2}' <&5)"
+
+ if [ -z "${CODE_VERSION}" ]; then
+ echo >&2 "Unable to find the code version."
exit 1
fi
- if [ -e "${DB_VERSION}" ]; then
- echo "Unable to find the code version"
+ if [ -z "${DB_VERSION}" ]; then
+ echo >&2 "Unable to find the database version."
exit 1
fi
- if [ "$DB_VERSION" -eq "$CODE_VERSION" ]; then
- echo "Database already configured at the latest version"
+ if [ "${DB_VERSION}" -eq "${CODE_VERSION}" ]; then
+ echo "Database already configured at the latest version."
else
- echo "Migration required from $DB_VERSION to $CODE_VERSION."
+ echo "Migration required from <${DB_VERSION}> to <${CODE_VERSION}>."
+ exit 2
fi
migrate-db-version.sh: |
#!/bin/bash
set -eu
- TEMP_FILE=/tmp/db-version.txt
-
- MODULE=$1
- CODE_VERSION=${2-""}
-
- if [ -z "${MODULE}" ]; then
- echo The environment variable must be defined with the module to check
- echo for example "storage"
- exit 1
+ if [ "$#" -lt 1 ]; then
+ echo >&2 "\
+ usage: $0 MODULE_NAME [CONFIG_FILE [CONFIG_KEY [TO_VERSION]]]]
+
+ Migrate the database for MODULE_NAME, configured in the CONFIG_KEY key
+ of CONFIG_FILE, to TO_VERSION.
+
+ CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
+ CONFIG_KEY sets which key is used in the config to find the database
+ config. It defaults to <MODULE_NAME>
+ TO_VERSION defaults to the latest schema version defined in <MODULE_NAME>
+ "
+ exit 127
fi
- if [ ! -z "${CODE_VERSION}" ]; then
- swh db --config-file=/etc/swh/config-${MODULE}.yml upgrade "${MODULE}" \
- --to-version ${CODE_VERSION}
- else
- swh db --config-file=/etc/swh/config-${MODULE}.yml upgrade "${MODULE}"
- fi
+ MODULE_NAME="$1"
+
+ CONFIG_FILE="${2:-/etc/swh/config-${MODULE_NAME}.yml}"
+ CONFIG_KEY="${3:-$MODULE_NAME}"
+ TO_VERSION="${4:-}"
+
+ exec swh db --config-file="${CONFIG_FILE}" upgrade "${MODULE_NAME}" \
+ --module-config-key="${CONFIG_KEY}" \
+ ${TO_VERSION:+"--to-version=${TO_VERSION}"}
check-scrubber-journal-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scrubber-journal
+ /opt/swh/bin/check-db-version.sh scrubber-journal /etc/swh/config-scrubber-journal.yml scrubber-journal
migrate-scrubber-journal-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scrubber-journal
+ exec /opt/swh/bin/migrate-db-version.sh scrubber-journal /etc/swh/config-scrubber-journal.yml scrubber-journal
check-scrubber-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scrubber-storage
+ /opt/swh/bin/check-db-version.sh scrubber /etc/swh/config-scrubber-storage.yml scrubber
migrate-scrubber-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scrubber-storage
+ exec /opt/swh/bin/migrate-db-version.sh scrubber /etc/swh/config-scrubber-storage.yml scrubber
check-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh storage
+ /opt/swh/bin/check-db-version.sh storage /etc/swh/config-storage.yml storage
migrate-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh storage
+ exec /opt/swh/bin/migrate-db-version.sh storage /etc/swh/config-storage.yml storage
---
# Source: swh/templates/utils/config-utils.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-utils
namespace: swh
data:
prepare-configuration.sh: |
#!/bin/bash
@@ -14494,21 +14524,21 @@
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: swh-toolbox
annotations:
# Force a rollout upgrade if the configuration changes
checksum/config: e97aeaf7d335204f17859d112f29e02d63964df65690c3445ede85921cac9910
- checksum/configScript: e3b6f8f32ad4eaa9e388b5203c750970c56b5dfa216c4fb5d2f2f176d8249d2a
+ checksum/configScript: da25c913eb4386cccc982a2d2ab342f0e90810ce16a4f7e39b6a049d9b0df54b
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/toolbox
operator: In
values:
- "true"
@@ -14518,31 +14548,33 @@
- name: prepare-configuration-scrubber-journal
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scrubber-journal.yml
env:
+
- name: BROKER_USER_PASSWORD
valueFrom:
secretKeyRef:
name: swh-archive-broker-secret
key: BROKER_USER_PASSWORD
# 'name' secret must exist & include that ^ key
optional: false
+
- name: SCRUBBER_POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scrubber-postgresql-common-secret
key: postgres-swh-scrubber-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -14555,30 +14587,34 @@
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scrubber-storage.yml
env:
+
+
+
- name: SCRUBBER_POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scrubber-postgresql-common-secret
key: postgres-swh-scrubber-password
# 'name' secret must exist & include that ^ key
optional: false
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-postgresql-common-secret
key: postgres-guest-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -14590,20 +14626,21 @@
- name: prepare-configuration-storage
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-storage.yml
env:
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-postgresql-common-secret
key: postgres-swh-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -14627,45 +14664,45 @@
- /opt/swh/entrypoint.sh
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: toolbox-script-utils
mountPath: /opt/swh/bin
readOnly: true
volumes:
- name: configuration
emptyDir: {}
-
+
- name: configuration-scrubber-journal-template
configMap:
name: toolbox-scrubber-journal-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-scrubber-storage-template
configMap:
name: toolbox-scrubber-storage-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-storage-template
configMap:
name: toolbox-storage-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: toolbox-script-utils
configMap:
name: toolbox-script-utils
defaultMode: 0555
---
# Source: swh/templates/web/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh
------------- diff for environment staging namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.0rLQdha5/staging-swh-cassandra.before 2024-02-21 17:40:31.928568577 +0100
+++ /tmp/swh-chart.swh.0rLQdha5/staging-swh-cassandra.after 2024-02-21 17:40:32.628567694 +0100
@@ -7362,151 +7362,181 @@
set -eux
export SWH_CONFIG_FILENAME=/etc/swh/config-scheduler.yml
swh scheduler -C $SWH_CONFIG_FILENAME task-type register
check-db-version.sh: |
#!/bin/bash
set -eu
+ set -o pipefail
- TEMP_FILE=/tmp/db-version.txt
+ if [ "$#" -lt 1 ]; then
+ echo >&2 "\
+ usage: $0 MODULE_NAME [CONFIG_FILE [CONFIG_KEY]]
+
+ Check whether MODULE_NAME, configured in the CONFIG_KEY key of
+ CONFIG_FILE, needs a migration.
+
+ CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
+ CONFIG_KEY sets which key is used in the config to find the database
+ config. It defaults to <MODULE_NAME>
+
+ This script exits with code 1 if the settings are wrong, and with code
+ 2 if a migration is needed.
+ "
+ exit 127
+ fi
- MODULE=$1
- CODE_VERSION=${2-""}
- DB_VERSION=${3-""}
+ MODULE_NAME="$1"
- if [ -z ${MODULE} ]; then
- echo The environment variable must be defined with the module to check
- echo for example "storage"
- exit 1
- fi
+ CONFIG_FILE="${2:-/etc/swh/config-${MODULE_NAME}.yml}"
+ CONFIG_KEY="${3:-$MODULE_NAME}"
- # checking the database status
- swh db --config-file=/etc/swh/config-${MODULE}.yml version "${MODULE}" | \
- tee "${TEMP_FILE}"
+ # Create a temporary file, open it for writing as fd 3, for reading as fd 4 and 5
+ TEMP_FILE="$(mktemp)"
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ exec 3>"${TEMP_FILE}"
+ exec 4<"${TEMP_FILE}"
+ exec 5<"${TEMP_FILE}"
- if [ -e "${CODE_VERSION}" ]; then
- echo "Unable to find the code version"
+ # and remove the tempfile
+ rm -f "${TEMP_FILE}"
+
+ # check the database status into the tempfile
+ swh db --config-file="${CONFIG_FILE}" version "${MODULE_NAME}" \
+ --module-config-key="${CONFIG_KEY}" | tee >&2 /dev/fd/3
+
+ # Read the code and db version from the tempfile
+ CODE_VERSION="$(awk -F': ' '/code/ {print $2}' <&4)"
+ DB_VERSION="$(awk -F': ' '/^version/ {print $2}' <&5)"
+
+ if [ -z "${CODE_VERSION}" ]; then
+ echo >&2 "Unable to find the code version."
exit 1
fi
- if [ -e "${DB_VERSION}" ]; then
- echo "Unable to find the code version"
+ if [ -z "${DB_VERSION}" ]; then
+ echo >&2 "Unable to find the database version."
exit 1
fi
- if [ "$DB_VERSION" -eq "$CODE_VERSION" ]; then
- echo "Database already configured at the latest version"
+ if [ "${DB_VERSION}" -eq "${CODE_VERSION}" ]; then
+ echo "Database already configured at the latest version."
else
- echo "Migration required from $DB_VERSION to $CODE_VERSION."
+ echo "Migration required from <${DB_VERSION}> to <${CODE_VERSION}>."
+ exit 2
fi
migrate-db-version.sh: |
#!/bin/bash
set -eu
- TEMP_FILE=/tmp/db-version.txt
-
- MODULE=$1
- CODE_VERSION=${2-""}
-
- if [ -z "${MODULE}" ]; then
- echo The environment variable must be defined with the module to check
- echo for example "storage"
- exit 1
+ if [ "$#" -lt 1 ]; then
+ echo >&2 "\
+ usage: $0 MODULE_NAME [CONFIG_FILE [CONFIG_KEY [TO_VERSION]]]]
+
+ Migrate the database for MODULE_NAME, configured in the CONFIG_KEY key
+ of CONFIG_FILE, to TO_VERSION.
+
+ CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
+ CONFIG_KEY sets which key is used in the config to find the database
+ config. It defaults to <MODULE_NAME>
+ TO_VERSION defaults to the latest schema version defined in <MODULE_NAME>
+ "
+ exit 127
fi
- if [ ! -z "${CODE_VERSION}" ]; then
- swh db --config-file=/etc/swh/config-${MODULE}.yml upgrade "${MODULE}" \
- --to-version ${CODE_VERSION}
- else
- swh db --config-file=/etc/swh/config-${MODULE}.yml upgrade "${MODULE}"
- fi
+ MODULE_NAME="$1"
+
+ CONFIG_FILE="${2:-/etc/swh/config-${MODULE_NAME}.yml}"
+ CONFIG_KEY="${3:-$MODULE_NAME}"
+ TO_VERSION="${4:-}"
+
+ exec swh db --config-file="${CONFIG_FILE}" upgrade "${MODULE_NAME}" \
+ --module-config-key="${CONFIG_KEY}" \
+ ${TO_VERSION:+"--to-version=${TO_VERSION}"}
check-indexer-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh indexer-storage
+ /opt/swh/bin/check-db-version.sh indexer /etc/swh/config-indexer-storage.yml indexer_storage
migrate-indexer-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh indexer-storage
+ exec /opt/swh/bin/migrate-db-version.sh indexer /etc/swh/config-indexer-storage.yml indexer_storage
check-scheduler-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scheduler
+ /opt/swh/bin/check-db-version.sh scheduler /etc/swh/config-scheduler.yml scheduler
migrate-scheduler-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scheduler
+ exec /opt/swh/bin/migrate-db-version.sh scheduler /etc/swh/config-scheduler.yml scheduler
check-scrubber-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scrubber-storage
+ /opt/swh/bin/check-db-version.sh scrubber /etc/swh/config-scrubber-storage.yml scrubber
migrate-scrubber-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scrubber-storage
+ exec /opt/swh/bin/migrate-db-version.sh scrubber /etc/swh/config-scrubber-storage.yml scrubber
check-vault-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh vault
+ /opt/swh/bin/check-db-version.sh vault /etc/swh/config-vault.yml vault
migrate-vault-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh vault
+ exec /opt/swh/bin/migrate-db-version.sh vault /etc/swh/config-vault.yml vault
check-webhooks-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh webhooks
+ /opt/swh/bin/check-db-version.sh webhooks /etc/swh/config-webhooks.yml webhooks
migrate-webhooks-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh webhooks
+ exec /opt/swh/bin/migrate-db-version.sh webhooks /etc/swh/config-webhooks.yml webhooks
---
# Source: swh/templates/utils/config-utils.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-utils
namespace: swh-cassandra
data:
prepare-configuration.sh: |
#!/bin/bash
@@ -32135,21 +32165,21 @@
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: swh-toolbox
annotations:
# Force a rollout upgrade if the configuration changes
checksum/config: df5d1aa136684d49d96cb247e608bd3c30b74943b7fd8ae9d0e30fca3c736d2a
- checksum/configScript: 2039091ccb94d9826414aed88d4b7a63d2be405fb14e935201ae449661b515b4
+ checksum/configScript: f702b4aeedc086e1acb59911db631e9b7de3e2e33a0361365aec5f66872a0eb1
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/toolbox
operator: In
values:
- "true"
@@ -32159,56 +32189,63 @@
- name: prepare-configuration-indexer-storage
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-indexer-storage.yml
env:
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-indexer-storage-postgresql-secret
key: postgres-swh-indexer-password
# 'name' secret must exist & include that ^ key
optional: false
+
+
+
+
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-indexer-storage-template
mountPath: /etc/swh/configuration-template
- name: prepare-configuration-scheduler
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scheduler.yml
env:
+
- name: AMQP_PASSWORD
valueFrom:
secretKeyRef:
name: amqp-secrets
key: swhproducer-password
# 'name' secret must exist & include that ^ key
optional: false
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scheduler-postgresql-common-secret
key: postgres-swh-scheduler-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -32221,30 +32258,34 @@
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scrubber-storage.yml
env:
+
+
+
- name: SCRUBBER_POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scrubber-postgresql-common-secret
key: postgres-swh-scrubber-password
# 'name' secret must exist & include that ^ key
optional: false
+
- name: CASSANDRA_PASSWORD
valueFrom:
secretKeyRef:
name: common-secrets
key: cassandra-swh-ro-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -32256,20 +32297,21 @@
- name: prepare-configuration-vault
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-vault.yml
env:
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-vault-postgresql-secret
key: postgres-swh-vault-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -32281,20 +32323,21 @@
- name: prepare-configuration-webhooks
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-webhooks.yml
env:
+
- name: SVIX_AUTH_TOKEN
valueFrom:
secretKeyRef:
name: common-secrets
key: svix-token
# 'name' secret must exist & include that ^ key
optional: false
@@ -32318,61 +32361,61 @@
- /opt/swh/entrypoint.sh
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: toolbox-script-utils
mountPath: /opt/swh/bin
readOnly: true
volumes:
- name: configuration
emptyDir: {}
-
+
- name: configuration-indexer-storage-template
configMap:
name: toolbox-indexer-storage-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-scheduler-template
configMap:
name: toolbox-scheduler-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-scrubber-storage-template
configMap:
name: toolbox-scrubber-storage-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-vault-template
configMap:
name: toolbox-vault-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-webhooks-template
configMap:
name: toolbox-webhooks-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: toolbox-script-utils
configMap:
name: toolbox-script-utils
defaultMode: 0555
---
# Source: swh/templates/vault/rpc-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra
------------- diff for environment staging namespace swh-cassandra-next-version -------------
No differences
------------- diff for environment production namespace swh -------------
--- /tmp/swh-chart.swh.0rLQdha5/production-swh.before 2024-02-21 17:40:33.196566977 +0100
+++ /tmp/swh-chart.swh.0rLQdha5/production-swh.after 2024-02-21 17:40:33.712566327 +0100
@@ -7367,181 +7367,211 @@
set -eux
export SWH_CONFIG_FILENAME=/etc/swh/config-scheduler.yml
swh scheduler -C $SWH_CONFIG_FILENAME task-type register
check-db-version.sh: |
#!/bin/bash
set -eu
+ set -o pipefail
- TEMP_FILE=/tmp/db-version.txt
+ if [ "$#" -lt 1 ]; then
+ echo >&2 "\
+ usage: $0 MODULE_NAME [CONFIG_FILE [CONFIG_KEY]]
+
+ Check whether MODULE_NAME, configured in the CONFIG_KEY key of
+ CONFIG_FILE, needs a migration.
+
+ CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
+ CONFIG_KEY sets which key is used in the config to find the database
+ config. It defaults to <MODULE_NAME>
+
+ This script exits with code 1 if the settings are wrong, and with code
+ 2 if a migration is needed.
+ "
+ exit 127
+ fi
- MODULE=$1
- CODE_VERSION=${2-""}
- DB_VERSION=${3-""}
+ MODULE_NAME="$1"
- if [ -z ${MODULE} ]; then
- echo The environment variable must be defined with the module to check
- echo for example "storage"
- exit 1
- fi
+ CONFIG_FILE="${2:-/etc/swh/config-${MODULE_NAME}.yml}"
+ CONFIG_KEY="${3:-$MODULE_NAME}"
- # checking the database status
- swh db --config-file=/etc/swh/config-${MODULE}.yml version "${MODULE}" | \
- tee "${TEMP_FILE}"
+ # Create a temporary file, open it for writing as fd 3, for reading as fd 4 and 5
+ TEMP_FILE="$(mktemp)"
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ exec 3>"${TEMP_FILE}"
+ exec 4<"${TEMP_FILE}"
+ exec 5<"${TEMP_FILE}"
- if [ -e "${CODE_VERSION}" ]; then
- echo "Unable to find the code version"
+ # and remove the tempfile
+ rm -f "${TEMP_FILE}"
+
+ # check the database status into the tempfile
+ swh db --config-file="${CONFIG_FILE}" version "${MODULE_NAME}" \
+ --module-config-key="${CONFIG_KEY}" | tee >&2 /dev/fd/3
+
+ # Read the code and db version from the tempfile
+ CODE_VERSION="$(awk -F': ' '/code/ {print $2}' <&4)"
+ DB_VERSION="$(awk -F': ' '/^version/ {print $2}' <&5)"
+
+ if [ -z "${CODE_VERSION}" ]; then
+ echo >&2 "Unable to find the code version."
exit 1
fi
- if [ -e "${DB_VERSION}" ]; then
- echo "Unable to find the code version"
+ if [ -z "${DB_VERSION}" ]; then
+ echo >&2 "Unable to find the database version."
exit 1
fi
- if [ "$DB_VERSION" -eq "$CODE_VERSION" ]; then
- echo "Database already configured at the latest version"
+ if [ "${DB_VERSION}" -eq "${CODE_VERSION}" ]; then
+ echo "Database already configured at the latest version."
else
- echo "Migration required from $DB_VERSION to $CODE_VERSION."
+ echo "Migration required from <${DB_VERSION}> to <${CODE_VERSION}>."
+ exit 2
fi
migrate-db-version.sh: |
#!/bin/bash
set -eu
- TEMP_FILE=/tmp/db-version.txt
-
- MODULE=$1
- CODE_VERSION=${2-""}
-
- if [ -z "${MODULE}" ]; then
- echo The environment variable must be defined with the module to check
- echo for example "storage"
- exit 1
+ if [ "$#" -lt 1 ]; then
+ echo >&2 "\
+ usage: $0 MODULE_NAME [CONFIG_FILE [CONFIG_KEY [TO_VERSION]]]]
+
+ Migrate the database for MODULE_NAME, configured in the CONFIG_KEY key
+ of CONFIG_FILE, to TO_VERSION.
+
+ CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
+ CONFIG_KEY sets which key is used in the config to find the database
+ config. It defaults to <MODULE_NAME>
+ TO_VERSION defaults to the latest schema version defined in <MODULE_NAME>
+ "
+ exit 127
fi
- if [ ! -z "${CODE_VERSION}" ]; then
- swh db --config-file=/etc/swh/config-${MODULE}.yml upgrade "${MODULE}" \
- --to-version ${CODE_VERSION}
- else
- swh db --config-file=/etc/swh/config-${MODULE}.yml upgrade "${MODULE}"
- fi
+ MODULE_NAME="$1"
+
+ CONFIG_FILE="${2:-/etc/swh/config-${MODULE_NAME}.yml}"
+ CONFIG_KEY="${3:-$MODULE_NAME}"
+ TO_VERSION="${4:-}"
+
+ exec swh db --config-file="${CONFIG_FILE}" upgrade "${MODULE_NAME}" \
+ --module-config-key="${CONFIG_KEY}" \
+ ${TO_VERSION:+"--to-version=${TO_VERSION}"}
check-indexer-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh indexer-storage
+ /opt/swh/bin/check-db-version.sh indexer /etc/swh/config-indexer-storage.yml indexer_storage
migrate-indexer-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh indexer-storage
+ exec /opt/swh/bin/migrate-db-version.sh indexer /etc/swh/config-indexer-storage.yml indexer_storage
check-scheduler-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scheduler
+ /opt/swh/bin/check-db-version.sh scheduler /etc/swh/config-scheduler.yml scheduler
migrate-scheduler-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scheduler
+ exec /opt/swh/bin/migrate-db-version.sh scheduler /etc/swh/config-scheduler.yml scheduler
check-scrubber-journal-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scrubber-journal
+ /opt/swh/bin/check-db-version.sh scrubber-journal /etc/swh/config-scrubber-journal.yml scrubber-journal
migrate-scrubber-journal-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scrubber-journal
+ exec /opt/swh/bin/migrate-db-version.sh scrubber-journal /etc/swh/config-scrubber-journal.yml scrubber-journal
check-scrubber-storage-primary-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scrubber-storage-primary
+ /opt/swh/bin/check-db-version.sh scrubber /etc/swh/config-scrubber-storage-primary.yml scrubber
migrate-scrubber-storage-primary-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scrubber-storage-primary
+ exec /opt/swh/bin/migrate-db-version.sh scrubber /etc/swh/config-scrubber-storage-primary.yml scrubber
check-scrubber-storage-secondary-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scrubber-storage-secondary
+ /opt/swh/bin/check-db-version.sh scrubber /etc/swh/config-scrubber-storage-secondary.yml scrubber
migrate-scrubber-storage-secondary-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scrubber-storage-secondary
+ exec /opt/swh/bin/migrate-db-version.sh scrubber /etc/swh/config-scrubber-storage-secondary.yml scrubber
check-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh storage
+ /opt/swh/bin/check-db-version.sh storage /etc/swh/config-storage.yml storage
migrate-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh storage
+ exec /opt/swh/bin/migrate-db-version.sh storage /etc/swh/config-storage.yml storage
check-storage-secondary-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh storage-secondary
+ /opt/swh/bin/check-db-version.sh storage /etc/swh/config-storage-secondary.yml storage
migrate-storage-secondary-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh storage-secondary
+ exec /opt/swh/bin/migrate-db-version.sh storage /etc/swh/config-storage-secondary.yml storage
---
# Source: swh/templates/utils/config-utils.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-utils
namespace: swh
data:
prepare-configuration.sh: |
#!/bin/bash
@@ -33749,21 +33779,21 @@
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: swh-toolbox
annotations:
# Force a rollout upgrade if the configuration changes
checksum/config: ad08aa1e482a1681ce6d3bf1f53a4bbd00d15bedffbf39844a779c7cd1d63281
- checksum/configScript: 663c64a77cb64ac413bb3014e6a87dbd2c528b0b92f716d79ebaeb200d76c6da
+ checksum/configScript: 75080e9c349ee6ac2b0ea94c23869f3192dc02380f47cab3370aadd2240ccdc1
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/toolbox
operator: In
values:
- "true"
@@ -33773,56 +33803,63 @@
- name: prepare-configuration-indexer-storage
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-indexer-storage.yml
env:
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-indexer-storage-postgresql-secret
key: postgres-swh-indexer-password
# 'name' secret must exist & include that ^ key
optional: false
+
+
+
+
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-indexer-storage-template
mountPath: /etc/swh/configuration-template
- name: prepare-configuration-scheduler
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scheduler.yml
env:
+
- name: AMQP_PASSWORD
valueFrom:
secretKeyRef:
name: amqp-secrets
key: swhproducer-password
# 'name' secret must exist & include that ^ key
optional: false
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scheduler-postgresql-common-secret
key: postgres-swh-scheduler-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -33834,38 +33871,40 @@
- name: prepare-configuration-scrubber-journal
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scrubber-journal.yml
env:
+
- name: BROKER_USER
valueFrom:
secretKeyRef:
name: swh-archive-broker-secret
key: BROKER_USER
# 'name' secret must exist & include that ^ key
optional: false
- name: BROKER_USER_PASSWORD
valueFrom:
secretKeyRef:
name: swh-archive-broker-secret
key: BROKER_USER_PASSWORD
# 'name' secret must exist & include that ^ key
optional: false
+
- name: SCRUBBER_POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scrubber-postgresql-common-secret
key: postgres-swh-scrubber-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -33878,30 +33917,34 @@
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scrubber-storage-primary.yml
env:
+
+
+
- name: SCRUBBER_POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scrubber-postgresql-common-secret
key: postgres-swh-scrubber-password
# 'name' secret must exist & include that ^ key
optional: false
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-storage-postgresql-common-secret
key: postgres-guest-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -33914,30 +33957,34 @@
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scrubber-storage-secondary.yml
env:
+
+
+
- name: SCRUBBER_POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scrubber-postgresql-common-secret
key: postgres-swh-scrubber-password
# 'name' secret must exist & include that ^ key
optional: false
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-storage-postgresql-common-secret
key: postgres-guest-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -33949,20 +33996,21 @@
- name: prepare-configuration-storage
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-storage.yml
env:
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-storage-postgresql-common-secret
key: postgres-swh-storage-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -33975,20 +34023,23 @@
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-storage-secondary.yml
env:
+
+
+
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-storage-postgresql-common-secret
key: postgres-swh-storage-password
# 'name' secret must exist & include that ^ key
optional: false
volumeMounts:
@@ -34011,77 +34062,77 @@
- /opt/swh/entrypoint.sh
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: toolbox-script-utils
mountPath: /opt/swh/bin
readOnly: true
volumes:
- name: configuration
emptyDir: {}
-
+
- name: configuration-indexer-storage-template
configMap:
name: toolbox-indexer-storage-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-scheduler-template
configMap:
name: toolbox-scheduler-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-scrubber-journal-template
configMap:
name: toolbox-scrubber-journal-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-scrubber-storage-primary-template
configMap:
name: toolbox-scrubber-storage-primary-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-scrubber-storage-secondary-template
configMap:
name: toolbox-scrubber-storage-secondary-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-storage-template
configMap:
name: toolbox-storage-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: configuration-storage-secondary-template
configMap:
name: toolbox-storage-secondary-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: toolbox-script-utils
configMap:
name: toolbox-script-utils
defaultMode: 0555
---
# Source: swh/templates/vault/rpc-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh
------------- diff for environment production namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.0rLQdha5/production-swh-cassandra.before 2024-02-21 17:40:33.368566760 +0100
+++ /tmp/swh-chart.swh.0rLQdha5/production-swh-cassandra.after 2024-02-21 17:40:33.896566094 +0100
@@ -1367,91 +1367,121 @@
set -eux
export SWH_CONFIG_FILENAME=/etc/swh/config-scheduler.yml
swh scheduler -C $SWH_CONFIG_FILENAME task-type register
check-db-version.sh: |
#!/bin/bash
set -eu
+ set -o pipefail
- TEMP_FILE=/tmp/db-version.txt
+ if [ "$#" -lt 1 ]; then
+ echo >&2 "\
+ usage: $0 MODULE_NAME [CONFIG_FILE [CONFIG_KEY]]
+
+ Check whether MODULE_NAME, configured in the CONFIG_KEY key of
+ CONFIG_FILE, needs a migration.
+
+ CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
+ CONFIG_KEY sets which key is used in the config to find the database
+ config. It defaults to <MODULE_NAME>
+
+ This script exits with code 1 if the settings are wrong, and with code
+ 2 if a migration is needed.
+ "
+ exit 127
+ fi
- MODULE=$1
- CODE_VERSION=${2-""}
- DB_VERSION=${3-""}
+ MODULE_NAME="$1"
- if [ -z ${MODULE} ]; then
- echo The environment variable must be defined with the module to check
- echo for example "storage"
- exit 1
- fi
+ CONFIG_FILE="${2:-/etc/swh/config-${MODULE_NAME}.yml}"
+ CONFIG_KEY="${3:-$MODULE_NAME}"
- # checking the database status
- swh db --config-file=/etc/swh/config-${MODULE}.yml version "${MODULE}" | \
- tee "${TEMP_FILE}"
+ # Create a temporary file, open it for writing as fd 3, for reading as fd 4 and 5
+ TEMP_FILE="$(mktemp)"
- CODE_VERSION=$(awk -F':' '/code/ {print $2}' ${TEMP_FILE})
- DB_VERSION=$(awk -F':' '/^version/ {print $2}' ${TEMP_FILE})
+ exec 3>"${TEMP_FILE}"
+ exec 4<"${TEMP_FILE}"
+ exec 5<"${TEMP_FILE}"
- if [ -e "${CODE_VERSION}" ]; then
- echo "Unable to find the code version"
+ # and remove the tempfile
+ rm -f "${TEMP_FILE}"
+
+ # check the database status into the tempfile
+ swh db --config-file="${CONFIG_FILE}" version "${MODULE_NAME}" \
+ --module-config-key="${CONFIG_KEY}" | tee >&2 /dev/fd/3
+
+ # Read the code and db version from the tempfile
+ CODE_VERSION="$(awk -F': ' '/code/ {print $2}' <&4)"
+ DB_VERSION="$(awk -F': ' '/^version/ {print $2}' <&5)"
+
+ if [ -z "${CODE_VERSION}" ]; then
+ echo >&2 "Unable to find the code version."
exit 1
fi
- if [ -e "${DB_VERSION}" ]; then
- echo "Unable to find the code version"
+ if [ -z "${DB_VERSION}" ]; then
+ echo >&2 "Unable to find the database version."
exit 1
fi
- if [ "$DB_VERSION" -eq "$CODE_VERSION" ]; then
- echo "Database already configured at the latest version"
+ if [ "${DB_VERSION}" -eq "${CODE_VERSION}" ]; then
+ echo "Database already configured at the latest version."
else
- echo "Migration required from $DB_VERSION to $CODE_VERSION."
+ echo "Migration required from <${DB_VERSION}> to <${CODE_VERSION}>."
+ exit 2
fi
migrate-db-version.sh: |
#!/bin/bash
set -eu
- TEMP_FILE=/tmp/db-version.txt
-
- MODULE=$1
- CODE_VERSION=${2-""}
-
- if [ -z "${MODULE}" ]; then
- echo The environment variable must be defined with the module to check
- echo for example "storage"
- exit 1
+ if [ "$#" -lt 1 ]; then
+ echo >&2 "\
+ usage: $0 MODULE_NAME [CONFIG_FILE [CONFIG_KEY [TO_VERSION]]]]
+
+ Migrate the database for MODULE_NAME, configured in the CONFIG_KEY key
+ of CONFIG_FILE, to TO_VERSION.
+
+ CONFIG_FILE defaults to /etc/swh/config-<MODULE_NAME>.yml
+ CONFIG_KEY sets which key is used in the config to find the database
+ config. It defaults to <MODULE_NAME>
+ TO_VERSION defaults to the latest schema version defined in <MODULE_NAME>
+ "
+ exit 127
fi
- if [ ! -z "${CODE_VERSION}" ]; then
- swh db --config-file=/etc/swh/config-${MODULE}.yml upgrade "${MODULE}" \
- --to-version ${CODE_VERSION}
- else
- swh db --config-file=/etc/swh/config-${MODULE}.yml upgrade "${MODULE}"
- fi
+ MODULE_NAME="$1"
+
+ CONFIG_FILE="${2:-/etc/swh/config-${MODULE_NAME}.yml}"
+ CONFIG_KEY="${3:-$MODULE_NAME}"
+ TO_VERSION="${4:-}"
+
+ exec swh db --config-file="${CONFIG_FILE}" upgrade "${MODULE_NAME}" \
+ --module-config-key="${CONFIG_KEY}" \
+ ${TO_VERSION:+"--to-version=${TO_VERSION}"}
check-scrubber-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/check-db-version.sh scrubber-storage
+ /opt/swh/bin/check-db-version.sh scrubber /etc/swh/config-scrubber-storage.yml scrubber
migrate-scrubber-storage-db-version.sh: |
#!/bin/bash
set -eu
- /opt/swh/bin/migrate-db-version.sh scrubber-storage
+ exec /opt/swh/bin/migrate-db-version.sh scrubber /etc/swh/config-scrubber-storage.yml scrubber
---
# Source: swh/templates/utils/config-utils.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-utils
namespace: swh-cassandra
data:
prepare-configuration.sh: |
#!/bin/bash
@@ -15026,21 +15056,21 @@
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: swh-toolbox
annotations:
# Force a rollout upgrade if the configuration changes
checksum/config: 88d71f4d0110718e59c2c354b59e119df7d2fbd8d5169bdc9217c880083c99cd
- checksum/configScript: e524fc0d85bca4929459b068d030f3d7dd3680cd450d39c51791420840539736
+ checksum/configScript: 4f326541ba4fb8bd129b38bacb919003a3d6e27f66965b88a182105dcc3cf048
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/toolbox
operator: In
values:
- "true"
@@ -15051,30 +15081,34 @@
image: debian:bullseye
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- -c
- eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config-scrubber-storage.yml
env:
+
+
+
- name: SCRUBBER_POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: swh-scrubber-postgresql-common-secret
key: postgres-swh-scrubber-password
# 'name' secret must exist & include that ^ key
optional: false
+
- name: CASSANDRA_PASSWORD
valueFrom:
secretKeyRef:
name: common-secrets
key: cassandra-swh-ro-password
# 'name' secret must exist & include that ^ key
optional: false
@@ -15098,29 +15132,29 @@
- /opt/swh/entrypoint.sh
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: toolbox-script-utils
mountPath: /opt/swh/bin
readOnly: true
volumes:
- name: configuration
emptyDir: {}
-
+
- name: configuration-scrubber-storage-template
configMap:
name: toolbox-scrubber-storage-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
-
+
- name: toolbox-script-utils
configMap:
name: toolbox-script-utils
defaultMode: 0555
---
# Source: swh/templates/web/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandraEdited by Antoine R. Dumont