Staging instance, all changes can be removed at any time

Skip to content

Move indexer_journal_client to scheduler instances

No need for those journal client to be on kafka nodes.

This is also an attempt to clarify where to deploy our backfill configuration.

In effect, this moves the service from:

  • prod: getty -> saatchi
  • staging: journal0 -> scheduler0

Deployment:

  • deactivate puppet on saatchi, scheduler0
  • stop old services on getty and journal0
  • (clean up)
  • puppet run on saatchi, scheduler0

Test Plan

octocatalog-diff:

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging getty
...
I, [2021-02-10T17:09:55.240238 #14284]  INFO -- : Catalogs compiled for getty.internal.softwareheritage.org
I, [2021-02-10T17:09:56.576597 #14284]  INFO -- : Diffs computed for getty.internal.softwareheritage.org
diff origin/production/getty.internal.softwareheritage.org current/getty.internal.softwareheritage.org
*******************************************
- File[/etc/softwareheritage/indexer/journal_client.yml]
*******************************************
- File[/etc/softwareheritage/indexer]
*******************************************
- File[/etc/softwareheritage/journal]
*******************************************
- File[/etc/systemd/system/swh-indexer-journal-client.service]
*******************************************
- Package[python3-swh.indexer]
*******************************************
- Package[python3-swh.journal]
*******************************************
- Service[swh-indexer-journal-client]
*******************************************
- Systemd::Unit_file[swh-indexer-journal-client.service]
*******************************************
*** End octocatalog-diff on getty.internal.softwareheritage.org
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging journal0.internal.staging.swh.network
...
*** Running octocatalog-diff on host journal0.internal.staging.swh.network
I, [2021-02-10T17:10:26.109041 #19467]  INFO -- : Catalogs compiled for journal0.internal.staging.swh.network
I, [2021-02-10T17:10:27.498780 #19467]  INFO -- : Diffs computed for journal0.internal.staging.swh.network
diff origin/production/journal0.internal.staging.swh.network current/journal0.internal.staging.swh.network
*******************************************
- File[/etc/softwareheritage/indexer/journal_client.yml]
*******************************************
- File[/etc/softwareheritage/indexer]
*******************************************
- File[/etc/softwareheritage/journal]
*******************************************
- File[/etc/systemd/system/swh-indexer-journal-client.service]
*******************************************
- Package[python3-swh.indexer]
*******************************************
- Package[python3-swh.journal]
*******************************************
- Service[swh-indexer-journal-client]
*******************************************
- Systemd::Unit_file[swh-indexer-journal-client.service]
*******************************************
*** End octocatalog-diff on journal0.internal.staging.swh.network
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging scheduler0.internal.staging.swh.network
...
*** Running octocatalog-diff on host scheduler0.internal.staging.swh.network
I, [2021-02-10T17:10:57.460041 #24611]  INFO -- : Catalogs compiled for scheduler0.internal.staging.swh.network
I, [2021-02-10T17:10:59.194738 #24611]  INFO -- : Diffs computed for scheduler0.internal.staging.swh.network
diff origin/production/scheduler0.internal.staging.swh.network current/scheduler0.internal.staging.swh.network
*******************************************
+ File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhdev"
      "mode": "0644"
      "notify": "Service[swh-indexer-journal-client]"
      "owner": "root"
      "content": >>>
---
journal:
  brokers:
  - journal0.internal.staging.swh.network
  group_id: swh.indexer.journal_client
  prefix: swh.journal.objects
scheduler:
  cls: remote
  args:
    url: http://scheduler0.internal.staging.swh.network:5008/
<<<
*******************************************
+ File[/etc/softwareheritage/indexer] =>
   parameters =>
      "ensure": "directory"
      "group": "swhworker"
      "mode": "0755"
      "owner": "swhworker"
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Package[python3-swh.indexer] =>
   parameters =>
      "ensure": "present"
*******************************************
+ Service[swh-indexer-journal-client] =>
   parameters =>
      "enable": true
      "ensure": "running"
*******************************************
+ Systemd::Unit_file[swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "notify": ["Service[swh-indexer-journal-client]"]
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
*** End octocatalog-diff on scheduler0.internal.staging.swh.network
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging saatchi
...
*** Running octocatalog-diff on host saatchi.internal.softwareheritage.org
I, [2021-02-10T17:15:05.562029 #30505]  INFO -- : Catalogs compiled for saatchi.internal.softwareheritage.org
I, [2021-02-10T17:15:07.186209 #30505]  INFO -- : Diffs computed for saatchi.internal.softwareheritage.org
diff origin/production/saatchi.internal.softwareheritage.org current/saatchi.internal.softwareheritage.org
*******************************************
+ File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhdev"
      "mode": "0644"
      "notify": "Service[swh-indexer-journal-client]"
      "owner": "root"
      "content": >>>
---
journal:
  brokers:
  - kafka1.internal.softwareheritage.org
  - kafka2.internal.softwareheritage.org
  - kafka3.internal.softwareheritage.org
  - kafka4.internal.softwareheritage.org
  group_id: swh.indexer.journal_client
  prefix: swh.journal.objects
scheduler:
  cls: remote
  url: http://saatchi.internal.softwareheritage.org:5008/
<<<
*******************************************
+ File[/etc/softwareheritage/indexer] =>
   parameters =>
      "ensure": "directory"
      "group": "swhworker"
      "mode": "0755"
      "owner": "swhworker"
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Package[python3-swh.indexer] =>
   parameters =>
      "ensure": "present"
*******************************************
+ Service[swh-indexer-journal-client] =>
   parameters =>
      "enable": true
      "ensure": "running"
*******************************************
+ Systemd::Unit_file[swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "notify": ["Service[swh-indexer-journal-client]"]
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
*** End octocatalog-diff on saatchi.internal.softwareheritage.org

Migrated from D5059 (view on Phabricator)

Merge request reports

Loading