Save code now status not updated since the 2024-05-07
It seems the svix server replies a 404 on api requests:
│ webhooks INFO:httpx:HTTP Request: GET https://svix.internal.staging.swh.network/api/v1/event-type/origin.visit "HTTP/1.1 404 Not Found"
│ webhooks INFO:httpx:HTTP Request: GET https://svix.internal.staging.swh.network/api/v1/event-type/origin.visit "HTTP/1.1 404 Not Found"
│ webhooks INFO:httpx:HTTP Request: GET https://svix.internal.staging.swh.network/api/v1/event-type/origin.visit "HTTP/1.1 404 Not Found"
│ webhooks INFO:httpx:HTTP Request: GET https://svix.internal.staging.swh.network/api/v1/event-type/origin.visit "HTTP/1.1 404 Not Found"
The problem is present in staging and production so it seems it's related to the last deployment made this day. The messages are acknowledged by the webhook journal client.
A possible cause is the svix api version was updated in the journal client image from 1.21 to 1.23 swh-apps@7ed1a447
Current mitigation:
- webhook journal client manually stopped in production
- status.io incident created
Identified actions:
-
Update svix in staging in 1.23 -
Update svix in production to 1.23 -
The kafka messages should not be acknowledged by the journal client after a 404 issue in swh-webhook -
Freeze the version of the library to match the svix server deployed in production
Edited by Vincent Sellier