Staging instance, all changes can be removed at any time

Skip to content

journal_client: Ensure queue position does not overflow

Antoine R. Dumont requested to merge generated-differential-D6136-source into generated-differential-D6136-target

Queue positions are date and the current next_position_offset used to compute the new queue position was not bounded. This has the side-effect of making overflow error.

This commit adapts the journal client computations to limit such next_position_offset to 10 [1]. This value was chosen because above that exponent the dates overflow (and we are way in the future already).

Currently in production, we already exceed such value (staging, prod: 14 [2]) which makes the scheduler journal client fail.

  • [1]
In [10]: utcnow() + timedelta(days=4 ** 10)
Out[10]: datetime.datetime(4892, 7, 20, 16, 13, 7, 345452, tzinfo=datetime.timezone.utc)

In [11]: utcnow() + timedelta(days=4 ** 11)
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-11-42dc34eb0713> in <module>
----> 1 utcnow() + timedelta(days=4 ** 11)

OverflowError: date value out of range
  • [2]
18:19:22 swh-scheduler@db1:5432=> select max(next_position_offset) from origin_visit_stats where next_position_offset > 10;
+-----+
| max |
+-----+
|  14 |
+-----+
(1 row)

Time: 145.075 ms

18:19:40 softwareheritage-scheduler@belvedere:5432=> select max(next_position_offset) from origin_visit_stats where next_position_offset > 10;
+-----+
| max |
+-----+
|  14 |
+-----+
(1 row)

Time: 83509.421 ms (01:23.509)

Related to swh/infra/sysadm-environment#3502 (closed)

Test Plan

tox

staging scheduler got patched with the following and it restarted with the following (db migrated as well)

Migrated from D6136 (view on Phabricator)

Merge request reports