Staging instance, all changes can be removed at any time

Skip to content

journal_client: Compute next position for origin visit

Antoine R. Dumont requested to merge generated-differential-D5950-source into generated-differential-D5950-target

For origin without any last_update information [1], the journal client is now also in charge of moving their next position in the queue for rescheduling. Depending on their status, the next position offset and next_visit_queue_position are updated after each visit completes:

  • if the visit has failed, increase the next visit target by the minimal visit interval (to take into account transient loading issues)
  • if the visit is successful, and records some changes, decrease the visit interval index by 2 (visit the origin way more often).
  • if the visit is successful, and records no changes, increase the visit interval index by 1 (visit the origin less often).

We then set the next visit target to its current value + the new visit interval multiplied by a random fudge factor (picked in the -/+ 10% range).

The fudge factor allows the visits to spread out, avoiding "bursts" of loaded origins e.g. when a number of origins from a single hoster are processed at once.

Note that the computations happen for all origins for simplicity and code maintenance but it will only be used by a new soon-to-be scheduling policy.

  • [1] Lister cannot provide it for some reason.

Depends on D5919 Related to #2345

Test Plan

tox (failing as tests need update)

Migrated from D5950 (view on Phabricator)

Merge request reports