Implement removal notification, notification watcher and propagation tooling for mirrors
To implement the policy on takedown requests propagation for mirrors, we need to send notifications to mirrors when a removal happen, and add processing tools for these notifications.
Notifications will take the form of messages in a dedicated Kafka topic. As previously thought, they should contain: an identifier for the removal operation, the reason for the removal, and a list of SWHID of the removed objects.
On mirrors, a process should watch for new messages. When one arrives, it should, in order:
- Create a new masking request in the masking proxy database, masking the SWHIDs with status “decision-pending”.
- Send an email to the mirror operator with a command to remove the objects from the mirror storage and a command to make the objects visible again.
XXX: Do we want a command to mask the objects permanently but not remove them?
If a masking request with the same slug already exists, it means we are catching up on old messages for some reason, and we stop there.
The email will be able to list removed origins by retrieving the URL from the storage using the swh:1:ori:…
SWHIDs in the notification.
Please note that a similar process will be needed when objects are masked permanently in the main archive…