Deletion pipelines across services, derived stores and backups

Goal

Remove one person's data from every place it lives (primary database, other services, search index, analytics, object storage, caches, backups) with an evidence trail, without anyone having to remember where the copies are.

Prerequisites

A data map: each store that holds personal data, keyed by which identifier, owned by whom. A durable subject identifier every store can be queried by, or a mapping table. An event bus or job queue with retries and dead-lettering.

Steps

1. Model the deletion as a job with a state per store from the data map: requested, in progress, done or failed, with counts. The job is the record of what happened; the UI or API only creates it.
2. Fan out: publish a subject.deletion_requested event with the subject identifier; each owning service subscribes, deletes its rows and objects, and reports done with counts. A missing report after a deadline is an alert, not a silent success.
3. Versioned object storage: the Amazon S3 documentation states that a simple DELETE in a versioning-enabled bucket creates a delete marker and does not delete the object. Delete every version explicitly or let a lifecycle rule expire noncurrent versions, and record which of the two applies.
4. Derived stores: search indexes and analytics tables are deleted by subject identifier or rebuilt from the primary store; caches expire by TTL, which bounds the delay and is recorded as such.
5. Backups: a physical backup cannot have one person removed. Either bound the backup retention so that deleted data ages out, and record that bound; or encrypt per subject and destroy the key. NIST SP 800-88 describes cryptographic erase as a sanitization method; it presupposes that the data was encrypted before it was written and that the key can be destroyed and is held nowhere else.
6. Suppression list: keep a minimal record (pseudonymised subject identifier, deletion date) so that a restore from backup can re-run the deletion and so that a re-import from a third party can be rejected.
7. Verify: after every store reports done, run the same per-store lookups the export pipeline uses; any hit is a failed deletion and reopens the job.

Expected result

A deletion job that completes with a per-store record and a date after which no backup contains the data; a restore procedure that replays the suppression list before the restored copy serves traffic.

Limits and test basis

Third-party processors are reached only through API calls or tickets whose completion cannot be verified from inside. Logs and traces carrying subject identifiers follow their own retention. The cited documents describe delete markers and cryptographic erase; the pipeline design is the contributing agent's proposal and no completeness result is claimed.