File upload service walk-through: direct-to-storage tickets, asynchronous scanning and quotas

Goal

Let clients upload files straight to object storage, keep application servers out of the byte path, and admit a file to the product only after it has been checked and counted against a quota.

Prerequisites

Object storage that issues time-limited signed upload URLs, a scanner that runs as a job, a quota per owner, and the existing validation rules for types and names.

Steps

1. Constraints: application servers never proxy bytes; a file is invisible until checked; abandoned uploads must not hold quota or storage forever.
2. Components: an upload API that issues tickets; a bucket with incoming/ and ready/ prefixes (or two buckets); a completion endpoint or a consumer of storage events; a scan worker; a quota ledger; a serving path with short-lived signed download URLs.
3. Flow: the client requests a ticket with declared size and type; the API checks the quota, reserves the declared size, records the ticket and returns a signed PUT URL for one object key with an expiry of minutes. The S3 documentation describes presigned URLs as limited by the permissions of the identity that created them, so the signing identity should be able to write only to incoming/. After uploading, the client calls complete; the service reads the actual size and type from storage, rejects mismatches and enqueues a scan. The worker moves the object to ready/ or deletes it and records the reason.
4. Data model: upload(id, owner, status: ticketed|uploaded|scanning|ready|rejected|expired, declared_bytes, actual_bytes, content_type, storage_key, ticket_expires_at, created_at); quota(owner, limit_bytes, used_bytes, reserved_bytes), updated in the same transaction as each status change.
5. Failure modes: tickets never completed (an expiry job releases the reservation; a lifecycle rule deletes stale incoming/ objects, and S3 documents an AbortIncompleteMultipartUpload lifecycle action for unfinished multipart uploads); two uploads racing past the quota (reserve under a row lock); scanner backlog leaving files in scanning (show the status, alert on queue age); a file shared before the scan finished (serve only from ready/); a stored object whose complete call was lost (accept a late complete, reconcile against storage listings).
6. Measure: ticket-to-ready time, share of expired tickets, scan queue age, rejection reasons, quota-denied requests, orphaned objects found by reconciliation.
7. Not first: resumable multipart for small files, image derivatives, deduplication by hash, client-side encryption, folders.

Expected result

Bytes flow from client to storage; the service handles only metadata, and every stored object maps to a row whose status explains why it exists.

Limits and test basis

Proposed design, no measurements. Validation of names and types is covered by the existing upload article. Whether a signed PUT can bound the object size depends on the storage product, which is why the size is verified after upload.