Object Storage & Large Blobs

An object store is a flat namespace of immutable-ish blobs with metadata, addressed by key, served over HTTP, at roughly a hundredth the cost of database storage and effectively unlimited in size. Keys look hierarchical but aren't; there are no directories.

Don't put files in the database. A large bytea column bloats the table, wrecks backup and restore times, blows up memory on every query that forgets to exclude it, and makes replication expensive. Store the key, the content type, the size, a checksum, and the owning entity.

Don't proxy uploads through your API either. A presigned URL lets the client PUT directly to the bucket with a short-lived, scope-limited credential your server generates — no bandwidth through your service, no request-size limits, no blocked worker for the duration of a 500MB upload. Use presigned GETs for private downloads, with an expiry. Large files use multipart upload so a failure resumes rather than restarts.

The reconciliation problem is the interesting part. Two writes — an object and a row — with no shared transaction, so any failure leaves them inconsistent. The workable pattern: create a pending row first, hand out the presigned URL, and mark it ready when the client confirms or when a bucket event notification arrives. Orphan objects are then cleaned by a sweeper over old pending rows, and orphan rows never happen because the row comes first. Deletion runs the other way: delete the row, queue the object deletion, tolerate retries.

The rest is policy. Never trust the client's content type or filename — sniff, constrain by extension, and serve user content from a separate origin so a malicious upload can't run as your site. Versioning and lifecycle rules move old objects to cheaper classes and delete what expires. Egress costs are the line item that surprises people, which is why a CDN sits in front of anything public. And immutability plus a content hash in the key makes caching trivial and makes an overwrite impossible to get wrong.

File upload is one of the standard frontend system design prompts, and presigned direct upload plus the pending/ready state machine is the expected answer. Getting it wrong in production means either a memory-exhausting API or a bucket full of orphaned files nobody can attribute.

• Files belong in object storage; the database holds a row with the key, size, type and checksum.
• Presigned URLs let clients upload and download directly, keeping bandwidth and long-lived requests out of your API.
• Multipart upload makes large transfers resumable instead of all-or-nothing.
• There is no transaction spanning the bucket and the database — use a pending row, confirm on completion, and sweep orphans.
• Create the row before the object so failures leave orphaned bytes, which are cheap, rather than dangling references.
• Never trust client-supplied content types or filenames, and serve user content from a separate origin.
• Content-hash keys make objects immutable and cacheable forever; lifecycle rules control the storage bill.