Ensiklopedia VibeKoding: An Introduction to File and Object Storage.Ensiklopedia VibeKoding: An Introduction to File and Object Storage.
A user uploads an avatar, and you save it in the server's /uploads directory — then the server's disk fills up, or you add a second server and the user finds their avatar appears and disappears. File storage seems simple, but in distributed environments it's an architectural problem that requires serious attention. Object storage is the standard answer to this problem in the internet age.A user uploads an avatar, and you save it in the server's /uploads directory — then the server's disk fills up, or you add a second server and the user finds their avatar appears and disappears. File storage seems simple, but in distributed environments it's an architectural problem that requires serious attention. Object storage is the standard answer to this problem in the internet age.
What will you learn in this article?What will you learn in this article?
After reading this chapter, you will gain:After reading this chapter, you will gain:
| Chapter | Content | Core Concepts |
|---|---|---|
| Chapter 1 | Storage type comparison | Block storage, file storage, object storage |
| Chapter 2 | Object storage core concepts | Bucket, Object, Key, Metadata |
| Chapter 3 | File upload solutions | Client-side direct upload, Pre-signed URL |
| Chapter 4 | CDN acceleration | Edge nodes, cache policies, origin fetch |
| Chapter 5 | Best practices | Naming conventions, permissions, lifecycle |
------
When starting a project, saving user-uploaded files in a local server directory is the most intuitive approach. But as the project grows, you'll encounter a series of problems:When starting a project, saving user-uploaded files in a local server directory is the most intuitive approach. But as the project grows, you'll encounter a series of problems:
Object storage (like AWS S3, Alibaba Cloud OSS) solves all these problems: infinite capacity, globally accessible, automatic backup, native CDN support. It has become the de facto standard for file storage in internet applications.Object storage (like AWS S3, Alibaba Cloud OSS) solves all these problems: infinite capacity, globally accessible, automatic backup, native CDN support. It has become the de facto standard for file storage in internet applications.
------
There are three main storage types in the computing world, each solving problems at different levels.There are three main storage types in the computing world, each solving problems at different levels.
| Dimension | Block Storage | File Storage | Object Storage |
|---|---|---|---|
| Data unit | Fixed-size blocks | Files + directories | Objects (Key-Value) |
| Access protocol | iSCSI/FC | NFS/SMB | HTTP REST API |
| Performance | Highest (millisecond) | Medium | Lower (but sufficient) |
| Scalability | Limited | Medium | Nearly infinite |
| Cost | Highest | Medium | Lowest |
| Typical use case | Databases | Shared files | Images/video/backups |
- Block storage is like a hard drive — for databases - File storage is like a network shared folder — for sharing configurations across multiple servers - Object storage is like cloud storage — for user-uploaded images and videos- Block storage is like a hard drive — for databases - File storage is like a network shared folder — for sharing configurations across multiple servers - Object storage is like cloud storage — for user-uploaded images and videos
------
The data model of object storage is very simple: a Bucket is the container, an Object is the file, and each object is identified by a unique Key.The data model of object storage is very simple: a Bucket is the container, an Object is the file, and each object is identified by a unique Key.
CODE my-app-bucket/ ← Bucket ├── avatars/user-123.jpg ← Object Key ├── avatars/user-456.png ← Object Key ├── reports/2024/q1-report.pdf ← Object Key ("directory" is just a Key prefix) └── uploads/temp/file.zip ← Object Key
| Concept | Description | Example |
|---|---|---|
| Bucket | Storage container, globally unique name | my-app-prod, company-assets |
| Object | The stored file itself + metadata | An image, a PDF |
| Key | The object's unique identifier | avatars/user-123.jpg |
| Metadata | Additional information about the object | Content-Type, custom tags |
| ACL | Access Control List | public-read, private |
| Pre-signed URL | Temporary authorized access link | Upload/download link valid for 15 minutes |
avatars/user-123.jpg — the avatars/ part is not a directory, just a Key prefix. Object storage is a flat structure; all objects are at the same level. The "folders" displayed in the console are just visual grouping by prefix.avatars/user-123.jpg — the avatars/ part is not a directory, just a Key prefix. Object storage is a flat structure; all objects are at the same level. The "folders" displayed in the console are just visual grouping by prefix.
------
There are two mainstream approaches for file upload: server-side relay and client-side direct upload. For most scenarios, client-side direct upload is the better choice.There are two mainstream approaches for file upload: server-side relay and client-side direct upload. For most scenarios, client-side direct upload is the better choice.
1. Saves server bandwidth: Files don't pass through your server; they go directly to OSS 2. Avoids timeouts: Large file uploads won't trigger Nginx/gateway timeout limits 3. Reduces server load: The server only needs to issue credentials, not process file streams 4. Supports resumable uploads: OSS natively supports multipart upload; the frontend can implement resumable uploads Implementation steps: Frontend requests a Pre-signed URL from the backend → Frontend uses this URL to upload directly to OSS → OSS callback notifies the backend1. Saves server bandwidth: Files don't pass through your server; they go directly to OSS 2. Avoids timeouts: Large file uploads won't trigger Nginx/gateway timeout limits 3. Reduces server load: The server only needs to issue credentials, not process file streams 4. Supports resumable uploads: OSS natively supports multipart upload; the frontend can implement resumable uploads Implementation steps: Frontend requests a Pre-signed URL from the backend → Frontend uses this URL to upload directly to OSS → OSS callback notifies the backend
------
When your users are distributed globally, downloading files from a single origin server is slow. CDN (Content Delivery Network) deploys edge nodes worldwide, caching files at nodes closest to users, dramatically reducing access latency.When your users are distributed globally, downloading files from a single origin server is slow. CDN (Content Delivery Network) deploys edge nodes worldwide, caching files at nodes closest to users, dramatically reducing access latency.
| CDN Concept | Description |
|---|---|
| Edge node | Cache servers distributed around the world |
| Origin fetch | When an edge node doesn't have a cached copy, it requests the file from the origin server |
| Cache hit rate | The proportion of requests served directly by edge nodes; the higher the better |
| TTL | Cache validity period; after expiration, a new origin fetch is needed |
| Cache invalidation | Proactively clearing edge node caches to make new files take effect |
- Add hash to filenames: logo.a3f2b1.png instead of logo.png, so you don't need to invalidate cache when updating files - Set reasonable TTLs: Static assets (JS/CSS/images) get long TTLs (1 year), HTML gets short TTLs (5 minutes) - Enable Gzip/Brotli compression: Text-based resources can be reduced by 60-80% after compression- Add hash to filenames: logo.a3f2b1.png instead of logo.png, so you don't need to invalidate cache when updating files - Set reasonable TTLs: Static assets (JS/CSS/images) get long TTLs (1 year), HTML gets short TTLs (5 minutes) - Enable Gzip/Brotli compression: Text-based resources can be reduced by 60-80% after compression
------
| Practice | Description | Example |
|---|---|---|
| Key naming convention | Use meaningful prefixes to organize files | {type}/{date}/{uuid}.{ext} |
| Avoid hot spot Keys | Don't start with incrementing numbers | Use UUID or hash prefixes |
| Minimum necessary permissions | Bucket defaults to private | Only set public-read for files that need to be public |
| Lifecycle rules | Automatically clean up expired files | Temp files auto-deleted after 7 days |
| CORS configuration | Frontend direct upload requires CORS setup | Allow your domain to PUT/POST |
| Server-side encryption | Enable SSE for sensitive files | SSE-S3 or SSE-KMS |
------
File storage is a fundamental issue that every web application encounters. Object storage, with its infinite capacity, low cost, and high availability, has become the standard choice for internet applications.File storage is a fundamental issue that every web application encounters. Object storage, with its infinite capacity, low cost, and high availability, has become the standard choice for internet applications.
Key takeaways from this chapter:Key takeaways from this chapter: