Managed Storage

Managed storage is selected by access pattern: object storage for immutable blobs and datasets, block storage for attached disks, file storage for POSIX-like shared paths, warehouses for analytical tables, databases for serving state, and caches for repeated low-latency reads. The wrong abstraction creates both performance and cost management problems.

Storage service types

Each type is defined by its access pattern, and mismatches are the usual source of cost and latency surprises:

Type	Access pattern	Good fit	Poor fit
Object	key-addressed blob GET/PUT	datasets, model artifacts, backups	low-latency mutation, POSIX I/O
Block	attached disk, random R/W	database volumes, boot disks	shared multi-host access
File	POSIX shared path	shared home dirs, legacy apps	massive object throughput
Warehouse	analytical SQL over columns	facts and dimensions, BI queries	transactional row serving
Database	indexed row or key serving	application state, low-latency reads	large analytical scans
Cache	in-memory key lookup	hot repeated reads	durable storage of record

Object stores such as S3 and Cloud Storage expose buckets, object keys, metadata, IAM, lifecycle rules, and storage classes. They are excellent for cloud storage, model artifacts, and distributed data processing inputs. They are not low-latency mutable filesystems. A practical storage contract should state:

flowchart LR
  Shape[Data shape] --> Pattern[Read and write pattern]
  Pattern --> Consistency[Consistency need]
  Consistency --> Retention[Retention]
  Retention --> Recovery[Recovery target]
  Recovery --> Class[Storage class]

Lifecycle policy is part of the mechanism, not cleanup afterthought. For example, training checkpoints might stay in frequent-access storage for 14 days, transition to cold storage for 90 days, then expire after model governance requirements are met.

Worked small-object check

Some infrequent-access object classes have minimum billable object sizes. For 50 million feature fragments of 32 KiB each, the physical payload is

$$\frac{50,000,000\times 32\text{ KiB}}{1024^3}\approx 1.49\text{ TiB}.$$

If each object is billed as at least 128 KiB, the billable storage becomes

$$\frac{50,000,000\times 128\text{ KiB}}{1024^3}\approx 5.96\text{ TiB},$$

which is a $5.96/1.49=4.0\times$ multiplier. The fix is architectural: compact small records into Parquet/Avro shards or a table format before moving them to colder classes. Otherwise storage and decoding bottlenecks show up as slow listing, excess requests, and poor scan throughput.

Caveats

Durability and availability are different. Archive classes can be durable but have minimum durations, retrieval fees, or lower availability. Replication improves recovery but adds write amplification and egress. Backups are only reliable after restore tests prove that credentials, schemas, and dependencies still work.

References

• What is Amazon S3?
• Amazon S3 pricing
• Google Cloud Storage classes

Nav

Section — Cloud and Distributed Systems

← Managed Compute GPU Systems →