Ceph

Use it when

• On-premises S3-compatible storage at real scale is the requirement; RADOS Gateway is the default answer under OpenStack and most private clouds.
• One cluster should serve three interfaces: S3 objects, RBD block devices for VMs, and CephFS POSIX filesystems.
• Multi-site is serious: RGW multisite replicates asynchronously, active-active, with per-bucket sync policies.
• The S3 surface must be deep: Object Lock, storage classes mapped to pools, lifecycle transitions, and bucket notifications to Kafka or AMQP.

Think twice when

• The team cannot own a large C++ distributed system; CRUSH maps, placement-group counts, and recovery behavior under failure are the operator's problem, and ops complexity is the highest in this capability.
• The deployment is small or edge-shaped, where MinIO or Garage does the job with a fraction of the surface.
• Hardware is thin or links are slow; Ceph assumes datacenter-grade nodes and networks.
• Cloud object storage is an option at your scale and egress profile.

How it runs

Self-hosted on commodity hardware: monitors and managers for quorum (three nodes minimum, five recommended), OSDs per disk, and gateways per interface. Placement is computed from the CRUSH map rather than looked up centrally, which is what lets the cluster grow without a central index becoming the limit. Durability is 3x replication or erasure coding, your choice per pool.