Object Storage Shard Calculator

When most folks think of building an object storage cluster they focus on amount of disk space available. Buy four servers, slap 16 drives into each one, hit your terabyte goal and call it a day. On paper this looks great, but when put under load the cluster is likely to fail as metadata layer buckles long before disks are full. The beauty of object storage is that it’s less concerned with raw capacity then it is with how you slice up the namespace.

Enter shard planning. After plugging in your protection scheme and workload profile, the calculator do the math for you. It turns abstract concepts (such as erasure coding overhead and bucket indexing) into real numbers, and spares you the guesswork involved in converting units and fiddling with coefficients. Most importantly, it takes into account how many objects there is (not just how much data).

How to Plan Your Object Storage Cluster

A single-node system can handle a two-terabyte archive of large video files without breaking a sweat, but what about a two-terabyte telemetry bucket of kilobyte-sized JSON files? Not so much. Those tiny objects create index keys that consume CPU cycles and memory unrelated to disk space. Without considering the object count, you’ll starve your metadata service.

Then there’s the additional complexity of erasure coding, which a lot of home lab admins gloss over. Keeping a copy (or two, or three) is nice, it’s replication, and replication is simple. Keep two or three copies and move along. It wastes some space, but math is easy. With erasure coding, your data gets sliced up into chunks and spread out across multiple nodes. It saves space and often cuts raw overhead in half. But if one drive go bad, it puts a heavy burden on rebuilds.

That’s made clear in the calculator: you see the footprint, and if you opt for something wide such as eight plus three parity, then you’re saving quite a bit of disk capacity. The price you pay for that savings? Rebuild time, wide erasure coding takes quite a while to rebuild a node, because it has to read from lots of other nodes simultaneously. If your network isn’t fast, that rebuild window can be a dangerous time, leaving the cluster exposed to another failure.

The big danger is metadata memory. For every object there’s an entry in index which contains its placement info, tags, version markers and key. So as your objects gets smaller, your metadata footprint gets bigger compared to your data. That’s why the tool will estimate how much RAM it requires and help you properly size your servers. It’s all too easy to grab plenty of disk but not realize that the server hosting the bucket index also has to have sufficient RAM to hold those keys in memory. When it starts spilling over into disk, things start getting slower and it will take a while to list a bucket containing millions of objects.

To see this play out for different workloads check out the reference table on the page. You’ll notice that for equal amounts of total data, telemetry logs requires many more shards than media archives.

Another trap is bucket concentration. That means if you have one app writing 90 percent of your objects, then that one bucket is now a hotspot. Everything else looks like it is running at a good level in terms of average cluster loads, but that index shard are still overheating. Your sharding plan needs to address the peak write rate, not simply the average. A safety buffer will help here as well; it provides breathing room for the system when things aren’t perfectly distributed. Production workloads rarely grows or spike linearly.

The other side of the equation is planning for growth. Fifty percent growth is fine; maybe that’s how much data you’ll have tomorrow or this month. But what about next year? How many more objects will you have then? That should of been reflected in how many shards you’ve planned for. Can you rebalance shards on a running cluster? Yes, but it’s hard. It takes time (slowing down I/O) and it sends data across the wire. Better to err a little toward having too many shards initially rather than having to add some later on. The calculator takes into account your anticipated annual growth rate: you get to view the current situation, plus the prediction.

There’s nothing like object storage. There’s no free lunch. You trade raw capacity for resilience. You pay for metadata overhead by trading away performance. You pay for rebuild speed by trading away disk savings. Balance is the name of the game. Your cluster should be able to handle your worst case scenario without wasting money on idle hardware.

Think about what you want to store first: object count. Respect the metadata layer. Plan for the rebuild. If your cluster lasts, it won’t be because of how much you stored; it’ll be how gracefully you were able to handle data loss without losing the cluster itself.