Erasure Coding Overhead Calculator

After you’ve got a bunch of disks stuffed inside a server chassis, how many are left over? And what’s the server holding back as part of the file system protection?

If you use erasure coding, it’ll break up your data into shards that get spreaded out onto different disks. So if some goes missing, you won’t lose your files. But there’s no free lunch here. Rebuilding from scratch takes time, and there’s an overhead cost in terms of matching. Raw disk capacity comes with a price tag, too.

How to Use the Erasure Coding Calculator

Plug in your number of drives and type of coding on top and the calculator will work the numbers out for you… No need to guess at what conversion factor or coefficient applies.

K plus m is the core idea here. K equals the data shards containing your real data. M equals the parity shards needed for rebuilding when a drive go down. A typical home lab deployment will use four drives for data and two for parity. That’s sixty-six percent efficiency, i.e., two-thirds of available disk space. But amount of parity data used to protect information is 50% larger.

By moving to a six plus two layout, you achieve seventy-five percent efficiency. You get more bang for your buck on parity. But more expense during rebuild: Each time a drive die in a six plus two stripe set, the system needs to draw from five surviving data drives to rebuild. It is efficient at rest but expensive in motion.

But also take into account what kind of media it’s going to be. A fast seven-thousand-two-hundred RPM enterprise disk will rebuild far faster than a slower five-thousand four-hundred RPM NAS drive. These throughput numbers can vary widely. According to the table on the page, a mechanical drive could rebuild as slow as ninety-five megabytes per second, while an NVMe SSD can reach one thousand two hundred.

Why does this matter? Because the longer your rebuild takes, the higher your chance of having another drive fail before the initial one has been fixed. If you have a big parity group rebuilding over a long time and two drives goes down in the process, you’re out of luck. This is why calculator asks for both a degraded rebuild reserve and target maximum fill percentage.

I’m not being conservative when I leave my pool 80% full. This leaves room for storage engine to breathe. To do garbage collection of SSDs. It handles metadata updates. It also performs backfill operations where data are moved around in distributed systems. If you pack every terabyte without leaving any space, the system slows down as it looks for empty slots where it can write its new parity information.

Another strong way is adding a compression ratio. If you’re storing un-compressed video, databases or text logs, then you may get a ten percent gain. But if you are already compressing media files or want to encrypt backups, the ratio doesn’t change. It stays one to one, which means capacity is actualy what the math says.

What about layout? You will want different layouts depending on the type of failure you expect in your fault domain. If you’re storing a media archive and don’t change much, you can get four-shard tolerance and huge capacity by going 10 plus four. It takes ten drives, but they all function as a single unit.

If space isn’t an issue, then go with safety: 3 plus 2 protects against two drives in a very small subset failing at once. If you have a busy home server that does lots of writes, go with a lower k, it’ll keep the write penalty down. If you’re doing more long-haul stuff, go with a higher m and rest easy knowing that you’ve got some peace of mind.

The tool also figures out how long it would of take to rebuild one shard, so you can kind of visualize how long that’s taking. That makes a difference when planning your maintenance windows, too. It could be a six-hour rebuild instead of a 12-hour rebuild.

In conclusion, erasure coding means sacrificing some space to make it safer. Storage is not free. You can’t get maximum speed + maximum safety + maximum density, you’ve got to choose two. Play with the presets; they’ll show you what impact various profiles have on the numbers. Try a deep archive vault versus a mini Ceph cluster, and look at how the usable capacity scales when the parity changes in size.

The point here is to configure a pool that withstands expected hardware failure while not forcing you to buy more disk space than you wanted. Begin with real-world hardware inputs, tweak the fill levels based off comfort, and then let the numbers inform your buying decisions. Your data will appreciate the foresight.