Replicated network storage vs local NVMe

The oldest storage argument in hosting: local NVMe is fast but dies with the host; network storage survives anything but costs you latency. We put that to a measurement. One identical fio battery, four disks: the local NVMe of a premium Hetzner instance, the local NVMe of a premium DigitalOcean instance, and ServersCamp's two replicated network pools.

Every number below is from a run on 2026-07-18, fio 3.41, one pass, no retries, raw JSON kept for every test. The battery and methodology are published so you can run the same thing on your own machines.

TL;DR

The contenders

Both competitor instances are from their premium dedicated-CPU lines, not budget tiers, and both test their local NVMe: the fastest storage either provider sells at this size. Neither had a network volume attached; this is deliberately the hardest comparison for network storage, their strongest disk against ours. The ServersCamp disks are network-attached and replicated: every write lands on independent storage nodes before it is acknowledged.

Methodology

One battery, 16 tests, 30 seconds each with a 5-second ramp: {4k random, 1M sequential} × {QD1, parallel} × {read, write, 50/50 mixed, fsync'd write}. All tests run with direct=1, ioengine=libaio, on an 8 GiB file on a fresh ext4 filesystem. Parallel means 8 jobs × QD64 for 4k (512 outstanding IOs, enough to saturate a 200k IOPS limit) and 4 jobs × QD16 for sequential, spread across the file with offset_increment.

• This is a buyer-facing benchmark, not a normalized hardware lab test. Each provider was measured on the best comparable instance and storage option it sells in this segment. CPU, hypervisor and IO stack are part of the product, so they are part of the measurement: the thing under test is what a customer VM actually gets.
• Same fio version everywhere (3.41), same battery script, one run per disk, no cherry-picking between runs.
• fsync tests issue fsync after every write: the write-ahead-log pattern of every transactional database.
• Raw fio JSON and system info for every test on every machine are preserved and available on request.

4k random, QD1: the latency a query feels

QD1 is one operation at a time, each waiting for the previous: the pattern of index lookups and any chain of dependent reads. This is where storage latency turns directly into application latency.

Reads: the network disk wins outright. 81 µs on a replicated network volume against 112 and 131 µs on local NVMe. The fabric moves data over a direct memory path between machines, and the round trip comes in under the local IO stacks of both premium instances. This is the result that breaks the intuition this article set out to test.

Writes: local caches win, and it is fair to say why. A local NVMe acknowledges a write the moment it lands in the drive's cache, no network involved. Our write is acknowledged after it lands on independent storage nodes. 39 µs versus 65 µs is the price of the copy existing somewhere else, and the fsync section below shows what each of those acknowledgements is actually worth.

4k random, parallel: throughput under load

Both ServersCamp disks carry a 200 000 IOPS tier limit, and both ran into it: 199 982 reads on R5, 199 617 writes on W5. The limits are enforced by the hypervisor and they are exact; the number on the card is a delivery, and here it is also higher than what either local NVMe produced. The local disks are not throttled at all, this is simply as fast as they went.

fsync: the result we did not expect to win

An fsync'd write is a promise: the data is durable before the call returns. Databases live and die by this pattern; it is also where network storage is supposed to be hopeless, because every promise costs a round trip.

A single committer is fastest on DO's local disk: 8 826 durable writes per second at 65 µs per fsync. Our W5 does 3 560. If your workload is one process committing serially, local NVMe is the right tool and this table says so.

Parallel committers flip the table. With multiple writers, the pattern of any real database under load, W5 pushed 147 041 fsync'd writes per second: 1.4x DO's local disk and 1.5x Hetzner's. The Database pool runs on enterprise NVMe with power-loss protection: a flush into protected cache is already durable, so commits complete at full device speed without waiting on media, and the fabric aggregates many writers better than a single local drive. A replicated network disk out-committing the local NVMe of instances costing €86 and $161 is the headline we did not plan for.

1M sequential: bandwidth

* Sequential bandwidth is a property of the shared fabric and identical for both pools; figures from the Read-pool volume run.

Raw parallel bandwidth belongs to the local disks: Hetzner's NVMe streams 9.2 GB/s and DO's reads at a spectacular 14.2 GB/s, while our tier tops out at its advertised 4 GB/s ceiling (which the run hit exactly: 4.29 GB/s measured against a 4 GiB/s limit). Two footnotes worth reading, though: at a single stream, the pattern of one backup or one large file copy, the network disk's 3.89 GB/s beats both locals; and DO's local disk collapses to 1.9 GB/s on parallel writes at 35 ms latency, below even our throttled figure. Fast local reads do not guarantee fast local writes.

Credit where it is due. On these premium lines both providers ship genuinely good disks: tight QD1 tails (p99 146 µs on the c5, 183 µs on the CCX23), six-figure IOPS, and rates that held flat across every 30-second run with no burst-credit cliffs. The numbers look like unshared, visibly un-oversold hardware, exactly what €86 and $161 a month should buy. The budget section below shows how quickly that stops being a given further down the price list.

What the numbers cannot show

• The local disks in this test die with their host. One hardware failure and the 14 GB/s NVMe is a paperweight; recovery is a restore onto new hardware. The ServersCamp volumes are replicated across nodes: a host failure restarts the VM elsewhere with the disk intact, and planned maintenance live-migrates VMs with storage attached.
• The local disks are fixed. 160 GB on the CCX23, 250 GB on the c5. The network volumes resize online and change speed tiers in place, with the data staying put.
• Every ServersCamp figure here is a throttle, not a peak. The same fabric serves higher and lower tiers; you buy the number you need and the hypervisor enforces exactly that.

The budget tier: the same battery on cheap VMs

The premium tier is where providers put their best silicon. The budget tier is where storage corners get cut first, so we ran the identical battery there too.

On Basic-generation hardware the ServersCamp disks are clamped at 50 000 IOPS and 1 GB/s: the real ceiling of what these guests can drive, stated on the card. The competitor budget disks carry no published limits at all, which cuts both ways, as the table shows.

Three things stand out:

• The group-commit gap widens at the bottom. W3 pushed 46 593 parallel fsync'd writes per second against 10 695 on the CX43 and 16 573 on the DO droplet: a 2.8-4.4x lead for the replicated volume with power-loss protection, on the cheapest hardware we sell.
• The clamps held again. 49 981, 49 980 and 50 030 IOPS against a 50 000 limit; 1 074 MB/s against a 1 GiB/s limit. Two tiers, four disks, every limit delivered to a fraction of a percent.
• The $48 droplet trails the €12 legacy box on everything. 371 µs QD1 reads with a p99 near 2 ms, 33k IOPS, 464 single commits per second. Budget-tier storage varies wildly between providers, which is exactly why published limits are worth having.

The cheap VM is not a toy

€30-33 a month buys 4 Sustained vCPUs, 8 GB of RAM and a replicated disk that just did 46 593 durable commits per second. That is a base for a small production system, not a sandbox. And that is the point of the platform: the €2 entry VM exists, but the path upward is not a migration project. You start cheap, switch compute profiles live, retier the disk in place, and keep the same operational model all the way to 200k IOPS on High Frequency hardware.

The verdict: what to pick for what

The pattern is consistent: local NVMe keeps its two physics advantages, cache-speed single writes and unthrottled sequential streaming. Everything a database does under load, random reads, parallel writes, durable commits, went to the replicated network volumes in this test, on top of the durability they exist for. If your bulk-streaming data is disposable, ServersCamp's answer to that use case is the single-copy Solo disk class, which was not part of this battery.

Reproduce it

The whole battery, complete and copy-pasteable. Each test runs 30 seconds after a 5-second ramp, on an 8 GiB file:

# shared flags
C="--direct=1 --ioengine=libaio --time_based --runtime=30 --ramp_time=5 \
   --size=8G --filename=/mnt/bench/fiotest --group_reporting"

# 4k random, QD1
fio --name=4k-qd1-read   $C --rw=randread  --bs=4k --iodepth=1
fio --name=4k-qd1-write  $C --rw=randwrite --bs=4k --iodepth=1
fio --name=4k-qd1-rw     $C --rw=randrw --rwmixread=50 --bs=4k --iodepth=1
fio --name=4k-qd1-fsync  $C --rw=randwrite --bs=4k --iodepth=1 --fsync=1

# 4k random, parallel (512 outstanding on the IOPS tests)
fio --name=4k-multi-read  $C --rw=randread  --bs=4k --iodepth=64 --numjobs=8
fio --name=4k-multi-write $C --rw=randwrite --bs=4k --iodepth=64 --numjobs=8
fio --name=4k-multi-rw    $C --rw=randrw --rwmixread=50 --bs=4k --iodepth=64 --numjobs=8
fio --name=4k-multi-fsync $C --rw=randwrite --bs=4k --iodepth=64 --numjobs=4 --fsync=1

# 1M sequential, QD1
fio --name=seq-qd1-read   $C --rw=read  --bs=1M --iodepth=1
fio --name=seq-qd1-write  $C --rw=write --bs=1M --iodepth=1
fio --name=seq-qd1-rw     $C --rw=rw --rwmixread=50 --bs=1M --iodepth=1
fio --name=seq-qd1-fsync  $C --rw=write --bs=1M --iodepth=1 --fsync=1

# 1M sequential, parallel
fio --name=seq-multi-read  $C --rw=read  --bs=1M --iodepth=16 --numjobs=4 --offset_increment=2G
fio --name=seq-multi-write $C --rw=write --bs=1M --iodepth=16 --numjobs=4 --offset_increment=2G
fio --name=seq-multi-rw    $C --rw=rw --rwmixread=50 --bs=1M --iodepth=16 --numjobs=4 --offset_increment=2G
fio --name=seq-multi-fsync $C --rw=write --bs=1M --iodepth=16 --numjobs=4 --offset_increment=2G --fsync=1

Run it on any ServersCamp VM and compare against your current provider. If your numbers on our disks come out below what this page claims, that is a bug: tell us in a ticket.

Disclaimers

• One run per disk, on 2026-07-18. Storage performance varies with fabric load and neighbors; we publish single honest passes, not the best of N.
• Disk sizes differ where the providers fix them (160 GB Hetzner, 250 GB elsewhere). All tests use the same 8 GiB working set, so the size difference does not affect the IO paths measured.
• Both competitor disks are local NVMe, ours are replicated network volumes: a deliberate apples-to-durable-oranges comparison, stated openly. Their network volume products (Hetzner Volumes, DO Volumes) are slower than their local disks and were not attached to these instances.
• Guest CPUs differ (EPYC-Milan, Xeon Platinum 8358, EPYC 4565P), and that is by design, not an apology. At 200k IOPS the guest's silicon and IO stack matter, and each provider sells the stack it sells. A buyer does not get to normalize the CPU out of the VM they paid for; neither did we.
• Prices as shown on the providers' pages on the test date. The ServersCamp totals include the Sustained-profile VM, the disk and a public IP.
• Budget-tier runs used the same battery and fio 3.41 everywhere except the CX43 (CentOS Stream 9, fio 3.35 from its repos). The CX43 is a legacy plan running production-adjacent software; its numbers may flatter or hurt it accordingly.