True - the great thing is that we have been optimizing for this type of interface for the last decade. I.e., due to the benefits of making writes sequential (both for HDDs and SSDs)
We, the industry, have just been missing the interface to actually perfectly align our workloads to the media that we store the data on. The zone interface bridges this gab.
As SSDs get denser, moving to QLC/PLC, the number af writes to the same place on media gets to the low hundreds.
By using zones, the total amount of available host writes increases by 4-5x (when considering non-optimized file-systems). Technically the device write amplification is reduced from 4-5x to ~1x.
Half of the world bits from HDDs are estimated to be on SMR in 2023 - the gains are significant when deploying at scale.
For SSDs, it gets even more fun as zones aligns with the characteristics of the media and you get this effect of significant increase in capacity (20% with a 28% OP drive), order of magnitude reduction in dram, and eliminates device side garbage collection on the drive (commonly between >1-5), which improves the QoS considerably.
Additionally, one can now run the drives at 100% capacity utilization - conventional drives becomes slower due to increasing device write amplification.
It is worth mentioning that the Linux kernel has a new kernel API (io_uring) that changes the whole argument around using libos designs. With the new io_uring library (available with Linux kernel 5.1), peak IOPS per core is 1.7M IOPS... Which beats or is close to SPDK performance[0]. Later updates to the patches improves the throughput even more.
Jens' benchmark for SPDK quoted there is far off from the numbers we (the SPDK community) measure. We are able to replicate his io_uring numbers though, so we agree that the new interface is a large improvement. We're working to make full benchmarking data available shortly.
io_uring dumps data directly into a ring buffer mapped into the user-level address space. User code is notified by (at least) an updated atomic counter. The user process must be finished with the data before the kernel comes around again to overwrite it. Often that demands the user process or thread is bound to a core which the OS has been forbidden to run anything else on, and the thread does a carefully circumscribed amount of work, rarely including memory allocation, i/o, or even system calls, that may cause it to be "lapped" by subsequent writes.
The idea is that the average time to process a packet absolutely must not exceed the average arrival rate, and the sum of spikes in arrival rate must average out over the size of the ring buffer to less than the process rate.
The hamster process pulling from the ring may just be load balancing to a herd of other threads operating under less stringent conditions, so they might be permitted i/o.
This statement should be seen as there's no way to scale the old block layer to new devices. To current SSDs, its already useful, in that it decreases latency and CPU usage for current generation of drives.
It's currently only enabled using the virtioblk driver. But there's work underway to make the scsi layer and all the others drivers use it (already patches out for the mtip and nvme driver).
Thanks for the clarification. IIRC you are a co-author of the paper, so perhaps you can answer a follow-up question.
What kind of latency or CPU usage change should a typical modern SSD on an amd64 class multicore processor observe when using the new block layer?
Also correct me if I'm wrong but since Linux aggressively caches already and SSDs are already way faster than older drives for normal (ie. ~random access) loads, plus RAM is cheap and plentiful these days, I am guessing that very few applications will honestly be IO-bound enough to see that benefit.
One thread issuing IOs: A reduction of 2x in the IO path latency isn't unusual. The overhead of the code path drops from 5us to around 2us. When there's multiple IO threads, the gain is much higher (to 38x in the 8 socket setup). Thus, the more complex workload, the better performance.
I don't have any up-to-date numbers on CPU usage. When we did the experiments on the mtip drive, it was around 20% less CPU usage when performing roughly the same IOs.
For a typical workstation workload, the SSDs access times are still too high to feel the reduced latency. A typical modern SSD is around 50-100us for an IO access. The win there will be the lesser CPU usage that free up resources for other things to do.
Applications are still bound by the round-trip time of getting IOs. Just because we get more memory, we still have to persist data at intervals to prevent data loss, and everything that can help in decreasing the overhead is a win.
There are certainly plenty of contexts where SSD's in general provide limited if any performance wins because disk I/O is largely not involved. However, in cases where SSD's are being used for performance reasons, particularly for random reads, I would expect this would make a fair bit of difference.
