After our last upgrade to our storage devices with multiple NVMe disks we run into the issue that the disk bandwidth supported overcome the capacity of our storage network connections. The new storage does 4714.3 MB/s (Read) and 4363.1 MB/s (Write) per Storage Volume. These volumes are then multiplexed per availability zone and can reach up to 21GB/s for Read and 17GB/s for Write operations.
These new storage zones had the side effect of completely saturating the 40Gbps connections on each storage zone, during the initialisation of our storage zones this morning we managed to run in full capacity the local backbone (2x40Gbps) which resulted in minor delays in packet delivery for http(s) traffic, it was fortunate that our QoS rules were automatically applied by Halcyon and the packet loss was less than 0.7%.
Today we will be adding another 40Gbps on the storage nodes (20G per zone) and we will be ordering the needed hardware to reach the needed 80GB per zone that we require to fully support our new storage nodes, till then we will be rate limiting our storage nodes to a maximum of 7.5GB/s.
There will be no downtime on any of our services during the network maintenance and storage changes that will be performed today.
Regards,
eSG NOC
Thursday, March 15, 2018
