SOLUTIONS FOR ANALYSIS OF NETWORK SECURITY DREAMS – SANDVINE Copy

EBOF Storage is an Evolution of JBOF

As we are all aware, a huge amount of data is created every day. This represents a problem for hyperscale data centers that have to store that data and make it available to servers and applications. Data centers are under pressure to manage their data storage and are constantly seeking any advantage that new technology may provide. Data center servers can store a limited amount of data internally, this “direct-attached” storage delivers fast, high bandwidth, and low-latency access to that data. In some data centers, more storage is pooled in devices that are connected to servers in the same rack. However, larger centralized data storage is typically made available to data center servers over storage area networks (SANs), which are often based on expensive Fibre Channel or InfiniBand standards rather than Ethernet.

Storage devices have rapidly evolved over recent years, older hard disk drives (HDDs) based on electro-mechanical magnetic storage technologies are now increasingly being replaced with new solid-state drives (SSDs) that use fast NAND flash memory. SSD devices provide significantly improved performance over HDDs, use less power, and are more reliable. For hyperscale data centers looking to improve efficiency, new storage solutions that implement SSDs are of great interest. Ideally, data centers need large centralized SSD storage that can be accessed over a network at speeds as fast as any direct-attached storage. This has driven the development of “Just a Bunch of Flash” (JBOF) storage devices that deliver fast access to data on pooled SSDs over a network fabric. An extension to JBOF storage is Ethernet Bunch of Flash (EBOF), which provides an Ethernet network connection directly to each pooled SSD device. EBOF storage solutions deliver high performance at lower cost, offer huge scale-out capability, and further enable the disaggregation of storage and compute in the data center.

EBOF and NVMe Storage Speed

Three main factors have an impact on speed when data is transferred from any storage to a server CPU for application processing:

• IOPS: Storage device speed is often measured in IOPS, or input/output operations per second. That is, how fast data can be read or written to or from the storage. Generally, HDDs have an IOPS up to a few hundred, whereas SSDs have IOPS in the hundreds of thousands.
• Bandwidth: The connecting bus and network bandwidth can limit the speed with which data is transferred. The Serial Advanced Technology Attachment (SATA) bus has a maximum bandwidth of 6 Gbps, and the Serial Attached SCSI (SAS) bus is 12 Gbps. Storage that has a Non-Volatile Memory Express (NVMe) interface that directly connects to a host Peripheral Component Interconnect Express (PCIe) bus is limited by the PCIe bandwidth, which is 16 Gbps per lane for PCIe 4.0. Compared to bus bandwidths, network fabric (such as Ethernet) can be as high as 100 Gbps.
• Latency: The time that it takes to execute an I/O storage command. For any storage device and in the connections to a host, there is latency in the device, host adapter, switches, and network protocols, etc. The average latency in HDDs is typically about 10 milliseconds, but for SSDs it is measured in microseconds, although there is a large difference between read and write operations. Compared to storage device latency, network device latency can be less than a microsecond, so even several network switch hops do not contribute much to latency. Other network latencies from adapters and protocol implementations can add up to another 10 microseconds.

For fast storage access, it is clear that SSD storage is many times faster than HDD. In addition, new NVMe interfaces and direct PCIe bus connections to host CPUs provide another significant bandwidth and speed advantage. Low latency is also important for storage performance and NVMe SSDs again deliver this. For remote storage, the latencies vary depending on the connecting network fabric, but most network devices are generally all low latency.

NVMe-oF™ Enables EBOF

As often happens with older protocols and standards, technology just overtakes them. This is definitely the case with older HDD drives that connect to a host system through SATA III or SAS interfaces. By replacing mechanical HDD drives with super-fast SSD drives, the interface becomes a bottleneck that limits the SSD performance. The only option is to replace the interface specification with a new faster standard. The Non-Volatile Memory Express (NVMe) standard was designed specifically for the fast NAND flash memory access in SSD drives. NVMe allows SSD drives to connect directly to host system high-speed PCIe bus providing low latency memory access without a high CPU overhead. In addition, NVMe supports parallel read/write requests and a massive 64K queue and command depth – that is, 64,000 queues with up to 64,000 commands per queue! The adoption of NVMe for SSD drives unleashes the full speed of the flash memory, enabling over a million IOPS to be achieved.

Accton’s EBOF Platform

To support and develop EBOF storage solutions, KIOXIA® (formerly Toshiba Memory America, Inc.) has started to produce Ethernet SSDs with an integrated Marvell® NVMe-oF Ethernet SSD converter controller that provides a 25 Gbps Ethernet interface connection. Accton’s EBOF platform includes 24 KIOXIA® Ethernet SSDs contained in a 2RU chassis. The Ethernet SSDs connect internally to dual switch units in the rear of the chassis that each provide six 200 Gbps connections to the Ethernet network. In addition, it is possible to use the 200 Gbps ports for daisy-chain links to other EBOF units.

The Accton EBOF device is one of the industry’s first native NVMe-oF over TCP/IP all-flash array systems, which offers a total 66.4 GB/s of storage throughput to the 24 Ethernet SSDs and 2.4 Tbps of network bandwidth connectivity. Each Ethernet SSD can support up to 7680 GB capacity and all 24 SSDs together deliver over 16M IOPS in total. The high-performance of the EBOF platform approaches that of PCIe direct-attached storage, delivering disaggregated storage at a lower total cost of ownership and reduced power consumption.