We tested the Bluefield-2 DPU’s capability to handle OVN/OVS offloading within the FishOS environment. The results showed smooth operation, reduced latency, and freed CPU resources, demonstrating the potential of FishOS and Bluefield DPUs to optimize modern data center operations.
FishOS: Enhancing Cloud Management
FishOS by Sardina Systems is an advanced cloud management platform based on OpenStack. Designed to boost the efficiency and scalability of modern data centers, FishOS leverages automation and intelligent resource management to streamline operations, cut costs, and improve overall performance.
Nvidia Bluefield DPUs: Revolutionizing Data Processing
Nvidia Bluefield DPUs are specialized processors that offload and accelerate networking, storage, and security tasks, freeing up the main CPUs. These units enhance data center performance and efficiency by handling data-intensive workloads, offering programmability and high performance to meet the demands of modern computing.
Traditional vs. Modern Data Center Architectures
Traditional data centers rely heavily on general-purpose CPUs for all tasks, leading to performance bottlenecks, resource inefficiencies, and higher operational costs. The advent of cloud computing and virtualization has significantly transformed data centers. Cloud computing provides flexible resource allocation, on-demand scalability, and cost efficiency, while virtualization abstracts physical hardware, improving utilization and infrastructure management.
Integrating Bluefield DPUs into data centers addresses these challenges. By offloading intensive tasks from the CPU, they enhance performance and provide hardware-based security features.
Objective:
To test if Bluefield-2 (BF2) can seamlessly handle OVN/OVS offloading within the FishOS OpenStack environment, allowing users to provision VMs.
Test Conditions:
A test platform was deployed with a cloud service provider who allocated hardware with BF2 DPUs. Sardina Systems then deployed the FishOS OpenStack environment on this setup.
Step-by-Step Walkthrough
Deployment: FishOS OpenStack (Bobcat release) was deployed on server nodes running AlmaLinux 9 as the base OS.
BF2 DPU Mode and OS Options: The required BF2 DPU mode was “Embedded CPU.” Various OS options were tested on BF2 DPUs. After studying the documentation, we decided that accounting for all the caveats of a custom operating system for this test was not feasible. Therefore, an official Nvidia build of an Ubuntu-based operating system for BF2 was deployed via the RShim interface using the bfb-install command.
What we discovered during the Deployment stage:
– AlmaLinux 9 installation went on easily via PXE.
– Debian 12 required initramfs adjustments.
– Nvidia DOCA package installation was necessary for proper BF2 functionality.
– RShim interface is required for deployment of Nvidia Ubuntu-based OS on BF2 .
– Bandwidth Testing: Bandwidth between two BF2 DPUs was tested using iPerf3, achieving 100 Gbit/sec.
The BF2 DPU hosts were deployed in the cloud and configured to host OpenStack Neutron OVN agents. We followed the official documentation to enable SmartNIC DPU support in OpenStack: OpenStack Neutron SmartNIC DPU Support.
Offloaded Virtual Machine Networking:
To utilize offloaded virtual machine networking, SR-IOV had to be enabled. While BF2 supports up to 127 virtual functions, the bundled node hardware platform failed to boot with this configuration, so the limit was set to 63 in the BF2 firmware settings.
During virtual machine creation, it was found that OpenStack Nova and Neutron required additional patches for offloaded tenant networking to function correctly.
The Nova compute filtered out the SR-IOV device requested by the Nova conductor. This issue was mitigated by commenting out the filtering line.
The Neutron server refused to bind ports due to a capabilities mismatch, likely a bug.
The line in file neutron/common/ovn/utils.py was changed from this:
to this:
Virtual Machine Testing:
During the VM testing, no significant CPU load was detected on either the main node or the DPU. We also tested BF2 cards in “NIC mode,” where their functionality resembles a ConnectX 6 NIC. One advantage of using “DPU mode” is that no OFED drivers are needed on compute nodes, unlike in “NIC mode.” Therefore, to enable overlay networking offload, we installed the latest Mellanox OFED drivers on the main nodes. This introduced an additional administrative burden for future updates, particularly kernel upgrades (which require rebuilding of modules) and OFED upgrades (which are only supported through complete uninstallation and reinstallation).
The BF2 NIC mode virtual machine test was similar to the DPU mode test, except for using the «direct» type of Neutron ports. The 8-thread iPerf3 test between virtual machines showed approximately 70 Gbps.
Virtual Machine Testing Takeaways:
– Neutron ML2/OVN used GENEVE tunnels for tenant overlay networking.
– A separate tenant network was created with two Neutron ports of the remote-managed type.
– Two virtual machines running AlmaLinux 9 were deployed on separate nodes.
– Single-threaded iPerf3 test: ~25 Gbit/sec.
– 8-threaded iPerf3 test: ~86 Gbit/sec.
– Insignificant CPU load was detected on the main node and DPU.
Summary
The growing demand for high-bandwidth connections to handle massive data volumes, including AI operations, requires efficient solutions to eliminate bottlenecks. FishOS solves this challenge by offloading these bottlenecks to specialized hardware like Bluefield DPUs. By removing necessary networking software from the main node and leveraging advanced technologies like OFED drivers, FishOS significantly enhances performance, reduces latency, and frees up CPU resources for critical application processing. As a result, this provides seamless and efficient data center operations.