In the previous post I patched the NVIDIA driver to enable GPUDirect RDMA on my GeForce GPUs. The motivation was Priority Flow Control which requires the NIC’s receive buffer to fill up before it sends a PAUSE frame and with standard CPU-memory RDMA the NIC drains its buffer to DRAM faster than the wire can fill it. GPUDirect changes this by targeting the GPU memory instead of system RAM, introducing enough PCIe latency to create backpressure.
In the previous post I built a software congestion point using a Linux bridge with TC RED to exercise the full DCQCN feedback loop on my point-to-point lab. That experiment successfully triggered ECN marking and CNP generation but ended with a gap: no PFC. Priority Flow Control is the other half of lossless RoCEv2, the L2 backstop mechanism that prevents packet drops when DCQCN can’t reduce rates fast enough. Without PFC the congestion control picture was incomplete.
In the RDMA lab I built a two-node setup using dual-port ConnectX-4 cards with PF passthrough, connected point-to-point over fiber. That setup is great for simple RDMA performance testing but I wanted to go further and explore how congestion control works in RoCEv2 networks.
After my EPYC workstation I wanted to go further: experiment with multi-node GPU networking and learn RDMA, RoCE and the networking technologies behind today’s AI datacenters. My initial plan was ambitious but so were the prices. Even a small, already outdated setup was prohibitively expensive. Here’s how I ended up achieving similar learning goals for much less.
After running a power-efficient home lab setup for quite some time, I recently decided it was time for an upgrade. My existing machines were chosen primarily for their small form factors and energy efficiency, but new requirements—particularly the need for GPU acceleration—meant it was time to think bigger. Here’s how I approached building a more powerful workstation while maintaining a practical balance of performance, cost and expandability.