Session

AI serving workloads are pushing past host memory hierarchies. Long-context KV caches, embedding stores, and vector databases for RAG need large capacity, high bandwidth, and low random-access latency. Composable memory fabrics based on fabric-attached CXL memory move compute close to data, where smart memory devices run search, cache lookup, compression, and quantization near device-local DDR. This wins on data movement, latency, and memory scaling, but it creates a Linux networking problem. In deployment these functions become cloud native services such as Redis, KV-cache managers and Milvus QueryNodes. Those services need IP-reachable endpoints. The CXL Type-2 device that runs them exposes a memory window, not a NIC.

In this talk, we present the Linux virtual-interface path we built for that gap, using only stock kernel pieces. A host daemon and device daemon open /dev/net/tun, configure a virtual L3 interface, and mmap the same CXL HDM-H window as a shared packet ring. ping, ssh, Redis, and TCP services work end-to-end with no new module on either side. We walk through the shared-ring design and the ordering rules needed to carry packets reliably over CXL memory and also the new host-pod bridge pattern created for this which gives device-side Linux a cluster-facing service identity over the CXL-backed virtual link, so the service is discovered, scaled, and reached by service IP while clients stay unaware of the memory window underneath. We then run those services over this path without application changes. Data-intensive work stays on device-local DDR while Linux-native networking carries only control and result traffic where our measurements which show 60x less host-link traffic than a host-side compute path, and 2.56x FAISS vector-search throughput scaling across four cards.

Networking composable memory for AI is moving fast, and the design space is wide open. Getting it right in Linux will shape how AI services scale. We bring this deliberately simple path, with the data behind it, to Netdev to spark collaboration on Linux networking mechanisms for composable memory fabrics serving AI workloads.