Processing Particle Data Flows with SmartNICs
- Univ. of California, Santa Cruz, CA (United States)
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
- Sandia National Lab. (SNL-CA), Livermore, CA (United States)
Many distributed applications implement complex data flows and need a flexible mechanism for routing data between producers and consumers. Recent advances in programmable network interface cards, or SmartNICs, represent an opportunity to offload data-flow tasks into the network fabric, thereby freeing the hosts to perform other work. System architects in this space face multiple questions about the best way to leverage SmartNICs as processing elements in data flows. In this paper, we advocate the use of Apache Arrow as a foundation for implementing data-flow tasks on SmartNICs. We report on our experiences adapting a partitioning algorithm for particle data to Apache Arrow and measure the on-card processing performance for the BlueField-2 SmartNIC. Our experiments confirm that the BlueField-2’s (de)compression hardware can have a significant impact on in-transit workflows where data must be unpacked, processed, and repacked.
- Research Organization:
- Sandia National Lab. (SNL-CA), Livermore, CA (United States); Sandia National Laboratories,, Albuquerque, NM
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE National Nuclear Security Administration (NNSA)
- DOE Contract Number:
- NA0003525
- OSTI ID:
- 1892372
- Report Number(s):
- SAND2022-14225R; 710902
- Country of Publication:
- United States
- Language:
- English
Similar Records
OpenSNAPI: Toward a Unified API for SmartNICs
Computational Offload with BlueField Smart NICs