Show simple item record

dc.contributor.author
Narasimhakumar, Rajasimha
dc.contributor.supervisor
Alonso, Gustavo
dc.contributor.supervisor
Müller, Ingo
dc.date.accessioned
2021-09-02T09:13:41Z
dc.date.available
2021-08-16T13:15:53Z
dc.date.available
2021-09-02T07:09:14Z
dc.date.available
2021-09-02T09:13:41Z
dc.date.issued
2021
dc.identifier.uri
http://hdl.handle.net/20.500.11850/501090
dc.identifier.doi
10.3929/ethz-b-000501090
dc.description.abstract
For handling large scale data communication spanning gigabytes per second, Remote Direct Memory Access (RDMA) technology is significant. InfiniBand (IB) is a widely used RDMA point-to-point interconnect whose full potential can be realized using the most efficient networking library. In this thesis we address this challenge by benchmarking the bandwidth performance of two latest and popular open source RDMA libraries - NetIO from CERN and HPNL from Intel. Our main objective is to design a shuffle operator with these RDMA libraries and perform comprehensive benchmarking by considering two important dimensions in our study: scalability and platform consistency. We study network usage patterns with increasing complexity: from simple unidirectional and bidirectional streaming between two nodes extended to pairwise all-to-all bidirectional streaming with multiple nodes and finally to a data-dependent shuffling operation for the full cluster. We create multiple designs, execute and compare the results on two different IB clusters: euler (QDR) and r630 (FDR) to observe consistency on different cluster infrastructure. Using proper performance tuning factors such as message size and in-flight messages, our results show that both libraries do not scale assuredly well for shuffling, showing a lowering trend as the cluster size increases, but still analogous to their all-to-all streaming counterparts. Compared to the peak reference bidirectional bandwidth, the best performing shuffling design of NetIO with optimal parameters manages to achieve about 80-90% on euler and about 80% on r630 upto a node count of 6 and 5 respectively for a message size of 128 KB. Whereas, HPNL's best design reaches >70% on euler and around 50% on r630 for 1 MB messages upto a cluster size of 7, showing that NetIO marginally outperforms HPNL.
en_US
dc.format
application/pdf
en_US
dc.language.iso
en
en_US
dc.publisher
ETH Zurich
en_US
dc.rights.uri
http://creativecommons.org/licenses/by/4.0/
dc.title
Benchmarking and Optimizations of Data Shuffling on High-performance Networks
en_US
dc.type
Master Thesis
dc.rights.license
Creative Commons Attribution 4.0 International
ethz.size
84 p.
en_US
ethz.publication.place
Zurich
en_US
ethz.publication.status
published
en_US
ethz.leitzahl
ETH Zürich::00002 - ETH Zürich::00012 - Lehre und Forschung::00007 - Departemente::02150 - Dep. Informatik / Dep. of Computer Science::02663 - Institut für Computing Platforms / Institute for Computing Platforms::03506 - Alonso, Gustavo / Alonso, Gustavo
en_US
ethz.leitzahl.certified
ETH Zürich::00002 - ETH Zürich::00012 - Lehre und Forschung::00007 - Departemente::02150 - Dep. Informatik / Dep. of Computer Science::02663 - Institut für Computing Platforms / Institute for Computing Platforms::03506 - Alonso, Gustavo / Alonso, Gustavo
en_US
ethz.date.deposited
2021-08-16T13:15:58Z
ethz.source
FORM
ethz.eth
yes
en_US
ethz.availability
Open access
en_US
ethz.rosetta.installDate
2021-09-02T09:13:48Z
ethz.rosetta.lastUpdated
2022-03-29T11:27:11Z
ethz.rosetta.versionExported
true
ethz.COinS
ctx_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.atitle=Benchmarking%20and%20Optimizations%20of%20Data%20Shuffling%20on%20High-performance%20Networks&rft.date=2021&rft.au=Narasimhakumar,%20Rajasimha&rft.genre=unknown&rft.btitle=Benchmarking%20and%20Optimizations%20of%20Data%20Shuffling%20on%20High-performance%20Networks
 Search print copy at ETH Library

Files in this item

Thumbnail

Publication type

Show simple item record