Network Storage (NetworkIO) Implementation for SST Firefly NIC#2643
Network Storage (NetworkIO) Implementation for SST Firefly NIC#2643RishankPratikHPLabs wants to merge 3 commits into
Conversation
|
Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
RishankPratikHPLabs
force-pushed
the
nwStorage-feature
branch
from
630bdc2 to
f85e717
Compare
|
Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
feldergast
left a comment
feldergast
left a comment
There was a problem hiding this comment.
Code generally looks good. There are a few comments about some code clean-up. There will also need to be at least one test, preferably more to test the new code. The tests should use the new merlin.base derived python modules from the merlin library and should not use emberload.py as this uses the old deprecated merlin python libraries.
RishankPratikHPLabs
force-pushed
the
nwStorage-feature
branch
from
f85e717 to
c1014f6
Compare
|
Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
|
This PR now shows 0 changed files. It also has 104 commits. I would suggest rebasing this branch on the latest devel and overlaying the changes onto that. In general, we recommend you point your local devel branch to the sst-simulator devel branch as described in the contributing document (found in the Forked Development section): https://github.com/sstsimulator/sst-elements/blob/master/CONTRIBUTING.md This creates much cleaner PRs without all the extraneous merge commits from devel, and tends to have fewer merging issues. |
SSD module integration with Firefly NIC for realistic storage I/O modeling. End-to-end I/O path simulation: compute nodes -> network -> SSD storage nodes. Distributed address mapping with round-robin striping across storage devices. Direct integration with Ember motif framework for HPC workload simulation. New components: - NetworkIO API layer in Hermes interface - HadesNetworkIO with distributed address mapping - NIC-level packet handling for NetworkIO (read, write, ACK) - SimpleSSD storage device model (configurable bandwidth, latency, multi-queue bus) - Ember NetworkIO motif and test configurations Co-authored-by: Rishank Pratik <rishank.pratik@hpe.com> Co-authored-by: Pawan Kumar <pawan.kumar4@hpe.com> Co-authored-by: Sumant Kalra <sumant.kalra@hpe.com> Co-authored-by: Shridhar Joshi <shridhar@hpe.com>
Add SPDX-FileCopyrightText and SPDX-License-Identifier (BSD-3-Clause) headers to all new and modified files per HPE OSRB attribution requirements. Update CONTRIBUTORS.TXT with HPE entry. Fix corrupted copyright line in nicEvents.h.
RishankPratikHPLabs
force-pushed
the
nwStorage-feature
branch
from
c1014f6 to
e1e773d
Compare
|
Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
Using Repos:
Pull Request Author: RishankPratikHPLabs |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 8 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Job: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
|
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
Using Repos:
Pull Request Author: RishankPratikHPLabs |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 8 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Job: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
|
|
I've look through the code and it is almost ready to merge, but it fails the "make dist" test, which means that one or more of the added files didn't make it into the proper place in the relevant Makefile.am or Makefile.inc. These files will need to be updated and we can send it through testing again. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
Using Repos:
Pull Request Author: RishankPratikHPLabs |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 8 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Job: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
|
feldergast
left a comment
feldergast
left a comment
There was a problem hiding this comment.
Looks good for the most part. There are several places where NULL should be changed to nullptr, and a couple files need to be added to ember/Makefile.am. After those changes, it should pass testing and we can get it merged.
| class EmberNetworkIOEvent : public EmberEvent { | ||
| public: | ||
| EmberNetworkIOEvent(NetworkIO::Interface& api, Output* output, | ||
| EmberEventTimeStatistic* stat = NULL) : |
There was a problem hiding this comment.
This should use nullptr instead of NULL
| public: | ||
| EmberNetworkIOReadEvent(NetworkIO::Interface& api, Output* output, Hermes::MemAddr dest, | ||
| uint64_t offset, uint32_t length, | ||
| EmberEventTimeStatistic* stat = NULL) : |
| public: | ||
| EmberNetworkIOWriteEvent(NetworkIO::Interface& api, Output* output, uint64_t offset, | ||
| Hermes::MemAddr src, uint32_t length, | ||
| EmberEventTimeStatistic* stat = NULL) : |
| EmberNetworkIOGenerator::EmberNetworkIOGenerator(ComponentId_t id, Params& params, std::string name) | ||
| : EmberGenerator(id, params, name), m_networkIOLib(nullptr) | ||
| { | ||
| m_shmemLib = NULL; |
|
|
||
| HadesNetworkIO::HadesNetworkIO(ComponentId_t id, Params& params) : | ||
| Hermes::NetworkIO::Interface(id), | ||
| m_nicPtr(NULL) |
| } | ||
|
|
||
| void enque( int unit, int pid, std::vector< MemOp >* vec, FireflyNetworkEvent* ev, int vn, int dest, Callback callback = NULL ); | ||
| void enque( int unit, int pid, std::vector< MemOp >* vec, FireflyNetworkEvent* ev, int vn, int dest, Callback callback = NULL, SendEntryBase* entry=NULL ); |
There was a problem hiding this comment.
May as well change NULL to nullptr for both of these
|
|
||
| libember_la_LDFLAGS = -module -avoid-version | ||
|
|
||
| EXTRA_DIST = \ |
There was a problem hiding this comment.
Need to add these file to this section:
test/loadNetworkIO
test/networkIOParams.py
4 participants
Network Storage (NetworkIO) Implementation for SST Firefly NIC
What this does
Adds network-attached SSD storage simulation to the Firefly NIC. Compute nodes can now issue async read/write operations to storage nodes over the simulated network, with a SimpleSSD model handling the storage-side latency and bandwidth.
We needed this to study I/O performance in distributed storage setups — how network latency, SSD throughput, node placement, and access patterns interact — without needing real hardware. This plugs into the existing Ember/Firefly/Hermes stack, so existing MPI simulations are unaffected.
How it works
Here's what each layer does:
Hermes API (
networkIOapi.h) — defines two calls:networkIORead(dest, offset, length, callback)andnetworkIOWrite(offset, src, length, callback). Both are async — you pass a callback that fires when the op completes.Hades (
hadesNetworkIO) — takes the global byte offset and figures out which storage node to hit. Uses a simple modulo scheme:nodeIndex = (offset / storageNodeCapacity) % numStorageNodes. Then hands it down to the NIC.Firefly NIC (
nicNetworkIO,nicNetworkIOSendEntry,nicNetworkIOStream) — this is where most of the work happens:On the compute side: the NIC creates a send entry (read or write), generates a response key (
respKey) with a callback attached, and queues a small request packet:[MsgHdr::NetworkIO][Read|Write][offset][addr][length][respKey]. The respKey gets stored so the NIC can match the ACK later.On the storage side: a
NetworkIOStreamreceives the packet, extracts the op type and respKey, and passes it to SimpleSSD (or falls back to DMA if SimpleSSD isn't loaded). When the SSD finishes, the stream creates a tiny ACK packet containing just the respKey and sends it back.Back on the compute side: another
NetworkIOStreamreceives the ACK, pulls out the respKey, looks up the stored callback viagetRespKeyValue(), and invokes it. This completes the async operation.NetworkIO traffic stays separated from MPI — it uses
MsgHdr::NetworkIO(vsMsgHdr::Msg), its ownNetworkIOStream(vsMsgStream), and vNic 0.Ember (
TestNetworkIOmotif) — the workload layer. Issues configurable read/write calls withmessageSize,iterations,optype, andfileSizeparams. Randomizes offsets within the file range.SimpleSSD — sits on each storage node as a subcomponent. Has a multi-lane queue structure (
nSSDsPerNode × queuesCountPerSSDlanes). Requests go round-robin across lanes. Each request gets a delay ofoverheadLatency + (bytes / bandwidth), scheduled via a self-link timer. When the delay fires, the completion callback runs and the ACK gets sent.SimpleSSD model
firefly.SimpleSSDis a subcomponent with these params:nSSDsPerNodequeuesCountPerSSDreadBandwidthPerSSD_GBpswriteBandwidthPerSSD_GBpsreadOverheadLatency_nswriteOverheadLatency_nsDelay = overhead + (bytes / bandwidth). Requests go round-robin across all SSD×queue lanes.
Node setup
Some nodes are compute, some are storage. Configured in
loadNetworkIO:TestNetworkIO) and issue I/ONull(idle) and serve requests through SimpleSSDSSD_START_NODEandSSD_NODESvars in the load fileFiles
New files (20):
hermes/networkIOapi.h— NetworkIO APIfirefly/hadesNetworkIO.{cc,h}— offset-to-node mappingfirefly/nicNetworkIO.{cc,h}— NIC command handlingfirefly/nicNetworkIOSendEntry.h— packet constructionfirefly/nicNetworkIOStream.{cc,h}— incoming packet processingfirefly/storageModel/simpleSSD.{cc,h}— SSD simulationember/libs/emberNetworkIOLib.h— library wrapperember/libs/networkIOEvents/emberNetworkIO{Event,ReadEvent,WriteEvent}.h— eventsember/networkIO/emberNetworkIOGen.{cc,h}— generator baseember/networkIO/motifs/emberTestNetworkIO.{cc,h}— test motifember/test/loadNetworkIO— test configember/test/networkIOParams.py— platform paramsModified files (16):
firefly/nic.{cc,h}— NetworkIO handler + SimpleSSD setupfirefly/nicEvents.h— new event typesfirefly/nicRecvCtx.cc,nicRecvMachine.h— stream dispatchfirefly/nicSendEntry.h,nicSendMachine.{cc,h}— send supportfirefly/nicVirtNic.h,virtNic.{cc,h}— delegation methodsfirefly/Makefile.am,hermes/Makefile.am,ember/Makefile.am— build updatesember/.gitignore— ignore patternsCONTRIBUTORS.TXT— added HPE entryTesting
Tested with three configs, all pass:
Built with SST Core + Elements from
devel, GCC 14.How to run
Build:
Verify SimpleSSD registered:
Run the default test (2 compute + 2 SSD):
cd src/sst/elements/ember/test sst emberLoad.py -- --topo=torus --shape=2x2 --numNodes=4 --numCores=1 --platform=networkIO --loadFile=loadNetworkIOTo change the node layout, edit
loadNetworkIO:Then run with a matching topology:
Motif params in
loadNetworkIO:messageSize=<bytes>,iterations=<N>,op=read|write,fileSize=<bytes>.Co-authored-by: Rishank Pratik rishank.pratik@hpe.com
Co-authored-by: Pawan Kumar pawan.kumar4@hpe.com
Co-authored-by: Sumant Kalra sumant.kalra@hpe.com
Co-authored-by: Shridhar Joshi shridhar@hpe.com