Scan pdus by bmcdonald3 · Pull Request #107 · OpenCHAMI/magellan

bmcdonald3 · 2025-07-03T20:33:27Z

This PR adds support for scanning JAWS endpoints to the scan command. This allows the tool to discover both Redfish-based BMCs and JAWS-based PDUs in a single run. To help with development against test hardware, I also included the --insecure flag to handle self-signed certs.

While I was testing this internally, I ran into a couple of weird issues with the scanner that I've also fixed here. The first thing I noticed was that the scan would sometimes stop unexpectedly and not finish if it hit a machine that was offline (which was happening on this machine because of an upgrade happening at the same time). It turns out a single connection error was killing the entire worker thread. I changed it so it just logs the error and continues on with the rest of its list.

I also saw that things would slow down when running a lot of scans and I think this is because we weren't always closing the HTTP connections, especially on responses that weren't a 200 OK. This was causing a resource leak. I added a fix to make sure we always close the connection body no matter what the response is, which I think solves this problem.

Here is the way that I was testing this:

change from writing SQL file to printing to stdout (due to machine limitations...)

if len(foundAssets) == 0 {
    log.Info().Msg("Scan complete. No responsive assets were found.")
    return
}

log.Info().Msgf("Scan complete. Found %d responsive asset(s):", len(foundAssets))

output, err := json.MarshalIndent(foundAssets, "", "  ")
if err != nil {
    log.Error().Err(err).Msg("Failed to format results to JSON")
    return
}
fmt.Println(string(output))

build Magellan locally (but for architecture of test machine): GOOS=linux GOARCH=amd64 go build -o magellan-linux
scp to machine
run a scan on the node management network

./magellan-linux scan --subnet 10.254.1.0/17 --insecure
{"level":"info","time":"2025-07-03T20:28:17Z","message":"Scan complete. Found 15 responsive asset(s):"}
[
  {
    "host": "https://10.254.1.31",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.292671243Z",                                                                                                                 "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.29",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.261800343Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.5",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.213246997Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.3",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.211797421Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.7",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.212010565Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.21",
    "port": 443,
    "protocol": "tcp",
    "state": true,                                                                                                                                       [39/1904]
    "timestamp": "2025-07-03T20:27:58.292588111Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.11",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.211543276Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.9",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.2117599Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.32",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.292864507Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.23",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.211555246Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.15",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.260458899Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.25",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.21178194Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.13",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.21178627Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.26",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.21283304Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.27",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T20:27:58.26052832Z",
    "service_type": "JAWS"
  }
]

…dfish Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

davidallendj · 2025-07-03T21:02:26Z

A couple of notes:

Does the scan still timeout after the amount of time specified with the --timeout flag? Sometimes it just takes a while for a BMC to respond and I don't think we want to error out before that happens.
Can you make the scan for JAWS endpoints optional with a flag somehow? Not all systems are going to have PDUs to scan for and it wouldn't make sense if we have to scan for them if we know they're not there. Maybe add a new flag?

magellan scan --include=bmcs,pdus --subnet 172.16.0.0/24 --insecure
# ...or...
magellan scan --bmcs --pdus --subnet 172.16.0.0/24 --insecure

Can you change it back to use SQLite to store the scan data by default and add the JSON output as optional? This will maintain the original behavior while also allowing saving the scan output to a flat file without the database. If the --output/-o flag is specified, then it should write the JSON to a file. Something like...

# write JSON to standard output like mentioned in PR
magellan scan --subnet 172.16.0.0/24 -F json

# write JSON to file specified with `-o`
magellan scan --subnet 172.16.0.0/24 -F json -o assets.json

The --format/-F can be values like db/json/yaml or something like that similar to the -F flag for the other commands.

bmcdonald3 · 2025-07-03T21:19:16Z

Thanks for the comments.

Yes, assuming I understand what you are asking, that is right. The timeout is applied independently to each probe. If you set --timeout=30, the node probe gets its own 30-second timeout, and then the PDU probe gets its own separate 30-second timeout. They don't share the time.
This makes a lot of sense. I'll add a new --include flag like you mention. I'll set the default to --include=bmcs
Yup, I can add that.

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

bmcdonald3 · 2025-07-03T22:52:48Z

Alright, implemented that first part basing off what is done in the collect command: 

# ./magellan-linux scan --subnet 10.254.1.0/17 --insecure -F json -o ""
[
  {
    "host": "https://10.254.1.21",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:17.90890229Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.27",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:17.908405271Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.9",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:17.910311526Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.23",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:18.49311309Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.5",
    "port": 443,
    "p
  },
  {
    "host": "https://10.254.1.7",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:17.908627395Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.19",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:17.908209107Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.3",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:17.908170686Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.26",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:17.908305569Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.32",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03
    "host": "https://10.254.1.13",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:18.49147542Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.29",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:17.908496402Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.15",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:18.491976789Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.11",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:18.491394668Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.25",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:45:27.321554781Z",
    "service_type": "JAWS"
  }
]

And then writing to a file like this also works: ./magellan-linux scan --subnet 10.254.1.0/17 --insecure -F json -o assets.json  And then the includes:

# ./magellan-linux scan --subnet 10.254.1.0/17 --insecure -F json -o "" --include=pdus
[
  {
    "host": "https://10.254.1.3",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:54:19.509239146Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.11",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:54:28.669611854Z",
    "service_type": "JAWS"
  }
]

# ./magellan-linux scan --subnet 10.254.1.0/17 --insecure -F json -o "" --include=pdus,bmcs
[
  {
    "host": "https://10.254.1.23",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.019804629Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.3",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.020640064Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.19",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.021818165Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.9",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.065127117Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.29",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.06473621Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.27",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.064429734Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.5",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.022678981Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.32",
    "port": 443,
    "protocol": "tcp",
    "state": true,
  {
    "host": "https://10.254.1.26",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.023503506Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.7",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.064830882Z",
    "service_type": "JAWS"
  },
  {
    "host": "https://10.254.1.11",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.019903901Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.13",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.020488861Z",
    "service_type": "Redfish"
  },
  {
    "host": "https://10.254.1.25",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-03T21:55:15.020376619Z",
    "service_type": "Redfish"
  }
]

Probably good for me to look at this again on Monday when we get back, since I put this together last minute before a long weekend, I wouldn't be surprised if it isn't perfect...

davidallendj · 2025-07-03T23:17:49Z

I think it's looking good so far. Is it mandatory to specify -o to write to standard output? I would think you wouldn't need to put it explicitly if -F=json and -o is an empty string by default. Or is that just to showing it in the example?

bmcdonald3 · 2025-07-08T17:13:36Z

Whoops, sorry, missed this comment. Yeah, I don't think you need to put that, I just ran again and confirmed, I was just showing that in the example.

Edit: here is some output showing it's working without that "" I had

# ./magellan scan --subnet 172.24.0.0/24 --insecure -F json
[
  {
    "host": "https://172.24.0.3",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-08T12:17:30.360383042-05:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.24.0.2",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-08T12:17:30.359125496-05:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.24.0.4",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-08T12:17:30.359173925-05:00",
    "service_type": "Redfish"
  }
]

davidallendj · 2025-07-08T17:27:26Z

-		}
+		switch format {
+		case "json", "yaml":
+			if len(foundAssets) == 0 {


This can be moved outside above of the switch statement to check for all cases.

davidallendj · 2025-07-08T17:34:18Z

+			if len(foundAssets) > 0 && debug {
+				log.Info().Any("assets", foundAssets).Msgf("found assets from scan")
+			}


This can be removed if we moved the check above the switch.

davidallendj

One small change to move the found assets check.

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

davidallendj · 2025-07-08T18:50:44Z

-			// make the cache directory path if needed
-			err := os.MkdirAll(path.Dir(cachePath), 0755)
+		if len(foundAssets) == 0 {
+			log.Info().Msg("Scan complete. No responsive assets were found.")


Can you change this to a log.Warn() instead? That way it's not included whenever I redo the logging to use a --log-level flag.

davidallendj

The scan isn't working for me. For some reason, it's not returning anything.

[allend@re-head magellan-testing]$ ./magellan scan --subnet 172.16.0.0/24
{"level":"info","time":"2025-07-08T12:57:43-06:00","message":"Scan complete. No responsive assets were found."}
[allend@re-head magellan-testing]$ ./magellan list
{"level":"error","error":"no file found","time":"2025-07-08T12:59:22-06:00","message":"failed to get scanned assets"}

Hosts

[allend@re-head magellan-testing]$ cat /etc/hosts
...
# Compute node BMCs
172.16.0.101    bmc-nid001 bmc-re001
172.16.0.102    bmc-nid002 bmc-re002
172.16.0.103    bmc-nid003 bmc-re003
172.16.0.104    bmc-nid004 bmc-re004
172.16.0.105    bmc-nid005 bmc-re005
172.16.0.199    bmc-nid099 bmc-re099

It is working for the current main though.

[allend@re-head magellan]$ ./magellan scan --subnet 172.16.0.0/24
[allend@re-head magellan]$ ./magellan list
https://172.16.0.101:443 (tcp) @Tue Jul  8 12:59:57 -0600 2025
https://172.16.0.102:443 (tcp) @Tue Jul  8 12:59:57 -0600 2025
https://172.16.0.103:443 (tcp) @Tue Jul  8 12:59:57 -0600 2025
https://172.16.0.104:443 (tcp) @Tue Jul  8 12:59:57 -0600 2025
https://172.16.0.105:443 (tcp) @Tue Jul  8 12:59:57 -0600 2025
https://172.16.0.199:443 (tcp) @Tue Jul  8 12:59:57 -0600 2025

bmcdonald3 · 2025-07-08T22:09:41Z

Interesting... looking into it.

bmcdonald3 · 2025-07-08T22:18:30Z

Can you try it again? I think the issue was the addition of the tls validation:

transport := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: params.Insecure},
	}
	probeClient := &http.Client{
		Timeout:   time.Duration(params.Timeout) * time.Second,
		Transport: transport,
	}

but I switched the default to retain the old behavior.

Edit: in other words, I think if you used the --insecure flag, it would have worked as it does on main, but I understand we don't necessarily want to change the behavior for this, so I've switched the default.

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

davidallendj · 2025-07-10T17:14:03Z

Just tried again, and it looks like that fixed it.

./magellan scan --subnet 172.16.0.0/24
[allend@re-head magellan-testing]$ ./magellan list
https://172.16.0.101:443 (tcp) @Thu Jul 10 11:14:49 -0600 2025
https://172.16.0.102:443 (tcp) @Thu Jul 10 11:14:49 -0600 2025
https://172.16.0.103:443 (tcp) @Thu Jul 10 11:14:49 -0600 2025
https://172.16.0.104:443 (tcp) @Thu Jul 10 11:14:49 -0600 2025
https://172.16.0.105:443 (tcp) @Thu Jul 10 11:14:49 -0600 2025
https://172.16.0.199:443 (tcp) @Thu Jul 10 11:14:49 -0600 2025
[allend@re-head magellan-testing]$ ./magellan scan --subnet 172.16.0.0/24 -F json
[
  {
    "host": "https://172.16.0.199",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-10T11:16:20.827313696-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.105",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-10T11:16:20.827320306-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.101",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-10T11:16:20.827185044-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.102",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-10T11:16:20.827215305-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.103",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-10T11:16:20.827247775-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.104",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2025-07-10T11:16:20.827281146-06:00",
    "service_type": "Redfish"
  }
]

Looks like it's working like expected so I'll go ahead and approve.

davidallendj

LGTM

bmcdonald3 added 2 commits July 3, 2025 13:11

Add support for scanning PDU components by probing JAWS as well as re…

4cfd48d

…dfish Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

ADd secure, remove username and password

488ad24

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

bmcdonald3 added 2 commits July 3, 2025 15:51

Remove unnecessary insecure flag

7988ed6

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

Add printing to stdout or a file instead of only DB for scan

c413dd5

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

bmcdonald3 force-pushed the scan-pdus branch from e51f1fb to d72645c Compare July 3, 2025 22:51

davidallendj mentioned this pull request Jul 8, 2025

[FEATURE]Add Some Mechanism for Deriving Real BMC XNAMEs Instead of Synthesizing Them #106

Closed

davidallendj reviewed Jul 8, 2025

View reviewed changes

Comment thread cmd/scan.go

davidallendj reviewed Jul 8, 2025

View reviewed changes

davidallendj requested changes Jul 8, 2025

View reviewed changes

bmcdonald3 added 2 commits July 8, 2025 11:12

Add include flag

96931cd

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

Pull length of assets check out, per David's feedback

117783d

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

bmcdonald3 force-pushed the scan-pdus branch from d72645c to 117783d Compare July 8, 2025 18:12

davidallendj reviewed Jul 8, 2025

View reviewed changes

bmcdonald3 force-pushed the scan-pdus branch from 3427cb2 to b4c4038 Compare July 8, 2025 22:15

bmcdonald3 added 2 commits July 8, 2025 15:22

Switch to a warn

3a6ed5a

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

Switch insecure default to true

833fb80

Signed-off-by: Ben McDonald <ben.mcdonald@hpe.com>

bmcdonald3 force-pushed the scan-pdus branch from 53a5d43 to 833fb80 Compare July 8, 2025 22:22

davidallendj self-requested a review July 10, 2025 17:17

davidallendj approved these changes Jul 10, 2025

View reviewed changes

bmcdonald3 merged commit 7e8011c into OpenCHAMI:main Jul 10, 2025
1 check passed

bmcdonald3 deleted the scan-pdus branch July 10, 2025 21:14

Uh oh!

Conversation

bmcdonald3 commented Jul 3, 2025

Uh oh!

davidallendj commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bmcdonald3 commented Jul 3, 2025

Uh oh!

bmcdonald3 commented Jul 3, 2025

Uh oh!

davidallendj commented Jul 3, 2025

Uh oh!

bmcdonald3 commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidallendj Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidallendj Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

davidallendj left a comment

Choose a reason for hiding this comment

Uh oh!

davidallendj Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

davidallendj left a comment

Choose a reason for hiding this comment

Uh oh!

bmcdonald3 commented Jul 8, 2025

Uh oh!

bmcdonald3 commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidallendj commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidallendj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

davidallendj commented Jul 3, 2025 •

edited

Loading

bmcdonald3 commented Jul 8, 2025 •

edited

Loading

davidallendj Jul 8, 2025 •

edited

Loading

bmcdonald3 commented Jul 8, 2025 •

edited

Loading

davidallendj commented Jul 10, 2025 •

edited

Loading