Headless experience for Datum tunnels

[privateip]

High-Level Summary

Add a headless CLI that creates and manages tunnels to Datum Cloud without requiring a graphical interface. The CLI can run as a foreground process or be daemonized as a background process, and optionally install a tunnel as a system service (systemd on Linux, launchd on macOS, Windows Service on Windows) for persistent, boot-time tunnel management — mirroring the service lifecycle model familiar from Docker and Podman.

Motivation

The Datum desktop app currently requires a display environment, which excludes a large class of legitimate use cases:

• headless servers,
• CI/CD pipelines,
• container workloads,
• SSH-only remote machines, and
• automated infrastructure

where no GUI is available or desirable. Operators who want a persistent tunnel on a server today have no supported path. It's either run the GUI app in a degraded state, cobble together workarounds, or don't use Datum at all. The service installation feature directly addresses the operational gap where tunnels need to survive reboots and run unattended, the same problem Docker and Podman solve for container workloads with systemctl.

Without a CLI and service model, Datum is positioned as a developer convenience tool rather than production infrastructure.

Goals

• Users can invoke datum tunnel (or equivalent subcommand) from any terminal to establish a tunnel without a display environment
• The CLI accepts all tunnel configuration options currently available in the desktop UI (auth, endpoint, credentials, network settings, etc.)
• Users can run the tunnel in the foreground (blocking, logs to stdout/stderr) or detach it as a background daemon
• Users can install a named tunnel as a persistent system service (datum tunnel install --name ) that auto-starts on boot, using systemd on Linux, launchd on macOS, and Windows Service Manager on Windows
• Service lifecycle management commands are supported: install, uninstall, start, stop, status
• The CLI exits with standard POSIX exit codes so it composes cleanly with shell scripts and process supervisors

Non-Goals

• This does not replace or deprecate the existing desktop GUI, the GUI remains the primary UX for interactive desktop users
• No new tunnel types or networking capabilities are introduced; the CLI exposes existing functionality only
• Container-native packaging (e.g., an official Docker image with an entrypoint) is out of scope for this enhancement, though the headless CLI makes it straightforward to build one separately
• Cross-platform service installation is not required to ship simultaneously, Linux/systemd ships first; macOS and Windows follow
• Remote management of tunnels (controlling a daemon over a network socket or API) is out of scope

[richardhenwood]

Hi @drewr - thanks for sharing! This is a topic of great interest to me!

This is an excellent description of something I would love to have. I didn't really think about Mac and Windows - but I guess they deserve some attention :) I think what you've described covers my MVP of this tool. My use-cases that will be enabled by this tool include:

• Sprinkle this into my dockerfile and my container app is tunneled into Datum network platform.
• Add headless as a kubernetes Sidecar and my Kubernetes Pod is tunneled into Datum network platform.

I am completely in agreement with your design idea. It would seem that a case can be made for a single threaded process that constructs and manages the tunnel. headless-tunnel.exe if you will. headless-tunnel.exe can be inserted into a container, placed on a host etc and leverage appropriate supervisor (start, stop, etc) tooling appropriate to the environment (systemd, supervisord, fork, windows?, mac?).

I'm interested in your ideas about how headless-tunnel.exe receives the configuration packet that gives it appropriate identity, logging configuration, etc. i.e. an Identity Packet. I would tend towards environment variables as the only supported mechanism. This emphasizes 'headless' operation... But maybe this is design to the point of being obtuse and including command line alternatives is zero cost and simplifies initial setup :)

Another thought: the Identity Packet is baked into headless-tunnel.exe. Each use of the datium tunnel requires a bespoke download. This feels a little beyond the MVP to me. Nice because it is 'zero-config'... but significantly more complex to develop and debug?

A further thought (that I'm not a fan of): socket base communication. Ugh... I never feel very comfortable using sockets. Also, I might suggest that if you don't allow updating the Identity Packet on a running process, they might be the wrong mode. Also, is the socket in the container or on the host? Further opportunity for confusion for me! (but also an opportunity to learn something new!)

Thinking much further down the line, it might be interesting design the Identity Packet anticipating the idea of a locally hosted Identity-Packet forge. I.e. a User uses their own local secrets etc to issue an Identity Packet for their services running over Datium networks.

[privateip]

hi @richardhenwood

I would tend towards environment variables as the only supported mechanism. This emphasizes 'headless' operation... But maybe this is design to the point of being obtuse and including command line alternatives is zero cost and simplifies initial setup :)

My expectation would be we would follow 12 Factor App development here

[privateip]

Another thought: the Identity Packet is baked into headless-tunnel.exe. Each use of the datium tunnel requires a bespoke download. This feels a little beyond the MVP to me. Nice because it is 'zero-config'... but significantly more complex to develop and debug?

Agreed, I think we need to get here for sure. I think we don't want to pollute the initial MVP with this requirement just yet but I see this as an early feature addition going forward.

[drewr]

Quick update here:

• feat: add Headless Tunnel CLI enhancement #756 is an design for the connect datumctl plugin. This is the way we want tunnels to be managed going forward, but it's still baking.
• In the meantime, I cleaned up and released the derivative headless work from the desktop app just to have something useful. It's linux-only at the moment because that's the primary environment where they're needed.

Setting up tunnel...
  ✓ tunnel accepted (0.2s) [HTTPProxy/tunnel-9zllw]
  ✓ TLS certificate issued (0.2s) [HTTPProxy/tunnel-9zllw]
  ✓ connector ready (0.2s) [Connector/datum-connect-mhxj5]
  ✓ iroh DNS published (0.2s) [Connector/datum-connect-mhxj5]
  ✓ route programmed (0.2s) [HTTPProxy/tunnel-9zllw]
  ✓ envoy metadata propagated (1.0s) [HTTPProxy/tunnel-9zllw]
Verifying connectivity...
  ✓ origin reachable (0.1s) [http://localhost:9999]: HTTP 200
  ✓ proxy responding (0.1s) [https://tip-post-nqt7t.datumproxy.net]: HTTP 200
Tunnel ready after 1 sec: https://tip-post-nqt7t.datumproxy.net
Press Ctrl+C to stop...

https://github.com/datum-cloud/app/releases/tag/cli-v0.2.1

[drewr]

Latest here is that we have a POC of our connect plugin, which moves this functionality out of the app repo, leaving that as solely a desktop app. I've deleted the datum-connect releases from that repo to eliminate confusion.

Here's the sequence that needs to happen now:

1. Merge Connect plugin: initial implementation (POC) connect#1, feat: warn on stale/orphaned connectors in tunnel list connect#2, fix: don't print 'Tunnel ready' until proxy URL responds connect#3 so we can generate releases and test datumctl plugin install connect.
2. Concurrently, review feat: add Headless Tunnel CLI enhancement #756 for to make sure requirements are directionally correct.
3. Validate functionality against this issue and feat: add Headless Tunnel CLI enhancement #756 with testing.
4. Address gaps in future PRs to the plugin repo.
5. More testing.
6. Release the connect plugin, update changelog, and widen the test radius.

[drewr]

Change of plans. We're going to audit datum-cloud/connect#1 and break it up before merging the all-vibed version to the connect repo. In the meantime, if you want to test (apologies, nix is the easiest way right now):

% git clone git@github.com:datum-cloud/connect
% cd connect
% git checkout develop
% nix develop
% task install:release
% datumctl plugin trust connect
% datumctl login --no-browser
% datumctl connect tunnel listen --endpoint localhost:9999 # whatever you want to expose

We don't have this work scheduled beyond this point, so I'm unassigning myself for now.