Systems-minded builder working across infrastructure, Kubernetes, observability, networking, security, local AI, and automation.

19 years in manufacturing systems 🏭 → building and operating NovaLabs, my self-hosted infrastructure and engineering lab. Ontario, Canada 🍁

Philosophy: I feel more comfortable driving my car when I understand how the brakes work.

NovaLabs is my living systems lab.

It spans traditional infrastructure, networking, Kubernetes, observability, data engineering, local AI inference, automation, developer environments, security research, and experimental AI systems.

I use it to learn technologies by actually operating them: deploying workloads, breaking things, recovering them, instrumenting them, documenting them, and turning what worked into repeatable infrastructure.

A heterogeneous k3s cluster operated declaratively with Flux.

flowchart TD
    forgejo["Forgejo<br/>Source of Truth"]
    github["GitHub<br/>Public Mirror"]
    flux["Flux<br/>GitOps Reconciliation"]

    control["lovelace<br/>Raspberry Pi 5<br/>ARM64 Control Plane"]
    worker["k8s-worker-01<br/>Debian x86_64<br/>Workload Node"]

    forgejo -->|mirror| github
    forgejo -->|desired state| flux

    flux --> control
    flux --> worker

Current platform features include:

• Raspberry Pi 5 ARM64 control plane
• Debian x86_64 workload node
• Forgejo as the primary Git service
• GitHub as an external public mirror
• Flux reconciliation from main
• Kustomize base + environment overlays
• SOPS + age encrypted Kubernetes secrets
• Traefik ingress
• persistent local storage
• node-bound workloads where appropriate
• Renovate dependency automation
• Prometheus-based observability
• Grafana dashboards
• centralized log collection with Alloy and Loki
• externally exposed applications through purpose-specific tunnels

The rule is simple:

Git is desired state. Kubernetes is current state.

One of the largest stateful workloads running on Lovelace is a dedicated modded Minecraft Create server.

Rather than treating it like a manually managed game server, I built it as an actual Kubernetes application.

• Minecraft 1.21.1
• NeoForge
• All of Create
• dedicated Kubernetes namespace
• StatefulSet deployment
• dedicated x86_64 worker scheduling
• 8-vCPU worker capacity for faster JVM and modpack startup
• separate lifecycle from my Paper Minecraft environment

flowchart LR
    change["Configuration Change"]
    branch["Forgejo<br/>Branch / Pull Request"]
    main["main"]
    flux["Flux"]
    kustomize["Kustomize"]
    sts["Minecraft<br/>StatefulSet"]

    change --> branch
    branch --> main
    main --> flux
    flux --> kustomize
    kustomize --> sts

The deployment includes:

• reusable Kustomize base
• cluster-specific staging overlay
• Flux-controlled reconciliation
• Forgejo as source of truth
• GitHub mirror
• encrypted configuration using SOPS + age
• Git-controlled server configuration
• declarative workload scheduling

Minecraft world data lives on dedicated persistent storage attached to the Kubernetes worker.

flowchart LR
    sts["Minecraft StatefulSet"]
    pvc["PersistentVolumeClaim"]
    pv["Retained PersistentVolume"]
    disk["Dedicated Worker Storage"]

    sts --> pvc
    pvc --> pv
    pv --> disk

The design includes:

• dedicated host storage
• static persistent volume
• persistent volume claim
• retained world data across pod recreation
• node affinity to keep storage and workload together
• Kubernetes-safe handling of stateful data

The server can be destroyed and recreated without treating the Minecraft world itself as ephemeral container state.

Player traffic is exposed through a dedicated Playit tunnel rather than directly opening the game server to the Internet.

flowchart LR
    players["Players"]
    playit["Playit Tunnel"]
    service["Kubernetes Service"]
    server["Minecraft Create Server"]

    players --> playit
    playit --> service
    service --> server

Tunnel credentials remain encrypted in Git.

The server is integrated into the wider NovaLabs monitoring stack.

I can observe:

• Minecraft service availability
• JVM / server metrics
• worker CPU and memory
• filesystem capacity
• storage utilization
• network activity
• node health
• public tunnel reachability

Logs are collected through Grafana Alloy → Loki and explored alongside infrastructure metrics in Grafana.

Minecraft telemetry is also included in the durable NovaLabs telemetry pipeline for historical analysis.

flowchart LR
    mc["Minecraft Create"]
    metrics["Metrics Exporters"]
    alloy["Grafana Alloy"]
    prom["Prometheus"]
    loki["Loki"]
    grafana["Grafana"]
    timescale["TimescaleDB"]

    mc --> metrics
    mc --> alloy

    metrics --> prom
    alloy --> loki

    prom --> grafana
    loki --> grafana

    prom -->|remote write| timescale
    timescale --> grafana

It is, perhaps unnecessarily, a GitOps-operated, observable, stateful distributed-systems exercise disguised as Minecraft.

And that is precisely why I built it.

A Compose-first model operations control plane for NVIDIA DGX Spark.

Built to manage the operational side of local AI infrastructure:

• Hugging Face model inventory
• local model metadata inspection
• Ollama model discovery and management
• Hugging Face model search and downloads
• DGX-aware deployment planning
• vLLM deployment workflows
• SGLang deployment workflows
• llama.cpp support
• Docker Compose lifecycle management
• serving-engine discovery
• LiteLLM routing
• host and service diagnostics
• authentication
• role-based access
• API tokens
• audit logging
• multi-node DGX Spark architecture

The primary development target is an:

NVIDIA DGX Spark · GB10 Grace Blackwell · 128 GB unified memory

The goal is to treat local models like actual infrastructure rather than a collection of shell scripts.

A centralized configuration and userland management system for the NovaLabs fleet using ChezMoi + Mise.

The architecture deliberately separates ownership:

Current profiles cover:

Proxmox · Docker · Pi-hole · UbuntuLab · Lovelace · k8s-worker-01 · DGX Spark · OPNsense

The repository manages:

• Bash environment
• Neovim + LazyVim
• pinned plugin state
• Tree-sitter tooling
• Rust toolchain
• Starship
• Fastfetch
• btop
• LazyGit
• LazyDocker
• tmux
• Vim
• Git configuration
• portable CLI utilities

Each server retains its own visual identity through generated:

• host-colored Starship prompts
• Fastfetch configurations
• btop themes

The repository is maintained primarily in Forgejo, with GitHub serving as the public mirror.

A significant part of NovaLabs is not simply running services.

It is building the instrumentation required to understand what those systems are actually doing.

My DGX Spark is shared by several different inference consumers:

• my own interactive AI workloads
• Yuki / Hermes
• applications
• background services
• friends using private inference access

I built a Grafana dashboard specifically to see that shared inference pipeline as an operational system.

The dashboard observes:

• vLLM availability
• LiteLLM availability
• running requests
• waiting requests
• completed requests
• request success rate
• scheduler concurrency

Because the GB10 uses a unified-memory architecture, memory behavior is particularly important.

I monitor:

• GPU utilization
• GPU temperature
• unified memory usage
• model-serving memory pressure
• KV-cache utilization

The dashboard exposes:

• token throughput
• prompt activity
• generation activity
• latency
• request volume
• model activity

LiteLLM gives me visibility above the raw inference-engine layer.

That means I can distinguish traffic generated by different consumers rather than merely seeing:

GPU busy.

I can observe inference usage associated with individual users and services, including separate Hermes activity.

flowchart LR
    nova["Nova"]
    yuki["Yuki / Hermes"]
    apps["Applications"]
    friends["Friends"]
    services["Background Services"]

    litellm["LiteLLM<br/>Routing + Attribution"]
    vllm["vLLM<br/>Inference Engine"]
    gb10["NVIDIA GB10<br/>Unified Memory"]

    prom["Prometheus"]
    grafana["Grafana<br/>Inference Operations"]

    nova --> litellm
    yuki --> litellm
    apps --> litellm
    friends --> litellm
    services --> litellm

    litellm --> vllm
    vllm --> gb10

    litellm -->|usage metrics| prom
    vllm -->|serving metrics| prom
    gb10 -->|hardware metrics| prom

    prom --> grafana

The result is an operational view of who is using the inference platform, what the serving engine is doing, and what that workload is doing to the hardware.

I built a dedicated observation layer around my over-the-air TV streaming environment while troubleshooting live-TV reliability problems with Plex and later migrating the workload to Jellyfin.

The tuner itself is an HDHomeRun, but the interesting part of the project became understanding the entire path a television stream takes through the lab.

flowchart LR
    antenna["OTA Antenna"]
    tuner["HDHomeRun<br/>Network TV Tuner"]
    network["NovaLabs Network"]

    plex["Plex"]
    jellyfin["Jellyfin"]

    clients["TV / Streaming Clients"]

    metrics["Prometheus<br/>System + Storage Metrics"]
    logs["Loki<br/>Application Logs"]
    grafana["Grafana<br/>TV Streaming Observation"]

    antenna --> tuner
    tuner --> network

    network --> plex
    network --> jellyfin

    plex --> clients
    jellyfin --> clients

    network --> metrics
    plex --> metrics
    jellyfin --> metrics

    plex --> logs
    jellyfin --> logs

    metrics --> grafana
    logs --> grafana

The project started because intermittent TV-streaming problems are difficult to diagnose from the player alone.

A stalled or degraded live stream could originate from several different layers:

• tuner reception
• network transport
• media-server behavior
• transcoding
• storage
• CPU or memory pressure
• client playback
• application instability

The observation environment lets me correlate streaming problems with the rest of the system instead of treating every playback failure as a generic Plex or Jellyfin problem.

I can examine things such as:

• media-server availability
• host CPU and memory
• filesystem capacity
• storage utilization
• network activity
• container health
• application logs
• tuner-related infrastructure
• transcoding behavior
• events occurring at the same time as a playback problem

This became particularly useful during my migration from Plex to Jellyfin, where I could compare behavior while changing the media-serving layer without changing the tuner or underlying network.

The result is less:

“The TV froze again.”

and more:

“What changed in the system at the exact moment the stream failed?”

That distinction is a recurring theme throughout NovaLabs: if a problem is intermittent, make it observable.

Accurate time is one of those infrastructure dependencies that is almost invisible until it goes wrong.

I built a dedicated time synchronization service and Grafana observation dashboard to understand how clocks across NovaLabs behave rather than relying solely on a binary synchronized: yes status.

The system tracks time synchronization as measurable infrastructure.

flowchart TD
    upstream["Upstream NTP Sources"]

    timesvc["NovaLabs<br/>Time Service"]

    proxmox["Proxmox"]
    docker["Docker / Monitoring"]
    lovelace["Lovelace"]
    worker["k3s Worker"]
    hosts["Other NovaLabs Hosts"]

    exporter["NTP / Clock Metrics"]
    prom["Prometheus"]
    grafana["Grafana<br/>Time Observation"]

    upstream --> timesvc

    timesvc --> proxmox
    timesvc --> docker
    timesvc --> lovelace
    timesvc --> worker
    timesvc --> hosts

    timesvc --> exporter
    proxmox --> exporter
    docker --> exporter
    lovelace --> exporter
    worker --> exporter

    exporter --> prom
    prom --> grafana

The dashboard focuses on questions that ordinary system status commands do not answer very well:

• Are the hosts synchronized?
• Which time source is currently being selected?
• What stratum am I operating at?
• How far is the local clock from its reference?
• How much correction is being applied?
• Is clock offset stable over time?
• Did synchronization behavior change after moving to another source?

One of the most interesting measurements is the observed clock offset.

Rather than simply seeing that synchronization is active, I can watch the clock continuously move around its reference at microsecond-scale resolution and see how that behavior changes when the selected time source changes.

flowchart LR
    source["NTP Reference"]
    clock["Host Clock"]
    correction["Clock Discipline"]
    offset["Observed Offset"]
    history["Prometheus / TimescaleDB"]
    grafana["Grafana"]

    source --> correction
    clock --> correction

    correction --> clock
    correction --> offset

    offset --> history
    history --> grafana

This turns NTP from a background daemon into another observable distributed system.

The goal is not extreme precision for its own sake. It is to ensure that every system generating:

• metrics
• logs
• alerts
• database records
• Kubernetes events
• security events

has a trustworthy concept of when something actually happened.

That becomes increasingly important as NovaLabs grows into a distributed environment where events from many independent machines need to line up on the same timeline.

Security work in NovaLabs includes a dedicated CVE tracking dashboard.

I use it to maintain visibility into vulnerabilities relevant to systems I actually operate rather than relying solely on generic vulnerability news.

The broader workflow includes:

• vulnerability tracking
• affected-system investigation
• CVE research
• remediation status
• patch documentation
• historical notes on vulnerabilities that were actually addressed

This is evolving into a broader cyber intelligence pipeline built around authoritative structured sources.

Current source work includes:

• CISA KEV
• FIRST EPSS
• ENISA EUVD
• ThreatFox
• privacy and regulatory intelligence sources

flowchart LR
    kev["CISA KEV"]
    epss["FIRST EPSS"]
    euvd["ENISA EUVD"]
    threatfox["ThreatFox"]
    privacy["Privacy / Regulatory Sources"]

    ingest["NovaLabs<br/>Intelligence Ingestion"]
    normalize["Normalize + Enrich"]
    datastore["Historical Data Store"]
    dashboard["CVE Tracker<br/>Dashboard"]

    kev --> ingest
    epss --> ingest
    euvd --> ingest
    threatfox --> ingest
    privacy --> ingest

    ingest --> normalize
    normalize --> datastore
    datastore --> dashboard

The long-term architecture separates vulnerability intelligence, exploitation likelihood, known-active exploitation, and privacy / sovereignty events while retaining historical state for analysis.

Live monitoring and historical telemetry serve different purposes.

So NovaLabs uses both.

flowchart TD
    hosts["Hosts + Services"]
    node["node_exporter"]
    apps["Application Exporters"]
    kube["Kubernetes Metrics"]

    prom["Prometheus<br/>Operational Monitoring"]
    grafana["Grafana<br/>Live Dashboards"]

    telegraf["Telegraf<br/>Remote Write Receiver"]
    timescale["TimescaleDB<br/>Durable Telemetry"]

    hosts --> node
    hosts --> apps
    hosts --> kube

    node --> prom
    apps --> prom
    kube --> prom

    prom --> grafana
    prom -->|remote write| telegraf

    telegraf --> timescale
    timescale -->|historical queries| grafana

Prometheus remains the live operational monitoring system.

TimescaleDB provides durable historical telemetry for:

• host health
• virtualization
• storage
• network behavior
• containers
• Minecraft
• infrastructure availability
• telemetry warehouse health

This allows Grafana to move beyond short retention windows and answer longer-term questions about how the lab behaves.

Yuki is my locally hosted AI agent environment.

She runs through a private stack built around:

• Hermes Agent
• LiteLLM
• vLLM
• local speech services
• tools
• persistent memory
• local inference on DGX Spark

The larger goal is not simply to run an LLM locally.

It is to understand how to operate an AI system.

flowchart LR
    user["User"]
    speech["Speech"]
    tools["Tools"]
    memory["Persistent Memory"]

    agent["Hermes Agent"]
    routing["LiteLLM"]
    inference["vLLM"]
    gpu["DGX Spark / GB10"]

    observability["Prometheus + Grafana"]

    user --> agent
    speech --> agent
    tools <--> agent
    memory <--> agent

    agent --> routing
    routing --> inference
    inference --> gpu

    agent --> observability
    routing --> observability
    inference --> observability
    gpu --> observability

Many Ears extends Yuki beyond conversational text and gives her a pipeline for listening to and processing audio.

The project grew from a simple idea:

What would an AI system need in order to actually listen?

The pipeline handles audio as something that can be processed, analyzed, and understood rather than simply played back.

Current work includes:

• audio ingestion
• music listening
• audio processing
• speech transcription
• speaker diarization
• processing longer recordings
• producing structured information that downstream AI systems can use

The architecture separates listening from reasoning.

flowchart LR
    audio["Audio / Music"]
    capture["Audio Capture"]
    processing["Audio Processing"]

    transcription["Transcription"]
    diarization["Speaker Diarization"]
    analysis["Audio Analysis"]

    context["Structured Context"]
    yuki["Yuki"]

    audio --> capture
    capture --> processing

    processing --> transcription
    processing --> diarization
    processing --> analysis

    transcription --> context
    diarization --> context
    analysis --> context

    context --> yuki

The important distinction is that Yuki does not need to perform every expensive audio operation inside the conversational model itself.

Many Ears acts as the perception layer.

A separate multimodal experiment exploring an AI that can share the user's visual and auditory environment.

flowchart LR
    subgraph pc["Windows Client"]
        mic["Microphone"]
        screen["Screen Capture"]
        client["Conversation Client"]
        memory["Persistent Memory"]
    end

    subgraph dgx["DGX Spark"]
        stt["Whisper STT"]
        vision["Vision-Language Model"]
        litellm["LiteLLM"]
        vllm["vLLM"]
        tts["Local TTS"]
    end

    mic --> stt
    screen --> vision

    stt --> client
    vision --> client
    memory <--> client

    client --> litellm
    litellm --> vllm

    vllm --> tts
    tts --> client

The client combines:

• microphone input
• screen capture
• speech recognition
• visual context
• conversational inference
• persistent memory
• local text-to-speech

The project explores what locally hosted conversational AI looks like when it can perceive the same environment as the person using it.

Right now I'm particularly interested in:

• GitOps and Kubernetes platform engineering
• durable observability and telemetry architecture
• Grafana as an investigative interface rather than just a status screen
• local AI inference infrastructure
• multi-user AI serving
• AI observability and inference economics
• audio and multimodal AI perception
• cyber vulnerability intelligence
• Linux systems engineering
• networking and CCNA fundamentals
• security, segmentation, and infrastructure hardening
• self-hosted developer platforms
• reproducible workstation and server environments
• stateful workloads on Kubernetes
• turning manual operations into documented, version-controlled systems

My background is in manufacturing systems, where reliability is not theoretical: equipment has to run, problems need root causes, and fixes need to survive the next shift.

I bring the same mindset to infrastructure.

Inspect before changing. Understand ownership boundaries. Make the desired state explicit. Automate the repeatable parts. Instrument the system. Observe what actually happened. Document what you learned.

Most of my projects start with:

“I want to understand how this works.”

Eventually that tends to become a:

repository · dashboard · runbook · service · pipeline · automation

I don't want the lab to hide complexity from me.

I want it to make complexity observable, explainable, and manageable.

NovaLabs is intentionally a learning environment.

I use real infrastructure to explore:

• Kubernetes architecture and operations
• GitOps workflows
• routing and switching
• VLANs
• OSPF
• Linux administration
• PostgreSQL and time-series systems
• monitoring and telemetry
• infrastructure security
• vulnerability intelligence
• backend development
• containerization
• local AI systems
• inference serving
• multimodal AI
• audio processing

The repositories here aren't meant to present a magically finished environment.

They're a record of systems being:

designed → deployed → observed → broken → understood → improved

• GitHub: calico88x
• Portfolio: NovaLabs
• LinkedIn: Nova Peck
• Telegram: @nova88x
• X: @xCalico88x

“Learn to manage AI, or AI will learn to manage you.”