Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,25 @@ A time-based trigger that fires an action — sending a message or dispatching (
_Avoid_: cron job (recurring only), scheduled message (too narrow), reminder, timer
_See also_: Dispatch

## Observability

Scion produces two distinct families of metrics. They serve different audiences, use different prefixes, and flow through different pipelines — but both export to the same Cloud Monitoring backend.

**Infrastructure metrics**:
Operational health metrics for Scion as a system — the Hub process, its database connections, dispatch pipeline, broker authentication, and GCP token minting. These answer "is Scion itself healthy?" and are consumed by platform operators. Prefixes: `scion.hub.*`, `scion.db.*`, `scion.dispatch.*`. Produced by the Hub process; exported directly to Cloud Monitoring via an OTel MeterProvider with a GCP exporter.
_Avoid_: system metrics, platform metrics, server metrics
_See also_: Agent metrics (the other family)

**Agent metrics**:
Telemetry about what agents and their harnesses are doing — token usage, tool calls, model API latency, session counts, and cost signals. These answer "what are the agents doing and what do they cost?" and are consumed by users and project owners. Prefixes: `gen_ai.*`, `agent.*` (following OpenTelemetry Generative AI semantic conventions). Produced inside agent containers by the harness and sciontool; exported to Cloud Monitoring via the telemetry pipeline (`pkg/sciontool/telemetry`).
_Avoid_: harness metrics, user metrics, LLM metrics
_See also_: Infrastructure metrics (the other family), Telemetry pipeline

**Telemetry pipeline**:
The in-container OTLP receiver and forwarding pipeline (`pkg/sciontool/telemetry`) that collects traces, metrics, and logs from the harness and exports them to a cloud backend (GCP Cloud Monitoring, Cloud Trace, Cloud Logging). Requires the `scion-telemetry-gcp-credentials` secret for cloud export; runs in local-only mode without it.
_Avoid_: metrics pipeline, collector, OTel collector
_See also_: Agent metrics

## Potential Future Additions

Terms that recur in the codebase and may warrant canonical entries, but are **not yet defined** here. Listed so they aren't lost; promote to full entries (verified against the code) as the glossary matures.
Expand Down
14 changes: 1 addition & 13 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ GOLANGCI_LINT := $(shell command -v golangci-lint 2>/dev/null || echo $(shell go

.DEFAULT_GOAL := help

.PHONY: all build install test test-fast vet lint compat-literals golangci-lint web web-typecheck fmt fmt-check ci ci-full clean help container-sciontool container-scion container-binaries proto proto-check
.PHONY: all build install test test-fast vet lint compat-literals golangci-lint web web-typecheck fmt fmt-check ci ci-full clean help container-sciontool container-scion container-binaries

## all: Build the web frontend, then compile the Go binary with embedded assets
all: web install
Expand Down Expand Up @@ -142,18 +142,6 @@ ci-full: fmt-check web web-typecheck lint compat-literals golangci-lint test-fas
@echo ""
@echo "CI (full) passed."

## proto: Generate Go code from .proto files
proto:
@echo "Generating proto code..."
@protoc --go_out=. --go_opt=paths=source_relative \
--go-grpc_out=. --go-grpc_opt=paths=source_relative \
proto/broker/v1/broker.proto
@echo "Proto generation done."

## proto-check: Verify generated proto code is up to date
proto-check: proto
@git diff --exit-code proto/ || (echo "Proto generated code is stale. Run 'make proto'" && exit 1)

## clean: Remove build artifacts
clean:
@echo "Cleaning..."
Expand Down
181 changes: 181 additions & 0 deletions cmd/build.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
// Copyright 2026 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package cmd

import (
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"

"github.com/GoogleCloudPlatform/scion/pkg/config"
"github.com/GoogleCloudPlatform/scion/pkg/runtime"
"github.com/spf13/cobra"
"gopkg.in/yaml.v3"
)

var (
buildTag string
buildBaseImage string
buildPush bool
buildPlatform string
buildDryRun bool
)

var buildCmd = &cobra.Command{
Use: "build <harness-config-name>",
Short: "Build a container image from a harness-config Dockerfile",
Long: `Build a container image from a Dockerfile bundled inside a harness-config directory.

The base image is resolved from the image_registry setting unless --base-image
is provided. After a successful build the harness-config's config.yaml image
field is updated to reference the built image.`,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
harnessConfigName := args[0]

hcDir, err := config.FindHarnessConfigDir(harnessConfigName, projectPath)
if err != nil {
return fmt.Errorf("harness-config %q not found: %w", harnessConfigName, err)
}
if hcDir.Path == "" {
return fmt.Errorf("harness-config %q does not have a local directory path", harnessConfigName)
}

dockerfilePath := filepath.Join(hcDir.Path, "Dockerfile")
if _, err := os.Stat(dockerfilePath); err != nil {
if os.IsNotExist(err) {
return fmt.Errorf("harness-config %q does not contain a Dockerfile", harnessConfigName)
}
return fmt.Errorf("cannot access Dockerfile in harness-config %q: %w", harnessConfigName, err)
}

tag := buildTag

var settings *config.VersionedSettings
if buildBaseImage == "" || buildPush {
settings, _, err = config.LoadEffectiveSettings(projectPath)
if err != nil {
return fmt.Errorf("failed to load settings: %w", err)
}
}

baseImage := buildBaseImage
if baseImage == "" {
imageRegistry := ""
if settings != nil {
imageRegistry = settings.ResolveImageRegistry(profile)
}
baseImage = "scion-base:" + tag
if imageRegistry != "" {
baseImage = imageRegistry + "/scion-base:" + tag
}
}

runtimeBin := runtime.DetectContainerRuntime()
if runtimeBin == "" {
return fmt.Errorf("no container runtime found (tried docker, podman)")
}

outputImage := harnessConfigName + ":" + tag
if buildPush {
imageRegistry := ""
if settings != nil {
imageRegistry = settings.ResolveImageRegistry(profile)
}
if imageRegistry == "" {
return fmt.Errorf("--push requires image_registry to be configured")
}
outputImage = imageRegistry + "/" + harnessConfigName + ":" + tag
}

buildArgs := []string{"build",
"--build-arg", "BASE_IMAGE=" + baseImage,
"-t", outputImage,
}
if buildPlatform != "" {
buildArgs = append(buildArgs, "--platform", buildPlatform)
}
buildArgs = append(buildArgs, hcDir.Path)

if buildDryRun {
fmt.Println(runtimeBin + " " + strings.Join(buildArgs, " "))
return nil
}

buildExec := exec.CommandContext(cmd.Context(), runtimeBin, buildArgs...)
buildExec.Stdout = os.Stdout
buildExec.Stderr = os.Stderr
if err := buildExec.Run(); err != nil {
return fmt.Errorf("build failed: %w", err)
}

if buildPush {
pushExec := exec.CommandContext(cmd.Context(), runtimeBin, "push", outputImage)
pushExec.Stdout = os.Stdout
pushExec.Stderr = os.Stderr
if err := pushExec.Run(); err != nil {
return fmt.Errorf("push failed: %w", err)
}
}

configPath := filepath.Join(hcDir.Path, "config.yaml")
configData, err := os.ReadFile(configPath)
if err != nil {
return fmt.Errorf("failed to read config.yaml for update: %w", err)
}
var doc yaml.Node
if err := yaml.Unmarshal(configData, &doc); err != nil {
return fmt.Errorf("failed to parse config.yaml: %w", err)
}
if len(doc.Content) > 0 && doc.Content[0].Kind == yaml.MappingNode {
mapping := doc.Content[0]
found := false
for i := 0; i < len(mapping.Content)-1; i += 2 {
if mapping.Content[i].Value == "image" {
mapping.Content[i+1].Value = outputImage
found = true
break
}
}
if !found {
mapping.Content = append(mapping.Content,
&yaml.Node{Kind: yaml.ScalarNode, Value: "image"},
&yaml.Node{Kind: yaml.ScalarNode, Value: outputImage},
)
}
}
Comment on lines +144 to +160

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If config.yaml is empty or its root is not a mapping node, the image field update is silently skipped, but the command still prints a misleading success message. It is safer to validate that the root is a mapping node and return an error if it is not.

Suggested change
if len(doc.Content) > 0 && doc.Content[0].Kind == yaml.MappingNode {
mapping := doc.Content[0]
found := false
for i := 0; i < len(mapping.Content)-1; i += 2 {
if mapping.Content[i].Value == "image" {
mapping.Content[i+1].Value = outputImage
found = true
break
}
}
if !found {
mapping.Content = append(mapping.Content,
&yaml.Node{Kind: yaml.ScalarNode, Value: "image"},
&yaml.Node{Kind: yaml.ScalarNode, Value: outputImage},
)
}
}
if len(doc.Content) == 0 || doc.Content[0].Kind != yaml.MappingNode {
return fmt.Errorf("invalid config.yaml: root must be a mapping")
}
mapping := doc.Content[0]
found := false
for i := 0; i < len(mapping.Content)-1; i += 2 {
if mapping.Content[i].Value == "image" {
mapping.Content[i+1].Value = outputImage
found = true
break
}
}
if !found {
mapping.Content = append(mapping.Content,
&yaml.Node{Kind: yaml.ScalarNode, Value: "image"},
&yaml.Node{Kind: yaml.ScalarNode, Value: outputImage},
)
}

updatedData, err := yaml.Marshal(&doc)
if err != nil {
return fmt.Errorf("failed to marshal updated config.yaml: %w", err)
}
if err := os.WriteFile(configPath, updatedData, 0644); err != nil {
return fmt.Errorf("failed to write updated config.yaml: %w", err)
}
fmt.Printf("Updated %s image to %s\n", configPath, outputImage)

return nil
},
}

func init() {
rootCmd.AddCommand(buildCmd)
buildCmd.Flags().StringVar(&buildTag, "tag", "latest", "Image tag")
buildCmd.Flags().StringVar(&buildBaseImage, "base-image", "", "Override the base image (skips image_registry resolution)")
buildCmd.Flags().BoolVar(&buildPush, "push", false, "Push built image to image_registry after building")
buildCmd.Flags().StringVar(&buildPlatform, "platform", "", "Target platform (default: current architecture)")
buildCmd.Flags().BoolVar(&buildDryRun, "dry-run", false, "Show the docker build command without executing")
}
10 changes: 10 additions & 0 deletions cmd/cli_mode.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,16 @@ var agentAllowed = map[string]bool{
"template.push": true,
"template.pull": true,
"template.status": true,
"harness-config": true,
"harness-config.list": true,
"harness-config.show": true,
"harness-config.install": true,
"harness-config.sync": true,
"harness-config.push": true,
"harness-config.pull": true,
"harness-config.delete": true,
"harness-config.reset": true,
"harness-config.upgrade": true,
}

// resolveMode determines the active CLI mode from environment and settings.
Expand Down
7 changes: 5 additions & 2 deletions cmd/cli_mode_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@ func TestApplyModeRestrictions_Agent(t *testing.T) {
// These commands should be present in agent mode
expected := []string{
"create", "delete",
"harness-config", "harness-config.install", "harness-config.list",
"help",
"list", "logs", "look",
"message",
Expand Down Expand Up @@ -304,7 +305,7 @@ func TestApplyModeRestrictions_Agent(t *testing.T) {
// These should be removed
absent := []string{
"attach", "broker", "cdw", "clean", "completion", "config", "doctor",
"grove", "harness-config", "hub",
"grove", "hub",
"init", "messages", "restore", "server", "sync",
}
for _, cmd := range absent {
Expand Down Expand Up @@ -423,6 +424,9 @@ func TestAgentAllowedList(t *testing.T) {
"template", "template.list", "template.show", "template.clone",
"template.delete", "template.import", "template.sync",
"template.push", "template.pull", "template.status",
"harness-config", "harness-config.list", "harness-config.show", "harness-config.install",
"harness-config.sync", "harness-config.push", "harness-config.pull",
"harness-config.delete", "harness-config.reset", "harness-config.upgrade",
}
for _, path := range expectedAllowed {
assert.True(t, agentAllowed[path], "agentAllowed should contain %s", path)
Expand All @@ -432,7 +436,6 @@ func TestAgentAllowedList(t *testing.T) {
"attach", "restore", "sync", "clean", "cdw", "init",
"completion", "config", "doctor", "hub", "messages",
"server", "broker", "grove",
"harness-config",
"config.set", "config.validate", "config.migrate",
"config.list", "config.get", "config.dir", "config.schema",
"hub.enable", "hub.disable", "hub.link", "hub.unlink",
Expand Down
13 changes: 11 additions & 2 deletions cmd/sciontool/commands/secret.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,13 +73,22 @@ Examples:
// Handle @file syntax: read file and base64-encode contents.
if strings.HasPrefix(value, "@") {
filePath := value[1:]
if strings.HasPrefix(filePath, "~/") {
if filePath == "~" || strings.HasPrefix(filePath, "~/") {
home, err := os.UserHomeDir()
if err != nil {
log.Error("Failed to expand home directory: %v", err)
os.Exit(1)
}
filePath = home + filePath[1:]
filePath = filepath.Join(home, strings.TrimPrefix(filePath[1:], "/"))
}
info, err := os.Stat(filePath)
if err != nil {
log.Error("Failed to stat file %s: %v", filePath, err)
os.Exit(1)
}
if info.Size() > 64*1024 {
log.Error("File exceeds 64KB limit (%d bytes)", info.Size())
os.Exit(1)
}
Comment on lines +84 to 92

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If the resolved filePath is a directory, os.Stat will succeed but the subsequent os.ReadFile will fail. Adding an explicit check for info.IsDir() provides a clearer error message and prevents trying to read a directory as a secret file.

			info, err := os.Stat(filePath)
			if err != nil {
				log.Error("Failed to stat file %s: %v", filePath, err)
				os.Exit(1)
			}
			if info.IsDir() {
				log.Error("File %s is a directory", filePath)
				os.Exit(1)
			}
			if info.Size() > 64*1024 {
				log.Error("File exceeds 64KB limit (%d bytes)", info.Size())
				os.Exit(1)
			}

data, err := os.ReadFile(filePath)
if err != nil {
Expand Down
Loading
Loading