From 802d70e69e9828b76c2780dda9108640bc945d28 Mon Sep 17 00:00:00 2001 From: Vanshika Date: Mon, 18 May 2026 20:09:11 +0530 Subject: [PATCH] docs: translate the PipeCD plugin development book to English Signed-off-by: Vanshika --- .../plugin-development/01-introduction.md | 25 ++ .../plugin-development/02-plugin-features.md | 19 + .../plugin-development/03-tech-selection.md | 14 + .../plugin-development/04-project-init.md | 18 + .../plugin-development/05-dependencies.md | 14 + .../plugin-development/06-plugin-types.md | 20 + .../plugin-development/07-first-steps.md | 43 +++ .../08-deployment-plugin-interface.md | 27 ++ .../09-defining-config-types.md | 31 ++ .../10-empty-implementation.md | 57 +++ .../11-fetch-defined-stages.md | 28 ++ .../12-determine-versions.md | 22 ++ .../13-determine-strategy.md | 21 ++ .../14-build-pipeline-sync-stages.md | 64 ++++ .../15-build-quick-sync-stages.md | 40 ++ .../plugin-development/16-execute-stage.md | 44 +++ .../17-execute-stage-diff.md | 353 ++++++++++++++++++ .../18-execute-stage-sync.md | 233 ++++++++++++ .../19-execute-stage-rollback.md | 45 +++ .../plugin-development/20-updating-main.md | 33 ++ .../21-trying-with-piped.md | 171 +++++++++ .../plugin-development/22-conclusion.md | 17 + .../en/docs-dev/plugin-development/_index.md | 36 ++ 23 files changed, 1375 insertions(+) create mode 100644 docs/content/en/docs-dev/plugin-development/01-introduction.md create mode 100644 docs/content/en/docs-dev/plugin-development/02-plugin-features.md create mode 100644 docs/content/en/docs-dev/plugin-development/03-tech-selection.md create mode 100644 docs/content/en/docs-dev/plugin-development/04-project-init.md create mode 100644 docs/content/en/docs-dev/plugin-development/05-dependencies.md create mode 100644 docs/content/en/docs-dev/plugin-development/06-plugin-types.md create mode 100644 docs/content/en/docs-dev/plugin-development/07-first-steps.md create mode 100644 docs/content/en/docs-dev/plugin-development/08-deployment-plugin-interface.md create mode 100644 docs/content/en/docs-dev/plugin-development/09-defining-config-types.md create mode 100644 docs/content/en/docs-dev/plugin-development/10-empty-implementation.md create mode 100644 docs/content/en/docs-dev/plugin-development/11-fetch-defined-stages.md create mode 100644 docs/content/en/docs-dev/plugin-development/12-determine-versions.md create mode 100644 docs/content/en/docs-dev/plugin-development/13-determine-strategy.md create mode 100644 docs/content/en/docs-dev/plugin-development/14-build-pipeline-sync-stages.md create mode 100644 docs/content/en/docs-dev/plugin-development/15-build-quick-sync-stages.md create mode 100644 docs/content/en/docs-dev/plugin-development/16-execute-stage.md create mode 100644 docs/content/en/docs-dev/plugin-development/17-execute-stage-diff.md create mode 100644 docs/content/en/docs-dev/plugin-development/18-execute-stage-sync.md create mode 100644 docs/content/en/docs-dev/plugin-development/19-execute-stage-rollback.md create mode 100644 docs/content/en/docs-dev/plugin-development/20-updating-main.md create mode 100644 docs/content/en/docs-dev/plugin-development/21-trying-with-piped.md create mode 100644 docs/content/en/docs-dev/plugin-development/22-conclusion.md create mode 100644 docs/content/en/docs-dev/plugin-development/_index.md diff --git a/docs/content/en/docs-dev/plugin-development/01-introduction.md b/docs/content/en/docs-dev/plugin-development/01-introduction.md new file mode 100644 index 0000000000..59afc63ac2 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/01-introduction.md @@ -0,0 +1,25 @@ +--- +title: "Introduction" +weight: 1 +description: > + Introduction to the PipeCD Pluggable Architecture and the goal of this book. +--- + +### About this Book + +PipeCD is an open-source, GitOps-style Continuous Delivery platform. Recently, the PipeCD project introduced a new **Plugin Architecture** in its Alpha release to allow developers to flexibly extend PipeCD's deployment capabilities to support arbitrary platforms and tools. + +In this book, we will learn how to build a custom PipeCD plugin from scratch by implementing a plugin named `file`. This plugin will allow PipeCD to manage and sync files on a local file system. + +### Reference Links + +Here are some useful reference links regarding PipeCD and its Plugin Architecture: + +- [Official PipeCD Website](https://pipecd.dev/) +- [Official Plugin SDK for Go](https://pkg.go.dev/github.com/pipe-cd/piped-plugin-sdk-go) +- [Official Blog Post: Overview of the Plan for Pluggable PipeCD](https://pipecd.dev/blog/2024/11/28/overview-of-the-plan-for-pluginnable-pipecd/) +- [Japanese Article: Explaining the Plugin Architecture](https://zenn.dev/cadp/articles/pipecd-plugin-intro) + +### Note on SDK Stability + +This book is based on the PipeCD Plugin SDK version `v0.0.0-20250619080234-1ee9423d23c1` released during the Alpha stage. As pluggable architecture is continuously evolving, we recommend verifying Go interfaces, RPC definitions, and the `pipedv1` architecture against the latest default branch in the [`pipecd`](https://github.com/pipe-cd/pipecd) repository when implementing production-grade plugins. diff --git a/docs/content/en/docs-dev/plugin-development/02-plugin-features.md b/docs/content/en/docs-dev/plugin-development/02-plugin-features.md new file mode 100644 index 0000000000..2aac34b3b5 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/02-plugin-features.md @@ -0,0 +1,19 @@ +--- +title: "Plugin Features to Implement" +weight: 2 +description: > + Determining the features and stages our custom plugin will provide. +--- + +First, let's decide what kind of features our custom plugin will provide. + +In this book, we will implement a plugin that manages files on the local machine where `piped` is running. While `piped` is often run in containerized environments (such as Kubernetes), it can also be run directly on VMs or bare-metal machines. Practically, a local file-system plugin like this could be used to deploy configuration files onto a VM—taking over some of the tasks typically handled by tools like Chef or Ansible. + +### Specific Features to Implement + +Each PipeCD Application will have a designated target deployment directory. The plugin's job is to copy files located under the same directory as the `app.pipecd.yaml` configuration file from the Git repository directly into that target directory. + +We will define two deployment stages for this plugin: + +1. **`FILE_DIFF`**: This stage compares the files in the Git repository with the files actually deployed in the target directory, printing the differences to the deployment log. This is similar to running `terraform plan` or `kubectl diff`. +2. **`FILE_SYNC`**: This stage executes the actual sync operation by copying all files from the Git repository to the target directory and deleting any orphaned files in the target directory that do not exist in the Git repository. diff --git a/docs/content/en/docs-dev/plugin-development/03-tech-selection.md b/docs/content/en/docs-dev/plugin-development/03-tech-selection.md new file mode 100644 index 0000000000..ee4893a760 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/03-tech-selection.md @@ -0,0 +1,14 @@ +--- +title: "Technology Selection" +weight: 3 +description: > + Selecting the language and development tools for our plugin. +--- + +Next, let's select our development language and tools. + +When developing plugins for PipeCD, **Go** is the most natural choice because PipeCD officially provides the [piped-plugin-sdk-go](https://pkg.go.dev/github.com/pipe-cd/piped-plugin-sdk-go). Therefore, we will build our plugin using Go. + +To keep things educational and focused on understanding the core SDK, we will use the standard library as much as possible alongside the official SDK, avoiding any third-party dependencies. + +The version of the SDK used in this book is `v0.0.0-20250619080234-1ee9423d23c1`. Since pluggable architecture is continuously evolving, always check the latest official documentation when developing production-ready plugins. diff --git a/docs/content/en/docs-dev/plugin-development/04-project-init.md b/docs/content/en/docs-dev/plugin-development/04-project-init.md new file mode 100644 index 0000000000..fc8432d6a3 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/04-project-init.md @@ -0,0 +1,18 @@ +--- +title: "Initializing the Project" +weight: 4 +description: > + Initializing the Git repository and Go module. +--- + +Let's begin by creating our Git repository and initializing a Go module. + +Run the following commands in your terminal in a suitable working directory. Replace `` with your GitHub username or any appropriate identifier: + +```console +$ git init pipecd-plugin-file +$ cd pipecd-plugin-file +$ go mod init github.com//pipecd-plugin-file +``` + +Once you have initialized the project, make your initial git commit to save your progress. We highly recommend committing your work at small, logical milestones throughout this guide. diff --git a/docs/content/en/docs-dev/plugin-development/05-dependencies.md b/docs/content/en/docs-dev/plugin-development/05-dependencies.md new file mode 100644 index 0000000000..2d3872cc31 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/05-dependencies.md @@ -0,0 +1,14 @@ +--- +title: "Adding Dependency Libraries" +weight: 5 +description: > + Adding the official PipeCD plugin SDK to our project dependencies. +--- + +To enable autocompletion and type checking in your editor, add the official PipeCD Plugin SDK to your Go module dependencies by running: + +```console +$ go get github.com/pipe-cd/piped-plugin-sdk-go +``` + +As mentioned earlier, we are focusing on building our understanding of the core SDK, so we will not be adding any other third-party dependencies. diff --git a/docs/content/en/docs-dev/plugin-development/06-plugin-types.md b/docs/content/en/docs-dev/plugin-development/06-plugin-types.md new file mode 100644 index 0000000000..ce89c9585c --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/06-plugin-types.md @@ -0,0 +1,20 @@ +--- +title: "Understanding Plugin Types" +weight: 6 +description: > + Deep dive into the types of plugins supported by the PipeCD SDK. +--- + +While PipeCD's core agent `Piped` interacts with plugins transparently, the official Go SDK categorizes plugins into distinct types to simplify development. + +### 1. [StagePlugin](https://pkg.go.dev/github.com/pipe-cd/piped-plugin-sdk-go#StagePlugin) + +A utility plugin that does not manage resources but provides utility stages to be executed within a deployment pipeline. An example is a plugin that provides a `WAIT` stage to pause the deployment pipeline for a defined duration. + +### 2. [DeploymentPlugin](https://pkg.go.dev/github.com/pipe-cd/piped-plugin-sdk-go#DeploymentPlugin) + +A plugin that manages actual target resources and performs state synchronization (syncing, diffing, rolling back). A Kubernetes plugin is a prime example. + +In addition to core sync capabilities, a `DeploymentPlugin` can also implement the [LiveStatePlugin](https://pkg.go.dev/github.com/pipe-cd/piped-plugin-sdk-go#LiveStatePlugin) interface to report the live status of active resources on the target platform and render differences in the PipeCD Web UI. + +When starting a plugin project, choose the type that fits your goals and implement its required interface. Since our `file` plugin manages local files as target resources, it will be implemented as a **`DeploymentPlugin`**. diff --git a/docs/content/en/docs-dev/plugin-development/07-first-steps.md b/docs/content/en/docs-dev/plugin-development/07-first-steps.md new file mode 100644 index 0000000000..582d251e83 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/07-first-steps.md @@ -0,0 +1,43 @@ +--- +title: "First Step of Plugin Implementation" +weight: 7 +description: > + Creating the main function and starting the gRPC server. +--- + +Let's begin writing our plugin code by implementing the `main` function. + +Create a file named `main.go` and save the following code: + +```go +package main + +import ( + "log" + + sdk "github.com/pipe-cd/piped-plugin-sdk-go" +) + +func main() { + plugin, err := sdk.NewPlugin[any, any, any]("0.0.1") + if err != nil { + log.Fatalln(err) + } + + if err := plugin.Run(); err != nil { + log.Fatalln(err) + } +} +``` + +The type parameters passed to `sdk.NewPlugin` are temporary placeholders. By the time we finish implementing the plugin, Go's type inference will determine these automatically, but for now we must explicitly write them to compile. + +### Note on Plugin Naming + +You might have noticed that the string `file` (the name of our plugin) does not appear anywhere in this code. + +Indeed, a plugin binary has no concept of its own name. The naming is entirely determined in the Piped configuration written by the user. A user could configure Piped to recognize this plugin binary as `file`, or `filesystem`, or any other name. The name itself is decoupled from the binary. + +### Running this Code + +At this point, if you attempt to run `go run main.go`, it will fail because `sdk.NewPlugin` returns an error if no plugin implementation is registered. In the next section, we will define a struct and satisfy the `DeploymentPlugin` interface. diff --git a/docs/content/en/docs-dev/plugin-development/08-deployment-plugin-interface.md b/docs/content/en/docs-dev/plugin-development/08-deployment-plugin-interface.md new file mode 100644 index 0000000000..e60d32df52 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/08-deployment-plugin-interface.md @@ -0,0 +1,27 @@ +--- +title: "Satisfying the DeploymentPlugin Interface" +weight: 8 +description: > + Analyzing the Go interface methods required to implement a DeploymentPlugin. +--- + +To register a plugin as a [DeploymentPlugin](https://pkg.go.dev/github.com/pipe-cd/piped-plugin-sdk-go#DeploymentPlugin) via `sdk.NewPlugin`, we must implement its interface methods. + +First, let's understand the three type parameters required by `DeploymentPlugin`: + +1. **`Config`**: The global configuration common to all instances of this plugin. This is defined in Piped's global configuration file. +2. **`DeployTargetConfig`**: The configuration unique to each deployment target. For example, in a Kubernetes plugin, this represents the target cluster credentials and context. +3. **`ApplicationConfigSpec`**: The configuration specified per application in `app.pipecd.yaml`. For example, a list of target manifest files or resource configurations. + +### Interface Methods + +To fully implement a `DeploymentPlugin` (including the embedded `StagePlugin` methods), we must define the following methods on our plugin struct: + +- `FetchDefinedStages() []string` +- `DetermineVersions(context.Context, *Config, *DetermineVersionsInput[ApplicationConfigSpec]) (*DetermineVersionsResponse, error)` +- `DetermineStrategy(context.Context, *Config, *DetermineStrategyInput[ApplicationConfigSpec]) (*DetermineStrategyResponse, error)` +- `BuildPipelineSyncStages(context.Context, *Config, *BuildPipelineSyncStagesInput) (*BuildPipelineSyncStagesResponse, error)` +- `BuildQuickSyncStages(context.Context, *Config, *BuildQuickSyncStagesInput) (*BuildQuickSyncStagesResponse, error)` +- `ExecuteStage(context.Context, *Config, []*DeployTarget[DeployTargetConfig], *ExecuteStageInput[ApplicationConfigSpec]) (*ExecuteStageResponse, error)` + +We will implement each of these methods step-by-step. diff --git a/docs/content/en/docs-dev/plugin-development/09-defining-config-types.md b/docs/content/en/docs-dev/plugin-development/09-defining-config-types.md new file mode 100644 index 0000000000..a377755ca0 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/09-defining-config-types.md @@ -0,0 +1,31 @@ +--- +title: "Implementation: Defining Configuration Types" +weight: 9 +description: > + Defining the configuration Go structs for our file plugin. +--- + +Our `file` plugin needs to allow users to specify a unique target directory path per application. While we could theoretically define this in a Deploy Target, requiring a separate Deploy Target for every single directory would be tedious for users. Therefore, we will define the target directory path within each application's configuration. + +For future extensions—for instance, if we want this plugin to deploy files not just to the local machine but to remote hosts via SSH—the SSH connection details (host, keys, etc.) would be defined within the Deploy Target configuration, whereas the specific path on that host would remain in the Application configuration. + +Add the following type definitions to `main.go` (or a separate file under `package main`): + +```go +type ( + // config represents the global plugin configuration. + // Since our file plugin does not require global settings, we leave it empty. + config struct{} + + // deployTargetConfig represents the deployment target settings. + // Since our file plugin only interacts with the local machine, we leave it empty. + deployTargetConfig struct{} + + // applicationConfig represents the configuration defined per application in app.pipecd.yaml. + // Here, we define the target directory path where the files should be synchronized. + applicationConfig struct { + // Path specifies the absolute path on the local file system where the files should be synced. + Path string `json:"path"` + } +) +``` diff --git a/docs/content/en/docs-dev/plugin-development/10-empty-implementation.md b/docs/content/en/docs-dev/plugin-development/10-empty-implementation.md new file mode 100644 index 0000000000..d08526c5f1 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/10-empty-implementation.md @@ -0,0 +1,57 @@ +--- +title: "Implementation: Empty Implementation to Satisfy the Interface" +weight: 10 +description: > + Scaffolding the plugin struct and satisfying the interface with stub methods. +--- + +Before implementing the logic for each method, let's establish a skeleton structure in `main.go`. This guarantees our method signatures match the expected interface and allows the project to compile. + +Add the following code to `main.go` (or a separate file under `package main`): + +```go +package main + +import ( + "context" + + sdk "github.com/pipe-cd/piped-plugin-sdk-go/pkg/plugin/sdk" +) + +// Ensure that plugin implements the sdk.DeploymentPlugin interface. +// If any methods are missing or have incorrect signatures, this will trigger a compile-time error. +var _ sdk.DeploymentPlugin[config, deployTargetConfig, applicationConfig] = plugin{} + +// plugin provides the implementation of the DeploymentPlugin interface. +type plugin struct{} + +// FetchDefinedStages returns the list of stages supported by this plugin. +func (plugin) FetchDefinedStages() []string { + panic("unimplemented") +} + +// DetermineVersions determines the version of the resource being deployed. +func (plugin) DetermineVersions(ctx context.Context, cfg *config, input *sdk.DetermineVersionsInput[applicationConfig]) (*sdk.DetermineVersionsResponse, error) { + panic("unimplemented") +} + +// DetermineStrategy decides whether to execute a Quick Sync or a Pipeline Sync. +func (plugin) DetermineStrategy(ctx context.Context, cfg *config, input *sdk.DetermineStrategyInput[applicationConfig]) (*sdk.DetermineStrategyResponse, error) { + panic("unimplemented") +} + +// BuildPipelineSyncStages constructs the pipeline stages for a Pipeline Sync. +func (plugin) BuildPipelineSyncStages(ctx context.Context, cfg *config, input *sdk.BuildPipelineSyncStagesInput) (*sdk.BuildPipelineSyncStagesResponse, error) { + panic("unimplemented") +} + +// BuildQuickSyncStages constructs the stages for a Quick Sync. +func (plugin) BuildQuickSyncStages(ctx context.Context, cfg *config, input *sdk.BuildQuickSyncStagesInput) (*sdk.BuildQuickSyncStagesResponse, error) { + panic("unimplemented") +} + +// ExecuteStage runs the actual sync logic for a specific stage. +func (plugin) ExecuteStage(ctx context.Context, cfg *config, targets []*sdk.DeployTarget[deployTargetConfig], input *sdk.ExecuteStageInput[applicationConfig]) (*sdk.ExecuteStageResponse, error) { + panic("unimplemented") +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/11-fetch-defined-stages.md b/docs/content/en/docs-dev/plugin-development/11-fetch-defined-stages.md new file mode 100644 index 0000000000..cfdafb479a --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/11-fetch-defined-stages.md @@ -0,0 +1,28 @@ +--- +title: "Implementation: FetchDefinedStages" +weight: 11 +description: > + Implementing FetchDefinedStages to define custom deployment stages. +--- + +`FetchDefinedStages` returns the names of the stages supported by this plugin. + +Our `file` plugin will support three stages: `FILE_DIFF`, `FILE_SYNC`, and `FILE_ROLLBACK`. Defining these stage names as constants is best practice so we can reuse them across other methods. + +Update the `FetchDefinedStages` implementation and define the constants as follows: + +```go +const ( + stageDiff = "FILE_DIFF" + stageSync = "FILE_SYNC" + stageRollback = "FILE_ROLLBACK" +) + +func (plugin) FetchDefinedStages() []string { + return []string{ + stageDiff, + stageSync, + stageRollback, + } +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/12-determine-versions.md b/docs/content/en/docs-dev/plugin-development/12-determine-versions.md new file mode 100644 index 0000000000..0cfd9c720e --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/12-determine-versions.md @@ -0,0 +1,22 @@ +--- +title: "Implementation: DetermineVersions" +weight: 12 +description: > + Implementing DetermineVersions to manage version tracking. +--- + +`DetermineVersions` returns the version of the resource being deployed. + +Since our plugin synchronizes files from a Git repository, we will use the Git commit hash of the deployment source as our resource version. + +The deployment source details are passed via `input.Request.DeploymentSource`. We will fetch `CommitHash` from it. We do not need to specify the artifact name or URL for this local plugin. + +Update `DetermineVersions` as follows: + +```go +func (plugin) DetermineVersions(_ context.Context, _ *config, input *sdk.DetermineVersionsInput[applicationConfig]) (*sdk.DetermineVersionsResponse, error) { + return &sdk.DetermineVersionsResponse{ + Versions: []sdk.ArtifactVersion{{Version: input.Request.DeploymentSource.CommitHash}}, + }, nil +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/13-determine-strategy.md b/docs/content/en/docs-dev/plugin-development/13-determine-strategy.md new file mode 100644 index 0000000000..a008f8b293 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/13-determine-strategy.md @@ -0,0 +1,21 @@ +--- +title: "Implementation: DetermineStrategy" +weight: 13 +description: > + Implementing DetermineStrategy to configure sync execution strategies. +--- + +`DetermineStrategy` decides whether to run a `PipelineSync` or a `QuickSync` strategy. + +- **`PipelineSync`**: Executes the deployment using a pipeline explicitly defined by the user in `app.pipecd.yaml`. +- **`QuickSync`**: Bypasses any user-defined pipelines and executes a default set of sync stages defined by the plugin. For example, in Kubernetes deployments, a minor change (like updating replicas) might bypass canary release pipelines using a Quick Sync. + +If your plugin does not contain complex logic to dynamically choose between `QuickSync` and `PipelineSync`, you can simply return `nil, nil`. In this case, Piped will fall back to the strategy configured in the application configuration. + +Update `DetermineStrategy` as follows: + +```go +func (plugin) DetermineStrategy(context.Context, *config, *sdk.DetermineStrategyInput[applicationConfig]) (*sdk.DetermineStrategyResponse, error) { + return nil, nil +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/14-build-pipeline-sync-stages.md b/docs/content/en/docs-dev/plugin-development/14-build-pipeline-sync-stages.md new file mode 100644 index 0000000000..652ff5149a --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/14-build-pipeline-sync-stages.md @@ -0,0 +1,64 @@ +--- +title: "Implementation: BuildPipelineSyncStages" +weight: 14 +description: > + Implementing BuildPipelineSyncStages to construct user-defined pipelines. +--- + +`BuildPipelineSyncStages` is triggered when a `PipelineSync` strategy is selected. This method takes the pipeline defined by the user in `app.pipecd.yaml` and constructs the actual execution graph. + +### Key Considerations + +1. **`Index` Matching**: The returned stages must maintain the same `Index` values provided in `input.Request.Stages`. Returning an invalid or mismatched `Index` will result in a deployment error. +2. **Rollback Stage Integration**: If `input.Request.Rollback` is `true`, we must append a rollback stage (`FILE_ROLLBACK`). To ensure rollback starts immediately in case of failure, set its `Index` to the minimum index among all defined stages. + +Since our plugin does not require custom stage metadata, we can map stage names to our constants using a straightforward `switch` block. + +Update `BuildPipelineSyncStages` as follows: + +```go +import "fmt" + +func (plugin) BuildPipelineSyncStages(_ context.Context, _ *config, input *sdk.BuildPipelineSyncStagesInput) (*sdk.BuildPipelineSyncStagesResponse, error) { + if len(input.Request.Stages) == 0 { + return nil, fmt.Errorf("no stages defined in the request") + } + + stages := make([]sdk.PipelineStage, 0, len(input.Request.Stages)+1) // +1 for the rollback stage + for _, s := range input.Request.Stages { + switch s.Name { + case stageDiff: + stages = append(stages, sdk.PipelineStage{ + Index: s.Index, + Name: stageDiff, + }) + case stageSync: + stages = append(stages, sdk.PipelineStage{ + Index: s.Index, + Name: stageSync, + }) + default: + return nil, fmt.Errorf("unknown stage: %s", s.Name) + } + } + + if input.Request.Rollback { + // Find the minimum index to assign to the rollback stage + idx := input.Request.Stages[0].Index + for _, s := range input.Request.Stages[1:] { + if s.Index < idx { + idx = s.Index + } + } + stages = append(stages, sdk.PipelineStage{ + Index: idx, + Name: stageRollback, + Rollback: true, + }) + } + + return &sdk.BuildPipelineSyncStagesResponse{ + Stages: stages, + }, nil +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/15-build-quick-sync-stages.md b/docs/content/en/docs-dev/plugin-development/15-build-quick-sync-stages.md new file mode 100644 index 0000000000..c54bcf89b5 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/15-build-quick-sync-stages.md @@ -0,0 +1,40 @@ +--- +title: "Implementation: BuildQuickSyncStages" +weight: 15 +description: > + Implementing BuildQuickSyncStages to configure the fallback sync pipeline. +--- + +`BuildQuickSyncStages` is called when `QuickSync` is selected. It returns a default set of execution stages defined entirely by the plugin, bypassing the user's pipeline. + +### Differences from BuildPipelineSyncStages + +1. **No `Index` Field**: In a Quick Sync, stages are executed concurrently or without a rigid order, so `Index` is not required. +2. **`Description` Requirement**: Since there is no user-defined pipeline configuration, the plugin must supply a human-readable `Description` for each stage. + +For our `file` plugin's Quick Sync, we will execute a single `FILE_SYNC` stage, along with a `FILE_ROLLBACK` stage if rollback is requested. + +Update `BuildQuickSyncStages` as follows: + +```go +func (plugin) BuildQuickSyncStages(_ context.Context, _ *config, input *sdk.BuildQuickSyncStagesInput) (*sdk.BuildQuickSyncStagesResponse, error) { + stages := make([]sdk.QuickSyncStage, 0, 2) + + stages = append(stages, sdk.QuickSyncStage{ + Name: stageSync, + Description: "Synchronize local files from the Git repository", + }) + + if input.Request.Rollback { + stages = append(stages, sdk.QuickSyncStage{ + Name: stageRollback, + Description: "Rollback local files to the previous successful version", + Rollback: true, + }) + } + + return &sdk.BuildQuickSyncStagesResponse{ + Stages: stages, + }, nil +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/16-execute-stage.md b/docs/content/en/docs-dev/plugin-development/16-execute-stage.md new file mode 100644 index 0000000000..cd663447b9 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/16-execute-stage.md @@ -0,0 +1,44 @@ +--- +title: "Implementation: ExecuteStage" +weight: 16 +description: > + Implementing the main entrypoint for stage execution routing. +--- + +`ExecuteStage` is the core method where actual synchronization, diffing, or rolling back takes place. + +We will route execution to dedicated helper methods based on the `input.Request.StageName`. + +### Crucial Implementation Notes + +1. **Web UI Logs**: To persist logs that are viewable by users in the PipeCD Web Console, do **not** use `input.Logger`. Instead, fetch the specialized log persister using `input.Client.LogPersister()`. +2. **Stage Failure Handling**: When a stage fails due to a soft error (such as a deployment conflict), do **not** return a Go `error`. Instead, return a successful response with `Status` set to `sdk.StageStatusFailure`. Returning a Go `error` indicates a hard plugin crash. + +Update `ExecuteStage` and scaffold the helper methods as follows: + +```go +func (p plugin) ExecuteStage(ctx context.Context, _ *config, _ []*sdk.DeployTarget[deployTargetConfig], input *sdk.ExecuteStageInput[applicationConfig]) (*sdk.ExecuteStageResponse, error) { + switch input.Request.StageName { + case stageDiff: + return p.executeStageDiff(ctx, input) + case stageSync: + return p.executeStageSync(ctx, input) + case stageRollback: + return p.executeStageRollback(ctx, input) + default: + return nil, fmt.Errorf("unknown stage: %s", input.Request.StageName) + } +} + +func (plugin) executeStageDiff(ctx context.Context, input *sdk.ExecuteStageInput[applicationConfig]) (*sdk.ExecuteStageResponse, error) { + panic("unimplemented") +} + +func (plugin) executeStageSync(ctx context.Context, input *sdk.ExecuteStageInput[applicationConfig]) (*sdk.ExecuteStageResponse, error) { + panic("unimplemented") +} + +func (plugin) executeStageRollback(ctx context.Context, input *sdk.ExecuteStageInput[applicationConfig]) (*sdk.ExecuteStageResponse, error) { + panic("unimplemented") +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/17-execute-stage-diff.md b/docs/content/en/docs-dev/plugin-development/17-execute-stage-diff.md new file mode 100644 index 0000000000..e26bcf9595 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/17-execute-stage-diff.md @@ -0,0 +1,353 @@ +--- +title: "ExecuteStage Implementation: DIFF Stage" +weight: 17 +description: > + Implementing the DIFF stage to log and preview local file changes. +--- + +In the `FILE_DIFF` stage, we walk both the target directory on Piped's host and the Git repository's application directory to detect added, modified, or deleted files, printing the results to the log. + +We will divide the implementation into three helper functions: + +1. **`listFiles`**: Lists all relative file paths under a directory. +2. **`differenceFiles`**: Finds paths present in one file set but missing in another. +3. **`isFileContentDifferent`**: Checks if the content of a file has changed. + +--- + +### 1. Listing Files (`listFiles`) + +Let's start by writing a test. Create or append to a file named `main_test.go`: + +```go +package main + +import ( + "os" + "testing" +) + +func TestListFiles(t *testing.T) { + path := "./testdata/list_files" + expectedFiles := []string{"file1.txt", "file2.txt", "subdir/file3.txt"} + + files, err := listFiles(os.DirFS(path)) + if err != nil { + t.Fatalf("failed to list files: %v", err) + } + + if len(files) != len(expectedFiles) { + t.Fatalf("expected %d files, got %d", len(expectedFiles), len(files)) + } + + for _, expectedFile := range expectedFiles { + if _, found := files[expectedFile]; !found { + t.Errorf("expected file %s not found in the list", expectedFile) + } + } +} +``` + +To make the test pass, create the test files and directories: + +```console +$ mkdir -p testdata/list_files/subdir +$ touch testdata/list_files/file1.txt testdata/list_files/file2.txt testdata/list_files/subdir/file3.txt +``` + +Now, implement `listFiles` using Go's standard `io/fs` package: + +```go +import ( + "fmt" + "io/fs" +) + +func listFiles(f fs.FS) (map[string]struct{}, error) { + files := make(map[string]struct{}) + + if err := fs.WalkDir(f, ".", func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if !d.IsDir() { + files[path] = struct{}{} + } + return nil + }); err != nil { + return nil, fmt.Errorf("error walking through files: %w", err) + } + + return files, nil +} +``` + +Run `go test` to confirm it passes. + +--- + +### 2. Finding File Key Differences (`differenceFiles`) + +Next, let's write a test to find files that exist in one map but not in another: + +```go +import "reflect" + +func TestDifferenceFiles(t *testing.T) { + path1 := "./testdata/difference_files/path1" + path2 := "./testdata/difference_files/path2" + + expectedDifferences1 := map[string]struct{}{ + "file1.txt": {}, + "file2.txt": {}, + } + expectedDifferences2 := map[string]struct{}{ + "file3.txt": {}, + "file4.txt": {}, + } + + files1, err := listFiles(os.DirFS(path1)) + if err != nil { + t.Fatalf("failed to list files: %v", err) + } + + files2, err := listFiles(os.DirFS(path2)) + if err != nil { + t.Fatalf("failed to list files: %v", err) + } + + differences1 := differenceFiles(files1, files2) + if !reflect.DeepEqual(differences1, expectedDifferences1) { + t.Fatalf("expected %v differences, got %v", expectedDifferences1, differences1) + } + + differences2 := differenceFiles(files2, files1) + if !reflect.DeepEqual(differences2, expectedDifferences2) { + t.Fatalf("expected %v differences, got %v", expectedDifferences2, differences2) + } +} +``` + +Create the necessary test data: + +```console +$ mkdir -p testdata/difference_files/path1 testdata/difference_files/path2 +$ touch testdata/difference_files/path1/file0.txt testdata/difference_files/path1/file1.txt testdata/difference_files/path1/file2.txt testdata/difference_files/path1/file5.txt +$ touch testdata/difference_files/path2/file0.txt testdata/difference_files/path2/file3.txt testdata/difference_files/path2/file4.txt testdata/difference_files/path2/file5.txt +``` + +Implement `differenceFiles` in `main.go`: + +```go +// differenceFiles compares map a and map b, returning keys that exist in a but not in b. +func differenceFiles(a, b map[string]struct{}) map[string]struct{} { + differences := make(map[string]struct{}) + + for path := range a { + if _, ok := b[path]; !ok { + differences[path] = struct{}{} + } + } + + return differences +} +``` + +--- + +### 3. Comparing File Content (`isFileContentDifferent`) + +For educational simplicity, we will check if the content has changed using direct byte comparison rather than implementing a full line-by-line diff. + +Write the content comparison test: + +```go +func TestIsFileContentDifferent(t *testing.T) { + fs1 := os.DirFS("./testdata/difference_file_content/path1") + fs2 := os.DirFS("./testdata/difference_file_content/path2") + + testCases := []struct { + name string + fsA fs.FS + fsB fs.FS + path string + wantDifferent bool + wantErr bool + }{ + { + name: "same content", + fsA: fs1, + fsB: fs2, + path: "file1.txt", + wantDifferent: false, + wantErr: false, + }, + { + name: "different content", + fsA: fs1, + fsB: fs2, + path: "file2.txt", + wantDifferent: true, + wantErr: false, + }, + { + name: "file not found", + fsA: fs1, + fsB: fs2, + path: "file3.txt", + wantDifferent: false, + wantErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotDifferent, err := isFileContentDifferent(tc.fsA, tc.fsB, tc.path) + if (err != nil) != tc.wantErr { + t.Fatalf("isFileContentDifferent() error = %v, wantErr %v", err, tc.wantErr) + } + if gotDifferent != tc.wantDifferent { + t.Errorf("isFileContentDifferent() = %v, want %v", gotDifferent, tc.wantDifferent) + } + }) + } +} +``` + +Create the content test data: + +```console +$ mkdir -p testdata/difference_file_content/path1 testdata/difference_file_content/path2 +$ echo a > testdata/difference_file_content/path1/file1.txt +$ echo a > testdata/difference_file_content/path1/file2.txt +$ echo a > testdata/difference_file_content/path2/file1.txt +$ echo b > testdata/difference_file_content/path2/file2.txt +``` + +Implement `isFileContentDifferent` in `main.go`: + +```go +import ( + "bytes" + "io" +) + +func isFileContentDifferent(a, b fs.FS, path string) (bool, error) { + aFile, err := a.Open(path) + if err != nil { + return false, fmt.Errorf("error opening file %s: %w", path, err) + } + defer aFile.Close() + + bFile, err := b.Open(path) + if err != nil { + return false, fmt.Errorf("error opening file %s: %w", path, err) + } + defer bFile.Close() + + aContent, err := io.ReadAll(aFile) + if err != nil { + return false, fmt.Errorf("error reading file %s: %w", path, err) + } + + bContent, err := io.ReadAll(bFile) + if err != nil { + return false, fmt.Errorf("error reading file %s: %w", path, err) + } + + return !bytes.Equal(aContent, bContent), nil +} +``` + +Verify that all tests pass using `go test`. + +--- + +### 4. Implementing `executeStageDiff` + +Now, let's combine these three helpers into `executeStageDiff`. + +> [!IMPORTANT] +> Make sure to **exclude** PipeCD's application configuration file (`app.pipecd.yaml`) from the file comparison. Piped passes this filename under `input.Request.TargetDeploymentSource.ApplicationConfigFilename`, so we can simply use the Go builtin `delete` to remove it from our file lists. + +```go +import ( + "maps" + "slices" +) + +func (plugin) executeStageDiff(ctx context.Context, input *sdk.ExecuteStageInput[applicationConfig]) (*sdk.ExecuteStageResponse, error) { + lp := input.Client.LogPersister() + + lp.Info("Listing files in the Git repository...") + sourceFiles, err := listFiles(os.DirFS(input.Request.TargetDeploymentSource.ApplicationDirectory)) + if err != nil { + return nil, fmt.Errorf("error listing files: %w", err) + } + + // Exclude the application config file + delete(sourceFiles, input.Request.TargetDeploymentSource.ApplicationConfigFilename) + + lp.Info("Listing files in the target deployment directory...") + targetFiles, err := listFiles(os.DirFS(input.Request.TargetDeploymentSource.ApplicationConfig.Spec.Path)) + if err != nil { + return nil, fmt.Errorf("error listing files: %w", err) + } + + addedFiles := differenceFiles(sourceFiles, targetFiles) + removedFiles := differenceFiles(targetFiles, sourceFiles) + + mergedFiles := maps.Clone(sourceFiles) + maps.Copy(mergedFiles, targetFiles) + + diffFiles := make(map[string]struct{}) + for path := range mergedFiles { + if _, ok := addedFiles[path]; ok { + continue + } + if _, ok := removedFiles[path]; ok { + continue + } + + different, err := isFileContentDifferent( + os.DirFS(input.Request.TargetDeploymentSource.ApplicationDirectory), + os.DirFS(input.Request.TargetDeploymentSource.ApplicationConfig.Spec.Path), + path, + ) + if err != nil { + return nil, fmt.Errorf("error checking file diff for %s: %w", path, err) + } + + if different { + diffFiles[path] = struct{}{} + } + } + + // Output results to the LogPersister + lp.Info("Summary of the file diff:") + lp.Info("--------------------------------") + lp.Info("Added files:") + for _, path := range slices.Sorted(maps.Keys(addedFiles)) { + lp.Info(path) + } + + lp.Info("--------------------------------") + lp.Info("Removed files:") + for _, path := range slices.Sorted(maps.Keys(removedFiles)) { + lp.Info(path) + } + + lp.Info("--------------------------------") + lp.Info("Changed files:") + for _, path := range slices.Sorted(maps.Keys(diffFiles)) { + lp.Info(path) + } + lp.Info("--------------------------------") + + lp.Success("File diff completed successfully") + + return &sdk.ExecuteStageResponse{ + Status: sdk.StageStatusSuccess, + }, nil +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/18-execute-stage-sync.md b/docs/content/en/docs-dev/plugin-development/18-execute-stage-sync.md new file mode 100644 index 0000000000..d0c430eef7 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/18-execute-stage-sync.md @@ -0,0 +1,233 @@ +--- +title: "ExecuteStage Implementation: SYNC Stage" +weight: 18 +description: > + Implementing the SYNC stage to copy and synchronize local files. +--- + +In the `FILE_SYNC` stage, we execute the actual deployment operations: + +1. Copying all files from the Git repository's application directory to the target deployment path. +2. Deleting any orphaned files in the target path that are no longer tracked in Git. + +--- + +### 1. Copying Files (`copyFiles`) + +First, write a test in `main_test.go`. Since this involves disk manipulation, we will isolate it using `t.TempDir()`: + +```go +import "path/filepath" + +func TestCopyFiles(t *testing.T) { + srcDir := "testdata/list_files" + dstDir := t.TempDir() + + // Exclude file2.txt for testing + if err := copyFiles(dstDir, os.DirFS(srcDir), map[string]struct{}{"file2.txt": {}}); err != nil { + t.Fatalf("copyFiles() error = %v", err) + } + + srcFiles, err := listFiles(os.DirFS(srcDir)) + if err != nil { + t.Fatalf("listFiles() on source dir failed: %v", err) + } + + dstFiles, err := listFiles(os.DirFS(dstDir)) + if err != nil { + t.Fatalf("listFiles() on dest dir failed: %v", err) + } + + delete(srcFiles, "file2.txt") // file2.txt was excluded + + if !reflect.DeepEqual(srcFiles, dstFiles) { + t.Errorf("copied files list differs. got %v, want %v", dstFiles, srcFiles) + } + + for path := range srcFiles { + srcContent, err := os.ReadFile(filepath.Join(srcDir, path)) + if err != nil { + t.Fatalf("failed to read source file %s: %v", path, err) + } + + dstContent, err := os.ReadFile(filepath.Join(dstDir, path)) + if err != nil { + t.Fatalf("failed to read destination file %s: %v", path, err) + } + + if !bytes.Equal(srcContent, dstContent) { + t.Errorf("content of %s is different", path) + } + } +} +``` + +Now, implement `copyFiles` in `main.go`. We will create parent directories dynamically as files are written: + +```go +func copyFiles(dstDir string, files fs.FS, exclude map[string]struct{}) error { + walkDirFunc := func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + return nil + } + + if _, ok := exclude[path]; ok { + return nil + } + + dstPath := filepath.Join(dstDir, path) + + if err := os.MkdirAll(filepath.Dir(dstPath), 0755); err != nil { + return err + } + + srcFile, err := files.Open(path) + if err != nil { + return err + } + defer srcFile.Close() + + dstFile, err := os.Create(dstPath) + if err != nil { + return err + } + defer dstFile.Close() + + if _, err := io.Copy(dstFile, srcFile); err != nil { + return err + } + + return nil + } + + if err := fs.WalkDir(files, ".", walkDirFunc); err != nil { + return fmt.Errorf("walking through files: %w", err) + } + + return nil +} +``` + +--- + +### 2. Deleting Orphaned Files (`removeFiles`) + +Write the deletion test in `main_test.go`: + +```go +func TestRemoveFiles(t *testing.T) { + srcDir := "testdata/remove_files/src" + dstDir := t.TempDir() + + if err := copyFiles(dstDir, os.DirFS("testdata/remove_files/dst_before"), nil); err != nil { + t.Fatalf("failed to copy dst_before: %v", err) + } + + if err := removeFiles(dstDir, os.DirFS(srcDir), map[string]struct{}{"excluded_file.txt": {}}); err != nil { + t.Fatalf("removeFiles() error = %v", err) + } + + srcFS := os.DirFS(srcDir) + expectedFiles, err := listFiles(srcFS) + if err != nil { + t.Fatalf("failed to list files in src dir: %v", err) + } + + delete(expectedFiles, "excluded_file.txt") + + dstFiles, err := listFiles(os.DirFS(dstDir)) + if err != nil { + t.Fatalf("failed to list files in dst dir: %v", err) + } + + if !reflect.DeepEqual(dstFiles, expectedFiles) { + t.Errorf("file list differs. got %v, want %v", dstFiles, expectedFiles) + } + + if _, err := os.Stat(filepath.Join(dstDir, "file_to_remove.txt")); !os.IsNotExist(err) { + t.Errorf("file_to_remove.txt was not removed") + } +} +``` + +Create the required test data: + +```console +$ mkdir -p testdata/remove_files/src/subdir testdata/remove_files/dst_before/subdir +$ touch testdata/remove_files/src/file1.txt testdata/remove_files/src/subdir/file2.txt testdata/remove_files/src/excluded_file.txt +$ touch testdata/remove_files/dst_before/file1.txt testdata/remove_files/dst_before/subdir/file2.txt testdata/remove_files/dst_before/file_to_remove.txt testdata/remove_files/dst_before/excluded_file.txt +``` + +Implement `removeFiles` in `main.go`: + +```go +func removeFiles(dstDir string, files fs.FS, exclude map[string]struct{}) error { + sourceFiles, err := listFiles(files) + if err != nil { + return fmt.Errorf("listing files: %w", err) + } + + for path := range exclude { + delete(sourceFiles, path) + } + + dstFiles, err := listFiles(os.DirFS(dstDir)) + if err != nil { + return fmt.Errorf("listing files: %w", err) + } + + removedFiles := differenceFiles(dstFiles, sourceFiles) + + for path := range removedFiles { + if err := os.Remove(filepath.Join(dstDir, path)); err != nil { + return fmt.Errorf("removing file %s: %w", path, err) + } + } + + return nil +} +``` + +Run `go test` to confirm everything works properly. + +--- + +### 3. Implementing `executeStageSync` + +Now, let's orchestrate these in `executeStageSync`: + +```go +func (plugin) executeStageSync(ctx context.Context, input *sdk.ExecuteStageInput[applicationConfig]) (*sdk.ExecuteStageResponse, error) { + lp := input.Client.LogPersister() + + lp.Info("Copying files to the target directory...") + if err := copyFiles( + input.Request.TargetDeploymentSource.ApplicationConfig.Spec.Path, + os.DirFS(input.Request.TargetDeploymentSource.ApplicationDirectory), + map[string]struct{}{ + input.Request.TargetDeploymentSource.ApplicationConfigFilename: {}, + }, + ); err != nil { + return nil, fmt.Errorf("error copying files: %w", err) + } + + lp.Info("Removing orphaned files from target directory...") + if err := removeFiles( + input.Request.TargetDeploymentSource.ApplicationConfig.Spec.Path, + os.DirFS(input.Request.TargetDeploymentSource.ApplicationDirectory), + map[string]struct{}{ + input.Request.TargetDeploymentSource.ApplicationConfigFilename: {}, + }, + ); err != nil { + return nil, fmt.Errorf("error removing files: %w", err) + } + + lp.Success("File synchronization completed successfully") + return &sdk.ExecuteStageResponse{ + Status: sdk.StageStatusSuccess, + }, nil +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/19-execute-stage-rollback.md b/docs/content/en/docs-dev/plugin-development/19-execute-stage-rollback.md new file mode 100644 index 0000000000..9a8e5774e3 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/19-execute-stage-rollback.md @@ -0,0 +1,45 @@ +--- +title: "ExecuteStage Implementation: ROLLBACK Stage" +weight: 19 +description: > + Implementing the ROLLBACK stage to restore local files in case of failure. +--- + +The final step in our sync engine is the `FILE_ROLLBACK` stage. + +Fortunately, rollback is structurally identical to the sync operation. The only difference is that instead of syncing the target directory with the _newly requested_ deployment source (`input.Request.TargetDeploymentSource`), we sync it with the _previously successful_ deployment source (`input.Request.RunningDeploymentSource`). + +Update `executeStageRollback` as follows: + +```go +func (plugin) executeStageRollback(ctx context.Context, input *sdk.ExecuteStageInput[applicationConfig]) (*sdk.ExecuteStageResponse, error) { + lp := input.Client.LogPersister() + + lp.Info("Restoring files to target directory (rolling back)...") + if err := copyFiles( + input.Request.RunningDeploymentSource.ApplicationConfig.Spec.Path, + os.DirFS(input.Request.RunningDeploymentSource.ApplicationDirectory), + map[string]struct{}{ + input.Request.RunningDeploymentSource.ApplicationConfigFilename: {}, + }, + ); err != nil { + return nil, fmt.Errorf("error copying files during rollback: %w", err) + } + + lp.Info("Removing newer files from target directory...") + if err := removeFiles( + input.Request.RunningDeploymentSource.ApplicationConfig.Spec.Path, + os.DirFS(input.Request.RunningDeploymentSource.ApplicationDirectory), + map[string]struct{}{ + input.Request.RunningDeploymentSource.ApplicationConfigFilename: {}, + }, + ); err != nil { + return nil, fmt.Errorf("error removing files during rollback: %w", err) + } + + lp.Success("File rollback completed successfully") + return &sdk.ExecuteStageResponse{ + Status: sdk.StageStatusSuccess, + }, nil +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/20-updating-main.md b/docs/content/en/docs-dev/plugin-development/20-updating-main.md new file mode 100644 index 0000000000..ba2938bcf4 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/20-updating-main.md @@ -0,0 +1,33 @@ +--- +title: "Modifying the Main Function" +weight: 20 +description: > + Updating the main function to register the completed DeploymentPlugin. +--- + +Now that our plugin struct satisfies all interface methods, we can complete the `main` function in `main.go`. + +We will supply our `plugin{}` instance to `sdk.NewPlugin` using the registration option `sdk.WithDeploymentPlugin(plugin{})`. + +Update `main.go` as follows: + +```go +package main + +import ( + "log" + + sdk "github.com/pipe-cd/piped-plugin-sdk-go/pkg/plugin/sdk" +) + +func main() { + plugin, err := sdk.NewPlugin("0.0.1", sdk.WithDeploymentPlugin(plugin{})) + if err != nil { + log.Fatalln(err) + } + + if err := plugin.Run(); err != nil { + log.Fatalln(err) + } +} +``` diff --git a/docs/content/en/docs-dev/plugin-development/21-trying-with-piped.md b/docs/content/en/docs-dev/plugin-development/21-trying-with-piped.md new file mode 100644 index 0000000000..52175393ba --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/21-trying-with-piped.md @@ -0,0 +1,171 @@ +--- +title: "Running Locally with Piped" +weight: 21 +description: > + Step-by-step walkthrough to run, test, and deploy using the custom plugin. +--- + +Now that the custom plugin is fully built, let's run it locally and test it against a real PipeCD deployment agent! + +--- + +### 1. Setting Up the Control Plane + +To deploy applications, we need a running PipeCD Control Plane and a Piped agent. While production instances are typically deployed on Kubernetes, we will use Docker Compose for our local environment. + +Clone the official `pipe-cd/tutorial` repository: + +```console +$ git clone https://github.com/pipe-cd/tutorial.git +$ cd tutorial/src/install/control-plane +``` + +Open `docker-compose.yaml` and update the `ghcr.io/pipe-cd/pipecd` image tag to `v0.52.0-54-g8a12400` (the version compiled with Pluggable Architecture support). + +Start the control plane: + +```console +$ docker compose up -d +``` + +Open [http://localhost:8080/login](http://localhost:8080/login) in your browser. Log in with the default credentials: + +- **Project Name**: `tutorial` +- **Username**: `hello-pipecd` +- **Password**: `hello-pipecd` + +--- + +### 2. Registering Piped + +1. Open [http://localhost:8080/settings/piped?project=tutorial](http://localhost:8080/settings/piped?project=tutorial). +2. Click **+ ADD** in the top left corner. +3. Fill in a descriptive Name and Description, and click **SAVE**. +4. Securely copy the generated **Piped Id** and **Base64 Encoded Piped Key**. + +--- + +### 3. Setting Up the Git Manifest Repository + +Piped continuously syncs state with a Git repository. We will set up a local bare Git repository so we do not have to push to GitHub during local testing. + +In a new terminal window: + +```console +$ git init --bare pipecd-manifest.git +$ git clone ./pipecd-manifest.git +$ cd pipecd-manifest +$ mkdir demo-file-app +``` + +Under `demo-file-app`, create an `app.pipecd.yaml` configuration file specifying our `file` plugin: + +```yaml +apiVersion: pipecd.dev/v1beta1 +kind: Application +spec: + name: demo-file-app + description: | + Demo application synchronized using our custom file plugin. + pipeline: + stages: + - name: FILE_DIFF + - name: FILE_SYNC + plugins: + file: + path: /tmp/try-pipecd-file-plugin +``` + +Commit and push these changes: + +```console +$ git add demo-file-app/app.pipecd.yaml +$ git commit -m "Initialize demo file application" +$ git branch -M main +$ git push origin main +``` + +--- + +### 4. Compiling the Plugin Binary + +Compile the plugin in your plugin development directory: + +```console +$ go build -o pipecd-plugin-file +``` + +--- + +### 5. Configuring Piped + +Create Piped's local configuration file (`piped.config.yaml`). Ensure the directory paths are absolute: + +```yaml +apiVersion: pipecd.dev/v1beta1 +kind: Piped +spec: + projectID: tutorial + pipedID: <> + pipedKeyData: <> + apiAddress: localhost:8080 + repositories: + - repoId: local-manifest + remote: file://<> + branch: main + plugins: + - name: file + port: 7001 + url: file://<> + deployTargets: + - name: local + config: {} +``` + +--- + +### 6. Starting Piped + +Download the Pluggable Architecture Piped binary released under [kubecon-jp-2025](https://github.com/pipe-cd/pipecd/releases/tag/kubecon-jp-2025) matching your OS and CPU architecture. + +Make it executable and launch Piped: + +```console +$ chmod +x piped_kubecon_jp_2025_${os}_${arch} +$ ./piped_kubecon_jp_2025_${os}_${arch} piped --config-file=./piped.config.yaml --insecure +``` + +Check the logs to verify that Piped successfully connects to the Control Plane and registers the `file` plugin via localhost port `7001`. + +--- + +### 7. Registering the Application + +1. Open [http://localhost:8080/applications?project=tutorial](http://localhost:8080/applications?project=tutorial). +2. Click **+ ADD** and navigate to the **ADD FROM SUGGESTIONS** tab. +3. Select your Piped and the suggested application, then click **SAVE**. + +--- + +### 8. Testing Synchronization + +Since no sync files exist in our Git repository yet, the first deployment will create an empty `/tmp/try-pipecd-file-plugin` folder. + +Add a file to the Git repository to test syncing: + +```console +$ cd pipecd-manifest/demo-file-app +$ echo "Hello from PipeCD Custom Plugin!" > hello.txt +$ git add hello.txt +$ git commit -m "Add hello.txt" +$ git push origin main +``` + +Piped will automatically detect the commit, trigger a deployment, and run the `FILE_DIFF` and `FILE_SYNC` stages. + +Verify that the file is synced successfully: + +```console +$ cat /tmp/try-pipecd-file-plugin/hello.txt +Hello from PipeCD Custom Plugin! +``` diff --git a/docs/content/en/docs-dev/plugin-development/22-conclusion.md b/docs/content/en/docs-dev/plugin-development/22-conclusion.md new file mode 100644 index 0000000000..9fac769c86 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/22-conclusion.md @@ -0,0 +1,17 @@ +--- +title: "Conclusion" +weight: 22 +description: > + Concluding remarks and next steps in custom plugin development. +--- + +Congratulations! You have successfully implemented, tested, and run a custom PipeCD plugin from scratch using Go and the official SDK! + +While our `file` plugin is basic, it walks through all key concepts of Pluggable Continuous Delivery in PipeCD. You can build on this foundation to implement robust plugins for other infrastructure tools, cloud platforms, or proprietary systems. + +### Further Resources + +- The complete source code built in this book can be found in the [Warashi/pipecd-plugin-file](https://github.com/Warashi/pipecd-plugin-file/) repository, with logical commits mirroring the chapters in this guide. +- For questions or troubleshooting, join the community channels on CNCF Slack: `#pipecd` and `#pipecd-jp`. + +Thank you for reading, and happy deploying! diff --git a/docs/content/en/docs-dev/plugin-development/_index.md b/docs/content/en/docs-dev/plugin-development/_index.md new file mode 100644 index 0000000000..c7f4cbeaf2 --- /dev/null +++ b/docs/content/en/docs-dev/plugin-development/_index.md @@ -0,0 +1,36 @@ +--- +title: "Plugin Development Book" +linkTitle: "Plugin Development Book" +weight: 10 +description: > + A comprehensive, step-by-step guide to understanding, building, and implementing custom plugins for PipeCD. +--- + +Welcome to the **PipeCD Plugin Development Book**. This book is a translated and adapted version of the excellent Japanese resource _作って学ぶ PipeCD プラグイン_ (Try and Learn PipeCD Plugins) by Warashi. + +In this book, you will learn the internal mechanisms of PipeCD's Pluggable Architecture and walk through building a fully functional custom plugin from scratch using the Go SDK. + +### Table of Contents + +1. [Introduction](01-introduction/) +2. [Plugin Features to Implement](02-plugin-features/) +3. [Technology Selection](03-tech-selection/) +4. [Initializing the Project](04-project-init/) +5. [Adding Dependency Libraries](05-dependencies/) +6. [Understanding Plugin Types](06-plugin-types/) +7. [First Step of Plugin Implementation](07-first-steps/) +8. [Satisfying the DeploymentPlugin Interface](08-deployment-plugin-interface/) +9. [Implementation: Defining Configuration Types](09-defining-config-types/) +10. [Implementation: Empty Implementation to Satisfy the Interface](10-empty-implementation/) +11. [Implementation: FetchDefinedStages](11-fetch-defined-stages/) +12. [Implementation: DetermineVersions](12-determine-versions/) +13. [Implementation: DetermineStrategy](13-determine-strategy/) +14. [Implementation: BuildPipelineSyncStages](14-build-pipeline-sync-stages/) +15. [Implementation: BuildQuickSyncStages](15-build-quick-sync-stages/) +16. [Implementation: ExecuteStage](16-execute-stage/) +17. [ExecuteStage Implementation: DIFF Stage](17-execute-stage-diff/) +18. [ExecuteStage Implementation: SYNC Stage](18-execute-stage-sync/) +19. [ExecuteStage Implementation: ROLLBACK Stage](19-execute-stage-rollback/) +20. [Modifying the Main Function](20-updating-main/) +21. [Running Locally with Piped](21-trying-with-piped/) +22. [Conclusion](22-conclusion/)