Skip to content

Commit 8918c27

Browse files
committed
feat: Enhance AWS SSM integration and add unit tests for storage functionality
- Updated `get_ssm_parameter` to accept optional AWS profile and region parameters. - Introduced a new `tests/test_storage.py` file containing comprehensive unit tests for the storage backend, covering timestamp parsing, configuration resolution, and S3/SSM interactions. - Implemented the `storage.py` module to manage environment store configurations, including reading and writing to S3 and SSM, with dual-backend support for reading the latest values. - Added functions for handling AWS CLI commands and managing configurations, ensuring robust error handling and user prompts for configuration creation.
1 parent 9699c83 commit 8918c27

9 files changed

Lines changed: 1331 additions & 819 deletions

File tree

ARCHITECTURE.md

Lines changed: 24 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ toast-cli/
3838
├── prompt_plugin.py
3939
├── region_plugin.py
4040
├── ssm_plugin.py
41+
├── storage.py
4142
└── utils.py
4243
```
4344

@@ -102,10 +103,10 @@ Each plugin:
102103
| am | Show AWS caller identity |
103104
| cdw | Navigate to workspace directories |
104105
| ctx | Manage Kubernetes contexts (switch, add EKS clusters, delete) |
105-
| dot | Manage .env.local files with AWS SSM integration |
106+
| dot | Manage .env.local files with S3 env-store (SSM transition) |
106107
| env | Manage AWS profiles |
107108
| git | Manage Git repositories (clone, branch, pull, push, rm, mirror) |
108-
| prompt | Manage .prompt.md files with AWS SSM integration |
109+
| prompt | Manage .prompt.md files with S3 env-store (SSM transition) |
109110
| region | Set AWS region |
110111
| ssm | AWS SSM Parameter Store operations (get, put, delete, list) |
111112

@@ -127,21 +128,34 @@ Each plugin:
127128
- Interactive selection of contexts and clusters
128129

129130
#### DotPlugin (dot)
130-
- Manages .env.local files with AWS SSM Parameter Store
131-
- Default behavior: `sync` (compare local/remote, show diff, choose upload/download)
131+
- Manages .env.local files via the S3 env-store (`storage.py`)
132+
- Default behavior: `sync` (compare local/store, show diff, choose upload/download)
132133
- Commands: `sync` (default), `up` (upload), `down`/`dn` (download), `ls` (list)
133-
- Uploads/downloads environment variables as SecureString
134-
- SSM path: `/toast/local/{org}/{project}/env-local`
134+
- S3 key: `local/{org}/{project}/env-local` (SSE-KMS)
135135
- Validates workspace path structure (`workspace/github.com/{org}/{project}`)
136136

137137
#### PromptPlugin (prompt)
138-
- Manages .prompt.md files with AWS SSM Parameter Store
139-
- Default behavior: `sync` (compare local/remote, show diff, choose upload/download)
138+
- Manages .prompt.md files via the S3 env-store (`storage.py`)
139+
- Default behavior: `sync` (compare local/store, show diff, choose upload/download)
140140
- Commands: `sync` (default), `up` (upload), `down`/`dn` (download), `ls` (list)
141-
- Uploads/downloads prompt files as SecureString
142-
- SSM path: `/toast/local/{org}/{project}/prompt-md`
141+
- S3 key: `local/{org}/{project}/prompt-md` (SSE-KMS)
143142
- Validates workspace path structure (`workspace/github.com/{org}/{project}`)
144143

144+
#### Env-store backend (storage.py)
145+
- Shared storage layer for the dot/prompt plugins
146+
- Dual-backend during SSM → S3 transition: reads check both S3 and SSM and use
147+
the newest copy (ties prefer S3); writes always go to S3
148+
- SSM is a read-only fallback and is harvested into S3 on the next upload
149+
- All access uses a dedicated AWS profile (`TOAST_ENV_STORE_PROFILE`, default
150+
`{username}-admin`)
151+
- Profile defaults to the OS username + `-admin`; bucket defaults to
152+
`env-store-{account-id}` of that profile (account id via
153+
`aws sts get-caller-identity`)
154+
- Configurable via env vars (`TOAST_ENV_STORE_PROFILE`, `TOAST_ENV_STORE_BUCKET`,
155+
`TOAST_ENV_STORE_KMS_KEY`, `TOAST_ENV_STORE_REGION`) or the config file
156+
`~/.config/toast/config` (`KEY=VALUE`, created on first run by prompting the
157+
user); precedence: env var > config file > default
158+
145159
#### EnvPlugin (env)
146160
- Manages AWS profiles from `~/.aws/credentials`
147161
- Interactive selection of profiles

CLAUDE.md

Lines changed: 39 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -73,13 +73,14 @@ class MyPlugin(BasePlugin):
7373
### Key Components
7474
- **Helpers** (`toast/helpers.py`): Version handling, custom Click classes for UI
7575
- **Plugin Utils** (`toast/plugins/utils.py`): Common utilities for subprocess execution, fzf integration
76+
- **Env-store** (`toast/plugins/storage.py`): S3/SSM dual-backend storage for dot/prompt (newest-wins reads, S3 writes)
7677
- **Dynamic Loading**: No hardcoded plugin imports - all plugins discovered at runtime
7778

7879
### Plugin Categories
7980
- **AWS plugins**: `am_plugin.py` (identity), `env_plugin.py` (profiles), `region_plugin.py` (regions), `ssm_plugin.py` (SSM Parameter Store)
8081
- **Kubernetes**: `ctx_plugin.py` (context management)
8182
- **Git**: `git_plugin.py` (repository operations: clone, branch, pull, push, rm)
82-
- **Environment**: `dot_plugin.py` (SSM integration for .env.local files), `prompt_plugin.py` (SSM integration for .prompt.md files)
83+
- **Environment**: `dot_plugin.py` (.env.local files), `prompt_plugin.py` (.prompt.md files), backed by `storage.py` (S3 env-store with SSM transition)
8384
- **Navigation**: `cdw_plugin.py` (workspace directory navigation)
8485

8586
## Adding New Plugins
@@ -127,36 +128,61 @@ When working on git_plugin.py:
127128
- Handle subprocess calls properly - avoid mixing `capture_output=True` with explicit `stdout`/`stderr`
128129
- Supported commands: `clone` (cl), `rm`, `branch` (b), `pull` (p), `push` (ps) (with optional flags like `-b`, `-t`, `-r`, `-m`/`--mirror`)
129130

130-
### SSM Parameter Store Integration
131+
### Env-store Integration (S3 with SSM transition)
131132

132-
The `dot` and `prompt` plugins use AWS SSM Parameter Store for secure file storage:
133+
The `dot` and `prompt` plugins store secure files in an S3 env-store bucket.
134+
The backend logic lives in `toast/plugins/storage.py`; the plugins themselves
135+
are thin wrappers that call `run_file_sync(kind, filename, command)`.
133136

134137
**Storage structure**:
135138
```
136-
/toast/local/{org}/{project}/env-local # DotPlugin
137-
/toast/local/{org}/{project}/prompt-md # PromptPlugin
139+
s3://{bucket}/local/{org}/{project}/env-local # DotPlugin (kind=env-local)
140+
s3://{bucket}/local/{org}/{project}/prompt-md # PromptPlugin (kind=prompt-md)
141+
142+
/toast/local/{org}/{project}/{kind} # legacy SSM (read-only fallback)
138143
```
139144

145+
**Transition semantics** (`storage.store_read` / `store_write`):
146+
- Reads check both S3 and SSM and use whichever copy is newest (by
147+
`LastModified`); ties prefer S3
148+
- Writes always go to S3 (the bucket is the source of truth)
149+
- SSM copies are never written; they go stale and are harvested into S3 on the
150+
next upload (`up`, or choosing upload in `sync`)
151+
152+
**Configuration** (`storage.resolve_config`) — precedence: environment variable
153+
> config file > built-in default. The config file `~/.config/toast/config`
154+
(`KEY=VALUE`) is created on first run by prompting the user for values
155+
(`prompt_and_create_config`); skipped in non-interactive sessions.
156+
157+
| Env variable | Config key | Default |
158+
|--------------|------------|---------|
159+
| `TOAST_ENV_STORE_PROFILE` | `ENV_STORE_PROFILE` | `{username}-admin` (`default_profile`) |
160+
| `TOAST_ENV_STORE_BUCKET` | `ENV_STORE_BUCKET` | `env-store-{account-id}` of the profile (`default_bucket`, via STS) |
161+
| `TOAST_ENV_STORE_KMS_KEY` | `ENV_STORE_KMS_KEY` | bucket/account default KMS key |
162+
| `TOAST_ENV_STORE_REGION` | `ENV_STORE_REGION` | profile's region |
163+
140164
**Commands**:
141165
| Command | Description |
142166
|---------|-------------|
143167
| (none) | Default: same as `sync` |
144-
| `sync` | Compare local and SSM, then choose action (upload/download) |
145-
| `up` | Upload local file to SSM |
146-
| `down`/`dn` | Download from SSM to local |
147-
| `ls` | List all parameters in SSM |
168+
| `sync` | Compare local and env-store (newest), then choose action |
169+
| `up` | Upload local file to env-store (S3) |
170+
| `down`/`dn` | Download newest from env-store to local |
171+
| `ls` | List entries across S3 + SSM with current source |
148172

149173
**Sync command features**:
150-
- Compares local file content with SSM parameter value
151-
- Displays SHA256 hash (first 12 chars) for both versions
174+
- Compares local file content with the newest env-store copy
175+
- Displays SHA256 hash (first 12 chars) for local, S3, and SSM, plus the
176+
current (newest) source
152177
- Shows unified diff when contents differ
153178
- Interactive selection via fzf: Upload / Download / Cancel
154179
- Handles cases: identical, different, local_only, remote_only
155180

156181
**Common patterns**:
157182
- Validate workspace path: `workspace/github.com/{org}/{project}`
158-
- Store as SecureString type for encryption
159-
- Use temporary files for large content uploads
183+
- All AWS calls use the env-store profile via `storage._aws()`
184+
- Write S3 objects with `--sse aws:kms`
185+
- Use temporary files for content upload/download
160186
- Include confirmation prompts before overwriting files
161187

162188
## Project Code Guidelines

README.md

Lines changed: 48 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ Python-based CLI utility with plugin architecture for AWS, Kubernetes, and Git o
2424
* **Git**: Repository management (clone, branch, pull, push, rm, mirror), organization-specific GitHub hosts
2525
* **Workspace**: Directory navigation, environment file management (.env.local, .prompt.md)
2626
* **Interface**: FZF-powered interactive menus, formatted output with Rich
27-
* **Security**: AWS SSM SecureString storage for sensitive files
27+
* **Security**: S3 env-store with SSE-KMS for sensitive files (SSM fallback during transition)
2828

2929
## Architecture
3030

@@ -94,16 +94,16 @@ toast ctx # Switch contexts
9494
# Select [Del...] to delete contexts (individual or all)
9595

9696
# Environment Files (.env.local)
97-
toast dot # Compare local and SSM, choose action (default: sync)
98-
toast dot up # Upload .env.local to SSM
99-
toast dot down # Download .env.local from SSM (alias: dn)
100-
toast dot ls # List all .env.local files in SSM
97+
toast dot # Compare local and env-store, choose action (default: sync)
98+
toast dot up # Upload .env.local to env-store (S3)
99+
toast dot down # Download .env.local from env-store (alias: dn)
100+
toast dot ls # List all .env.local files in env-store (S3 + SSM)
101101

102102
# Prompt Files (.prompt.md)
103-
toast prompt # Compare local and SSM, choose action (default: sync)
104-
toast prompt up # Upload .prompt.md to SSM
105-
toast prompt down # Download .prompt.md from SSM (alias: dn)
106-
toast prompt ls # List all .prompt.md files in SSM
103+
toast prompt # Compare local and env-store, choose action (default: sync)
104+
toast prompt up # Upload .prompt.md to env-store (S3)
105+
toast prompt down # Download .prompt.md from env-store (alias: dn)
106+
toast prompt ls # List all .prompt.md files in env-store (S3 + SSM)
107107

108108
# SSM Parameter Store
109109
toast ssm # Interactive mode: browse and select parameters
@@ -183,16 +183,50 @@ Host myorg-github.com
183183
- Automatic host detection based on workspace location
184184
- Seamless switching between GitHub accounts
185185

186-
### AWS SSM Storage Paths
186+
### Env-store (S3) Storage Paths
187187

188-
Toast-cli stores files in AWS SSM Parameter Store with the following structure:
188+
The `dot` and `prompt` plugins store files in the S3 env-store bucket. During
189+
the transition from AWS SSM Parameter Store, reads check both backends and use
190+
whichever copy is newest; writes always go to S3 (the bucket is the source of
191+
truth), and SSM copies become stale and are harvested into S3 on the next
192+
upload.
189193

190194
```
191-
/toast/local/{org}/{project}/env-local # .env.local files
192-
/toast/local/{org}/{project}/prompt-md # .prompt.md files
195+
s3://env-store-{account-id}/local/{org}/{project}/env-local # .env.local files
196+
s3://env-store-{account-id}/local/{org}/{project}/prompt-md # .prompt.md files
197+
198+
/toast/local/{org}/{project}/env-local # legacy SSM (read-only fallback)
199+
/toast/local/{org}/{project}/prompt-md # legacy SSM (read-only fallback)
193200
```
194201

195-
Files are stored as SecureString type for encryption at rest.
202+
S3 objects are written with SSE-KMS encryption. All env-store access uses a
203+
dedicated AWS profile so it is decoupled from your current default profile.
204+
205+
**Configuration** — precedence: environment variable > config file > default.
206+
207+
Environment variables:
208+
209+
```bash
210+
TOAST_ENV_STORE_PROFILE # default: {username}-admin
211+
TOAST_ENV_STORE_BUCKET # default: env-store-{account-id of the profile}
212+
TOAST_ENV_STORE_KMS_KEY # default: bucket/account default KMS key
213+
TOAST_ENV_STORE_REGION # default: profile's region
214+
```
215+
216+
The profile defaults to your OS username + `-admin`, and the bucket defaults to
217+
`env-store-` + the AWS account id of that profile (looked up via
218+
`aws sts get-caller-identity`).
219+
220+
Config file `~/.config/toast/config` (`KEY=VALUE` format). On first run, if it
221+
is missing, toast prompts for the values and saves them (interactive sessions
222+
only):
223+
224+
```
225+
ENV_STORE_BUCKET=env-store-{account-id}
226+
ENV_STORE_PROFILE={username}-admin
227+
ENV_STORE_KMS_KEY=
228+
ENV_STORE_REGION=
229+
```
196230

197231
## Creating Plugins
198232

VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
v4.0.0.dev0
1+
v4.1.0.dev0

0 commit comments

Comments
 (0)