diff --git a/README.md b/README.md index 098873c..818a05f 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ npx skills add oracle/skills/graal ## Domains - `db/` is the active Oracle Database domain and includes database, ORDS, SQLcl, framework, container, and agent workflow skills. -- `oci/` is the root for future Oracle Cloud Infrastructure skills. +- `oci/` contains Oracle Cloud Infrastructure skills, starting with OCI Kubernetes Engine cluster design, troubleshooting, Generic VNIC Attachment, and Multus pod networking. - `fusion/` is the root for future Oracle Fusion skills. - `apex/` is the root for future Oracle APEX skills. - `graal/` contains GraalVM skills, starting with Native Image. @@ -70,7 +70,18 @@ npx skills add oracle/skills/graal │ ├── reachability-metadata.md │ └── troubleshooting.md └── oci/ - └── SKILL.md + ├── SKILL.md + └── oke/ + ├── cluster-design.md + ├── troubleshooting.md + ├── gva-node-pools.md + ├── multus-multihome.md + ├── skills/ + ├── scripts/ + ├── agents/ + ├── shared/ + ├── examples/ + └── tests/ ``` Each domain has its own `SKILL.md` and any supporting index files it needs. @@ -90,3 +101,8 @@ For stub domains, keep `SKILL.md` minimal and point users back to this `README.m - Skills that include version-specific behavior must include a section named `## Oracle Version Notes (19c vs 26ai)`. - Use Oracle Database 19c as the baseline compatibility target unless stated otherwise. - Explicitly call out features that require newer releases and provide 19c-compatible alternatives where practical. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://www.graalvm.org/latest/reference-manual/native-image/ diff --git a/oci/SKILL.md b/oci/SKILL.md index 2251c21..29f869f 100644 --- a/oci/SKILL.md +++ b/oci/SKILL.md @@ -1,4 +1,76 @@ --- name: oci -description: Placeholder for future OCI skills. ---- \ No newline at end of file +description: Oracle Cloud Infrastructure guidance for designing, operating, and troubleshooting OCI services, starting with OCI Kubernetes Engine (OKE). Use when the user asks about OKE cluster design, Terraform or Resource Manager planning, OKE incident troubleshooting, Generic VNIC Attachment, Multus, pod networking, node pools, add-ons, ingress, load balancers, OCIR image pulls, Workload Identity, or Kubernetes workloads on OCI. +--- + +# Oracle Cloud Infrastructure Skills + +Use this domain for practical Oracle Cloud Infrastructure guidance. The current content focuses on OCI Kubernetes Engine (OKE): cluster design, operational troubleshooting, Generic VNIC Attachment (GVA), and Multus multi-interface pod validation. + +## How to Use This Domain + +1. Start with the routing table below. +2. Read only the OKE file that matches the user's task. +3. Prefer official Oracle documentation and live read-only discovery commands before making design or remediation recommendations. +4. Ask before running commands that create, update, delete, restart, scale, drain, or otherwise mutate OCI or Kubernetes resources. + +## Directory Structure + +```text +oci/ +|-- SKILL.md +`-- oke/ + |-- cluster-design.md + |-- troubleshooting.md + |-- gva-node-pools.md + |-- multus-multihome.md + |-- skills/ + |-- scripts/ + |-- agents/ + |-- shared/ + |-- examples/ + `-- tests/ +``` + +## Category Routing + +| Topic | File | +|-------|------| +| Design or scaffold an OKE cluster, Terraform stack, or OCI Resource Manager stack | Start with `oci/oke/cluster-design.md`, then load `oci/oke/skills/oke-cluster-generator/SKILL.md` | +| Troubleshoot OKE workloads, pods, services, DNS, add-ons, ingress, load balancers, image pulls, storage, Workload Identity, or cluster access | Start with `oci/oke/troubleshooting.md`, then load `oci/oke/skills/oke-troubleshooter/SKILL.md` | +| Configure OKE managed node pools with Generic VNIC Attachment secondary VNIC profiles and Application Resources | Start with `oci/oke/gva-node-pools.md`, then load `oci/oke/skills/oke-gva-deployer/SKILL.md` | +| Deploy or validate Multus NetworkAttachmentDefinitions and multi-interface pods on OKE | Start with `oci/oke/multus-multihome.md`, then load `oci/oke/skills/oke-multihome-deployer/SKILL.md` | + +## Key Starting Points + +- `oci/oke/cluster-design.md` +- `oci/oke/troubleshooting.md` +- `oci/oke/gva-node-pools.md` +- `oci/oke/multus-multihome.md` + +## Operational Tools + +The OKE operational skills include deterministic helper tools under `oci/oke/scripts/` and skill-specific helper scripts under `oci/oke/skills/*/scripts/`. + +- Read-only discovery and evidence tools may be used to collect context. +- Generate-only tools may produce manifests, commands, Terraform snippets, or reports. +- Any tool or command that creates, updates, deletes, patches, restarts, scales, drains, debugs, assigns IPs, applies manifests, or otherwise mutates OCI or Kubernetes resources requires explicit user approval first. +- `oci/oke/scripts/gva-menu.sh` is allowed to create an OKE node pool for the GVA workflow only after the user approves execution and completes its final `CREATE` confirmation. +- `oci/oke/scripts/node-doctor-run.sh` requires approval before execution because it creates a temporary debug pod and may delete that pod during cleanup. + +## Common Multi-Step Flows + +| Task | Recommended Sequence | +|------|----------------------| +| Plan a production OKE cluster | `oke/cluster-design.md` | +| Diagnose an OKE service with no load balancer IP | `oke/troubleshooting.md` | +| Build a node pool with workload-specific secondary VNIC profiles | `oke/gva-node-pools.md` -> `oke/multus-multihome.md` if pods need multiple interfaces | +| Validate Multus pod networking on GVA-enabled nodes | `oke/multus-multihome.md` -> `oke/troubleshooting.md` if symptoms remain | +| Investigate OKE workload access to OCI APIs | `oke/troubleshooting.md` | + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +- https://github.com/oracle-terraform-modules/terraform-oci-oke diff --git a/oci/oke/agents/oke-evidence-collector.md b/oci/oke/agents/oke-evidence-collector.md new file mode 100644 index 0000000..9c30409 --- /dev/null +++ b/oci/oke/agents/oke-evidence-collector.md @@ -0,0 +1,80 @@ +--- +name: oke-evidence-collector +description: Collects Kubernetes and OCI evidence for the OKE troubleshooter. +default-tool: bash +--- + +You run shell commands to gather evidence for specified diagnostic domains. Always sanitize output: +- Redact tokens, passwords, and private keys. +- Trim logs to the most recent and relevant lines (target under 200 lines per command). + +## Input Contract + +You receive JSON with the following structure: +```json +{ + "symptom": "pods pending", + "domains": ["Pod Scheduling", "Node Health"], + "namespace": "ml-team", + "time_window": "1h", + "selectors": {"pod": "trainer-0", "deployment": "nginx", "label": "app=nginx"}, + "fallbacks": {"kubectl": true, "oci": false}, + "compartment_ocid": "ocid1.compartment..." +} +``` + +Fields: +- `domains`: list of diagnostic domains to investigate. +- `namespace`: namespace string or empty when cluster-wide. +- `time_window`: preferred lookback (e.g., `15m`, `1h`); use for log queries where possible. +- `selectors`: optional keys (`pod`, `service`, `deployment`, `pvc`, `node`, `label`) to scope commands. +- `fallbacks`: booleans indicating unavailable CLIs. Skip commands that require a missing CLI and mark the fallback. +- `compartment_ocid`: optional; when present, include in OCI commands. + +## Command Guidelines +- Batch commands by domain. Print each command before execution prefixed with `>>>`. +- Prefer `-o json` or `--query` flags for parsable output. +- Respect namespace: include `-n ` when provided. +- Do **not** prompt for confirmation; commands must run non-interactively. +- When a command fails, capture a concise error summary and continue. + +## Output Format +Return compact JSON on stdout: +```json +[ + { + "domain": "Pod Scheduling", + "findings": [ + "Pod trainer-0 Pending: 0/3 nodes available: Insufficient nvidia.com/gpu" + ], + "raw_snippets": [ + "Warning FailedScheduling ... 0/3 nodes available: 3 Insufficient nvidia.com/gpu" + ], + "anomalies": [ + "Node pool np-gpu has max size 3 and no headroom" + ], + "fallback_used": false + } +] +``` + +Keep each `raw_snippets` entry under 500 characters. If command output is huge, store only the most relevant fragment. Mark `fallback_used` as `true` when you had to skip or downgrade evidence due to missing tooling. + +## Error Handling +- On unexpected failures, exit with code `2` and print a JSON error to stderr: + ```json + {"error_code":"EVIDENCE_COLLECTOR_FAILURE","message":"...","remediation":"...","docs_url":""} + ``` +- For anticipated issues (missing CLI, permission denied), still exit `0` and include the problem in `anomalies`. + +## Completion Checklist +- Domains processed sequentially. +- Output JSON well-formed and parseable. +- Sensitive values redacted. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/ +- https://kubernetes.io/docs/reference/kubectl/ +- https://kubernetes.io/docs/tasks/debug/debug-application/ diff --git a/oci/oke/agents/oke-hypothesis-analyst.md b/oci/oke/agents/oke-hypothesis-analyst.md new file mode 100644 index 0000000..5771bd4 --- /dev/null +++ b/oci/oke/agents/oke-hypothesis-analyst.md @@ -0,0 +1,87 @@ +--- +name: oke-hypothesis-analyst +description: Scores troubleshooting hypotheses for OKE incidents using collected evidence. +--- + +You receive evidence from the `/oke-troubleshooter` skill and must produce a ranked list of hypotheses with remediation guidance. + +## Input Contract +JSON payload: +```json +{ + "symptom": "pods stuck Pending in ml namespace", + "domains": ["Pod Scheduling", "Node Health"], + "evidence": [ + { + "domain": "Pod Scheduling", + "findings": ["Pod trainer-0 Pending: 0/3 nodes available"], + "raw_snippets": ["Warning FailedScheduling ... Insufficient nvidia.com/gpu"], + "anomalies": ["Node pool np-gpu has max size 3"], + "fallback_used": false + } + ], + "fallbacks": {"kubectl": false, "oci": false} +} +``` + +## Analysis Requirements +- Synthesize cross-domain patterns; explicitly cite the most relevant `raw_snippets` entries using short quotes. +- Produce 1–3 hypotheses ordered by confidence (score 0–10). +- Each hypothesis must include: + - `title`: concise statement of the root cause. + - `score`: integer 0–10 (10 = conclusive, 5 = plausible, ≤3 = weak signal). + - `evidence`: bullet list referencing snippets (e.g., `"Warning FailedScheduling: Insufficient nvidia.com/gpu"`). + - `remediation`: actionable commands or steps (kubectl/oci as needed). + - `prevention`: long-term recommendation (autoscaling, alerts, policy adjustments). +- When evidence is insufficient, add a hypothesis with low confidence explaining what data is missing and suggest additional evidence requests. +- If fallbacks limited analysis (e.g., OCI CLI unavailable), call this out in the report header and downgrade scores accordingly. + +## Output Format +Return JSON adhering to: +```json +{ + "summary": "High confidence GPU quota exhaustion causing Pending pods.", + "hypotheses": [ + { + "title": "GPU node pool exhausted", + "score": 9, + "evidence": [ + "FailedScheduling: Insufficient nvidia.com/gpu on all nodes", + "Node pool np-gpu max size reached (3 nodes)" + ], + "remediation": [ + "oci ce node-pool update --node-pool-id --size 5", + "kubectl cordon if draining required before scale" + ], + "prevention": [ + "Enable autoscaler with GPU headroom", + "Set OCI budget alarms for GPU OCPUs" + ] + } + ], + "warnings": [ + "OCI CLI unavailable: network diagnostics skipped" + ] +} +``` + +Ensure `hypotheses` contains at least one entry even when all scores are low. + +## Tone & Style +- Be direct and operational. No fluff. +- Reference CLI commands precisely; include `--region` or `--compartment-id` flags when needed. +- Avoid repeating identical evidence across multiple hypotheses; if the same data supports multiple causes, explain the distinction. + +## Error Handling +- If the payload is malformed, exit with code `2` and emit JSON error on stderr: + ```json + {"error_code":"HYPOTHESIS_ANALYST_INPUT","message":"...","remediation":"Provide valid evidence payload.","docs_url":""} + ``` + +Complete the analysis in English unless explicitly asked otherwise. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://kubernetes.io/docs/tasks/debug/debug-application/ +- https://kubernetes.io/docs/concepts/cluster-administration/logging/ diff --git a/oci/oke/agents/oke-lb-log-collector.md b/oci/oke/agents/oke-lb-log-collector.md new file mode 100644 index 0000000..4ec2c69 --- /dev/null +++ b/oci/oke/agents/oke-lb-log-collector.md @@ -0,0 +1,150 @@ +--- +name: oke-lb-log-collector +description: Resolves OCI load balancer identity and collects LB logging status and issue signals for OKE troubleshooting. +default-tool: bash +--- + +You gather load balancer logging evidence for OKE incidents. Use read-only commands by default. + +## Input Contract +JSON payload: +```json +{ + "namespace": "default", + "service": "sample-web-app-svc", + "region": "us-sanjose-1", + "compartment_ocid": "ocid1.compartment...", + "time_window": "15m", + "enable_logging_mode": "report_only" +} +``` + +Fields: +- `namespace` and `service`: Kubernetes Service identity. +- `region`: OCI region for API calls. +- `compartment_ocid`: compartment to search for LB/NLB resources. +- `time_window`: lookback (default `15m`) for log queries. +- `enable_logging_mode`: one of `report_only`, `print_command_only`, `run_now`. + +## Procedure +1. Resolve service external IP: + - `kubectl get svc -n -o jsonpath='{.status.loadBalancer.ingress[0].ip}'` +2. Resolve LB OCID from IP: + - `oci lb load-balancer list --compartment-id --region --all --output json` + - Match `ip-addresses[].ip-address == ` +3. If not found, try NLB: + - `oci nlb network-load-balancer list --compartment-id --region --all --output json` +4. For classic LB, perform dual logging checks: + - LB config check: + - `oci lb load-balancer get --load-balancer-id --region --query 'data."access-log"' --output json` + - Logging service object check: + - `oci logging log-group list --compartment-id --region --all --output json` + - For each returned log-group OCID: + - `oci logging log list --log-group-id --all --query "data[?configuration.source.resource=='' && configuration.source.service=='loadbalancer'].[\"display-name\",id,\"is-enabled\",configuration]" --output json` +5. Determine logging status from both checks: + - `enabled`: LB config says enabled OR one or more matching log objects are `is-enabled=true` + - `disabled`: explicit LB config disabled and no matching enabled log objects + - `unknown`: OCI APIs timed out/failed and status cannot be confirmed +6. If access logging is enabled and log identifiers are present, query recent logs: + - `oci logging search search-logs --region --search-query "search \"/\" | where data.loadBalancerId = '' | sort by datetime desc" --time-start --time-end ` +7. Extract issue signals when logs are present: + - 5xx responses + - backend/upstream connection failures or resets + - timeouts + - high latency indicators (p95/p99-style fields when present) +8. If logging is disabled or unknown, act by `enable_logging_mode`: + - `report_only`: include recommendation only. + - `print_command_only`: include exact enable command. + - `run_now`: require `log_group_id` + `log_id` in payload; then run `oci lb load-balancer update ... --access-log ...`. + +## Command Rules +- Print every command prefixed with `>>>`. +- Do not mutate resources unless `enable_logging_mode == "run_now"`. +- Use retry/backoff for OCI read calls (`oci lb ...`, `oci nlb ...`, `oci logging ...`): + - max attempts: `5` + - base delay: `2s` + - backoff: exponential (`2, 4, 8, 16, 32`) + - add small jitter (0-1s) between retries + - cap total retry wait at `~70s` +- Retry only transient failures: + - connection timeout, temporary DNS/network errors, HTTP 429/5xx + - do **not** retry auth/permission failures (401/403) or malformed requests +- On final failure after retries, continue and mark fallback. +- Redact secrets and tokens. + +### Retry Helper Pattern +When executing OCI commands, use this shell pattern: +```bash +run_oci_retry() { + local attempt max=5 + for attempt in 1 2 3 4 5; do + out="$("$@" 2>&1)" && { printf "%s" "$out"; return 0; } + rc=$? + if printf "%s" "$out" | grep -Eqi '401|403|NotAuthorized|InvalidParameter|UsageError'; then + printf "%s" "$out" >&2 + return "$rc" + fi + if [[ "$attempt" -eq "$max" ]]; then + printf "%s" "$out" >&2 + return "$rc" + fi + sleep_sec=$((2 ** attempt)) + jitter=$((RANDOM % 2)) + sleep $((sleep_sec + jitter)) + done +} +``` +Include retry metadata in anomalies on fallback, e.g.: +- `"OCI LB list timed out after 5 attempts (~65s total wait)"` +- `"LB config check succeeded but logging log list timed out; reporting partial status"` + +## Output Format +Return JSON on stdout: +```json +{ + "service": "sample-web-app-svc", + "namespace": "default", + "external_ip": "192.0.2.10", + "lb_type": "lb", + "lb_ocid": "ocid1.loadbalancer...", + "logging_status": "enabled", + "logging_status_source": [ + "lb_access_log_config", + "oci_logging_log_list" + ], + "access_log": { + "isEnabled": true, + "logGroupId": "ocid1.loggroup...", + "logId": "ocid1.log..." + }, + "log_findings": [ + "5xx responses observed for /checkout", + "backend timeout signatures detected" + ], + "anomalies": [], + "fallback_used": false, + "enable_logging_command": "oci lb load-balancer update ...", + "executed_enablement": false +} +``` + +If unresolved, set: +- `logging_status`: `disabled` or `unknown` +- `logging_status_source`: include whichever checks ran successfully +- `fallback_used`: `true` +- include rerun guidance in `anomalies`. + +## Error Handling +- Malformed input: exit `2` and emit JSON error to stderr: + ```json + {"error_code":"LB_LOG_COLLECTOR_INPUT","message":"...","remediation":"Provide valid payload.","docs_url":""} + ``` +- Expected environment/API issues should not hard-fail the whole run. Return JSON with `fallback_used=true`. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/Balance/home.htm +- https://docs.oracle.com/en-us/iaas/Content/Logging/home.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/ +- https://kubernetes.io/docs/concepts/services-networking/service/ diff --git a/oci/oke/cluster-design.md b/oci/oke/cluster-design.md new file mode 100644 index 0000000..11c4936 --- /dev/null +++ b/oci/oke/cluster-design.md @@ -0,0 +1,150 @@ +# OKE Cluster Design and Terraform Planning + +## Overview + +Use this skill to design an OCI Kubernetes Engine (OKE) cluster and translate the design into Terraform or an OCI Resource Manager stack. It is intended for cluster creation, cluster modernization, and review of OKE infrastructure plans before deployment. + +Start from official OKE requirements, then map the chosen architecture to Terraform variables, Resource Manager schema inputs, and validation checks. + +For full operational behavior, load `oci/oke/skills/oke-cluster-generator/SKILL.md` and its supporting files before generating artifacts. That skill preserves the OKE generator workflow, including live preflight discovery, CIDR validation, Terraform output templates, and OCI Resource Manager schema guidance. + +Supporting tools and references: + +- `oci/oke/scripts/preflight-check.sh` +- `oci/oke/scripts/validate-cidr.sh` +- `oci/oke/skills/oke-cluster-generator/references/questionnaire.md` +- `oci/oke/skills/oke-cluster-generator/reference.md` +- `oci/oke/skills/oke-cluster-generator/output-templates/terraform.md` +- `oci/oke/skills/oke-cluster-generator/output-templates/schema.md` + +## Tool Use + +Use the included tools during the design workflow: + +| Phase | Tool | When to use | +|-------|------|-------------| +| Preflight | `oci/oke/scripts/preflight-check.sh` | Run before asking detailed design questions when OCI CLI is available. Use the output to identify tenancy OCID, home region, subscribed regions, and active compartments. | +| Networking | `oci/oke/scripts/validate-cidr.sh` | Run before finalizing VCN, pod, and service CIDRs. Re-run if the user revises any CIDR late in the flow. | + +Example preflight: + +```bash +bash oci/oke/scripts/preflight-check.sh +``` + +Example CIDR validation: + +```bash +bash oci/oke/scripts/validate-cidr.sh 10.0.0.0/16 10.244.0.0/16 10.96.0.0/16 +``` + +If a tool fails because local OCI access is unavailable, continue with explicit manual prompts and record that generated artifacts rely on manually supplied values. + +## Design Sequence + +1. Confirm tenancy context: + - Region + - Compartment + - Existing VCN or new VCN + - Required environments such as dev, test, prod, or regulated workloads + +2. Choose cluster model: + - Enhanced cluster when using enhanced-cluster features such as managed add-ons and Workload Identity. + - Basic cluster only when the workload does not need enhanced-cluster features. + - Kubernetes version aligned with OKE-supported versions and node image compatibility. + +3. Choose API endpoint access: + - Private endpoint for restricted administration paths. + - Public endpoint only when external access is required and NSG/security-list rules are tightly scoped. + - Separate API endpoint, worker, pod, and load balancer subnets where practical. + +4. Choose pod networking: + - VCN-native pod networking when pods need VCN-routable IPs, virtual nodes, or multiple secondary VNIC pod networking. + - Flannel overlay only when VCN-routable pod IPs and VCN-native features are not required. + - Check pod CIDR and service CIDR overlap before finalizing the network plan. + +5. Design node pools: + - Use separate node pools for materially different workload profiles. + - Capture shape, OCPU, memory, node count, autoscaling boundaries, image source, labels, taints, boot volume, and placement subnets. + - Confirm shape availability, quota, and VNIC/pod capacity in the target region and availability domains. + +6. Select add-ons and integrations: + - CoreDNS and kube-proxy are baseline cluster components. + - Consider Cluster Autoscaler, OCI Native Ingress Controller, CSI drivers, monitoring, and logging based on workload needs. + - For OCI API access from pods, use Workload Identity on enhanced clusters. + +7. Define security and operations: + - Prefer private worker nodes. + - Use NSGs where possible for API endpoint, worker, pod, and load balancer traffic. + - Use least-privilege IAM policies. + - Decide whether to use Oracle-managed encryption or customer-managed keys. + - Define tagging, backup, upgrade, observability, and incident response expectations. + +## Fast Path Starter Design + +When the user asks for a quick starter stack, use conservative defaults and ask only for values that cannot be inferred: + +| Area | Starter value | +|------|---------------| +| Cluster type | Enhanced | +| API endpoint | Private | +| VCN | New VCN | +| VCN CIDR | `10.0.0.0/16` | +| CNI | VCN-native pod networking | +| Worker nodes | Private managed nodes | +| Node pool | 3 nodes, flexible general-purpose shape where available | +| Gateways | NAT gateway and service gateway | +| Add-ons | CoreDNS and kube-proxy, plus autoscaler when autoscaling is enabled | +| IAM | Create required policies or document manual policies | +| Workload Identity | Enabled when pods need OCI API access | +| Encryption | Oracle-managed unless a customer key is required | + +Before generating Terraform, show the starter design back to the user and ask whether they want to revise networking, node pools, access, add-ons, or security. + +## Terraform and Resource Manager Guidance + +When generating Terraform: + +- Prefer the official Oracle Terraform OKE module when it meets the design requirements. +- Pin or record the module version used for variable mapping. +- Keep required values explicit: region, tenancy OCID, compartment OCID, cluster name, Kubernetes version, network settings, and node pools. +- Do not guess module variable names. Verify them against the selected module version. +- Generate `terraform.tfvars.example` with placeholder OCIDs rather than real secrets. +- Include outputs for cluster OCID, node pool OCIDs, kubeconfig guidance, VCN/subnet IDs, and any created policy or key resources. +- If building an OCI Resource Manager stack, include a `schema.yaml` that groups inputs by audience and hides irrelevant fields when possible. + +## Validation Checklist + +Before presenting the final stack, check: + +- No CIDR overlap among VCN, pods, services, and connected networks. +- Kubernetes control plane and node versions are compatible. +- Chosen node images match the node Kubernetes version and shape architecture. +- Quotas and service limits are called out for node shapes, GPUs, load balancers, block volumes, and public IPs. +- Private nodes have egress through NAT and service gateway paths when needed. +- API endpoint access rules match the intended administration path. +- NSG and route-table assumptions are documented. +- Add-ons are valid for the selected cluster type. +- Workload Identity is only recommended for enhanced clusters. +- Customer-managed keys include required KMS permissions. +- Generated Terraform has clear validation steps such as `terraform fmt`, `terraform validate`, and `terraform plan`. + +## Common Mistakes + +- Selecting Flannel when the design requires VCN-routable pod IPs, virtual nodes, or multiple secondary VNIC pod networking. +- Using a public API endpoint without tightly scoped source CIDRs. +- Reusing one node pool for workloads with different security, cost, GPU, RDMA, or scaling requirements. +- Assuming shape availability or quota without checking the target region. +- Enabling autoscaling without confirming Cluster Autoscaler installation and node-pool min/max values. +- Mixing real tenancy-specific OCIDs into examples that will be committed to source control. +- Claiming a Terraform plan has been validated when only static generation was performed. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingclusterusingoke_topic-Using_the_Console_to_create_a_Custom_Cluster_with_Explicitly_Defined_Settings.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengworkingwithenhancedclusters.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengintroducingclusteraddons.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +- https://github.com/oracle-terraform-modules/terraform-oci-oke diff --git a/oci/oke/examples/outputs/oke-cluster-generator/main.tf b/oci/oke/examples/outputs/oke-cluster-generator/main.tf new file mode 100644 index 0000000..2657afe --- /dev/null +++ b/oci/oke/examples/outputs/oke-cluster-generator/main.tf @@ -0,0 +1,42 @@ +module "oke" { + source = "oracle-terraform-modules/oke/oci" + version = "~> 5.0" + + tenancy_id = var.tenancy_ocid + compartment_id = var.compartment_ocid + region = var.region + cluster_name = "demo-private-oke" + kubernetes_version = var.kubernetes_version + cluster_type = "enhanced" + control_plane_is_public = false + + create_vcn = true + vcn_cidrs = ["10.0.0.0/16"] + pods_cidr = "10.244.0.0/16" + services_cidr = "10.96.0.0/16" + cni_type = "npn" + create_bastion = false + create_operator = false + create_nat_gateway = true + create_service_gateway = true + create_internet_gateway = false + + node_pools = { + general = { + shape = "VM.Standard.E5.Flex" + ocpus = 2 + memory = 16 + node_pool_size = 3 + } + } + + block_volume_csi_enabled = true + fss_enabled = false + create_policies = true + workload_identity_enabled = true + + cluster_addons = { + coredns = { remove = false } + kube-proxy = { remove = false } + } +} diff --git a/oci/oke/examples/outputs/oke-gva-deployer/node-pool-command.sh b/oci/oke/examples/outputs/oke-gva-deployer/node-pool-command.sh new file mode 100644 index 0000000..8dcf1a7 --- /dev/null +++ b/oci/oke/examples/outputs/oke-gva-deployer/node-pool-command.sh @@ -0,0 +1,12 @@ +oci ce node-pool create \ + --compartment-id "" \ + --cluster-id "" \ + --name "gva-workers" \ + --kubernetes-version "v1.31.1" \ + --node-shape "VM.Standard.E5.Flex" \ + --node-shape-config '{"ocpus":2,"memoryInGBs":16}' \ + --size 3 \ + --cni-type OCI_VCN_IP_NATIVE \ + --placement-configs '[{"availabilityDomain":"GrCh:US-ASHBURN-AD-1","subnetId":""}]' \ + --node-source-details '{"sourceType":"IMAGE","imageId":""}' \ + --secondary-vnics '[{"createVnicDetails":{"ipCount":32,"applicationResources":["frontend"],"assignPublicIp":false,"displayName":"frontend-vnic","nsgIds":null,"subnetId":"","skipSourceDestCheck":false},"displayName":"frontend-vnic"}]' diff --git a/oci/oke/examples/outputs/oke-multihome-deployer/gva-multihome-pods.yaml b/oci/oke/examples/outputs/oke-multihome-deployer/gva-multihome-pods.yaml new file mode 100644 index 0000000..9dd067c --- /dev/null +++ b/oci/oke/examples/outputs/oke-multihome-deployer/gva-multihome-pods.yaml @@ -0,0 +1,97 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: gva-multihome-test +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: gva-default-network + namespace: kube-system +spec: + config: | + { + "name": "gva-default-network", + "cniVersion": "0.3.1", + "plugins": [ + { + "cniVersion": "0.3.1", + "type": "oci-ipvlan", + "mode": "l2", + "ipam": { + "type": "oci-ipam", + "deviceSelector": { + "interfaceName": "enp1s0" + } + } + }, + { + "cniVersion": "0.3.1", + "type": "oci-ptp", + "containerInterface": "ptp-veth0", + "mtu": 9000 + } + ] + } +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: gva-secondary-network + namespace: gva-multihome-test +spec: + config: | + { + "cniVersion": "0.3.1", + "plugins": [ + { + "type": "ipvlan", + "mode": "l2", + "master": "enp2s0", + "ipam": { + "type": "oci-ipam", + "deviceSelector": { + "interfaceName": "enp2s0" + } + } + } + ] + } +--- +apiVersion: v1 +kind: Pod +metadata: + name: gva-multihome-a + namespace: gva-multihome-test + labels: + app: gva-multihome + annotations: + v1.multus-cni.io/default-network: kube-system/gva-default-network + k8s.v1.cni.cncf.io/networks: gva-multihome-test/gva-secondary-network +spec: + nodeName: "node-a" + containers: + - name: netshoot + image: "docker.io/nicolaka/netshoot:v0.13" + command: + - sleep + - "3600" +--- +apiVersion: v1 +kind: Pod +metadata: + name: gva-multihome-b + namespace: gva-multihome-test + labels: + app: gva-multihome + annotations: + v1.multus-cni.io/default-network: kube-system/gva-default-network + k8s.v1.cni.cncf.io/networks: gva-multihome-test/gva-secondary-network +spec: + nodeName: "node-b" + containers: + - name: netshoot + image: "docker.io/nicolaka/netshoot:v0.13" + command: + - sleep + - "3600" diff --git a/oci/oke/examples/outputs/oke-troubleshooter/final-report.md b/oci/oke/examples/outputs/oke-troubleshooter/final-report.md new file mode 100644 index 0000000..d1c4c12 --- /dev/null +++ b/oci/oke/examples/outputs/oke-troubleshooter/final-report.md @@ -0,0 +1,31 @@ +# OKE Troubleshooting Report + +## Summary + +Pods in namespace `prod` are Pending because the current node pool cannot fit the +requested CPU and the cluster autoscaler cannot grow the node pool further. + +## Top Hypotheses + +| Rank | Hypothesis | Confidence | Score | Evidence | +| ---: | --- | --- | ---: | --- | +| 1 | Node pool max size reached | High | 9 | `FailedScheduling` reports insufficient CPU; autoscaler reports max node group size reached | +| 2 | OCI quota or capacity limit | Medium | 5 | Scale-out did not proceed; no direct quota error was collected | + +## Recommended Actions + +```bash +kubectl -n prod describe pod +oci ce node-pool update --node-pool-id --size +``` + +## Evidence Gaps + +- OCI quota and capacity checks were not available in this sample. +- Re-run after increasing node-pool size to confirm pods schedule. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/update.html +- https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/ diff --git a/oci/oke/examples/transcripts/oke-cluster-generator.md b/oci/oke/examples/transcripts/oke-cluster-generator.md new file mode 100644 index 0000000..f598092 --- /dev/null +++ b/oci/oke/examples/transcripts/oke-cluster-generator.md @@ -0,0 +1,59 @@ +# OKE Cluster Generator Transcript + +This is a shortened successful run that shows the shape of the interaction and the +artifacts produced. Values are placeholders. + +## Prompt + +```text +Use the OKE cluster generator fast-path us-ashburn-1 demo-private-oke +``` + +## Transcript + +**User:** Generate an OKE Terraform stack quickly for a private worker cluster in +`us-ashburn-1`. + +**Agent:** I will use fast path defaults for a production-friendly starter stack: +Enhanced cluster, private API endpoint, VCN-native pod networking, one Flex node pool, +Block Volume CSI enabled, Oracle-managed encryption, managed CoreDNS and kube-proxy, +and no public worker nodes. I still need your tenancy OCID and compartment OCID. + +**User:** Use `` and ``. + +**Agent:** Summary before generation: + +| Area | Choice | +| --- | --- | +| Region | `us-ashburn-1` | +| Cluster | `demo-private-oke` | +| Cluster type | Enhanced | +| API endpoint | Private | +| CNI | VCN-native pod networking | +| VCN | Create new VCN, `10.0.0.0/16` | +| Pod CIDR | `10.244.0.0/16` | +| Service CIDR | `10.96.0.0/16` | +| Node pool | `general`, 3 nodes, `VM.Standard.E5.Flex`, 2 OCPUs, 16 GB | +| Storage | Block Volume CSI | +| Add-ons | CoreDNS, kube-proxy | + +**User:** Generate it. + +**Agent:** Generated Terraform and ORM assets: + +- `provider.tf` +- `main.tf` +- `variables.tf` +- `outputs.tf` +- `terraform.tfvars.example` +- `schema.yaml` + +## What This Produces + +See [sample Terraform output](../outputs/oke-cluster-generator/main.tf). + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://github.com/oracle-terraform-modules/terraform-oci-oke +- https://docs.oracle.com/en-us/iaas/Content/ResourceManager/home.htm diff --git a/oci/oke/examples/transcripts/oke-gva-deployer.md b/oci/oke/examples/transcripts/oke-gva-deployer.md new file mode 100644 index 0000000..19792b2 --- /dev/null +++ b/oci/oke/examples/transcripts/oke-gva-deployer.md @@ -0,0 +1,44 @@ +# OKE GVA Deployer Transcript + +This is a shortened successful run for creating a GVA-enabled node pool. + +## Prompt + +```text +Use the OKE GVA deployer +``` + +## Transcript + +**Agent:** Which OKE cluster should I configure? + +**User:** `demo-oke`, region `us-ashburn-1`. + +**Agent:** I found the cluster OCID in kubeconfig and discovered VCN, subnet, and NSG +options. I will ask for one mutable node-pool value at a time. + +**User choices:** + +| Field | Value | +| --- | --- | +| Node pool name | `gva-workers` | +| Shape | `VM.Standard.E5.Flex` | +| OCPUs / memory | `2` / `16` | +| Node count | `3` | +| Primary subnet | `` | +| Application Resource | `frontend` | +| Secondary subnet | `` | +| `ipCount` | `32` | + +**Agent:** Generated a create command and a validation workload manifest. The command +is printed for review; it is not run unless you explicitly choose to run it. + +## What This Produces + +See [sample GVA command output](../outputs/oke-gva-deployer/node-pool-command.sh). + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/create.html +- https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ diff --git a/oci/oke/examples/transcripts/oke-multihome-deployer.md b/oci/oke/examples/transcripts/oke-multihome-deployer.md new file mode 100644 index 0000000..913a5b0 --- /dev/null +++ b/oci/oke/examples/transcripts/oke-multihome-deployer.md @@ -0,0 +1,46 @@ +# OKE Multihome Deployer Transcript + +This is a shortened successful run for validating Multus pods on a GVA node pool. + +## Prompt + +```text +Use the OKE multihome deployer +``` + +## Transcript + +**User:** Deploy Multus multi-home test pods on my GVA node pool and validate `net1`. + +**Agent:** I will discover the OKE cluster, node pool, placement subnets, secondary +VNIC subnets, and ready node names before generating YAML. + +**Agent:** Discovery found a GVA-enabled node pool with secondary VNIC interfaces +`enp1s0` and `enp2s0`. I will generate two pinned netshoot pods. + +**Agent:** Generated manifest: + +- Namespace `gva-multihome-test` +- Default NAD `kube-system/gva-default-network` +- Secondary NAD `gva-multihome-test/gva-secondary-network` +- Test pods `gva-multihome-a` and `gva-multihome-b` + +**Agent:** Validation passed: + +| Check | Result | +| --- | --- | +| Multus daemonset | Running | +| Pod network-status | `eth0` and `net1` present | +| Pod interfaces | `eth0`, `net1` | +| `net1` ping both directions | Passed | + +## What This Produces + +See [sample Multus manifest](../outputs/oke-multihome-deployer/gva-multihome-pods.yaml). + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://github.com/k8snetworkplumbingwg/multus-cni +- https://github.com/k8snetworkplumbingwg/network-attachment-definition-client diff --git a/oci/oke/examples/transcripts/oke-troubleshooter.md b/oci/oke/examples/transcripts/oke-troubleshooter.md new file mode 100644 index 0000000..42c444a --- /dev/null +++ b/oci/oke/examples/transcripts/oke-troubleshooter.md @@ -0,0 +1,47 @@ +# OKE Troubleshooter Transcript + +This is a shortened successful troubleshooting run with redacted identifiers. + +## Prompt + +```text +Use the OKE troubleshooter for pods stuck Pending in prod namespace +``` + +## Transcript + +**User:** Pods are stuck Pending in the `prod` namespace. + +**Agent:** I mapped the symptom to Pod scheduling, Node health, Cluster Autoscaler, +and OCI limits. I will collect read-only evidence with `kubectl` and OCI CLI. + +**Agent:** Evidence collected: + +- `kubectl -n prod get pods --field-selector=status.phase=Pending -o wide` +- `kubectl -n prod get events --field-selector reason=FailedScheduling` +- `kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200` +- `oci ce node-pool list --compartment-id --cluster-id ` + +**Agent:** Findings: + +| Hypothesis | Score | Evidence | +| --- | ---: | --- | +| Node pool max size reached | 9 | FailedScheduling reports insufficient CPU; autoscaler logs show `max node group size reached` | +| OCI compute capacity or quota limit | 5 | No direct quota error found, but scale-out was blocked | + +**Agent:** Next steps: + +```bash +kubectl -n prod describe pod +oci ce node-pool update --node-pool-id --size +``` + +## What This Produces + +See [sample troubleshooting report](../outputs/oke-troubleshooter/final-report.md). + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/list.html +- https://kubernetes.io/docs/tasks/debug/debug-application/ diff --git a/oci/oke/gva-node-pools.md b/oci/oke/gva-node-pools.md new file mode 100644 index 0000000..c857d73 --- /dev/null +++ b/oci/oke/gva-node-pools.md @@ -0,0 +1,221 @@ +# OKE Generic VNIC Attachment Node Pools + +## Overview + +Use this skill to plan, create, or review OCI Kubernetes Engine (OKE) managed node pools that use Generic VNIC Attachment (GVA) secondary VNIC profiles for pod networking. + +GVA lets a node pool expose multiple secondary VNIC profiles, each with its own subnet, NSGs, display name, and pod IP allocation. Optional Application Resources let pods request one specific secondary VNIC profile through Kubernetes extended resources. + +For full operational behavior, load `oci/oke/skills/oke-gva-deployer/SKILL.md` and its supporting files before generating or running commands. That skill preserves the guided GVA workflow, including discovery, menu-driven command generation, validation templates, and approval-gated node-pool creation. + +Supporting tools and references: + +- `oci/oke/scripts/gva-discover.sh` +- `oci/oke/scripts/gva-menu.sh` +- `oci/oke/scripts/gva-cli-resolve.sh` +- `oci/oke/skills/oke-gva-deployer/references/gva.md` +- `oci/oke/skills/oke-gva-deployer/validation-report-template.md` + +## Tool Use + +Use the included tools during the GVA workflow: + +| Phase | Tool | When to use | +|-------|------|-------------| +| Discovery | `oci/oke/scripts/gva-discover.sh` | Run after the user identifies the cluster and region. Use it to discover cluster metadata, subnets, and NSGs before prompting manually. | +| Guided node-pool build | `oci/oke/scripts/gva-menu.sh` | Use for the interactive GVA node-pool workflow. It prompts in the terminal, collects fresh values, generates an `oci ce node-pool create` command, and can run it only after explicit approval and final `CREATE` confirmation. | +| Optional local GVA CLI resolution | `oci/oke/scripts/gva-cli-resolve.sh` | Use only when the workflow needs to locate a local GVA CLI helper installation. | + +Example discovery: + +```bash +bash oci/oke/scripts/gva-discover.sh --cluster --region +``` + +Example guided builder: + +```bash +bash oci/oke/scripts/gva-menu.sh +``` + +Run `gva-menu.sh` only after the user has approved entering the interactive guided creation flow. It is not a passive help command. Choose print-only mode unless the user explicitly approves running the create command. + +## When to Use + +Use this skill when the task involves: + +- Secondary VNIC profiles on OKE managed node pools. +- Application Resources such as `oke-application-resource.oci.oraclecloud.com/frontend`. +- Workload-specific pod network segmentation. +- Creating new node pools with multiple pod networking paths, or reviewing update plans for existing node pools. +- Validating pods pinned to a single secondary VNIC profile. + +Use `oci/oke/multus-multihome.md` instead when the pod needs multiple interfaces at the same time. + +## Prerequisites + +Confirm: + +- The cluster uses VCN-native pod networking. +- The target node pool is a managed node pool. +- The selected shape supports the required number of VNICs and pod IP capacity. +- Secondary VNIC subnets and NSGs already exist or are included in the infrastructure plan. +- Subnet CIDR capacity is sufficient for the requested `ipCount`. +- OCI CLI and `kubectl` are available when live discovery or validation is requested. +- IAM permissions allow the operator to manage the node-pool and networking resources. + +Multiple secondary VNIC pod networking is not supported with Flannel overlay networking. + +## Design Inputs + +Collect: + +| Input | Notes | +|-------|-------| +| Cluster OCID and region | Use the user-provided cluster; do not assume the current context is correct. | +| Operation type | Interactive creation is tool-backed; updates require separate review of the exact update command before execution. New node pools are safer when changing network model. | +| Node shape | Validate architecture and VNIC limits. | +| Node count and placement | Include availability domains and worker subnet. | +| Kubernetes version and image | Node image version must match the selected node Kubernetes version. | +| Secondary VNIC profile names | Use stable names that describe the network tier. | +| Secondary subnet OCIDs | Each profile can use a different subnet. | +| NSG OCIDs | Apply least-privilege rules per profile. | +| `ipCount` | Size pod IP allocation with headroom. | +| Application Resource name | Use only when pods must select one specific secondary VNIC profile. | + +## Application Resource Model + +When a secondary VNIC profile has an Application Resource name, OKE exposes a Kubernetes extended resource on the nodes: + +```text +oke-application-resource.oci.oraclecloud.com/ +``` + +Pods that use this model must: + +- Request exactly one Application Resource type. +- Request and limit one unit of that resource. +- Include the toleration for the node taint: + +```yaml +tolerations: + - key: oci.oraclecloud.com/application-resource-only + operator: Exists + effect: NoSchedule +``` + +Example container resource request: + +```yaml +resources: + requests: + oke-application-resource.oci.oraclecloud.com/frontend: "1" + limits: + oke-application-resource.oci.oraclecloud.com/frontend: "1" +``` + +Use Application Resources when a pod should be pinned to one selected secondary VNIC profile. Do not use this pod-level Application Resource model for pods that need multiple interfaces through Multus. + +## Approval-Gated Creation + +Creating an OKE node pool is allowed in this skill because it is part of the GVA workflow. The agent must still ask for explicit approval before running a mutating command. + +`oci/oke/scripts/gva-menu.sh` supports this approval model by generating the command first, offering a print-only path, and requiring the user to type `CREATE` before it runs `oci ce node-pool create`. + +## CLI Planning Pattern + +When generating an OCI CLI command, build the secondary VNIC profile list explicitly and show it for review before running anything. + +Abbreviated JSON shape: + +```json +[ + { + "displayName": "frontend-vnic-attachment", + "createVnicDetails": { + "displayName": "frontend-vnic", + "subnetId": "ocid1.subnet...", + "nsgIds": ["ocid1.networksecuritygroup..."], + "assignPublicIp": false, + "skipSourceDestCheck": false, + "ipCount": 32, + "applicationResources": ["frontend"] + } + } +] +``` + +Before execution, confirm: + +- Cluster and region. +- Node pool name. +- Placement subnet. +- Shape, OCPUs, memory, and node count. +- Each secondary subnet and NSG. +- Each Application Resource name. +- `ipCount` per profile. +- Whether this is an interactive create flow or an update review. + +## Validation + +After the node pool is active, validate in this order: + +1. Confirm nodes joined: + +```bash +kubectl get nodes -o wide +``` + +2. Pick one node from the GVA node pool and inspect capacity: + +```bash +kubectl describe node +``` + +Check for: + +```text +Capacity: + oke-application-resource.oci.oraclecloud.com/: +``` + +3. Confirm the node taint when Application Resources are configured: + +```text +oci.oraclecloud.com/application-resource-only:NoSchedule +``` + +4. Deploy or review a test pod that requests exactly one Application Resource and has the matching toleration. + +5. Inspect pod scheduling events and assigned IP: + +```bash +kubectl describe pod -n +kubectl get pod -n -o wide +``` + +6. If the pod remains Pending, check: + - Resource name spelling and case. + - Available extended resource capacity. + - Required toleration. + - Node selectors or affinity. + - Subnet pod IP capacity. + - Shape VNIC limits. + +## Common Mistakes + +- Trying to use GVA with Flannel networking. +- Combining pod-level Application Resource requests with Multus annotations in the same pod. +- Requesting more than one Application Resource type in a pod. +- Forgetting the `oci.oraclecloud.com/application-resource-only:NoSchedule` toleration. +- Setting `ipCount` without checking subnet capacity and shape VNIC limits. +- Reusing old node-pool JSON payloads without revalidating cluster, region, subnets, NSGs, image, and shape. +- Assuming Application Resource names are case-insensitive. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingclusterusingoke_topic-Using_the_Console_to_create_a_Custom_Cluster_with_Explicitly_Defined_Settings.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/create.html +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/update.html diff --git a/oci/oke/multus-multihome.md b/oci/oke/multus-multihome.md new file mode 100644 index 0000000..4e0f1ab --- /dev/null +++ b/oci/oke/multus-multihome.md @@ -0,0 +1,244 @@ +# OKE Multus Multi-Interface Pods + +## Overview + +Use this skill to plan, deploy, or troubleshoot Multus NetworkAttachmentDefinitions (NADs) and multi-interface pods on OCI Kubernetes Engine (OKE), especially on node pools with multiple secondary VNIC profiles. + +Use this when a pod needs more than one network interface, such as `eth0` plus `net1`, or when validating that a Multus attachment uses the intended OKE secondary VNIC path. + +For full operational behavior, load `oci/oke/skills/oke-multihome-deployer/SKILL.md` and its supporting files before generating manifests or validating pods. That skill preserves the Multus multihome workflow, including OKE/GVA discovery, manifest generation, validation reports, and DPDK/SR-IOV notes. + +Supporting tools and references: + +- `oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py` +- `oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py` +- `oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md` +- `oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md` +- `oci/oke/skills/oke-multihome-deployer/validation-report-template.md` + +## Tool Use + +Use the included tools during the Multus multihome workflow: + +| Phase | Tool | When to use | +|-------|------|-------------| +| Discovery | `oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py` | Run after the user identifies the cluster, context, region, or compartment. Use it to discover node pools, placement subnets, secondary VNIC subnets, and suggested manifest generator arguments. | +| Manifest generation | `oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py` | Run after the user confirms namespace, NAD names, interfaces, image, and target nodes. It generates YAML but does not apply it. | + +Example discovery: + +```bash +python3 oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py \ + --cluster-id \ + --region \ + --compartment-id \ + --pretty +``` + +Example manifest generation: + +```bash +python3 oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py \ + --namespace gva-multihome-test \ + --default-interface enp1s0 \ + --secondary-interface enp2s0 \ + --pod gva-multihome-a= \ + --pod gva-multihome-b= +``` + +Show generated YAML to the user before applying it. Applying manifests requires explicit approval. + +## When to Use + +Use this skill for: + +- Multus installation and health checks on OKE. +- NetworkAttachmentDefinition review. +- Multi-interface pod manifests. +- `k8s.v1.cni.cncf.io/networks` annotations. +- `v1.multus-cni.io/default-network` annotations. +- Pod `network-status` validation. +- Missing `net1` interfaces. +- OKE secondary VNIC profile selection from NAD configuration. + +Use `oci/oke/gva-node-pools.md` instead when a pod only needs to select one secondary VNIC profile through an Application Resource request. + +## Core Model + +OKE supports different pod-networking models: + +| Model | Selection mechanism | Use case | +|-------|---------------------|----------| +| Single pinned secondary VNIC profile | Pod-level Application Resource request | Pod needs exactly one selected secondary VNIC path | +| Multi-interface pod | Multus annotations and NADs | Pod needs multiple interfaces | +| Secondary VNIC profiles without Application Resources | NAD device selection or default CNI behavior | Capacity or interface-based selection without scheduler resource pinning | + +Do not combine pod-level Application Resource requests with Multus pod network annotations in the same pod spec. If a multi-interface pod needs to select a secondary VNIC profile, define that selection in the NAD, such as with `deviceSelector.appResource` when supported. + +## Prerequisites + +Confirm: + +- The cluster uses VCN-native pod networking. +- Worker nodes are in a node pool configured with the required secondary VNIC profiles. +- Multus is installed and healthy. +- Required CNI binaries exist on every target worker node. +- NADs are created in the correct namespace. +- Subnets and NSGs allow the traffic being tested. +- The pod image is available from a reachable registry. + +## Health Checks + +Check Multus: + +```bash +kubectl get pods -n kube-system -l app=multus -o wide +kubectl get daemonset -n kube-system +``` + +Check NADs: + +```bash +kubectl get network-attachment-definitions -A +kubectl get network-attachment-definition -n -o yaml +``` + +Check the target pod: + +```bash +kubectl describe pod -n +kubectl get pod -n -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' +``` + +Check interfaces inside the pod: + +```bash +kubectl exec -n -- ip addr +kubectl exec -n -- ip route +``` + +## NAD Review + +Review each NAD for: + +- Correct namespace. +- Correct CNI type for the intended path. +- Correct subnet or IPAM configuration. +- Correct parent interface or `deviceSelector`. +- Valid JSON inside the NAD `config`. +- Security model and expected traffic path. + +A default-network NAD can be used to make `eth0` selection explicit: + +```yaml +metadata: + annotations: + v1.multus-cni.io/default-network: kube-system/oci-vcn-native-network + k8s.v1.cni.cncf.io/networks: default/secondary-network +``` + +The additional network annotation attaches extra interfaces such as `net1`. + +## Validation Flow + +1. Confirm the pod is scheduled onto the intended node pool: + +```bash +kubectl get pod -n -o wide +``` + +2. Confirm Multus network status exists: + +```bash +kubectl describe pod -n +``` + +Look for the `k8s.v1.cni.cncf.io/network-status` annotation. + +3. Confirm expected interfaces: + +```bash +kubectl exec -n -- ip -brief addr +``` + +Expected result: + +```text +lo UNKNOWN +eth0 UP +net1 UP +``` + +4. Confirm IP addresses are from the intended subnets. + +5. Confirm routes match the expected traffic path: + +```bash +kubectl exec -n -- ip route +``` + +6. Test connectivity only to approved targets: + +```bash +kubectl exec -n -- ping -c 3 +``` + +## Common Failure Patterns + +### Pod Has Only `eth0` + +Likely causes: + +- NAD name or namespace does not match the pod annotation. +- Multus is not running or not injected as expected. +- The additional NAD failed validation. +- The pod annotation is malformed. + +### Pod Sandbox Creation Fails + +Likely causes: + +- CNI binary referenced by the NAD is missing on the node. +- NAD JSON is invalid. +- OCI IPAM cannot allocate an address. +- Selected secondary VNIC profile has no available pod IP capacity. +- Node is missing the expected parent interface. + +### Pod Schedules But Traffic Fails + +Likely causes: + +- NSG or security-list rules block traffic. +- Route table does not support the path. +- The pod IP is from a different subnet than expected. +- The wrong interface is being used. +- Return traffic is blocked. + +### DPDK or SR-IOV Workload Fails + +First separate the layers: + +- Kubernetes device-plugin resources only prove resource allocation. +- Multus `network-status` proves interface attachment. +- NAD configuration defines what each attached network means. +- Driver, RDMA, hugepages, and DPDK application configuration are separate checks. + +Do not assume the application failure is a network attachment failure until `network-status`, pod interfaces, and device visibility have been checked. + +## Common Mistakes + +- Combining Multus annotations with pod-level Application Resource requests. +- Creating NADs in `default` while the workload runs in another namespace. +- Referencing a short NAD name when a fully qualified `/` reference is needed. +- Assuming `net1` exists without checking `network-status`. +- Testing connectivity before confirming interface IPs and routes. +- Replacing node CNI binaries broadly when only one missing plugin binary needs investigation. +- Treating SR-IOV or DPDK resource allocation as proof that Multus attached the expected network. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://github.com/k8snetworkplumbingwg/multus-cni +- https://github.com/k8snetworkplumbingwg/network-attachment-definition-client diff --git a/oci/oke/scripts/gva-cli-resolve.sh b/oci/oke/scripts/gva-cli-resolve.sh new file mode 100755 index 0000000..a144cf6 --- /dev/null +++ b/oci/oke/scripts/gva-cli-resolve.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +WHEEL_NAME="oci_cli-3.65.2+preview.1.1355-py2.py3-none-any.whl" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +mode="json" +while [[ $# -gt 0 ]]; do + case "$1" in + --json) + mode="json" + shift + ;; + --print-home) + mode="home" + shift + ;; + --print-activate) + mode="activate" + shift + ;; + --print-wheel) + mode="wheel" + shift + ;; + -h|--help) + cat <<'EOF' +usage: gva-cli-resolve.sh [--json|--print-home|--print-activate|--print-wheel] + +Search order: +1. $OKE_GVA_CLI_HOME +2. /gva-cli +EOF + exit 0 + ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." \ + "Run with --help to view usage." + ;; + esac +done + +declare -a candidates=() +if [[ -n "${OKE_GVA_CLI_HOME:-}" ]]; then + candidates+=("$OKE_GVA_CLI_HOME") +fi +candidates+=( + "$REPO_ROOT/gva-cli" +) + +resolved_home="" +resolved_activate="" +resolved_wheel="" + +for candidate in "${candidates[@]}"; do + [[ -d "$candidate" ]] || continue + activate_path="$candidate/bin/activate" + wheel_path="$candidate/$WHEEL_NAME" + resolved_home="$candidate" + if [[ -f "$activate_path" ]]; then + resolved_activate="$activate_path" + fi + if [[ -f "$wheel_path" ]]; then + resolved_wheel="$wheel_path" + fi + break +done + +if [[ -z "$resolved_home" ]]; then + emit_error 1 "GVA_CLI_HOME_NOT_FOUND" \ + "Could not locate the preview GVA CLI workspace." \ + "Set OKE_GVA_CLI_HOME or place gva-cli under the OCI OKE skill root." \ + "https://github.com/oracle/skills/tree/main/oci/oke" +fi + +case "$mode" in + home) + printf '%s\n' "$resolved_home" + ;; + activate) + if [[ -z "$resolved_activate" ]]; then + emit_error 1 "GVA_CLI_ACTIVATE_NOT_FOUND" \ + "Resolved GVA CLI home does not contain bin/activate." \ + "Recreate the virtual environment under the resolved GVA CLI home." + fi + printf '%s\n' "$resolved_activate" + ;; + wheel) + if [[ -z "$resolved_wheel" ]]; then + emit_error 1 "GVA_CLI_WHEEL_NOT_FOUND" \ + "Resolved GVA CLI home does not contain the preview OCI CLI wheel." \ + "Place the preview wheel in the resolved GVA CLI home or update the skill instructions." + fi + printf '%s\n' "$resolved_wheel" + ;; + json) + python3 - "$resolved_home" "$resolved_activate" "$resolved_wheel" <<'PY' +import json +import sys + +home = sys.argv[1] +activate = sys.argv[2] +wheel = sys.argv[3] + +print(json.dumps({ + "home": home, + "activate": activate, + "wheel": wheel, + "has_activate": bool(activate), + "has_wheel": bool(wheel), +}, indent=2)) +PY + ;; +esac diff --git a/oci/oke/scripts/gva-discover.sh b/oci/oke/scripts/gva-discover.sh new file mode 100755 index 0000000..97c386c --- /dev/null +++ b/oci/oke/scripts/gva-discover.sh @@ -0,0 +1,488 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Discover GVA-relevant data for an OKE cluster. +# Usage: +# ./scripts/gva-discover.sh --cluster [--region ] [--compartment-id ] [--profile ] [--timeout ] [--kubeconfig ] + +cluster_ref="" +region_arg="" +compartment_arg="" +profile_arg="" +timeout_arg="" +kubeconfig_arg="" +cache_ttl_arg="0" +cache_dir_arg="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." \ + "Run with --help to view usage." + fi +} + +read_lines_into_array() { + local array_name="$1" + local line="" + eval "$array_name=()" + while IFS= read -r line; do + eval "$array_name+=(\"\$line\")" + done +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --cluster) + require_value "$1" "${2:-}" + cluster_ref="$2"; shift 2 ;; + --region) + require_value "$1" "${2:-}" + region_arg="$2"; shift 2 ;; + --compartment-id) + require_value "$1" "${2:-}" + compartment_arg="$2"; shift 2 ;; + --profile) + require_value "$1" "${2:-}" + profile_arg="$2"; shift 2 ;; + --timeout) + require_value "$1" "${2:-}" + timeout_arg="$2"; shift 2 ;; + --kubeconfig) + require_value "$1" "${2:-}" + kubeconfig_arg="$2"; shift 2 ;; + --cache-ttl) + require_value "$1" "${2:-}" + cache_ttl_arg="$2"; shift 2 ;; + --cache-dir) + require_value "$1" "${2:-}" + cache_dir_arg="$2"; shift 2 ;; + -h|--help) + echo "usage: $0 --cluster [--region ] [--compartment-id ] [--profile ] [--timeout ] [--kubeconfig ] [--cache-ttl ] [--cache-dir ]" >&2 + exit 0 ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." \ + "Run with --help to view usage." ;; + esac +done + +if [[ -z "$cluster_ref" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --cluster." \ + "Provide --cluster ." +fi + +if ! command -v oci >/dev/null 2>&1; then + emit_error 2 "OCI_CLI_NOT_INSTALLED" "OCI CLI not found on PATH." \ + "Install OCI CLI and retry." \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm" +fi + +# Read tenancy and default region from OCI config if present +config_file="$HOME/.oci/config" +config_region="" +if [[ -f "$config_file" ]]; then + config_region=$(awk -F= '/^region=/{print $2; exit}' "$config_file" | tr -d ' ') +fi + +region="${region_arg:-$config_region}" +if [[ -z "$region" ]]; then + emit_error 2 "REGION_NOT_PROVIDED" "Region not provided and not found in OCI config." \ + "Provide --region or set region in ~/.oci/config." +fi + +if ! [[ "$cache_ttl_arg" =~ ^[0-9]+$ ]]; then + emit_error 2 "INVALID_ARGUMENT" "--cache-ttl must be a non-negative integer." \ + "Use --cache-ttl ." +fi + +cache_dir="${cache_dir_arg:-${TMPDIR:-/tmp}/gva-discover-cache}" +cache_enabled="no" +cache_file="" +if [[ "$cache_ttl_arg" -gt 0 ]]; then + mkdir -p "$cache_dir" + cache_key=$(python3 - "$cluster_ref" "$region" "$compartment_arg" "$profile_arg" <<'PY' +import hashlib +import sys +raw = "|".join(sys.argv[1:]) +print(hashlib.sha256(raw.encode("utf-8")).hexdigest()) +PY + ) + cache_file="${cache_dir}/${cache_key}.json" + cache_enabled="yes" + if [[ -f "$cache_file" ]]; then + if python3 - "$cache_file" "$cache_ttl_arg" <<'PY' +import os +import sys +import time +path = sys.argv[1] +ttl = int(sys.argv[2]) +age = time.time() - os.path.getmtime(path) +sys.exit(0 if age <= ttl else 1) +PY + then + cat "$cache_file" + exit 0 + fi + fi +fi + +# Helper to run oci with region/profile/timeout +have_timeout="no" +if command -v timeout >/dev/null 2>&1; then + have_timeout="yes" +fi + +use_py_timeout="no" +declare -a timeout_prefix=() +if [[ -n "$timeout_arg" ]]; then + if [[ "$have_timeout" == "yes" ]]; then + timeout_prefix=(timeout "$timeout_arg") + else + use_py_timeout="yes" + fi +fi + +declare -a profile_args=() +if [[ -n "$profile_arg" ]]; then + profile_args=(--profile "$profile_arg") +elif [[ -n "${OCI_CLI_PROFILE:-}" ]]; then + profile_args=(--profile "$OCI_CLI_PROFILE") +fi + +oci_json() { + local out err rc + local -a cmd + err="$(mktemp)" + cmd=(oci --region "$region") + if [[ ${#profile_args[@]} -gt 0 ]]; then + cmd+=("${profile_args[@]}") + fi + cmd+=("$@") + if [[ ${#timeout_prefix[@]} -gt 0 ]]; then + cmd=("${timeout_prefix[@]}" "${cmd[@]}") + fi + if [[ "$use_py_timeout" == "yes" ]]; then + set +e + out="$(python3 - "$timeout_arg" "$err" "${cmd[@]}" <<'PY' +import subprocess, sys +timeout = float(sys.argv[1]) +err_path = sys.argv[2] +cmd = sys.argv[3:] +try: + p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout, text=True) + if p.stderr: + with open(err_path, "w") as f: + f.write(p.stderr) + sys.stdout.write(p.stdout or "") + sys.exit(p.returncode) +except subprocess.TimeoutExpired: + with open(err_path, "w") as f: + f.write("Command timed out after %ss\\n" % timeout) + sys.exit(124) +PY +)" + rc=$? + set -e + else + set +e + out="$("${cmd[@]}" 2>"$err")" + rc=$? + set -e + fi + if [[ $rc -ne 0 ]]; then + echo "error: oci $* failed (exit $rc)" >&2 + sed -e 's/^/oci stderr: /' "$err" >&2 + rm -f "$err" + return "$rc" + fi + rm -f "$err" + printf "%s" "$out" +} + +json_get_field() { + local field="$1" + python3 -c ' +import json +import sys + +field = sys.argv[1] +try: + data = json.load(sys.stdin) +except Exception: + data = {} + +if isinstance(data, dict): + value = data.get(field, "") + if value is None: + value = "" + print(value) +else: + print("") +' "$field" +} + +# If cluster ref is OCID, use it directly. Otherwise require compartment to search by name. +cluster_ocid="" +cluster_name="" +cluster_k8s="" +compartment_ocid="" +cluster_vcn_id="" + +is_ocid="no" +if [[ "$cluster_ref" == ocid1.* ]]; then + is_ocid="yes" +fi + +if [[ "$is_ocid" == "yes" ]]; then + cluster_ocid="$cluster_ref" + cluster_json=$(oci_json ce cluster get --cluster-id "$cluster_ocid" --query 'data.{name:"name",k8s:"kubernetes-version",compartment:"compartment-id",vcn:"vcn-id"}' --output json || true) + if [[ -z "$cluster_json" || "$cluster_json" == "{}" ]]; then + echo "warning: failed to fetch cluster details for $cluster_ocid (check auth/region/profile)" >&2 + else + cluster_name=$(printf '%s' "$cluster_json" | json_get_field name) + cluster_k8s=$(printf '%s' "$cluster_json" | json_get_field k8s) + compartment_ocid=$(printf '%s' "$cluster_json" | json_get_field compartment) + cluster_vcn_id=$(printf '%s' "$cluster_json" | json_get_field vcn) + fi +else + # Try to resolve cluster OCID from kubeconfig if no compartment was provided. + if [[ -z "$compartment_arg" ]]; then + kubeconfig_path="${kubeconfig_arg:-$HOME/.kube/config}" + if [[ -f "$kubeconfig_path" ]]; then + kube_cluster_id=$(python3 - "$kubeconfig_path" "$cluster_ref" <<'PY' +import sys +from pathlib import Path +try: + import yaml +except Exception: + print("") + raise SystemExit(0) + +path = Path(sys.argv[1]) +name = sys.argv[2] +try: + data = yaml.safe_load(path.read_text()) +except Exception: + print("") + raise SystemExit(0) + +contexts = data.get("contexts", []) or [] +users = {u.get("name"): u.get("user", {}) for u in (data.get("users", []) or [])} + +def pick_user(contexts, target): + exact = [] + exact_ci = [] + contains = [] + for c in contexts: + cname = c.get("name", "") + cuser = (c.get("context", {}) or {}).get("user") + if not cuser: + continue + if cname == target: + exact.append((len(cname), cname, cuser)) + elif cname.lower() == target.lower(): + exact_ci.append((len(cname), cname, cuser)) + elif target and target in cname: + contains.append((len(cname), cname, cuser)) + for bucket in (exact, exact_ci, contains): + if bucket: + bucket.sort() + return bucket[0][2] + return None + +match_user = pick_user(contexts, name) + +if not match_user: + print("") + raise SystemExit(0) + +exec_cfg = users.get(match_user, {}).get("exec", {}) +args = exec_cfg.get("args", []) if isinstance(exec_cfg, dict) else [] +try: + idx = args.index("--cluster-id") + print(args[idx + 1]) +except Exception: + print("") +PY + ) + if [[ -n "$kube_cluster_id" ]]; then + cluster_ocid="$kube_cluster_id" + is_ocid="yes" + else + # Fallback: iterate all cluster-ids in kubeconfig and match by name via oci ce cluster get + read_lines_into_array kube_cluster_ids < <(python3 - "$kubeconfig_path" <<'PY' +import sys +from pathlib import Path +try: + import yaml +except Exception: + sys.exit(0) +path = Path(sys.argv[1]) +try: + data = yaml.safe_load(path.read_text()) +except Exception: + sys.exit(0) +users = {u.get("name"): u.get("user", {}) for u in (data.get("users", []) or [])} +ids = [] +for u in users.values(): + exec_cfg = u.get("exec", {}) + args = exec_cfg.get("args", []) if isinstance(exec_cfg, dict) else [] + for i, a in enumerate(args): + if a == "--cluster-id" and i + 1 < len(args): + ids.append(args[i + 1]) +print("\\n".join(sorted(set(ids)))) +PY + ) + if [[ ${#kube_cluster_ids[@]} -gt 0 ]]; then + for cid in "${kube_cluster_ids[@]}"; do + match_json=$(oci_json ce cluster get --cluster-id "$cid" --query 'data.{name:"name",k8s:"kubernetes-version",compartment:"compartment-id",vcn:"vcn-id"}' --output json || true) + if [[ -n "$match_json" && "$match_json" != "{}" ]]; then + match_name=$(printf '%s' "$match_json" | json_get_field name) + if [[ "$match_name" == "$cluster_ref" ]]; then + cluster_ocid="$cid" + cluster_name="$match_name" + cluster_k8s=$(printf '%s' "$match_json" | json_get_field k8s) + compartment_ocid=$(printf '%s' "$match_json" | json_get_field compartment) + cluster_vcn_id=$(printf '%s' "$match_json" | json_get_field vcn) + is_ocid="yes" + break + fi + fi + done + fi + fi + fi + fi + + if [[ "$is_ocid" == "yes" ]]; then + cluster_json=$(oci_json ce cluster get --cluster-id "$cluster_ocid" --query 'data.{name:"name",k8s:"kubernetes-version",compartment:"compartment-id",vcn:"vcn-id"}' --output json || true) + if [[ -z "$cluster_json" || "$cluster_json" == "{}" ]]; then + echo "warning: failed to fetch cluster details for $cluster_ocid (check auth/region/profile)" >&2 + else + cluster_name=$(printf '%s' "$cluster_json" | json_get_field name) + cluster_k8s=$(printf '%s' "$cluster_json" | json_get_field k8s) + compartment_ocid=$(printf '%s' "$cluster_json" | json_get_field compartment) + cluster_vcn_id=$(printf '%s' "$cluster_json" | json_get_field vcn) + fi + fi + + # Require a compartment to search by name (avoid tenancy-wide scans). + if [[ -z "$compartment_arg" && -z "$compartment_ocid" && "$is_ocid" != "yes" ]]; then + emit_error 2 "COMPARTMENT_REQUIRED_FOR_CLUSTER_NAME" \ + "Compartment ID is required when using a cluster name." \ + "Provide --compartment-id, use cluster OCID, or use kubeconfig that includes --cluster-id." + fi + + comp_to_search="${compartment_arg:-$compartment_ocid}" + clusters_json=$(oci_json ce cluster list --compartment-id "$comp_to_search" --output json || true) + hit="" + if [[ -n "$clusters_json" ]]; then + hit=$(printf '%s' "$clusters_json" | CLUSTER_REF="$cluster_ref" python3 -c ' +import json +import os +import sys + +target = os.environ.get("CLUSTER_REF", "") +try: + payload = json.loads(sys.stdin.read()) +except Exception: + print("") + raise SystemExit(0) + +items = payload.get("data", []) if isinstance(payload, dict) else [] +for item in items: + if (item or {}).get("name", "") == target: + print(json.dumps(item)) + break +') + fi + if [[ -n "$hit" && "$hit" != "null" ]]; then + cluster_ocid=$(printf '%s' "$hit" | json_get_field id) + cluster_name=$(printf '%s' "$hit" | json_get_field name) + cluster_k8s=$(printf '%s' "$hit" | json_get_field kubernetes-version) + compartment_ocid="$comp_to_search" + fi +fi + +if [[ -z "$compartment_ocid" && -n "$compartment_arg" ]]; then + compartment_ocid="$compartment_arg" +fi + +# Pull subnets and NSGs in the compartment +subnets_json="[]" +nsGs_json="[]" +if [[ -n "$compartment_ocid" ]]; then + subnets_tmp="$(mktemp)" + nsgs_tmp="$(mktemp)" + + if [[ -n "$cluster_vcn_id" ]]; then + ( + oci_json network subnet list --compartment-id "$compartment_ocid" --vcn-id "$cluster_vcn_id" --query 'data[*].{"name":"display-name","id":"id","cidr":"cidr-block","ipv6":"ipv6-cidr-blocks"}' --output json || true + ) >"$subnets_tmp" & + subnets_pid=$! + ( + oci_json network nsg list --compartment-id "$compartment_ocid" --vcn-id "$cluster_vcn_id" --query 'data[*].{"name":"display-name","id":"id"}' --output json || true + ) >"$nsgs_tmp" & + nsgs_pid=$! + else + ( + oci_json network subnet list --compartment-id "$compartment_ocid" --query 'data[*].{"name":"display-name","id":"id","cidr":"cidr-block","ipv6":"ipv6-cidr-blocks"}' --output json || true + ) >"$subnets_tmp" & + subnets_pid=$! + ( + oci_json network nsg list --compartment-id "$compartment_ocid" --query 'data[*].{"name":"display-name","id":"id"}' --output json || true + ) >"$nsgs_tmp" & + nsgs_pid=$! + fi + + wait "$subnets_pid" || true + wait "$nsgs_pid" || true + subnets_json="$(cat "$subnets_tmp")" + nsGs_json="$(cat "$nsgs_tmp")" + rm -f "$subnets_tmp" "$nsgs_tmp" + + if [[ -z "$subnets_json" ]]; then + subnets_json="[]" + echo "warning: failed to list subnets for compartment $compartment_ocid" >&2 + fi + if [[ -z "$nsGs_json" ]]; then + nsGs_json="[]" + echo "warning: failed to list NSGs for compartment $compartment_ocid" >&2 + fi +fi + +# Output consolidated JSON +result_json=$(python3 - < "$cache_file" +fi + +printf "%s\n" "$result_json" diff --git a/oci/oke/scripts/gva-menu.sh b/oci/oke/scripts/gva-menu.sh new file mode 100755 index 0000000..aef22f3 --- /dev/null +++ b/oci/oke/scripts/gva-menu.sh @@ -0,0 +1,802 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +GVA_DISCOVER_SCRIPT="$SCRIPT_DIR/gva-discover.sh" + +say() { printf "%s\n" "$*"; } + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + python3 - "$error_code" "$message" "$remediation" "$docs_url" <<'PY' >&2 +import json +import sys + +print(json.dumps({ + "error_code": sys.argv[1], + "message": sys.argv[2], + "remediation": sys.argv[3], + "docs_url": sys.argv[4], +})) +PY + exit "$exit_code" +} + +ask() { + local prompt="$1" var + read -r -p "$prompt" var + printf "%s" "$var" +} + +ask_required() { + local prompt="$1" + local value="" + while [[ -z "$value" ]]; do + value=$(ask "$prompt") + if [[ -z "$value" ]]; then + say "A value is required." + fi + done + printf "%s" "$value" +} + +read_lines_into_array() { + local array_name="$1" + local line="" + eval "$array_name=()" + while IFS= read -r line; do + eval "$array_name+=(\"\$line\")" + done +} + +select_from_list() { + local prompt="$1" + local allow_manual="$2" + shift 2 + local items=("$@") + local options=("${items[@]}") + local choice="" + if [[ "$allow_manual" == "yes" ]]; then + options+=("Manual entry") + fi + say "$prompt" + select choice in "${options[@]}"; do + if [[ "$allow_manual" == "yes" && "$choice" == "Manual entry" ]]; then + choice="" + break + elif [[ -n "$choice" ]]; then + break + else + say "Choose one of the listed options." + fi + done + printf "%s" "$choice" +} + +normalize_none() { + local v="$1" + local v_lower + v_lower="$(printf "%s" "$v" | tr '[:upper:]' '[:lower:]')" + case "$v_lower" in + ""|"0"|"none"|"no"|"n"|"skip") echo "" ;; + *) echo "$v" ;; + esac +} + +lower() { + printf "%s" "$1" | tr '[:upper:]' '[:lower:]' +} + +oci_node_pool_create_supports_gva() { + oci ce node-pool create --help 2>/dev/null | grep -Eq 'secondary-vnics|cni-type' +} + +verify_gva_cli_before_create() { + if ! command -v oci >/dev/null 2>&1; then + emit_error 1 "OCI_CLI_NOT_FOUND" \ + "OCI CLI is required to create a GVA node pool." \ + "Install OCI CLI or choose Print command only." + fi + + if ! oci_node_pool_create_supports_gva; then + emit_error 1 "GVA_CLI_FLAGS_UNAVAILABLE" \ + "OCI CLI does not expose --secondary-vnics or --cni-type for node-pool create." \ + "Upgrade OCI CLI to a version with GA GVA support, or choose Print command only." + fi +} + +line_field() { + local line="$1" + local field_no="$2" + printf "%s" "$line" | cut -d'|' -f"$field_no" | xargs +} + +ip_count_valid() { + local value="$1" + [[ "$value" =~ ^[0-9]+$ ]] || return 1 + (( value >= 1 && value <= 256 )) +} + +oci_available="yes" +if ! command -v oci >/dev/null 2>&1; then + oci_available="no" +fi + +say "GVA Node Pool Builder (OKE)" +say "This wizard collects fresh values for every new node pool creation." +say "It does not reuse prior drafts or previous mutable node-pool settings." +say "Saved JSON payloads are ignored for new node pool creation." +say "Answer the prompts to generate an OCI CLI command." + +cluster_name=$(ask_required "Cluster name (required, no default): ") +region=$(ask_required "Region for the selected cluster (e.g., us-ashburn-1): ") + +profile_name=$(ask "OCI CLI profile (optional): ") + +# Try to resolve cluster OCID from kubeconfig +kubeconfig_path="$HOME/.kube/config" +cluster_ocid="" +if [[ -f "$kubeconfig_path" ]]; then + cluster_ocid=$(python3 - "$kubeconfig_path" "$cluster_name" <<'PY' +import sys +from pathlib import Path +try: + import yaml +except Exception: + print("") + raise SystemExit(0) +path = Path(sys.argv[1]) +name = sys.argv[2] +try: + data = yaml.safe_load(path.read_text()) +except Exception: + print("") + raise SystemExit(0) + +contexts = data.get("contexts", []) or [] +users = {u.get("name"): u.get("user", {}) for u in (data.get("users", []) or [])} + +def pick_user(contexts, target): + exact = [] + exact_ci = [] + contains = [] + for c in contexts: + cname = c.get("name", "") + cuser = (c.get("context", {}) or {}).get("user") + if not cuser: + continue + if cname == target: + exact.append((len(cname), cname, cuser)) + elif cname.lower() == target.lower(): + exact_ci.append((len(cname), cname, cuser)) + elif target and target in cname: + contains.append((len(cname), cname, cuser)) + for bucket in (exact, exact_ci, contains): + if bucket: + bucket.sort() + return bucket[0][2] + return None + +match_user = pick_user(contexts, name) + +if not match_user: + print("") + raise SystemExit(0) + +exec_cfg = users.get(match_user, {}).get("exec", {}) +args = exec_cfg.get("args", []) if isinstance(exec_cfg, dict) else [] +try: + idx = args.index("--cluster-id") + print(args[idx + 1]) +except Exception: + print("") +PY + ) +fi + +if [[ -z "$cluster_ocid" ]]; then + cluster_ocid=$(ask_required "Cluster OCID (not found in kubeconfig): ") +else + say "Detected cluster OCID from kubeconfig." +fi + +compartment_ocid="" +discovery_json="" +if [[ "$oci_available" == "yes" ]]; then + discover_cmd=("$GVA_DISCOVER_SCRIPT" --cluster "$cluster_ocid" --timeout 10) + if [[ -n "$region" ]]; then + discover_cmd+=(--region "$region") + fi + if [[ -n "$profile_name" ]]; then + discover_cmd+=(--profile "$profile_name") + fi + discovery_json=$("${discover_cmd[@]}" 2>/dev/null || true) +fi + +cluster_k8s="" +subnet_lines=() +secondary_subnet_lines=() +nsg_lines=() +vcn_lines=() + +if [[ -n "$discovery_json" ]]; then + cluster_k8s=$(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +print(d.get('cluster',{}).get('kubernetes_version','')) +PY + ) + + comp_from_discovery=$(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +print(d.get('cluster',{}).get('compartment_id','')) +PY + ) + + region_from_discovery=$(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +print(d.get('cluster',{}).get('region','')) +PY + ) + + if [[ -z "$compartment_ocid" && -n "$comp_from_discovery" ]]; then + compartment_ocid="$comp_from_discovery" + fi + if [[ -z "$region" && -n "$region_from_discovery" ]]; then + region="$region_from_discovery" + fi + + read_lines_into_array subnet_lines < <(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +subnets = d.get('subnets', []) +if isinstance(subnets, dict): + subnets = subnets.get('data', []) +for s in subnets: + name=s.get('name') or '' + sid=s.get('id') or '' + cidr=s.get('cidr') or '' + ipv6=s.get('ipv6') or [] + ipv6_status='has-ipv6' if ipv6 else 'ipv4-only' + if name and sid: + print(f"{name} | {cidr} | {ipv6_status} | {sid}") +PY + ) + + read_lines_into_array nsg_lines < <(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +nsgs = d.get('nsgs', []) +if isinstance(nsgs, dict): + nsgs = nsgs.get('data', []) +for n in nsgs: + name=n.get('name') or '' + nid=n.get('id') or '' + if name and nid: + print(f"{name} | {nid}") +PY + ) +fi + +if [[ -z "$compartment_ocid" ]]; then + compartment_ocid=$(ask_required "Compartment OCID: ") +fi + +config_file="$HOME/.oci/config" +tenancy_ocid="" +if [[ -f "$config_file" ]]; then + tenancy_ocid=$(awk -F= '/^tenancy=/{print $2; exit}' "$config_file" | tr -d ' ') +fi + +ad_lines=() +if [[ "$oci_available" == "yes" ]]; then + ad_compartment="${tenancy_ocid:-$compartment_ocid}" + ad_json=$(oci iam availability-domain list --compartment-id "$ad_compartment" --region "$region" --query 'data[*].name' --output json 2>/dev/null || true) + if [[ -n "$ad_json" && "$ad_json" != "[]" ]]; then + read_lines_into_array ad_lines < <(python3 - <<'PY' +import json,sys +try: + data=json.loads(sys.stdin.read()) +except Exception: + data=[] +for item in data: + if item: + print(item) +PY + <<<"$ad_json") + fi +fi + +if [[ ${#ad_lines[@]} -gt 0 ]]; then + ad=$(select_from_list "Select Availability Domain:" "no" "${ad_lines[@]}") +else + ad=$(ask_required "Availability Domain (e.g., GrCh:US-ASHBURN-AD-1): ") +fi + +vcn_id="" +if [[ "$oci_available" == "yes" && -n "$compartment_ocid" ]]; then + vcn_json=$(oci network vcn list --compartment-id "$compartment_ocid" --region "$region" --query 'data[*].{name:"display-name",id:id,cidr:"cidr-block"}' --output json 2>/dev/null || true) + if [[ -n "$vcn_json" && "$vcn_json" != "[]" ]]; then + read_lines_into_array vcn_lines < <(python3 - "$vcn_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +for v in d if isinstance(d, list) else []: + name=v.get('name') or '' + vid=v.get('id') or '' + cidr=v.get('cidr') or '' + if name and vid: + print(f"{name} | {cidr} | {vid}") +PY + ) + fi +fi + +if [[ ${#vcn_lines[@]} -gt 0 ]]; then + selection=$(select_from_list "Select VCN:" "yes" "${vcn_lines[@]}") + if [[ -n "$selection" ]]; then + vcn_id=$(printf "%s" "$selection" | cut -d'|' -f3 | xargs) + fi +fi +if [[ -z "$vcn_id" ]]; then + vcn_id=$(ask_required "VCN OCID: ") +fi + +if [[ -n "$vcn_id" ]]; then + subnet_lines=() + secondary_subnet_lines=() + subnet_json=$(oci network subnet list --compartment-id "$compartment_ocid" --vcn-id "$vcn_id" --region "$region" --query 'data[*].{"name":"display-name","id":"id","cidr":"cidr-block","ipv4Cidrs":"ipv4-cidr-blocks","ipv6":"ipv6-cidr-blocks"}' --output json 2>/dev/null || true) + if [[ -n "$subnet_json" && "$subnet_json" != "[]" ]]; then + read_lines_into_array subnet_lines < <(python3 - "$subnet_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +for s in d.get('data', d) if isinstance(d, dict) else d: + name=s.get('name') or s.get('display-name') or '' + sid=s.get('id') or '' + cidr=s.get('cidr') or s.get('cidr-block') or '' + ipv6=s.get('ipv6') or s.get('ipv6-cidr-blocks') or [] + ipv6_status='has-ipv6' if ipv6 else 'ipv4-only' + if name and sid: + print(f"{name} | {cidr} | {ipv6_status} | {sid}") +PY + ) + read_lines_into_array secondary_subnet_lines < <(python3 - "$subnet_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +for s in d.get('data', d) if isinstance(d, dict) else d: + name=s.get('name') or s.get('display-name') or '' + sid=s.get('id') or '' + cidr=s.get('cidr') or s.get('cidr-block') or '' + ipv6=s.get('ipv6') or s.get('ipv6-cidr-block') + ipv4_cidrs=s.get('ipv4Cidrs') or s.get('ipv4-cidr-blocks') or [] + cidr_count=len(ipv4_cidrs) if isinstance(ipv4_cidrs, list) else 0 + if name and sid and not ipv6 and cidr_count > 1: + print(f"{name} | {cidr} | {sid}") +PY + ) + fi +fi + +primary_subnet="" +if [[ ${#subnet_lines[@]} -gt 0 ]]; then + selection=$(select_from_list "Select primary subnet (node placement):" "yes" "${subnet_lines[@]}") + if [[ -n "$selection" ]]; then + primary_subnet=$(line_field "$selection" 4) + fi +fi +if [[ -z "$primary_subnet" ]]; then + primary_subnet=$(ask_required "Primary subnet OCID (node placement subnet): ") +fi + +node_pool_name=$(ask_required "Node pool name: ") + +k8s_version="" +if [[ -n "$cluster_k8s" ]]; then + k8s_version=$cluster_k8s + say "Using cluster Kubernetes version: $k8s_version" +fi +if [[ -z "$k8s_version" ]]; then + k8s_version=$(ask_required "Kubernetes version (e.g., v1.34.1): ") +fi + +shape=$(ask_required "Node shape (e.g., VM.Standard.E5.Flex): ") + +type_is_flex="no" +case "$shape" in + *.Flex) type_is_flex="yes" ;; + *) type_is_flex="no" ;; + esac + +ocpus="" +mem_gb="" +if [[ "$type_is_flex" == "yes" ]]; then + ocpus=$(ask_required "OCPUs per node: ") + mem_gb=$(ask_required "Memory GB per node: ") +fi + +node_count=$(ask_required "Node count: ") + +# Image selection: list images for k8s version and prompt +image_ocid="" +if [[ "$oci_available" == "yes" && -n "$k8s_version" ]]; then + img_json=$(oci ce node-pool-options get --node-pool-option-id "$cluster_ocid" --region "$region" --query 'data.sources[*].{"image":"image-id","name":"source-name"}' --output json 2>/dev/null || true) + if [[ -n "$img_json" && "$img_json" != "[]" ]]; then + read_lines_into_array img_lines < <(python3 - "$img_json" "$k8s_version" <<'PY' +import json,sys,re +try: + d=json.loads(sys.argv[1]) +except Exception: + d=[] +pattern=re.compile(rf'OKE-{re.escape(sys.argv[2])}') +items=[x for x in d if pattern.search(x.get('name',''))] +for x in items: + print(f"{x.get('name')} | {x.get('image')}") +PY + ) + if [[ ${#img_lines[@]} -gt 0 ]]; then + selection=$(select_from_list "Select OKE image for $k8s_version:" "yes" "${img_lines[@]}") + if [[ -n "$selection" ]]; then + image_ocid=$(printf "%s" "$selection" | cut -d'|' -f2 | xargs) + fi + fi + fi +fi + +if [[ -z "$image_ocid" ]]; then + image_ocid=$(ask_required "Image OCID: ") +fi + +say "" +say "CNI must be OCI_VCN_IP_NATIVE for GVA." +say "1) Yes" +say "2) No" +while true; do + say "Choose 1 or 2:" + IFS= read -r cni_ok + case "$cni_ok" in + 1|Yes|yes|Y|y) break ;; + 2|No|no|N|n) + emit_error 1 "GVA_REQUIRES_VCN_NATIVE_CNI" \ + "GVA is unsupported without OCI_VCN_IP_NATIVE." \ + "Use an OKE cluster with VCN-native pod networking before creating a GVA node pool." + ;; + *) say "Choose one of the listed options." ;; + esac +done + +# Collect VNIC profiles +profiles=() +profile_summaries=() +application_resources=() +while true; do + say "" + say "Add a secondary VNIC profile (GVA tier)" + while true; do + app_res=$(ask " applicationResource (label): ") + if [[ -z "$app_res" ]]; then + say " applicationResource is required." + continue + fi + duplicate="no" + if [[ ${#application_resources[@]} -gt 0 ]]; then + for existing in "${application_resources[@]}"; do + if [[ "$existing" == "$app_res" ]]; then + duplicate="yes" + break + fi + done + fi + if [[ "$duplicate" == "yes" ]]; then + say " applicationResource values must be unique across profiles." + continue + fi + break + done + + subnet_id="" + subnet_summary="" + while [[ -z "$subnet_id" ]]; do + if [[ ${#secondary_subnet_lines[@]} -gt 0 ]]; then + selection=$(select_from_list " Select subnet for this profile (IPv4-only, >1 IPv4 CIDR block):" "yes" "${secondary_subnet_lines[@]}") + if [[ -n "$selection" ]]; then + subnet_id=$(line_field "$selection" 3) + subnet_summary="$(line_field "$selection" 1) ($(line_field "$selection" 2))" + fi + fi + if [[ -z "$subnet_id" ]]; then + subnet_id=$(ask " subnetId OCID (must be IPv4-only and have >1 IPv4 CIDR block): ") + if [[ -n "$subnet_id" ]]; then + say " Manual subnet entry cannot be OCI-validated. Ensure the subnet is IPv4-only and has more than one IPv4 CIDR block." + subnet_summary="$subnet_id" + fi + fi + done + + while true; do + ip_count=$(ask " ipCount (1-256): ") + if ip_count_valid "$ip_count"; then + break + fi + say " ipCount must be an integer between 1 and 256." + done + + nsg_ids="" + if [[ ${#nsg_lines[@]} -gt 0 ]]; then + selection=$(select_from_list " Select NSG (or Manual entry for none/custom):" "yes" "${nsg_lines[@]}") + if [[ -n "$selection" ]]; then + nsg_ids=$(printf "%s" "$selection" | cut -d'|' -f2 | xargs) + fi + nsg_ids=$(normalize_none "$nsg_ids") + else + nsg_ids=$(normalize_none "$(ask " nsgIds (comma-separated OCIDs, optional): ")") + fi + + display_name=$(ask " displayName (optional): ") + + profile_json="$( + APP_RES="$app_res" \ + SUBNET_ID="$subnet_id" \ + IP_COUNT="$ip_count" \ + NSG_IDS="$nsg_ids" \ + DISPLAY_NAME="$display_name" \ + python3 - <<'PY' +import json +import os +import sys + +app_res = os.environ.get("APP_RES", "") +subnet_id = os.environ.get("SUBNET_ID", "") +display_name = os.environ.get("DISPLAY_NAME", "") +nsg_ids_raw = os.environ.get("NSG_IDS", "") +ip_count_raw = os.environ.get("IP_COUNT", "") + +try: + ip_count = int(ip_count_raw) +except ValueError: + print("Invalid ipCount. Expected an integer.", file=sys.stderr) + sys.exit(2) + +nsg_ids = [x.strip() for x in nsg_ids_raw.split(",") if x.strip()] if nsg_ids_raw else None +display_field = display_name if display_name else None + +obj = { + "createVnicDetails": { + "ipCount": ip_count, + "applicationResources": [app_res], + "assignPublicIp": False, + "displayName": display_field, + "nsgIds": nsg_ids, + "subnetId": subnet_id, + "skipSourceDestCheck": False, + }, + "displayName": display_field, +} + +print(json.dumps(obj, separators=(",", ":"))) +PY + )" + + profiles+=("$profile_json") + application_resources+=("$app_res") + profile_summaries+=("$app_res | ipCount=$ip_count | subnet=$subnet_summary") + + say "" + select more in "Add another profile" "Finish"; do + case "$more" in + "Add another profile") break ;; + "Finish") more=""; break 2 ;; + esac + done + done + +secondary_vnics_json="[" +for i in "${!profiles[@]}"; do + secondary_vnics_json+="${profiles[$i]}" + if [[ $i -lt $((${#profiles[@]}-1)) ]]; then + secondary_vnics_json+=","; + fi + done +secondary_vnics_json+="]" + +# Build node shape config +shape_config="{}" +if [[ "$type_is_flex" == "yes" ]]; then + shape_config="{\"ocpus\":$ocpus,\"memoryInGBs\":$mem_gb}" +fi + +optional_args=() +add_optional=$(ask "Add optional node-pool parameters? (y/N): ") +if [[ "$(lower "$add_optional")" == "y" || "$(lower "$add_optional")" == "yes" ]]; then + defined_tags=$(ask " --defined-tags (JSON or file://path, optional): ") + if [[ -n "$defined_tags" ]]; then + optional_args+=("--defined-tags" "$defined_tags") + fi + + freeform_tags=$(ask " --freeform-tags (JSON or file://path, optional): ") + if [[ -n "$freeform_tags" ]]; then + optional_args+=("--freeform-tags" "$freeform_tags") + fi + + initial_labels=$(ask " --initial-node-labels (JSON or file://path, optional): ") + if [[ -n "$initial_labels" ]]; then + optional_args+=("--initial-node-labels" "$initial_labels") + fi + + max_pods=$(ask " --max-pods-per-node (integer, optional): ") + if [[ -n "$max_pods" ]]; then + optional_args+=("--max-pods-per-node" "$max_pods") + fi + + pv_encrypt=$(ask " --is-pv-encryption-in-transit-enabled? (y/N): ") + if [[ "$(lower "$pv_encrypt")" == "y" || "$(lower "$pv_encrypt")" == "yes" ]]; then + optional_args+=("--is-pv-encryption-in-transit-enabled" "true") + fi + + kms_key=$(ask " --kms-key-id (OCID, optional): ") + if [[ -n "$kms_key" ]]; then + optional_args+=("--kms-key-id" "$kms_key") + fi + + boot_size=$(ask " --node-boot-volume-size-in-gbs (integer, optional): ") + if [[ -n "$boot_size" ]]; then + optional_args+=("--node-boot-volume-size-in-gbs" "$boot_size") + fi + + node_metadata=$(ask " --node-metadata (JSON or file://path, optional): ") + if [[ -n "$node_metadata" ]]; then + optional_args+=("--node-metadata" "$node_metadata") + fi + + ssh_key=$(ask " --ssh-public-key (string or file://path, optional): ") + if [[ -n "$ssh_key" ]]; then + optional_args+=("--ssh-public-key" "$ssh_key") + fi +fi + +say "" +say "Summary:" +say "- Cluster: $cluster_name" +say "- Cluster OCID: $cluster_ocid" +say "- Region: $region" +say "- Compartment: $compartment_ocid" +say "- AD: $ad" +say "- Node pool: $node_pool_name" +say "- Shape: $shape" +say "- Node count: $node_count" +say "- K8s version: $k8s_version" +say "- Primary subnet: $primary_subnet" +for p in "${profile_summaries[@]}"; do + say "- VNIC: $p" + done + +confirm_generate=$(ask "Generate a create command with these values? (y/N): ") +confirm_generate=$(lower "$confirm_generate") +if [[ "$confirm_generate" != "y" && "$confirm_generate" != "yes" ]]; then + say "Aborted before command generation." + exit 0 +fi + +say "" +say "Choose next action:" +say "1) Run command now" +say "2) Print command only" +say "3) Exit without output" +choice=$(ask "Select (1/2/3): ") + +if [[ "$choice" == "1" ]]; then + run_now="yes" +elif [[ "$choice" == "2" ]]; then + run_now="no" +elif [[ "$choice" == "3" ]]; then + say "Aborted." + exit 0 +else + say "Unknown choice; printing command only." + run_now="no" +fi + +say "" +say "Generated OCI CLI command:" +cmd=(oci) +if [[ -n "$profile_name" ]]; then + cmd+=(--profile "$profile_name") +fi +cmd+=( + --region "$region" + ce node-pool create + --compartment-id "$compartment_ocid" + --cluster-id "$cluster_ocid" + --name "$node_pool_name" + --kubernetes-version "$k8s_version" + --node-shape "$shape" + --node-shape-config "$shape_config" + --size "$node_count" + --cni-type OCI_VCN_IP_NATIVE + --placement-configs "[{\"availabilityDomain\":\"$ad\",\"subnetId\":\"$primary_subnet\"}]" + --node-source-details "{\"sourceType\":\"IMAGE\",\"imageId\":\"$image_ocid\"}" + --secondary-vnics "$secondary_vnics_json" +) +if [[ ${#optional_args[@]} -gt 0 ]]; then + cmd+=("${optional_args[@]}") +fi +printf '%q ' "${cmd[@]}" +printf '\n' + +if [[ "$run_now" == "yes" ]]; then + final_confirmation=$(ask "Type CREATE to run the node-pool create command now: ") + if [[ "$final_confirmation" != "CREATE" ]]; then + say "Run cancelled. Command was not executed." + exit 0 + fi + say "" + say "Verifying OCI CLI support for GVA node-pool flags..." + verify_gva_cli_before_create + say "Running command..." + "${cmd[@]}" +fi + +say "" +say "Sample test Deployment (replace ResourceName/image):" +cat <<'YAML' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gva-test +spec: + replicas: 1 + selector: + matchLabels: + app: gva-test + template: + metadata: + labels: + app: gva-test + spec: + tolerations: + - key: "oci.oraclecloud.com/application-resource-only" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: app + image: + resources: + requests: + oke-application-resource.oci.oraclecloud.com/ResourceName: "1" + limits: + oke-application-resource.oci.oraclecloud.com/ResourceName: "1" +YAML + +say "" +say "Next steps:" +say "1) Verify resources on a node: kubectl describe node " +say "2) Apply the test Deployment (with your chosen ResourceName)." diff --git a/oci/oke/scripts/node-doctor-run.sh b/oci/oke/scripts/node-doctor-run.sh new file mode 100755 index 0000000..29eb085 --- /dev/null +++ b/oci/oke/scripts/node-doctor-run.sh @@ -0,0 +1,226 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Run OKE node doctor through kubectl debug and emit normalized JSON. +# +# Usage: +# ./scripts/node-doctor-run.sh --node --image [--namespace ] [--cleanup true|false] + +node_name="" +image_name="" +namespace="default" +cleanup="true" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." \ + "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --node) + require_value "$1" "${2:-}" + node_name="$2" + shift 2 + ;; + --image) + require_value "$1" "${2:-}" + image_name="$2" + shift 2 + ;; + --namespace) + require_value "$1" "${2:-}" + namespace="$2" + shift 2 + ;; + --cleanup) + require_value "$1" "${2:-}" + cleanup="$2" + shift 2 + ;; + -h|--help) + cat <<'EOF' +usage: node-doctor-run.sh --node --image [--namespace ] [--cleanup true|false] +EOF + exit 0 + ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." + ;; + esac +done + +if [[ -z "$node_name" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --node." "Provide --node ." +fi + +if [[ -z "$image_name" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --image." "Provide --image ." +fi + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +debug_cmd=(kubectl -n "$namespace" debug "node/${node_name}" --image="${image_name}" -- chroot /host sudo /usr/local/bin/node-doctor.sh --check) + +set +e +raw_out="$("${debug_cmd[@]}" 2>&1)" +cmd_rc=$? +set -e + +pod_name="$(printf "%s\n" "$raw_out" | sed -n 's/^Creating debugging pod \([^ ]*\) .*/\1/p' | head -n 1)" + +# When kubectl debug cannot attach, execution output may only exist in pod logs. +if [[ -n "$pod_name" ]]; then + # Wait for the debug pod to finish so node-doctor output is complete. + for _ in $(seq 1 120); do + set +e + pod_phase="$(kubectl -n "$namespace" get pod "$pod_name" -o jsonpath='{.status.phase}' 2>/dev/null)" + phase_rc=$? + set -e + if [[ $phase_rc -eq 0 && ( "$pod_phase" == "Succeeded" || "$pod_phase" == "Failed" ) ]]; then + break + fi + sleep 1 + done + + set +e + pod_logs="$(kubectl -n "$namespace" logs "$pod_name" --all-containers=true 2>&1)" + logs_rc=$? + set -e + if [[ $logs_rc -eq 0 && -n "$pod_logs" ]]; then + raw_out="${raw_out}"$'\n'"${pod_logs}" + fi +fi + +cleanup_attempted="false" +cleanup_message="" + +cleanup_lc="$(printf "%s" "$cleanup" | tr '[:upper:]' '[:lower:]')" +if [[ "$cleanup_lc" == "true" && -n "$pod_name" ]]; then + cleanup_attempted="true" + set +e + cleanup_out="$(kubectl -n "$namespace" delete pod "$pod_name" --ignore-not-found 2>&1)" + cleanup_rc=$? + set -e + if [[ $cleanup_rc -eq 0 ]]; then + cleanup_message="$cleanup_out" + else + cleanup_message="cleanup failed: $cleanup_out" + fi +fi + +RAW_OUT="$raw_out" python3 - "$node_name" "$image_name" "$namespace" "$cmd_rc" "$pod_name" "$cleanup_attempted" "$cleanup_message" <<'PY' +import json +import os +import re +import sys + +node = sys.argv[1] +image = sys.argv[2] +namespace = sys.argv[3] +cmd_rc = int(sys.argv[4]) +pod_name = sys.argv[5] +cleanup_attempted = sys.argv[6].lower() == "true" +cleanup_message = sys.argv[7] +raw = os.environ.get("RAW_OUT", "") + +# Strip ANSI escapes so snippets remain readable. +raw = re.sub(r"\x1B\[[0-9;]*[A-Za-z]", "", raw) +lines = raw.splitlines() + +pass_count = sum(1 for l in lines if l.startswith("PASS ")) +fail_count = sum(1 for l in lines if l.startswith("FAIL ")) +warn_count = sum(1 for l in lines if l.startswith("WARN ")) +skip_count = sum(1 for l in lines if l.startswith("SKIP ")) + +checks_line = "" +signals_line = "" +for l in lines: + if "checks passed" in l: + checks_line = l.strip() + if "Signal(s) generated" in l: + signals_line = l.strip() + +has_check_lines = any( + l.startswith(("PASS ", "FAIL ", "WARN ", "SKIP ")) + for l in lines +) +executed = ( + "Running node doctor..." in raw + or "NODE DOCTOR REPORT" in raw + or checks_line != "" + or has_check_lines +) + +if not executed: + result = "unknown" + fallback_reason = "node doctor checks did not execute or produced no parsable output" +elif cmd_rc != 0: + result = "unknown" + fallback_reason = f"node doctor command exited with code {cmd_rc}" +elif fail_count > 0: + result = "fail" + fallback_reason = "" +else: + result = "pass" + fallback_reason = "" + +findings = [] +if checks_line: + findings.append(checks_line) +if signals_line: + findings.append(signals_line) + +signal_pat = re.compile(r"^Signal \d+:\s+(.+)$") +for l in lines: + m = signal_pat.match(l.strip()) + if m: + findings.append(f"Signal: {m.group(1)}") + +snippet_lines = lines[-120:] +raw_snippet = "\n".join(snippet_lines) + +out = { + "node_doctor_attempted": True, + "node_doctor_executed": executed, + "node_doctor_node": node, + "node_doctor_image": image, + "node_doctor_namespace": namespace, + "node_doctor_debug_pod": pod_name, + "node_doctor_result": result, + "node_doctor_command_rc": cmd_rc, + "node_doctor_findings": findings, + "node_doctor_counts": { + "pass": pass_count, + "fail": fail_count, + "warn": warn_count, + "skip": skip_count + }, + "node_doctor_raw_snippet": raw_snippet, + "node_doctor_fallback_reason": fallback_reason, + "cleanup_attempted": cleanup_attempted, + "cleanup_message": cleanup_message +} + +print(json.dumps(out, indent=2)) +PY + +# Runtime failures are represented in JSON fields above. Exit 0 so callers can +# always parse structured output and decide fallback behavior. +exit 0 diff --git a/oci/oke/scripts/oke-addon-health.sh b/oci/oke/scripts/oke-addon-health.sh new file mode 100755 index 0000000..67d9e55 --- /dev/null +++ b/oci/oke/scripts/oke-addon-health.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect OKE add-on health signals from kube-system. +# +# Usage: +# bash scripts/oke-addon-health.sh [--namespace kube-system] + +namespace="kube-system" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace) + require_value "$1" "${2:-}" + namespace="$2" + shift 2 + ;; + -h|--help) + echo "usage: oke-addon-health.sh [--namespace kube-system]" + exit 0 + ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." + ;; + esac +done + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1" + shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json +import os +import sys + +record = { + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], +} +with open(sys.argv[1], "a") as f: + f.write(json.dumps(record) + "\n") +PY +} + +run_check "kube-system pods" kubectl -n "$namespace" get pods -o wide +run_check "kube-system deployments" kubectl -n "$namespace" get deploy -o wide +run_check "kube-system daemonsets" kubectl -n "$namespace" get ds -o wide +run_check "CoreDNS deployment" kubectl -n "$namespace" get deploy coredns -o wide +run_check "recent kube-system warnings" kubectl -n "$namespace" get events --field-selector type=Warning --sort-by=.lastTimestamp + +python3 - "$records" "$namespace" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace = sys.argv[2] +anomaly_re = re.compile( + r"(CrashLoopBackOff|ImagePullBackOff|ErrImagePull|OOMKilled|NotReady|Pending|Error|Failed|0/[1-9])", + re.I, +) + +findings = [] +anomalies = [] +snippets = [] +for item in records: + output = item["output"].strip() + if item["rc"] == 0: + findings.append(f"{item['name']} collected") + else: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if output: + snippets.append(f"$ {item['cmd']}\n{output[-800:]}") + for line in output.splitlines(): + if anomaly_re.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") + +print(json.dumps({ + "domain": "OKE Add-ons Health", + "namespace": namespace, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-8:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-autoscaler-check.sh b/oci/oke/scripts/oke-autoscaler-check.sh new file mode 100755 index 0000000..e29a364 --- /dev/null +++ b/oci/oke/scripts/oke-autoscaler-check.sh @@ -0,0 +1,148 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect cluster-autoscaler, Pending pod, and optional OKE node-pool evidence. +# +# Usage: +# bash scripts/oke-autoscaler-check.sh --namespace [--deployment ] [--cluster-id ] [--compartment-id ] [--region ] + +namespace="" +deployment="" +cluster_id="" +compartment_id="" +region="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) + require_value "$1" "${2:-}" + namespace="$2"; shift 2 ;; + --deployment) + require_value "$1" "${2:-}" + deployment="$2"; shift 2 ;; + --cluster-id) + require_value "$1" "${2:-}" + cluster_id="$2"; shift 2 ;; + --compartment-id) + require_value "$1" "${2:-}" + compartment_id="$2"; shift 2 ;; + --region) + require_value "$1" "${2:-}" + region="$2"; shift 2 ;; + -h|--help) + echo "usage: oke-autoscaler-check.sh --namespace [--deployment ] [--cluster-id ] [--compartment-id ] [--region ]" + exit 0 ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done + +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --namespace." "Provide --namespace ." +fi + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1" + shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json +import os +import sys + +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], + }) + "\n") +PY +} + +run_check "pending pods" kubectl -n "$namespace" get pods --field-selector=status.phase=Pending -o wide +run_check "scheduling warning events" kubectl -n "$namespace" get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp +run_check "cluster-autoscaler deployment" kubectl -n kube-system get deploy cluster-autoscaler -o wide +run_check "cluster-autoscaler logs" kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200 + +if [[ -n "$deployment" ]]; then + run_check "target deployment" kubectl -n "$namespace" describe deployment "$deployment" + run_check "target deployment hpa" kubectl -n "$namespace" get hpa --selector "app=$deployment" -o wide +fi + +if [[ -n "$cluster_id" && -n "$compartment_id" ]] && command -v oci >/dev/null 2>&1; then + oci_cmd=(oci) + if [[ -n "$region" ]]; then + oci_cmd+=(--region "$region") + fi + run_check "OKE node pools" "${oci_cmd[@]}" ce node-pool list --compartment-id "$compartment_id" --cluster-id "$cluster_id" --all --output json +fi + +python3 - "$records" "$namespace" "$deployment" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, deployment = sys.argv[2], sys.argv[3] +anomaly_re = re.compile( + r"(FailedScheduling|Insufficient|max node group size reached|scale.?up|NotTriggerScaleUp|quota|limit|no node group|didn.t trigger|unremovable|Pending|Error|Failed)", + re.I, +) + +findings = [] +anomalies = [] +snippets = [] +for item in records: + output = item["output"].strip() + if item["rc"] == 0: + findings.append(f"{item['name']} collected") + else: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if output: + snippets.append(f"$ {item['cmd']}\n{output[-800:]}") + for line in output.splitlines(): + if anomaly_re.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") + +print(json.dumps({ + "domain": "Cluster Autoscaler / Node Pool Scaling", + "namespace": namespace, + "deployment": deployment, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-discover.sh b/oci/oke/scripts/oke-discover.sh new file mode 100755 index 0000000..e3eccfc --- /dev/null +++ b/oci/oke/scripts/oke-discover.sh @@ -0,0 +1,312 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Discover OKE cluster context for troubleshooting. +# Usage: +# ./scripts/oke-discover.sh --cluster [--region ] [--profile ] [--timeout ] [--kubeconfig ] [--deployment ] + +cluster_ref="" +region_arg="" +profile_arg="" +timeout_arg="" +kubeconfig_arg="" +deployment_name="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." \ + "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --cluster) + require_value "$1" "${2:-}" + cluster_ref="$2"; shift 2 ;; + --region) + require_value "$1" "${2:-}" + region_arg="$2"; shift 2 ;; + --profile) + require_value "$1" "${2:-}" + profile_arg="$2"; shift 2 ;; + --timeout) + require_value "$1" "${2:-}" + timeout_arg="$2"; shift 2 ;; + --kubeconfig) + require_value "$1" "${2:-}" + kubeconfig_arg="$2"; shift 2 ;; + --deployment) + require_value "$1" "${2:-}" + deployment_name="$2"; shift 2 ;; + -h|--help) + echo "usage: $0 --cluster [--region ] [--profile ] [--timeout ] [--kubeconfig ] [--deployment ]" >&2 + exit 0 ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." \ + "Run with --help to view usage." ;; + esac +done + +if [[ -z "$cluster_ref" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --cluster." \ + "Provide --cluster ." +fi + +if ! command -v oci >/dev/null 2>&1; then + emit_error 2 "OCI_CLI_NOT_INSTALLED" "OCI CLI not found on PATH." \ + "Install OCI CLI and retry." \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm" +fi + +# Read defaults from OCI config if present +config_file="$HOME/.oci/config" +config_region="" +if [[ -f "$config_file" ]]; then + config_region=$(awk -F= '/^region=/{print $2; exit}' "$config_file" | tr -d ' ') +fi + +have_timeout="no" +if command -v timeout >/dev/null 2>&1; then + have_timeout="yes" +fi + +use_py_timeout="no" +declare -a timeout_prefix=() +if [[ -n "$timeout_arg" ]]; then + if [[ "$have_timeout" == "yes" ]]; then + timeout_prefix=(timeout "$timeout_arg") + else + use_py_timeout="yes" + fi +fi + +declare -a profile_args=() +if [[ -n "$profile_arg" ]]; then + profile_args=(--profile "$profile_arg") +elif [[ -n "${OCI_CLI_PROFILE:-}" ]]; then + profile_args=(--profile "$OCI_CLI_PROFILE") +fi + +oci_json() { + local out err rc + local -a cmd + err="$(mktemp)" + cmd=(oci --region "$region") + if [[ ${#profile_args[@]} -gt 0 ]]; then + cmd+=("${profile_args[@]}") + fi + cmd+=("$@") + if [[ ${#timeout_prefix[@]} -gt 0 ]]; then + cmd=("${timeout_prefix[@]}" "${cmd[@]}") + fi + if [[ "$use_py_timeout" == "yes" ]]; then + set +e + out="$(python3 - "$timeout_arg" "$err" "${cmd[@]}" <<'PY' +import subprocess, sys +timeout = float(sys.argv[1]) +err_path = sys.argv[2] +cmd = sys.argv[3:] +try: + p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout, text=True) + if p.stderr: + with open(err_path, "w") as f: + f.write(p.stderr) + sys.stdout.write(p.stdout or "") + sys.exit(p.returncode) +except subprocess.TimeoutExpired: + with open(err_path, "w") as f: + f.write("Command timed out after %ss\n" % timeout) + sys.exit(124) +PY +)" + rc=$? + set -e + else + set +e + out="$("${cmd[@]}" 2>"$err")" + rc=$? + set -e + fi + if [[ $rc -ne 0 ]]; then + echo "error: oci $* failed (exit $rc)" >&2 + sed -e 's/^/oci stderr: /' "$err" >&2 + rm -f "$err" + return "$rc" + fi + rm -f "$err" + printf "%s" "$out" +} + +json_get_field() { + local field="$1" + python3 -c ' +import json +import sys + +field = sys.argv[1] +try: + data = json.load(sys.stdin) +except Exception: + data = {} + +if isinstance(data, dict): + value = data.get(field, "") + if value is None: + value = "" + print(value) +else: + print("") +' "$field" +} + +cluster_ocid="" +cluster_name="" +cluster_k8s="" +compartment_ocid="" + +if [[ "$cluster_ref" == ocid1.* ]]; then + cluster_ocid="$cluster_ref" +else + # Resolve from kubeconfig + kubeconfig_path="${kubeconfig_arg:-$HOME/.kube/config}" + if [[ -f "$kubeconfig_path" ]]; then + cluster_ocid=$(python3 - "$kubeconfig_path" "$cluster_ref" <<'PY' +import sys +from pathlib import Path +try: + import yaml +except Exception: + print("") + raise SystemExit(0) +path = Path(sys.argv[1]) +name = sys.argv[2] +try: + data = yaml.safe_load(path.read_text()) +except Exception: + print("") + raise SystemExit(0) + +contexts = data.get("contexts", []) or [] +users = {u.get("name"): u.get("user", {}) for u in (data.get("users", []) or [])} + +def pick_user(contexts, target): + exact = [] + exact_ci = [] + contains = [] + for c in contexts: + cname = c.get("name", "") + cuser = (c.get("context", {}) or {}).get("user") + if not cuser: + continue + if cname == target: + exact.append((len(cname), cname, cuser)) + elif cname.lower() == target.lower(): + exact_ci.append((len(cname), cname, cuser)) + elif target and target in cname: + contains.append((len(cname), cname, cuser)) + for bucket in (exact, exact_ci, contains): + if bucket: + bucket.sort() + return bucket[0][2] + return None + +match_user = pick_user(contexts, name) + +if not match_user: + print("") + raise SystemExit(0) + +exec_cfg = users.get(match_user, {}).get("exec", {}) +args = exec_cfg.get("args", []) if isinstance(exec_cfg, dict) else [] +try: + idx = args.index("--cluster-id") + print(args[idx + 1]) +except Exception: + print("") +PY + ) + fi +fi + +if [[ -z "$cluster_ocid" ]]; then + emit_error 1 "CLUSTER_OCID_NOT_RESOLVED" \ + "Could not resolve cluster OCID from kubeconfig." \ + "Provide cluster OCID directly with --cluster ." +fi + +region_from_cluster_ocid="$(python3 - "$cluster_ocid" <<'PY' +import re +import sys + +ocid = sys.argv[1].strip() +match = re.match(r"^ocid1\.cluster\.oc1\.([a-z0-9-]+)\..+$", ocid) +print(match.group(1) if match else "") +PY +)" + +region="${region_arg:-${region_from_cluster_ocid:-$config_region}}" +if [[ -z "$region" ]]; then + emit_error 2 "REGION_NOT_PROVIDED" \ + "Region not provided and could not be inferred from cluster OCID or OCI config." \ + "Provide --region or set region in ~/.oci/config." +fi + +# Fetch cluster details if possible +cluster_json=$(oci_json ce cluster get --cluster-id "$cluster_ocid" --query 'data.{name:"name",k8s:"kubernetes-version",compartment:"compartment-id"}' --output json || true) +if [[ -n "$cluster_json" && "$cluster_json" != "{}" ]]; then + cluster_name=$(printf '%s' "$cluster_json" | json_get_field name) + cluster_k8s=$(printf '%s' "$cluster_json" | json_get_field k8s) + compartment_ocid=$(printf '%s' "$cluster_json" | json_get_field compartment) +else + echo "warning: failed to fetch cluster details; returning partial context" >&2 + cluster_name="$cluster_ref" +fi + +# Optional: try to resolve deployment namespace if kubectl is available +deployment_namespace="" +deployment_namespaces="" +if [[ -n "$deployment_name" ]]; then + if command -v kubectl >/dev/null 2>&1; then + deployment_namespaces=$(kubectl get deploy -A --no-headers 2>/dev/null | awk -v d="$deployment_name" '$2==d {print $1}' | paste -sd "," -) + if [[ -n "$deployment_namespaces" ]]; then + if [[ "$deployment_namespaces" == *,* ]]; then + deployment_namespace="" + else + deployment_namespace="$deployment_namespaces" + fi + fi + else + echo "warning: kubectl not found; cannot resolve deployment namespace" >&2 + fi +fi + +python3 - < [--service ] [--pod ] [--lookup ] + +namespace="" +service="" +pod="" +lookup="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) + require_value "$1" "${2:-}" + namespace="$2"; shift 2 ;; + --service) + require_value "$1" "${2:-}" + service="$2"; shift 2 ;; + --pod) + require_value "$1" "${2:-}" + pod="$2"; shift 2 ;; + --lookup) + require_value "$1" "${2:-}" + lookup="$2"; shift 2 ;; + -h|--help) + echo "usage: oke-dns-check.sh --namespace [--service ] [--pod ] [--lookup ]" + exit 0 ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done + +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --namespace." "Provide --namespace ." +fi + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1" + shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json +import os +import sys + +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], + }) + "\n") +PY +} + +run_check "CoreDNS pods" kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide +run_check "CoreDNS deployment" kubectl -n kube-system get deploy coredns -o wide +run_check "CoreDNS config" kubectl -n kube-system get configmap coredns -o yaml +run_check "CoreDNS logs" kubectl -n kube-system logs deployment/coredns --tail=200 + +if [[ -n "$service" ]]; then + run_check "target service" kubectl -n "$namespace" get svc "$service" -o yaml + run_check "target endpoints" kubectl -n "$namespace" get endpoints "$service" -o yaml + run_check "target endpoint slices" kubectl -n "$namespace" get endpointslices -l "kubernetes.io/service-name=$service" -o yaml +fi + +if [[ -n "$pod" && -n "$lookup" ]]; then + run_check "pod DNS lookup" kubectl -n "$namespace" exec "$pod" -- nslookup "$lookup" +fi + +python3 - "$records" "$namespace" "$service" "$pod" "$lookup" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, service, pod, lookup = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +anomaly_re = re.compile( + r"(SERVFAIL|NXDOMAIN|timeout|no such host|connection refused|no endpoints|CrashLoopBackOff|NotReady|Error|Failed)", + re.I, +) + +findings = [] +anomalies = [] +snippets = [] +for item in records: + output = item["output"].strip() + if item["rc"] == 0: + findings.append(f"{item['name']} collected") + else: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if output: + snippets.append(f"$ {item['cmd']}\n{output[-800:]}") + for line in output.splitlines(): + if anomaly_re.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") + +print(json.dumps({ + "domain": "DNS / Service Discovery", + "namespace": namespace, + "service": service, + "pod": pod, + "lookup": lookup, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-incident-timeline.sh b/oci/oke/scripts/oke-incident-timeline.sh new file mode 100755 index 0000000..d2f772c --- /dev/null +++ b/oci/oke/scripts/oke-incident-timeline.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Build a lightweight incident timeline from Kubernetes events and selected objects. +# +# Usage: +# bash scripts/oke-incident-timeline.sh --namespace [--pod ] [--deployment ] [--service ] [--compartment-id ] [--region ] + +namespace="" +pod="" +deployment="" +service="" +compartment_id="" +region="" + +emit_error() { + local exit_code="$1"; local error_code="$2"; local message="$3"; local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --pod) require_value "$1" "${2:-}"; pod="$2"; shift 2 ;; + --deployment) require_value "$1" "${2:-}"; deployment="$2"; shift 2 ;; + --service) require_value "$1" "${2:-}"; service="$2"; shift 2 ;; + --compartment-id) require_value "$1" "${2:-}"; compartment_id="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + -h|--help) echo "usage: oke-incident-timeline.sh --namespace [--pod ] [--deployment ] [--service ] [--compartment-id ] [--region ]"; exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --namespace." "Provide --namespace ." +fi +command -v kubectl >/dev/null 2>&1 || emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." + +tmp_dir="$(mktemp -d)"; trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl"; : > "$records" +run_check() { + local name="$1"; shift + local out rc + set +e; out="$("$@" 2>&1)"; rc=$?; set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({"name": os.environ["NAME"], "cmd": os.environ["CMD"], "rc": int(os.environ["RC"]), "output": os.environ["OUT"][-6000:]}) + "\n") +PY +} + +run_check "namespace events" kubectl -n "$namespace" get events --sort-by=.lastTimestamp +if [[ -n "$pod" ]]; then + run_check "pod describe" kubectl -n "$namespace" describe pod "$pod" +fi +if [[ -n "$deployment" ]]; then + run_check "deployment rollout history" kubectl -n "$namespace" rollout history deployment/"$deployment" + run_check "deployment describe" kubectl -n "$namespace" describe deployment "$deployment" +fi +if [[ -n "$service" ]]; then + run_check "service describe" kubectl -n "$namespace" describe service "$service" +fi +if [[ -n "$compartment_id" && -n "$region" ]] && command -v oci >/dev/null 2>&1; then + run_check "firing alarms" oci --region "$region" monitoring alarm-status list-alarms-status --compartment-id "$compartment_id" --status FIRING --all --output json +fi + +python3 - "$records" "$namespace" "$pod" "$deployment" "$service" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, pod, deployment, service = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +signal = re.compile(r"(Warning|Failed|BackOff|Killing|Pulling|Created|Started|Scaled|Successful|Unhealthy|NotReady|FIRING)", re.I) +timeline = [] +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if signal.search(line): + timeline.append({"source": item["name"], "event": line.strip()[:300]}) + if re.search(r"(Warning|Failed|BackOff|Unhealthy|NotReady|FIRING)", line, re.I): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "Incident Timeline", + "namespace": namespace, + "pod": pod, + "deployment": deployment, + "service": service, + "findings": findings, + "timeline": timeline[-30:], + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-ingress-check.sh b/oci/oke/scripts/oke-ingress-check.sh new file mode 100755 index 0000000..997aff1 --- /dev/null +++ b/oci/oke/scripts/oke-ingress-check.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect OCI Native Ingress and Kubernetes Ingress evidence. +# +# Usage: +# bash scripts/oke-ingress-check.sh --namespace --ingress [--region ] + +namespace="" +ingress="" +region="" + +emit_error() { + local exit_code="$1"; local error_code="$2"; local message="$3"; local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --ingress) require_value "$1" "${2:-}"; ingress="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + -h|--help) echo "usage: oke-ingress-check.sh --namespace --ingress [--region ]"; exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done +if [[ -z "$namespace" || -z "$ingress" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --namespace or --ingress." "Provide both --namespace and --ingress." +fi +command -v kubectl >/dev/null 2>&1 || emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." + +tmp_dir="$(mktemp -d)"; trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl"; : > "$records" +run_check() { + local name="$1"; shift + local out rc + set +e; out="$("$@" 2>&1)"; rc=$?; set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({"name": os.environ["NAME"], "cmd": os.environ["CMD"], "rc": int(os.environ["RC"]), "output": os.environ["OUT"][-4000:]}) + "\n") +PY +} + +run_check "ingress classes" kubectl get ingressclass -o yaml +run_check "target ingress" kubectl -n "$namespace" get ingress "$ingress" -o yaml +run_check "target ingress describe" kubectl -n "$namespace" describe ingress "$ingress" +run_check "ingress controller pods" kubectl -n kube-system get pods -l app.kubernetes.io/name=oci-native-ingress-controller -o wide +run_check "ingress controller logs" kubectl -n kube-system logs -l app.kubernetes.io/name=oci-native-ingress-controller --tail=200 + +python3 - "$records" "$namespace" "$ingress" "$region" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, ingress, region = sys.argv[2], sys.argv[3], sys.argv[4] +pat = re.compile(r"(Failed|Error|Warning|certificate|TLS|listener|backend|reconcile|NotFound|forbidden|load balancer|ingressclass)", re.I) +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if pat.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "Ingress / OCI Native Ingress", + "namespace": namespace, + "ingress": ingress, + "region": region, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-object-correlator.sh b/oci/oke/scripts/oke-object-correlator.sh new file mode 100755 index 0000000..159dbb8 --- /dev/null +++ b/oci/oke/scripts/oke-object-correlator.sh @@ -0,0 +1,1015 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Build a read-only Kubernetes-to-OCI object graph for OKE troubleshooting. +# +# Usage: +# bash scripts/oke-object-correlator.sh --namespace --cluster-id --compartment-id --region [--pod ] [--deployment ] [--service ] [--ingress ] [--pvc ] [--node ] + +namespace="" +cluster_id="" +compartment_id="" +region="" +pod="" +deployment="" +service="" +ingress="" +pvc="" +node="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --cluster-id) require_value "$1" "${2:-}"; cluster_id="$2"; shift 2 ;; + --compartment-id) require_value "$1" "${2:-}"; compartment_id="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + --pod) require_value "$1" "${2:-}"; pod="$2"; shift 2 ;; + --deployment) require_value "$1" "${2:-}"; deployment="$2"; shift 2 ;; + --service) require_value "$1" "${2:-}"; service="$2"; shift 2 ;; + --ingress) require_value "$1" "${2:-}"; ingress="$2"; shift 2 ;; + --pvc) require_value "$1" "${2:-}"; pvc="$2"; shift 2 ;; + --node) require_value "$1" "${2:-}"; node="$2"; shift 2 ;; + -h|--help) + echo "usage: oke-object-correlator.sh --namespace --cluster-id --compartment-id --region [--pod ] [--deployment ] [--service ] [--ingress ] [--pvc ] [--node ]" + exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done + +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --namespace." "Provide --namespace ." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +record_check() { + local name="$1" + local kind="$2" + local cmd="$3" + local rc="$4" + local out="$5" + NAME="$name" KIND="$kind" CMD="$cmd" RC="$rc" OUT="$out" python3 - "$records" <<'PY' +import json +import os +import sys + +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "kind": os.environ["KIND"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-12000:], + }) + "\n") +PY +} + +run_check() { + local name="$1" + local kind="$2" + shift 2 + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + record_check "$name" "$kind" "$*" "$rc" "$out" +} + +kubectl_available="yes" +oci_available="yes" +if ! command -v kubectl >/dev/null 2>&1; then + kubectl_available="no" + record_check "kubectl availability" "tool" "kubectl" 127 "kubectl is not installed or not on PATH" +fi +if ! command -v oci >/dev/null 2>&1; then + oci_available="no" + record_check "oci availability" "tool" "oci" 127 "oci is not installed or not on PATH" +fi + +if [[ "$kubectl_available" == "yes" ]]; then + if [[ -n "$pod" ]]; then + run_check "pod" "k8s_pod" kubectl -n "$namespace" get pod "$pod" -o json + fi + if [[ -n "$deployment" ]]; then + run_check "deployment" "k8s_deployment" kubectl -n "$namespace" get deployment "$deployment" -o json + fi + if [[ -n "$service" ]]; then + run_check "service" "k8s_service" kubectl -n "$namespace" get service "$service" -o json + run_check "endpoints" "k8s_endpoints" kubectl -n "$namespace" get endpoints "$service" -o json + fi + if [[ -n "$ingress" ]]; then + run_check "ingress" "k8s_ingress" kubectl -n "$namespace" get ingress "$ingress" -o json + fi + if [[ -n "$pvc" ]]; then + run_check "pvc" "k8s_pvc" kubectl -n "$namespace" get pvc "$pvc" -o json + fi + if [[ -n "$node" ]]; then + run_check "node" "k8s_node" kubectl get node "$node" -o json + fi +fi + +oci_cmd=(oci) +if [[ -n "$region" ]]; then + oci_cmd+=(--region "$region") +fi + +python3 - "$records" "$namespace" "$cluster_id" "$compartment_id" "$region" "$pod" "$deployment" "$service" "$ingress" "$pvc" "$node" <<'PY' >"$tmp_dir/plan.json" +import json +import re +import sys +from pathlib import Path + +records_path = Path(sys.argv[1]) +namespace, cluster_id, compartment_id, region = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +pod_arg, deployment_arg, service_arg, ingress_arg, pvc_arg, node_arg = sys.argv[6:12] +records = [json.loads(line) for line in records_path.read_text().splitlines() if line.strip()] + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +def ocid(value): + return isinstance(value, str) and value.startswith("ocid1.") + +def provider_instance_id(obj): + provider_id = ((obj.get("spec") or {}).get("providerID") or "") + match = re.search(r"(ocid1\.instance\.[A-Za-z0-9_.-]+)", provider_id) + return match.group(1) if match else "" + +def lb_addresses(item): + values = [] + for address in item.get("ip-addresses", []) or []: + ip = address.get("ip-address") or address.get("ip") + if ip: + values.append(ip) + return values + +def match_lb_by_ip(items, target_ip): + if not target_ip: + return None + for item in items: + if target_ip in lb_addresses(item): + return item + return None + +def ann(obj, key): + return (((obj or {}).get("metadata") or {}).get("annotations") or {}).get(key, "") + +def meta_name(obj): + return ((obj or {}).get("metadata") or {}).get("name", "") + +def first(value): + if isinstance(value, list) and value: + return value[0] + return None + +plan = [] +derived = {"node": node_arg, "instance_id": "", "lb_id": "", "lb_ip": "", "pv": "", "volume_id": ""} + +for rec in records: + obj = parse_json(rec.get("output", "")) + if rec["kind"] == "k8s_pod" and obj: + node = ((obj.get("spec") or {}).get("nodeName") or "") + if node and not derived["node"]: + derived["node"] = node + for vol in (obj.get("spec") or {}).get("volumes", []) or []: + claim = (vol.get("persistentVolumeClaim") or {}).get("claimName", "") + if claim: + plan.append({"name": f"pvc {claim}", "kind": "k8s_pvc", "cmd": ["kubectl", "-n", namespace, "get", "pvc", claim, "-o", "json"]}) + elif rec["kind"] == "k8s_node" and obj: + annotations = ((obj.get("metadata") or {}).get("annotations") or {}) + instance_id = ( + annotations.get("node.oci.oraclecloud.com/instance-id") + or annotations.get("oci.oraclecloud.com/instance-id") + or annotations.get("oci.oraclecloud.com/instance_id") + or provider_instance_id(obj) + or "" + ) + if instance_id: + derived["instance_id"] = instance_id + elif rec["kind"] == "k8s_service" and obj: + lb_id = ( + ann(obj, "oci.oraclecloud.com/load-balancer-id") + or ann(obj, "service.beta.kubernetes.io/oci-load-balancer-id") + or ann(obj, "oci.oraclecloud.com/network-load-balancer-id") + or "" + ) + if lb_id: + derived["lb_id"] = lb_id + ingress = (((obj.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if ingress: + derived["lb_ip"] = ingress[0].get("ip") or ingress[0].get("hostname") or "" + elif rec["kind"] == "k8s_ingress" and obj: + lb_id = ( + ann(obj, "oci.oraclecloud.com/load-balancer-id") + or ann(obj, "service.beta.kubernetes.io/oci-load-balancer-id") + or "" + ) + if lb_id: + derived["lb_id"] = lb_id + status_ing = (((obj.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if status_ing and not derived["lb_ip"]: + derived["lb_ip"] = status_ing[0].get("ip") or status_ing[0].get("hostname") or "" + elif rec["kind"] == "k8s_pvc" and obj: + pv = ((obj.get("spec") or {}).get("volumeName") or "") + if pv and not derived["pv"]: + derived["pv"] = pv + elif rec["kind"] == "k8s_pv" and obj: + spec = obj.get("spec") or {} + handle = ( + (spec.get("csi") or {}).get("volumeHandle") + or spec.get("ociVolumeID") + or spec.get("volumeID") + or "" + ) + if handle and not derived["volume_id"]: + derived["volume_id"] = handle + +if derived["node"] and not any(r["kind"] == "k8s_node" for r in records): + plan.append({"name": f"node {derived['node']}", "kind": "k8s_node", "cmd": ["kubectl", "get", "node", derived["node"], "-o", "json"]}) +if derived["pv"] and not any(r["kind"] == "k8s_pv" for r in records): + plan.append({"name": f"pv {derived['pv']}", "kind": "k8s_pv", "cmd": ["kubectl", "get", "pv", derived["pv"], "-o", "json"]}) +if derived["instance_id"]: + plan.append({"name": "compute instance", "kind": "oci_instance", "cmd": ["oci", "compute", "instance", "get", "--instance-id", derived["instance_id"], "--output", "json"]}) + if compartment_id: + plan.append({"name": "vnic attachments", "kind": "oci_vnic_attachments", "cmd": ["oci", "compute", "vnic-attachment", "list", "--compartment-id", compartment_id, "--instance-id", derived["instance_id"], "--all", "--output", "json"]}) +if derived["lb_id"]: + if ".networkloadbalancer." in derived["lb_id"]: + plan.append({"name": "network load balancer", "kind": "oci_nlb", "cmd": ["oci", "nlb", "network-load-balancer", "get", "--network-load-balancer-id", derived["lb_id"], "--output", "json"]}) + else: + plan.append({"name": "load balancer", "kind": "oci_lb", "cmd": ["oci", "lb", "load-balancer", "get", "--load-balancer-id", derived["lb_id"], "--output", "json"]}) + plan.append({"name": "load balancer health", "kind": "oci_lb_health", "cmd": ["oci", "lb", "load-balancer-health", "get", "--load-balancer-id", derived["lb_id"], "--output", "json"]}) +elif derived["lb_ip"] and compartment_id: + plan.append({"name": "load balancers by IP", "kind": "oci_lb_list", "cmd": ["oci", "lb", "load-balancer", "list", "--compartment-id", compartment_id, "--all", "--output", "json"]}) + plan.append({"name": "network load balancers by IP", "kind": "oci_nlb_list", "cmd": ["oci", "nlb", "network-load-balancer", "list", "--compartment-id", compartment_id, "--all", "--output", "json"]}) +if derived["volume_id"]: + plan.append({"name": "block volume", "kind": "oci_volume", "cmd": ["oci", "bv", "volume", "get", "--volume-id", derived["volume_id"], "--output", "json"]}) + if compartment_id: + plan.append({"name": "volume attachments", "kind": "oci_volume_attachments", "cmd": ["oci", "compute", "volume-attachment", "list", "--compartment-id", compartment_id, "--volume-id", derived["volume_id"], "--all", "--output", "json"]}) +if service_arg or ingress_arg: + if compartment_id: + plan.append({"name": "network security groups", "kind": "oci_nsgs", "cmd": ["oci", "network", "nsg", "list", "--compartment-id", compartment_id, "--all", "--output", "json"]}) +if cluster_id: + plan.append({"name": "cluster node pools", "kind": "oci_node_pools", "cmd": ["oci", "ce", "node-pool", "list", "--cluster-id", cluster_id, "--compartment-id", compartment_id, "--all", "--output", "json"]}) + +print(json.dumps(plan)) +PY + +execute_plan() { + local plan_file="$1" + python3 - "$plan_file" "$region" <<'PY' >"$tmp_dir/commands.jsonl" +import json +import sys +from pathlib import Path + +plan = json.loads(Path(sys.argv[1]).read_text()) +region = sys.argv[2] +for item in plan: + cmd = item["cmd"] + if cmd and cmd[0] == "oci" and region: + cmd = ["oci", "--region", region] + cmd[1:] + print(json.dumps({"name": item["name"], "kind": item["kind"], "cmd": cmd})) +PY + while IFS= read -r line; do + [[ -n "$line" ]] || continue + name="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["name"])' <<<"$line")" + kind="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["kind"])' <<<"$line")" + cmd_parts=() + while IFS= read -r part; do + cmd_parts+=("$part") + done < <(python3 -c 'import json,sys; [print(x) for x in json.loads(sys.stdin.read())["cmd"]]' <<<"$line") + if [[ "${cmd_parts[0]:-}" == "oci" && "$oci_available" != "yes" ]]; then + continue + fi + run_check "$name" "$kind" "${cmd_parts[@]}" + done <"$tmp_dir/commands.jsonl" +} + +execute_plan "$tmp_dir/plan.json" +python3 - "$records" "$namespace" "$cluster_id" "$compartment_id" "$region" "$pod" "$deployment" "$service" "$ingress" "$pvc" "$node" <<'PY' >"$tmp_dir/plan2.json" +import json +import re +import sys +from pathlib import Path + +records_path = Path(sys.argv[1]) +namespace, cluster_id, compartment_id, region = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +node_arg = sys.argv[11] +records = [json.loads(line) for line in records_path.read_text().splitlines() if line.strip()] + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +def ann(obj, key): + return (((obj or {}).get("metadata") or {}).get("annotations") or {}).get(key, "") + +def provider_instance_id(obj): + provider_id = ((obj.get("spec") or {}).get("providerID") or "") + match = re.search(r"(ocid1\.instance\.[A-Za-z0-9_.-]+)", provider_id) + return match.group(1) if match else "" + +derived = {"instance_id": "", "volume_id": "", "lb_ip": "", "lb_id": ""} +existing = {(r["kind"], r["name"]) for r in records} +plan = [] + +for rec in records: + obj = parse_json(rec.get("output", "")) + if not obj: + continue + data = obj.get("data", obj) if isinstance(obj, dict) else obj + if rec["kind"] == "k8s_node" and isinstance(data, dict): + annotations = ((data.get("metadata") or {}).get("annotations") or {}) + derived["instance_id"] = ( + annotations.get("node.oci.oraclecloud.com/instance-id") + or annotations.get("oci.oraclecloud.com/instance-id") + or annotations.get("oci.oraclecloud.com/instance_id") + or provider_instance_id(data) + or derived["instance_id"] + ) + elif rec["kind"] == "k8s_pvc" and isinstance(data, dict): + pv = ((data.get("spec") or {}).get("volumeName") or "") + if pv and ("k8s_pv", f"pv {pv}") not in existing: + plan.append({"name": f"pv {pv}", "kind": "k8s_pv", "cmd": ["kubectl", "get", "pv", pv, "-o", "json"]}) + elif rec["kind"] == "k8s_pv" and isinstance(data, dict): + spec = data.get("spec") or {} + derived["volume_id"] = ( + (spec.get("csi") or {}).get("volumeHandle") + or spec.get("ociVolumeID") + or spec.get("volumeID") + or derived["volume_id"] + ) + elif rec["kind"] in ("k8s_service", "k8s_ingress") and isinstance(data, dict): + annotations = ((data.get("metadata") or {}).get("annotations") or {}) + derived["lb_id"] = ( + annotations.get("oci.oraclecloud.com/load-balancer-id") + or annotations.get("service.beta.kubernetes.io/oci-load-balancer-id") + or annotations.get("oci.oraclecloud.com/network-load-balancer-id") + or derived["lb_id"] + ) + ingress = (((data.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if ingress and not derived["lb_ip"]: + derived["lb_ip"] = ingress[0].get("ip") or ingress[0].get("hostname") or "" + elif rec["kind"] == "oci_lb_list" and derived["lb_ip"]: + items = data.get("data", []) if isinstance(data, dict) else data + items = items or [] + for item in items: + for address in item.get("ip-addresses", []) or []: + if address.get("ip-address") == derived["lb_ip"]: + derived["lb_id"] = item.get("id", derived["lb_id"]) + break + elif rec["kind"] == "oci_nlb_list" and derived["lb_ip"]: + items = data.get("data", []) if isinstance(data, dict) else data + items = items or [] + for item in items: + for address in item.get("ip-addresses", []) or []: + if address.get("ip-address") == derived["lb_ip"]: + derived["lb_id"] = item.get("id", derived["lb_id"]) + break + +if derived["instance_id"] and ("oci_instance", "compute instance") not in existing: + plan.append({"name": "compute instance", "kind": "oci_instance", "cmd": ["oci", "compute", "instance", "get", "--instance-id", derived["instance_id"], "--output", "json"]}) + if compartment_id and ("oci_vnic_attachments", "vnic attachments") not in existing: + plan.append({"name": "vnic attachments", "kind": "oci_vnic_attachments", "cmd": ["oci", "compute", "vnic-attachment", "list", "--compartment-id", compartment_id, "--instance-id", derived["instance_id"], "--all", "--output", "json"]}) +if derived["volume_id"] and ("oci_volume", "block volume") not in existing: + plan.append({"name": "block volume", "kind": "oci_volume", "cmd": ["oci", "bv", "volume", "get", "--volume-id", derived["volume_id"], "--output", "json"]}) + if compartment_id and ("oci_volume_attachments", "volume attachments") not in existing: + plan.append({"name": "volume attachments", "kind": "oci_volume_attachments", "cmd": ["oci", "compute", "volume-attachment", "list", "--compartment-id", compartment_id, "--volume-id", derived["volume_id"], "--all", "--output", "json"]}) +if derived["lb_id"] and ("oci_lb", "load balancer") not in existing and ("oci_nlb", "network load balancer") not in existing: + if ".networkloadbalancer." in derived["lb_id"]: + plan.append({"name": "network load balancer", "kind": "oci_nlb", "cmd": ["oci", "nlb", "network-load-balancer", "get", "--network-load-balancer-id", derived["lb_id"], "--output", "json"]}) + else: + plan.append({"name": "load balancer", "kind": "oci_lb", "cmd": ["oci", "lb", "load-balancer", "get", "--load-balancer-id", derived["lb_id"], "--output", "json"]}) + plan.append({"name": "load balancer health", "kind": "oci_lb_health", "cmd": ["oci", "lb", "load-balancer-health", "get", "--load-balancer-id", derived["lb_id"], "--output", "json"]}) + +print(json.dumps(plan)) +PY +execute_plan "$tmp_dir/plan2.json" + +python3 - "$records" "$compartment_id" <<'PY' >"$tmp_dir/plan3.json" +import json +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +compartment_id = sys.argv[2] +existing = {(r["kind"], r["name"]) for r in records} +plan = [] + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +for rec in records: + if rec["kind"] != "k8s_pv": + continue + obj = parse_json(rec.get("output", "")) + if not isinstance(obj, dict): + continue + spec = obj.get("spec") or {} + volume_id = ( + (spec.get("csi") or {}).get("volumeHandle") + or spec.get("ociVolumeID") + or spec.get("volumeID") + or "" + ) + if volume_id and ("oci_volume", "block volume") not in existing: + plan.append({"name": "block volume", "kind": "oci_volume", "cmd": ["oci", "bv", "volume", "get", "--volume-id", volume_id, "--output", "json"]}) + if volume_id and compartment_id and ("oci_volume_attachments", "volume attachments") not in existing: + plan.append({"name": "volume attachments", "kind": "oci_volume_attachments", "cmd": ["oci", "compute", "volume-attachment", "list", "--compartment-id", compartment_id, "--volume-id", volume_id, "--all", "--output", "json"]}) + +print(json.dumps(plan)) +PY +execute_plan "$tmp_dir/plan3.json" + +python3 - "$records" <<'PY' >"$tmp_dir/plan4.json" +import json +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +existing = {(r["kind"], r["name"]) for r in records} +requested = set() +plan = [] + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +def data(obj): + if isinstance(obj, dict) and isinstance(obj.get("data"), dict): + return obj["data"] + return obj + +def items(obj): + if isinstance(obj, dict): + return obj.get("data", []) + if isinstance(obj, list): + return obj + return [] + +def add(name, kind, cmd): + key = (kind, name) + if key in existing or key in requested: + return + requested.add(key) + plan.append({"name": name, "kind": kind, "cmd": cmd}) + +for rec in records: + obj = parse_json(rec.get("output", "")) + if not obj: + continue + obj_data = data(obj) + if rec["kind"] == "oci_vnic_attachments": + for item in items(obj_data): + vnic_id = item.get("vnic-id") + subnet_id = item.get("subnet-id") + if vnic_id: + add(f"vnic {vnic_id}", "oci_vnic", ["oci", "network", "vnic", "get", "--vnic-id", vnic_id, "--output", "json"]) + if subnet_id: + add(f"subnet {subnet_id}", "oci_subnet", ["oci", "network", "subnet", "get", "--subnet-id", subnet_id, "--output", "json"]) + elif rec["kind"] in ("oci_lb", "oci_nlb"): + lb = data(obj_data) + for subnet_id in lb.get("subnet-ids", []) or []: + add(f"subnet {subnet_id}", "oci_subnet", ["oci", "network", "subnet", "get", "--subnet-id", subnet_id, "--output", "json"]) + +print(json.dumps(plan)) +PY +execute_plan "$tmp_dir/plan4.json" + +python3 - "$records" <<'PY' >"$tmp_dir/plan5.json" +import json +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +existing = {(r["kind"], r["name"]) for r in records} +requested = set() +plan = [] + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +def data(obj): + if isinstance(obj, dict) and isinstance(obj.get("data"), dict): + return obj["data"] + return obj + +def add(name, kind, cmd): + key = (kind, name) + if key in existing or key in requested: + return + requested.add(key) + plan.append({"name": name, "kind": kind, "cmd": cmd}) + +for rec in records: + obj = parse_json(rec.get("output", "")) + if not obj: + continue + obj_data = data(obj) + if rec["kind"] == "oci_vnic": + vnic = data(obj_data) + subnet_id = vnic.get("subnet-id") + if subnet_id: + add(f"subnet {subnet_id}", "oci_subnet", ["oci", "network", "subnet", "get", "--subnet-id", subnet_id, "--output", "json"]) + for nsg_id in vnic.get("nsg-ids", []) or []: + add(f"nsg {nsg_id}", "oci_nsg", ["oci", "network", "nsg", "get", "--nsg-id", nsg_id, "--output", "json"]) + elif rec["kind"] == "oci_subnet": + subnet = data(obj_data) + route_table_id = subnet.get("route-table-id") + if route_table_id: + add(f"route-table {route_table_id}", "oci_route_table", ["oci", "network", "route-table", "get", "--rt-id", route_table_id, "--output", "json"]) + for sl_id in subnet.get("security-list-ids", []) or []: + add(f"security-list {sl_id}", "oci_security_list", ["oci", "network", "security-list", "get", "--security-list-id", sl_id, "--output", "json"]) + +print(json.dumps(plan)) +PY +execute_plan "$tmp_dir/plan5.json" + +python3 - "$records" "$namespace" "$cluster_id" "$compartment_id" "$region" "$pod" "$deployment" "$service" "$ingress" "$pvc" "$node" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, cluster_id, compartment_id, region = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +pod_arg, deployment_arg, service_arg, ingress_arg, pvc_arg, node_arg = sys.argv[6:12] + +nodes = {} +edges = [] +findings = [] +anomalies = [] +snippets = [] +seen_edges = set() + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +def add_node(node_id, node_type, name="", source="", **attrs): + if not node_id: + return + existing = nodes.setdefault(node_id, {"id": node_id, "type": node_type, "name": name or node_id, "source": source, "attributes": {}}) + existing["type"] = existing.get("type") or node_type + if name: + existing["name"] = name + if source: + existing["source"] = source + for key, value in attrs.items(): + if value not in ("", None, [], {}): + existing["attributes"][key] = value + +def add_edge(src, dst, relation, evidence=""): + if not src or not dst: + return + key = (src, dst, relation) + if key in seen_edges: + return + seen_edges.add(key) + edge = {"from": src, "to": dst, "relation": relation} + if evidence: + edge["evidence"] = evidence + edges.append(edge) + +def data(obj): + if isinstance(obj, dict) and isinstance(obj.get("data"), dict): + return obj["data"] + return obj + +def ann(obj, key): + return (((obj or {}).get("metadata") or {}).get("annotations") or {}).get(key, "") + +def labels(obj): + return ((obj or {}).get("metadata") or {}).get("labels") or {} + +def meta_name(obj): + return ((obj or {}).get("metadata") or {}).get("name", "") + +def k8s_id(kind, name, ns=None): + if not name: + return "" + return f"k8s:{kind}:{ns + '/' if ns else ''}{name}" + +def oci_id(kind, ocid): + if not ocid: + return "" + return f"oci:{kind}:{ocid}" + +def ocid(value): + return isinstance(value, str) and value.startswith("ocid1.") + +def provider_instance_id(obj): + provider_id = ((obj.get("spec") or {}).get("providerID") or "") + match = re.search(r"(ocid1\.instance\.[A-Za-z0-9_.-]+)", provider_id) + return match.group(1) if match else "" + +def lb_addresses(item): + values = [] + for address in item.get("ip-addresses", []) or []: + ip = address.get("ip-address") or address.get("ip") + if ip: + values.append(ip) + return values + +def match_lb_by_ip(items, target_ip): + if not target_ip: + return None + for item in items: + if target_ip in lb_addresses(item): + return item + return None + +def network_entity_type(ocid_value): + if not ocid_value: + return "oci.network.entity" + mapping = { + ".internetgateway.": "oci.network.internetgateway", + ".natgateway.": "oci.network.natgateway", + ".servicegateway.": "oci.network.servicegateway", + ".drg.": "oci.network.drg", + ".localpeeringgateway.": "oci.network.localpeeringgateway", + ".remotepeeringconnection.": "oci.network.remotepeeringconnection", + ".privateip.": "oci.network.privateip", + ".ipsecconnection.": "oci.network.ipsecconnection", + ".networksecuritygroup.": "oci.network.nsg", + } + return next((node_type for marker, node_type in mapping.items() if marker in ocid_value), "oci.network.entity") + +derived = {"node": node_arg, "instance_id": "", "lb_id": "", "lb_ip": "", "pv": "", "volume_id": ""} + +for rec in records: + out = rec.get("output", "").strip() + if rec.get("rc") != 0: + anomalies.append(f"{rec['name']} failed with rc={rec['rc']}") + if out: + snippets.append(f"$ {rec['cmd']}\n{out[-1000:]}") + +for rec in records: + obj = parse_json(rec.get("output", "")) + if not obj: + continue + obj_data = data(obj) + kind = rec["kind"] + if kind == "k8s_pod": + name = meta_name(obj_data) or pod_arg + pod_id = k8s_id("pod", name, namespace) + add_node(pod_id, "kubernetes.pod", name, "kubectl", namespace=namespace, service_account=(obj_data.get("spec") or {}).get("serviceAccountName")) + node = (obj_data.get("spec") or {}).get("nodeName", "") + if node: + derived["node"] = node + node_id = k8s_id("node", node) + add_node(node_id, "kubernetes.node", node, "kubectl") + add_edge(pod_id, node_id, "scheduled_on", "pod.spec.nodeName") + sa = (obj_data.get("spec") or {}).get("serviceAccountName", "") + if sa: + sa_id = k8s_id("serviceaccount", sa, namespace) + add_node(sa_id, "kubernetes.serviceaccount", sa, "kubectl", namespace=namespace) + add_edge(pod_id, sa_id, "uses_service_account", "pod.spec.serviceAccountName") + for vol in (obj_data.get("spec") or {}).get("volumes", []) or []: + claim = (vol.get("persistentVolumeClaim") or {}).get("claimName", "") + if claim: + pvc_id = k8s_id("pvc", claim, namespace) + add_node(pvc_id, "kubernetes.pvc", claim, "kubectl", namespace=namespace) + add_edge(pod_id, pvc_id, "mounts_claim", f"pod volume {vol.get('name', '')}") + elif kind == "k8s_node": + name = meta_name(obj_data) or derived["node"] + node_id = k8s_id("node", name) + annotations = ((obj_data.get("metadata") or {}).get("annotations") or {}) + annotation_instance_id = annotations.get("node.oci.oraclecloud.com/instance-id") or annotations.get("oci.oraclecloud.com/instance-id") or annotations.get("oci.oraclecloud.com/instance_id") or "" + provider_instance = provider_instance_id(obj_data) + instance_id = annotation_instance_id or provider_instance + pool = labels(obj_data).get("oke.oraclecloud.com/nodepool") or labels(obj_data).get("oci.oraclecloud.com/oke-nodepool") + add_node(node_id, "kubernetes.node", name, "kubectl", nodepool=pool, provider_id=(obj_data.get("spec") or {}).get("providerID")) + if instance_id: + derived["instance_id"] = instance_id + inst_id = oci_id("instance", instance_id) + source = "kubectl node annotation" if annotation_instance_id else "kubectl node providerID" + evidence = "node instance annotation" if annotation_instance_id else "node.spec.providerID" + add_node(inst_id, "oci.compute.instance", instance_id, source) + add_edge(node_id, inst_id, "runs_on_instance", evidence) + elif kind == "k8s_deployment": + name = meta_name(obj_data) or deployment_arg + dep_id = k8s_id("deployment", name, namespace) + add_node(dep_id, "kubernetes.deployment", name, "kubectl", namespace=namespace, replicas=(obj_data.get("spec") or {}).get("replicas")) + elif kind == "k8s_service": + name = meta_name(obj_data) or service_arg + svc_id = k8s_id("service", name, namespace) + spec = obj_data.get("spec") or {} + add_node(svc_id, "kubernetes.service", name, "kubectl", namespace=namespace, service_type=spec.get("type"), selector=spec.get("selector")) + lb_id = ann(obj_data, "oci.oraclecloud.com/load-balancer-id") or ann(obj_data, "service.beta.kubernetes.io/oci-load-balancer-id") or ann(obj_data, "oci.oraclecloud.com/network-load-balancer-id") + status_ing = (((obj_data.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if status_ing and not lb_id: + derived["lb_ip"] = status_ing[0].get("ip") or status_ing[0].get("hostname") or "" + if lb_id: + derived["lb_id"] = lb_id + lb_kind = "networkloadbalancer" if ".networkloadbalancer." in lb_id else "loadbalancer" + lb_node = oci_id(lb_kind, lb_id) + add_node(lb_node, f"oci.{lb_kind}", lb_id, "service annotation") + add_edge(svc_id, lb_node, "provisions", "service LB annotation") + elif kind == "k8s_endpoints": + name = meta_name(obj_data) or service_arg + ep_id = k8s_id("endpoints", name, namespace) + svc_id = k8s_id("service", name, namespace) + add_node(ep_id, "kubernetes.endpoints", name, "kubectl", namespace=namespace) + add_edge(svc_id, ep_id, "selects_endpoints", "endpoints object") + addresses = [] + for subset in obj_data.get("subsets", []) or []: + for addr in subset.get("addresses", []) or []: + ip = addr.get("ip") + if ip: + addresses.append(ip) + if not addresses: + anomalies.append(f"Service {namespace}/{name} has no ready endpoint addresses") + else: + findings.append(f"Service {namespace}/{name} has {len(addresses)} ready endpoint address(es)") + elif kind == "k8s_ingress": + name = meta_name(obj_data) or ingress_arg + ing_id = k8s_id("ingress", name, namespace) + add_node(ing_id, "kubernetes.ingress", name, "kubectl", namespace=namespace, ingress_class=(obj_data.get("spec") or {}).get("ingressClassName")) + lb_id = ann(obj_data, "oci.oraclecloud.com/load-balancer-id") or ann(obj_data, "service.beta.kubernetes.io/oci-load-balancer-id") + if lb_id: + derived["lb_id"] = lb_id + lb_node = oci_id("loadbalancer", lb_id) + add_node(lb_node, "oci.loadbalancer", lb_id, "ingress annotation") + add_edge(ing_id, lb_node, "provisions", "ingress LB annotation") + status_ing = (((obj_data.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if status_ing and not derived["lb_ip"]: + derived["lb_ip"] = status_ing[0].get("ip") or status_ing[0].get("hostname") or "" + for rule in (obj_data.get("spec") or {}).get("rules", []) or []: + for path in (((rule.get("http") or {}).get("paths")) or []): + backend_svc = ((((path.get("backend") or {}).get("service") or {}).get("name")) or "") + if backend_svc: + svc_id = k8s_id("service", backend_svc, namespace) + add_node(svc_id, "kubernetes.service", backend_svc, "ingress backend", namespace=namespace) + add_edge(ing_id, svc_id, "routes_to", f"host {rule.get('host', '')}") + elif kind == "k8s_pvc": + name = meta_name(obj_data) or pvc_arg + pvc_id = k8s_id("pvc", name, namespace) + pv = (obj_data.get("spec") or {}).get("volumeName", "") + add_node(pvc_id, "kubernetes.pvc", name, "kubectl", namespace=namespace, phase=(obj_data.get("status") or {}).get("phase")) + if pv: + derived["pv"] = pv + pv_id = k8s_id("pv", pv) + add_node(pv_id, "kubernetes.pv", pv, "pvc.spec.volumeName") + add_edge(pvc_id, pv_id, "bound_to", "pvc.spec.volumeName") + elif kind == "k8s_pv": + name = meta_name(obj_data) or derived["pv"] + pv_id = k8s_id("pv", name) + spec = obj_data.get("spec") or {} + handle = (spec.get("csi") or {}).get("volumeHandle") or spec.get("ociVolumeID") or spec.get("volumeID") or "" + add_node(pv_id, "kubernetes.pv", name, "kubectl", storage_class=spec.get("storageClassName")) + if handle: + derived["volume_id"] = handle + vol_id = oci_id("volume", handle) + add_node(vol_id, "oci.blockvolume.volume", handle, "pv volume handle") + add_edge(pv_id, vol_id, "backs_onto", "PV volume handle") + elif kind == "oci_instance": + inst = data(obj_data) + inst_id_raw = inst.get("id", derived["instance_id"]) + inst_id = oci_id("instance", inst_id_raw) + add_node(inst_id, "oci.compute.instance", inst_id_raw, "oci", lifecycle_state=inst.get("lifecycle-state"), shape=inst.get("shape"), availability_domain=inst.get("availability-domain")) + if inst.get("lifecycle-state") and inst.get("lifecycle-state") != "RUNNING": + anomalies.append(f"Compute instance {inst_id_raw} lifecycle-state is {inst.get('lifecycle-state')}") + elif kind == "oci_vnic_attachments": + items = obj_data.get("data", obj_data if isinstance(obj_data, list) else []) or [] + inst_id = oci_id("instance", derived["instance_id"]) + for item in items: + vnic_id = item.get("vnic-id") + subnet_id = item.get("subnet-id") + if vnic_id: + vnic_node = oci_id("vnic", vnic_id) + add_node(vnic_node, "oci.network.vnic", vnic_id, "oci", lifecycle_state=item.get("lifecycle-state"), display_name=item.get("display-name")) + add_edge(inst_id, vnic_node, "has_vnic", "vnic attachment") + if subnet_id: + subnet_node = oci_id("subnet", subnet_id) + add_node(subnet_node, "oci.network.subnet", subnet_id, "oci vnic attachment") + add_edge(oci_id("vnic", vnic_id), subnet_node, "attached_to_subnet", "vnic attachment subnet") + elif kind == "oci_vnic": + vnic = data(obj_data) + vnic_id_raw = vnic.get("id", "") + vnic_node = oci_id("vnic", vnic_id_raw) + add_node( + vnic_node, + "oci.network.vnic", + vnic.get("display-name") or vnic_id_raw, + "oci", + lifecycle_state=vnic.get("lifecycle-state"), + private_ip=vnic.get("private-ip"), + public_ip=vnic.get("public-ip"), + is_primary=vnic.get("is-primary"), + subnet_id=vnic.get("subnet-id"), + ) + subnet_id = vnic.get("subnet-id") + if subnet_id: + subnet_node = oci_id("subnet", subnet_id) + add_node(subnet_node, "oci.network.subnet", subnet_id, "oci vnic") + add_edge(vnic_node, subnet_node, "attached_to_subnet", "vnic.subnet-id") + for nsg_id in vnic.get("nsg-ids", []) or []: + nsg_node = oci_id("nsg", nsg_id) + add_node(nsg_node, "oci.network.nsg", nsg_id, "vnic.nsg-ids") + add_edge(vnic_node, nsg_node, "uses_nsg", "vnic.nsg-ids") + elif kind == "oci_subnet": + subnet = data(obj_data) + subnet_id_raw = subnet.get("id", "") + subnet_node = oci_id("subnet", subnet_id_raw) + add_node( + subnet_node, + "oci.network.subnet", + subnet.get("display-name") or subnet_id_raw, + "oci", + cidr_block=subnet.get("cidr-block"), + ipv6_cidr_block=subnet.get("ipv6-cidr-block"), + dns_label=subnet.get("dns-label"), + route_table_id=subnet.get("route-table-id"), + vcn_id=subnet.get("vcn-id"), + ) + vcn_id = subnet.get("vcn-id") + if vcn_id: + vcn_node = oci_id("vcn", vcn_id) + add_node(vcn_node, "oci.network.vcn", vcn_id, "subnet.vcn-id") + add_edge(subnet_node, vcn_node, "in_vcn", "subnet.vcn-id") + route_table_id = subnet.get("route-table-id") + if route_table_id: + rt_node = oci_id("route-table", route_table_id) + add_node(rt_node, "oci.network.route_table", route_table_id, "subnet.route-table-id") + add_edge(subnet_node, rt_node, "uses_route_table", "subnet.route-table-id") + for sl_id in subnet.get("security-list-ids", []) or []: + sl_node = oci_id("security-list", sl_id) + add_node(sl_node, "oci.network.security_list", sl_id, "subnet.security-list-ids") + add_edge(subnet_node, sl_node, "uses_security_list", "subnet.security-list-ids") + elif kind == "oci_nsg": + nsg = data(obj_data) + nsg_id_raw = nsg.get("id", "") + add_node(oci_id("nsg", nsg_id_raw), "oci.network.nsg", nsg.get("display-name") or nsg_id_raw, "oci", lifecycle_state=nsg.get("lifecycle-state"), vcn_id=nsg.get("vcn-id")) + elif kind == "oci_security_list": + sl = data(obj_data) + sl_id_raw = sl.get("id", "") + add_node( + oci_id("security-list", sl_id_raw), + "oci.network.security_list", + sl.get("display-name") or sl_id_raw, + "oci", + lifecycle_state=sl.get("lifecycle-state"), + vcn_id=sl.get("vcn-id"), + ingress_rules=len(sl.get("ingress-security-rules", []) or []), + egress_rules=len(sl.get("egress-security-rules", []) or []), + ) + elif kind == "oci_route_table": + rt = data(obj_data) + rt_id_raw = rt.get("id", "") + rt_node = oci_id("route-table", rt_id_raw) + add_node(rt_node, "oci.network.route_table", rt.get("display-name") or rt_id_raw, "oci", lifecycle_state=rt.get("lifecycle-state"), vcn_id=rt.get("vcn-id")) + for idx, rule in enumerate(rt.get("route-rules", []) or [], start=1): + target_id = rule.get("network-entity-id") or rule.get("networkEntityId") or "" + destination = rule.get("destination") or rule.get("cidr-block") or rule.get("destination-type") or f"rule {idx}" + if target_id: + target_node = oci_id("network-entity", target_id) + add_node(target_node, network_entity_type(target_id), target_id, "route rule", destination=destination) + add_edge(rt_node, target_node, "routes_to", str(destination)) + elif kind in ("oci_lb", "oci_nlb"): + lb = data(obj_data) + lb_id_raw = lb.get("id", derived["lb_id"]) + lb_type = "oci.networkloadbalancer" if kind == "oci_nlb" else "oci.loadbalancer" + lb_node = oci_id("networkloadbalancer" if kind == "oci_nlb" else "loadbalancer", lb_id_raw) + add_node(lb_node, lb_type, lb_id_raw, "oci", lifecycle_state=lb.get("lifecycle-state") or lb.get("lifecycle-state-details"), shape=lb.get("shape-name"), ip_addresses=lb_addresses(lb)) + for subnet_id in lb.get("subnet-ids", []) or []: + subnet_node = oci_id("subnet", subnet_id) + add_node(subnet_node, "oci.network.subnet", subnet_id, "oci load balancer") + add_edge(lb_node, subnet_node, "uses_subnet", "load balancer subnet") + if service_arg and (derived["lb_id"] == lb_id_raw or derived["lb_ip"] in lb_addresses(lb)): + add_edge(k8s_id("service", service_arg, namespace), lb_node, "provisions", "matched service external IP or LB ID") + if ingress_arg and (derived["lb_id"] == lb_id_raw or derived["lb_ip"] in lb_addresses(lb)): + add_edge(k8s_id("ingress", ingress_arg, namespace), lb_node, "provisions", "matched ingress external IP or LB ID") + if lb.get("lifecycle-state") and lb.get("lifecycle-state") not in ("ACTIVE", "SUCCEEDED"): + anomalies.append(f"Load balancer {lb_id_raw} lifecycle-state is {lb.get('lifecycle-state')}") + elif kind in ("oci_lb_list", "oci_nlb_list"): + items = obj_data.get("data", []) if isinstance(obj_data, dict) else obj_data + items = items or [] + lb = match_lb_by_ip(items, derived["lb_ip"]) + if lb: + lb_id_raw = lb.get("id", "") + lb_kind = "networkloadbalancer" if kind == "oci_nlb_list" else "loadbalancer" + lb_type = "oci.networkloadbalancer" if kind == "oci_nlb_list" else "oci.loadbalancer" + lb_node = oci_id(lb_kind, lb_id_raw) + derived["lb_id"] = lb_id_raw + add_node(lb_node, lb_type, lb_id_raw, "oci ip match", lifecycle_state=lb.get("lifecycle-state") or lb.get("lifecycle-state-details"), shape=lb.get("shape-name"), ip_addresses=lb_addresses(lb), display_name=lb.get("display-name")) + if service_arg: + add_edge(k8s_id("service", service_arg, namespace), lb_node, "provisions", f"matched external IP {derived['lb_ip']}") + if ingress_arg: + add_edge(k8s_id("ingress", ingress_arg, namespace), lb_node, "provisions", f"matched external IP {derived['lb_ip']}") + for subnet_id in lb.get("subnet-ids", []) or []: + subnet_node = oci_id("subnet", subnet_id) + add_node(subnet_node, "oci.network.subnet", subnet_id, "oci load balancer") + add_edge(lb_node, subnet_node, "uses_subnet", "load balancer subnet") + findings.append(f"Matched external IP {derived['lb_ip']} to OCI {'Network ' if kind == 'oci_nlb_list' else ''}Load Balancer {lb_id_raw}") + if lb.get("lifecycle-state") and lb.get("lifecycle-state") not in ("ACTIVE", "SUCCEEDED"): + anomalies.append(f"Load balancer {lb_id_raw} lifecycle-state is {lb.get('lifecycle-state')}") + elif derived["lb_ip"]: + anomalies.append(f"No OCI {'Network ' if kind == 'oci_nlb_list' else ''}Load Balancer matched external IP {derived['lb_ip']}") + elif kind == "oci_lb_health": + health = data(obj_data) + status = health.get("status") or health.get("overall-health") + lb_node = oci_id("loadbalancer", derived["lb_id"]) + if status: + add_node(f"{lb_node}:health", "oci.loadbalancer.health", status, "oci", status=status) + add_edge(lb_node, f"{lb_node}:health", "has_health", "LB health API") + if str(status).upper() not in ("OK", "SUCCEEDED", "HEALTHY"): + anomalies.append(f"Load balancer health status is {status}") + elif kind == "oci_volume": + vol = data(obj_data) + vol_id_raw = vol.get("id", derived["volume_id"]) + add_node(oci_id("volume", vol_id_raw), "oci.blockvolume.volume", vol_id_raw, "oci", lifecycle_state=vol.get("lifecycle-state"), availability_domain=vol.get("availability-domain"), size_in_gbs=vol.get("size-in-gbs")) + elif kind == "oci_volume_attachments": + items = obj_data.get("data", obj_data if isinstance(obj_data, list) else []) or [] + for item in items: + vol_id_raw = item.get("volume-id", derived["volume_id"]) + inst_id_raw = item.get("instance-id") + if vol_id_raw and inst_id_raw: + add_node(oci_id("volume", vol_id_raw), "oci.blockvolume.volume", vol_id_raw, "oci volume attachment") + add_node(oci_id("instance", inst_id_raw), "oci.compute.instance", inst_id_raw, "oci volume attachment") + add_edge(oci_id("volume", vol_id_raw), oci_id("instance", inst_id_raw), "attached_to_instance", "volume attachment") + if item.get("lifecycle-state") and item.get("lifecycle-state") != "ATTACHED": + anomalies.append(f"Volume attachment {item.get('id', '')} lifecycle-state is {item.get('lifecycle-state')}") + elif kind == "oci_node_pools": + items = obj_data.get("data", obj_data if isinstance(obj_data, list) else []) or [] + cluster_node = oci_id("cluster", cluster_id) + if cluster_id: + add_node(cluster_node, "oci.oke.cluster", cluster_id, "input", region=region, compartment_id=compartment_id) + for item in items: + np_id = item.get("id") + if np_id: + np_node = oci_id("nodepool", np_id) + add_node(np_node, "oci.oke.nodepool", item.get("name", np_id), "oci", size=item.get("size"), shape=item.get("node-shape")) + add_edge(cluster_node, np_node, "has_node_pool", "node pool list") + +if cluster_id: + add_node(oci_id("cluster", cluster_id), "oci.oke.cluster", cluster_id, "input", region=region, compartment_id=compartment_id) + +if not findings: + findings.append(f"Built object graph with {len(nodes)} node(s) and {len(edges)} edge(s)") +else: + findings.insert(0, f"Built object graph with {len(nodes)} node(s) and {len(edges)} edge(s)") + +if not nodes: + anomalies.append("No Kubernetes or OCI objects could be correlated from the provided selectors") + +print(json.dumps({ + "domain": "OCI Object Correlation", + "namespace": namespace, + "cluster_id": cluster_id, + "compartment_id": compartment_id, + "region": region, + "selectors": { + "pod": pod_arg, + "deployment": deployment_arg, + "service": service_arg, + "ingress": ingress_arg, + "pvc": pvc_arg, + "node": node_arg, + }, + "graph": { + "kubernetes": [node for node in nodes.values() if node["type"].startswith("kubernetes.")], + "oci": [node for node in nodes.values() if node["type"].startswith("oci.")], + "edges": edges, + }, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-12:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-ocir-image-pull-check.sh b/oci/oke/scripts/oke-ocir-image-pull-check.sh new file mode 100755 index 0000000..594dbee --- /dev/null +++ b/oci/oke/scripts/oke-ocir-image-pull-check.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect OCIR image pull failure evidence. +# +# Usage: +# bash scripts/oke-ocir-image-pull-check.sh --namespace --pod [--image ] [--compartment-id ] [--region ] + +namespace="" +pod="" +image="" +compartment_id="" +region="" + +emit_error() { + local exit_code="$1"; local error_code="$2"; local message="$3"; local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --pod) require_value "$1" "${2:-}"; pod="$2"; shift 2 ;; + --image) require_value "$1" "${2:-}"; image="$2"; shift 2 ;; + --compartment-id) require_value "$1" "${2:-}"; compartment_id="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + -h|--help) echo "usage: oke-ocir-image-pull-check.sh --namespace --pod [--image ] [--compartment-id ] [--region ]"; exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done +if [[ -z "$namespace" || -z "$pod" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --namespace or --pod." "Provide both --namespace and --pod." +fi +command -v kubectl >/dev/null 2>&1 || emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." + +tmp_dir="$(mktemp -d)"; trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl"; : > "$records" +run_check() { + local name="$1"; shift + local out rc + set +e; out="$("$@" 2>&1)"; rc=$?; set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({"name": os.environ["NAME"], "cmd": os.environ["CMD"], "rc": int(os.environ["RC"]), "output": os.environ["OUT"][-4000:]}) + "\n") +PY +} + +run_check "pod describe" kubectl -n "$namespace" describe pod "$pod" +run_check "pod yaml" kubectl -n "$namespace" get pod "$pod" -o yaml +run_check "pod warning events" kubectl -n "$namespace" get events --field-selector "involvedObject.name=$pod" --sort-by=.lastTimestamp +run_check "service accounts" kubectl -n "$namespace" get serviceaccount -o yaml +run_check "image pull secrets" kubectl -n "$namespace" get secret -o yaml + +if [[ -n "$compartment_id" && -n "$region" ]] && command -v oci >/dev/null 2>&1; then + run_check "OCIR repositories" oci --region "$region" artifacts container repository list --compartment-id "$compartment_id" --all --output json +fi + +python3 - "$records" "$namespace" "$pod" "$image" "$region" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, pod, image, region = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +pat = re.compile(r"(ImagePullBackOff|ErrImagePull|unauthorized|denied|not found|repository|pull access denied|x509|i/o timeout|no basic auth|OCIR)", re.I) +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if pat.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "OCIR / Image Pull", + "namespace": namespace, + "pod": pod, + "image": image, + "region": region, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-pod-network-check.sh b/oci/oke/scripts/oke-pod-network-check.sh new file mode 100755 index 0000000..ed6907d --- /dev/null +++ b/oci/oke/scripts/oke-pod-network-check.sh @@ -0,0 +1,149 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect pod networking, OCI CNI/IPAM, and Multus signals. +# +# Usage: +# bash scripts/oke-pod-network-check.sh --namespace [--pod ] [--selector ] + +namespace="" +pod="" +selector="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) + require_value "$1" "${2:-}" + namespace="$2" + shift 2 + ;; + --pod) + require_value "$1" "${2:-}" + pod="$2" + shift 2 + ;; + --selector|-l) + require_value "$1" "${2:-}" + selector="$2" + shift 2 + ;; + -h|--help) + echo "usage: oke-pod-network-check.sh --namespace [--pod ] [--selector ]" + exit 0 + ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." + ;; + esac +done + +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --namespace." "Provide --namespace ." +fi + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1" + shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json +import os +import sys + +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], + }) + "\n") +PY +} + +if [[ -n "$pod" ]]; then + run_check "target pod" kubectl -n "$namespace" get pod "$pod" -o wide + run_check "target pod describe" kubectl -n "$namespace" describe pod "$pod" + run_check "target pod network-status" kubectl -n "$namespace" get pod "$pod" -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' + run_check "target pod node" kubectl -n "$namespace" get pod "$pod" -o jsonpath='{.spec.nodeName}' +else + pod_args=() + if [[ -n "$selector" ]]; then + pod_args=(-l "$selector") + fi + run_check "namespace pods" kubectl -n "$namespace" get pods "${pod_args[@]}" -o wide +fi + +run_check "namespace warning events" kubectl -n "$namespace" get events --field-selector type=Warning --sort-by=.lastTimestamp +run_check "OCI CNI pods" kubectl -n kube-system get pods -l app=oci-cni -o wide +run_check "Multus pods" kubectl -n kube-system get pods -l name=multus -o wide +run_check "network attachment definitions" kubectl get network-attachment-definitions -A + +python3 - "$records" "$namespace" "$pod" "$selector" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, pod, selector = sys.argv[2], sys.argv[3], sys.argv[4] +anomaly_re = re.compile( + r"(FailedCreatePodSandBox|failed to find plugin|CNI|multus|ipam|network-status|ImageInspectError|EOF|timeout|NotReady|Pending|Error|Failed)", + re.I, +) + +findings = [] +anomalies = [] +snippets = [] +for item in records: + output = item["output"].strip() + if item["rc"] == 0: + findings.append(f"{item['name']} collected") + else: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if output: + snippets.append(f"$ {item['cmd']}\n{output[-800:]}") + for line in output.splitlines(): + if anomaly_re.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") + +print(json.dumps({ + "domain": "Pod Networking / OCI CNI / IPAM", + "namespace": namespace, + "pod": pod, + "selector": selector, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-private-endpoint-check.sh b/oci/oke/scripts/oke-private-endpoint-check.sh new file mode 100755 index 0000000..ed78bd3 --- /dev/null +++ b/oci/oke/scripts/oke-private-endpoint-check.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect private OKE API endpoint and kubeconfig connectivity signals. +# +# Usage: +# bash scripts/oke-private-endpoint-check.sh --cluster-id --region [--compartment-id ] + +cluster_id="" +region="" +compartment_id="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --cluster-id) require_value "$1" "${2:-}"; cluster_id="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + --compartment-id) require_value "$1" "${2:-}"; compartment_id="$2"; shift 2 ;; + -h|--help) + echo "usage: oke-private-endpoint-check.sh --cluster-id --region [--compartment-id ]" + exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done + +if [[ -z "$cluster_id" || -z "$region" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --cluster-id or --region." "Provide both --cluster-id and --region." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1"; shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], + }) + "\n") +PY +} + +command -v kubectl >/dev/null 2>&1 && { + run_check "current context" kubectl config current-context + run_check "cluster info" kubectl cluster-info + run_check "readyz" kubectl get --raw=/readyz?verbose + run_check "kubectl version" kubectl version --client +} + +if command -v oci >/dev/null 2>&1; then + run_check "OKE cluster" oci --region "$region" ce cluster get --cluster-id "$cluster_id" --output json + if [[ -n "$compartment_id" ]]; then + run_check "NSGs" oci --region "$region" network nsg list --compartment-id "$compartment_id" --all --output json + fi +fi + +python3 - "$records" "$cluster_id" "$region" "$compartment_id" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +cluster_id, region, compartment_id = sys.argv[2], sys.argv[3], sys.argv[4] +pat = re.compile(r"(Unable to connect|i/o timeout|connection refused|x509|Unauthorized|Forbidden|exec: executable oci failed|NotAuthorized|private|endpoint|NSG|security)", re.I) +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if pat.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "Private Cluster / API Endpoint Connectivity", + "cluster_id": cluster_id, + "region": region, + "compartment_id": compartment_id, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-workload-identity-check.sh b/oci/oke/scripts/oke-workload-identity-check.sh new file mode 100755 index 0000000..263ffb4 --- /dev/null +++ b/oci/oke/scripts/oke-workload-identity-check.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect OKE workload identity and pod OCI API auth evidence. +# +# Usage: +# bash scripts/oke-workload-identity-check.sh --namespace --serviceaccount [--pod ] [--tenancy-id ] + +namespace="" +serviceaccount="" +pod="" +tenancy_id="" + +emit_error() { + local exit_code="$1"; local error_code="$2"; local message="$3"; local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --serviceaccount|--sa) require_value "$1" "${2:-}"; serviceaccount="$2"; shift 2 ;; + --pod) require_value "$1" "${2:-}"; pod="$2"; shift 2 ;; + --tenancy-id) require_value "$1" "${2:-}"; tenancy_id="$2"; shift 2 ;; + -h|--help) echo "usage: oke-workload-identity-check.sh --namespace --serviceaccount [--pod ] [--tenancy-id ]"; exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done +if [[ -z "$namespace" || -z "$serviceaccount" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --namespace or --serviceaccount." "Provide both --namespace and --serviceaccount." +fi +command -v kubectl >/dev/null 2>&1 || emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." + +tmp_dir="$(mktemp -d)"; trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl"; : > "$records" +run_check() { + local name="$1"; shift + local out rc + set +e; out="$("$@" 2>&1)"; rc=$?; set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({"name": os.environ["NAME"], "cmd": os.environ["CMD"], "rc": int(os.environ["RC"]), "output": os.environ["OUT"][-4000:]}) + "\n") +PY +} + +run_check "service account" kubectl -n "$namespace" get serviceaccount "$serviceaccount" -o yaml +if [[ -n "$pod" ]]; then + run_check "pod describe" kubectl -n "$namespace" describe pod "$pod" + run_check "pod logs" sh -c "kubectl -n '$namespace' logs '$pod' --tail=200 2>&1 | egrep -i 'notauthorized|forbidden|401|403|principal|workload|token|oci' || true" +fi +if [[ -n "$tenancy_id" ]] && command -v oci >/dev/null 2>&1; then + run_check "IAM policies" oci iam policy list --compartment-id "$tenancy_id" --all --output json +fi + +python3 - "$records" "$namespace" "$serviceaccount" "$pod" "$tenancy_id" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, serviceaccount, pod, tenancy_id = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +pat = re.compile(r"(NotAuthorized|Forbidden|401|403|principal|workload|token|policy|serviceaccount|namespace|cluster|oci)", re.I) +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if pat.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "Workload Identity / OCI API From Pods", + "namespace": namespace, + "serviceaccount": serviceaccount, + "pod": pod, + "tenancy_id": tenancy_id, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/preflight-check.sh b/oci/oke/scripts/preflight-check.sh new file mode 100755 index 0000000..6fbf2f5 --- /dev/null +++ b/oci/oke/scripts/preflight-check.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# preflight-check.sh +# OCI CLI auth + tenancy + region + compartment validation +# +# Usage: bash preflight-check.sh +# Exit: 0 = success +# 1 = expected error (CLI not auth'd, no compartments found) +# 2 = unexpected error (CLI not installed, parse failure) +# +# Stderr: JSON {error_code, message, remediation, docs_url} +# Stdout: JSON {tenancy_ocid, home_region, regions:[], compartments:[]} + +set -euo pipefail + +# ── helpers ────────────────────────────────────────────────────────────────── + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="$5" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +TMP_SUBSCRIPTIONS_JSON="$(mktemp)" +TMP_COMPARTMENTS_JSON="$(mktemp)" +TMP_REGIONS_JSON="$(mktemp)" +TMP_COMPARTMENTS_OUT_JSON="$(mktemp)" +cleanup_tmp() { + rm -f "$TMP_SUBSCRIPTIONS_JSON" "$TMP_COMPARTMENTS_JSON" "$TMP_REGIONS_JSON" "$TMP_COMPARTMENTS_OUT_JSON" +} +trap cleanup_tmp EXIT + +# ── step 1: verify OCI CLI is installed ────────────────────────────────────── + +if ! command -v oci >/dev/null 2>&1; then + emit_error 2 \ + "OCI_CLI_NOT_INSTALLED" \ + "The OCI CLI is not installed or not on PATH." \ + "Install OCI CLI from https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm" \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm" +fi + +# ── step 2: verify OCI CLI is authenticated ─────────────────────────────────── + +if ! oci iam region-subscription list --output json >/dev/null 2>&1; then + emit_error 1 \ + "OCI_CLI_NOT_AUTHENTICATED" \ + "The OCI CLI is installed but not authenticated. Check ~/.oci/config." \ + "Run: oci setup config" \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliconfigure.htm" +fi + +# ── step 3: extract tenancy OCID and home region ───────────────────────────── + +RAW_SUBSCRIPTIONS=$(oci iam region-subscription list --output json 2>/dev/null) || RAW_SUBSCRIPTIONS='{"data":[]}' +printf '%s' "$RAW_SUBSCRIPTIONS" > "$TMP_SUBSCRIPTIONS_JSON" + +TENANCY_INFO=$(python3 - "$TMP_SUBSCRIPTIONS_JSON" <<'PYEOF' +import json +import sys +from pathlib import Path +raw = Path(sys.argv[1]).read_text() or '{"data":[]}' +data = json.loads(raw).get("data", []) +home = next((r for r in data if r.get("is-home-region")), None) +if not home: + print("ERROR") + sys.exit(1) +print(home.get("region-name", ""), home.get("tenancy-id", "")) +PYEOF +) || true + +if [ -z "$TENANCY_INFO" ] || [ "$TENANCY_INFO" = "ERROR" ]; then + emit_error 1 \ + "TENANCY_DISCOVERY_FAILED" \ + "Could not extract tenancy OCID and home region from OCI CLI output." \ + "Verify your ~/.oci/config has a valid tenancy OCID and region." \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliconfigure.htm" +fi + +HOME_REGION=$(echo "$TENANCY_INFO" | awk '{print $1}') +TENANCY_OCID=$(echo "$TENANCY_INFO" | awk '{print $2}') + +if [ -z "$TENANCY_OCID" ]; then + emit_error 1 \ + "TENANCY_OCID_MISSING" \ + "Tenancy OCID could not be determined from region-subscription output." \ + "Ensure your OCI CLI profile includes a valid tenancy_ocid in ~/.oci/config." \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliconfigure.htm" +fi + +# ── step 4: list subscribed regions ────────────────────────────────────────── + +REGIONS_JSON=$(python3 - "$TMP_SUBSCRIPTIONS_JSON" <<'PYEOF' +import json +import sys +from pathlib import Path +raw = Path(sys.argv[1]).read_text() or '{"data":[]}' +data = json.loads(raw).get("data", []) +regions = [{"region": r.get("region-name",""), "status": r.get("status","")} for r in data] +print(json.dumps(regions)) +PYEOF +) || REGIONS_JSON='[]' +printf '%s' "$REGIONS_JSON" > "$TMP_REGIONS_JSON" + +# ── step 5: list active compartments ───────────────────────────────────────── + +RAW_COMPARTMENTS=$(oci iam compartment list \ + --compartment-id "$TENANCY_OCID" \ + --compartment-id-in-subtree true \ + --all \ + --lifecycle-state ACTIVE \ + --output json 2>/dev/null) || RAW_COMPARTMENTS='{"data":[]}' +printf '%s' "$RAW_COMPARTMENTS" > "$TMP_COMPARTMENTS_JSON" + +COMPARTMENTS_JSON=$(python3 - "$TMP_COMPARTMENTS_JSON" "$TENANCY_OCID" <<'PYEOF' +import json +import sys +from pathlib import Path +raw = Path(sys.argv[1]).read_text() or '{"data":[]}' +data = json.loads(raw).get("data", []) +tenancy_ocid = sys.argv[2] +compartments = [{"name": c.get("name",""), "ocid": c.get("id",""), "parent": c.get("compartment-id","")} for c in data] +# prepend root compartment (the tenancy itself) +compartments.insert(0, {"name": "root (tenancy)", "ocid": tenancy_ocid, "parent": ""}) +print(json.dumps(compartments)) +PYEOF +) || COMPARTMENTS_JSON='[]' +printf '%s' "$COMPARTMENTS_JSON" > "$TMP_COMPARTMENTS_OUT_JSON" + +# ── step 6: emit structured output ─────────────────────────────────────────── + +python3 - "$TENANCY_OCID" "$HOME_REGION" "$TMP_REGIONS_JSON" "$TMP_COMPARTMENTS_OUT_JSON" <<'PYEOF' +import sys, json +from pathlib import Path +regions_raw = Path(sys.argv[3]).read_text() +compartments_raw = Path(sys.argv[4]).read_text() +result = { + "tenancy_ocid": sys.argv[1], + "home_region": sys.argv[2], + "regions": json.loads(regions_raw), + "compartments": json.loads(compartments_raw) +} +print(json.dumps(result, indent=2)) +PYEOF + +exit 0 diff --git a/oci/oke/scripts/validate-cidr.sh b/oci/oke/scripts/validate-cidr.sh new file mode 100755 index 0000000..c32dac3 --- /dev/null +++ b/oci/oke/scripts/validate-cidr.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# validate-cidr.sh +# Detect CIDR overlap between 2 or more CIDR blocks +# +# Usage: bash validate-cidr.sh [cidr3 ...] +# Exit: 0 = no overlaps found +# 1 = overlaps detected +# 2 = invalid input (bad CIDR format or insufficient arguments) +# +# Stderr: JSON {error_code, message, remediation, docs_url} (exit 2 only) +# Stdout: JSON {valid: bool, overlaps: [{cidr_a, cidr_b}], cidrs_checked: []} + +set -euo pipefail + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +if [ "$#" -lt 2 ]; then + emit_error 2 \ + "INSUFFICIENT_ARGUMENTS" \ + "validate-cidr.sh requires at least 2 CIDR arguments." \ + "Usage: bash validate-cidr.sh 10.0.0.0/16 10.244.0.0/16 10.96.0.0/16" +fi + +# Delegate all parsing and overlap detection to Python +python3 - "$@" <<'PYEOF' +import sys, json, ipaddress + +cidrs_raw = sys.argv[1:] +cidrs = [] + +for raw in cidrs_raw: + try: + net = ipaddress.ip_network(raw, strict=False) + cidrs.append((raw, net)) + except ValueError as e: + error = { + "error_code": "INVALID_CIDR", + "message": f"Invalid CIDR block: {raw}. {e}", + "remediation": "Provide a valid CIDR in the form A.B.C.D/prefix (e.g. 10.0.0.0/16).", + "docs_url": "" + } + print(json.dumps(error), file=sys.stderr) + sys.exit(2) + +overlaps = [] +for i in range(len(cidrs)): + for j in range(i + 1, len(cidrs)): + raw_a, net_a = cidrs[i] + raw_b, net_b = cidrs[j] + if net_a.overlaps(net_b): + overlaps.append({"cidr_a": raw_a, "cidr_b": raw_b}) + +result = { + "valid": len(overlaps) == 0, + "overlaps": overlaps, + "cidrs_checked": cidrs_raw +} +print(json.dumps(result, indent=2)) +sys.exit(1 if overlaps else 0) +PYEOF diff --git a/oci/oke/shared/oci-resource-map.md b/oci/oke/shared/oci-resource-map.md new file mode 100644 index 0000000..c0a0d12 --- /dev/null +++ b/oci/oke/shared/oci-resource-map.md @@ -0,0 +1,138 @@ +# OCI Resource Mapping Cheatsheet + +Use these command chains to relate Kubernetes objects to Oracle Cloud Infrastructure resources during troubleshooting. + +## Automated Correlation +Use the object correlator first when you have a namespace and one or more target selectors. It runs read-only `kubectl` and `oci` lookups and returns a normalized graph with Kubernetes nodes, OCI nodes, and edges between them. + +```bash +./scripts/oke-object-correlator.sh \ + --namespace \ + --cluster-id \ + --compartment-id \ + --region \ + [--pod ] \ + [--deployment ] \ + [--service ] \ + [--ingress ] \ + [--pvc ] \ + [--node ] +``` + +Expected graph edges include: +- `scheduled_on`: Pod to Kubernetes node. +- `runs_on_instance`: Kubernetes node to OCI Compute instance. +- `has_vnic`: OCI instance to each primary or secondary VNIC attachment. +- `attached_to_subnet`: VNIC to subnet. +- `uses_nsg`: VNIC to Network Security Group. +- `uses_security_list`: Subnet to security list. +- `uses_route_table`: Subnet to route table. +- `routes_to`: Route table to gateway, DRG, private IP, or peering target. +- `provisions`: Kubernetes Service or Ingress to OCI Load Balancer. +- `bound_to` and `backs_onto`: PVC to PV to OCI Block Volume. + +Use the manual command chains below when the correlator cannot resolve a link or when you need to inspect a specific object in more detail. + +## Pod → Node → Instance +1. Identify the node hosting the pod: + ```bash + kubectl get pod -n -o jsonpath='{.spec.nodeName}' + ``` +2. Fetch node annotations for the instance OCID: + ```bash + kubectl get node -o jsonpath='{.metadata.annotations."node\.oci\.oraclecloud\.com/instance-id"}' + ``` +3. Inspect the OCI instance: + ```bash + oci compute instance get --instance-id + ``` +4. List VNIC attachments for the instance: + ```bash + oci compute vnic-attachment list \ + --compartment-id \ + --instance-id \ + --all + ``` +5. Inspect each VNIC and its subnet path: + ```bash + oci network vnic get --vnic-id + oci network subnet get --subnet-id + oci network security-list get --security-list-id + oci network route-table get --rt-id + ``` + +## Service / Ingress → Load Balancer +1. Obtain OCI load balancer OCID from annotations: + ```bash + kubectl get svc -n -o jsonpath='{.metadata.annotations."oci\.oraclecloud\.com/load-balancer-id"}' + ``` +2. Describe load balancer health: + ```bash + oci lb load-balancer get --load-balancer-id + ``` +3. Review backend set health: + ```bash + oci lb load-balancer-health get \ + --load-balancer-id + ``` + For a specific backend set: + ```bash + oci lb backend-set-health get \ + --load-balancer-id \ + --backend-set-name + ``` + +## PersistentVolumeClaim → Block Volume / File System +1. Retrieve PV name and volume handle: + ```bash + kubectl get pvc -n -o jsonpath='{.spec.volumeName} {.spec.volumeHandle}' + ``` +2. If Block Volume: + ```bash + oci bv volume get --volume-id + ``` + For attachment details: + ```bash + oci compute volume-attachment list --compartment-id --volume-id + ``` +3. If File Storage (FSS): + ```bash + oci fs file-system get --file-system-id + ``` + +## Namespace / Service Account → IAM Policies +1. Locate relevant IAM policy statements: + ```bash + oci iam policy list --compartment-id --all \ + --query "data[].{name:name, statements:statements}" + ``` +2. Cross-check Kubernetes service account annotations and projected token behavior: + ```bash + kubectl get serviceaccount -n -o yaml + ``` + +## Node Pool → Availability Domains +1. Identify node pool OCID: + ```bash + oci ce node-pool list --compartment-id --cluster-id + ``` +2. Fetch node pool details: + ```bash + oci ce node-pool get --node-pool-id + ``` +3. Map to AD and subnet IDs; confirm subnet health via: + ```bash + oci network subnet get --subnet-id + ``` + +Keep this document in sync with any future automation covering additional OCI resource types. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/Balance/home.htm +- https://docs.oracle.com/en-us/iaas/Content/Block/Concepts/overview.htm +- https://docs.oracle.com/en-us/iaas/Content/File/Concepts/filestorageoverview.htm +- https://docs.oracle.com/en-us/iaas/Content/Identity/policieshow/Policy_Basics.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/ +- https://kubernetes.io/docs/reference/kubectl/ diff --git a/oci/oke/skills/AGENTS.md b/oci/oke/skills/AGENTS.md new file mode 100644 index 0000000..66f540e --- /dev/null +++ b/oci/oke/skills/AGENTS.md @@ -0,0 +1,38 @@ +# Skills Directory Instructions + +## Skill Structure + +Every skill must have: + +- `SKILL.md` with YAML frontmatter containing `name` and `description`. +- A concise body that explains the core workflow and points to references or scripts only when needed. + +Optional files: + +- `references/` for longer notes, examples, constraints, and troubleshooting details. +- `scripts/` for deterministic helpers that are easier to test than prose-only workflows. +- `agents/openai.yaml` for Codex-facing skill UI metadata when useful. + +## Skill Authoring Rules + +- Keep `SKILL.md` focused on procedure, guardrails, and when to load extra resources. +- Put large examples, detailed troubleshooting notes, and edge-case explanations in `references/`. +- Prefer discovery helpers over asking users for raw OCIDs when the environment can enumerate choices. +- Make scripts safe to run offline where possible; destructive OCI or Kubernetes actions must require explicit user confirmation in the skill workflow. +- Public examples must use placeholders instead of real tenancy values, cluster OCIDs, private IPs, or customer names. +- When a skill relies on another skill, state the handoff clearly. For example, multihome pod validation should tell users to run the GVA deployer first if the node pool does not yet have GVA secondary VNIC profiles. + +## Validation + +After changing skill scripts, run: + +```bash +bash tests/scripts-smoke.sh +``` + +If adding a new OKE skill, update `oci/SKILL.md`, the root `README.md`, and the relevant `oci/oke/*.md` routing file. + +## Sources + +- https://github.com/oracle/skills/blob/main/SKILL_AUTHORING_GUIDE.md +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm diff --git a/oci/oke/skills/oke-cluster-generator/SKILL.md b/oci/oke/skills/oke-cluster-generator/SKILL.md new file mode 100644 index 0000000..1f351bc --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/SKILL.md @@ -0,0 +1,172 @@ +--- +name: oke-cluster-generator +description: Use this skill when the user asks to build, generate, create, design, or scaffold an OKE (OCI Kubernetes Engine) Terraform stack, OCI Kubernetes infrastructure, ORM schema, or Resource Manager template. Trigger phrases include "build an OKE stack", "create OKE Terraform", "generate ORM schema", "deploy OKE cluster", "OKE infrastructure", "terraform-oci-oke", or any request to design OCI Kubernetes infrastructure with Terraform. Do not use it for active incident RCA, GVA secondary VNIC node-pool creation, or Multus pod manifest validation; use the troubleshooting, GVA, or multihome skills for those surfaces. +--- + +## OCI OKE Domain Context + +This operational skill is part of the OCI OKE domain. Route here from `oci/SKILL.md` or `oci/oke/cluster-design.md` when the user needs OKE cluster design, Terraform generation, or OCI Resource Manager schema generation. + +Supporting files: +- `references/questionnaire.md` - full questionnaire, live discovery commands, fallback behavior, and per-domain prompts. Read before running the full workflow. +- `reference.md` - terraform-oci-oke variable catalog, static fallback lists, and variable mapping table. Read before code generation. +- `output-templates/terraform.md` - provider.tf, main.tf module call, and outputs.tf templates. +- `output-templates/schema.md` - ORM schema.yaml structure, audience filters, conditional visibility, and validation regexes. + +Utility scripts: +- `../../scripts/preflight-check.sh` - OCI CLI auth, tenancy, region, and compartment discovery. +- `../../scripts/validate-cidr.sh` - CIDR overlap detection for VCN, pod, and service CIDRs. + +## Role + +You are an expert OCI Infrastructure Architect specializing in OCI Kubernetes Engine (OKE) cluster design and Terraform automation. Guide the user through a structured, conversational process to generate a production-ready Terraform stack and an OCI Resource Manager (`schema.yaml`) bundle. + +Use these authoritative sources for module structure and OCI behavior: +- OCI OKE Terraform Module: https://github.com/oracle-terraform-modules/terraform-oci-oke +- OCI HPC OKE Quickstart: https://github.com/oracle-quickstart/oci-hpc-oke +- OCI Terraform Provider: https://github.com/oracle/terraform-provider-oci +- OKE Official Documentation: https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm + +Only use these sources and pages linked from them. Do not perform general web searches for this workflow. + +## Argument Pre-fill + +If `$ARGUMENTS` is non-empty, parse it before preflight: + +| Pattern | Matches | Variable | +|---------|---------|----------| +| `FAST_PATH` | `fast-path`, `fastpath`, `quick-start`, `quickstart`, `starter-stack`, `starter`, `minimal` | `FAST_PATH_MODE = true`; consume before cluster-name parsing | +| `WORKLOAD_TYPE` | `ai`, `ai/ml`, `aiml`, `ml`, `gpu`, `hpc`, `rdma`, `microservices`, `micro`, `general` | `WORKLOAD_TYPE` | +| `TARGET_REGION` | OCI region pattern such as `us-ashburn-1` | `TARGET_REGION` | +| `CLUSTER_NAME` | Remaining token | Suggested cluster name; confirm with user | + +Canonical workload mapping: +- `ai`, `ai/ml`, `aiml`, `ml`, `gpu` -> `AI / ML` +- `hpc`, `rdma` -> `HPC` +- `microservices`, `micro` -> `Microservices` +- everything else -> `General Purpose` + +If any values were pre-filled, tell the user what was detected and say they can revise values at any domain summary. If `FAST_PATH_MODE = true`, do not treat the fast-path token as `CLUSTER_NAME`. + +## Fast Path Mode + +Use Fast Path Mode when the user asks for a quick start, starter stack, fast path, or minimal questions. Ask only for values that cannot be safely inferred: +- `TENANCY_OCID` +- `COMPARTMENT_OCID` +- `TARGET_REGION` +- `CLUSTER_NAME` + +Fast Path defaults: + +| Area | Default | +|------|---------| +| Workload type | General Purpose | +| Kubernetes version | Latest GA from OCI CLI, static fallback from `reference.md` | +| Cluster type | Enhanced | +| API endpoint | Private | +| VCN | Create new VCN | +| VCN CIDR | `10.0.0.0/16` | +| Pod CIDR | `10.244.0.0/16` | +| Service CIDR | `10.96.0.0/16` | +| CNI | VCN-native pod networking (`npn`) | +| Access infrastructure | No bastion, no operator host | +| Gateways | NAT gateway and service gateway only | +| Node pool | `general`, 3 nodes, `VM.Standard.E5.Flex`, 2 OCPUs, 16 GB | +| Storage | Block Volume CSI enabled, FSS disabled | +| Security | Create IAM policies, Oracle-managed encryption | +| Workload Identity | Enabled | +| Add-ons | CoreDNS and kube-proxy managed add-ons | +| ORM audience | Expert | + +Fast Path workflow: +1. Announce that Fast Path defaults are being used. +2. Run preflight when OCI CLI is available. +3. Collect missing required inputs directly. +4. Present a compact architecture summary table with every default. +5. Ask the user to confirm, revise, or switch to the full questionnaire. +6. On confirmation, proceed to code generation. + +## Full Workflow + +Read `references/questionnaire.md` before running the full questionnaire. It contains the detailed domain prompts, CLI commands, and fallback rules for: +- Cluster fundamentals. +- Networking. +- Node pools. +- Storage. +- Security and access. +- Add-ons and observability. +- ORM schema preferences. + +The full workflow has four phases: + +1. **Preflight and discovery** + - Run `bash ../../scripts/preflight-check.sh` when OCI CLI is available. + - Populate tenancy, home region, subscribed regions, and compartments. + - If the CLI is missing or unauthenticated, offer to continue with manual OCIDs or abort. + +2. **Guided design** + - Work one domain at a time. + - Prefer live tenancy data via OCI CLI before static fallback lists. + - Use `AskUserQuestion` for fixed-choice questions when available. + - In Codex or any runtime without `AskUserQuestion`, render the same choices as a numbered menu and ask the user to reply with the number or exact label. + - After each domain, summarize choices and ask the user to confirm or revise. + - Track session state variables named in `references/questionnaire.md`. + +3. **Architecture summary** + - Summarize cluster topology, key design decisions, cost or service-limit warnings, known constraints, and any `CLI_*_FALLBACK` flags. + - Ask the user to confirm or revise before generating artifacts. + +4. **Artifact generation** + - Read `reference.md` and map every user answer to the exact Terraform variable name. + - Read `output-templates/terraform.md` and `output-templates/schema.md`. + - Generate `provider.tf`, `variables.tf`, `main.tf`, `outputs.tf`, `terraform.tfvars.example`, and `schema.yaml`. + - Remove template lines that do not apply. Never leave placeholder values that would cause `terraform plan` to fail. + +## Behavioral Rules + +- Explain why a configuration choice matters before asking the user to decide. +- Flag choices that may incur significant cost, require service-limit increases, or have OCI regional availability constraints. +- Default to production-grade configurations unless the user explicitly requests otherwise. +- Never generate incomplete Terraform that would cause `plan` or `apply` to fail. +- Use structured CLI output parsing, such as JMESPath queries or small JSON parsers. Never dump raw OCI JSON to the user. +- Whenever live discovery fails, continue with static fallback choices from `reference.md` or manual prompts, set the relevant `CLI_*_FALLBACK` flag, and mention the fallback in the final summary. +- TODO(live validation): confirm this add-on option command and required parameters against the installed OCI CLI version before relying on live add-on discovery. + +## Code Generation Rules + +Before generating any code: +1. Read `reference.md` for the Variable Mapping table. +2. Read `output-templates/terraform.md` for Terraform file structure. +3. Read `output-templates/schema.md` for Resource Manager schema structure and conditional visibility. + +Generate: +- `provider.tf` +- `variables.tf` +- `main.tf` +- `outputs.tf` +- `terraform.tfvars.example` +- `schema.yaml` + +For ORM audience filtering: + +| Audience | Expose | Hide | +|----------|--------|------| +| Expert | All variable groups | Nothing | +| App team | Cluster Fundamentals only | Networking, Storage, Security, Add-ons | +| Ops team | Cluster Fundamentals + Add-ons and Observability | Networking, Storage, Security | +| Minimal | tenancy_ocid, compartment_ocid, region only | All others, with safe defaults | + +After delivering artifacts, offer targeted refinements such as additional add-ons, GitOps bootstrapping, RDMA/SR-IOV device plugin manifests, or operational Makefile targets. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingclusterusingoke_topic-Using_the_Console_to_create_a_Custom_Cluster_with_Explicitly_Defined_Settings.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengworkingwithenhancedclusters.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengintroducingclusteraddons.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +- https://docs.oracle.com/en-us/iaas/Content/ResourceManager/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ResourceManager/Concepts/terraformconfigresourcemanager_topic-schema.htm +- https://github.com/oracle-terraform-modules/terraform-oci-oke +- https://registry.terraform.io/providers/oracle/oci/latest/docs diff --git a/oci/oke/skills/oke-cluster-generator/agents/openai.yaml b/oci/oke/skills/oke-cluster-generator/agents/openai.yaml new file mode 100644 index 0000000..4fe5700 --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "OKE Cluster Generator" + short_description: "Design OKE Terraform and Resource Manager stacks." + default_prompt: "Use $oke-cluster-generator to design an OKE cluster, validate network and node-pool choices, and generate Terraform plus OCI Resource Manager schema artifacts." diff --git a/oci/oke/skills/oke-cluster-generator/output-templates/schema.md b/oci/oke/skills/oke-cluster-generator/output-templates/schema.md new file mode 100644 index 0000000..4f7c61c --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/output-templates/schema.md @@ -0,0 +1,217 @@ +# ORM Schema Templates + +Use these structures and patterns during Phase 3 when generating `schema.yaml`. +Apply audience filtering rules from Domain 7 to decide which variable groups to surface. + +## Contents +- schema.yaml Top-Level Structure +- Audience Filtering (from Domain 7) +- Conditional Visibility Patterns +- Input Validation Patterns +- Per-Variable Schema Entry Example + +--- + +## schema.yaml Top-Level Structure + +```yaml +title: "OKE Terraform Stack" +description: "Deploy an OKE cluster with customizable networking, node pools, and security." +schemaVersion: "1.1.0" +version: "1.0.0" +locale: "en" +logoUrl: "" + +variableGroups: + - title: "Cluster Fundamentals" + variables: + - tenancy_ocid + - compartment_ocid + - region + - cluster_name + - kubernetes_version + - cluster_type + - control_plane_is_public + + - title: "Networking" + variables: + - create_vcn + - vcn_cidrs + - existing_vcn_ocid + - pods_cidr + - services_cidr + - cni_type + - create_bastion + - create_operator + - create_nat_gateway + - create_service_gateway + - create_internet_gateway + + - title: "Node Pools" + variables: + - node_pools + + - title: "Storage" + variables: + - block_volume_csi_enabled + - fss_enabled + + - title: "Security & Access" + variables: + - create_policies + - kms_key_id + - workload_identity_enabled + + - title: "Add-ons & Observability" + variables: + - cluster_addons + - oke_logging_enabled + - oke_monitoring_enabled +``` + +--- + +## Audience Filtering (from Domain 7) + +| Audience | Expose these groups | Hide these groups | +|----------|--------------------|--------------------| +| Expert — all variables | All 6 groups | None | +| App team — simplified | Cluster Fundamentals only | Networking, Storage, Security, Add-ons | +| Ops team — operational | Cluster Fundamentals, Add-ons & Observability | Networking, Storage, Security | +| Minimal — required only | Cluster Fundamentals (tenancy_ocid, compartment_ocid, region only) | All others; set everything else to `required: false` with a sensible default | + +--- + +## Conditional Visibility Patterns + +### Show field only for private control plane: + +```yaml +visible: + and: + - eq: + - ${control_plane_is_public} + - false +``` + +### Show field only for Enhanced cluster: + +```yaml +visible: + and: + - eq: + - ${cluster_type} + - "enhanced" +``` + +### Show field only when creating a new VCN: + +```yaml +visible: + and: + - eq: + - ${create_vcn} + - true +``` + +### Show field only when using an existing VCN: + +```yaml +visible: + and: + - eq: + - ${create_vcn} + - false +``` + +### Show GPU-specific fields (any node pool shape contains "GPU"): + +```yaml +visible: + or: + - contains: + - ${node_pool_shape} + - "GPU" + - contains: + - ${node_pool_shape} + - "H100" + - contains: + - ${node_pool_shape} + - "A100" +``` + +### Show KMS key field only when BYOK is enabled: + +```yaml +visible: + and: + - eq: + - ${byok_enabled} + - true +``` + +--- + +## Input Validation Patterns + +### CIDR block validation: + +```yaml +pattern: "^([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}$" +``` + +### Kubernetes version validation: + +```yaml +pattern: "^v[0-9]+\.[0-9]+(\.[0-9]+)?$" +``` + +### OCI region identifier validation: + +```yaml +pattern: "^[a-z]{2}-[a-z]+-[0-9]$" +``` + +### OCID validation: + +```yaml +pattern: "^ocid1\.[a-z]+\.oc[0-9]+\.[a-z0-9]*\.[a-z0-9]+" +``` + +--- + +## Per-Variable Schema Entry Example + +```yaml +variables: + kubernetes_version: + title: "Kubernetes Version" + description: "OKE Kubernetes version to deploy. Use the latest GA release unless a specific version is required." + type: string + required: true + default: "v1.32" + pattern: "^v[0-9]+\.[0-9]+(\.[0-9]+)?$" + + cni_type: + title: "CNI Plugin" + description: "Container network interface plugin. VCN-Native (npn) is recommended for production." + type: enum + required: true + default: "npn" + enum: + - npn + - flannel + + control_plane_is_public: + title: "API Endpoint Visibility" + description: "Whether the Kubernetes API server endpoint is publicly accessible." + type: boolean + required: true + default: false +``` + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ResourceManager/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ResourceManager/Concepts/terraformconfigresourcemanager_topic-schema.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm diff --git a/oci/oke/skills/oke-cluster-generator/output-templates/terraform.md b/oci/oke/skills/oke-cluster-generator/output-templates/terraform.md new file mode 100644 index 0000000..e3fd93b --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/output-templates/terraform.md @@ -0,0 +1,138 @@ +# Terraform Templates + +Use these as the base structure during Phase 3 code generation. Populate all variable +bindings from the user's answers using the mapping table in `reference.md`. +Never leave `# ...` placeholder comments or stub values that would cause `terraform plan` +to fail. + +## Contents +- provider.tf +- main.tf — Module Call +- outputs.tf + +--- + +## provider.tf + +```hcl +terraform { + required_version = ">= 1.3.0" + required_providers { + oci = { + source = "oracle/oci" + version = ">= 5.0.0" + } + } +} + +provider "oci" { + tenancy_ocid = var.tenancy_ocid + user_ocid = var.user_ocid + fingerprint = var.fingerprint + private_key_path = var.private_key_path + region = var.region +} +``` + +--- + +## main.tf — Module Call + +```hcl +module "oke" { + source = "oracle-terraform-modules/oke/oci" + version = "~> 5.0" + + # ── Cluster Fundamentals (Domain 1) ───────────────────────────────────── + tenancy_id = var.tenancy_ocid + compartment_id = var.compartment_ocid + region = var.region + cluster_name = var.cluster_name + kubernetes_version = var.kubernetes_version + cluster_type = var.cluster_type + control_plane_is_public = var.control_plane_is_public + + # ── Networking (Domain 2) ──────────────────────────────────────────────── + create_vcn = var.create_vcn + vcn_cidrs = var.vcn_cidrs # new VCN only + vcn_id = var.existing_vcn_ocid # existing VCN only + pods_cidr = var.pods_cidr + services_cidr = var.services_cidr + cni_type = var.cni_type + create_bastion = var.create_bastion + create_operator = var.create_operator + create_nat_gateway = var.create_nat_gateway + create_service_gateway = var.create_service_gateway + create_internet_gateway = var.create_internet_gateway + + # ── Node Pools (Domain 3) ──────────────────────────────────────────────── + # Expand this map for each pool defined during the questionnaire. + # See reference.md § Variable Mapping for per-pool variable structure. + node_pools = var.node_pools + + # ── Storage (Domain 4) ────────────────────────────────────────────────── + block_volume_csi_enabled = var.block_volume_csi_enabled + fss_enabled = var.fss_enabled + + # ── Security & Access (Domain 5) ──────────────────────────────────────── + create_policies = var.create_policies + kms_key_id = var.kms_key_id + workload_identity_enabled = var.workload_identity_enabled + + # ── Add-ons (Domain 6) ────────────────────────────────────────────────── + cluster_addons = var.cluster_addons +} +``` + +TODO(live validation): Before using this template for an operator-ready bundle, +confirm the current `terraform-oci-oke` module major version and variable names +against the approved module source. If a variable has changed, update +`reference.md` and this template together instead of guessing. + +Remove lines for variables not applicable to this deployment (e.g., remove `kms_key_id` +if Oracle-managed encryption was chosen, remove `vcn_id` if creating a new VCN, etc.). + +--- + +## outputs.tf + +```hcl +output "cluster_id" { + description = "OCID of the OKE cluster" + value = module.oke.cluster_id +} + +output "kubeconfig_cmd" { + description = "Command to generate kubeconfig for this cluster" + value = "oci ce cluster create-kubeconfig --cluster-id ${module.oke.cluster_id} --region ${var.region} --token-version 2.0.0" +} + +output "node_pool_ids" { + description = "Map of node pool name to OCID" + value = module.oke.node_pool_ids +} + +output "vcn_id" { + description = "OCID of the VCN (created or existing)" + value = module.oke.vcn_id +} + +output "bastion_public_ip" { + description = "Public IP of the bastion host (null if not created)" + value = try(module.oke.bastion_public_ip, null) +} + +output "operator_private_ip" { + description = "Private IP of the operator instance (null if not created)" + value = try(module.oke.operator_private_ip, null) +} +``` + +Omit `bastion_public_ip` and `operator_private_ip` if neither was created (both +`create_bastion` and `create_operator` are false). + +## Sources + +- https://github.com/oracle-terraform-modules/terraform-oci-oke +- https://registry.terraform.io/providers/oracle/oci/latest/docs +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm diff --git a/oci/oke/skills/oke-cluster-generator/reference.md b/oci/oke/skills/oke-cluster-generator/reference.md new file mode 100644 index 0000000..1ded700 --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/reference.md @@ -0,0 +1,110 @@ +# OKE Terraform Stack Builder — Reference Data + +This file contains static fallback lists and the authoritative variable mapping table. +Read this file during the questionnaire when a CLI call fails, and at the start of Phase 3 +code generation to ensure correct `terraform-oci-oke` module variable names. + +## Contents +- Static Kubernetes Versions +- Static OKE Managed Add-ons +- Variable Mapping: User Selection → terraform-oci-oke Module Variables + +--- + +## Static Kubernetes Versions + +Use this list when `oci ce cluster-options get` fails or returns empty output. +Mark the first entry as "Latest GA (Recommended)". + +| Label | Description | +|-------|-------------| +| `v1.32` (Latest GA) | Recommended — newest OKE-supported release with the longest support window. | +| `v1.31` | Previous GA release; stable and widely tested on OKE. | +| `v1.30` | Older GA release; choose only if a specific dependency requires it. | +| `Other (specify)` | Enter a custom version string in the follow-up free-text prompt. | + +--- + +## Static OKE Managed Add-ons + +Use this list when live add-on discovery fails or returns empty output. TODO(live validation): +confirm the exact `oci ce addon-option` command and required parameters for the installed +OCI CLI version before relying on live discovery. +These add-ons apply to Enhanced clusters only. + +| Name | Description | +|------|-------------| +| `CoreDNS` | Cluster DNS — always required; managed version receives automatic patch updates. | +| `Kube-proxy` | Network proxy — always required; managed version receives automatic patch updates. | +| `Cluster Autoscaler` | Automatically scales node pools based on pending pod demand. | +| `OCI Native Ingress Controller` | Provisions OCI Load Balancers directly from Ingress resources; no NGINX required. | +| `DCGM Exporter` | NVIDIA DCGM exporter DaemonSet for GPU metrics (AI/ML workloads). | +| `Multus` | Multiple network interfaces per pod via the Multus CNI meta-plugin. | + +--- + +## Variable Mapping: User Selection → terraform-oci-oke Module Variables + +Read this table during Phase 3 code generation. Use the exact variable names shown in the +"Terraform Variable" column — do not guess or infer variable names from the module README. +TODO(live validation): Reconcile this mapping with the currently approved +`terraform-oci-oke` module version before claiming generated Terraform has been +plan-verified. If live validation is unavailable, call out the mapping as a +module-version assumption in the generated summary. + +| Domain | User Selection / Answer | Terraform Variable | Type | Notes | +|--------|------------------------|-------------------|------|-------| +| D1 | Kubernetes version (e.g. `v1.32`) | `kubernetes_version` | string | Quote the string: `"v1.32.1"` | +| D1 | API visibility: Private | `control_plane_is_public = false` | bool | | +| D1 | API visibility: Public | `control_plane_is_public = true` | bool | | +| D1 | Cluster type: Enhanced | `cluster_type = "enhanced"` | string | | +| D1 | Cluster type: Basic | `cluster_type = "basic"` | string | | +| D2 | Create new VCN | `create_vcn = true` | bool | | +| D2 | Use existing VCN | `create_vcn = false`, `vcn_id = var.existing_vcn_ocid` | bool + string | | +| D2 | VCN CIDR | `vcn_cidrs = [""]` | list(string) | New VCN only | +| D2 | Pod CIDR | `pods_cidr = ""` | string | Always required | +| D2 | Service CIDR | `services_cidr = ""` | string | Always required | +| D2 | CNI: VCN-Native Pod Networking | `cni_type = "npn"` | string | | +| D2 | CNI: Flannel | `cni_type = "flannel"` | string | | +| D2 | Bastion + Operator | `create_bastion = true`, `create_operator = true` | bool | | +| D2 | Bastion only | `create_bastion = true`, `create_operator = false` | bool | | +| D2 | Operator only | `create_bastion = false`, `create_operator = true` | bool | | +| D2 | No access infra | `create_bastion = false`, `create_operator = false` | bool | | +| D2 | NAT Gateway | `create_nat_gateway = true` | bool | | +| D2 | Service Gateway | `create_service_gateway = true` | bool | | +| D2 | Internet Gateway | `create_internet_gateway = true` | bool | | +| D3 | Shape name (per pool) | `node_pools..shape` | string | e.g. `"VM.Standard.E4.Flex"` | +| D3 | Flex OCPUs (per pool) | `node_pools..ocpus` | number | Flex shapes only | +| D3 | Flex memory GB (per pool) | `node_pools..memory` | number | Flex shapes only | +| D3 | Fixed count (per pool) | `node_pools..node_pool_size` | number | | +| D3 | Autoscaling enabled | `node_pools..autoscale = true` | bool | | +| D3 | Autoscaling min | `node_pools..min_node_count` | number | | +| D3 | Autoscaling max | `node_pools..max_node_count` | number | | +| D3 | Boot: Higher Performance | `node_pools..boot_volume_vpus_per_gb = 20` | number | | +| D3 | Boot: Balanced | `node_pools..boot_volume_vpus_per_gb = 10` | number | | +| D3 | OS: OKE-optimised | (no override — use OKE default image) | — | | +| D3 | OS: Custom image OCID | `node_pools..image_id = ""` | string | | +| D4 | Block Volume CSI | `block_volume_csi_enabled = true` | bool | | +| D4 | OCI File Storage (FSS) | `fss_enabled = true` | bool | | +| D5 | Instance Principals | (default — no extra variable) | — | | +| D5 | Auto-generate IAM policies | `create_policies = true` | bool | | +| D5 | Manual IAM policies | `create_policies = false` | bool | | +| D5 | Kubernetes PSA | `psp_enabled = false` (use native K8s PSA) | bool | | +| D5 | BYOK encryption key | `kms_key_id = var.kms_key_id` | string | | +| D5 | Oracle-managed encryption | (no variable — OCI default) | — | | +| D5 | Workload Identity enabled | `workload_identity_enabled = true` | bool | Enhanced only | +| D6 | CoreDNS managed add-on | `cluster_addons = { coredns = { remove = false } }` | map | Enhanced only | +| D6 | Cluster Autoscaler add-on | `cluster_addons = { cluster-autoscaler = { remove = false } }` | map | Enhanced only | +| D6 | OCI Native Ingress add-on | `cluster_addons = { oci-native-ingress-controller = { remove = false } }` | map | Enhanced only | +| D6 | OCI Logging | `oke_logging_enabled = true` | bool | | +| D6 | OCI Monitoring | `oke_monitoring_enabled = true` | bool | | + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengworkingwithenhancedclusters.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengintroducingclusteraddons.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengconfiguringclusteraddons.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://github.com/oracle-terraform-modules/terraform-oci-oke +- https://registry.terraform.io/providers/oracle/oci/latest/docs diff --git a/oci/oke/skills/oke-cluster-generator/references/questionnaire.md b/oci/oke/skills/oke-cluster-generator/references/questionnaire.md new file mode 100644 index 0000000..4779804 --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/references/questionnaire.md @@ -0,0 +1,273 @@ +# OKE Cluster Generator Questionnaire + +Use this reference when running the full OKE cluster-generation workflow. Keep `SKILL.md` focused and load this file only when the user needs the full questionnaire rather than Fast Path Mode. + +## Contents +- Session State +- Preflight +- Domain 1 - Cluster Fundamentals +- Domain 2 - Networking +- Domain 3 - Node Pools +- Domain 4 - Storage +- Domain 5 - Security and Access +- Domain 6 - Add-ons and Observability +- Domain 7 - ORM Schema Preferences +- Architecture Summary +- Code Generation Notes + +## Session State + +Track these values across all phases: + +| Variable | Purpose | +|----------|---------| +| `WORKLOAD_TYPE` | `AI / ML`, `HPC`, `Microservices`, or `General Purpose` | +| `KUBERNETES_VERSION` | OKE Kubernetes version string | +| `CLUSTER_TYPE` | `Enhanced` or `Basic` | +| `TARGET_REGION` | OCI region for all OCI CLI calls | +| `TENANCY_OCID` | Root tenancy compartment OCID | +| `HOME_REGION` | Tenancy home region | +| `COMPARTMENT_OCID` | Target compartment OCID | +| `VCN_SOURCE` | `new` or `existing` | +| `EXISTING_VCN_OCID` | Existing VCN when `VCN_SOURCE=existing` | +| `CNI_TYPE` | `npn` or `flannel` | +| `RDMA_ROCE_SELECTED` | True when RDMA/RoCE is selected | +| `NODE_POOL_COUNT` | Number of node pools | +| `POOL_NAME_i` | Node pool name per pool | +| `POOL_SHAPE_i` | Shape per pool | +| `VAULT_MANAGEMENT_ENDPOINT` | KMS vault endpoint | +| `KMS_KEY_ID` | Customer-managed key OCID | +| `WORKLOAD_IDENTITY_ENABLED` | Enhanced cluster workload identity setting | + +## Preflight + +Run: + +```bash +bash ../../scripts/preflight-check.sh +``` + +On success, parse: +- `.tenancy_ocid` -> `TENANCY_OCID` +- `.home_region` -> `HOME_REGION` +- `.regions[]` -> region options +- `.compartments[]` -> compartment options + +If the OCI CLI is missing or unauthenticated, offer: +- Continue without CLI and enter OCIDs manually. +- Abort. + +Never abort because one live discovery command fails. Set a `CLI_*_FALLBACK` flag and continue with static options or manual prompts. + +## Domain 1 - Cluster Fundamentals + +Fetch supported Kubernetes versions: + +```bash +oci ce cluster-options get --cluster-option-id all \ + --query 'data."kubernetes-versions"' \ + --output json +``` + +Ask: + +| Question | Options | +|----------|---------| +| Workload type | General Purpose, AI / ML, HPC, Microservices | +| Kubernetes version | Latest supported versions from CLI; fallback to `reference.md` | +| API endpoint | Private, Public | +| Cluster type | Enhanced, Basic | + +If the user chooses a custom Kubernetes version, ask for the exact version string. + +## Domain 2 - Networking + +Ask: + +| Question | Options | +|----------|---------| +| VCN source | Create new VCN, Use existing VCN | +| CNI plugin | VCN-native pod networking (`npn`), Flannel | +| Bastion/operator access | Bastion + Operator, Bastion only, Operator only, None | + +If using an existing VCN, list VCNs: + +```bash +oci network vcn list \ + --compartment-id "$COMPARTMENT_OCID" \ + --lifecycle-state AVAILABLE \ + --query 'data[*].{Name:"display-name",OCID:id,CIDR:"cidr-block"}' \ + --output json +``` + +Always collect: +- VCN CIDR or existing VCN CIDR. +- Pod CIDR. +- Service CIDR. + +Validate CIDRs: + +```bash +bash ../../scripts/validate-cidr.sh "" "" "" +``` + +If overlaps are detected, show the conflicting ranges and ask the user to revise before continuing. + +Ask for gateways: +- NAT gateway. +- Service gateway. +- Internet gateway. + +For AI / ML or HPC workloads, ask about additional network interfaces: +- RDMA / RoCE. +- SR-IOV. +- Multus multi-homed. +- None. + +Set `RDMA_ROCE_SELECTED=true` when RDMA/RoCE is selected. + +## Domain 3 - Node Pools + +Ask for `NODE_POOL_COUNT`, then repeat this section for each node pool. + +Fetch shape context before the loop: + +```bash +oci compute shape list \ + --compartment-id "$COMPARTMENT_OCID" \ + --query 'data[*].shape' \ + --output json + +oci limits value list \ + --compartment-id "$TENANCY_OCID" \ + --service-name compute \ + --all \ + --query 'data[?contains(name, `gpu`) || contains(name, `hpc`) || contains(name, `rdma`)]' \ + --output json +``` + +TODO(live validation): confirm the exact OCI Limits command and limit-name mapping for GPU/HPC/RDMA shape families in the target tenancy before treating a zero or missing value as conclusive. + +Per pool, ask: +- Pool name. +- Shape family: VM Standard, VM GPU, BM GPU, BM HPC. +- Scaling strategy: fixed count or autoscaling. +- Exact shape name. +- OCPUs and memory for Flex shapes. +- Node count or min/max values. +- Boot volume performance: Higher Performance or Balanced. +- OS image: OKE-optimized or custom image OCID. +- Cloud-init: none, inline script, or file path. + +Cross-checks: +- If RDMA/RoCE was selected, require a BM GPU or BM HPC shape unless the user explicitly revises the network choice or accepts the warning. +- If a GPU shape is selected for a non-AI/HPC workload, call out the cost and workload mismatch. +- Repeat service-limit warnings for selected GPU/HPC/RDMA families. + +## Domain 4 - Storage + +Ask for persistent storage backends: +- OCI Block Volume CSI. +- OCI File Storage. +- Object Storage. +- None. + +If a DenseIO or BM.HPC shape was selected, ask whether to configure an NVMe StorageClass. + +## Domain 5 - Security and Access + +Ask: + +| Question | Options | +|----------|---------| +| Node identity | Instance Principals, User credentials | +| IAM policies | Auto-generate, Manual | +| Pod security | Kubernetes Pod Security Admission, OCI Security Zones, Both, None | +| Volume encryption | Customer-managed key, Oracle-managed key | + +If customer-managed keys are selected, list vaults: + +```bash +oci kms management vault list \ + --compartment-id "$COMPARTMENT_OCID" \ + --lifecycle-state ACTIVE \ + --query 'data[*].{Name:"display-name",OCID:id,Endpoint:"management-endpoint"}' \ + --output json +``` + +Store the selected vault management endpoint, then list AES keys: + +```bash +oci kms management key list \ + --compartment-id "$COMPARTMENT_OCID" \ + --endpoint "$VAULT_MANAGEMENT_ENDPOINT" \ + --protection-mode HSM \ + --algorithm AES \ + --lifecycle-state ENABLED \ + --query 'data[*].{Name:"display-name",OCID:id}' \ + --output json +``` + +If `CLUSTER_TYPE=Enhanced`, ask whether to enable Workload Identity. + +## Domain 6 - Add-ons and Observability + +If `CLUSTER_TYPE=Basic`, skip OKE managed add-ons and explain that managed add-ons require Enhanced clusters. + +For Enhanced clusters, discover add-ons: + +```bash +oci ce addon-option list \ + --kubernetes-version "$KUBERNETES_VERSION" \ + --query 'data[*].{Name:name,Description:description}' \ + --output json +``` + +TODO(live validation): confirm this add-on option command and required parameters against the installed OCI CLI version before relying on live add-on discovery. + +Ask: +- OKE managed add-ons, multi-select, from CLI output or `reference.md`. +- Observability: OCI Logging, OCI Monitoring, Container Insights, None. +- GPU metrics, only if a GPU shape was selected: DCGM Exporter, OCI GPU Scanner, Both, None. + +## Domain 7 - ORM Schema Preferences + +Ask: + +| Question | Options | +|----------|---------| +| Target audience | Expert, App team, Ops team, Minimal | +| Schema features | Variable groups, Help text and descriptions, Input validation rules, Conditional visibility | + +## Architecture Summary + +After all domains are confirmed, summarize: +- Cluster topology. +- Key design decisions and rationale. +- Cost and service-limit warnings. +- Known constraints. +- CLI fallback flags. + +Ask the user to confirm or revise before code generation. + +## Code Generation Notes + +Generate artifacts only after confirmation: +- Read `reference.md` for exact variable mapping. +- Read `output-templates/terraform.md` for Terraform file shapes. +- Read `output-templates/schema.md` for ORM schema shape. +- Remove irrelevant template lines. +- Do not leave placeholders that would cause `terraform plan` to fail. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingclusterusingoke_topic-Using_the_Console_to_create_a_Custom_Cluster_with_Explicitly_Defined_Settings.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengworkingwithenhancedclusters.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengintroducingclusteraddons.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengconfiguringclusteraddons.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +- https://docs.oracle.com/en-us/iaas/Content/ResourceManager/Concepts/terraformconfigresourcemanager_topic-schema.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/ +- https://github.com/oracle-terraform-modules/terraform-oci-oke diff --git a/oci/oke/skills/oke-gva-deployer/SKILL.md b/oci/oke/skills/oke-gva-deployer/SKILL.md new file mode 100644 index 0000000..b1fa2d1 --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/SKILL.md @@ -0,0 +1,285 @@ +--- +name: oke-gva-deployer +description: Use this skill when the user asks to enable, deploy, or configure Generic VNIC Attachment (GVA) for OCI Kubernetes Engine (OKE), create node pools with secondary VNIC profiles, review update plans for existing GVA node pools, map Application Resources to workloads, or explain GVA functionality, constraints, and scheduling behavior. Do not use it for general OKE incident RCA or for deploying Multus test pods after a GVA node pool already exists; use oke-troubleshooter or oke-multihome-deployer for those surfaces. +--- + +# OKE Generic VNIC Attachment (GVA) Deployer + +You are an OCI networking and OKE specialist. Help the user deploy GVA, validate prerequisites, configure node pools with secondary VNIC profiles, and roll out workloads that request Application Resources. Prefer live OCI discovery to reduce user input and confirm choices before generating commands. + +Hard constraint: +- Never query existing node pools for this workflow. Do not run `oci ce node-pool list` or `oci ce node-pool get`. +- Do not collect node-level information in this workflow. Do not run `kubectl get nodes`, `kubectl describe node`, or any per-node inspection commands unless the user explicitly asks for node details. +- Collect required values from cluster metadata (`oci ce cluster get`), networking discovery, and user-provided inputs only. +- Always use an interactive menu for node pool creation. For node-pool updates, prepare an explicit update review and do not execute a non-interactive update command unless an update-specific interactive flow is added and the user approves it. +- For every new node pool creation request, collect mutable node-pool inputs again. Never use saved JSON payload files, prior `tmp/` payloads, previous turn values, or previously generated commands without re-prompting and re-confirming them in the current workflow. +- Never start discovery on an implicit/default cluster. Discovery is allowed only after the user explicitly selects or provides a target cluster name/context/OCID in the current turn. +- Never use an OCI config default region for workflow execution. Always ask the user for the region in the current flow and use exactly the region they provide for all OCI CLI calls in that workflow. + +Supporting reference (load on demand): +- `references/gva.md` — concise feature summary, constraints, and example CLI / pod specs +- `validation-report-template.md` — standard node-pool validation report structure + +Scripts: +- `../../scripts/gva-discover.sh` — discover cluster, subnets, and NSGs to minimize prompts +- `../../scripts/gva-menu.sh` — guided interactive flow that consumes discovery data and prints CLI command + test manifest + +--- + +## Phase 0 — Intake +Flow requirements: +1) Confirm the target cluster first. If the user did not explicitly provide a cluster name/context/OCID in the prompt, ask which cluster to use before running discovery or generating commands. + - Do not assume a default cluster (for example `cluster3`) even if scripts offer one. + - If multiple kube contexts exist, require explicit selection before discovery. +2) Resolve **cluster OCID** from `~/.kube/config` when possible. +3) Resolve **tenancy defaults** from `~/.oci/config` only for non-region values if needed. + - Always ask the user for the region in the current flow. + - Use only the user-provided region for all OCI CLI calls in this workflow. +4) Use OCI CLI to retrieve cluster details, then **auto-populate** whatever is available. +5) Prompt only for missing information. + +If the cluster is not using VCN-Native CNI, stop and explain that GVA is unsupported for Flannel/Cilium. + +--- + +## Phase 1 — Fast Discovery (Mandatory) +For speed, use this sequence first before broader discovery: + +1) Resolve cluster OCID from kubeconfig. +2) Pull cluster details: + +```bash +oci ce cluster get --cluster-id --region +``` + +3) Pull VCNs only in the cluster compartment: + +```bash +oci network vcn list --compartment-id --region +``` + +4) Ask the user which VCN to use. +5) Pull subnets only for the selected VCN: + +```bash +oci network subnet list --compartment-id --vcn-id --region +``` + +6) Pull NSGs in the selected VCN (or compartment fallback if needed): + +```bash +oci network nsg list --compartment-id --vcn-id --region +``` + +Only if this flow fails should you fall back to the broader discovery helper below. + +## Phase 1b — Discovery Helper (Fallback) +When OCI CLI is available and authenticated, you may run: + +```bash +bash ../../scripts/gva-discover.sh --cluster [--region ] [--compartment-id ] [--profile ] [--timeout ] [--kubeconfig ] +``` + +Use the JSON output to populate: +- Cluster OCID, Kubernetes version, compartment OCID, region +- Subnet list (name, OCID, CIDR) +- NSG list (name, OCID) + +If any list is empty or the CLI call fails, fall back to manual prompts for that item. +Do not use node pool discovery as fallback. + +--- + +## Phase 2 — Conversational Menu UX (Mandatory) +Use a one-at-a-time menu flow in chat. Do not ask for multiple unrelated fields in a single prompt. + +Interaction rules: +- Ask exactly one configuration item per turn. +- For each menu, allow either: + - Numeric option selection (for example `1`, `2`, `3`), or + - Direct custom value typed by the user without a special keyword. +- Exception: for Availability Domain selection, only allow choosing from discovered AD options (no custom AD text). +- Prefer numeric menus consistently across all steps. +- Do not mark options as "recommended" unless the user explicitly asks for recommendations. +- If the user requests more options, expand the menu rather than truncating. +- Confirm and carry forward each accepted value before asking the next item. +- When the user starts a new node pool creation, always begin a fresh create flow for mutable values such as node pool name, shape, node count, placement subnet, AD, image, and each GVA profile, even if earlier runs already gathered similar values. +- Do not use an old `--from-json` request body as the source of truth for a new node pool create. + +Menu order: +1) VCN selection +2) `node_pool_name` +3) `node_shape` +4) shape config (OCPUs + memory) when shape is Flex +5) node count +6) Availability Domains (allow one, two, or all three; comma-separated) +7) primary node subnet +8) image selection (filter by Kubernetes version; validate shape architecture/family compatibility before finalizing) +9) Secondary VNIC profile fields: + - `applicationResource` label + - GVA secondary subnet + - NSG selection (`none` allowed) + - `ipCount` (1-256) +10) Ask: "Add another secondary VNIC profile?" and repeat step 9 until user says no. + +Data presentation rules: +- VCN menu must list all discovered VCNs in the target compartment (name + CIDR + OCID or selectable key). +- Subnet menus must list all discovered subnets in the user-selected VCN (name + CIDR + OCID or selectable key). +- Image menus must list OKE images matching the cluster Kubernetes version. If the helper cannot prove shape architecture/family compatibility from image metadata, mark that compatibility check as TODO/live validation and ask the user to confirm before finalizing. +- NSG menus must include all discovered NSGs and a "none" option. +- Availability Domain menu must list discovered ADs and accept only numeric multi-select input (`1,2,3` style); reject custom AD values. +- When a validation rule narrows choices (for example secondary subnet 2+ CIDR rule), render the filtered menu in the same format as the full menu (numeric key + name + CIDR), not key-only output. +- For every filtered menu, re-number options contiguously from `1..N` (do not reuse original indices like `4,10,11`). + +Compatibility guardrails: +- Validate image compatibility with node shape architecture/family before finalizing. +- If there is a mismatch (for example ARM image with x86 shape), stop and ask user to change either image or shape. +- Build one profile per secondary VNIC entry collected in step 9/10. +- After each secondary subnet selection, verify the subnet is IPv4-only (no IPv6 CIDR); if not, reject it and prompt for another subnet. +- After each secondary subnet selection, verify the subnet has more than one IPv4 CIDR block; if not, reject it and prompt for another subnet. + +Automation option: +- If user asks to use scripts, you may run: + - `bash ../../scripts/gva-menu.sh` + - `bash ../../scripts/gva-discover.sh ...` + But preserve the same conversational behavior above when operating in chat. + - When using scripts, pass/run only with the user-selected cluster; never accept or rely on script defaults for cluster selection. + +--- + +## Phase 3 — Design VNIC Profiles +Create a table of VNIC profiles with these fields: +- `applicationResource` (string label used by pods) +- `subnetId` (OCID) +- `ipCount` (integer, max 256) +- `nsgIds` (list, optional) +- `displayName` (optional) +- `assignPublicIp` (optional, default false) +- `tags` (optional) + +Validate: +- Each `applicationResource` is unique. +- Total IPs across VNICs fits expected pod count + buffer. +- Subnets align with intended traffic isolation. +- Each GVA secondary subnet is IPv4-only and has more than one IPv4 CIDR block. + +## Phase 3b — Required Variable Checklist (Always Collect) +Before generating create commands or update review commands, collect and confirm: +- Cluster context/name +- Cluster OCID +- Region +- Compartment OCID +- VCN OCID +- Kubernetes version +- Node pool name (must be explicitly provided; do not auto-finalize a default name without user confirmation) +- Node shape +- OCPUs (if Flex) +- Memory in GB (if Flex) +- Node count +- Availability Domain(s) (one or more) +- Primary node subnet (placement subnet) +- Image OCID matching cluster Kubernetes version +- Secondary VNIC profile list (one or more) +- `applicationResource` label per profile +- GVA secondary subnet per profile +- NSG IDs per profile +- `ipCount` per profile (1-256) +- Secondary VNIC display name (recommended) +- Whether additional GVA profiles are required (explicit yes/no loop) +- Optional node pool parameters (tags, labels, boot volume, SSH key, etc.) +- Final explicit create confirmation in the current turn before generating a runnable command + +--- + +## Phase 4 — Create Node Pool or Review Update (CLI) +Execution constraint: +- Keep node pool creation interactive (prompt/confirm driven). Command previews are allowed, but do not run non-interactive create or update commands directly. + +CLI runtime workflow (mandatory before create): +1) Check the installed OCI CLI for GA GVA node-pool flags: +```bash +oci --version +oci ce node-pool create --help | grep -E 'secondary-vnics|cni-type' +``` +2) If the flags are missing, stop before create and ask the operator to upgrade +the regular OCI CLI through their normal install path. A preview OCI CLI is no +longer required for this workflow. + +`../../scripts/gva-menu.sh` performs this check before executing a confirmed +create command. If the check fails, print the command only and ask the operator +to install an OCI CLI version with GVA flag support. + +If the user uses OCI CLI, generate a command using the prepared profiles. Use the template below and replace placeholders. + +```bash +oci ce node-pool create \ + --compartment-id "" \ + --cluster-id "" \ + --name "" \ + --kubernetes-version "" \ + --node-shape "" \ + --node-shape-config '{"ocpus":,"memoryInGBs":}' \ + --size \ + --cni-type OCI_VCN_IP_NATIVE \ + --placement-configs '[{"availabilityDomain":"","subnetId":""}]' \ + --node-source-details '{"sourceType":"IMAGE","imageId":""}' \ + --secondary-vnics '' +``` + +Use availability domains discovered from IAM for the tenant (for example `GrCh:US-ASHBURN-AD-1`), not guessed aliases. + +If the user uses Terraform, ask which module/resource they are using and map the same profile fields without guessing. + +--- + +## Phase 5 — Verify Node Resources +Provide user-run verification commands only (do not execute node queries from the agent). Instruct the user to confirm that GVA resources appear on nodes: + +```bash +kubectl describe node +``` + +Expected signals: +- Extended resources like `oke-application-resource.oci.oraclecloud.com/` +- Taint: `oci.oraclecloud.com/application-resource-only:NoSchedule` + +--- + +## Phase 6 — Deploy Workloads +Provide a pod/deployment snippet that: +- Requests exactly **1** unit of the chosen Application Resource +- Adds a toleration for the GVA taint + +Highlight validation rules: +- Exactly one Application Resource type per pod +- Resource count must be exactly 1 +- Pods without toleration will not schedule + +--- + +## Troubleshooting Quick Hits +- **Pods Pending**: No available IPs for requested resource, missing toleration, or wrong resource name +- **Validation webhook errors**: Pod requests multiple resources or incorrect count +- **Capacity issues**: Increase node pool size or rebalance `ipCount` across profiles + +--- + +## Output Expectations +Deliverables should include: +1. A short explanation of GVA functionality +2. A finalized VNIC profile table +3. A ready-to-run CLI command (or Terraform mapping notes) +4. A sample workload manifest +5. A verification checklist with pass/fail placeholders so the user can record node + resource validation after running `kubectl describe node ` +6. A final validation report using `validation-report-template.md` when post-create + validation evidence is available + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/create.html +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/update.html +- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ +- https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ diff --git a/oci/oke/skills/oke-gva-deployer/agents/openai.yaml b/oci/oke/skills/oke-gva-deployer/agents/openai.yaml new file mode 100644 index 0000000..4b24622 --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "OKE GVA Deployer" + short_description: "Create OKE GVA secondary VNIC node pools." + default_prompt: "Use $oke-gva-deployer to guide an interactive Generic VNIC Attachment workflow, discover cluster networking, build secondary VNIC profiles, and create or review node-pool commands with explicit approval." diff --git a/oci/oke/skills/oke-gva-deployer/references/gva.md b/oci/oke/skills/oke-gva-deployer/references/gva.md new file mode 100644 index 0000000..292ff75 --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/references/gva.md @@ -0,0 +1,97 @@ +# Generic VNIC Attachment (GVA) — Reference Summary + +Source baseline: public OKE documentation for attaching multiple secondary VNICs +for pod networking. + +## What GVA Does +- Provides per-secondary-VNIC configuration for OKE pods. +- Each secondary VNIC can have its own subnet, NSGs, tags, and IP allocation. +- Pods request a specific Application Resource label to select the VNIC type. + +## Why It Matters +- Enables workload isolation by network tier. +- Avoids identical secondary VNICs in standard VCN-Native CNI. +- Supports compliance and multi-tenant segmentation. + +## Key Concepts +- Primary VNIC: node management and control plane traffic. +- Secondary VNICs: pod networking. +- Application Resource: label that maps pod requests to a VNIC profile. +- Scheduler only places pods on nodes that have matching resources. + +## Prerequisites +- OKE cluster with **VCN-Native CNI** (GVA not supported on Flannel). +- Subnets and NSGs per workload tier. +- Each GVA secondary subnet must be IPv4-only and have more than one IPv4 CIDR block. +- Node pool permissions to create/manage VNICs. + +## Constraints and Limits +- `ipCount` per VNIC is capped at **256**. +- Pods can request **only one** Application Resource type. +- Pods must request **exactly 1** unit of that resource. +- Pod-level Application Resource selection is for a single secondary VNIC profile. + Use Multus and NetworkAttachmentDefinitions (NADs) for pods that need multiple + interfaces; do not combine pod-level Application Resource requests with Multus + network annotations in the same pod spec. +- VNIC attachment limits depend on instance shape. + +## Node Behavior +- Nodes expose extended resources: + `oke-application-resource.oci.oraclecloud.com/` +- Nodes are tainted: + `oci.oraclecloud.com/application-resource-only:NoSchedule` + +## CLI Version Note +- GVA node-pool create is expected to use the regular OCI CLI. Confirm local support with `oci ce node-pool create --help | grep -E 'secondary-vnics|cni-type'`. + Confirm availability in the user's environment before relying on it. + +## Example Secondary VNIC JSON (template) +Use as `--secondary-vnics ''` in CLI. + +```json +[ + { + "createVnicDetails": { + "ipCount": 256, + "applicationResources": ["Frontend"], + "assignPublicIp": false, + "displayName": "vnic-frontend", + "nsgIds": ["ocid1.nsg..."], + "subnetId": "ocid1.subnet...", + "skipSourceDestCheck": false + }, + "displayName": "vnicattachment-frontend" + } +] +``` + +## Example Pod Snippet (template) + +```yaml +spec: + tolerations: + - key: "oci.oraclecloud.com/application-resource-only" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: app + image: + resources: + requests: + oke-application-resource.oci.oraclecloud.com/Frontend: "1" + limits: + oke-application-resource.oci.oraclecloud.com/Frontend: "1" +``` + +## max-pods Guidance +- GVA reduces per-interface pod capacity (256 IPs per VNIC). +- Set kubelet `max-pods` via cloud-init accordingly. +- HostNetwork pods still count toward max-pods. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/create.html +- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ +- https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ diff --git a/oci/oke/skills/oke-gva-deployer/validation-report-template.md b/oci/oke/skills/oke-gva-deployer/validation-report-template.md new file mode 100644 index 0000000..8aeec0e --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/validation-report-template.md @@ -0,0 +1,56 @@ +# OKE GVA Node Pool Validation Report Template + +Use this structure after generating a GVA node-pool command or after the user reports +post-create validation results. + +## Summary + +State whether the GVA node-pool configuration is ready to create, created and +validated, or blocked by missing information. + +## Node Pool Configuration + +| Field | Value | +| --- | --- | +| Cluster | `` | +| Region | `` | +| Node pool | `` | +| Shape | `` | +| Node count | `` | +| Primary subnet | `` | +| Kubernetes version | `` | + +## Secondary VNIC Profiles + +| Application Resource | Subnet | `ipCount` | NSGs | Display name | +| --- | --- | ---: | --- | --- | +| `` | `` | `` | `` | `` | + +## Generated Artifacts + +| Artifact | Purpose | +| --- | --- | +| OCI CLI command | Creates the GVA-enabled node pool | +| Workload manifest | Requests one Application Resource and tolerates the GVA taint | +| Verification checklist | Confirms extended resources and scheduling constraints | + +## Validation Checklist + +| Check | Command | Result | +| --- | --- | --- | +| Extended resource present | `kubectl describe node ` | `` | +| GVA taint present | `kubectl describe node ` | `` | +| Test workload schedules | `kubectl -n get pods -o wide` | `` | +| Workload requests exactly one resource | `kubectl -n get pod -o yaml` | `` | + +## Remediation + +If validation fails, keep these causes separate: missing Application Resource on nodes, +wrong resource name in workload, missing toleration, invalid `ipCount`, subnet capacity, +or incompatible node-pool image/shape. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ +- https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ diff --git a/oci/oke/skills/oke-multihome-deployer/SKILL.md b/oci/oke/skills/oke-multihome-deployer/SKILL.md new file mode 100644 index 0000000..b050006 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/SKILL.md @@ -0,0 +1,139 @@ +--- +name: oke-multihome-deployer +description: Deploy and troubleshoot multihome networking on OCI Kubernetes Engine using existing GVA secondary VNIC node pools, Multus thick plugin, OCI CNI/IPAM, NetworkAttachmentDefinitions, and test pods. Use when the user asks to auto-discover OKE cluster subnets or node pools for multihome, create OKE multihome pod manifests, configure Multus NADs for secondary VNIC interfaces, validate multihomed pod connectivity, inspect OCI CNI IPAM allocation on OKE nodes, or package repeatable OKE GVA multihome deployment steps. Do not use it to create or update GVA node pools. For broad incident RCA, DPDK/SR-IOV failures, or non-Multus pod symptoms, use `oke-troubleshooter` first. +--- + +# OKE Multihome Deployer + +## Scope + +Use this skill after an OKE node pool already has GVA secondary VNIC profiles attached. This skill deploys and verifies Multus-based pod multihoming; it does not create or update GVA node pools. + +If the user needs to create or update the node pool secondary VNIC profiles, use `oke-gva-deployer` first, then return here. + +Supporting references: +- `validation-report-template.md` — standard final validation report structure. + +## Core Workflow + +1. Confirm target cluster/context and auth. + - Prefer explicit `OCI_CLI_PROFILE` and `OCI_CLI_AUTH`; when using OCI security-token auth, pass `--auth security_token` or set `OCI_CLI_AUTH=security_token`. + - If `kubectl` fails with `exec: executable oci failed`, ask the user to refresh the OCI session before applying anything. +2. Auto-discover cluster, node pool, and subnet data before generating YAML. + - Use `scripts/discover-oke-multihome.py`. + - Prefer `--cluster-id` when the user provides it; otherwise try `--context`, then `--cluster-name` with `--compartment-id`. + - The discovery script queries OKE node pools and fetches all referenced placement and secondary VNIC subnets. + - Use the discovered node names for `--pod NAME=NODE_NAME` entries. +3. Verify prerequisites from Kubernetes and node-side checks. + - OKE CNI must be `OCI_VCN_IP_NATIVE`. + - Worker nodes must expose secondary VNIC interfaces. Treat `enp1s0` and `enp2s0` as common defaults, then confirm the actual interface names from discovery or node-side evidence before applying manifests. + - OCI CNI binaries should exist on nodes: `oci-ipam`, `oci-ipvlan`, `oci-ptp`. + - The standard `ipvlan` CNI binary is needed when using a plain `ipvlan` secondary NAD. +4. Install Multus thick plugin if absent. + - Upstream manifest used in this workflow: + `https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml` + - TODO(live validation): pin this manifest to a tested Multus release tag after validating it against the target OKE Kubernetes version. Keep using the repo's current manifest reference until that live validation is complete. +5. Generate or edit the deployment manifest. + - Use `scripts/generate-multihome-manifest.py` for repeatable YAML. + - Default network NAD: `kube-system/gva-default-network`, using `oci-ipvlan` on the first secondary VNIC path. + - Secondary NAD: workload namespace scoped, using `ipvlan` on the second secondary VNIC path with `oci-ipam`. +6. Apply and verify. + - Use fully qualified images such as `docker.io/nicolaka/netshoot:v0.13`; CRI-O may reject short image names. + - Check pod annotation `k8s.v1.cni.cncf.io/network-status`. + - Exec `ip -br addr` in each pod and ping the peer `net1` IP both directions. + - Use `validation-report-template.md` for the final validation report, whether + validation passed, failed, or remains incomplete. + +## Discovery + +Run discovery before manifest generation: + +```bash +OCI_CLI_PROFILE= OCI_CLI_AUTH= \ +python3 /scripts/discover-oke-multihome.py \ + --cluster-id \ + --region \ + --pretty > oke-multihome-discovery.json +``` + +If the current kubeconfig contains the cluster OCID in its exec user args, use: + +```bash +OCI_CLI_PROFILE= OCI_CLI_AUTH= \ +python3 /scripts/discover-oke-multihome.py \ + --context \ + --region \ + --pretty +``` + +Use the JSON output to pick: + +- Node pool with GVA `secondaryVnics`. +- Primary placement subnet(s), for node placement context. +- Secondary VNIC subnet IDs/CIDRs, for deciding which interface path maps to each pod network. +- Up to two ready node names for pinned smoke-test pods. + +Do not guess subnet CIDRs or node names when discovery can retrieve them. + +## Manifest Generation + +Generate a two-pod test manifest: + +```bash +python3 /scripts/generate-multihome-manifest.py \ + --namespace gva-multihome-test \ + --default-interface \ + --secondary-interface \ + --pod gva-multihome-a= \ + --pod gva-multihome-b= \ + > gva-multihome-pods.yaml +``` + +Use Kubernetes node names exactly as returned by `kubectl get nodes`. In OKE clusters configured like the handoff, node names may be node private IPs. + +## Verification Commands + +Use these checks after applying manifests: + +```bash +kubectl -n kube-system get pods -l name=multus -o wide +kubectl -n gva-multihome-test get pods -o wide +kubectl -n gva-multihome-test get pod gva-multihome-a gva-multihome-b \ + -o jsonpath='{range .items[*]}{.metadata.name}{" on "}{.spec.nodeName}{"\n"}{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}{"\n\n"}{end}' +kubectl -n gva-multihome-test exec gva-multihome-a -- ip -br addr +kubectl -n gva-multihome-test exec gva-multihome-b -- ip -br addr +``` + +Then ping `net1` IPs across pods: + +```bash +kubectl -n gva-multihome-test exec gva-multihome-a -- ping -c 3 -W 2 +kubectl -n gva-multihome-test exec gva-multihome-b -- ping -c 3 -W 2 +``` + +## Troubleshooting + +Read `references/oke-multihome-notes.md` when: + +- Multus pods are missing or failing. +- Pods are stuck creating sandbox/network. +- `ipvlan` is missing on the host. +- OCI IPAM capacity or allocation behavior is unclear. +- Floating IP/VIP behavior is being explored. + +Read `references/oke-dpdk-mlx5-notes.md` when: + +- The workload uses DPDK, SR-IOV device-plugin resources, Mellanox/NVIDIA VFs, `mlx5`, `vfio-pci`, hugepages, or `/dev/infiniband`. +- A pod requests Multus networks and device-plugin resources but `network-status` only shows `eth0`. +- You need to decide whether the issue is Multus/NAD attachment, resource allocation, driver binding, RDMA/verbs exposure, hugepages, or app PCI/interface mapping. + +For node-side inspection, prefer a privileged diagnostic DaemonSet that mounts `/` at `/host` and runs commands with `chroot /host`, rather than SSHing to each worker. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://github.com/k8snetworkplumbingwg/multus-cni +- https://github.com/k8snetworkplumbingwg/network-attachment-definition-client +- https://www.cni.dev/plugins/current/main/ipvlan/ +- https://kubernetes.io/docs/tasks/debug/debug-cluster/kubectl-node-debug/ diff --git a/oci/oke/skills/oke-multihome-deployer/agents/openai.yaml b/oci/oke/skills/oke-multihome-deployer/agents/openai.yaml new file mode 100644 index 0000000..71751e9 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "OKE Multihome Deployer" + short_description: "Deploy OKE Multus multihome networking." + default_prompt: "Use $oke-multihome-deployer to auto-discover OKE node pools and subnets, then deploy or troubleshoot Multus multihome networking with GVA secondary VNICs and NetworkAttachmentDefinitions." diff --git a/oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md b/oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md new file mode 100644 index 0000000..cdd6247 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md @@ -0,0 +1,118 @@ +# OKE DPDK, Multus, and Mellanox mlx5 Notes + +Use this note when an OKE workload combines DPDK, Multus, SR-IOV device-plugin resources, Mellanox/NVIDIA VFs, `mlx5_core`, `vfio-pci`, hugepages, or RDMA/verbs devices. + +## Contents +- Decision Model +- mlx5 Driver Rule +- Read-Only Evidence Commands +- Expected Evidence +- Common Failure Pattern + +## Decision Model + +Do not collapse these mechanisms into one check: + +- Kubernetes device-plugin resource request: allocates a named resource to the pod. +- Multus annotation: asks for named network attachments. +- `NetworkAttachmentDefinition`: defines the requested network. +- `network-status`: proves what Multus actually attached. +- Driver binding: decides whether the VF is owned by Linux `mlx5_core`, `vfio-pci`, or another driver. +- DPDK app config: decides which PCI BDFs, ports, hugepage path, and PMD are used. + +A pod can be `Running` and still have no usable DPDK network path. + +## mlx5 Driver Rule + +For Mellanox/NVIDIA `mlx5` DPDK PMD, `vfio-pci` is not always the desired state. The bifurcated mlx5 model can require the VF to stay bound to Linux `mlx5_core` while DPDK uses mlx5 PMD libraries through the kernel/RDMA userspace stack. + +Treat `vfio-pci` as proven only when the node exposes numbered VFIO group devices such as `/dev/vfio/` and a controlled bind succeeds. If `/dev/vfio` contains only the control device and `/sys/kernel/iommu_groups` is empty, do not keep retrying disruptive binds on live nodes. + +## Read-Only Evidence Commands + +Collect the Kubernetes side first: + +```bash +kubectl get network-attachment-definitions -A | egrep '||noiommu|sriov|mlx' +kubectl -n get network-attachment-definition -o yaml +kubectl -n describe pod +kubectl -n get pod \ + -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' +``` + +Inside the pod: + +```bash +kubectl -n exec -- ip -br addr +kubectl -n exec -- ls -l /dev/infiniband +kubectl -n exec -- ibv_devices +kubectl -n exec -- lspci -nnk +kubectl -n exec -- dpdk-devbind.py -s +kubectl -n exec -- grep -i Huge /proc/meminfo +kubectl -n exec -- mount | grep -i huge +``` + +On the target node, after explicit approval for node-side inspection: + +```bash +ip -br addr +lspci -nnk | egrep -A3 'Mellanox|NVIDIA|Ethernet' +ls -l /dev/vfio +find /sys/kernel/iommu_groups -maxdepth 1 -mindepth 1 -type d | wc -l +dpdk-devbind.py -s +``` + +Capture DPDK startup logs: + +```bash +kubectl -n logs | egrep -i 'EAL|mlx5|ibv|verbs|huge|vfio|pci|rte' +``` + +## Expected Evidence + +Multus attachment is proven only when `network-status` lists the requested networks and interfaces, not just the default network: + +```text +eth0 ... +net1 ... +net2 ... +``` + +The mlx5 PMD path usually needs: + +```text +Kernel driver in use: mlx5_core +/dev/infiniband/uverbs* +ibv_devices shows mlx5 devices +HugePages_Total > 0 +``` + +The VFIO path usually needs: + +```text +/dev/vfio/vfio +/dev/vfio/ +non-empty /sys/kernel/iommu_groups +target PCI device bound to vfio-pci +``` + +## Common Failure Pattern + +If the pod requests networks such as `noiommu-0,noiommu-1` and device-plugin resources such as `mellanox.com/`, but `network-status` only shows `cbr0` or `eth0`, the immediate issue is not proven DPDK initialization. First verify: + +- The NADs exist in the correct namespace. +- The pod annotations reference the correct NAD names or namespace-qualified names. +- Each NAD maps to the intended device-plugin resource when using SR-IOV CNI. +- Pod events do not show Multus or CNI errors. +- The pod has the expected `net1`, `net2`, etc. interfaces. + +Only after this is correct should the investigation move to DPDK PMD, PCI allowlist, hugepage, or NUMA details. + +## Sources + +- https://github.com/k8snetworkplumbingwg/multus-cni +- https://github.com/k8snetworkplumbingwg/network-attachment-definition-client +- https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin +- https://doc.dpdk.org/guides/nics/mlx5.html +- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ +- https://kubernetes.io/docs/tasks/debug/debug-application/ diff --git a/oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md b/oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md new file mode 100644 index 0000000..0f1a7fc --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md @@ -0,0 +1,103 @@ +# OKE Multihome Notes + +## Known Working Pattern + +- OKE CNI: `OCI_VCN_IP_NATIVE` +- Multus: thick DaemonSet +- Default Multus network annotation: `v1.multus-cni.io/default-network` +- Additional network annotation: `k8s.v1.cni.cncf.io/networks` +- Default NAD uses `oci-ipvlan` plus `oci-ptp`. +- Secondary NAD can use standard `ipvlan` with OCI `oci-ipam`. +- Netshoot image should be fully qualified: `docker.io/nicolaka/netshoot:v0.13`. + +## Discovery Scope + +Use `scripts/discover-oke-multihome.py` to collect: + +- OKE cluster metadata. +- All node pools in the target cluster. +- Node pool placement subnets. +- GVA secondary VNIC profile subnets. +- Referenced subnet display names, CIDRs, VCN IDs, and availability domains. + +Discovery intentionally reads node pool details because secondary VNIC profile data lives on the node pool. It does not create or update any OCI resources. + +Example interface mapping from the observed OKE GVA nodes: + +```text +enp0s5 = primary node subnet +enp1s0 = first secondary VNIC path +enp2s0 = second secondary VNIC path +``` + +## CNI Binary Checks + +The OKE nodes should already have: + +```text +oci-ipam +oci-ipvlan +oci-ptp +``` + +The standard `ipvlan` binary may be missing. If the secondary NAD uses `"type": "ipvlan"`, install or copy the `ipvlan` CNI binary into host `/opt/cni/bin` on every target node. Avoid replacing the full CNI plugin bundle unless needed. + +## OCI IPAM State + +OCI private-IP API output may not show all addresses available to the node-local CNI pool. For allocation details, inspect node-side state: + +```text +/dev/shm/oci-cni/free +/dev/shm/oci-cni/used +``` + +Use a privileged diagnostic DaemonSet and `chroot /host` for repeated multi-node inspection. + +## Common Failures + +- `getting credentials: exec: executable oci failed`: refresh OCI CLI security token or set `OCI_CLI_PROFILE` and `OCI_CLI_AUTH` correctly. +- CRI-O rejects `nicolaka/netshoot:v0.13`: use `docker.io/nicolaka/netshoot:v0.13`. +- Pod sandbox creation fails with missing CNI plugin `ipvlan`: install the standard `ipvlan` CNI binary on the worker host. +- Pods do not receive the expected interfaces: check the exact NAD names, namespaces, and Multus annotations. +- Cross-pod `net1` ping fails: confirm both pods are on nodes with the secondary VNIC interface, check subnet security rules/NSGs, and verify each pod's `network-status` annotation. + +## DPDK / SR-IOV Boundary + +For DPDK workloads, keep these mechanisms separate: + +- `resources.requests` or `resources.limits` for a device-plugin resource only asks Kubernetes to allocate that resource. +- `k8s.v1.cni.cncf.io/networks` only asks Multus to attach named networks. +- A `NetworkAttachmentDefinition` defines what each named network means. +- `k8s.v1.cni.cncf.io/network-status` is the evidence that Multus actually attached the interfaces. + +If a pod requests SR-IOV or Mellanox resources but `network-status` only shows the default `eth0` network, scheduling may have succeeded while Multus attachment is still unproven or broken. First collect NAD YAML and pod `network-status` before debugging the DPDK application. + +For Mellanox/NVIDIA `mlx5` DPDK PMD, do not assume `vfio-pci` is the right driver. Some working mlx5 paths use the Linux `mlx5_core` driver plus RDMA/verbs devices. If the workload mentions `mlx5`, `/dev/infiniband`, `ibv_devices`, or DPDK mlx5 PMD libraries, read `oke-dpdk-mlx5-notes.md`. + +## Floating IP Note + +A VIP can be modeled as an OCI secondary private IP moved between matching secondary VNICs: + +```bash +oci network vnic assign-private-ip \ + --vnic-id \ + --ip-address \ + --unassign-if-already-assigned +``` + +Then add the /32 inside the active pod on the corresponding Multus interface: + +```bash +ip addr add /32 dev net1 +``` + +Reserve VIP ranges carefully. Confirm OCI IPAM allocation behavior first to avoid collisions with dynamically assigned pod addresses. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://github.com/k8snetworkplumbingwg/multus-cni +- https://github.com/k8snetworkplumbingwg/network-attachment-definition-client +- https://www.cni.dev/plugins/current/main/ipvlan/ +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/network/vnic/assign-private-ip.html diff --git a/oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py b/oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py new file mode 100755 index 0000000..2996412 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py @@ -0,0 +1,358 @@ +#!/usr/bin/env python3 +"""Discover OKE cluster, node pool, and subnet data for Multus multihome setup.""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys +from typing import Any + + +CLUSTER_OCID_RE = re.compile(r"ocid1\.cluster\.[^\s\"']+") +SUBNET_OCID_RE = re.compile(r"ocid1\.subnet\.[^\s\"']+") + + +def emit_error(exit_code: int, error_code: str, message: str, remediation: str, docs_url: str = "") -> int: + print( + json.dumps( + { + "error_code": error_code, + "message": message, + "remediation": remediation, + "docs_url": docs_url, + } + ), + file=sys.stderr, + ) + return exit_code + + +class JsonArgumentParser(argparse.ArgumentParser): + def error(self, message: str) -> None: + raise SystemExit( + emit_error( + 2, + "INVALID_ARGUMENT", + message, + "Run with --help to view usage.", + ) + ) + + +def run_json(cmd: list[str], env: dict[str, str]) -> dict[str, Any]: + result = subprocess.run(cmd, env=env, text=True, capture_output=True, check=False) + if result.returncode != 0: + raise RuntimeError( + "command failed: {}\nstdout:\n{}\nstderr:\n{}".format( + " ".join(cmd), result.stdout.strip(), result.stderr.strip() + ) + ) + try: + return json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise RuntimeError(f"command did not return JSON: {' '.join(cmd)}") from exc + + +def run_text(cmd: list[str], env: dict[str, str]) -> str: + result = subprocess.run(cmd, env=env, text=True, capture_output=True, check=False) + if result.returncode != 0: + raise RuntimeError( + "command failed: {}\nstdout:\n{}\nstderr:\n{}".format( + " ".join(cmd), result.stdout.strip(), result.stderr.strip() + ) + ) + return result.stdout + + +def oci_base(args: argparse.Namespace) -> list[str]: + cmd = ["oci"] + if args.profile: + cmd.extend(["--profile", args.profile]) + if args.auth: + cmd.extend(["--auth", args.auth]) + return cmd + + +def with_region(cmd: list[str], region: str | None) -> list[str]: + if region: + return [*cmd, "--region", region] + return cmd + + +def first_ocid(value: Any, regex: re.Pattern[str]) -> str | None: + text = json.dumps(value) + match = regex.search(text) + return match.group(0) if match else None + + +def cluster_id_from_kubeconfig(context: str | None, env: dict[str, str]) -> str | None: + try: + config = run_json(["kubectl", "config", "view", "--raw", "-o", "json"], env) + except Exception: + return None + + selected = context or config.get("current-context") + if not selected: + return first_ocid(config, CLUSTER_OCID_RE) + + contexts = { + item.get("name"): item.get("context", {}) + for item in config.get("contexts", []) + if isinstance(item, dict) + } + ctx = contexts.get(selected, {}) + related_names = {selected, ctx.get("cluster"), ctx.get("user")} + related = [] + for section in ("contexts", "clusters", "users"): + for item in config.get(section, []): + if item.get("name") in related_names: + related.append(item) + return first_ocid(related or config, CLUSTER_OCID_RE) + + +def cluster_id_from_name(args: argparse.Namespace, env: dict[str, str]) -> str | None: + if not args.cluster_name or not args.compartment_id: + return None + cmd = with_region( + [ + *oci_base(args), + "ce", + "cluster", + "list", + "--compartment-id", + args.compartment_id, + "--name", + args.cluster_name, + ], + args.region, + ) + data = run_json(cmd, env).get("data", []) + if not data: + return None + if len(data) > 1: + raise RuntimeError(f"multiple clusters named {args.cluster_name!r}; pass --cluster-id") + return data[0].get("id") + + +def get_cluster(args: argparse.Namespace, cluster_id: str, env: dict[str, str]) -> dict[str, Any]: + cmd = with_region([*oci_base(args), "ce", "cluster", "get", "--cluster-id", cluster_id], args.region) + return run_json(cmd, env)["data"] + + +def list_node_pools(args: argparse.Namespace, cluster: dict[str, Any], env: dict[str, str]) -> list[dict[str, Any]]: + compartment_id = args.compartment_id or cluster.get("compartment-id") + if not compartment_id: + raise RuntimeError("missing compartment id; pass --compartment-id") + cmd = with_region( + [ + *oci_base(args), + "ce", + "node-pool", + "list", + "--compartment-id", + compartment_id, + "--cluster-id", + cluster["id"], + "--all", + ], + args.region, + ) + pools = run_json(cmd, env).get("data", []) + full = [] + for pool in pools: + pool_id = pool.get("id") + if not pool_id: + continue + get_cmd = with_region([*oci_base(args), "ce", "node-pool", "get", "--node-pool-id", pool_id], args.region) + full.append(run_json(get_cmd, env)["data"]) + return full + + +def collect_subnet_ids(cluster: dict[str, Any], pools: list[dict[str, Any]]) -> list[str]: + ids: set[str] = set() + for ocid in re.findall(SUBNET_OCID_RE, json.dumps(cluster)): + ids.add(ocid) + for pool in pools: + for ocid in re.findall(SUBNET_OCID_RE, json.dumps(pool)): + ids.add(ocid) + return sorted(ids) + + +def get_subnets(args: argparse.Namespace, subnet_ids: list[str], env: dict[str, str]) -> dict[str, dict[str, Any]]: + subnets = {} + for subnet_id in subnet_ids: + cmd = with_region([*oci_base(args), "network", "subnet", "get", "--subnet-id", subnet_id], args.region) + subnets[subnet_id] = run_json(cmd, env)["data"] + return subnets + + +def subnet_summary(subnet: dict[str, Any]) -> dict[str, Any]: + return { + "id": subnet.get("id"), + "displayName": subnet.get("display-name"), + "cidrBlock": subnet.get("cidr-block"), + "dnsLabel": subnet.get("dns-label"), + "vcnId": subnet.get("vcn-id"), + "availabilityDomain": subnet.get("availability-domain"), + "prohibitPublicIpOnVnic": subnet.get("prohibit-public-ip-on-vnic"), + } + + +def node_pool_summary(pool: dict[str, Any], subnets: dict[str, dict[str, Any]]) -> dict[str, Any]: + placement_configs = pool.get("placement-configs") or [] + secondary_vnics = pool.get("secondary-vnics") or [] + return { + "id": pool.get("id"), + "name": pool.get("name"), + "kubernetesVersion": pool.get("kubernetes-version"), + "nodeShape": pool.get("node-shape"), + "size": pool.get("size"), + "placementSubnets": [ + { + "availabilityDomain": pc.get("availability-domain"), + "subnetId": pc.get("subnet-id"), + "subnet": subnet_summary(subnets[pc["subnet-id"]]) if pc.get("subnet-id") in subnets else None, + } + for pc in placement_configs + ], + "secondaryVnics": [ + { + "displayName": item.get("display-name"), + "attachmentDisplayName": item.get("attachment-display-name"), + "subnetId": item.get("subnet-id"), + "subnet": subnet_summary(subnets[item["subnet-id"]]) if item.get("subnet-id") in subnets else None, + "ipCount": item.get("ip-count"), + "nicIndex": item.get("nic-index"), + "applicationResource": item.get("application-resource"), + "nsgIds": item.get("nsg-ids") or [], + } + for item in secondary_vnics + ], + } + + +def suggested_generator_args(pools: list[dict[str, Any]]) -> list[str]: + args = [ + "python3", + "/scripts/generate-multihome-manifest.py", + "--default-interface", + "enp1s0", + "--secondary-interface", + "enp2s0", + ] + node_names: list[str] = [] + for pool in pools: + for node in pool.get("nodes") or []: + name = node.get("name") or node.get("private-ip") + if name and name not in node_names: + node_names.append(name) + for index, node_name in enumerate(node_names[:2], start=1): + args.extend(["--pod", f"gva-multihome-{index}={node_name}"]) + return args + + +def main() -> int: + parser = JsonArgumentParser( + description="Discover OKE cluster node pools, placement subnets, and GVA secondary VNIC subnets." + ) + parser.add_argument("--cluster-id") + parser.add_argument("--cluster-name") + parser.add_argument("--context", help="Kubernetes context to inspect for cluster OCID") + parser.add_argument("--compartment-id") + parser.add_argument("--region") + parser.add_argument("--profile", default=os.environ.get("OCI_CLI_PROFILE")) + parser.add_argument("--auth", default=os.environ.get("OCI_CLI_AUTH")) + parser.add_argument("--pretty", action="store_true", help="Pretty-print JSON output") + args = parser.parse_args() + + env = os.environ.copy() + if args.profile: + env["OCI_CLI_PROFILE"] = args.profile + if args.auth: + env["OCI_CLI_AUTH"] = args.auth + + if shutil.which("oci") is None: + return emit_error( + 1, + "OCI_CLI_NOT_FOUND", + "OCI CLI is not installed or not on PATH.", + "Install OCI CLI, authenticate to the target tenancy, and rerun discovery.", + ) + + needs_kubeconfig_lookup = not args.cluster_id and not (args.cluster_name and args.compartment_id) + if needs_kubeconfig_lookup and shutil.which("kubectl") is None: + return emit_error( + 1, + "KUBECTL_NOT_FOUND", + "kubectl is not installed or not on PATH, and discovery needs kubeconfig to resolve the cluster OCID.", + "Install kubectl, pass --cluster-id, or pass --cluster-name with --compartment-id.", + ) + + cluster_id = ( + args.cluster_id + or cluster_id_from_kubeconfig(args.context, env) + or cluster_id_from_name(args, env) + ) + if not cluster_id: + return emit_error( + 1, + "CLUSTER_OCID_NOT_RESOLVED", + "Could not resolve cluster OCID.", + "Pass --cluster-id, or pass --cluster-name with --compartment-id, or use a kubeconfig context that contains --cluster-id.", + ) + + cluster = get_cluster(args, cluster_id, env) + if not args.region: + args.region = cluster.get("region") + + pools = list_node_pools(args, cluster, env) + subnet_ids = collect_subnet_ids(cluster, pools) + subnets = get_subnets(args, subnet_ids, env) + output = { + "cluster": { + "id": cluster.get("id"), + "name": cluster.get("name"), + "compartmentId": cluster.get("compartment-id"), + "kubernetesVersion": cluster.get("kubernetes-version"), + "vcnId": cluster.get("vcn-id"), + "type": cluster.get("type"), + "endpointConfig": cluster.get("endpoint-config"), + "options": cluster.get("options"), + }, + "subnets": [subnet_summary(subnets[subnet_id]) for subnet_id in sorted(subnets)], + "nodePools": [node_pool_summary(pool, subnets) for pool in pools], + "suggestedGeneratorArgs": suggested_generator_args(pools), + } + indent = 2 if args.pretty else None + print(json.dumps(output, indent=indent, sort_keys=bool(indent))) + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except BrokenPipeError: + raise SystemExit(1) + except RuntimeError as exc: + raise SystemExit( + emit_error( + 1, + "DISCOVERY_COMMAND_FAILED", + str(exc), + "Verify OCI CLI authentication, region, compartment, and kubeconfig context, then rerun discovery.", + ) + ) + except Exception as exc: + raise SystemExit( + emit_error( + 2, + "UNEXPECTED_ERROR", + str(exc), + "Inspect the input arguments and environment, then rerun discovery.", + ) + ) diff --git a/oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py b/oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py new file mode 100755 index 0000000..c628007 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +"""Generate OKE GVA Multus multihome test manifests.""" + +from __future__ import annotations + +import argparse +import json +import sys +from typing import Iterable + + +def emit_error(exit_code: int, error_code: str, message: str, remediation: str, docs_url: str = "") -> int: + print( + json.dumps( + { + "error_code": error_code, + "message": message, + "remediation": remediation, + "docs_url": docs_url, + } + ), + file=sys.stderr, + ) + return exit_code + + +class JsonArgumentParser(argparse.ArgumentParser): + def error(self, message: str) -> None: + raise SystemExit( + emit_error( + 2, + "INVALID_ARGUMENT", + message, + "Run with --help to view usage.", + ) + ) + + +def q(value: str) -> str: + return json.dumps(value) + + +def block_json(value: dict) -> str: + text = json.dumps(value, indent=2) + return "\n".join(f" {line}" for line in text.splitlines()) + + +def parse_pod(value: str) -> tuple[str, str]: + if "=" not in value: + raise argparse.ArgumentTypeError("pod must use NAME=NODE_NAME") + name, node = value.split("=", 1) + if not name or not node: + raise argparse.ArgumentTypeError("pod must use NAME=NODE_NAME") + return name, node + + +def emit(lines: Iterable[str]) -> None: + sys.stdout.write("\n".join(lines)) + sys.stdout.write("\n") + + +def main() -> int: + parser = JsonArgumentParser( + description="Generate Multus NetworkAttachmentDefinitions and pinned test pods for OKE GVA multihome." + ) + parser.add_argument("--namespace", default="gva-multihome-test") + parser.add_argument("--default-nad-namespace", default="kube-system") + parser.add_argument("--default-nad-name", default="gva-default-network") + parser.add_argument("--secondary-nad-name", default="gva-secondary-network") + parser.add_argument("--default-interface", default="enp1s0") + parser.add_argument("--secondary-interface", default="enp2s0") + parser.add_argument("--image", default="docker.io/nicolaka/netshoot:v0.13") + parser.add_argument("--sleep", default="3600") + parser.add_argument( + "--pod", + action="append", + type=parse_pod, + required=True, + metavar="NAME=NODE_NAME", + help="Add a pinned test pod. Repeat for multiple pods.", + ) + args = parser.parse_args() + + default_config = { + "name": args.default_nad_name, + "cniVersion": "0.3.1", + "plugins": [ + { + "cniVersion": "0.3.1", + "type": "oci-ipvlan", + "mode": "l2", + "ipam": { + "type": "oci-ipam", + "deviceSelector": {"interfaceName": args.default_interface}, + }, + }, + { + "cniVersion": "0.3.1", + "type": "oci-ptp", + "containerInterface": "ptp-veth0", + "mtu": 9000, + }, + ], + } + + secondary_config = { + "cniVersion": "0.3.1", + "plugins": [ + { + "type": "ipvlan", + "mode": "l2", + "master": args.secondary_interface, + "ipam": { + "type": "oci-ipam", + "deviceSelector": {"interfaceName": args.secondary_interface}, + }, + } + ], + } + + lines: list[str] = [ + "apiVersion: v1", + "kind: Namespace", + "metadata:", + f" name: {args.namespace}", + "---", + "apiVersion: k8s.cni.cncf.io/v1", + "kind: NetworkAttachmentDefinition", + "metadata:", + f" name: {args.default_nad_name}", + f" namespace: {args.default_nad_namespace}", + "spec:", + " config: |", + block_json(default_config), + "---", + "apiVersion: k8s.cni.cncf.io/v1", + "kind: NetworkAttachmentDefinition", + "metadata:", + f" name: {args.secondary_nad_name}", + f" namespace: {args.namespace}", + "spec:", + " config: |", + block_json(secondary_config), + ] + + for pod_name, node_name in args.pod: + lines.extend( + [ + "---", + "apiVersion: v1", + "kind: Pod", + "metadata:", + f" name: {pod_name}", + f" namespace: {args.namespace}", + " labels:", + " app: gva-multihome", + " annotations:", + f" v1.multus-cni.io/default-network: {args.default_nad_namespace}/{args.default_nad_name}", + f" k8s.v1.cni.cncf.io/networks: {args.namespace}/{args.secondary_nad_name}", + "spec:", + f" nodeName: {q(node_name)}", + " containers:", + " - name: netshoot", + f" image: {q(args.image)}", + " command:", + " - sleep", + f" - {q(args.sleep)}", + ] + ) + + emit(lines) + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except BrokenPipeError: + raise SystemExit(1) + except Exception as exc: + raise SystemExit( + emit_error( + 2, + "UNEXPECTED_ERROR", + str(exc), + "Inspect the input arguments and rerun the manifest generator.", + ) + ) diff --git a/oci/oke/skills/oke-multihome-deployer/validation-report-template.md b/oci/oke/skills/oke-multihome-deployer/validation-report-template.md new file mode 100644 index 0000000..305529d --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/validation-report-template.md @@ -0,0 +1,50 @@ +# OKE Multihome Validation Report Template + +Use this structure after generating or applying Multus/GVA validation manifests. + +## Summary + +State whether multihome validation passed, failed, or is incomplete. + +## Environment + +| Field | Value | +| --- | --- | +| Cluster | `` | +| Region | `` | +| Namespace | `` | +| Node pool | `` | +| Default interface | `` | +| Secondary interface | `` | + +## Generated Artifacts + +| Artifact | Purpose | +| --- | --- | +| Namespace | Test namespace for validation pods | +| Default NAD | Multus default network backed by OCI CNI/IPAM | +| Secondary NAD | Secondary `net1` network backed by OCI IPAM | +| Test pods | Pinned netshoot pods for interface and ping checks | + +## Validation Checks + +| Check | Command | Result | +| --- | --- | --- | +| Multus daemonset | `kubectl -n kube-system get pods -l name=multus -o wide` | `` | +| Pod placement | `kubectl -n get pods -o wide` | `` | +| Network status | `kubectl -n get pod -o jsonpath=...` | `` | +| Interfaces | `kubectl -n exec -- ip -br addr` | `` | +| Peer ping | `kubectl -n exec -- ping -c 3 ` | `` | + +## Remediation + +List the first failed check and the most likely fix. Keep Multus/NAD failures, +OCI CNI/IPAM allocation failures, node-interface issues, and DPDK/SR-IOV driver +issues as separate facts. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://github.com/k8snetworkplumbingwg/multus-cni +- https://github.com/k8snetworkplumbingwg/network-attachment-definition-client +- https://kubernetes.io/docs/reference/kubectl/ diff --git a/oci/oke/skills/oke-troubleshooter/SKILL.md b/oci/oke/skills/oke-troubleshooter/SKILL.md new file mode 100644 index 0000000..f581d6d --- /dev/null +++ b/oci/oke/skills/oke-troubleshooter/SKILL.md @@ -0,0 +1,400 @@ +--- +name: oke-troubleshooter +description: Use this skill when the user wants to diagnose or root-cause issues with an OCI Kubernetes Engine cluster or workload. Trigger phrases include "pods pending", "troubleshoot OKE", "service has no IP", "cluster unhealthy", DPDK/SR-IOV mlx5 pod failures, Multus network-status issues, or broad incident RCA across Kubernetes and OCI resources. Do not use it for greenfield Terraform generation, GVA node-pool creation or update review, or routine Multus manifest deployment when no incident is being investigated; route those to oke-cluster-generator, oke-gva-deployer, or oke-multihome-deployer. +--- + +You are an experienced Site Reliability Engineer for OCI Kubernetes Engine. Guide the user through an evidence-driven investigation that spans Kubernetes signals and OCI infrastructure. + +Supporting references (load on demand): +- `symptom-triage.md` — initial mapping of symptom → diagnostic domains. +- `evidence-collectors.md` — command recipes for each domain. +- `final-report-template.md` — standard final report structure. +- `../../shared/oci-resource-map.md` — K8s-to-OCI mapping commands. +- `../oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md` — DPDK, Multus, Mellanox mlx5, `vfio-pci`, hugepage, and RDMA/verbs diagnostic rules. + +Optional accelerators (use only when the runtime supports delegation; never block on them): +- `../../agents/oke-evidence-collector.md` — agent for command execution and evidence normalization. +- `../../agents/oke-hypothesis-analyst.md` — agent for scoring hypotheses. +- `../../agents/oke-lb-log-collector.md` — agent for LB OCID resolution, logging-status checks, and LB log signal extraction. + +Scripts rely on the global error contract: exit 0 success, exit 1 expected issues, exit 2 unexpected. Emit JSON errors on stderr in failure scenarios. + +Helper scripts: +- `../../scripts/oke-discover.sh` — resolve cluster OCID from kubeconfig and fetch compartment/region via OCI CLI +- `../../scripts/oke-addon-health.sh` — collect kube-system add-on health signals +- `../../scripts/oke-pod-network-check.sh` — collect OCI CNI/IPAM, Multus, pod sandbox, and NAD signals +- `../../scripts/oke-autoscaler-check.sh` — collect Pending pod, cluster-autoscaler, and node-pool scaling signals +- `../../scripts/oke-dns-check.sh` — collect CoreDNS, Service, EndpointSlice, and pod DNS lookup signals +- `../../scripts/oke-ingress-check.sh` — collect OCI Native Ingress controller and Ingress object signals +- `../../scripts/oke-private-endpoint-check.sh` — collect private endpoint, kubeconfig, and API reachability signals +- `../../scripts/oke-ocir-image-pull-check.sh` — collect OCIR image pull, secret, service account, and repository signals +- `../../scripts/oke-workload-identity-check.sh` — collect service account, pod log, token projection, and workload identity IAM policy signals +- `../../scripts/oke-incident-timeline.sh` — merge Kubernetes events, rollout history, object descriptions, and OCI alarms into a timeline +- `../../scripts/oke-object-correlator.sh` — build a Kubernetes-to-OCI object graph for pods, nodes, services, ingress, PVCs, load balancers, instances, VNICs, volumes, and node pools + +--- + +## Execution Mode + +- Default to **local execution in the parent skill**. +- Use the optional agents above only as accelerators when the current runtime clearly supports agent delegation. +- If agents are unavailable, disabled, or return malformed output, continue locally with the same command list and payload shape. Do not stop the investigation solely because delegation is unavailable. +- Normalize local evidence to the same JSON shape documented in `evidence-collectors.md`. +- Never execute a mutating Kubernetes or OCI action unless the exact command or action has been shown to the user and explicitly approved in the current session. +- Treat `kubectl apply`, `kubectl patch`, `kubectl annotate`, `kubectl delete`, `kubectl rollout restart`, `kubectl scale`, node cordon/drain/debug flows, OCI create/update/delete operations, and LB logging enablement as approval-required. Approval for one command does not approve follow-up mutations. + +## Phase 0 — Input & Preflight +1. **Parse Arguments** + - `$ARGUMENTS` holds an optional symptom string. If empty, ask the user for a concise description (e.g., `"pods stuck Pending in prod namespace"`). + - Extract namespace hints (`-n`, `namespace:`) and resource names when present. +2. **Auto-Discover Cluster Context** + - Ask for **cluster name** if not provided. + - First list kubeconfig contexts to identify managed clusters and current context: + ```bash + kubectl config get-contexts + ``` + - Use this output to suggest likely cluster/context names before prompting for manual input. + - Derive `active_cluster_region` from the active kube context (`kubectl config view --minify`, user exec args, or cluster metadata tied to the current context) and treat it as authoritative. + - Resolve **cluster OCID** from `~/.kube/config` when possible. + - Use tenancy defaults from `~/.oci/config` only for auth/profile hints, not for region selection. + - Run: + ```bash + bash ../../scripts/oke-discover.sh --cluster [--region ] [--profile ] [--timeout ] [--kubeconfig ] [--deployment ] + ``` + - Always pass `--region ` to discovery and all OCI CLI calls in later phases. + - Never use implicit OCI CLI region or fallback/default region. + - Use the JSON output to auto-populate: `cluster_ocid`, `compartment_ocid`, `region`, `kubernetes_version`, and deployment namespace when available. + - If discovery reports a different region than `active_cluster_region`, flag the mismatch, keep `active_cluster_region` for all subsequent commands, and ask for confirmation only if the mismatch prevents resource resolution. + - Prompt only for fields that remain missing after discovery. + - **Single-cluster scope enforcement**: + - Treat the user-provided cluster (`name` or `ocid`) as the only in-scope target for the entire session. + - Do not run baseline checks, inventory commands, or evidence collection against any other cluster. + - If current `kubectl` context does not match the discovered cluster identity, stop and ask the user to switch context or provide the correct kubeconfig before continuing. + - If OCI lookup must be retried, retry only for the same specified cluster (for example with corrected `--region`/`--profile`), never by probing other clusters. +3. **Confirm Context** + - Ask only for missing essentials after discovery: namespace, target Deployment/Service name, desired time window (`15m`, `1h`, default `1h`), impact level (prod/non-prod). +4. **Tool Availability Checks** + - Run `kubectl version --client` and `oci --version`. + - Record `KUBECTL_AVAILABLE`/`OCI_AVAILABLE` booleans. If a CLI is missing, inform the user that evidence will be partial and continue with available tools. +5. **Session State** + - Initialize state structure: + ```json + { + "symptom": "...", + "namespace": "...", + "time_window": "1h", + "cluster_ocid": "...", + "compartment_ocid": "...", + "region": "...", + "domains": [], + "dependency_map": { + "entrypoint": "", + "hops": [], + "critical_path": [], + "latency_budget_ms": {} + }, + "fallbacks": {"kubectl": false, "oci": false}, + "evidence": [], + "node_doctor": { + "enabled": false, + "execution_mode": "ask_then_execute", + "image": "", + "targets": [], + "results": [] + } + } + ``` + +--- + +## Phase 1 — Symptom Triage +1. Load `symptom-triage.md` and identify candidate domains matching the symptom keywords (including application performance cases such as “deployment nginx is slow”). +2. Present the suggested domains to the user with brief rationales. Allow them to: + - Confirm the list. + - Add or remove domains. + - Provide additional focus (specific pod, service, node pool, PVC, IAM entity). +3. For application latency symptoms, model dependency context before evidence collection: + - Capture request entrypoint (Ingress/API/Job), target deployment, and downstream services (internal and external). + - Mark critical-path dependencies vs optional/background calls. + - Capture baseline latency and per-hop budget when known. +4. Capture clarifying answers (from the table's questions) and store them in session state (e.g., `POD_NAME`, `SERVICE_NAME`, `DEPLOYMENT_NAME`, `LABEL_SELECTOR`, `BASELINE_LATENCY`, `DEPENDENCY_MAP`). + +--- + +## Phase 2 — Dependency Path Modeling +1. Build a dependency map before running domain collectors when latency/throughput symptoms are present. +2. Dependency map structure: + ```json + { + "entrypoint": "ingress/payments", + "hops": [ + {"from": "ingress/payments", "to": "deployment/payments-api", "protocol": "HTTP"}, + {"from": "deployment/payments-api", "to": "svc/orders", "protocol": "gRPC"}, + {"from": "deployment/payments-api", "to": "svc/redis", "protocol": "TCP"} + ], + "critical_path": ["ingress/payments->deployment/payments-api", "deployment/payments-api->svc/orders"], + "latency_budget_ms": { + "end_to_end_p99": 500, + "ingress/payments->deployment/payments-api": 120, + "deployment/payments-api->svc/orders": 220 + } + } + ``` +3. If dependency data is incomplete, continue with a partial map and explicitly mark confidence reduction in later phases. + +--- + +## Phase 3 — Evidence Collection +1. Build the OCI object correlation graph before domain-specific collectors when enough selectors are known. + - Run the correlator with all discovered selectors, even if only one target object is known: + ```bash + bash ../../scripts/oke-object-correlator.sh \ + --namespace \ + --cluster-id \ + --compartment-id \ + --region \ + [--pod ] \ + [--deployment ] \ + [--service ] \ + [--ingress ] \ + [--pvc ] \ + [--node ] + ``` + - Treat the output as evidence with fields: `domain`, `graph.kubernetes`, `graph.oci`, `graph.edges`, `findings`, `anomalies`, `raw_snippets`, and `fallback_used`. + - Use the graph to narrow follow-on checks. Examples: + - If a Service maps to an OCI Load Balancer with unhealthy backend health, focus on backend set, node subnets, NSGs, security lists, route tables, gateways or peering paths, and endpoint readiness. + - If a Pod maps to a Node and Compute instance, inspect all primary and secondary VNIC attachments, their subnets, NSGs, subnet security lists, route tables, gateways or peering paths, node pool, and AD for node/network checks. + - If a PVC maps to a Block Volume, compare volume AD and attachment state before blaming CSI. + - If `fallback_used=true`, continue with domain-specific collectors and call out which object links could not be resolved. +2. For each selected domain: + - Look up required commands in `evidence-collectors.md`. + - Build command batches with placeholders filled (namespace, resource names, compartment OCID, time window, and dependency hop identifiers when present). + - **Auto-run read-only evidence commands without prompting** when tools are available. + - Never auto-run remediation or mutating commands. Present the exact command, explain the expected impact, and wait for explicit user approval before running it. + - Example command item: + ```json + { + "cmd": "kubectl describe pod trainer-0 -n ml-team", + "purpose": "Inspect scheduling events" + } + ``` + - For Networking/LB investigations, prefer the dedicated LB collector when delegation is available. Otherwise run the LB commands from `evidence-collectors.md` locally and normalize the same output fields in the parent skill. + - Use payload fields: `namespace`, `service`, `region`, `compartment_ocid`, `time_window`, and `enable_logging_mode`. + - Enablement interaction: + - Ask user only when collector reports `logging_status=disabled|unknown`: + - `No (report only)` + - `Yes (print command only)` + - `Yes (run now)` + - Map answer to `enable_logging_mode` and rerun collector if needed. + - Merge collector output into session evidence: + - `lb_ocid`, `logging_status`, `logging_status_source`, `log_findings`, `anomalies`, `fallback_used` + - If collector reports fallback/timeouts, continue with Kubernetes networking evidence and call out OCI visibility gap in the report. + - For OKE-specific domains, prefer the dedicated helper script before generic command batches: + - OKE Add-ons Health: + ```bash + bash ../../scripts/oke-addon-health.sh --namespace kube-system + ``` + - Pod Networking / OCI CNI / IPAM: + ```bash + bash ../../scripts/oke-pod-network-check.sh --namespace [--pod ] [--selector ] + ``` + - Cluster Autoscaler / Node Pool Scaling: + ```bash + bash ../../scripts/oke-autoscaler-check.sh --namespace [--deployment ] --cluster-id --compartment-id --region + ``` + - DNS / Service Discovery: + ```bash + bash ../../scripts/oke-dns-check.sh --namespace [--service ] [--pod ] [--lookup ] + ``` + - Ingress / OCI Native Ingress: + ```bash + bash ../../scripts/oke-ingress-check.sh --namespace --ingress [--region ] + ``` + - Private Cluster / API Endpoint Connectivity: + ```bash + bash ../../scripts/oke-private-endpoint-check.sh --cluster-id --region [--compartment-id ] + ``` + - OCIR / Image Pull: + ```bash + bash ../../scripts/oke-ocir-image-pull-check.sh --namespace --pod [--image ] [--compartment-id ] [--region ] + ``` + - Workload Identity / OCI API From Pods: + ```bash + bash ../../scripts/oke-workload-identity-check.sh --namespace --serviceaccount [--pod ] [--tenancy-id ] + ``` + - Incident Timeline: + ```bash + bash ../../scripts/oke-incident-timeline.sh --namespace [--pod ] [--deployment ] [--service ] [--compartment-id ] [--region ] + ``` + - Treat helper JSON output as evidence with fields: `domain`, `findings`, `anomalies`, `raw_snippets`, and `fallback_used`. + - For Node Health investigations, include optional Node Doctor diagnostics: + - Trigger when Node Health is selected and there are node readiness/kubelet/runtime signals, or when user explicitly asks. + - Scope starts with one candidate node first, then ask whether to continue to additional nodes. + - Default debug image to `docker.io/library/ubuntu` each run (`kubectl debug ... --image=`), and allow user override. Keep the selected image in session for additional nodes unless user changes it. + - Before execution, present the exact sequence and ask explicit confirmation per node: + 1) `bash ../../scripts/node-doctor-run.sh --node --image ` + 2) (script executes `kubectl debug` + `chroot /host` + `sudo /usr/local/bin/node-doctor.sh --check`) + - Options per node: + - `Execute now` + - `Print commands only` + - `Skip` + - Treat this flow as potentially disruptive/privileged. Never auto-run without confirmation. + - Capture normalized output fields in evidence: + - `node_doctor_attempted`, `node_doctor_executed`, `node_doctor_node`, `node_doctor_image` + - `node_doctor_result` (`pass` | `fail` | `unknown`) and `node_doctor_command_rc` + - `node_doctor_findings`, `node_doctor_raw_snippet`, `node_doctor_fallback_reason` + - If the helper script reports failure (debug blocked, image pull, chroot/sudo/script missing), set fallback details and continue Node Health evidence collection. +3. Assemble collector input payload: + ```json + { + "symptom": "...", + "domains": ["Pod Scheduling"], + "namespace": "...", + "time_window": "...", + "selectors": {"pod": "...", "service": "...", "deployment": "...", "label": "..."}, + "dependency_map": { + "entrypoint": "...", + "hops": [], + "critical_path": [], + "latency_budget_ms": {} + }, + "object_graph": {...}, + "fallbacks": {"kubectl": false, "oci": true}, + "compartment_ocid": "..." + } + ``` +4. Execute the prepared command list. + - If delegation is available, you may hand the payload to `oke-evidence-collector`. + - Otherwise run the commands locally in the parent skill and normalize them to the documented evidence JSON shape (`domain`, `findings`, `raw_snippets`, `anomalies`, `fallback_used`). + - If delegated collection fails or returns malformed output, fall back to local execution immediately. +5. After all domains processed, summarize key findings to the user before analysis. Note any `fallback_used` signals or missing data. + +--- + +## Phase 4 — Hypothesis Ranking +1. Construct analyst payload containing: + ```json + { + "symptom": "...", + "domains": [...], + "dependency_map": {...}, + "object_graph": {...}, + "evidence": [...], + "fallbacks": {"kubectl": false, "oci": true} + } + ``` +2. Rank hypotheses. + - If delegation is available, you may use `oke-hypothesis-analyst`. + - Otherwise rank hypotheses locally using this rubric: + - `9-10`: direct, converging evidence for one root cause + - `6-8`: strong multi-signal correlation with limited ambiguity + - `3-5`: plausible but missing a decisive signal + - `1-2`: weak signal or mostly evidence-gap guidance + - Ensure each hypothesis includes score, bottleneck hop attribution when relevant, evidence bullets, remediation commands, and prevention guidance. + - Prefer hypotheses that are supported by explicit graph edges across Kubernetes and OCI resources over hypotheses supported only by isolated symptoms. + - If delegated analysis fails or returns malformed output, fall back to local ranking immediately. +3. Validate that evidence quotes reference actual snippets collected. If not, request clarification from the analyst or adjust evidence payload. + +--- + +## Phase 5 — Report & Next Steps +1. Load `final-report-template.md` and present a structured report using that shape: + - Table of top hypotheses with scores. + - Highlight confidence level (e.g., `High`, `Medium`, `Low` based on score thresholds). + - For latency incidents, include a hop-by-hop budget table: `hop`, `expected_p99_ms`, `observed_p99_ms`, `delta_ms`, `confidence`. + - Remediation commands rendered in fenced code blocks, prefixed with comments where necessary. Do not execute them unless the user approves the exact command or action after seeing it. + - Prevention recommendations as concise bullet points. +2. Call out any limitations: missing tooling, commands that failed, domains not yet explored, and missing dependency telemetry. +3. Offer next actions: + - Rerun for another namespace/resource. + - Deep-dive into IAM, capacity, or service-limit evidence when the collected data points there. + - Export findings to a file (future enhancement). +4. Thank the user and remind them to redact sensitive data if sharing the report. + +--- + +## Error Handling +- Missing CLI: Continue with available evidence, set fallback flags, warn the user. +- Permission denied or forbidden: include remediation (e.g., "ensure tenancy OCID has access to compartment"). +- Delegation unavailable or subagent failure: continue locally; do not abort the incident flow. +- Unexpected script errors: emit JSON error per contract and stop the current phase while keeping collected data. + +## Security & Logging +- Do not echo secret values or service account tokens. Redact with `***`. +- Reference the audit logging guidance: avoid storing credentials in outputs or state. +- Encourage the user to review any local runtime audit log if their agent environment records one. + +--- + +## Invocation Examples +- `Use the OKE troubleshooter for pods stuck Pending in prod namespace` +- `Use the OKE troubleshooter for lb service has no IP us-phoenix-1` +- `Use the OKE troubleshooter for cluster api timing out` +- `Use the OKE troubleshooter for customer is indicating poor performance for deployment` +- `Use the OKE troubleshooter for CoreDNS timeouts in prod` +- `Use the OKE troubleshooter for cluster autoscaler is not adding nodes` +- `Use the OKE troubleshooter for pods fail sandbox creation with OCI CNI IPAM errors` +- `Use the OKE troubleshooter for OCIR ImagePullBackOff unauthorized` +- `/oke-troubleshooter "workload identity pod gets NotAuthorized"` +- `/oke-troubleshooter "private OKE API endpoint unreachable"` +- `/oke-troubleshooter "OCI native ingress TLS backend errors"` + +## Latency Walkthrough (Dependency-Aware) +Use this pattern when the incident is "deployment is slow" and the deployment depends on other services. + +1. **Input Example** + - Symptom: `"payments API p99 jumped from 350ms to 1.8s"` + - Namespace: `prod` + - Deployment: `payments-api` + - Time window: `1h` +2. **Dependency Map Example** + ```json + { + "entrypoint": "ingress/payments", + "hops": [ + {"from": "ingress/payments", "to": "deployment/payments-api", "protocol": "HTTP"}, + {"from": "deployment/payments-api", "to": "svc/orders", "protocol": "gRPC"}, + {"from": "deployment/payments-api", "to": "svc/redis", "protocol": "TCP"} + ], + "critical_path": [ + "ingress/payments->deployment/payments-api", + "deployment/payments-api->svc/orders" + ], + "latency_budget_ms": { + "end_to_end_p99": 500, + "ingress/payments->deployment/payments-api": 120, + "deployment/payments-api->svc/orders": 220, + "deployment/payments-api->svc/redis": 80 + } + } + ``` +3. **Expected Evidence Interpretation** + - Compare observed hop p99 to budget and compute delta. + - Identify the largest over-budget hop on critical path first. + - Validate with both client-side and server-side evidence when possible. +4. **Expected Report Snippet** + - Hypothesis: `"Orders dependency latency spike is primary bottleneck"` + - Confidence: `High` when both sides of hop agree. + - Budget table: + + | Hop | Expected p99 (ms) | Observed p99 (ms) | Delta (ms) | Confidence | + |-----|-------------------|-------------------|------------|------------| + | ingress/payments->payments-api | 120 | 140 | +20 | Medium | + | payments-api->orders | 220 | 980 | +760 | High | + | payments-api->redis | 80 | 95 | +15 | Medium | + + - Remediation should target `payments-api->orders` first, then re-measure end-to-end p99. + +The skill should deliver actionable insight even when only partial data is available. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengdownloadkubeconfigfile.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengconfiguringclusteraddons.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +- https://docs.oracle.com/en-us/iaas/Content/Balance/home.htm +- https://docs.oracle.com/en-us/iaas/Content/Logging/home.htm +- https://kubernetes.io/docs/tasks/debug/debug-application/ +- https://kubernetes.io/docs/reference/kubectl/ diff --git a/oci/oke/skills/oke-troubleshooter/agents/openai.yaml b/oci/oke/skills/oke-troubleshooter/agents/openai.yaml new file mode 100644 index 0000000..c9b8604 --- /dev/null +++ b/oci/oke/skills/oke-troubleshooter/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "OKE Troubleshooter" + short_description: "Troubleshoot OKE workloads and clusters." + default_prompt: "Use $oke-troubleshooter to collect read-only OKE evidence, correlate Kubernetes objects to OCI resources, rank likely causes, and recommend approval-gated remediation steps." diff --git a/oci/oke/skills/oke-troubleshooter/evidence-collectors.md b/oci/oke/skills/oke-troubleshooter/evidence-collectors.md new file mode 100644 index 0000000..b781a80 --- /dev/null +++ b/oci/oke/skills/oke-troubleshooter/evidence-collectors.md @@ -0,0 +1,387 @@ +# `/oke-troubleshooter` — Evidence Collection Recipes + +For each diagnostic domain, gather the following evidence. Prefer JSON output (`-o json`) when available. Summaries returned to the parent skill should follow the structure: + +```json +{ + "domain": "", + "findings": ["Short bullet summary"], + "raw_snippets": ["Trimmed command output"], + "anomalies": ["Detected warnings/errors"], + "fallback_used": false +} +``` + +When a command fails, set `fallback_used` to `true`, capture stderr (sanitized), and continue with other evidence. + +## Contents +- Pod Scheduling +- Pod Runtime +- Node Health +- OKE Add-ons Health +- Pod Networking / OCI CNI / IPAM +- DPDK / SR-IOV / Mellanox mlx5 +- Cluster Autoscaler / Node Pool Scaling +- Networking / CNI / Load Balancer +- DNS / Service Discovery +- Ingress / OCI Native Ingress +- Private Cluster / API Endpoint Connectivity +- OCIR / Image Pull +- Workload Identity / OCI API From Pods +- Incident Timeline +- Application Performance +- Dependency Path +- Storage / CSI +- Control Plane +- IAM / RBAC +- OCI Infrastructure / Capacity and Service Limits + +## Pod Scheduling +- **Kubernetes** + - `kubectl get pods -n -o wide` + - `kubectl describe pod -n ` + - `kubectl get events -n --field-selector involvedObject.name= --sort-by=.lastTimestamp` +- **OCI** + - `oci ce node-pool list --compartment-id ` (ensure node pool capacity) + - `oci limits resource-availability get --compartment-id --service-name compute --limit-name --availability-domain ` + - TODO(live validation): resolve the exact compute `limit-name` for the target shape from `oci limits value list` before treating remaining capacity as conclusive. +- **Normalization tips**: Highlight scheduling failure reasons (`0/3 nodes available`, taints), summarize resource requests vs. node capacity, include current node pool size. + +## Pod Runtime +- **Kubernetes** + - `kubectl describe pod -n ` + - `kubectl logs -n --previous` (when restart count > 0) + - `kubectl get events -n --field-selector type=Warning` +- **OCI** + - `oci logging search --time-start --time-end --search-query "search where podName = ''"` +- **Normalization tips**: Capture container state (`Waiting`, `CrashLoopBackOff`), include last log lines causing failure, flag missing secrets or configmaps. + +## Node Health +- **Kubernetes** + - `kubectl get nodes -o wide` + - `kubectl describe node ` + - `kubectl top node ` (requires metrics server) +- **OCI** + - `oci ce node-pool get --node-pool-id ` + - `oci compute instance get --instance-id ` + - `oci health-check probe-result get --probe-configuration-id ` (if using health checks) +- **Normalization tips**: Surface conditions not `True`, kubelet versions, OCI lifecycle state, recent maintenance events. +- **Advanced: Node Doctor (OKE node deep diagnostics)** + - Use when Node Health symptoms indicate node readiness/runtime faults, or when user explicitly requests deep node checks. + - Potentially disruptive/privileged; require explicit confirmation per node before execution. + - Start with one affected node first, then ask whether to continue on more nodes. + - Use default debug image `docker.io/library/ubuntu` each run (allow user override) and execute via helper script: + - `bash ../../scripts/node-doctor-run.sh --node --image docker.io/library/ubuntu [--namespace ]` + - Under the hood this runs: + 1) `kubectl -n debug node/ --image=` + 2) `chroot /host` + 3) `sudo /usr/local/bin/node-doctor.sh --check` + - If execution is not approved, print commands only and continue other Node Health evidence. + - Failure handling: + - capture and continue when `kubectl debug` is blocked, image pull fails, `chroot` fails, `sudo` missing, or `/usr/local/bin/node-doctor.sh` not found. + - Script output is normalized JSON, including: + - `node_doctor_attempted`, `node_doctor_executed`, `node_doctor_node`, `node_doctor_image` + - `node_doctor_result` (`pass` | `fail` | `unknown`) and `node_doctor_command_rc` + - `node_doctor_findings`, `node_doctor_raw_snippet`, `node_doctor_fallback_reason` + - `node_doctor_counts` (`pass`/`fail`/`warn`/`skip`) + +## OKE Add-ons Health +- **Helper script** + - `bash ../../scripts/oke-addon-health.sh --namespace kube-system` +- **Kubernetes** + - `kubectl -n kube-system get pods -o wide` + - `kubectl -n kube-system get deploy -o wide` + - `kubectl -n kube-system get ds -o wide` + - `kubectl -n kube-system get events --field-selector type=Warning --sort-by=.lastTimestamp` + - `kubectl -n kube-system logs deployment/coredns --tail=200` when DNS symptoms are present +- **Normalization tips**: Flag kube-system pods that are not Running/Ready, add-on deployments with unavailable replicas, daemonsets missing scheduled pods, and warning events after cluster upgrades or node pool changes. + +## Pod Networking / OCI CNI / IPAM +- **Helper script** + - `bash ../../scripts/oke-pod-network-check.sh --namespace [--pod ] [--selector ]` +- **Kubernetes** + - `kubectl -n describe pod ` + - `kubectl -n get pod -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'` + - `kubectl -n get events --field-selector type=Warning --sort-by=.lastTimestamp` + - `kubectl -n kube-system get pods -l app=oci-cni -o wide` + - `kubectl -n kube-system get pods -l name=multus -o wide` + - `kubectl get network-attachment-definitions -A` +- **Node-side checks** + - Use a privileged diagnostic pod or `kubectl debug node/` only after confirmation. + - Inspect `/opt/cni/bin` for required binaries such as `oci-ipam`, `oci-ipvlan`, `oci-ptp`, and `ipvlan`. + - Inspect `/dev/shm/oci-cni/free` and `/dev/shm/oci-cni/used` when OCI IPAM allocation state is in question. +- **Normalization tips**: Separate OCI CNI primary-pod-network failures from Multus secondary-network failures. Flag `FailedCreatePodSandBox`, missing CNI binaries, subnet/IP exhaustion, stale or missing `network-status`, and NAD namespace/name mismatches. + +## DPDK / SR-IOV / Mellanox mlx5 +- **When to use** + - User mentions DPDK, SR-IOV, Mellanox/NVIDIA, `mlx5`, `vfio-pci`, `/dev/infiniband`, `ibv_devices`, `dpdk-devbind.py`, hugepages, or a pod that requests device-plugin resources but lacks expected Multus interfaces. + - Also load `../oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md`. +- **Kubernetes** + - `kubectl -n describe pod ` + - `kubectl -n get pod -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'` + - `kubectl get network-attachment-definitions -A | egrep '|noiommu|sriov|mlx|dpdk'` + - `kubectl -n get network-attachment-definition -o yaml` + - `kubectl -n get events --field-selector involvedObject.name= --sort-by=.lastTimestamp` +- **Pod exec checks** + - `kubectl -n exec -- ip -br addr` + - `kubectl -n exec -- ls -l /dev/infiniband` + - `kubectl -n exec -- ibv_devices` + - `kubectl -n exec -- lspci -nnk` + - `kubectl -n exec -- dpdk-devbind.py -s` + - `kubectl -n exec -- grep -i Huge /proc/meminfo` + - `kubectl -n exec -- mount | grep -i huge` + - `kubectl -n logs | egrep -i 'EAL|mlx5|ibv|verbs|huge|vfio|pci|rte'` +- **Node-side checks** + - Use a privileged diagnostic pod, `kubectl debug node/`, or SSH only after confirmation. + - `ip -br addr` + - `lspci -nnk | egrep -A3 'Mellanox|NVIDIA|Ethernet'` + - `ls -l /dev/vfio` + - `find /sys/kernel/iommu_groups -maxdepth 1 -mindepth 1 -type d | wc -l` + - `dpdk-devbind.py -s` +- **Normalization tips**: Keep resource allocation, Multus attachment, driver binding, RDMA/verbs exposure, hugepage visibility, and application PCI/interface mapping as separate facts. If `network-status` only shows `eth0`, do not claim SR-IOV/Multus attachment is working just because the pod is `Running`. For Mellanox `mlx5` PMD, do not assume `vfio-pci` is correct; `mlx5_core` plus RDMA/verbs may be the intended model. + +## Cluster Autoscaler / Node Pool Scaling +- **Helper script** + - `bash ../../scripts/oke-autoscaler-check.sh --namespace [--deployment ] --cluster-id --compartment-id --region ` +- **Kubernetes** + - `kubectl -n get pods --field-selector=status.phase=Pending -o wide` + - `kubectl -n get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp` + - `kubectl -n kube-system get deploy cluster-autoscaler -o wide` + - `kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200` + - `kubectl -n describe deployment ` +- **OCI** + - `oci ce node-pool list --compartment-id --cluster-id --region --all --output json` + - `oci ce node-pool get --node-pool-id --region ` + - `oci limits resource-availability list --compartment-id --service-name compute --region ` +- **Normalization tips**: Distinguish Kubernetes scheduling constraints from autoscaler refusal. Flag max node pool size reached, no matching node group, shape capacity/limit errors, subnet IP exhaustion, taints/tolerations mismatch, and missing autoscaler deployment. + +## Networking / CNI / Load Balancer +- **Kubernetes** + - `kubectl get svc -n -o yaml` + - `kubectl get svc -n -o jsonpath='{.status.loadBalancer.ingress[0].ip}'` (capture LB public IP when type is `LoadBalancer`) + - `kubectl get ingress -n -o yaml` + - `kubectl describe networkpolicy -n ` + - `kubectl get pods -n kube-system -l k8s-app=cilium-agent` (or corresponding CNI) +- **OCI** + - `oci lb load-balancer list --compartment-id --region --all --output json | jq -r '.data[] | select((."ip-addresses" // []) | any(."ip-address"=="")) | .id'` (resolve LB OCID from service external IP) + - `oci lb load-balancer get --load-balancer-id --region ` + - `oci lb load-balancer get --load-balancer-id --region --query 'data."access-log"' --output json` (check whether LB access logging is enabled) + - `oci logging log-group list --compartment-id --region --all --output json` (list candidate log groups) + - `oci logging log list --log-group-id --all --query "data[?configuration.source.resource=='' && configuration.source.service=='loadbalancer'].[\"display-name\",id,\"is-enabled\",configuration]" --output json` (second check for logging objects tied to LB OCID) + - `oci logging search search-logs --region --search-query "search \"/\" | where data.loadBalancerId = '' | sort by datetime desc" --time-start --time-end ` (when LB logs are enabled) + - `oci nlb network-load-balancer list --compartment-id --region --all --output json | jq -r '.data[] | select((."ip-addresses" // []) | any(."ip-address"=="")) | .id'` (fallback if classic LB lookup is empty) + - `oci network nsg list --compartment-id ` + - `oci network subnet get --subnet-id ` + - `oci network security-list get --security-list-id ` + - `oci network route-table get --rt-id ` +- **Subagent preference** + - Prefer delegating LB-specific discovery/log retrieval to `oke-lb-log-collector`: + - resolves LB/NLB OCID from Service IP + - checks access-log status + - optionally enables logging (with explicit user approval path) + - extracts log issue signals for ranking +- **Normalization tips**: Note load balancer lifecycle (`PROVISIONING`, `FAILED`), security list/NSG rules, route table targets, gateway or peering route targets, CNI pod status, and service annotations impacting provisioning. Explicitly record LB logging status as `enabled`, `disabled`, or `unknown`, and include `logging_status_source` showing which check(s) succeeded. +- **TODO(live validation)**: Confirm the exact OCI CLI fields and Logging Search syntax for LB access-log discovery in the target OCI CLI version before treating log absence as conclusive. +- **If LB logs are disabled or unknown**: recommend enabling access logs before closing the incident so future RCA has request-level evidence. + - Offer operator action: + - `No (report only)` + - `Yes (print enable command)` + - `Yes (execute enable command now)` + - Enable command template: + ```bash + oci lb load-balancer update \ + --load-balancer-id \ + --region \ + --access-log '{"isEnabled":true,"logGroupId":"","logId":""}' + ``` + - Post-check: + ```bash + oci lb load-balancer get \ + --load-balancer-id \ + --region \ + --query 'data."access-log"' \ + --output json + ``` +- **If LB logs are enabled**: summarize concrete issue signals from log lines: + - 5xx rate and top failing paths/backends + - timeout/reset/error signatures + - highest observed latency fields in the selected window + +## DNS / Service Discovery +- **Helper script** + - `bash ../../scripts/oke-dns-check.sh --namespace [--service ] [--pod ] [--lookup ]` +- **Kubernetes** + - `kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide` + - `kubectl -n kube-system get deploy coredns -o wide` + - `kubectl -n kube-system get configmap coredns -o yaml` + - `kubectl -n kube-system logs deployment/coredns --tail=200` + - `kubectl -n get svc -o yaml` + - `kubectl -n get endpoints -o yaml` + - `kubectl -n get endpointslices -l kubernetes.io/service-name= -o yaml` + - `kubectl -n exec -- nslookup ` +- **Normalization tips**: Separate DNS server health from service object problems. Flag CoreDNS unavailable, ConfigMap rewrite/stub-domain issues, Service without endpoints, EndpointSlice readiness issues, `NXDOMAIN`, `SERVFAIL`, timeout, and pod-local resolver failures. + +## Ingress / OCI Native Ingress +- **Helper script** + - `bash ../../scripts/oke-ingress-check.sh --namespace --ingress [--region ]` +- **Kubernetes** + - `kubectl get ingressclass -o yaml` + - `kubectl -n get ingress -o yaml` + - `kubectl -n describe ingress ` + - `kubectl -n kube-system get pods -l app.kubernetes.io/name=oci-native-ingress-controller -o wide` + - `kubectl -n kube-system logs -l app.kubernetes.io/name=oci-native-ingress-controller --tail=200` + - `kubectl -n get secret -o yaml` +- **OCI** + - `oci lb load-balancer get --load-balancer-id --region ` + - `oci lb listener list --load-balancer-id --region ` + - `oci lb backend-set-health get --load-balancer-id --backend-set-name --region ` +- **Normalization tips**: Flag missing ingress class, controller reconciliation errors, TLS secret/certificate mismatch, listener/backend-set mismatch, backend health failures, and annotation drift. + +## Private Cluster / API Endpoint Connectivity +- **Helper script** + - `bash ../../scripts/oke-private-endpoint-check.sh --cluster-id --region [--compartment-id ]` +- **Kubernetes / workstation** + - `kubectl config current-context` + - `kubectl cluster-info` + - `kubectl get --raw=/readyz?verbose` + - `kubectl version` +- **OCI** + - `oci ce cluster get --cluster-id --region ` + - `oci network subnet get --subnet-id --region ` + - `oci network nsg list --compartment-id --region ` + - `oci network security-list get --security-list-id --region ` + - `oci network route-table get --rt-id --region ` +- **Normalization tips**: Separate kubeconfig exec/auth failures from network reachability. Flag expired OCI security tokens, private endpoint DNS/routing problems, NSG/security-list blocks, missing gateway or peering routes, and public/private endpoint expectation mismatch. + +## OCIR / Image Pull +- **Helper script** + - `bash ../../scripts/oke-ocir-image-pull-check.sh --namespace --pod [--image ] [--compartment-id ] [--region ]` +- **Kubernetes** + - `kubectl -n describe pod ` + - `kubectl -n get secret -o yaml` + - `kubectl -n get serviceaccount -o yaml` + - `kubectl -n get events --field-selector involvedObject.name= --sort-by=.lastTimestamp` +- **OCI** + - `oci artifacts container repository list --compartment-id --region ` + - `oci artifacts container image list --compartment-id --repository-name --region ` +- **Normalization tips**: Flag image region mismatch, missing namespace-local pull secret, expired auth token, wrong tenancy namespace, repository not found, and node egress/DNS failures to OCIR. + +## Workload Identity / OCI API From Pods +- **Helper script** + - `bash ../../scripts/oke-workload-identity-check.sh --namespace --serviceaccount [--pod ] [--tenancy-id ]` +- **Kubernetes** + - `kubectl -n get serviceaccount -o yaml` + - `kubectl -n describe pod ` + - `kubectl -n logs --tail=200 | egrep -i "notauthorized|forbidden|401|403|principal|workload|token|oci"` +- **OCI** + - `oci iam policy list --compartment-id --all` +- **Normalization tips**: Flag missing service account annotations, workload identity policy condition gaps, token projection failures, and pod code using the wrong OCI auth provider. Do not treat dynamic groups as the primary authorization path for OKE Workload Identity. + +## Incident Timeline +- **Helper script** + - `bash ../../scripts/oke-incident-timeline.sh --namespace [--pod ] [--deployment ] [--service ] [--compartment-id ] [--region ]` +- **Purpose** + - Merge event timing from Kubernetes objects and OCI alarms so the final report can explain what changed first. +- **Kubernetes** + - `kubectl -n get events --sort-by=.lastTimestamp` + - `kubectl -n describe pod ` + - `kubectl -n rollout history deployment/` + - `kubectl -n describe deployment ` + - `kubectl -n describe service ` +- **OCI** + - `oci monitoring alarm-status list-alarms-status --compartment-id --status FIRING --region --all` +- **Normalization tips**: Prefer chronology over volume. Keep warning/failure events, rollout changes, object readiness changes, and firing alarms. Use the timeline to avoid blaming symptoms that occurred after the first failing signal. + +## Application Performance +- **Kubernetes** + - `kubectl get deployment -n -o yaml` + - `kubectl describe deployment -n ` + - `kubectl rollout history deployment/ -n ` + - `kubectl top pods -n -l app=