From d29d70ed963cfd4c67a0305f78b5ec6303cd5291 Mon Sep 17 00:00:00 2001 From: Chiping Hwang Date: Thu, 28 May 2026 10:03:53 -0700 Subject: [PATCH 1/6] Add OKE skills to OCI domain Signed-off-by: Chiping Hwang --- README.md | 9 +- oci/SKILL.md | 59 ++++++++++- oci/oke/cluster-design.md | 115 ++++++++++++++++++++ oci/oke/gva-node-pools.md | 181 ++++++++++++++++++++++++++++++++ oci/oke/multus-multihome.md | 202 ++++++++++++++++++++++++++++++++++++ oci/oke/troubleshooting.md | 164 +++++++++++++++++++++++++++++ 6 files changed, 726 insertions(+), 4 deletions(-) create mode 100644 oci/oke/cluster-design.md create mode 100644 oci/oke/gva-node-pools.md create mode 100644 oci/oke/multus-multihome.md create mode 100644 oci/oke/troubleshooting.md diff --git a/README.md b/README.md index 098873c..3db95ce 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ npx skills add oracle/skills/graal ## Domains - `db/` is the active Oracle Database domain and includes database, ORDS, SQLcl, framework, container, and agent workflow skills. -- `oci/` is the root for future Oracle Cloud Infrastructure skills. +- `oci/` contains Oracle Cloud Infrastructure skills, starting with OCI Kubernetes Engine cluster design, troubleshooting, Generic VNIC Attachment, and Multus pod networking. - `fusion/` is the root for future Oracle Fusion skills. - `apex/` is the root for future Oracle APEX skills. - `graal/` contains GraalVM skills, starting with Native Image. @@ -70,7 +70,12 @@ npx skills add oracle/skills/graal │ ├── reachability-metadata.md │ └── troubleshooting.md └── oci/ - └── SKILL.md + ├── SKILL.md + └── oke/ + ├── cluster-design.md + ├── troubleshooting.md + ├── gva-node-pools.md + └── multus-multihome.md ``` Each domain has its own `SKILL.md` and any supporting index files it needs. diff --git a/oci/SKILL.md b/oci/SKILL.md index 957f0cd..c2cc6c7 100644 --- a/oci/SKILL.md +++ b/oci/SKILL.md @@ -1,5 +1,60 @@ +--- +name: oci +description: Oracle Cloud Infrastructure guidance for designing, operating, and troubleshooting OCI services, starting with OCI Kubernetes Engine (OKE). Use when the user asks about OKE cluster design, Terraform or Resource Manager planning, OKE incident troubleshooting, Generic VNIC Attachment, Multus, pod networking, node pools, add-ons, ingress, load balancers, OCIR image pulls, Workload Identity, or Kubernetes workloads on OCI. +--- + # Oracle Cloud Infrastructure Skills -This file is a sample domain skeleton. +Use this domain for practical Oracle Cloud Infrastructure guidance. The current content focuses on OCI Kubernetes Engine (OKE): cluster design, operational troubleshooting, Generic VNIC Attachment (GVA), and Multus multi-interface pod validation. + +## How to Use This Domain + +1. Start with the routing table below. +2. Read only the OKE file that matches the user's task. +3. Prefer official Oracle documentation and live read-only discovery commands before making design or remediation recommendations. +4. Ask before running commands that create, update, delete, restart, scale, drain, or otherwise mutate OCI or Kubernetes resources. + +## Directory Structure + +```text +oci/ +|-- SKILL.md +`-- oke/ + |-- cluster-design.md + |-- troubleshooting.md + |-- gva-node-pools.md + `-- multus-multihome.md +``` + +## Category Routing + +| Topic | File | +|-------|------| +| Design or scaffold an OKE cluster, Terraform stack, or OCI Resource Manager stack | `oci/oke/cluster-design.md` | +| Troubleshoot OKE workloads, pods, services, DNS, add-ons, ingress, load balancers, image pulls, storage, Workload Identity, or cluster access | `oci/oke/troubleshooting.md` | +| Configure OKE managed node pools with Generic VNIC Attachment secondary VNIC profiles and Application Resources | `oci/oke/gva-node-pools.md` | +| Deploy or validate Multus NetworkAttachmentDefinitions and multi-interface pods on OKE | `oci/oke/multus-multihome.md` | + +## Key Starting Points + +- `oci/oke/cluster-design.md` +- `oci/oke/troubleshooting.md` +- `oci/oke/gva-node-pools.md` +- `oci/oke/multus-multihome.md` + +## Common Multi-Step Flows + +| Task | Recommended Sequence | +|------|----------------------| +| Plan a production OKE cluster | `oke/cluster-design.md` | +| Diagnose an OKE service with no load balancer IP | `oke/troubleshooting.md` | +| Build a node pool with workload-specific secondary VNIC profiles | `oke/gva-node-pools.md` -> `oke/multus-multihome.md` if pods need multiple interfaces | +| Validate Multus pod networking on GVA-enabled nodes | `oke/multus-multihome.md` -> `oke/troubleshooting.md` if symptoms remain | +| Investigate OKE workload access to OCI APIs | `oke/troubleshooting.md` | + +## Sources -For the repo-wide structure and authoring pattern, read the root `README.md` and `SKILL_AUTHORING_GUIDE.md`. +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +- https://github.com/oracle-terraform-modules/terraform-oci-oke diff --git a/oci/oke/cluster-design.md b/oci/oke/cluster-design.md new file mode 100644 index 0000000..d46eeef --- /dev/null +++ b/oci/oke/cluster-design.md @@ -0,0 +1,115 @@ +# OKE Cluster Design and Terraform Planning + +## Overview + +Use this skill to design an OCI Kubernetes Engine (OKE) cluster and translate the design into Terraform or an OCI Resource Manager stack. It is intended for cluster creation, cluster modernization, and review of OKE infrastructure plans before deployment. + +Start from official OKE requirements, then map the chosen architecture to Terraform variables, Resource Manager schema inputs, and validation checks. + +## Design Sequence + +1. Confirm tenancy context: + - Region + - Compartment + - Existing VCN or new VCN + - Required environments such as dev, test, prod, or regulated workloads + +2. Choose cluster model: + - Enhanced cluster when using enhanced-cluster features such as managed add-ons and Workload Identity. + - Basic cluster only when the workload does not need enhanced-cluster features. + - Kubernetes version aligned with OKE-supported versions and node image compatibility. + +3. Choose API endpoint access: + - Private endpoint for restricted administration paths. + - Public endpoint only when external access is required and NSG/security-list rules are tightly scoped. + - Separate API endpoint, worker, pod, and load balancer subnets where practical. + +4. Choose pod networking: + - VCN-native pod networking when pods need VCN-routable IPs, virtual nodes, or multiple secondary VNIC pod networking. + - Flannel overlay only when VCN-routable pod IPs and VCN-native features are not required. + - Check pod CIDR and service CIDR overlap before finalizing the network plan. + +5. Design node pools: + - Use separate node pools for materially different workload profiles. + - Capture shape, OCPU, memory, node count, autoscaling boundaries, image source, labels, taints, boot volume, and placement subnets. + - Confirm shape availability, quota, and VNIC/pod capacity in the target region and availability domains. + +6. Select add-ons and integrations: + - CoreDNS and kube-proxy are baseline cluster components. + - Consider Cluster Autoscaler, OCI Native Ingress Controller, CSI drivers, monitoring, and logging based on workload needs. + - For OCI API access from pods, use Workload Identity on enhanced clusters. + +7. Define security and operations: + - Prefer private worker nodes. + - Use NSGs where possible for API endpoint, worker, pod, and load balancer traffic. + - Use least-privilege IAM policies. + - Decide whether to use Oracle-managed encryption or customer-managed keys. + - Define tagging, backup, upgrade, observability, and incident response expectations. + +## Fast Path Starter Design + +When the user asks for a quick starter stack, use conservative defaults and ask only for values that cannot be inferred: + +| Area | Starter value | +|------|---------------| +| Cluster type | Enhanced | +| API endpoint | Private | +| VCN | New VCN | +| VCN CIDR | `10.0.0.0/16` | +| CNI | VCN-native pod networking | +| Worker nodes | Private managed nodes | +| Node pool | 3 nodes, flexible general-purpose shape where available | +| Gateways | NAT gateway and service gateway | +| Add-ons | CoreDNS and kube-proxy, plus autoscaler when autoscaling is enabled | +| IAM | Create required policies or document manual policies | +| Workload Identity | Enabled when pods need OCI API access | +| Encryption | Oracle-managed unless a customer key is required | + +Before generating Terraform, show the starter design back to the user and ask whether they want to revise networking, node pools, access, add-ons, or security. + +## Terraform and Resource Manager Guidance + +When generating Terraform: + +- Prefer the official Oracle Terraform OKE module when it meets the design requirements. +- Pin or record the module version used for variable mapping. +- Keep required values explicit: region, tenancy OCID, compartment OCID, cluster name, Kubernetes version, network settings, and node pools. +- Do not guess module variable names. Verify them against the selected module version. +- Generate `terraform.tfvars.example` with placeholder OCIDs rather than real secrets. +- Include outputs for cluster OCID, node pool OCIDs, kubeconfig guidance, VCN/subnet IDs, and any created policy or key resources. +- If building an OCI Resource Manager stack, include a `schema.yaml` that groups inputs by audience and hides irrelevant fields when possible. + +## Validation Checklist + +Before presenting the final stack, check: + +- No CIDR overlap among VCN, pods, services, and connected networks. +- Kubernetes control plane and node versions are compatible. +- Chosen node images match the node Kubernetes version and shape architecture. +- Quotas and service limits are called out for node shapes, GPUs, load balancers, block volumes, and public IPs. +- Private nodes have egress through NAT and service gateway paths when needed. +- API endpoint access rules match the intended administration path. +- NSG and route-table assumptions are documented. +- Add-ons are valid for the selected cluster type. +- Workload Identity is only recommended for enhanced clusters. +- Customer-managed keys include required KMS permissions. +- Generated Terraform has clear validation steps such as `terraform fmt`, `terraform validate`, and `terraform plan`. + +## Common Mistakes + +- Selecting Flannel when the design requires VCN-routable pod IPs, virtual nodes, or multiple secondary VNIC pod networking. +- Using a public API endpoint without tightly scoped source CIDRs. +- Reusing one node pool for workloads with different security, cost, GPU, RDMA, or scaling requirements. +- Assuming shape availability or quota without checking the target region. +- Enabling autoscaling without confirming Cluster Autoscaler installation and node-pool min/max values. +- Mixing real tenancy-specific OCIDs into examples that will be committed to source control. +- Claiming a Terraform plan has been validated when only static generation was performed. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingclusterusingoke_topic-Using_the_Console_to_create_a_Custom_Cluster_with_Explicitly_Defined_Settings.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengworkingwithenhancedclusters.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +- https://github.com/oracle-terraform-modules/terraform-oci-oke diff --git a/oci/oke/gva-node-pools.md b/oci/oke/gva-node-pools.md new file mode 100644 index 0000000..438512f --- /dev/null +++ b/oci/oke/gva-node-pools.md @@ -0,0 +1,181 @@ +# OKE Generic VNIC Attachment Node Pools + +## Overview + +Use this skill to plan, create, or review OCI Kubernetes Engine (OKE) managed node pools that use Generic VNIC Attachment (GVA) secondary VNIC profiles for pod networking. + +GVA lets a node pool expose multiple secondary VNIC profiles, each with its own subnet, NSGs, display name, and pod IP allocation. Optional Application Resources let pods request one specific secondary VNIC profile through Kubernetes extended resources. + +## When to Use + +Use this skill when the task involves: + +- Secondary VNIC profiles on OKE managed node pools. +- Application Resources such as `oke-application-resource.oci.oraclecloud.com/frontend`. +- Workload-specific pod network segmentation. +- Creating or updating node pools with multiple pod networking paths. +- Validating pods pinned to a single secondary VNIC profile. + +Use `oci/oke/multus-multihome.md` instead when the pod needs multiple interfaces at the same time. + +## Prerequisites + +Confirm: + +- The cluster uses VCN-native pod networking. +- The target node pool is a managed node pool. +- The selected shape supports the required number of VNICs and pod IP capacity. +- Secondary VNIC subnets and NSGs already exist or are included in the infrastructure plan. +- Subnet CIDR capacity is sufficient for the requested `ipCount`. +- OCI CLI and `kubectl` are available when live discovery or validation is requested. +- IAM permissions allow the operator to manage the node-pool and networking resources. + +Multiple secondary VNIC pod networking is not supported with Flannel overlay networking. + +## Design Inputs + +Collect: + +| Input | Notes | +|-------|-------| +| Cluster OCID and region | Use the user-provided cluster; do not assume the current context is correct. | +| Node pool create or update | New node pools are safer when changing network model. | +| Node shape | Validate architecture and VNIC limits. | +| Node count and placement | Include availability domains and worker subnet. | +| Kubernetes version and image | Node image version must match the selected node Kubernetes version. | +| Secondary VNIC profile names | Use stable names that describe the network tier. | +| Secondary subnet OCIDs | Each profile can use a different subnet. | +| NSG OCIDs | Apply least-privilege rules per profile. | +| `ipCount` | Size pod IP allocation with headroom. | +| Application Resource name | Use only when pods must select one specific secondary VNIC profile. | + +## Application Resource Model + +When a secondary VNIC profile has an Application Resource name, OKE exposes a Kubernetes extended resource on the nodes: + +```text +oke-application-resource.oci.oraclecloud.com/ +``` + +Pods that use this model must: + +- Request exactly one Application Resource type. +- Request and limit one unit of that resource. +- Include the toleration for the node taint: + +```yaml +tolerations: + - key: oci.oraclecloud.com/application-resource-only + operator: Exists + effect: NoSchedule +``` + +Example container resource request: + +```yaml +resources: + requests: + oke-application-resource.oci.oraclecloud.com/frontend: "1" + limits: + oke-application-resource.oci.oraclecloud.com/frontend: "1" +``` + +Use Application Resources when a pod should be pinned to one selected secondary VNIC profile. Do not use this pod-level Application Resource model for pods that need multiple interfaces through Multus. + +## CLI Planning Pattern + +When generating an OCI CLI command, build the secondary VNIC profile list explicitly and show it for review before running anything. + +Abbreviated JSON shape: + +```json +[ + { + "displayName": "frontend-vnic-attachment", + "createVnicDetails": { + "displayName": "frontend-vnic", + "subnetId": "ocid1.subnet...", + "nsgIds": ["ocid1.networksecuritygroup..."], + "assignPublicIp": false, + "skipSourceDestCheck": false, + "ipCount": 32, + "applicationResources": ["frontend"] + } + } +] +``` + +Before execution, confirm: + +- Cluster and region. +- Node pool name. +- Placement subnet. +- Shape, OCPUs, memory, and node count. +- Each secondary subnet and NSG. +- Each Application Resource name. +- `ipCount` per profile. +- Whether this is create or update. + +## Validation + +After the node pool is active, validate in this order: + +1. Confirm nodes joined: + +```bash +kubectl get nodes -o wide +``` + +2. Pick one node from the GVA node pool and inspect capacity: + +```bash +kubectl describe node +``` + +Check for: + +```text +Capacity: + oke-application-resource.oci.oraclecloud.com/: +``` + +3. Confirm the node taint when Application Resources are configured: + +```text +oci.oraclecloud.com/application-resource-only:NoSchedule +``` + +4. Deploy or review a test pod that requests exactly one Application Resource and has the matching toleration. + +5. Inspect pod scheduling events and assigned IP: + +```bash +kubectl describe pod -n +kubectl get pod -n -o wide +``` + +6. If the pod remains Pending, check: + - Resource name spelling and case. + - Available extended resource capacity. + - Required toleration. + - Node selectors or affinity. + - Subnet pod IP capacity. + - Shape VNIC limits. + +## Common Mistakes + +- Trying to use GVA with Flannel networking. +- Combining pod-level Application Resource requests with Multus annotations in the same pod. +- Requesting more than one Application Resource type in a pod. +- Forgetting the `oci.oraclecloud.com/application-resource-only:NoSchedule` toleration. +- Setting `ipCount` without checking subnet capacity and shape VNIC limits. +- Reusing old node-pool JSON payloads without revalidating cluster, region, subnets, NSGs, image, and shape. +- Assuming Application Resource names are case-insensitive. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingclusterusingoke_topic-Using_the_Console_to_create_a_Custom_Cluster_with_Explicitly_Defined_Settings.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/create.html +- https://docs.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/ce/node-pool/update.html diff --git a/oci/oke/multus-multihome.md b/oci/oke/multus-multihome.md new file mode 100644 index 0000000..2d9ec73 --- /dev/null +++ b/oci/oke/multus-multihome.md @@ -0,0 +1,202 @@ +# OKE Multus Multi-Interface Pods + +## Overview + +Use this skill to plan, deploy, or troubleshoot Multus NetworkAttachmentDefinitions (NADs) and multi-interface pods on OCI Kubernetes Engine (OKE), especially on node pools with multiple secondary VNIC profiles. + +Use this when a pod needs more than one network interface, such as `eth0` plus `net1`, or when validating that a Multus attachment uses the intended OKE secondary VNIC path. + +## When to Use + +Use this skill for: + +- Multus installation and health checks on OKE. +- NetworkAttachmentDefinition review. +- Multi-interface pod manifests. +- `k8s.v1.cni.cncf.io/networks` annotations. +- `v1.multus-cni.io/default-network` annotations. +- Pod `network-status` validation. +- Missing `net1` interfaces. +- OKE secondary VNIC profile selection from NAD configuration. + +Use `oci/oke/gva-node-pools.md` instead when a pod only needs to select one secondary VNIC profile through an Application Resource request. + +## Core Model + +OKE supports different pod-networking models: + +| Model | Selection mechanism | Use case | +|-------|---------------------|----------| +| Single pinned secondary VNIC profile | Pod-level Application Resource request | Pod needs exactly one selected secondary VNIC path | +| Multi-interface pod | Multus annotations and NADs | Pod needs multiple interfaces | +| Secondary VNIC profiles without Application Resources | NAD device selection or default CNI behavior | Capacity or interface-based selection without scheduler resource pinning | + +Do not combine pod-level Application Resource requests with Multus pod network annotations in the same pod spec. If a multi-interface pod needs to select a secondary VNIC profile, define that selection in the NAD, such as with `deviceSelector.appResource` when supported. + +## Prerequisites + +Confirm: + +- The cluster uses VCN-native pod networking. +- Worker nodes are in a node pool configured with the required secondary VNIC profiles. +- Multus is installed and healthy. +- Required CNI binaries exist on every target worker node. +- NADs are created in the correct namespace. +- Subnets and NSGs allow the traffic being tested. +- The pod image is available from a reachable registry. + +## Health Checks + +Check Multus: + +```bash +kubectl get pods -n kube-system -l app=multus -o wide +kubectl get daemonset -n kube-system +``` + +Check NADs: + +```bash +kubectl get network-attachment-definitions -A +kubectl get network-attachment-definition -n -o yaml +``` + +Check the target pod: + +```bash +kubectl describe pod -n +kubectl get pod -n -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' +``` + +Check interfaces inside the pod: + +```bash +kubectl exec -n -- ip addr +kubectl exec -n -- ip route +``` + +## NAD Review + +Review each NAD for: + +- Correct namespace. +- Correct CNI type for the intended path. +- Correct subnet or IPAM configuration. +- Correct parent interface or `deviceSelector`. +- Valid JSON inside the NAD `config`. +- Security model and expected traffic path. + +A default-network NAD can be used to make `eth0` selection explicit: + +```yaml +metadata: + annotations: + v1.multus-cni.io/default-network: kube-system/oci-vcn-native-network + k8s.v1.cni.cncf.io/networks: default/secondary-network +``` + +The additional network annotation attaches extra interfaces such as `net1`. + +## Validation Flow + +1. Confirm the pod is scheduled onto the intended node pool: + +```bash +kubectl get pod -n -o wide +``` + +2. Confirm Multus network status exists: + +```bash +kubectl describe pod -n +``` + +Look for the `k8s.v1.cni.cncf.io/network-status` annotation. + +3. Confirm expected interfaces: + +```bash +kubectl exec -n -- ip -brief addr +``` + +Expected result: + +```text +lo UNKNOWN +eth0 UP +net1 UP +``` + +4. Confirm IP addresses are from the intended subnets. + +5. Confirm routes match the expected traffic path: + +```bash +kubectl exec -n -- ip route +``` + +6. Test connectivity only to approved targets: + +```bash +kubectl exec -n -- ping -c 3 +``` + +## Common Failure Patterns + +### Pod Has Only `eth0` + +Likely causes: + +- NAD name or namespace does not match the pod annotation. +- Multus is not running or not injected as expected. +- The additional NAD failed validation. +- The pod annotation is malformed. + +### Pod Sandbox Creation Fails + +Likely causes: + +- CNI binary referenced by the NAD is missing on the node. +- NAD JSON is invalid. +- OCI IPAM cannot allocate an address. +- Selected secondary VNIC profile has no available pod IP capacity. +- Node is missing the expected parent interface. + +### Pod Schedules But Traffic Fails + +Likely causes: + +- NSG or security-list rules block traffic. +- Route table does not support the path. +- The pod IP is from a different subnet than expected. +- The wrong interface is being used. +- Return traffic is blocked. + +### DPDK or SR-IOV Workload Fails + +First separate the layers: + +- Kubernetes device-plugin resources only prove resource allocation. +- Multus `network-status` proves interface attachment. +- NAD configuration defines what each attached network means. +- Driver, RDMA, hugepages, and DPDK application configuration are separate checks. + +Do not assume the application failure is a network attachment failure until `network-status`, pod interfaces, and device visibility have been checked. + +## Common Mistakes + +- Combining Multus annotations with pod-level Application Resource requests. +- Creating NADs in `default` while the workload runs in another namespace. +- Referencing a short NAD name when a fully qualified `/` reference is needed. +- Assuming `net1` exists without checking `network-status`. +- Testing connectivity before confirming interface IPs and routes. +- Replacing node CNI binaries broadly when only one missing plugin binary needs investigation. +- Treating SR-IOV or DPDK resource allocation as proof that Multus attached the expected network. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://github.com/k8snetworkplumbingwg/multus-cni +- https://github.com/k8snetworkplumbingwg/network-attachment-definition-client diff --git a/oci/oke/troubleshooting.md b/oci/oke/troubleshooting.md new file mode 100644 index 0000000..8428029 --- /dev/null +++ b/oci/oke/troubleshooting.md @@ -0,0 +1,164 @@ +# OKE Troubleshooting + +## Overview + +Use this skill to troubleshoot OCI Kubernetes Engine (OKE) workload and cluster symptoms. Start with read-only evidence collection, map Kubernetes objects to OCI resources where possible, then rank likely causes before recommending fixes. + +Use this for symptoms such as Pending pods, unhealthy add-ons, CNI failures, DNS issues, LoadBalancer or Ingress problems, OCIR image pulls, Workload Identity errors, private endpoint access failures, storage issues, and node-pool scaling failures. + +## Safety Rules + +- Run read-only commands first. +- Ask before changing OCI or Kubernetes resources. +- Do not restart, scale, drain, delete, patch, apply, or update resources without explicit user approval. +- Keep the investigation scoped to the user-provided cluster, namespace, and workload. +- If `kubectl` or OCI CLI access is unavailable, continue with partial evidence and clearly mark the gap. + +## Intake + +Collect or infer: + +| Field | Examples | +|-------|----------| +| Symptom | `pods stuck Pending`, `service has no IP`, `ImagePullBackOff` | +| Cluster | OKE cluster name, OCID, or kubeconfig context | +| Region | `us-ashburn-1` | +| Namespace | `default`, `prod`, `payments` | +| Workload | Deployment, Pod, Service, Ingress, PVC, or ServiceAccount | +| Time window | `15m`, `1h`, `24h` | +| Impact | production, lower environment, single workload, cluster-wide | + +Confirm the active kubeconfig context before collecting evidence: + +```bash +kubectl config current-context +kubectl config get-contexts +``` + +If the context does not match the target cluster, stop and ask the user to switch context or provide the correct kubeconfig. + +## Symptom Routing + +| Symptom | Primary evidence | +|---------|------------------| +| Pods Pending or Unschedulable | Pod events, node capacity, taints/tolerations, node selectors, autoscaler logs, OCI limits | +| ImagePullBackOff or ErrImagePull | Pod events, image path, imagePullSecrets, service account, OCIR repository and IAM policy | +| CrashLoopBackOff or OOMKilled | Pod logs, previous logs, resource requests/limits, node pressure | +| CNI or pod sandbox failures | Pod events, kube-system CNI pods, OCI VCN-native CNI logs, subnet IP capacity, Multus NADs | +| Service has no LoadBalancer IP | Service events, OCI load balancer state, subnet and NSG rules, quota | +| Ingress failure | Ingress events, OCI Native Ingress controller logs, listener/certificate/backend health | +| DNS timeout or NXDOMAIN | CoreDNS health, Service and EndpointSlice state, pod lookup tests | +| PVC Pending or attach failure | PVC/PV events, CSI controller logs, volume AD, node AD, block volume limits | +| Workload cannot call OCI API | ServiceAccount, Workload Identity policy, SDK provider, namespace and cluster identity | +| Private API endpoint unreachable | Kubeconfig, endpoint subnet, NSGs, route tables, bastion or operator host path | + +## Evidence Collection Pattern + +Start broad, then narrow. + +1. Confirm client tools: + +```bash +kubectl version --client +oci --version +``` + +2. Capture cluster and namespace state: + +```bash +kubectl get nodes -o wide +kubectl get pods -A +kubectl get events -A --sort-by=.lastTimestamp +kubectl get deploy,rs,pod,svc,ingress,pvc -n -o wide +``` + +3. Inspect the failing object: + +```bash +kubectl describe pod -n +kubectl logs -n --previous +kubectl describe service -n +kubectl describe ingress -n +kubectl describe pvc -n +``` + +4. Check OKE system components: + +```bash +kubectl get pods -n kube-system -o wide +kubectl get daemonset,deployment -n kube-system +``` + +5. Use OCI CLI for matching OCI-side state when the cluster OCID, compartment OCID, and region are known: + +```bash +oci ce cluster get --cluster-id --region +oci ce node-pool list --compartment-id --cluster-id --region +``` + +Only run broader OCI inventory commands when they are needed to explain the Kubernetes symptom. + +## Kubernetes to OCI Correlation + +When enough selectors are known, map Kubernetes objects to OCI resources: + +| Kubernetes object | OCI resource to check | +|-------------------|-----------------------| +| Node | Compute instance, primary VNIC, subnet, NSGs, node pool | +| Service type `LoadBalancer` | OCI Load Balancer, backend set, listeners, subnets, NSGs | +| Ingress | OCI Load Balancer resources managed by the ingress controller | +| PVC/PV | Block Volume or File Storage resource and attachment state | +| Pod with VCN-native networking | Pod subnet IP capacity and OCI CNI allocation path | +| Pod using Workload Identity | Cluster, namespace, service account, IAM policy conditions | + +Use this correlation to avoid treating every symptom as only a Kubernetes problem. + +## Diagnosis Report + +Present findings in this order: + +1. Confirmed context: + - Cluster + - Region + - Namespace + - Workload + - Time window + +2. Top findings: + - Evidence + - Why it matters + - Confidence + +3. Likely causes: + - Rank from most likely to least likely. + - Link each cause to the evidence that supports it. + +4. Recommended actions: + - Separate read-only validation from mutating remediation. + - Ask before any remediation command. + +5. Residual risks: + - Missing permissions + - Missing logs + - Time-window gaps + - OCI visibility gaps + +## Common Mistakes + +- Troubleshooting the current kubeconfig context before confirming it is the target OKE cluster. +- Treating a Pending pod as only a Kubernetes scheduler issue without checking node-pool limits, OCI quota, and shape availability. +- Debugging LoadBalancer Services without checking the OCI load balancer, backend health, subnets, NSGs, and service events together. +- Assuming DNS is broken before checking Service selectors and EndpointSlices. +- Rotating image pull secrets before confirming the image path, region key, OCIR namespace, and repository permissions. +- Mixing Workload Identity with dynamic group assumptions. OKE Workload Identity uses workload identity policy conditions, not dynamic groups. +- Applying fixes before preserving the events and logs needed to explain the incident. + +## Sources + +- https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengdownloadkubeconfigfile.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengconfiguringclusteraddons.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengAttaching_Multiple_VNICs.htm From f79c8b5a11963553507e2af1f4337bf53f2306d5 Mon Sep 17 00:00:00 2001 From: Chiping Hwang Date: Thu, 28 May 2026 10:41:47 -0700 Subject: [PATCH 2/6] Add OKE operational tools and workflows Signed-off-by: Chiping Hwang --- README.md | 8 +- oci/SKILL.md | 26 +- oci/oke/agents/oke-evidence-collector.md | 74 ++ oci/oke/agents/oke-hypothesis-analyst.md | 82 ++ oci/oke/agents/oke-lb-log-collector.md | 143 +++ oci/oke/cluster-design.md | 10 + .../outputs/oke-cluster-generator/main.tf | 42 + .../oke-gva-deployer/node-pool-command.sh | 12 + .../gva-multihome-pods.yaml | 97 ++ .../oke-troubleshooter/final-report.md | 25 + .../transcripts/oke-cluster-generator.md | 53 ++ .../examples/transcripts/oke-gva-deployer.md | 38 + .../transcripts/oke-multihome-deployer.md | 39 + .../transcripts/oke-troubleshooter.md | 41 + oci/oke/gva-node-pools.md | 16 + oci/oke/multus-multihome.md | 10 + oci/oke/scripts/gva-cli-resolve.sh | 126 +++ oci/oke/scripts/gva-discover.sh | 488 ++++++++++ oci/oke/scripts/gva-menu.sh | 802 ++++++++++++++++ oci/oke/scripts/node-doctor-run.sh | 226 +++++ oci/oke/scripts/oke-addon-health.sh | 120 +++ oci/oke/scripts/oke-autoscaler-check.sh | 148 +++ oci/oke/scripts/oke-discover.sh | 312 ++++++ oci/oke/scripts/oke-dns-check.sh | 143 +++ oci/oke/scripts/oke-incident-timeline.sh | 104 ++ oci/oke/scripts/oke-ingress-check.sh | 84 ++ oci/oke/scripts/oke-object-correlator.sh | 808 ++++++++++++++++ oci/oke/scripts/oke-ocir-image-pull-check.sh | 93 ++ oci/oke/scripts/oke-pod-network-check.sh | 149 +++ oci/oke/scripts/oke-private-endpoint-check.sh | 111 +++ .../scripts/oke-workload-identity-check.sh | 91 ++ oci/oke/scripts/preflight-check.sh | 150 +++ oci/oke/scripts/validate-cidr.sh | 68 ++ oci/oke/shared/oci-resource-map.md | 121 +++ oci/oke/skills/AGENTS.md | 33 + oci/oke/skills/oke-cluster-generator/SKILL.md | 809 ++++++++++++++++ .../oke-cluster-generator/implementation.md | 85 ++ .../output-templates/schema.md | 211 +++++ .../output-templates/terraform.md | 132 +++ .../skills/oke-cluster-generator/reference.md | 100 ++ oci/oke/skills/oke-gva-deployer/SKILL.md | 276 ++++++ oci/oke/skills/oke-gva-deployer/USAGE.md | 87 ++ .../skills/oke-gva-deployer/implementation.md | 58 ++ .../skills/oke-gva-deployer/references/gva.md | 85 ++ .../validation-report-template.md | 50 + .../skills/oke-multihome-deployer/SKILL.md | 130 +++ .../oke-multihome-deployer/agents/openai.yaml | 4 + .../references/oke-dpdk-mlx5-notes.md | 102 ++ .../references/oke-multihome-notes.md | 94 ++ .../scripts/discover-oke-multihome.py | 358 +++++++ .../scripts/generate-multihome-manifest.py | 188 ++++ .../validation-report-template.md | 43 + oci/oke/skills/oke-troubleshooter/SKILL.md | 386 ++++++++ oci/oke/skills/oke-troubleshooter/USAGE.md | 186 ++++ .../oke-troubleshooter/evidence-collectors.md | 346 +++++++ .../final-report-template.md | 45 + .../oke-troubleshooter/implementation.md | 55 ++ .../oke-troubleshooter/symptom-triage.md | 35 + oci/oke/tests/golden/gva-multihome-pods.yaml | 97 ++ oci/oke/tests/golden/gva-node-pool-command.sh | 12 + oci/oke/tests/golden/oke-cluster-main.tf | 42 + oci/oke/tests/scripts-smoke.sh | 892 ++++++++++++++++++ oci/oke/troubleshooting.md | 33 +- 63 files changed, 9821 insertions(+), 13 deletions(-) create mode 100644 oci/oke/agents/oke-evidence-collector.md create mode 100644 oci/oke/agents/oke-hypothesis-analyst.md create mode 100644 oci/oke/agents/oke-lb-log-collector.md create mode 100644 oci/oke/examples/outputs/oke-cluster-generator/main.tf create mode 100644 oci/oke/examples/outputs/oke-gva-deployer/node-pool-command.sh create mode 100644 oci/oke/examples/outputs/oke-multihome-deployer/gva-multihome-pods.yaml create mode 100644 oci/oke/examples/outputs/oke-troubleshooter/final-report.md create mode 100644 oci/oke/examples/transcripts/oke-cluster-generator.md create mode 100644 oci/oke/examples/transcripts/oke-gva-deployer.md create mode 100644 oci/oke/examples/transcripts/oke-multihome-deployer.md create mode 100644 oci/oke/examples/transcripts/oke-troubleshooter.md create mode 100755 oci/oke/scripts/gva-cli-resolve.sh create mode 100755 oci/oke/scripts/gva-discover.sh create mode 100755 oci/oke/scripts/gva-menu.sh create mode 100755 oci/oke/scripts/node-doctor-run.sh create mode 100755 oci/oke/scripts/oke-addon-health.sh create mode 100755 oci/oke/scripts/oke-autoscaler-check.sh create mode 100755 oci/oke/scripts/oke-discover.sh create mode 100755 oci/oke/scripts/oke-dns-check.sh create mode 100755 oci/oke/scripts/oke-incident-timeline.sh create mode 100755 oci/oke/scripts/oke-ingress-check.sh create mode 100755 oci/oke/scripts/oke-object-correlator.sh create mode 100755 oci/oke/scripts/oke-ocir-image-pull-check.sh create mode 100755 oci/oke/scripts/oke-pod-network-check.sh create mode 100755 oci/oke/scripts/oke-private-endpoint-check.sh create mode 100755 oci/oke/scripts/oke-workload-identity-check.sh create mode 100755 oci/oke/scripts/preflight-check.sh create mode 100755 oci/oke/scripts/validate-cidr.sh create mode 100644 oci/oke/shared/oci-resource-map.md create mode 100644 oci/oke/skills/AGENTS.md create mode 100644 oci/oke/skills/oke-cluster-generator/SKILL.md create mode 100644 oci/oke/skills/oke-cluster-generator/implementation.md create mode 100644 oci/oke/skills/oke-cluster-generator/output-templates/schema.md create mode 100644 oci/oke/skills/oke-cluster-generator/output-templates/terraform.md create mode 100644 oci/oke/skills/oke-cluster-generator/reference.md create mode 100644 oci/oke/skills/oke-gva-deployer/SKILL.md create mode 100644 oci/oke/skills/oke-gva-deployer/USAGE.md create mode 100644 oci/oke/skills/oke-gva-deployer/implementation.md create mode 100644 oci/oke/skills/oke-gva-deployer/references/gva.md create mode 100644 oci/oke/skills/oke-gva-deployer/validation-report-template.md create mode 100644 oci/oke/skills/oke-multihome-deployer/SKILL.md create mode 100644 oci/oke/skills/oke-multihome-deployer/agents/openai.yaml create mode 100644 oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md create mode 100644 oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md create mode 100755 oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py create mode 100755 oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py create mode 100644 oci/oke/skills/oke-multihome-deployer/validation-report-template.md create mode 100644 oci/oke/skills/oke-troubleshooter/SKILL.md create mode 100644 oci/oke/skills/oke-troubleshooter/USAGE.md create mode 100644 oci/oke/skills/oke-troubleshooter/evidence-collectors.md create mode 100644 oci/oke/skills/oke-troubleshooter/final-report-template.md create mode 100644 oci/oke/skills/oke-troubleshooter/implementation.md create mode 100644 oci/oke/skills/oke-troubleshooter/symptom-triage.md create mode 100644 oci/oke/tests/golden/gva-multihome-pods.yaml create mode 100644 oci/oke/tests/golden/gva-node-pool-command.sh create mode 100644 oci/oke/tests/golden/oke-cluster-main.tf create mode 100755 oci/oke/tests/scripts-smoke.sh diff --git a/README.md b/README.md index 3db95ce..cd84ec5 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,13 @@ npx skills add oracle/skills/graal ├── cluster-design.md ├── troubleshooting.md ├── gva-node-pools.md - └── multus-multihome.md + ├── multus-multihome.md + ├── skills/ + ├── scripts/ + ├── agents/ + ├── shared/ + ├── examples/ + └── tests/ ``` Each domain has its own `SKILL.md` and any supporting index files it needs. diff --git a/oci/SKILL.md b/oci/SKILL.md index c2cc6c7..29f869f 100644 --- a/oci/SKILL.md +++ b/oci/SKILL.md @@ -23,17 +23,23 @@ oci/ |-- cluster-design.md |-- troubleshooting.md |-- gva-node-pools.md - `-- multus-multihome.md + |-- multus-multihome.md + |-- skills/ + |-- scripts/ + |-- agents/ + |-- shared/ + |-- examples/ + `-- tests/ ``` ## Category Routing | Topic | File | |-------|------| -| Design or scaffold an OKE cluster, Terraform stack, or OCI Resource Manager stack | `oci/oke/cluster-design.md` | -| Troubleshoot OKE workloads, pods, services, DNS, add-ons, ingress, load balancers, image pulls, storage, Workload Identity, or cluster access | `oci/oke/troubleshooting.md` | -| Configure OKE managed node pools with Generic VNIC Attachment secondary VNIC profiles and Application Resources | `oci/oke/gva-node-pools.md` | -| Deploy or validate Multus NetworkAttachmentDefinitions and multi-interface pods on OKE | `oci/oke/multus-multihome.md` | +| Design or scaffold an OKE cluster, Terraform stack, or OCI Resource Manager stack | Start with `oci/oke/cluster-design.md`, then load `oci/oke/skills/oke-cluster-generator/SKILL.md` | +| Troubleshoot OKE workloads, pods, services, DNS, add-ons, ingress, load balancers, image pulls, storage, Workload Identity, or cluster access | Start with `oci/oke/troubleshooting.md`, then load `oci/oke/skills/oke-troubleshooter/SKILL.md` | +| Configure OKE managed node pools with Generic VNIC Attachment secondary VNIC profiles and Application Resources | Start with `oci/oke/gva-node-pools.md`, then load `oci/oke/skills/oke-gva-deployer/SKILL.md` | +| Deploy or validate Multus NetworkAttachmentDefinitions and multi-interface pods on OKE | Start with `oci/oke/multus-multihome.md`, then load `oci/oke/skills/oke-multihome-deployer/SKILL.md` | ## Key Starting Points @@ -42,6 +48,16 @@ oci/ - `oci/oke/gva-node-pools.md` - `oci/oke/multus-multihome.md` +## Operational Tools + +The OKE operational skills include deterministic helper tools under `oci/oke/scripts/` and skill-specific helper scripts under `oci/oke/skills/*/scripts/`. + +- Read-only discovery and evidence tools may be used to collect context. +- Generate-only tools may produce manifests, commands, Terraform snippets, or reports. +- Any tool or command that creates, updates, deletes, patches, restarts, scales, drains, debugs, assigns IPs, applies manifests, or otherwise mutates OCI or Kubernetes resources requires explicit user approval first. +- `oci/oke/scripts/gva-menu.sh` is allowed to create an OKE node pool for the GVA workflow only after the user approves execution and completes its final `CREATE` confirmation. +- `oci/oke/scripts/node-doctor-run.sh` requires approval before execution because it creates a temporary debug pod and may delete that pod during cleanup. + ## Common Multi-Step Flows | Task | Recommended Sequence | diff --git a/oci/oke/agents/oke-evidence-collector.md b/oci/oke/agents/oke-evidence-collector.md new file mode 100644 index 0000000..fd4c247 --- /dev/null +++ b/oci/oke/agents/oke-evidence-collector.md @@ -0,0 +1,74 @@ +--- +name: oke-evidence-collector +description: Collects Kubernetes and OCI evidence for the OKE troubleshooter. +model: claude-3-haiku-20240307 +default-tool: bash +--- + +You run shell commands to gather evidence for specified diagnostic domains. Always sanitize output: +- Redact tokens, passwords, and private keys. +- Trim logs to the most recent and relevant lines (target under 200 lines per command). + +## Input Contract + +You receive JSON with the following structure: +```json +{ + "symptom": "pods pending", + "domains": ["Pod Scheduling", "Node Health"], + "namespace": "ml-team", + "time_window": "1h", + "selectors": {"pod": "trainer-0", "deployment": "nginx", "label": "app=nginx"}, + "fallbacks": {"kubectl": true, "oci": false}, + "compartment_ocid": "ocid1.compartment..." +} +``` + +Fields: +- `domains`: list of diagnostic domains to investigate. +- `namespace`: namespace string or empty when cluster-wide. +- `time_window`: preferred lookback (e.g., `15m`, `1h`); use for log queries where possible. +- `selectors`: optional keys (`pod`, `service`, `deployment`, `pvc`, `node`, `label`) to scope commands. +- `fallbacks`: booleans indicating unavailable CLIs. Skip commands that require a missing CLI and mark the fallback. +- `compartment_ocid`: optional; when present, include in OCI commands. + +## Command Guidelines +- Batch commands by domain. Print each command before execution prefixed with `>>>`. +- Prefer `-o json` or `--query` flags for parsable output. +- Respect namespace: include `-n ` when provided. +- Do **not** prompt for confirmation; commands must run non-interactively. +- When a command fails, capture a concise error summary and continue. + +## Output Format +Return compact JSON on stdout: +```json +[ + { + "domain": "Pod Scheduling", + "findings": [ + "Pod trainer-0 Pending: 0/3 nodes available: Insufficient nvidia.com/gpu" + ], + "raw_snippets": [ + "Warning FailedScheduling ... 0/3 nodes available: 3 Insufficient nvidia.com/gpu" + ], + "anomalies": [ + "Node pool np-gpu has max size 3 and no headroom" + ], + "fallback_used": false + } +] +``` + +Keep each `raw_snippets` entry under 500 characters. If command output is huge, store only the most relevant fragment. Mark `fallback_used` as `true` when you had to skip or downgrade evidence due to missing tooling. + +## Error Handling +- On unexpected failures, exit with code `2` and print a JSON error to stderr: + ```json + {"error_code":"EVIDENCE_COLLECTOR_FAILURE","message":"...","remediation":"...","docs_url":""} + ``` +- For anticipated issues (missing CLI, permission denied), still exit `0` and include the problem in `anomalies`. + +## Completion Checklist +- Domains processed sequentially. +- Output JSON well-formed and parseable. +- Sensitive values redacted. diff --git a/oci/oke/agents/oke-hypothesis-analyst.md b/oci/oke/agents/oke-hypothesis-analyst.md new file mode 100644 index 0000000..7c40f7a --- /dev/null +++ b/oci/oke/agents/oke-hypothesis-analyst.md @@ -0,0 +1,82 @@ +--- +name: oke-hypothesis-analyst +description: Scores troubleshooting hypotheses for OKE incidents using collected evidence. +model: claude-3-5-sonnet-20241022 +--- + +You receive evidence from the `/oke-troubleshooter` skill and must produce a ranked list of hypotheses with remediation guidance. + +## Input Contract +JSON payload: +```json +{ + "symptom": "pods stuck Pending in ml namespace", + "domains": ["Pod Scheduling", "Node Health"], + "evidence": [ + { + "domain": "Pod Scheduling", + "findings": ["Pod trainer-0 Pending: 0/3 nodes available"], + "raw_snippets": ["Warning FailedScheduling ... Insufficient nvidia.com/gpu"], + "anomalies": ["Node pool np-gpu has max size 3"], + "fallback_used": false + } + ], + "fallbacks": {"kubectl": false, "oci": false} +} +``` + +## Analysis Requirements +- Synthesize cross-domain patterns; explicitly cite the most relevant `raw_snippets` entries using short quotes. +- Produce 1–3 hypotheses ordered by confidence (score 0–10). +- Each hypothesis must include: + - `title`: concise statement of the root cause. + - `score`: integer 0–10 (10 = conclusive, 5 = plausible, ≤3 = weak signal). + - `evidence`: bullet list referencing snippets (e.g., `"Warning FailedScheduling: Insufficient nvidia.com/gpu"`). + - `remediation`: actionable commands or steps (kubectl/oci as needed). + - `prevention`: long-term recommendation (autoscaling, alerts, policy adjustments). +- When evidence is insufficient, add a hypothesis with low confidence explaining what data is missing and suggest additional evidence requests. +- If fallbacks limited analysis (e.g., OCI CLI unavailable), call this out in the report header and downgrade scores accordingly. + +## Output Format +Return JSON adhering to: +```json +{ + "summary": "High confidence GPU quota exhaustion causing Pending pods.", + "hypotheses": [ + { + "title": "GPU node pool exhausted", + "score": 9, + "evidence": [ + "FailedScheduling: Insufficient nvidia.com/gpu on all nodes", + "Node pool np-gpu max size reached (3 nodes)" + ], + "remediation": [ + "oci ce node-pool update --node-pool-id --size 5", + "kubectl cordon if draining required before scale" + ], + "prevention": [ + "Enable autoscaler with GPU headroom", + "Set OCI budget alarms for GPU OCPUs" + ] + } + ], + "warnings": [ + "OCI CLI unavailable: network diagnostics skipped" + ] +} +``` + +Ensure `hypotheses` contains at least one entry even when all scores are low. + +## Tone & Style +- Be direct and operational. No fluff. +- Reference CLI commands precisely; include `--region` or `--compartment-id` flags when needed. +- Avoid repeating identical evidence across multiple hypotheses; if the same data supports multiple causes, explain the distinction. + +## Error Handling +- If the payload is malformed, exit with code `2` and emit JSON error on stderr: + ```json + {"error_code":"HYPOTHESIS_ANALYST_INPUT","message":"...","remediation":"Provide valid evidence payload.","docs_url":""} + ``` + +Complete the analysis in English unless explicitly asked otherwise. diff --git a/oci/oke/agents/oke-lb-log-collector.md b/oci/oke/agents/oke-lb-log-collector.md new file mode 100644 index 0000000..f0579dd --- /dev/null +++ b/oci/oke/agents/oke-lb-log-collector.md @@ -0,0 +1,143 @@ +--- +name: oke-lb-log-collector +description: Resolves OCI load balancer identity and collects LB logging status and issue signals for OKE troubleshooting. +model: claude-3-haiku-20240307 +default-tool: bash +--- + +You gather load balancer logging evidence for OKE incidents. Use read-only commands by default. + +## Input Contract +JSON payload: +```json +{ + "namespace": "default", + "service": "sample-web-app-svc", + "region": "us-sanjose-1", + "compartment_ocid": "ocid1.compartment...", + "time_window": "15m", + "enable_logging_mode": "report_only" +} +``` + +Fields: +- `namespace` and `service`: Kubernetes Service identity. +- `region`: OCI region for API calls. +- `compartment_ocid`: compartment to search for LB/NLB resources. +- `time_window`: lookback (default `15m`) for log queries. +- `enable_logging_mode`: one of `report_only`, `print_command_only`, `run_now`. + +## Procedure +1. Resolve service external IP: + - `kubectl get svc -n -o jsonpath='{.status.loadBalancer.ingress[0].ip}'` +2. Resolve LB OCID from IP: + - `oci lb load-balancer list --compartment-id --region --all --output json` + - Match `ip-addresses[].ip-address == ` +3. If not found, try NLB: + - `oci nlb network-load-balancer list --compartment-id --region --all --output json` +4. For classic LB, perform dual logging checks: + - LB config check: + - `oci lb load-balancer get --load-balancer-id --region --query 'data."access-log"' --output json` + - Logging service object check: + - `oci logging log-group list --compartment-id --region --all --output json` + - For each returned log-group OCID: + - `oci logging log list --log-group-id --all --query "data[?configuration.source.resource=='' && configuration.source.service=='loadbalancer'].[\"display-name\",id,\"is-enabled\",configuration]" --output json` +5. Determine logging status from both checks: + - `enabled`: LB config says enabled OR one or more matching log objects are `is-enabled=true` + - `disabled`: explicit LB config disabled and no matching enabled log objects + - `unknown`: OCI APIs timed out/failed and status cannot be confirmed +6. If access logging is enabled and log identifiers are present, query recent logs: + - `oci logging search search-logs --region --search-query "search \"/\" | where data.loadBalancerId = '' | sort by datetime desc" --time-start --time-end ` +7. Extract issue signals when logs are present: + - 5xx responses + - backend/upstream connection failures or resets + - timeouts + - high latency indicators (p95/p99-style fields when present) +8. If logging is disabled or unknown, act by `enable_logging_mode`: + - `report_only`: include recommendation only. + - `print_command_only`: include exact enable command. + - `run_now`: require `log_group_id` + `log_id` in payload; then run `oci lb load-balancer update ... --access-log ...`. + +## Command Rules +- Print every command prefixed with `>>>`. +- Do not mutate resources unless `enable_logging_mode == "run_now"`. +- Use retry/backoff for OCI read calls (`oci lb ...`, `oci nlb ...`, `oci logging ...`): + - max attempts: `5` + - base delay: `2s` + - backoff: exponential (`2, 4, 8, 16, 32`) + - add small jitter (0-1s) between retries + - cap total retry wait at `~70s` +- Retry only transient failures: + - connection timeout, temporary DNS/network errors, HTTP 429/5xx + - do **not** retry auth/permission failures (401/403) or malformed requests +- On final failure after retries, continue and mark fallback. +- Redact secrets and tokens. + +### Retry Helper Pattern +When executing OCI commands, use this shell pattern: +```bash +run_oci_retry() { + local attempt max=5 + for attempt in 1 2 3 4 5; do + out="$("$@" 2>&1)" && { printf "%s" "$out"; return 0; } + rc=$? + if printf "%s" "$out" | grep -Eqi '401|403|NotAuthorized|InvalidParameter|UsageError'; then + printf "%s" "$out" >&2 + return "$rc" + fi + if [[ "$attempt" -eq "$max" ]]; then + printf "%s" "$out" >&2 + return "$rc" + fi + sleep_sec=$((2 ** attempt)) + jitter=$((RANDOM % 2)) + sleep $((sleep_sec + jitter)) + done +} +``` +Include retry metadata in anomalies on fallback, e.g.: +- `"OCI LB list timed out after 5 attempts (~65s total wait)"` +- `"LB config check succeeded but logging log list timed out; reporting partial status"` + +## Output Format +Return JSON on stdout: +```json +{ + "service": "sample-web-app-svc", + "namespace": "default", + "external_ip": "192.0.2.10", + "lb_type": "lb", + "lb_ocid": "ocid1.loadbalancer...", + "logging_status": "enabled", + "logging_status_source": [ + "lb_access_log_config", + "oci_logging_log_list" + ], + "access_log": { + "isEnabled": true, + "logGroupId": "ocid1.loggroup...", + "logId": "ocid1.log..." + }, + "log_findings": [ + "5xx responses observed for /checkout", + "backend timeout signatures detected" + ], + "anomalies": [], + "fallback_used": false, + "enable_logging_command": "oci lb load-balancer update ...", + "executed_enablement": false +} +``` + +If unresolved, set: +- `logging_status`: `disabled` or `unknown` +- `logging_status_source`: include whichever checks ran successfully +- `fallback_used`: `true` +- include rerun guidance in `anomalies`. + +## Error Handling +- Malformed input: exit `2` and emit JSON error to stderr: + ```json + {"error_code":"LB_LOG_COLLECTOR_INPUT","message":"...","remediation":"Provide valid payload.","docs_url":""} + ``` +- Expected environment/API issues should not hard-fail the whole run. Return JSON with `fallback_used=true`. diff --git a/oci/oke/cluster-design.md b/oci/oke/cluster-design.md index d46eeef..5cf2a8f 100644 --- a/oci/oke/cluster-design.md +++ b/oci/oke/cluster-design.md @@ -6,6 +6,16 @@ Use this skill to design an OCI Kubernetes Engine (OKE) cluster and translate th Start from official OKE requirements, then map the chosen architecture to Terraform variables, Resource Manager schema inputs, and validation checks. +For full operational behavior, load `oci/oke/skills/oke-cluster-generator/SKILL.md` and its supporting files before generating artifacts. That skill preserves the OKE generator workflow, including live preflight discovery, CIDR validation, Terraform output templates, and OCI Resource Manager schema guidance. + +Supporting tools and references: + +- `oci/oke/scripts/preflight-check.sh` +- `oci/oke/scripts/validate-cidr.sh` +- `oci/oke/skills/oke-cluster-generator/reference.md` +- `oci/oke/skills/oke-cluster-generator/output-templates/terraform.md` +- `oci/oke/skills/oke-cluster-generator/output-templates/schema.md` + ## Design Sequence 1. Confirm tenancy context: diff --git a/oci/oke/examples/outputs/oke-cluster-generator/main.tf b/oci/oke/examples/outputs/oke-cluster-generator/main.tf new file mode 100644 index 0000000..2657afe --- /dev/null +++ b/oci/oke/examples/outputs/oke-cluster-generator/main.tf @@ -0,0 +1,42 @@ +module "oke" { + source = "oracle-terraform-modules/oke/oci" + version = "~> 5.0" + + tenancy_id = var.tenancy_ocid + compartment_id = var.compartment_ocid + region = var.region + cluster_name = "demo-private-oke" + kubernetes_version = var.kubernetes_version + cluster_type = "enhanced" + control_plane_is_public = false + + create_vcn = true + vcn_cidrs = ["10.0.0.0/16"] + pods_cidr = "10.244.0.0/16" + services_cidr = "10.96.0.0/16" + cni_type = "npn" + create_bastion = false + create_operator = false + create_nat_gateway = true + create_service_gateway = true + create_internet_gateway = false + + node_pools = { + general = { + shape = "VM.Standard.E5.Flex" + ocpus = 2 + memory = 16 + node_pool_size = 3 + } + } + + block_volume_csi_enabled = true + fss_enabled = false + create_policies = true + workload_identity_enabled = true + + cluster_addons = { + coredns = { remove = false } + kube-proxy = { remove = false } + } +} diff --git a/oci/oke/examples/outputs/oke-gva-deployer/node-pool-command.sh b/oci/oke/examples/outputs/oke-gva-deployer/node-pool-command.sh new file mode 100644 index 0000000..8dcf1a7 --- /dev/null +++ b/oci/oke/examples/outputs/oke-gva-deployer/node-pool-command.sh @@ -0,0 +1,12 @@ +oci ce node-pool create \ + --compartment-id "" \ + --cluster-id "" \ + --name "gva-workers" \ + --kubernetes-version "v1.31.1" \ + --node-shape "VM.Standard.E5.Flex" \ + --node-shape-config '{"ocpus":2,"memoryInGBs":16}' \ + --size 3 \ + --cni-type OCI_VCN_IP_NATIVE \ + --placement-configs '[{"availabilityDomain":"GrCh:US-ASHBURN-AD-1","subnetId":""}]' \ + --node-source-details '{"sourceType":"IMAGE","imageId":""}' \ + --secondary-vnics '[{"createVnicDetails":{"ipCount":32,"applicationResources":["frontend"],"assignPublicIp":false,"displayName":"frontend-vnic","nsgIds":null,"subnetId":"","skipSourceDestCheck":false},"displayName":"frontend-vnic"}]' diff --git a/oci/oke/examples/outputs/oke-multihome-deployer/gva-multihome-pods.yaml b/oci/oke/examples/outputs/oke-multihome-deployer/gva-multihome-pods.yaml new file mode 100644 index 0000000..9dd067c --- /dev/null +++ b/oci/oke/examples/outputs/oke-multihome-deployer/gva-multihome-pods.yaml @@ -0,0 +1,97 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: gva-multihome-test +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: gva-default-network + namespace: kube-system +spec: + config: | + { + "name": "gva-default-network", + "cniVersion": "0.3.1", + "plugins": [ + { + "cniVersion": "0.3.1", + "type": "oci-ipvlan", + "mode": "l2", + "ipam": { + "type": "oci-ipam", + "deviceSelector": { + "interfaceName": "enp1s0" + } + } + }, + { + "cniVersion": "0.3.1", + "type": "oci-ptp", + "containerInterface": "ptp-veth0", + "mtu": 9000 + } + ] + } +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: gva-secondary-network + namespace: gva-multihome-test +spec: + config: | + { + "cniVersion": "0.3.1", + "plugins": [ + { + "type": "ipvlan", + "mode": "l2", + "master": "enp2s0", + "ipam": { + "type": "oci-ipam", + "deviceSelector": { + "interfaceName": "enp2s0" + } + } + } + ] + } +--- +apiVersion: v1 +kind: Pod +metadata: + name: gva-multihome-a + namespace: gva-multihome-test + labels: + app: gva-multihome + annotations: + v1.multus-cni.io/default-network: kube-system/gva-default-network + k8s.v1.cni.cncf.io/networks: gva-multihome-test/gva-secondary-network +spec: + nodeName: "node-a" + containers: + - name: netshoot + image: "docker.io/nicolaka/netshoot:v0.13" + command: + - sleep + - "3600" +--- +apiVersion: v1 +kind: Pod +metadata: + name: gva-multihome-b + namespace: gva-multihome-test + labels: + app: gva-multihome + annotations: + v1.multus-cni.io/default-network: kube-system/gva-default-network + k8s.v1.cni.cncf.io/networks: gva-multihome-test/gva-secondary-network +spec: + nodeName: "node-b" + containers: + - name: netshoot + image: "docker.io/nicolaka/netshoot:v0.13" + command: + - sleep + - "3600" diff --git a/oci/oke/examples/outputs/oke-troubleshooter/final-report.md b/oci/oke/examples/outputs/oke-troubleshooter/final-report.md new file mode 100644 index 0000000..5030b66 --- /dev/null +++ b/oci/oke/examples/outputs/oke-troubleshooter/final-report.md @@ -0,0 +1,25 @@ +# OKE Troubleshooting Report + +## Summary + +Pods in namespace `prod` are Pending because the current node pool cannot fit the +requested CPU and the cluster autoscaler cannot grow the node pool further. + +## Top Hypotheses + +| Rank | Hypothesis | Confidence | Score | Evidence | +| ---: | --- | --- | ---: | --- | +| 1 | Node pool max size reached | High | 9 | `FailedScheduling` reports insufficient CPU; autoscaler reports max node group size reached | +| 2 | OCI quota or capacity limit | Medium | 5 | Scale-out did not proceed; no direct quota error was collected | + +## Recommended Actions + +```bash +kubectl -n prod describe pod +oci ce node-pool update --node-pool-id --size +``` + +## Evidence Gaps + +- OCI quota and capacity checks were not available in this sample. +- Re-run after increasing node-pool size to confirm pods schedule. diff --git a/oci/oke/examples/transcripts/oke-cluster-generator.md b/oci/oke/examples/transcripts/oke-cluster-generator.md new file mode 100644 index 0000000..977982f --- /dev/null +++ b/oci/oke/examples/transcripts/oke-cluster-generator.md @@ -0,0 +1,53 @@ +# OKE Cluster Generator Transcript + +This is a shortened successful run that shows the shape of the interaction and the +artifacts produced. Values are placeholders. + +## Prompt + +```text +Use the OKE cluster generator fast-path us-ashburn-1 demo-private-oke +``` + +## Transcript + +**User:** Generate an OKE Terraform stack quickly for a private worker cluster in +`us-ashburn-1`. + +**Agent:** I will use fast path defaults for a production-friendly starter stack: +Enhanced cluster, private API endpoint, VCN-native pod networking, one Flex node pool, +Block Volume CSI enabled, Oracle-managed encryption, managed CoreDNS and kube-proxy, +and no public worker nodes. I still need your tenancy OCID and compartment OCID. + +**User:** Use `` and ``. + +**Agent:** Summary before generation: + +| Area | Choice | +| --- | --- | +| Region | `us-ashburn-1` | +| Cluster | `demo-private-oke` | +| Cluster type | Enhanced | +| API endpoint | Private | +| CNI | VCN-native pod networking | +| VCN | Create new VCN, `10.0.0.0/16` | +| Pod CIDR | `10.244.0.0/16` | +| Service CIDR | `10.96.0.0/16` | +| Node pool | `general`, 3 nodes, `VM.Standard.E5.Flex`, 2 OCPUs, 16 GB | +| Storage | Block Volume CSI | +| Add-ons | CoreDNS, kube-proxy | + +**User:** Generate it. + +**Agent:** Generated Terraform and ORM assets: + +- `provider.tf` +- `main.tf` +- `variables.tf` +- `outputs.tf` +- `terraform.tfvars.example` +- `schema.yaml` + +## What This Produces + +See [sample Terraform output](../outputs/oke-cluster-generator/main.tf). diff --git a/oci/oke/examples/transcripts/oke-gva-deployer.md b/oci/oke/examples/transcripts/oke-gva-deployer.md new file mode 100644 index 0000000..e2c6ec1 --- /dev/null +++ b/oci/oke/examples/transcripts/oke-gva-deployer.md @@ -0,0 +1,38 @@ +# OKE GVA Deployer Transcript + +This is a shortened successful run for creating a GVA-enabled node pool. + +## Prompt + +```text +Use the OKE GVA deployer +``` + +## Transcript + +**Agent:** Which OKE cluster should I configure? + +**User:** `demo-oke`, region `us-ashburn-1`. + +**Agent:** I found the cluster OCID in kubeconfig and discovered VCN, subnet, and NSG +options. I will ask for one mutable node-pool value at a time. + +**User choices:** + +| Field | Value | +| --- | --- | +| Node pool name | `gva-workers` | +| Shape | `VM.Standard.E5.Flex` | +| OCPUs / memory | `2` / `16` | +| Node count | `3` | +| Primary subnet | `` | +| Application Resource | `frontend` | +| Secondary subnet | `` | +| `ipCount` | `32` | + +**Agent:** Generated a create command and a validation workload manifest. The command +is printed for review; it is not run unless you explicitly choose to run it. + +## What This Produces + +See [sample GVA command output](../outputs/oke-gva-deployer/node-pool-command.sh). diff --git a/oci/oke/examples/transcripts/oke-multihome-deployer.md b/oci/oke/examples/transcripts/oke-multihome-deployer.md new file mode 100644 index 0000000..b83e5aa --- /dev/null +++ b/oci/oke/examples/transcripts/oke-multihome-deployer.md @@ -0,0 +1,39 @@ +# OKE Multihome Deployer Transcript + +This is a shortened successful run for validating Multus pods on a GVA node pool. + +## Prompt + +```text +Use the OKE multihome deployer +``` + +## Transcript + +**User:** Deploy Multus multi-home test pods on my GVA node pool and validate `net1`. + +**Agent:** I will discover the OKE cluster, node pool, placement subnets, secondary +VNIC subnets, and ready node names before generating YAML. + +**Agent:** Discovery found a GVA-enabled node pool with secondary VNIC interfaces +`enp1s0` and `enp2s0`. I will generate two pinned netshoot pods. + +**Agent:** Generated manifest: + +- Namespace `gva-multihome-test` +- Default NAD `kube-system/gva-default-network` +- Secondary NAD `gva-multihome-test/gva-secondary-network` +- Test pods `gva-multihome-a` and `gva-multihome-b` + +**Agent:** Validation passed: + +| Check | Result | +| --- | --- | +| Multus daemonset | Running | +| Pod network-status | `eth0` and `net1` present | +| Pod interfaces | `eth0`, `net1` | +| `net1` ping both directions | Passed | + +## What This Produces + +See [sample Multus manifest](../outputs/oke-multihome-deployer/gva-multihome-pods.yaml). diff --git a/oci/oke/examples/transcripts/oke-troubleshooter.md b/oci/oke/examples/transcripts/oke-troubleshooter.md new file mode 100644 index 0000000..dc27380 --- /dev/null +++ b/oci/oke/examples/transcripts/oke-troubleshooter.md @@ -0,0 +1,41 @@ +# OKE Troubleshooter Transcript + +This is a shortened successful troubleshooting run with redacted identifiers. + +## Prompt + +```text +Use the OKE troubleshooter for pods stuck Pending in prod namespace +``` + +## Transcript + +**User:** Pods are stuck Pending in the `prod` namespace. + +**Agent:** I mapped the symptom to Pod scheduling, Node health, Cluster Autoscaler, +and OCI limits. I will collect read-only evidence with `kubectl` and OCI CLI. + +**Agent:** Evidence collected: + +- `kubectl -n prod get pods --field-selector=status.phase=Pending -o wide` +- `kubectl -n prod get events --field-selector reason=FailedScheduling` +- `kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200` +- `oci ce node-pool list --compartment-id --cluster-id ` + +**Agent:** Findings: + +| Hypothesis | Score | Evidence | +| --- | ---: | --- | +| Node pool max size reached | 9 | FailedScheduling reports insufficient CPU; autoscaler logs show `max node group size reached` | +| OCI compute capacity or quota limit | 5 | No direct quota error found, but scale-out was blocked | + +**Agent:** Next steps: + +```bash +kubectl -n prod describe pod +oci ce node-pool update --node-pool-id --size +``` + +## What This Produces + +See [sample troubleshooting report](../outputs/oke-troubleshooter/final-report.md). diff --git a/oci/oke/gva-node-pools.md b/oci/oke/gva-node-pools.md index 438512f..05909fb 100644 --- a/oci/oke/gva-node-pools.md +++ b/oci/oke/gva-node-pools.md @@ -6,6 +6,16 @@ Use this skill to plan, create, or review OCI Kubernetes Engine (OKE) managed no GVA lets a node pool expose multiple secondary VNIC profiles, each with its own subnet, NSGs, display name, and pod IP allocation. Optional Application Resources let pods request one specific secondary VNIC profile through Kubernetes extended resources. +For full operational behavior, load `oci/oke/skills/oke-gva-deployer/SKILL.md` and its supporting files before generating or running commands. That skill preserves the guided GVA workflow, including discovery, menu-driven command generation, validation templates, and approval-gated node-pool creation. + +Supporting tools and references: + +- `oci/oke/scripts/gva-discover.sh` +- `oci/oke/scripts/gva-menu.sh` +- `oci/oke/scripts/gva-cli-resolve.sh` +- `oci/oke/skills/oke-gva-deployer/references/gva.md` +- `oci/oke/skills/oke-gva-deployer/validation-report-template.md` + ## When to Use Use this skill when the task involves: @@ -82,6 +92,12 @@ resources: Use Application Resources when a pod should be pinned to one selected secondary VNIC profile. Do not use this pod-level Application Resource model for pods that need multiple interfaces through Multus. +## Approval-Gated Creation + +Creating an OKE node pool is allowed in this skill because it is part of the GVA workflow. The agent must still ask for explicit approval before running a mutating command. + +`oci/oke/scripts/gva-menu.sh` supports this approval model by generating the command first, offering a print-only path, and requiring the user to type `CREATE` before it runs `oci ce node-pool create`. + ## CLI Planning Pattern When generating an OCI CLI command, build the secondary VNIC profile list explicitly and show it for review before running anything. diff --git a/oci/oke/multus-multihome.md b/oci/oke/multus-multihome.md index 2d9ec73..ee89c9c 100644 --- a/oci/oke/multus-multihome.md +++ b/oci/oke/multus-multihome.md @@ -6,6 +6,16 @@ Use this skill to plan, deploy, or troubleshoot Multus NetworkAttachmentDefiniti Use this when a pod needs more than one network interface, such as `eth0` plus `net1`, or when validating that a Multus attachment uses the intended OKE secondary VNIC path. +For full operational behavior, load `oci/oke/skills/oke-multihome-deployer/SKILL.md` and its supporting files before generating manifests or validating pods. That skill preserves the Multus multihome workflow, including OKE/GVA discovery, manifest generation, validation reports, and DPDK/SR-IOV notes. + +Supporting tools and references: + +- `oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py` +- `oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py` +- `oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md` +- `oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md` +- `oci/oke/skills/oke-multihome-deployer/validation-report-template.md` + ## When to Use Use this skill for: diff --git a/oci/oke/scripts/gva-cli-resolve.sh b/oci/oke/scripts/gva-cli-resolve.sh new file mode 100755 index 0000000..a144cf6 --- /dev/null +++ b/oci/oke/scripts/gva-cli-resolve.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +WHEEL_NAME="oci_cli-3.65.2+preview.1.1355-py2.py3-none-any.whl" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +mode="json" +while [[ $# -gt 0 ]]; do + case "$1" in + --json) + mode="json" + shift + ;; + --print-home) + mode="home" + shift + ;; + --print-activate) + mode="activate" + shift + ;; + --print-wheel) + mode="wheel" + shift + ;; + -h|--help) + cat <<'EOF' +usage: gva-cli-resolve.sh [--json|--print-home|--print-activate|--print-wheel] + +Search order: +1. $OKE_GVA_CLI_HOME +2. /gva-cli +EOF + exit 0 + ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." \ + "Run with --help to view usage." + ;; + esac +done + +declare -a candidates=() +if [[ -n "${OKE_GVA_CLI_HOME:-}" ]]; then + candidates+=("$OKE_GVA_CLI_HOME") +fi +candidates+=( + "$REPO_ROOT/gva-cli" +) + +resolved_home="" +resolved_activate="" +resolved_wheel="" + +for candidate in "${candidates[@]}"; do + [[ -d "$candidate" ]] || continue + activate_path="$candidate/bin/activate" + wheel_path="$candidate/$WHEEL_NAME" + resolved_home="$candidate" + if [[ -f "$activate_path" ]]; then + resolved_activate="$activate_path" + fi + if [[ -f "$wheel_path" ]]; then + resolved_wheel="$wheel_path" + fi + break +done + +if [[ -z "$resolved_home" ]]; then + emit_error 1 "GVA_CLI_HOME_NOT_FOUND" \ + "Could not locate the preview GVA CLI workspace." \ + "Set OKE_GVA_CLI_HOME or place gva-cli under the OCI OKE skill root." \ + "https://github.com/oracle/skills/tree/main/oci/oke" +fi + +case "$mode" in + home) + printf '%s\n' "$resolved_home" + ;; + activate) + if [[ -z "$resolved_activate" ]]; then + emit_error 1 "GVA_CLI_ACTIVATE_NOT_FOUND" \ + "Resolved GVA CLI home does not contain bin/activate." \ + "Recreate the virtual environment under the resolved GVA CLI home." + fi + printf '%s\n' "$resolved_activate" + ;; + wheel) + if [[ -z "$resolved_wheel" ]]; then + emit_error 1 "GVA_CLI_WHEEL_NOT_FOUND" \ + "Resolved GVA CLI home does not contain the preview OCI CLI wheel." \ + "Place the preview wheel in the resolved GVA CLI home or update the skill instructions." + fi + printf '%s\n' "$resolved_wheel" + ;; + json) + python3 - "$resolved_home" "$resolved_activate" "$resolved_wheel" <<'PY' +import json +import sys + +home = sys.argv[1] +activate = sys.argv[2] +wheel = sys.argv[3] + +print(json.dumps({ + "home": home, + "activate": activate, + "wheel": wheel, + "has_activate": bool(activate), + "has_wheel": bool(wheel), +}, indent=2)) +PY + ;; +esac diff --git a/oci/oke/scripts/gva-discover.sh b/oci/oke/scripts/gva-discover.sh new file mode 100755 index 0000000..97c386c --- /dev/null +++ b/oci/oke/scripts/gva-discover.sh @@ -0,0 +1,488 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Discover GVA-relevant data for an OKE cluster. +# Usage: +# ./scripts/gva-discover.sh --cluster [--region ] [--compartment-id ] [--profile ] [--timeout ] [--kubeconfig ] + +cluster_ref="" +region_arg="" +compartment_arg="" +profile_arg="" +timeout_arg="" +kubeconfig_arg="" +cache_ttl_arg="0" +cache_dir_arg="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." \ + "Run with --help to view usage." + fi +} + +read_lines_into_array() { + local array_name="$1" + local line="" + eval "$array_name=()" + while IFS= read -r line; do + eval "$array_name+=(\"\$line\")" + done +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --cluster) + require_value "$1" "${2:-}" + cluster_ref="$2"; shift 2 ;; + --region) + require_value "$1" "${2:-}" + region_arg="$2"; shift 2 ;; + --compartment-id) + require_value "$1" "${2:-}" + compartment_arg="$2"; shift 2 ;; + --profile) + require_value "$1" "${2:-}" + profile_arg="$2"; shift 2 ;; + --timeout) + require_value "$1" "${2:-}" + timeout_arg="$2"; shift 2 ;; + --kubeconfig) + require_value "$1" "${2:-}" + kubeconfig_arg="$2"; shift 2 ;; + --cache-ttl) + require_value "$1" "${2:-}" + cache_ttl_arg="$2"; shift 2 ;; + --cache-dir) + require_value "$1" "${2:-}" + cache_dir_arg="$2"; shift 2 ;; + -h|--help) + echo "usage: $0 --cluster [--region ] [--compartment-id ] [--profile ] [--timeout ] [--kubeconfig ] [--cache-ttl ] [--cache-dir ]" >&2 + exit 0 ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." \ + "Run with --help to view usage." ;; + esac +done + +if [[ -z "$cluster_ref" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --cluster." \ + "Provide --cluster ." +fi + +if ! command -v oci >/dev/null 2>&1; then + emit_error 2 "OCI_CLI_NOT_INSTALLED" "OCI CLI not found on PATH." \ + "Install OCI CLI and retry." \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm" +fi + +# Read tenancy and default region from OCI config if present +config_file="$HOME/.oci/config" +config_region="" +if [[ -f "$config_file" ]]; then + config_region=$(awk -F= '/^region=/{print $2; exit}' "$config_file" | tr -d ' ') +fi + +region="${region_arg:-$config_region}" +if [[ -z "$region" ]]; then + emit_error 2 "REGION_NOT_PROVIDED" "Region not provided and not found in OCI config." \ + "Provide --region or set region in ~/.oci/config." +fi + +if ! [[ "$cache_ttl_arg" =~ ^[0-9]+$ ]]; then + emit_error 2 "INVALID_ARGUMENT" "--cache-ttl must be a non-negative integer." \ + "Use --cache-ttl ." +fi + +cache_dir="${cache_dir_arg:-${TMPDIR:-/tmp}/gva-discover-cache}" +cache_enabled="no" +cache_file="" +if [[ "$cache_ttl_arg" -gt 0 ]]; then + mkdir -p "$cache_dir" + cache_key=$(python3 - "$cluster_ref" "$region" "$compartment_arg" "$profile_arg" <<'PY' +import hashlib +import sys +raw = "|".join(sys.argv[1:]) +print(hashlib.sha256(raw.encode("utf-8")).hexdigest()) +PY + ) + cache_file="${cache_dir}/${cache_key}.json" + cache_enabled="yes" + if [[ -f "$cache_file" ]]; then + if python3 - "$cache_file" "$cache_ttl_arg" <<'PY' +import os +import sys +import time +path = sys.argv[1] +ttl = int(sys.argv[2]) +age = time.time() - os.path.getmtime(path) +sys.exit(0 if age <= ttl else 1) +PY + then + cat "$cache_file" + exit 0 + fi + fi +fi + +# Helper to run oci with region/profile/timeout +have_timeout="no" +if command -v timeout >/dev/null 2>&1; then + have_timeout="yes" +fi + +use_py_timeout="no" +declare -a timeout_prefix=() +if [[ -n "$timeout_arg" ]]; then + if [[ "$have_timeout" == "yes" ]]; then + timeout_prefix=(timeout "$timeout_arg") + else + use_py_timeout="yes" + fi +fi + +declare -a profile_args=() +if [[ -n "$profile_arg" ]]; then + profile_args=(--profile "$profile_arg") +elif [[ -n "${OCI_CLI_PROFILE:-}" ]]; then + profile_args=(--profile "$OCI_CLI_PROFILE") +fi + +oci_json() { + local out err rc + local -a cmd + err="$(mktemp)" + cmd=(oci --region "$region") + if [[ ${#profile_args[@]} -gt 0 ]]; then + cmd+=("${profile_args[@]}") + fi + cmd+=("$@") + if [[ ${#timeout_prefix[@]} -gt 0 ]]; then + cmd=("${timeout_prefix[@]}" "${cmd[@]}") + fi + if [[ "$use_py_timeout" == "yes" ]]; then + set +e + out="$(python3 - "$timeout_arg" "$err" "${cmd[@]}" <<'PY' +import subprocess, sys +timeout = float(sys.argv[1]) +err_path = sys.argv[2] +cmd = sys.argv[3:] +try: + p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout, text=True) + if p.stderr: + with open(err_path, "w") as f: + f.write(p.stderr) + sys.stdout.write(p.stdout or "") + sys.exit(p.returncode) +except subprocess.TimeoutExpired: + with open(err_path, "w") as f: + f.write("Command timed out after %ss\\n" % timeout) + sys.exit(124) +PY +)" + rc=$? + set -e + else + set +e + out="$("${cmd[@]}" 2>"$err")" + rc=$? + set -e + fi + if [[ $rc -ne 0 ]]; then + echo "error: oci $* failed (exit $rc)" >&2 + sed -e 's/^/oci stderr: /' "$err" >&2 + rm -f "$err" + return "$rc" + fi + rm -f "$err" + printf "%s" "$out" +} + +json_get_field() { + local field="$1" + python3 -c ' +import json +import sys + +field = sys.argv[1] +try: + data = json.load(sys.stdin) +except Exception: + data = {} + +if isinstance(data, dict): + value = data.get(field, "") + if value is None: + value = "" + print(value) +else: + print("") +' "$field" +} + +# If cluster ref is OCID, use it directly. Otherwise require compartment to search by name. +cluster_ocid="" +cluster_name="" +cluster_k8s="" +compartment_ocid="" +cluster_vcn_id="" + +is_ocid="no" +if [[ "$cluster_ref" == ocid1.* ]]; then + is_ocid="yes" +fi + +if [[ "$is_ocid" == "yes" ]]; then + cluster_ocid="$cluster_ref" + cluster_json=$(oci_json ce cluster get --cluster-id "$cluster_ocid" --query 'data.{name:"name",k8s:"kubernetes-version",compartment:"compartment-id",vcn:"vcn-id"}' --output json || true) + if [[ -z "$cluster_json" || "$cluster_json" == "{}" ]]; then + echo "warning: failed to fetch cluster details for $cluster_ocid (check auth/region/profile)" >&2 + else + cluster_name=$(printf '%s' "$cluster_json" | json_get_field name) + cluster_k8s=$(printf '%s' "$cluster_json" | json_get_field k8s) + compartment_ocid=$(printf '%s' "$cluster_json" | json_get_field compartment) + cluster_vcn_id=$(printf '%s' "$cluster_json" | json_get_field vcn) + fi +else + # Try to resolve cluster OCID from kubeconfig if no compartment was provided. + if [[ -z "$compartment_arg" ]]; then + kubeconfig_path="${kubeconfig_arg:-$HOME/.kube/config}" + if [[ -f "$kubeconfig_path" ]]; then + kube_cluster_id=$(python3 - "$kubeconfig_path" "$cluster_ref" <<'PY' +import sys +from pathlib import Path +try: + import yaml +except Exception: + print("") + raise SystemExit(0) + +path = Path(sys.argv[1]) +name = sys.argv[2] +try: + data = yaml.safe_load(path.read_text()) +except Exception: + print("") + raise SystemExit(0) + +contexts = data.get("contexts", []) or [] +users = {u.get("name"): u.get("user", {}) for u in (data.get("users", []) or [])} + +def pick_user(contexts, target): + exact = [] + exact_ci = [] + contains = [] + for c in contexts: + cname = c.get("name", "") + cuser = (c.get("context", {}) or {}).get("user") + if not cuser: + continue + if cname == target: + exact.append((len(cname), cname, cuser)) + elif cname.lower() == target.lower(): + exact_ci.append((len(cname), cname, cuser)) + elif target and target in cname: + contains.append((len(cname), cname, cuser)) + for bucket in (exact, exact_ci, contains): + if bucket: + bucket.sort() + return bucket[0][2] + return None + +match_user = pick_user(contexts, name) + +if not match_user: + print("") + raise SystemExit(0) + +exec_cfg = users.get(match_user, {}).get("exec", {}) +args = exec_cfg.get("args", []) if isinstance(exec_cfg, dict) else [] +try: + idx = args.index("--cluster-id") + print(args[idx + 1]) +except Exception: + print("") +PY + ) + if [[ -n "$kube_cluster_id" ]]; then + cluster_ocid="$kube_cluster_id" + is_ocid="yes" + else + # Fallback: iterate all cluster-ids in kubeconfig and match by name via oci ce cluster get + read_lines_into_array kube_cluster_ids < <(python3 - "$kubeconfig_path" <<'PY' +import sys +from pathlib import Path +try: + import yaml +except Exception: + sys.exit(0) +path = Path(sys.argv[1]) +try: + data = yaml.safe_load(path.read_text()) +except Exception: + sys.exit(0) +users = {u.get("name"): u.get("user", {}) for u in (data.get("users", []) or [])} +ids = [] +for u in users.values(): + exec_cfg = u.get("exec", {}) + args = exec_cfg.get("args", []) if isinstance(exec_cfg, dict) else [] + for i, a in enumerate(args): + if a == "--cluster-id" and i + 1 < len(args): + ids.append(args[i + 1]) +print("\\n".join(sorted(set(ids)))) +PY + ) + if [[ ${#kube_cluster_ids[@]} -gt 0 ]]; then + for cid in "${kube_cluster_ids[@]}"; do + match_json=$(oci_json ce cluster get --cluster-id "$cid" --query 'data.{name:"name",k8s:"kubernetes-version",compartment:"compartment-id",vcn:"vcn-id"}' --output json || true) + if [[ -n "$match_json" && "$match_json" != "{}" ]]; then + match_name=$(printf '%s' "$match_json" | json_get_field name) + if [[ "$match_name" == "$cluster_ref" ]]; then + cluster_ocid="$cid" + cluster_name="$match_name" + cluster_k8s=$(printf '%s' "$match_json" | json_get_field k8s) + compartment_ocid=$(printf '%s' "$match_json" | json_get_field compartment) + cluster_vcn_id=$(printf '%s' "$match_json" | json_get_field vcn) + is_ocid="yes" + break + fi + fi + done + fi + fi + fi + fi + + if [[ "$is_ocid" == "yes" ]]; then + cluster_json=$(oci_json ce cluster get --cluster-id "$cluster_ocid" --query 'data.{name:"name",k8s:"kubernetes-version",compartment:"compartment-id",vcn:"vcn-id"}' --output json || true) + if [[ -z "$cluster_json" || "$cluster_json" == "{}" ]]; then + echo "warning: failed to fetch cluster details for $cluster_ocid (check auth/region/profile)" >&2 + else + cluster_name=$(printf '%s' "$cluster_json" | json_get_field name) + cluster_k8s=$(printf '%s' "$cluster_json" | json_get_field k8s) + compartment_ocid=$(printf '%s' "$cluster_json" | json_get_field compartment) + cluster_vcn_id=$(printf '%s' "$cluster_json" | json_get_field vcn) + fi + fi + + # Require a compartment to search by name (avoid tenancy-wide scans). + if [[ -z "$compartment_arg" && -z "$compartment_ocid" && "$is_ocid" != "yes" ]]; then + emit_error 2 "COMPARTMENT_REQUIRED_FOR_CLUSTER_NAME" \ + "Compartment ID is required when using a cluster name." \ + "Provide --compartment-id, use cluster OCID, or use kubeconfig that includes --cluster-id." + fi + + comp_to_search="${compartment_arg:-$compartment_ocid}" + clusters_json=$(oci_json ce cluster list --compartment-id "$comp_to_search" --output json || true) + hit="" + if [[ -n "$clusters_json" ]]; then + hit=$(printf '%s' "$clusters_json" | CLUSTER_REF="$cluster_ref" python3 -c ' +import json +import os +import sys + +target = os.environ.get("CLUSTER_REF", "") +try: + payload = json.loads(sys.stdin.read()) +except Exception: + print("") + raise SystemExit(0) + +items = payload.get("data", []) if isinstance(payload, dict) else [] +for item in items: + if (item or {}).get("name", "") == target: + print(json.dumps(item)) + break +') + fi + if [[ -n "$hit" && "$hit" != "null" ]]; then + cluster_ocid=$(printf '%s' "$hit" | json_get_field id) + cluster_name=$(printf '%s' "$hit" | json_get_field name) + cluster_k8s=$(printf '%s' "$hit" | json_get_field kubernetes-version) + compartment_ocid="$comp_to_search" + fi +fi + +if [[ -z "$compartment_ocid" && -n "$compartment_arg" ]]; then + compartment_ocid="$compartment_arg" +fi + +# Pull subnets and NSGs in the compartment +subnets_json="[]" +nsGs_json="[]" +if [[ -n "$compartment_ocid" ]]; then + subnets_tmp="$(mktemp)" + nsgs_tmp="$(mktemp)" + + if [[ -n "$cluster_vcn_id" ]]; then + ( + oci_json network subnet list --compartment-id "$compartment_ocid" --vcn-id "$cluster_vcn_id" --query 'data[*].{"name":"display-name","id":"id","cidr":"cidr-block","ipv6":"ipv6-cidr-blocks"}' --output json || true + ) >"$subnets_tmp" & + subnets_pid=$! + ( + oci_json network nsg list --compartment-id "$compartment_ocid" --vcn-id "$cluster_vcn_id" --query 'data[*].{"name":"display-name","id":"id"}' --output json || true + ) >"$nsgs_tmp" & + nsgs_pid=$! + else + ( + oci_json network subnet list --compartment-id "$compartment_ocid" --query 'data[*].{"name":"display-name","id":"id","cidr":"cidr-block","ipv6":"ipv6-cidr-blocks"}' --output json || true + ) >"$subnets_tmp" & + subnets_pid=$! + ( + oci_json network nsg list --compartment-id "$compartment_ocid" --query 'data[*].{"name":"display-name","id":"id"}' --output json || true + ) >"$nsgs_tmp" & + nsgs_pid=$! + fi + + wait "$subnets_pid" || true + wait "$nsgs_pid" || true + subnets_json="$(cat "$subnets_tmp")" + nsGs_json="$(cat "$nsgs_tmp")" + rm -f "$subnets_tmp" "$nsgs_tmp" + + if [[ -z "$subnets_json" ]]; then + subnets_json="[]" + echo "warning: failed to list subnets for compartment $compartment_ocid" >&2 + fi + if [[ -z "$nsGs_json" ]]; then + nsGs_json="[]" + echo "warning: failed to list NSGs for compartment $compartment_ocid" >&2 + fi +fi + +# Output consolidated JSON +result_json=$(python3 - < "$cache_file" +fi + +printf "%s\n" "$result_json" diff --git a/oci/oke/scripts/gva-menu.sh b/oci/oke/scripts/gva-menu.sh new file mode 100755 index 0000000..aef22f3 --- /dev/null +++ b/oci/oke/scripts/gva-menu.sh @@ -0,0 +1,802 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +GVA_DISCOVER_SCRIPT="$SCRIPT_DIR/gva-discover.sh" + +say() { printf "%s\n" "$*"; } + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + python3 - "$error_code" "$message" "$remediation" "$docs_url" <<'PY' >&2 +import json +import sys + +print(json.dumps({ + "error_code": sys.argv[1], + "message": sys.argv[2], + "remediation": sys.argv[3], + "docs_url": sys.argv[4], +})) +PY + exit "$exit_code" +} + +ask() { + local prompt="$1" var + read -r -p "$prompt" var + printf "%s" "$var" +} + +ask_required() { + local prompt="$1" + local value="" + while [[ -z "$value" ]]; do + value=$(ask "$prompt") + if [[ -z "$value" ]]; then + say "A value is required." + fi + done + printf "%s" "$value" +} + +read_lines_into_array() { + local array_name="$1" + local line="" + eval "$array_name=()" + while IFS= read -r line; do + eval "$array_name+=(\"\$line\")" + done +} + +select_from_list() { + local prompt="$1" + local allow_manual="$2" + shift 2 + local items=("$@") + local options=("${items[@]}") + local choice="" + if [[ "$allow_manual" == "yes" ]]; then + options+=("Manual entry") + fi + say "$prompt" + select choice in "${options[@]}"; do + if [[ "$allow_manual" == "yes" && "$choice" == "Manual entry" ]]; then + choice="" + break + elif [[ -n "$choice" ]]; then + break + else + say "Choose one of the listed options." + fi + done + printf "%s" "$choice" +} + +normalize_none() { + local v="$1" + local v_lower + v_lower="$(printf "%s" "$v" | tr '[:upper:]' '[:lower:]')" + case "$v_lower" in + ""|"0"|"none"|"no"|"n"|"skip") echo "" ;; + *) echo "$v" ;; + esac +} + +lower() { + printf "%s" "$1" | tr '[:upper:]' '[:lower:]' +} + +oci_node_pool_create_supports_gva() { + oci ce node-pool create --help 2>/dev/null | grep -Eq 'secondary-vnics|cni-type' +} + +verify_gva_cli_before_create() { + if ! command -v oci >/dev/null 2>&1; then + emit_error 1 "OCI_CLI_NOT_FOUND" \ + "OCI CLI is required to create a GVA node pool." \ + "Install OCI CLI or choose Print command only." + fi + + if ! oci_node_pool_create_supports_gva; then + emit_error 1 "GVA_CLI_FLAGS_UNAVAILABLE" \ + "OCI CLI does not expose --secondary-vnics or --cni-type for node-pool create." \ + "Upgrade OCI CLI to a version with GA GVA support, or choose Print command only." + fi +} + +line_field() { + local line="$1" + local field_no="$2" + printf "%s" "$line" | cut -d'|' -f"$field_no" | xargs +} + +ip_count_valid() { + local value="$1" + [[ "$value" =~ ^[0-9]+$ ]] || return 1 + (( value >= 1 && value <= 256 )) +} + +oci_available="yes" +if ! command -v oci >/dev/null 2>&1; then + oci_available="no" +fi + +say "GVA Node Pool Builder (OKE)" +say "This wizard collects fresh values for every new node pool creation." +say "It does not reuse prior drafts or previous mutable node-pool settings." +say "Saved JSON payloads are ignored for new node pool creation." +say "Answer the prompts to generate an OCI CLI command." + +cluster_name=$(ask_required "Cluster name (required, no default): ") +region=$(ask_required "Region for the selected cluster (e.g., us-ashburn-1): ") + +profile_name=$(ask "OCI CLI profile (optional): ") + +# Try to resolve cluster OCID from kubeconfig +kubeconfig_path="$HOME/.kube/config" +cluster_ocid="" +if [[ -f "$kubeconfig_path" ]]; then + cluster_ocid=$(python3 - "$kubeconfig_path" "$cluster_name" <<'PY' +import sys +from pathlib import Path +try: + import yaml +except Exception: + print("") + raise SystemExit(0) +path = Path(sys.argv[1]) +name = sys.argv[2] +try: + data = yaml.safe_load(path.read_text()) +except Exception: + print("") + raise SystemExit(0) + +contexts = data.get("contexts", []) or [] +users = {u.get("name"): u.get("user", {}) for u in (data.get("users", []) or [])} + +def pick_user(contexts, target): + exact = [] + exact_ci = [] + contains = [] + for c in contexts: + cname = c.get("name", "") + cuser = (c.get("context", {}) or {}).get("user") + if not cuser: + continue + if cname == target: + exact.append((len(cname), cname, cuser)) + elif cname.lower() == target.lower(): + exact_ci.append((len(cname), cname, cuser)) + elif target and target in cname: + contains.append((len(cname), cname, cuser)) + for bucket in (exact, exact_ci, contains): + if bucket: + bucket.sort() + return bucket[0][2] + return None + +match_user = pick_user(contexts, name) + +if not match_user: + print("") + raise SystemExit(0) + +exec_cfg = users.get(match_user, {}).get("exec", {}) +args = exec_cfg.get("args", []) if isinstance(exec_cfg, dict) else [] +try: + idx = args.index("--cluster-id") + print(args[idx + 1]) +except Exception: + print("") +PY + ) +fi + +if [[ -z "$cluster_ocid" ]]; then + cluster_ocid=$(ask_required "Cluster OCID (not found in kubeconfig): ") +else + say "Detected cluster OCID from kubeconfig." +fi + +compartment_ocid="" +discovery_json="" +if [[ "$oci_available" == "yes" ]]; then + discover_cmd=("$GVA_DISCOVER_SCRIPT" --cluster "$cluster_ocid" --timeout 10) + if [[ -n "$region" ]]; then + discover_cmd+=(--region "$region") + fi + if [[ -n "$profile_name" ]]; then + discover_cmd+=(--profile "$profile_name") + fi + discovery_json=$("${discover_cmd[@]}" 2>/dev/null || true) +fi + +cluster_k8s="" +subnet_lines=() +secondary_subnet_lines=() +nsg_lines=() +vcn_lines=() + +if [[ -n "$discovery_json" ]]; then + cluster_k8s=$(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +print(d.get('cluster',{}).get('kubernetes_version','')) +PY + ) + + comp_from_discovery=$(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +print(d.get('cluster',{}).get('compartment_id','')) +PY + ) + + region_from_discovery=$(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +print(d.get('cluster',{}).get('region','')) +PY + ) + + if [[ -z "$compartment_ocid" && -n "$comp_from_discovery" ]]; then + compartment_ocid="$comp_from_discovery" + fi + if [[ -z "$region" && -n "$region_from_discovery" ]]; then + region="$region_from_discovery" + fi + + read_lines_into_array subnet_lines < <(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +subnets = d.get('subnets', []) +if isinstance(subnets, dict): + subnets = subnets.get('data', []) +for s in subnets: + name=s.get('name') or '' + sid=s.get('id') or '' + cidr=s.get('cidr') or '' + ipv6=s.get('ipv6') or [] + ipv6_status='has-ipv6' if ipv6 else 'ipv4-only' + if name and sid: + print(f"{name} | {cidr} | {ipv6_status} | {sid}") +PY + ) + + read_lines_into_array nsg_lines < <(python3 - "$discovery_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +nsgs = d.get('nsgs', []) +if isinstance(nsgs, dict): + nsgs = nsgs.get('data', []) +for n in nsgs: + name=n.get('name') or '' + nid=n.get('id') or '' + if name and nid: + print(f"{name} | {nid}") +PY + ) +fi + +if [[ -z "$compartment_ocid" ]]; then + compartment_ocid=$(ask_required "Compartment OCID: ") +fi + +config_file="$HOME/.oci/config" +tenancy_ocid="" +if [[ -f "$config_file" ]]; then + tenancy_ocid=$(awk -F= '/^tenancy=/{print $2; exit}' "$config_file" | tr -d ' ') +fi + +ad_lines=() +if [[ "$oci_available" == "yes" ]]; then + ad_compartment="${tenancy_ocid:-$compartment_ocid}" + ad_json=$(oci iam availability-domain list --compartment-id "$ad_compartment" --region "$region" --query 'data[*].name' --output json 2>/dev/null || true) + if [[ -n "$ad_json" && "$ad_json" != "[]" ]]; then + read_lines_into_array ad_lines < <(python3 - <<'PY' +import json,sys +try: + data=json.loads(sys.stdin.read()) +except Exception: + data=[] +for item in data: + if item: + print(item) +PY + <<<"$ad_json") + fi +fi + +if [[ ${#ad_lines[@]} -gt 0 ]]; then + ad=$(select_from_list "Select Availability Domain:" "no" "${ad_lines[@]}") +else + ad=$(ask_required "Availability Domain (e.g., GrCh:US-ASHBURN-AD-1): ") +fi + +vcn_id="" +if [[ "$oci_available" == "yes" && -n "$compartment_ocid" ]]; then + vcn_json=$(oci network vcn list --compartment-id "$compartment_ocid" --region "$region" --query 'data[*].{name:"display-name",id:id,cidr:"cidr-block"}' --output json 2>/dev/null || true) + if [[ -n "$vcn_json" && "$vcn_json" != "[]" ]]; then + read_lines_into_array vcn_lines < <(python3 - "$vcn_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +for v in d if isinstance(d, list) else []: + name=v.get('name') or '' + vid=v.get('id') or '' + cidr=v.get('cidr') or '' + if name and vid: + print(f"{name} | {cidr} | {vid}") +PY + ) + fi +fi + +if [[ ${#vcn_lines[@]} -gt 0 ]]; then + selection=$(select_from_list "Select VCN:" "yes" "${vcn_lines[@]}") + if [[ -n "$selection" ]]; then + vcn_id=$(printf "%s" "$selection" | cut -d'|' -f3 | xargs) + fi +fi +if [[ -z "$vcn_id" ]]; then + vcn_id=$(ask_required "VCN OCID: ") +fi + +if [[ -n "$vcn_id" ]]; then + subnet_lines=() + secondary_subnet_lines=() + subnet_json=$(oci network subnet list --compartment-id "$compartment_ocid" --vcn-id "$vcn_id" --region "$region" --query 'data[*].{"name":"display-name","id":"id","cidr":"cidr-block","ipv4Cidrs":"ipv4-cidr-blocks","ipv6":"ipv6-cidr-blocks"}' --output json 2>/dev/null || true) + if [[ -n "$subnet_json" && "$subnet_json" != "[]" ]]; then + read_lines_into_array subnet_lines < <(python3 - "$subnet_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +for s in d.get('data', d) if isinstance(d, dict) else d: + name=s.get('name') or s.get('display-name') or '' + sid=s.get('id') or '' + cidr=s.get('cidr') or s.get('cidr-block') or '' + ipv6=s.get('ipv6') or s.get('ipv6-cidr-blocks') or [] + ipv6_status='has-ipv6' if ipv6 else 'ipv4-only' + if name and sid: + print(f"{name} | {cidr} | {ipv6_status} | {sid}") +PY + ) + read_lines_into_array secondary_subnet_lines < <(python3 - "$subnet_json" <<'PY' +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + d={} +for s in d.get('data', d) if isinstance(d, dict) else d: + name=s.get('name') or s.get('display-name') or '' + sid=s.get('id') or '' + cidr=s.get('cidr') or s.get('cidr-block') or '' + ipv6=s.get('ipv6') or s.get('ipv6-cidr-block') + ipv4_cidrs=s.get('ipv4Cidrs') or s.get('ipv4-cidr-blocks') or [] + cidr_count=len(ipv4_cidrs) if isinstance(ipv4_cidrs, list) else 0 + if name and sid and not ipv6 and cidr_count > 1: + print(f"{name} | {cidr} | {sid}") +PY + ) + fi +fi + +primary_subnet="" +if [[ ${#subnet_lines[@]} -gt 0 ]]; then + selection=$(select_from_list "Select primary subnet (node placement):" "yes" "${subnet_lines[@]}") + if [[ -n "$selection" ]]; then + primary_subnet=$(line_field "$selection" 4) + fi +fi +if [[ -z "$primary_subnet" ]]; then + primary_subnet=$(ask_required "Primary subnet OCID (node placement subnet): ") +fi + +node_pool_name=$(ask_required "Node pool name: ") + +k8s_version="" +if [[ -n "$cluster_k8s" ]]; then + k8s_version=$cluster_k8s + say "Using cluster Kubernetes version: $k8s_version" +fi +if [[ -z "$k8s_version" ]]; then + k8s_version=$(ask_required "Kubernetes version (e.g., v1.34.1): ") +fi + +shape=$(ask_required "Node shape (e.g., VM.Standard.E5.Flex): ") + +type_is_flex="no" +case "$shape" in + *.Flex) type_is_flex="yes" ;; + *) type_is_flex="no" ;; + esac + +ocpus="" +mem_gb="" +if [[ "$type_is_flex" == "yes" ]]; then + ocpus=$(ask_required "OCPUs per node: ") + mem_gb=$(ask_required "Memory GB per node: ") +fi + +node_count=$(ask_required "Node count: ") + +# Image selection: list images for k8s version and prompt +image_ocid="" +if [[ "$oci_available" == "yes" && -n "$k8s_version" ]]; then + img_json=$(oci ce node-pool-options get --node-pool-option-id "$cluster_ocid" --region "$region" --query 'data.sources[*].{"image":"image-id","name":"source-name"}' --output json 2>/dev/null || true) + if [[ -n "$img_json" && "$img_json" != "[]" ]]; then + read_lines_into_array img_lines < <(python3 - "$img_json" "$k8s_version" <<'PY' +import json,sys,re +try: + d=json.loads(sys.argv[1]) +except Exception: + d=[] +pattern=re.compile(rf'OKE-{re.escape(sys.argv[2])}') +items=[x for x in d if pattern.search(x.get('name',''))] +for x in items: + print(f"{x.get('name')} | {x.get('image')}") +PY + ) + if [[ ${#img_lines[@]} -gt 0 ]]; then + selection=$(select_from_list "Select OKE image for $k8s_version:" "yes" "${img_lines[@]}") + if [[ -n "$selection" ]]; then + image_ocid=$(printf "%s" "$selection" | cut -d'|' -f2 | xargs) + fi + fi + fi +fi + +if [[ -z "$image_ocid" ]]; then + image_ocid=$(ask_required "Image OCID: ") +fi + +say "" +say "CNI must be OCI_VCN_IP_NATIVE for GVA." +say "1) Yes" +say "2) No" +while true; do + say "Choose 1 or 2:" + IFS= read -r cni_ok + case "$cni_ok" in + 1|Yes|yes|Y|y) break ;; + 2|No|no|N|n) + emit_error 1 "GVA_REQUIRES_VCN_NATIVE_CNI" \ + "GVA is unsupported without OCI_VCN_IP_NATIVE." \ + "Use an OKE cluster with VCN-native pod networking before creating a GVA node pool." + ;; + *) say "Choose one of the listed options." ;; + esac +done + +# Collect VNIC profiles +profiles=() +profile_summaries=() +application_resources=() +while true; do + say "" + say "Add a secondary VNIC profile (GVA tier)" + while true; do + app_res=$(ask " applicationResource (label): ") + if [[ -z "$app_res" ]]; then + say " applicationResource is required." + continue + fi + duplicate="no" + if [[ ${#application_resources[@]} -gt 0 ]]; then + for existing in "${application_resources[@]}"; do + if [[ "$existing" == "$app_res" ]]; then + duplicate="yes" + break + fi + done + fi + if [[ "$duplicate" == "yes" ]]; then + say " applicationResource values must be unique across profiles." + continue + fi + break + done + + subnet_id="" + subnet_summary="" + while [[ -z "$subnet_id" ]]; do + if [[ ${#secondary_subnet_lines[@]} -gt 0 ]]; then + selection=$(select_from_list " Select subnet for this profile (IPv4-only, >1 IPv4 CIDR block):" "yes" "${secondary_subnet_lines[@]}") + if [[ -n "$selection" ]]; then + subnet_id=$(line_field "$selection" 3) + subnet_summary="$(line_field "$selection" 1) ($(line_field "$selection" 2))" + fi + fi + if [[ -z "$subnet_id" ]]; then + subnet_id=$(ask " subnetId OCID (must be IPv4-only and have >1 IPv4 CIDR block): ") + if [[ -n "$subnet_id" ]]; then + say " Manual subnet entry cannot be OCI-validated. Ensure the subnet is IPv4-only and has more than one IPv4 CIDR block." + subnet_summary="$subnet_id" + fi + fi + done + + while true; do + ip_count=$(ask " ipCount (1-256): ") + if ip_count_valid "$ip_count"; then + break + fi + say " ipCount must be an integer between 1 and 256." + done + + nsg_ids="" + if [[ ${#nsg_lines[@]} -gt 0 ]]; then + selection=$(select_from_list " Select NSG (or Manual entry for none/custom):" "yes" "${nsg_lines[@]}") + if [[ -n "$selection" ]]; then + nsg_ids=$(printf "%s" "$selection" | cut -d'|' -f2 | xargs) + fi + nsg_ids=$(normalize_none "$nsg_ids") + else + nsg_ids=$(normalize_none "$(ask " nsgIds (comma-separated OCIDs, optional): ")") + fi + + display_name=$(ask " displayName (optional): ") + + profile_json="$( + APP_RES="$app_res" \ + SUBNET_ID="$subnet_id" \ + IP_COUNT="$ip_count" \ + NSG_IDS="$nsg_ids" \ + DISPLAY_NAME="$display_name" \ + python3 - <<'PY' +import json +import os +import sys + +app_res = os.environ.get("APP_RES", "") +subnet_id = os.environ.get("SUBNET_ID", "") +display_name = os.environ.get("DISPLAY_NAME", "") +nsg_ids_raw = os.environ.get("NSG_IDS", "") +ip_count_raw = os.environ.get("IP_COUNT", "") + +try: + ip_count = int(ip_count_raw) +except ValueError: + print("Invalid ipCount. Expected an integer.", file=sys.stderr) + sys.exit(2) + +nsg_ids = [x.strip() for x in nsg_ids_raw.split(",") if x.strip()] if nsg_ids_raw else None +display_field = display_name if display_name else None + +obj = { + "createVnicDetails": { + "ipCount": ip_count, + "applicationResources": [app_res], + "assignPublicIp": False, + "displayName": display_field, + "nsgIds": nsg_ids, + "subnetId": subnet_id, + "skipSourceDestCheck": False, + }, + "displayName": display_field, +} + +print(json.dumps(obj, separators=(",", ":"))) +PY + )" + + profiles+=("$profile_json") + application_resources+=("$app_res") + profile_summaries+=("$app_res | ipCount=$ip_count | subnet=$subnet_summary") + + say "" + select more in "Add another profile" "Finish"; do + case "$more" in + "Add another profile") break ;; + "Finish") more=""; break 2 ;; + esac + done + done + +secondary_vnics_json="[" +for i in "${!profiles[@]}"; do + secondary_vnics_json+="${profiles[$i]}" + if [[ $i -lt $((${#profiles[@]}-1)) ]]; then + secondary_vnics_json+=","; + fi + done +secondary_vnics_json+="]" + +# Build node shape config +shape_config="{}" +if [[ "$type_is_flex" == "yes" ]]; then + shape_config="{\"ocpus\":$ocpus,\"memoryInGBs\":$mem_gb}" +fi + +optional_args=() +add_optional=$(ask "Add optional node-pool parameters? (y/N): ") +if [[ "$(lower "$add_optional")" == "y" || "$(lower "$add_optional")" == "yes" ]]; then + defined_tags=$(ask " --defined-tags (JSON or file://path, optional): ") + if [[ -n "$defined_tags" ]]; then + optional_args+=("--defined-tags" "$defined_tags") + fi + + freeform_tags=$(ask " --freeform-tags (JSON or file://path, optional): ") + if [[ -n "$freeform_tags" ]]; then + optional_args+=("--freeform-tags" "$freeform_tags") + fi + + initial_labels=$(ask " --initial-node-labels (JSON or file://path, optional): ") + if [[ -n "$initial_labels" ]]; then + optional_args+=("--initial-node-labels" "$initial_labels") + fi + + max_pods=$(ask " --max-pods-per-node (integer, optional): ") + if [[ -n "$max_pods" ]]; then + optional_args+=("--max-pods-per-node" "$max_pods") + fi + + pv_encrypt=$(ask " --is-pv-encryption-in-transit-enabled? (y/N): ") + if [[ "$(lower "$pv_encrypt")" == "y" || "$(lower "$pv_encrypt")" == "yes" ]]; then + optional_args+=("--is-pv-encryption-in-transit-enabled" "true") + fi + + kms_key=$(ask " --kms-key-id (OCID, optional): ") + if [[ -n "$kms_key" ]]; then + optional_args+=("--kms-key-id" "$kms_key") + fi + + boot_size=$(ask " --node-boot-volume-size-in-gbs (integer, optional): ") + if [[ -n "$boot_size" ]]; then + optional_args+=("--node-boot-volume-size-in-gbs" "$boot_size") + fi + + node_metadata=$(ask " --node-metadata (JSON or file://path, optional): ") + if [[ -n "$node_metadata" ]]; then + optional_args+=("--node-metadata" "$node_metadata") + fi + + ssh_key=$(ask " --ssh-public-key (string or file://path, optional): ") + if [[ -n "$ssh_key" ]]; then + optional_args+=("--ssh-public-key" "$ssh_key") + fi +fi + +say "" +say "Summary:" +say "- Cluster: $cluster_name" +say "- Cluster OCID: $cluster_ocid" +say "- Region: $region" +say "- Compartment: $compartment_ocid" +say "- AD: $ad" +say "- Node pool: $node_pool_name" +say "- Shape: $shape" +say "- Node count: $node_count" +say "- K8s version: $k8s_version" +say "- Primary subnet: $primary_subnet" +for p in "${profile_summaries[@]}"; do + say "- VNIC: $p" + done + +confirm_generate=$(ask "Generate a create command with these values? (y/N): ") +confirm_generate=$(lower "$confirm_generate") +if [[ "$confirm_generate" != "y" && "$confirm_generate" != "yes" ]]; then + say "Aborted before command generation." + exit 0 +fi + +say "" +say "Choose next action:" +say "1) Run command now" +say "2) Print command only" +say "3) Exit without output" +choice=$(ask "Select (1/2/3): ") + +if [[ "$choice" == "1" ]]; then + run_now="yes" +elif [[ "$choice" == "2" ]]; then + run_now="no" +elif [[ "$choice" == "3" ]]; then + say "Aborted." + exit 0 +else + say "Unknown choice; printing command only." + run_now="no" +fi + +say "" +say "Generated OCI CLI command:" +cmd=(oci) +if [[ -n "$profile_name" ]]; then + cmd+=(--profile "$profile_name") +fi +cmd+=( + --region "$region" + ce node-pool create + --compartment-id "$compartment_ocid" + --cluster-id "$cluster_ocid" + --name "$node_pool_name" + --kubernetes-version "$k8s_version" + --node-shape "$shape" + --node-shape-config "$shape_config" + --size "$node_count" + --cni-type OCI_VCN_IP_NATIVE + --placement-configs "[{\"availabilityDomain\":\"$ad\",\"subnetId\":\"$primary_subnet\"}]" + --node-source-details "{\"sourceType\":\"IMAGE\",\"imageId\":\"$image_ocid\"}" + --secondary-vnics "$secondary_vnics_json" +) +if [[ ${#optional_args[@]} -gt 0 ]]; then + cmd+=("${optional_args[@]}") +fi +printf '%q ' "${cmd[@]}" +printf '\n' + +if [[ "$run_now" == "yes" ]]; then + final_confirmation=$(ask "Type CREATE to run the node-pool create command now: ") + if [[ "$final_confirmation" != "CREATE" ]]; then + say "Run cancelled. Command was not executed." + exit 0 + fi + say "" + say "Verifying OCI CLI support for GVA node-pool flags..." + verify_gva_cli_before_create + say "Running command..." + "${cmd[@]}" +fi + +say "" +say "Sample test Deployment (replace ResourceName/image):" +cat <<'YAML' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gva-test +spec: + replicas: 1 + selector: + matchLabels: + app: gva-test + template: + metadata: + labels: + app: gva-test + spec: + tolerations: + - key: "oci.oraclecloud.com/application-resource-only" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: app + image: + resources: + requests: + oke-application-resource.oci.oraclecloud.com/ResourceName: "1" + limits: + oke-application-resource.oci.oraclecloud.com/ResourceName: "1" +YAML + +say "" +say "Next steps:" +say "1) Verify resources on a node: kubectl describe node " +say "2) Apply the test Deployment (with your chosen ResourceName)." diff --git a/oci/oke/scripts/node-doctor-run.sh b/oci/oke/scripts/node-doctor-run.sh new file mode 100755 index 0000000..29eb085 --- /dev/null +++ b/oci/oke/scripts/node-doctor-run.sh @@ -0,0 +1,226 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Run OKE node doctor through kubectl debug and emit normalized JSON. +# +# Usage: +# ./scripts/node-doctor-run.sh --node --image [--namespace ] [--cleanup true|false] + +node_name="" +image_name="" +namespace="default" +cleanup="true" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." \ + "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --node) + require_value "$1" "${2:-}" + node_name="$2" + shift 2 + ;; + --image) + require_value "$1" "${2:-}" + image_name="$2" + shift 2 + ;; + --namespace) + require_value "$1" "${2:-}" + namespace="$2" + shift 2 + ;; + --cleanup) + require_value "$1" "${2:-}" + cleanup="$2" + shift 2 + ;; + -h|--help) + cat <<'EOF' +usage: node-doctor-run.sh --node --image [--namespace ] [--cleanup true|false] +EOF + exit 0 + ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." + ;; + esac +done + +if [[ -z "$node_name" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --node." "Provide --node ." +fi + +if [[ -z "$image_name" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --image." "Provide --image ." +fi + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +debug_cmd=(kubectl -n "$namespace" debug "node/${node_name}" --image="${image_name}" -- chroot /host sudo /usr/local/bin/node-doctor.sh --check) + +set +e +raw_out="$("${debug_cmd[@]}" 2>&1)" +cmd_rc=$? +set -e + +pod_name="$(printf "%s\n" "$raw_out" | sed -n 's/^Creating debugging pod \([^ ]*\) .*/\1/p' | head -n 1)" + +# When kubectl debug cannot attach, execution output may only exist in pod logs. +if [[ -n "$pod_name" ]]; then + # Wait for the debug pod to finish so node-doctor output is complete. + for _ in $(seq 1 120); do + set +e + pod_phase="$(kubectl -n "$namespace" get pod "$pod_name" -o jsonpath='{.status.phase}' 2>/dev/null)" + phase_rc=$? + set -e + if [[ $phase_rc -eq 0 && ( "$pod_phase" == "Succeeded" || "$pod_phase" == "Failed" ) ]]; then + break + fi + sleep 1 + done + + set +e + pod_logs="$(kubectl -n "$namespace" logs "$pod_name" --all-containers=true 2>&1)" + logs_rc=$? + set -e + if [[ $logs_rc -eq 0 && -n "$pod_logs" ]]; then + raw_out="${raw_out}"$'\n'"${pod_logs}" + fi +fi + +cleanup_attempted="false" +cleanup_message="" + +cleanup_lc="$(printf "%s" "$cleanup" | tr '[:upper:]' '[:lower:]')" +if [[ "$cleanup_lc" == "true" && -n "$pod_name" ]]; then + cleanup_attempted="true" + set +e + cleanup_out="$(kubectl -n "$namespace" delete pod "$pod_name" --ignore-not-found 2>&1)" + cleanup_rc=$? + set -e + if [[ $cleanup_rc -eq 0 ]]; then + cleanup_message="$cleanup_out" + else + cleanup_message="cleanup failed: $cleanup_out" + fi +fi + +RAW_OUT="$raw_out" python3 - "$node_name" "$image_name" "$namespace" "$cmd_rc" "$pod_name" "$cleanup_attempted" "$cleanup_message" <<'PY' +import json +import os +import re +import sys + +node = sys.argv[1] +image = sys.argv[2] +namespace = sys.argv[3] +cmd_rc = int(sys.argv[4]) +pod_name = sys.argv[5] +cleanup_attempted = sys.argv[6].lower() == "true" +cleanup_message = sys.argv[7] +raw = os.environ.get("RAW_OUT", "") + +# Strip ANSI escapes so snippets remain readable. +raw = re.sub(r"\x1B\[[0-9;]*[A-Za-z]", "", raw) +lines = raw.splitlines() + +pass_count = sum(1 for l in lines if l.startswith("PASS ")) +fail_count = sum(1 for l in lines if l.startswith("FAIL ")) +warn_count = sum(1 for l in lines if l.startswith("WARN ")) +skip_count = sum(1 for l in lines if l.startswith("SKIP ")) + +checks_line = "" +signals_line = "" +for l in lines: + if "checks passed" in l: + checks_line = l.strip() + if "Signal(s) generated" in l: + signals_line = l.strip() + +has_check_lines = any( + l.startswith(("PASS ", "FAIL ", "WARN ", "SKIP ")) + for l in lines +) +executed = ( + "Running node doctor..." in raw + or "NODE DOCTOR REPORT" in raw + or checks_line != "" + or has_check_lines +) + +if not executed: + result = "unknown" + fallback_reason = "node doctor checks did not execute or produced no parsable output" +elif cmd_rc != 0: + result = "unknown" + fallback_reason = f"node doctor command exited with code {cmd_rc}" +elif fail_count > 0: + result = "fail" + fallback_reason = "" +else: + result = "pass" + fallback_reason = "" + +findings = [] +if checks_line: + findings.append(checks_line) +if signals_line: + findings.append(signals_line) + +signal_pat = re.compile(r"^Signal \d+:\s+(.+)$") +for l in lines: + m = signal_pat.match(l.strip()) + if m: + findings.append(f"Signal: {m.group(1)}") + +snippet_lines = lines[-120:] +raw_snippet = "\n".join(snippet_lines) + +out = { + "node_doctor_attempted": True, + "node_doctor_executed": executed, + "node_doctor_node": node, + "node_doctor_image": image, + "node_doctor_namespace": namespace, + "node_doctor_debug_pod": pod_name, + "node_doctor_result": result, + "node_doctor_command_rc": cmd_rc, + "node_doctor_findings": findings, + "node_doctor_counts": { + "pass": pass_count, + "fail": fail_count, + "warn": warn_count, + "skip": skip_count + }, + "node_doctor_raw_snippet": raw_snippet, + "node_doctor_fallback_reason": fallback_reason, + "cleanup_attempted": cleanup_attempted, + "cleanup_message": cleanup_message +} + +print(json.dumps(out, indent=2)) +PY + +# Runtime failures are represented in JSON fields above. Exit 0 so callers can +# always parse structured output and decide fallback behavior. +exit 0 diff --git a/oci/oke/scripts/oke-addon-health.sh b/oci/oke/scripts/oke-addon-health.sh new file mode 100755 index 0000000..67d9e55 --- /dev/null +++ b/oci/oke/scripts/oke-addon-health.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect OKE add-on health signals from kube-system. +# +# Usage: +# bash scripts/oke-addon-health.sh [--namespace kube-system] + +namespace="kube-system" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace) + require_value "$1" "${2:-}" + namespace="$2" + shift 2 + ;; + -h|--help) + echo "usage: oke-addon-health.sh [--namespace kube-system]" + exit 0 + ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." + ;; + esac +done + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1" + shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json +import os +import sys + +record = { + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], +} +with open(sys.argv[1], "a") as f: + f.write(json.dumps(record) + "\n") +PY +} + +run_check "kube-system pods" kubectl -n "$namespace" get pods -o wide +run_check "kube-system deployments" kubectl -n "$namespace" get deploy -o wide +run_check "kube-system daemonsets" kubectl -n "$namespace" get ds -o wide +run_check "CoreDNS deployment" kubectl -n "$namespace" get deploy coredns -o wide +run_check "recent kube-system warnings" kubectl -n "$namespace" get events --field-selector type=Warning --sort-by=.lastTimestamp + +python3 - "$records" "$namespace" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace = sys.argv[2] +anomaly_re = re.compile( + r"(CrashLoopBackOff|ImagePullBackOff|ErrImagePull|OOMKilled|NotReady|Pending|Error|Failed|0/[1-9])", + re.I, +) + +findings = [] +anomalies = [] +snippets = [] +for item in records: + output = item["output"].strip() + if item["rc"] == 0: + findings.append(f"{item['name']} collected") + else: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if output: + snippets.append(f"$ {item['cmd']}\n{output[-800:]}") + for line in output.splitlines(): + if anomaly_re.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") + +print(json.dumps({ + "domain": "OKE Add-ons Health", + "namespace": namespace, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-8:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-autoscaler-check.sh b/oci/oke/scripts/oke-autoscaler-check.sh new file mode 100755 index 0000000..e29a364 --- /dev/null +++ b/oci/oke/scripts/oke-autoscaler-check.sh @@ -0,0 +1,148 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect cluster-autoscaler, Pending pod, and optional OKE node-pool evidence. +# +# Usage: +# bash scripts/oke-autoscaler-check.sh --namespace [--deployment ] [--cluster-id ] [--compartment-id ] [--region ] + +namespace="" +deployment="" +cluster_id="" +compartment_id="" +region="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) + require_value "$1" "${2:-}" + namespace="$2"; shift 2 ;; + --deployment) + require_value "$1" "${2:-}" + deployment="$2"; shift 2 ;; + --cluster-id) + require_value "$1" "${2:-}" + cluster_id="$2"; shift 2 ;; + --compartment-id) + require_value "$1" "${2:-}" + compartment_id="$2"; shift 2 ;; + --region) + require_value "$1" "${2:-}" + region="$2"; shift 2 ;; + -h|--help) + echo "usage: oke-autoscaler-check.sh --namespace [--deployment ] [--cluster-id ] [--compartment-id ] [--region ]" + exit 0 ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done + +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --namespace." "Provide --namespace ." +fi + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1" + shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json +import os +import sys + +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], + }) + "\n") +PY +} + +run_check "pending pods" kubectl -n "$namespace" get pods --field-selector=status.phase=Pending -o wide +run_check "scheduling warning events" kubectl -n "$namespace" get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp +run_check "cluster-autoscaler deployment" kubectl -n kube-system get deploy cluster-autoscaler -o wide +run_check "cluster-autoscaler logs" kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200 + +if [[ -n "$deployment" ]]; then + run_check "target deployment" kubectl -n "$namespace" describe deployment "$deployment" + run_check "target deployment hpa" kubectl -n "$namespace" get hpa --selector "app=$deployment" -o wide +fi + +if [[ -n "$cluster_id" && -n "$compartment_id" ]] && command -v oci >/dev/null 2>&1; then + oci_cmd=(oci) + if [[ -n "$region" ]]; then + oci_cmd+=(--region "$region") + fi + run_check "OKE node pools" "${oci_cmd[@]}" ce node-pool list --compartment-id "$compartment_id" --cluster-id "$cluster_id" --all --output json +fi + +python3 - "$records" "$namespace" "$deployment" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, deployment = sys.argv[2], sys.argv[3] +anomaly_re = re.compile( + r"(FailedScheduling|Insufficient|max node group size reached|scale.?up|NotTriggerScaleUp|quota|limit|no node group|didn.t trigger|unremovable|Pending|Error|Failed)", + re.I, +) + +findings = [] +anomalies = [] +snippets = [] +for item in records: + output = item["output"].strip() + if item["rc"] == 0: + findings.append(f"{item['name']} collected") + else: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if output: + snippets.append(f"$ {item['cmd']}\n{output[-800:]}") + for line in output.splitlines(): + if anomaly_re.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") + +print(json.dumps({ + "domain": "Cluster Autoscaler / Node Pool Scaling", + "namespace": namespace, + "deployment": deployment, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-discover.sh b/oci/oke/scripts/oke-discover.sh new file mode 100755 index 0000000..e3eccfc --- /dev/null +++ b/oci/oke/scripts/oke-discover.sh @@ -0,0 +1,312 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Discover OKE cluster context for troubleshooting. +# Usage: +# ./scripts/oke-discover.sh --cluster [--region ] [--profile ] [--timeout ] [--kubeconfig ] [--deployment ] + +cluster_ref="" +region_arg="" +profile_arg="" +timeout_arg="" +kubeconfig_arg="" +deployment_name="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="${5:-}" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." \ + "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --cluster) + require_value "$1" "${2:-}" + cluster_ref="$2"; shift 2 ;; + --region) + require_value "$1" "${2:-}" + region_arg="$2"; shift 2 ;; + --profile) + require_value "$1" "${2:-}" + profile_arg="$2"; shift 2 ;; + --timeout) + require_value "$1" "${2:-}" + timeout_arg="$2"; shift 2 ;; + --kubeconfig) + require_value "$1" "${2:-}" + kubeconfig_arg="$2"; shift 2 ;; + --deployment) + require_value "$1" "${2:-}" + deployment_name="$2"; shift 2 ;; + -h|--help) + echo "usage: $0 --cluster [--region ] [--profile ] [--timeout ] [--kubeconfig ] [--deployment ]" >&2 + exit 0 ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." \ + "Run with --help to view usage." ;; + esac +done + +if [[ -z "$cluster_ref" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --cluster." \ + "Provide --cluster ." +fi + +if ! command -v oci >/dev/null 2>&1; then + emit_error 2 "OCI_CLI_NOT_INSTALLED" "OCI CLI not found on PATH." \ + "Install OCI CLI and retry." \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm" +fi + +# Read defaults from OCI config if present +config_file="$HOME/.oci/config" +config_region="" +if [[ -f "$config_file" ]]; then + config_region=$(awk -F= '/^region=/{print $2; exit}' "$config_file" | tr -d ' ') +fi + +have_timeout="no" +if command -v timeout >/dev/null 2>&1; then + have_timeout="yes" +fi + +use_py_timeout="no" +declare -a timeout_prefix=() +if [[ -n "$timeout_arg" ]]; then + if [[ "$have_timeout" == "yes" ]]; then + timeout_prefix=(timeout "$timeout_arg") + else + use_py_timeout="yes" + fi +fi + +declare -a profile_args=() +if [[ -n "$profile_arg" ]]; then + profile_args=(--profile "$profile_arg") +elif [[ -n "${OCI_CLI_PROFILE:-}" ]]; then + profile_args=(--profile "$OCI_CLI_PROFILE") +fi + +oci_json() { + local out err rc + local -a cmd + err="$(mktemp)" + cmd=(oci --region "$region") + if [[ ${#profile_args[@]} -gt 0 ]]; then + cmd+=("${profile_args[@]}") + fi + cmd+=("$@") + if [[ ${#timeout_prefix[@]} -gt 0 ]]; then + cmd=("${timeout_prefix[@]}" "${cmd[@]}") + fi + if [[ "$use_py_timeout" == "yes" ]]; then + set +e + out="$(python3 - "$timeout_arg" "$err" "${cmd[@]}" <<'PY' +import subprocess, sys +timeout = float(sys.argv[1]) +err_path = sys.argv[2] +cmd = sys.argv[3:] +try: + p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout, text=True) + if p.stderr: + with open(err_path, "w") as f: + f.write(p.stderr) + sys.stdout.write(p.stdout or "") + sys.exit(p.returncode) +except subprocess.TimeoutExpired: + with open(err_path, "w") as f: + f.write("Command timed out after %ss\n" % timeout) + sys.exit(124) +PY +)" + rc=$? + set -e + else + set +e + out="$("${cmd[@]}" 2>"$err")" + rc=$? + set -e + fi + if [[ $rc -ne 0 ]]; then + echo "error: oci $* failed (exit $rc)" >&2 + sed -e 's/^/oci stderr: /' "$err" >&2 + rm -f "$err" + return "$rc" + fi + rm -f "$err" + printf "%s" "$out" +} + +json_get_field() { + local field="$1" + python3 -c ' +import json +import sys + +field = sys.argv[1] +try: + data = json.load(sys.stdin) +except Exception: + data = {} + +if isinstance(data, dict): + value = data.get(field, "") + if value is None: + value = "" + print(value) +else: + print("") +' "$field" +} + +cluster_ocid="" +cluster_name="" +cluster_k8s="" +compartment_ocid="" + +if [[ "$cluster_ref" == ocid1.* ]]; then + cluster_ocid="$cluster_ref" +else + # Resolve from kubeconfig + kubeconfig_path="${kubeconfig_arg:-$HOME/.kube/config}" + if [[ -f "$kubeconfig_path" ]]; then + cluster_ocid=$(python3 - "$kubeconfig_path" "$cluster_ref" <<'PY' +import sys +from pathlib import Path +try: + import yaml +except Exception: + print("") + raise SystemExit(0) +path = Path(sys.argv[1]) +name = sys.argv[2] +try: + data = yaml.safe_load(path.read_text()) +except Exception: + print("") + raise SystemExit(0) + +contexts = data.get("contexts", []) or [] +users = {u.get("name"): u.get("user", {}) for u in (data.get("users", []) or [])} + +def pick_user(contexts, target): + exact = [] + exact_ci = [] + contains = [] + for c in contexts: + cname = c.get("name", "") + cuser = (c.get("context", {}) or {}).get("user") + if not cuser: + continue + if cname == target: + exact.append((len(cname), cname, cuser)) + elif cname.lower() == target.lower(): + exact_ci.append((len(cname), cname, cuser)) + elif target and target in cname: + contains.append((len(cname), cname, cuser)) + for bucket in (exact, exact_ci, contains): + if bucket: + bucket.sort() + return bucket[0][2] + return None + +match_user = pick_user(contexts, name) + +if not match_user: + print("") + raise SystemExit(0) + +exec_cfg = users.get(match_user, {}).get("exec", {}) +args = exec_cfg.get("args", []) if isinstance(exec_cfg, dict) else [] +try: + idx = args.index("--cluster-id") + print(args[idx + 1]) +except Exception: + print("") +PY + ) + fi +fi + +if [[ -z "$cluster_ocid" ]]; then + emit_error 1 "CLUSTER_OCID_NOT_RESOLVED" \ + "Could not resolve cluster OCID from kubeconfig." \ + "Provide cluster OCID directly with --cluster ." +fi + +region_from_cluster_ocid="$(python3 - "$cluster_ocid" <<'PY' +import re +import sys + +ocid = sys.argv[1].strip() +match = re.match(r"^ocid1\.cluster\.oc1\.([a-z0-9-]+)\..+$", ocid) +print(match.group(1) if match else "") +PY +)" + +region="${region_arg:-${region_from_cluster_ocid:-$config_region}}" +if [[ -z "$region" ]]; then + emit_error 2 "REGION_NOT_PROVIDED" \ + "Region not provided and could not be inferred from cluster OCID or OCI config." \ + "Provide --region or set region in ~/.oci/config." +fi + +# Fetch cluster details if possible +cluster_json=$(oci_json ce cluster get --cluster-id "$cluster_ocid" --query 'data.{name:"name",k8s:"kubernetes-version",compartment:"compartment-id"}' --output json || true) +if [[ -n "$cluster_json" && "$cluster_json" != "{}" ]]; then + cluster_name=$(printf '%s' "$cluster_json" | json_get_field name) + cluster_k8s=$(printf '%s' "$cluster_json" | json_get_field k8s) + compartment_ocid=$(printf '%s' "$cluster_json" | json_get_field compartment) +else + echo "warning: failed to fetch cluster details; returning partial context" >&2 + cluster_name="$cluster_ref" +fi + +# Optional: try to resolve deployment namespace if kubectl is available +deployment_namespace="" +deployment_namespaces="" +if [[ -n "$deployment_name" ]]; then + if command -v kubectl >/dev/null 2>&1; then + deployment_namespaces=$(kubectl get deploy -A --no-headers 2>/dev/null | awk -v d="$deployment_name" '$2==d {print $1}' | paste -sd "," -) + if [[ -n "$deployment_namespaces" ]]; then + if [[ "$deployment_namespaces" == *,* ]]; then + deployment_namespace="" + else + deployment_namespace="$deployment_namespaces" + fi + fi + else + echo "warning: kubectl not found; cannot resolve deployment namespace" >&2 + fi +fi + +python3 - < [--service ] [--pod ] [--lookup ] + +namespace="" +service="" +pod="" +lookup="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) + require_value "$1" "${2:-}" + namespace="$2"; shift 2 ;; + --service) + require_value "$1" "${2:-}" + service="$2"; shift 2 ;; + --pod) + require_value "$1" "${2:-}" + pod="$2"; shift 2 ;; + --lookup) + require_value "$1" "${2:-}" + lookup="$2"; shift 2 ;; + -h|--help) + echo "usage: oke-dns-check.sh --namespace [--service ] [--pod ] [--lookup ]" + exit 0 ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done + +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --namespace." "Provide --namespace ." +fi + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1" + shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json +import os +import sys + +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], + }) + "\n") +PY +} + +run_check "CoreDNS pods" kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide +run_check "CoreDNS deployment" kubectl -n kube-system get deploy coredns -o wide +run_check "CoreDNS config" kubectl -n kube-system get configmap coredns -o yaml +run_check "CoreDNS logs" kubectl -n kube-system logs deployment/coredns --tail=200 + +if [[ -n "$service" ]]; then + run_check "target service" kubectl -n "$namespace" get svc "$service" -o yaml + run_check "target endpoints" kubectl -n "$namespace" get endpoints "$service" -o yaml + run_check "target endpoint slices" kubectl -n "$namespace" get endpointslices -l "kubernetes.io/service-name=$service" -o yaml +fi + +if [[ -n "$pod" && -n "$lookup" ]]; then + run_check "pod DNS lookup" kubectl -n "$namespace" exec "$pod" -- nslookup "$lookup" +fi + +python3 - "$records" "$namespace" "$service" "$pod" "$lookup" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, service, pod, lookup = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +anomaly_re = re.compile( + r"(SERVFAIL|NXDOMAIN|timeout|no such host|connection refused|no endpoints|CrashLoopBackOff|NotReady|Error|Failed)", + re.I, +) + +findings = [] +anomalies = [] +snippets = [] +for item in records: + output = item["output"].strip() + if item["rc"] == 0: + findings.append(f"{item['name']} collected") + else: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if output: + snippets.append(f"$ {item['cmd']}\n{output[-800:]}") + for line in output.splitlines(): + if anomaly_re.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") + +print(json.dumps({ + "domain": "DNS / Service Discovery", + "namespace": namespace, + "service": service, + "pod": pod, + "lookup": lookup, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-incident-timeline.sh b/oci/oke/scripts/oke-incident-timeline.sh new file mode 100755 index 0000000..d2f772c --- /dev/null +++ b/oci/oke/scripts/oke-incident-timeline.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Build a lightweight incident timeline from Kubernetes events and selected objects. +# +# Usage: +# bash scripts/oke-incident-timeline.sh --namespace [--pod ] [--deployment ] [--service ] [--compartment-id ] [--region ] + +namespace="" +pod="" +deployment="" +service="" +compartment_id="" +region="" + +emit_error() { + local exit_code="$1"; local error_code="$2"; local message="$3"; local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --pod) require_value "$1" "${2:-}"; pod="$2"; shift 2 ;; + --deployment) require_value "$1" "${2:-}"; deployment="$2"; shift 2 ;; + --service) require_value "$1" "${2:-}"; service="$2"; shift 2 ;; + --compartment-id) require_value "$1" "${2:-}"; compartment_id="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + -h|--help) echo "usage: oke-incident-timeline.sh --namespace [--pod ] [--deployment ] [--service ] [--compartment-id ] [--region ]"; exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --namespace." "Provide --namespace ." +fi +command -v kubectl >/dev/null 2>&1 || emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." + +tmp_dir="$(mktemp -d)"; trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl"; : > "$records" +run_check() { + local name="$1"; shift + local out rc + set +e; out="$("$@" 2>&1)"; rc=$?; set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({"name": os.environ["NAME"], "cmd": os.environ["CMD"], "rc": int(os.environ["RC"]), "output": os.environ["OUT"][-6000:]}) + "\n") +PY +} + +run_check "namespace events" kubectl -n "$namespace" get events --sort-by=.lastTimestamp +if [[ -n "$pod" ]]; then + run_check "pod describe" kubectl -n "$namespace" describe pod "$pod" +fi +if [[ -n "$deployment" ]]; then + run_check "deployment rollout history" kubectl -n "$namespace" rollout history deployment/"$deployment" + run_check "deployment describe" kubectl -n "$namespace" describe deployment "$deployment" +fi +if [[ -n "$service" ]]; then + run_check "service describe" kubectl -n "$namespace" describe service "$service" +fi +if [[ -n "$compartment_id" && -n "$region" ]] && command -v oci >/dev/null 2>&1; then + run_check "firing alarms" oci --region "$region" monitoring alarm-status list-alarms-status --compartment-id "$compartment_id" --status FIRING --all --output json +fi + +python3 - "$records" "$namespace" "$pod" "$deployment" "$service" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, pod, deployment, service = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +signal = re.compile(r"(Warning|Failed|BackOff|Killing|Pulling|Created|Started|Scaled|Successful|Unhealthy|NotReady|FIRING)", re.I) +timeline = [] +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if signal.search(line): + timeline.append({"source": item["name"], "event": line.strip()[:300]}) + if re.search(r"(Warning|Failed|BackOff|Unhealthy|NotReady|FIRING)", line, re.I): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "Incident Timeline", + "namespace": namespace, + "pod": pod, + "deployment": deployment, + "service": service, + "findings": findings, + "timeline": timeline[-30:], + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-ingress-check.sh b/oci/oke/scripts/oke-ingress-check.sh new file mode 100755 index 0000000..997aff1 --- /dev/null +++ b/oci/oke/scripts/oke-ingress-check.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect OCI Native Ingress and Kubernetes Ingress evidence. +# +# Usage: +# bash scripts/oke-ingress-check.sh --namespace --ingress [--region ] + +namespace="" +ingress="" +region="" + +emit_error() { + local exit_code="$1"; local error_code="$2"; local message="$3"; local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --ingress) require_value "$1" "${2:-}"; ingress="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + -h|--help) echo "usage: oke-ingress-check.sh --namespace --ingress [--region ]"; exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done +if [[ -z "$namespace" || -z "$ingress" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --namespace or --ingress." "Provide both --namespace and --ingress." +fi +command -v kubectl >/dev/null 2>&1 || emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." + +tmp_dir="$(mktemp -d)"; trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl"; : > "$records" +run_check() { + local name="$1"; shift + local out rc + set +e; out="$("$@" 2>&1)"; rc=$?; set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({"name": os.environ["NAME"], "cmd": os.environ["CMD"], "rc": int(os.environ["RC"]), "output": os.environ["OUT"][-4000:]}) + "\n") +PY +} + +run_check "ingress classes" kubectl get ingressclass -o yaml +run_check "target ingress" kubectl -n "$namespace" get ingress "$ingress" -o yaml +run_check "target ingress describe" kubectl -n "$namespace" describe ingress "$ingress" +run_check "ingress controller pods" kubectl -n kube-system get pods -l app.kubernetes.io/name=oci-native-ingress-controller -o wide +run_check "ingress controller logs" kubectl -n kube-system logs -l app.kubernetes.io/name=oci-native-ingress-controller --tail=200 + +python3 - "$records" "$namespace" "$ingress" "$region" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, ingress, region = sys.argv[2], sys.argv[3], sys.argv[4] +pat = re.compile(r"(Failed|Error|Warning|certificate|TLS|listener|backend|reconcile|NotFound|forbidden|load balancer|ingressclass)", re.I) +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if pat.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "Ingress / OCI Native Ingress", + "namespace": namespace, + "ingress": ingress, + "region": region, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-object-correlator.sh b/oci/oke/scripts/oke-object-correlator.sh new file mode 100755 index 0000000..f5ad5d0 --- /dev/null +++ b/oci/oke/scripts/oke-object-correlator.sh @@ -0,0 +1,808 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Build a read-only Kubernetes-to-OCI object graph for OKE troubleshooting. +# +# Usage: +# bash scripts/oke-object-correlator.sh --namespace --cluster-id --compartment-id --region [--pod ] [--deployment ] [--service ] [--ingress ] [--pvc ] [--node ] + +namespace="" +cluster_id="" +compartment_id="" +region="" +pod="" +deployment="" +service="" +ingress="" +pvc="" +node="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --cluster-id) require_value "$1" "${2:-}"; cluster_id="$2"; shift 2 ;; + --compartment-id) require_value "$1" "${2:-}"; compartment_id="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + --pod) require_value "$1" "${2:-}"; pod="$2"; shift 2 ;; + --deployment) require_value "$1" "${2:-}"; deployment="$2"; shift 2 ;; + --service) require_value "$1" "${2:-}"; service="$2"; shift 2 ;; + --ingress) require_value "$1" "${2:-}"; ingress="$2"; shift 2 ;; + --pvc) require_value "$1" "${2:-}"; pvc="$2"; shift 2 ;; + --node) require_value "$1" "${2:-}"; node="$2"; shift 2 ;; + -h|--help) + echo "usage: oke-object-correlator.sh --namespace --cluster-id --compartment-id --region [--pod ] [--deployment ] [--service ] [--ingress ] [--pvc ] [--node ]" + exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done + +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --namespace." "Provide --namespace ." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +record_check() { + local name="$1" + local kind="$2" + local cmd="$3" + local rc="$4" + local out="$5" + NAME="$name" KIND="$kind" CMD="$cmd" RC="$rc" OUT="$out" python3 - "$records" <<'PY' +import json +import os +import sys + +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "kind": os.environ["KIND"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-12000:], + }) + "\n") +PY +} + +run_check() { + local name="$1" + local kind="$2" + shift 2 + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + record_check "$name" "$kind" "$*" "$rc" "$out" +} + +kubectl_available="yes" +oci_available="yes" +if ! command -v kubectl >/dev/null 2>&1; then + kubectl_available="no" + record_check "kubectl availability" "tool" "kubectl" 127 "kubectl is not installed or not on PATH" +fi +if ! command -v oci >/dev/null 2>&1; then + oci_available="no" + record_check "oci availability" "tool" "oci" 127 "oci is not installed or not on PATH" +fi + +if [[ "$kubectl_available" == "yes" ]]; then + if [[ -n "$pod" ]]; then + run_check "pod" "k8s_pod" kubectl -n "$namespace" get pod "$pod" -o json + fi + if [[ -n "$deployment" ]]; then + run_check "deployment" "k8s_deployment" kubectl -n "$namespace" get deployment "$deployment" -o json + fi + if [[ -n "$service" ]]; then + run_check "service" "k8s_service" kubectl -n "$namespace" get service "$service" -o json + run_check "endpoints" "k8s_endpoints" kubectl -n "$namespace" get endpoints "$service" -o json + fi + if [[ -n "$ingress" ]]; then + run_check "ingress" "k8s_ingress" kubectl -n "$namespace" get ingress "$ingress" -o json + fi + if [[ -n "$pvc" ]]; then + run_check "pvc" "k8s_pvc" kubectl -n "$namespace" get pvc "$pvc" -o json + fi + if [[ -n "$node" ]]; then + run_check "node" "k8s_node" kubectl get node "$node" -o json + fi +fi + +oci_cmd=(oci) +if [[ -n "$region" ]]; then + oci_cmd+=(--region "$region") +fi + +python3 - "$records" "$namespace" "$cluster_id" "$compartment_id" "$region" "$pod" "$deployment" "$service" "$ingress" "$pvc" "$node" <<'PY' >"$tmp_dir/plan.json" +import json +import re +import sys +from pathlib import Path + +records_path = Path(sys.argv[1]) +namespace, cluster_id, compartment_id, region = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +pod_arg, deployment_arg, service_arg, ingress_arg, pvc_arg, node_arg = sys.argv[6:12] +records = [json.loads(line) for line in records_path.read_text().splitlines() if line.strip()] + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +def ocid(value): + return isinstance(value, str) and value.startswith("ocid1.") + +def provider_instance_id(obj): + provider_id = ((obj.get("spec") or {}).get("providerID") or "") + match = re.search(r"(ocid1\.instance\.[A-Za-z0-9_.-]+)", provider_id) + return match.group(1) if match else "" + +def lb_addresses(item): + values = [] + for address in item.get("ip-addresses", []) or []: + ip = address.get("ip-address") or address.get("ip") + if ip: + values.append(ip) + return values + +def match_lb_by_ip(items, target_ip): + if not target_ip: + return None + for item in items: + if target_ip in lb_addresses(item): + return item + return None + +def ann(obj, key): + return (((obj or {}).get("metadata") or {}).get("annotations") or {}).get(key, "") + +def meta_name(obj): + return ((obj or {}).get("metadata") or {}).get("name", "") + +def first(value): + if isinstance(value, list) and value: + return value[0] + return None + +plan = [] +derived = {"node": node_arg, "instance_id": "", "lb_id": "", "lb_ip": "", "pv": "", "volume_id": ""} + +for rec in records: + obj = parse_json(rec.get("output", "")) + if rec["kind"] == "k8s_pod" and obj: + node = ((obj.get("spec") or {}).get("nodeName") or "") + if node and not derived["node"]: + derived["node"] = node + for vol in (obj.get("spec") or {}).get("volumes", []) or []: + claim = (vol.get("persistentVolumeClaim") or {}).get("claimName", "") + if claim: + plan.append({"name": f"pvc {claim}", "kind": "k8s_pvc", "cmd": ["kubectl", "-n", namespace, "get", "pvc", claim, "-o", "json"]}) + elif rec["kind"] == "k8s_node" and obj: + annotations = ((obj.get("metadata") or {}).get("annotations") or {}) + instance_id = ( + annotations.get("node.oci.oraclecloud.com/instance-id") + or annotations.get("oci.oraclecloud.com/instance-id") + or annotations.get("oci.oraclecloud.com/instance_id") + or provider_instance_id(obj) + or "" + ) + if instance_id: + derived["instance_id"] = instance_id + elif rec["kind"] == "k8s_service" and obj: + lb_id = ( + ann(obj, "oci.oraclecloud.com/load-balancer-id") + or ann(obj, "service.beta.kubernetes.io/oci-load-balancer-id") + or ann(obj, "oci.oraclecloud.com/network-load-balancer-id") + or "" + ) + if lb_id: + derived["lb_id"] = lb_id + ingress = (((obj.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if ingress: + derived["lb_ip"] = ingress[0].get("ip") or ingress[0].get("hostname") or "" + elif rec["kind"] == "k8s_ingress" and obj: + lb_id = ( + ann(obj, "oci.oraclecloud.com/load-balancer-id") + or ann(obj, "service.beta.kubernetes.io/oci-load-balancer-id") + or "" + ) + if lb_id: + derived["lb_id"] = lb_id + status_ing = (((obj.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if status_ing and not derived["lb_ip"]: + derived["lb_ip"] = status_ing[0].get("ip") or status_ing[0].get("hostname") or "" + elif rec["kind"] == "k8s_pvc" and obj: + pv = ((obj.get("spec") or {}).get("volumeName") or "") + if pv and not derived["pv"]: + derived["pv"] = pv + elif rec["kind"] == "k8s_pv" and obj: + spec = obj.get("spec") or {} + handle = ( + (spec.get("csi") or {}).get("volumeHandle") + or spec.get("ociVolumeID") + or spec.get("volumeID") + or "" + ) + if handle and not derived["volume_id"]: + derived["volume_id"] = handle + +if derived["node"] and not any(r["kind"] == "k8s_node" for r in records): + plan.append({"name": f"node {derived['node']}", "kind": "k8s_node", "cmd": ["kubectl", "get", "node", derived["node"], "-o", "json"]}) +if derived["pv"] and not any(r["kind"] == "k8s_pv" for r in records): + plan.append({"name": f"pv {derived['pv']}", "kind": "k8s_pv", "cmd": ["kubectl", "get", "pv", derived["pv"], "-o", "json"]}) +if derived["instance_id"]: + plan.append({"name": "compute instance", "kind": "oci_instance", "cmd": ["oci", "compute", "instance", "get", "--instance-id", derived["instance_id"], "--output", "json"]}) + if compartment_id: + plan.append({"name": "vnic attachments", "kind": "oci_vnic_attachments", "cmd": ["oci", "compute", "vnic-attachment", "list", "--compartment-id", compartment_id, "--instance-id", derived["instance_id"], "--all", "--output", "json"]}) +if derived["lb_id"]: + if ".networkloadbalancer." in derived["lb_id"]: + plan.append({"name": "network load balancer", "kind": "oci_nlb", "cmd": ["oci", "nlb", "network-load-balancer", "get", "--network-load-balancer-id", derived["lb_id"], "--output", "json"]}) + else: + plan.append({"name": "load balancer", "kind": "oci_lb", "cmd": ["oci", "lb", "load-balancer", "get", "--load-balancer-id", derived["lb_id"], "--output", "json"]}) + plan.append({"name": "load balancer health", "kind": "oci_lb_health", "cmd": ["oci", "lb", "load-balancer-health", "get", "--load-balancer-id", derived["lb_id"], "--output", "json"]}) +elif derived["lb_ip"] and compartment_id: + plan.append({"name": "load balancers by IP", "kind": "oci_lb_list", "cmd": ["oci", "lb", "load-balancer", "list", "--compartment-id", compartment_id, "--all", "--output", "json"]}) + plan.append({"name": "network load balancers by IP", "kind": "oci_nlb_list", "cmd": ["oci", "nlb", "network-load-balancer", "list", "--compartment-id", compartment_id, "--all", "--output", "json"]}) +if derived["volume_id"]: + plan.append({"name": "block volume", "kind": "oci_volume", "cmd": ["oci", "bv", "volume", "get", "--volume-id", derived["volume_id"], "--output", "json"]}) + if compartment_id: + plan.append({"name": "volume attachments", "kind": "oci_volume_attachments", "cmd": ["oci", "compute", "volume-attachment", "list", "--compartment-id", compartment_id, "--volume-id", derived["volume_id"], "--all", "--output", "json"]}) +if service_arg or ingress_arg: + if compartment_id: + plan.append({"name": "network security groups", "kind": "oci_nsgs", "cmd": ["oci", "network", "nsg", "list", "--compartment-id", compartment_id, "--all", "--output", "json"]}) +if cluster_id: + plan.append({"name": "cluster node pools", "kind": "oci_node_pools", "cmd": ["oci", "ce", "node-pool", "list", "--cluster-id", cluster_id, "--compartment-id", compartment_id, "--all", "--output", "json"]}) + +print(json.dumps(plan)) +PY + +execute_plan() { + local plan_file="$1" + python3 - "$plan_file" "$region" <<'PY' >"$tmp_dir/commands.jsonl" +import json +import sys +from pathlib import Path + +plan = json.loads(Path(sys.argv[1]).read_text()) +region = sys.argv[2] +for item in plan: + cmd = item["cmd"] + if cmd and cmd[0] == "oci" and region: + cmd = ["oci", "--region", region] + cmd[1:] + print(json.dumps({"name": item["name"], "kind": item["kind"], "cmd": cmd})) +PY + while IFS= read -r line; do + [[ -n "$line" ]] || continue + name="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["name"])' <<<"$line")" + kind="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["kind"])' <<<"$line")" + cmd_parts=() + while IFS= read -r part; do + cmd_parts+=("$part") + done < <(python3 -c 'import json,sys; [print(x) for x in json.loads(sys.stdin.read())["cmd"]]' <<<"$line") + if [[ "${cmd_parts[0]:-}" == "oci" && "$oci_available" != "yes" ]]; then + continue + fi + run_check "$name" "$kind" "${cmd_parts[@]}" + done <"$tmp_dir/commands.jsonl" +} + +execute_plan "$tmp_dir/plan.json" +python3 - "$records" "$namespace" "$cluster_id" "$compartment_id" "$region" "$pod" "$deployment" "$service" "$ingress" "$pvc" "$node" <<'PY' >"$tmp_dir/plan2.json" +import json +import re +import sys +from pathlib import Path + +records_path = Path(sys.argv[1]) +namespace, cluster_id, compartment_id, region = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +node_arg = sys.argv[11] +records = [json.loads(line) for line in records_path.read_text().splitlines() if line.strip()] + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +def ann(obj, key): + return (((obj or {}).get("metadata") or {}).get("annotations") or {}).get(key, "") + +def provider_instance_id(obj): + provider_id = ((obj.get("spec") or {}).get("providerID") or "") + match = re.search(r"(ocid1\.instance\.[A-Za-z0-9_.-]+)", provider_id) + return match.group(1) if match else "" + +derived = {"instance_id": "", "volume_id": "", "lb_ip": "", "lb_id": ""} +existing = {(r["kind"], r["name"]) for r in records} +plan = [] + +for rec in records: + obj = parse_json(rec.get("output", "")) + if not obj: + continue + data = obj.get("data", obj) if isinstance(obj, dict) else obj + if rec["kind"] == "k8s_node" and isinstance(data, dict): + annotations = ((data.get("metadata") or {}).get("annotations") or {}) + derived["instance_id"] = ( + annotations.get("node.oci.oraclecloud.com/instance-id") + or annotations.get("oci.oraclecloud.com/instance-id") + or annotations.get("oci.oraclecloud.com/instance_id") + or provider_instance_id(data) + or derived["instance_id"] + ) + elif rec["kind"] == "k8s_pvc" and isinstance(data, dict): + pv = ((data.get("spec") or {}).get("volumeName") or "") + if pv and ("k8s_pv", f"pv {pv}") not in existing: + plan.append({"name": f"pv {pv}", "kind": "k8s_pv", "cmd": ["kubectl", "get", "pv", pv, "-o", "json"]}) + elif rec["kind"] == "k8s_pv" and isinstance(data, dict): + spec = data.get("spec") or {} + derived["volume_id"] = ( + (spec.get("csi") or {}).get("volumeHandle") + or spec.get("ociVolumeID") + or spec.get("volumeID") + or derived["volume_id"] + ) + elif rec["kind"] in ("k8s_service", "k8s_ingress") and isinstance(data, dict): + annotations = ((data.get("metadata") or {}).get("annotations") or {}) + derived["lb_id"] = ( + annotations.get("oci.oraclecloud.com/load-balancer-id") + or annotations.get("service.beta.kubernetes.io/oci-load-balancer-id") + or annotations.get("oci.oraclecloud.com/network-load-balancer-id") + or derived["lb_id"] + ) + ingress = (((data.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if ingress and not derived["lb_ip"]: + derived["lb_ip"] = ingress[0].get("ip") or ingress[0].get("hostname") or "" + elif rec["kind"] == "oci_lb_list" and derived["lb_ip"]: + items = data.get("data", []) if isinstance(data, dict) else data + items = items or [] + for item in items: + for address in item.get("ip-addresses", []) or []: + if address.get("ip-address") == derived["lb_ip"]: + derived["lb_id"] = item.get("id", derived["lb_id"]) + break + elif rec["kind"] == "oci_nlb_list" and derived["lb_ip"]: + items = data.get("data", []) if isinstance(data, dict) else data + items = items or [] + for item in items: + for address in item.get("ip-addresses", []) or []: + if address.get("ip-address") == derived["lb_ip"]: + derived["lb_id"] = item.get("id", derived["lb_id"]) + break + +if derived["instance_id"] and ("oci_instance", "compute instance") not in existing: + plan.append({"name": "compute instance", "kind": "oci_instance", "cmd": ["oci", "compute", "instance", "get", "--instance-id", derived["instance_id"], "--output", "json"]}) + if compartment_id and ("oci_vnic_attachments", "vnic attachments") not in existing: + plan.append({"name": "vnic attachments", "kind": "oci_vnic_attachments", "cmd": ["oci", "compute", "vnic-attachment", "list", "--compartment-id", compartment_id, "--instance-id", derived["instance_id"], "--all", "--output", "json"]}) +if derived["volume_id"] and ("oci_volume", "block volume") not in existing: + plan.append({"name": "block volume", "kind": "oci_volume", "cmd": ["oci", "bv", "volume", "get", "--volume-id", derived["volume_id"], "--output", "json"]}) + if compartment_id and ("oci_volume_attachments", "volume attachments") not in existing: + plan.append({"name": "volume attachments", "kind": "oci_volume_attachments", "cmd": ["oci", "compute", "volume-attachment", "list", "--compartment-id", compartment_id, "--volume-id", derived["volume_id"], "--all", "--output", "json"]}) +if derived["lb_id"] and ("oci_lb", "load balancer") not in existing and ("oci_nlb", "network load balancer") not in existing: + if ".networkloadbalancer." in derived["lb_id"]: + plan.append({"name": "network load balancer", "kind": "oci_nlb", "cmd": ["oci", "nlb", "network-load-balancer", "get", "--network-load-balancer-id", derived["lb_id"], "--output", "json"]}) + else: + plan.append({"name": "load balancer", "kind": "oci_lb", "cmd": ["oci", "lb", "load-balancer", "get", "--load-balancer-id", derived["lb_id"], "--output", "json"]}) + plan.append({"name": "load balancer health", "kind": "oci_lb_health", "cmd": ["oci", "lb", "load-balancer-health", "get", "--load-balancer-id", derived["lb_id"], "--output", "json"]}) + +print(json.dumps(plan)) +PY +execute_plan "$tmp_dir/plan2.json" + +python3 - "$records" "$compartment_id" <<'PY' >"$tmp_dir/plan3.json" +import json +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +compartment_id = sys.argv[2] +existing = {(r["kind"], r["name"]) for r in records} +plan = [] + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +for rec in records: + if rec["kind"] != "k8s_pv": + continue + obj = parse_json(rec.get("output", "")) + if not isinstance(obj, dict): + continue + spec = obj.get("spec") or {} + volume_id = ( + (spec.get("csi") or {}).get("volumeHandle") + or spec.get("ociVolumeID") + or spec.get("volumeID") + or "" + ) + if volume_id and ("oci_volume", "block volume") not in existing: + plan.append({"name": "block volume", "kind": "oci_volume", "cmd": ["oci", "bv", "volume", "get", "--volume-id", volume_id, "--output", "json"]}) + if volume_id and compartment_id and ("oci_volume_attachments", "volume attachments") not in existing: + plan.append({"name": "volume attachments", "kind": "oci_volume_attachments", "cmd": ["oci", "compute", "volume-attachment", "list", "--compartment-id", compartment_id, "--volume-id", volume_id, "--all", "--output", "json"]}) + +print(json.dumps(plan)) +PY +execute_plan "$tmp_dir/plan3.json" + +python3 - "$records" "$namespace" "$cluster_id" "$compartment_id" "$region" "$pod" "$deployment" "$service" "$ingress" "$pvc" "$node" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, cluster_id, compartment_id, region = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +pod_arg, deployment_arg, service_arg, ingress_arg, pvc_arg, node_arg = sys.argv[6:12] + +nodes = {} +edges = [] +findings = [] +anomalies = [] +snippets = [] +seen_edges = set() + +def parse_json(text): + try: + return json.loads(text) + except Exception: + return None + +def add_node(node_id, node_type, name="", source="", **attrs): + if not node_id: + return + existing = nodes.setdefault(node_id, {"id": node_id, "type": node_type, "name": name or node_id, "source": source, "attributes": {}}) + existing["type"] = existing.get("type") or node_type + if name: + existing["name"] = name + if source: + existing["source"] = source + for key, value in attrs.items(): + if value not in ("", None, [], {}): + existing["attributes"][key] = value + +def add_edge(src, dst, relation, evidence=""): + if not src or not dst: + return + key = (src, dst, relation) + if key in seen_edges: + return + seen_edges.add(key) + edge = {"from": src, "to": dst, "relation": relation} + if evidence: + edge["evidence"] = evidence + edges.append(edge) + +def data(obj): + if isinstance(obj, dict) and isinstance(obj.get("data"), dict): + return obj["data"] + return obj + +def ann(obj, key): + return (((obj or {}).get("metadata") or {}).get("annotations") or {}).get(key, "") + +def labels(obj): + return ((obj or {}).get("metadata") or {}).get("labels") or {} + +def meta_name(obj): + return ((obj or {}).get("metadata") or {}).get("name", "") + +def k8s_id(kind, name, ns=None): + if not name: + return "" + return f"k8s:{kind}:{ns + '/' if ns else ''}{name}" + +def oci_id(kind, ocid): + if not ocid: + return "" + return f"oci:{kind}:{ocid}" + +def ocid(value): + return isinstance(value, str) and value.startswith("ocid1.") + +def provider_instance_id(obj): + provider_id = ((obj.get("spec") or {}).get("providerID") or "") + match = re.search(r"(ocid1\.instance\.[A-Za-z0-9_.-]+)", provider_id) + return match.group(1) if match else "" + +def lb_addresses(item): + values = [] + for address in item.get("ip-addresses", []) or []: + ip = address.get("ip-address") or address.get("ip") + if ip: + values.append(ip) + return values + +def match_lb_by_ip(items, target_ip): + if not target_ip: + return None + for item in items: + if target_ip in lb_addresses(item): + return item + return None + +derived = {"node": node_arg, "instance_id": "", "lb_id": "", "lb_ip": "", "pv": "", "volume_id": ""} + +for rec in records: + out = rec.get("output", "").strip() + if rec.get("rc") != 0: + anomalies.append(f"{rec['name']} failed with rc={rec['rc']}") + if out: + snippets.append(f"$ {rec['cmd']}\n{out[-1000:]}") + +for rec in records: + obj = parse_json(rec.get("output", "")) + if not obj: + continue + obj_data = data(obj) + kind = rec["kind"] + if kind == "k8s_pod": + name = meta_name(obj_data) or pod_arg + pod_id = k8s_id("pod", name, namespace) + add_node(pod_id, "kubernetes.pod", name, "kubectl", namespace=namespace, service_account=(obj_data.get("spec") or {}).get("serviceAccountName")) + node = (obj_data.get("spec") or {}).get("nodeName", "") + if node: + derived["node"] = node + node_id = k8s_id("node", node) + add_node(node_id, "kubernetes.node", node, "kubectl") + add_edge(pod_id, node_id, "scheduled_on", "pod.spec.nodeName") + sa = (obj_data.get("spec") or {}).get("serviceAccountName", "") + if sa: + sa_id = k8s_id("serviceaccount", sa, namespace) + add_node(sa_id, "kubernetes.serviceaccount", sa, "kubectl", namespace=namespace) + add_edge(pod_id, sa_id, "uses_service_account", "pod.spec.serviceAccountName") + for vol in (obj_data.get("spec") or {}).get("volumes", []) or []: + claim = (vol.get("persistentVolumeClaim") or {}).get("claimName", "") + if claim: + pvc_id = k8s_id("pvc", claim, namespace) + add_node(pvc_id, "kubernetes.pvc", claim, "kubectl", namespace=namespace) + add_edge(pod_id, pvc_id, "mounts_claim", f"pod volume {vol.get('name', '')}") + elif kind == "k8s_node": + name = meta_name(obj_data) or derived["node"] + node_id = k8s_id("node", name) + annotations = ((obj_data.get("metadata") or {}).get("annotations") or {}) + annotation_instance_id = annotations.get("node.oci.oraclecloud.com/instance-id") or annotations.get("oci.oraclecloud.com/instance-id") or annotations.get("oci.oraclecloud.com/instance_id") or "" + provider_instance = provider_instance_id(obj_data) + instance_id = annotation_instance_id or provider_instance + pool = labels(obj_data).get("oke.oraclecloud.com/nodepool") or labels(obj_data).get("oci.oraclecloud.com/oke-nodepool") + add_node(node_id, "kubernetes.node", name, "kubectl", nodepool=pool, provider_id=(obj_data.get("spec") or {}).get("providerID")) + if instance_id: + derived["instance_id"] = instance_id + inst_id = oci_id("instance", instance_id) + source = "kubectl node annotation" if annotation_instance_id else "kubectl node providerID" + evidence = "node instance annotation" if annotation_instance_id else "node.spec.providerID" + add_node(inst_id, "oci.compute.instance", instance_id, source) + add_edge(node_id, inst_id, "runs_on_instance", evidence) + elif kind == "k8s_deployment": + name = meta_name(obj_data) or deployment_arg + dep_id = k8s_id("deployment", name, namespace) + add_node(dep_id, "kubernetes.deployment", name, "kubectl", namespace=namespace, replicas=(obj_data.get("spec") or {}).get("replicas")) + elif kind == "k8s_service": + name = meta_name(obj_data) or service_arg + svc_id = k8s_id("service", name, namespace) + spec = obj_data.get("spec") or {} + add_node(svc_id, "kubernetes.service", name, "kubectl", namespace=namespace, service_type=spec.get("type"), selector=spec.get("selector")) + lb_id = ann(obj_data, "oci.oraclecloud.com/load-balancer-id") or ann(obj_data, "service.beta.kubernetes.io/oci-load-balancer-id") or ann(obj_data, "oci.oraclecloud.com/network-load-balancer-id") + status_ing = (((obj_data.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if status_ing and not lb_id: + derived["lb_ip"] = status_ing[0].get("ip") or status_ing[0].get("hostname") or "" + if lb_id: + derived["lb_id"] = lb_id + lb_kind = "networkloadbalancer" if ".networkloadbalancer." in lb_id else "loadbalancer" + lb_node = oci_id(lb_kind, lb_id) + add_node(lb_node, f"oci.{lb_kind}", lb_id, "service annotation") + add_edge(svc_id, lb_node, "provisions", "service LB annotation") + elif kind == "k8s_endpoints": + name = meta_name(obj_data) or service_arg + ep_id = k8s_id("endpoints", name, namespace) + svc_id = k8s_id("service", name, namespace) + add_node(ep_id, "kubernetes.endpoints", name, "kubectl", namespace=namespace) + add_edge(svc_id, ep_id, "selects_endpoints", "endpoints object") + addresses = [] + for subset in obj_data.get("subsets", []) or []: + for addr in subset.get("addresses", []) or []: + ip = addr.get("ip") + if ip: + addresses.append(ip) + if not addresses: + anomalies.append(f"Service {namespace}/{name} has no ready endpoint addresses") + else: + findings.append(f"Service {namespace}/{name} has {len(addresses)} ready endpoint address(es)") + elif kind == "k8s_ingress": + name = meta_name(obj_data) or ingress_arg + ing_id = k8s_id("ingress", name, namespace) + add_node(ing_id, "kubernetes.ingress", name, "kubectl", namespace=namespace, ingress_class=(obj_data.get("spec") or {}).get("ingressClassName")) + lb_id = ann(obj_data, "oci.oraclecloud.com/load-balancer-id") or ann(obj_data, "service.beta.kubernetes.io/oci-load-balancer-id") + if lb_id: + derived["lb_id"] = lb_id + lb_node = oci_id("loadbalancer", lb_id) + add_node(lb_node, "oci.loadbalancer", lb_id, "ingress annotation") + add_edge(ing_id, lb_node, "provisions", "ingress LB annotation") + status_ing = (((obj_data.get("status") or {}).get("loadBalancer") or {}).get("ingress") or []) + if status_ing and not derived["lb_ip"]: + derived["lb_ip"] = status_ing[0].get("ip") or status_ing[0].get("hostname") or "" + for rule in (obj_data.get("spec") or {}).get("rules", []) or []: + for path in (((rule.get("http") or {}).get("paths")) or []): + backend_svc = ((((path.get("backend") or {}).get("service") or {}).get("name")) or "") + if backend_svc: + svc_id = k8s_id("service", backend_svc, namespace) + add_node(svc_id, "kubernetes.service", backend_svc, "ingress backend", namespace=namespace) + add_edge(ing_id, svc_id, "routes_to", f"host {rule.get('host', '')}") + elif kind == "k8s_pvc": + name = meta_name(obj_data) or pvc_arg + pvc_id = k8s_id("pvc", name, namespace) + pv = (obj_data.get("spec") or {}).get("volumeName", "") + add_node(pvc_id, "kubernetes.pvc", name, "kubectl", namespace=namespace, phase=(obj_data.get("status") or {}).get("phase")) + if pv: + derived["pv"] = pv + pv_id = k8s_id("pv", pv) + add_node(pv_id, "kubernetes.pv", pv, "pvc.spec.volumeName") + add_edge(pvc_id, pv_id, "bound_to", "pvc.spec.volumeName") + elif kind == "k8s_pv": + name = meta_name(obj_data) or derived["pv"] + pv_id = k8s_id("pv", name) + spec = obj_data.get("spec") or {} + handle = (spec.get("csi") or {}).get("volumeHandle") or spec.get("ociVolumeID") or spec.get("volumeID") or "" + add_node(pv_id, "kubernetes.pv", name, "kubectl", storage_class=spec.get("storageClassName")) + if handle: + derived["volume_id"] = handle + vol_id = oci_id("volume", handle) + add_node(vol_id, "oci.blockvolume.volume", handle, "pv volume handle") + add_edge(pv_id, vol_id, "backs_onto", "PV volume handle") + elif kind == "oci_instance": + inst = data(obj_data) + inst_id_raw = inst.get("id", derived["instance_id"]) + inst_id = oci_id("instance", inst_id_raw) + add_node(inst_id, "oci.compute.instance", inst_id_raw, "oci", lifecycle_state=inst.get("lifecycle-state"), shape=inst.get("shape"), availability_domain=inst.get("availability-domain")) + if inst.get("lifecycle-state") and inst.get("lifecycle-state") != "RUNNING": + anomalies.append(f"Compute instance {inst_id_raw} lifecycle-state is {inst.get('lifecycle-state')}") + elif kind == "oci_vnic_attachments": + items = obj_data.get("data", obj_data if isinstance(obj_data, list) else []) or [] + inst_id = oci_id("instance", derived["instance_id"]) + for item in items: + vnic_id = item.get("vnic-id") + subnet_id = item.get("subnet-id") + if vnic_id: + vnic_node = oci_id("vnic", vnic_id) + add_node(vnic_node, "oci.network.vnic", vnic_id, "oci", lifecycle_state=item.get("lifecycle-state"), display_name=item.get("display-name")) + add_edge(inst_id, vnic_node, "has_vnic", "vnic attachment") + if subnet_id: + subnet_node = oci_id("subnet", subnet_id) + add_node(subnet_node, "oci.network.subnet", subnet_id, "oci vnic attachment") + add_edge(oci_id("vnic", vnic_id), subnet_node, "attached_to_subnet", "vnic attachment subnet") + elif kind in ("oci_lb", "oci_nlb"): + lb = data(obj_data) + lb_id_raw = lb.get("id", derived["lb_id"]) + lb_type = "oci.networkloadbalancer" if kind == "oci_nlb" else "oci.loadbalancer" + lb_node = oci_id("networkloadbalancer" if kind == "oci_nlb" else "loadbalancer", lb_id_raw) + add_node(lb_node, lb_type, lb_id_raw, "oci", lifecycle_state=lb.get("lifecycle-state") or lb.get("lifecycle-state-details"), shape=lb.get("shape-name"), ip_addresses=lb_addresses(lb)) + for subnet_id in lb.get("subnet-ids", []) or []: + subnet_node = oci_id("subnet", subnet_id) + add_node(subnet_node, "oci.network.subnet", subnet_id, "oci load balancer") + add_edge(lb_node, subnet_node, "uses_subnet", "load balancer subnet") + if service_arg and (derived["lb_id"] == lb_id_raw or derived["lb_ip"] in lb_addresses(lb)): + add_edge(k8s_id("service", service_arg, namespace), lb_node, "provisions", "matched service external IP or LB ID") + if ingress_arg and (derived["lb_id"] == lb_id_raw or derived["lb_ip"] in lb_addresses(lb)): + add_edge(k8s_id("ingress", ingress_arg, namespace), lb_node, "provisions", "matched ingress external IP or LB ID") + if lb.get("lifecycle-state") and lb.get("lifecycle-state") not in ("ACTIVE", "SUCCEEDED"): + anomalies.append(f"Load balancer {lb_id_raw} lifecycle-state is {lb.get('lifecycle-state')}") + elif kind in ("oci_lb_list", "oci_nlb_list"): + items = obj_data.get("data", []) if isinstance(obj_data, dict) else obj_data + items = items or [] + lb = match_lb_by_ip(items, derived["lb_ip"]) + if lb: + lb_id_raw = lb.get("id", "") + lb_kind = "networkloadbalancer" if kind == "oci_nlb_list" else "loadbalancer" + lb_type = "oci.networkloadbalancer" if kind == "oci_nlb_list" else "oci.loadbalancer" + lb_node = oci_id(lb_kind, lb_id_raw) + derived["lb_id"] = lb_id_raw + add_node(lb_node, lb_type, lb_id_raw, "oci ip match", lifecycle_state=lb.get("lifecycle-state") or lb.get("lifecycle-state-details"), shape=lb.get("shape-name"), ip_addresses=lb_addresses(lb), display_name=lb.get("display-name")) + if service_arg: + add_edge(k8s_id("service", service_arg, namespace), lb_node, "provisions", f"matched external IP {derived['lb_ip']}") + if ingress_arg: + add_edge(k8s_id("ingress", ingress_arg, namespace), lb_node, "provisions", f"matched external IP {derived['lb_ip']}") + for subnet_id in lb.get("subnet-ids", []) or []: + subnet_node = oci_id("subnet", subnet_id) + add_node(subnet_node, "oci.network.subnet", subnet_id, "oci load balancer") + add_edge(lb_node, subnet_node, "uses_subnet", "load balancer subnet") + findings.append(f"Matched external IP {derived['lb_ip']} to OCI {'Network ' if kind == 'oci_nlb_list' else ''}Load Balancer {lb_id_raw}") + if lb.get("lifecycle-state") and lb.get("lifecycle-state") not in ("ACTIVE", "SUCCEEDED"): + anomalies.append(f"Load balancer {lb_id_raw} lifecycle-state is {lb.get('lifecycle-state')}") + elif derived["lb_ip"]: + anomalies.append(f"No OCI {'Network ' if kind == 'oci_nlb_list' else ''}Load Balancer matched external IP {derived['lb_ip']}") + elif kind == "oci_lb_health": + health = data(obj_data) + status = health.get("status") or health.get("overall-health") + lb_node = oci_id("loadbalancer", derived["lb_id"]) + if status: + add_node(f"{lb_node}:health", "oci.loadbalancer.health", status, "oci", status=status) + add_edge(lb_node, f"{lb_node}:health", "has_health", "LB health API") + if str(status).upper() not in ("OK", "SUCCEEDED", "HEALTHY"): + anomalies.append(f"Load balancer health status is {status}") + elif kind == "oci_volume": + vol = data(obj_data) + vol_id_raw = vol.get("id", derived["volume_id"]) + add_node(oci_id("volume", vol_id_raw), "oci.blockvolume.volume", vol_id_raw, "oci", lifecycle_state=vol.get("lifecycle-state"), availability_domain=vol.get("availability-domain"), size_in_gbs=vol.get("size-in-gbs")) + elif kind == "oci_volume_attachments": + items = obj_data.get("data", obj_data if isinstance(obj_data, list) else []) or [] + for item in items: + vol_id_raw = item.get("volume-id", derived["volume_id"]) + inst_id_raw = item.get("instance-id") + if vol_id_raw and inst_id_raw: + add_node(oci_id("volume", vol_id_raw), "oci.blockvolume.volume", vol_id_raw, "oci volume attachment") + add_node(oci_id("instance", inst_id_raw), "oci.compute.instance", inst_id_raw, "oci volume attachment") + add_edge(oci_id("volume", vol_id_raw), oci_id("instance", inst_id_raw), "attached_to_instance", "volume attachment") + if item.get("lifecycle-state") and item.get("lifecycle-state") != "ATTACHED": + anomalies.append(f"Volume attachment {item.get('id', '')} lifecycle-state is {item.get('lifecycle-state')}") + elif kind == "oci_node_pools": + items = obj_data.get("data", obj_data if isinstance(obj_data, list) else []) or [] + cluster_node = oci_id("cluster", cluster_id) + if cluster_id: + add_node(cluster_node, "oci.oke.cluster", cluster_id, "input", region=region, compartment_id=compartment_id) + for item in items: + np_id = item.get("id") + if np_id: + np_node = oci_id("nodepool", np_id) + add_node(np_node, "oci.oke.nodepool", item.get("name", np_id), "oci", size=item.get("size"), shape=item.get("node-shape")) + add_edge(cluster_node, np_node, "has_node_pool", "node pool list") + +if cluster_id: + add_node(oci_id("cluster", cluster_id), "oci.oke.cluster", cluster_id, "input", region=region, compartment_id=compartment_id) + +if not findings: + findings.append(f"Built object graph with {len(nodes)} node(s) and {len(edges)} edge(s)") +else: + findings.insert(0, f"Built object graph with {len(nodes)} node(s) and {len(edges)} edge(s)") + +if not nodes: + anomalies.append("No Kubernetes or OCI objects could be correlated from the provided selectors") + +print(json.dumps({ + "domain": "OCI Object Correlation", + "namespace": namespace, + "cluster_id": cluster_id, + "compartment_id": compartment_id, + "region": region, + "selectors": { + "pod": pod_arg, + "deployment": deployment_arg, + "service": service_arg, + "ingress": ingress_arg, + "pvc": pvc_arg, + "node": node_arg, + }, + "graph": { + "kubernetes": [node for node in nodes.values() if node["type"].startswith("kubernetes.")], + "oci": [node for node in nodes.values() if node["type"].startswith("oci.")], + "edges": edges, + }, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-12:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-ocir-image-pull-check.sh b/oci/oke/scripts/oke-ocir-image-pull-check.sh new file mode 100755 index 0000000..594dbee --- /dev/null +++ b/oci/oke/scripts/oke-ocir-image-pull-check.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect OCIR image pull failure evidence. +# +# Usage: +# bash scripts/oke-ocir-image-pull-check.sh --namespace --pod [--image ] [--compartment-id ] [--region ] + +namespace="" +pod="" +image="" +compartment_id="" +region="" + +emit_error() { + local exit_code="$1"; local error_code="$2"; local message="$3"; local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --pod) require_value "$1" "${2:-}"; pod="$2"; shift 2 ;; + --image) require_value "$1" "${2:-}"; image="$2"; shift 2 ;; + --compartment-id) require_value "$1" "${2:-}"; compartment_id="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + -h|--help) echo "usage: oke-ocir-image-pull-check.sh --namespace --pod [--image ] [--compartment-id ] [--region ]"; exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done +if [[ -z "$namespace" || -z "$pod" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --namespace or --pod." "Provide both --namespace and --pod." +fi +command -v kubectl >/dev/null 2>&1 || emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." + +tmp_dir="$(mktemp -d)"; trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl"; : > "$records" +run_check() { + local name="$1"; shift + local out rc + set +e; out="$("$@" 2>&1)"; rc=$?; set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({"name": os.environ["NAME"], "cmd": os.environ["CMD"], "rc": int(os.environ["RC"]), "output": os.environ["OUT"][-4000:]}) + "\n") +PY +} + +run_check "pod describe" kubectl -n "$namespace" describe pod "$pod" +run_check "pod yaml" kubectl -n "$namespace" get pod "$pod" -o yaml +run_check "pod warning events" kubectl -n "$namespace" get events --field-selector "involvedObject.name=$pod" --sort-by=.lastTimestamp +run_check "service accounts" kubectl -n "$namespace" get serviceaccount -o yaml +run_check "image pull secrets" kubectl -n "$namespace" get secret -o yaml + +if [[ -n "$compartment_id" && -n "$region" ]] && command -v oci >/dev/null 2>&1; then + run_check "OCIR repositories" oci --region "$region" artifacts container repository list --compartment-id "$compartment_id" --all --output json +fi + +python3 - "$records" "$namespace" "$pod" "$image" "$region" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, pod, image, region = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +pat = re.compile(r"(ImagePullBackOff|ErrImagePull|unauthorized|denied|not found|repository|pull access denied|x509|i/o timeout|no basic auth|OCIR)", re.I) +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if pat.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "OCIR / Image Pull", + "namespace": namespace, + "pod": pod, + "image": image, + "region": region, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-pod-network-check.sh b/oci/oke/scripts/oke-pod-network-check.sh new file mode 100755 index 0000000..ed6907d --- /dev/null +++ b/oci/oke/scripts/oke-pod-network-check.sh @@ -0,0 +1,149 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect pod networking, OCI CNI/IPAM, and Multus signals. +# +# Usage: +# bash scripts/oke-pod-network-check.sh --namespace [--pod ] [--selector ] + +namespace="" +pod="" +selector="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) + require_value "$1" "${2:-}" + namespace="$2" + shift 2 + ;; + --pod) + require_value "$1" "${2:-}" + pod="$2" + shift 2 + ;; + --selector|-l) + require_value "$1" "${2:-}" + selector="$2" + shift 2 + ;; + -h|--help) + echo "usage: oke-pod-network-check.sh --namespace [--pod ] [--selector ]" + exit 0 + ;; + *) + emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." + ;; + esac +done + +if [[ -z "$namespace" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing required --namespace." "Provide --namespace ." +fi + +if ! command -v kubectl >/dev/null 2>&1; then + emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1" + shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json +import os +import sys + +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], + }) + "\n") +PY +} + +if [[ -n "$pod" ]]; then + run_check "target pod" kubectl -n "$namespace" get pod "$pod" -o wide + run_check "target pod describe" kubectl -n "$namespace" describe pod "$pod" + run_check "target pod network-status" kubectl -n "$namespace" get pod "$pod" -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' + run_check "target pod node" kubectl -n "$namespace" get pod "$pod" -o jsonpath='{.spec.nodeName}' +else + pod_args=() + if [[ -n "$selector" ]]; then + pod_args=(-l "$selector") + fi + run_check "namespace pods" kubectl -n "$namespace" get pods "${pod_args[@]}" -o wide +fi + +run_check "namespace warning events" kubectl -n "$namespace" get events --field-selector type=Warning --sort-by=.lastTimestamp +run_check "OCI CNI pods" kubectl -n kube-system get pods -l app=oci-cni -o wide +run_check "Multus pods" kubectl -n kube-system get pods -l name=multus -o wide +run_check "network attachment definitions" kubectl get network-attachment-definitions -A + +python3 - "$records" "$namespace" "$pod" "$selector" <<'PY' +import json +import re +import sys +from pathlib import Path + +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, pod, selector = sys.argv[2], sys.argv[3], sys.argv[4] +anomaly_re = re.compile( + r"(FailedCreatePodSandBox|failed to find plugin|CNI|multus|ipam|network-status|ImageInspectError|EOF|timeout|NotReady|Pending|Error|Failed)", + re.I, +) + +findings = [] +anomalies = [] +snippets = [] +for item in records: + output = item["output"].strip() + if item["rc"] == 0: + findings.append(f"{item['name']} collected") + else: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if output: + snippets.append(f"$ {item['cmd']}\n{output[-800:]}") + for line in output.splitlines(): + if anomaly_re.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") + +print(json.dumps({ + "domain": "Pod Networking / OCI CNI / IPAM", + "namespace": namespace, + "pod": pod, + "selector": selector, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-private-endpoint-check.sh b/oci/oke/scripts/oke-private-endpoint-check.sh new file mode 100755 index 0000000..ed78bd3 --- /dev/null +++ b/oci/oke/scripts/oke-private-endpoint-check.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect private OKE API endpoint and kubeconfig connectivity signals. +# +# Usage: +# bash scripts/oke-private-endpoint-check.sh --cluster-id --region [--compartment-id ] + +cluster_id="" +region="" +compartment_id="" + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --cluster-id) require_value "$1" "${2:-}"; cluster_id="$2"; shift 2 ;; + --region) require_value "$1" "${2:-}"; region="$2"; shift 2 ;; + --compartment-id) require_value "$1" "${2:-}"; compartment_id="$2"; shift 2 ;; + -h|--help) + echo "usage: oke-private-endpoint-check.sh --cluster-id --region [--compartment-id ]" + exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done + +if [[ -z "$cluster_id" || -z "$region" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --cluster-id or --region." "Provide both --cluster-id and --region." +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl" +: > "$records" + +run_check() { + local name="$1"; shift + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({ + "name": os.environ["NAME"], + "cmd": os.environ["CMD"], + "rc": int(os.environ["RC"]), + "output": os.environ["OUT"][-4000:], + }) + "\n") +PY +} + +command -v kubectl >/dev/null 2>&1 && { + run_check "current context" kubectl config current-context + run_check "cluster info" kubectl cluster-info + run_check "readyz" kubectl get --raw=/readyz?verbose + run_check "kubectl version" kubectl version --client +} + +if command -v oci >/dev/null 2>&1; then + run_check "OKE cluster" oci --region "$region" ce cluster get --cluster-id "$cluster_id" --output json + if [[ -n "$compartment_id" ]]; then + run_check "NSGs" oci --region "$region" network nsg list --compartment-id "$compartment_id" --all --output json + fi +fi + +python3 - "$records" "$cluster_id" "$region" "$compartment_id" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +cluster_id, region, compartment_id = sys.argv[2], sys.argv[3], sys.argv[4] +pat = re.compile(r"(Unable to connect|i/o timeout|connection refused|x509|Unauthorized|Forbidden|exec: executable oci failed|NotAuthorized|private|endpoint|NSG|security)", re.I) +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if pat.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "Private Cluster / API Endpoint Connectivity", + "cluster_id": cluster_id, + "region": region, + "compartment_id": compartment_id, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/oke-workload-identity-check.sh b/oci/oke/scripts/oke-workload-identity-check.sh new file mode 100755 index 0000000..509db25 --- /dev/null +++ b/oci/oke/scripts/oke-workload-identity-check.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Collect OKE workload identity and pod OCI API auth evidence. +# +# Usage: +# bash scripts/oke-workload-identity-check.sh --namespace --serviceaccount [--pod ] [--tenancy-id ] + +namespace="" +serviceaccount="" +pod="" +tenancy_id="" + +emit_error() { + local exit_code="$1"; local error_code="$2"; local message="$3"; local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} +require_value() { + local flag="$1" + if [[ $# -lt 2 || -z "${2:-}" || "${2:-}" == --* ]]; then + emit_error 2 "INVALID_ARGUMENT" "Missing value for ${flag}." "Run with --help to view usage." + fi +} +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace|-n) require_value "$1" "${2:-}"; namespace="$2"; shift 2 ;; + --serviceaccount|--sa) require_value "$1" "${2:-}"; serviceaccount="$2"; shift 2 ;; + --pod) require_value "$1" "${2:-}"; pod="$2"; shift 2 ;; + --tenancy-id) require_value "$1" "${2:-}"; tenancy_id="$2"; shift 2 ;; + -h|--help) echo "usage: oke-workload-identity-check.sh --namespace --serviceaccount [--pod ] [--tenancy-id ]"; exit 0 ;; + *) emit_error 2 "UNKNOWN_ARGUMENT" "Unknown argument: $1." "Run with --help to view usage." ;; + esac +done +if [[ -z "$namespace" || -z "$serviceaccount" ]]; then + emit_error 2 "MISSING_REQUIRED_ARGUMENT" "Missing --namespace or --serviceaccount." "Provide both --namespace and --serviceaccount." +fi +command -v kubectl >/dev/null 2>&1 || emit_error 2 "KUBECTL_NOT_INSTALLED" "kubectl is not installed or not on PATH." "Install kubectl and retry." + +tmp_dir="$(mktemp -d)"; trap 'rm -rf "$tmp_dir"' EXIT +records="$tmp_dir/records.jsonl"; : > "$records" +run_check() { + local name="$1"; shift + local out rc + set +e; out="$("$@" 2>&1)"; rc=$?; set -e + NAME="$name" RC="$rc" OUT="$out" CMD="$*" python3 - "$records" <<'PY' +import json, os, sys +with open(sys.argv[1], "a") as f: + f.write(json.dumps({"name": os.environ["NAME"], "cmd": os.environ["CMD"], "rc": int(os.environ["RC"]), "output": os.environ["OUT"][-4000:]}) + "\n") +PY +} + +run_check "service account" kubectl -n "$namespace" get serviceaccount "$serviceaccount" -o yaml +if [[ -n "$pod" ]]; then + run_check "pod describe" kubectl -n "$namespace" describe pod "$pod" + run_check "pod logs" sh -c "kubectl -n '$namespace' logs '$pod' --tail=200 2>&1 | egrep -i 'notauthorized|forbidden|401|403|principal|workload|token|oci' || true" +fi +if [[ -n "$tenancy_id" ]] && command -v oci >/dev/null 2>&1; then + run_check "dynamic groups" oci iam dynamic-group list --compartment-id "$tenancy_id" --all --output json + run_check "IAM policies" oci iam policy list --compartment-id "$tenancy_id" --all --output json +fi + +python3 - "$records" "$namespace" "$serviceaccount" "$pod" "$tenancy_id" <<'PY' +import json, re, sys +from pathlib import Path +records = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()] +namespace, serviceaccount, pod, tenancy_id = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] +pat = re.compile(r"(NotAuthorized|Forbidden|401|403|principal|workload|token|dynamic|policy|serviceaccount|oci)", re.I) +findings, anomalies, snippets = [], [], [] +for item in records: + out = item["output"].strip() + findings.append(f"{item['name']} {'collected' if item['rc'] == 0 else 'failed'}") + if item["rc"] != 0: + anomalies.append(f"{item['name']} failed with rc={item['rc']}") + if out: + snippets.append(f"$ {item['cmd']}\n{out[-800:]}") + for line in out.splitlines(): + if pat.search(line): + anomalies.append(f"{item['name']}: {line.strip()[:240]}") +print(json.dumps({ + "domain": "Workload Identity / OCI API From Pods", + "namespace": namespace, + "serviceaccount": serviceaccount, + "pod": pod, + "tenancy_id": tenancy_id, + "findings": findings, + "anomalies": anomalies, + "raw_snippets": snippets[-10:], + "fallback_used": any(item["rc"] != 0 for item in records), +}, indent=2)) +PY diff --git a/oci/oke/scripts/preflight-check.sh b/oci/oke/scripts/preflight-check.sh new file mode 100755 index 0000000..6fbf2f5 --- /dev/null +++ b/oci/oke/scripts/preflight-check.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# preflight-check.sh +# OCI CLI auth + tenancy + region + compartment validation +# +# Usage: bash preflight-check.sh +# Exit: 0 = success +# 1 = expected error (CLI not auth'd, no compartments found) +# 2 = unexpected error (CLI not installed, parse failure) +# +# Stderr: JSON {error_code, message, remediation, docs_url} +# Stdout: JSON {tenancy_ocid, home_region, regions:[], compartments:[]} + +set -euo pipefail + +# ── helpers ────────────────────────────────────────────────────────────────── + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + local docs_url="$5" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":"%s"}\n' \ + "$error_code" "$message" "$remediation" "$docs_url" >&2 + exit "$exit_code" +} + +TMP_SUBSCRIPTIONS_JSON="$(mktemp)" +TMP_COMPARTMENTS_JSON="$(mktemp)" +TMP_REGIONS_JSON="$(mktemp)" +TMP_COMPARTMENTS_OUT_JSON="$(mktemp)" +cleanup_tmp() { + rm -f "$TMP_SUBSCRIPTIONS_JSON" "$TMP_COMPARTMENTS_JSON" "$TMP_REGIONS_JSON" "$TMP_COMPARTMENTS_OUT_JSON" +} +trap cleanup_tmp EXIT + +# ── step 1: verify OCI CLI is installed ────────────────────────────────────── + +if ! command -v oci >/dev/null 2>&1; then + emit_error 2 \ + "OCI_CLI_NOT_INSTALLED" \ + "The OCI CLI is not installed or not on PATH." \ + "Install OCI CLI from https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm" \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm" +fi + +# ── step 2: verify OCI CLI is authenticated ─────────────────────────────────── + +if ! oci iam region-subscription list --output json >/dev/null 2>&1; then + emit_error 1 \ + "OCI_CLI_NOT_AUTHENTICATED" \ + "The OCI CLI is installed but not authenticated. Check ~/.oci/config." \ + "Run: oci setup config" \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliconfigure.htm" +fi + +# ── step 3: extract tenancy OCID and home region ───────────────────────────── + +RAW_SUBSCRIPTIONS=$(oci iam region-subscription list --output json 2>/dev/null) || RAW_SUBSCRIPTIONS='{"data":[]}' +printf '%s' "$RAW_SUBSCRIPTIONS" > "$TMP_SUBSCRIPTIONS_JSON" + +TENANCY_INFO=$(python3 - "$TMP_SUBSCRIPTIONS_JSON" <<'PYEOF' +import json +import sys +from pathlib import Path +raw = Path(sys.argv[1]).read_text() or '{"data":[]}' +data = json.loads(raw).get("data", []) +home = next((r for r in data if r.get("is-home-region")), None) +if not home: + print("ERROR") + sys.exit(1) +print(home.get("region-name", ""), home.get("tenancy-id", "")) +PYEOF +) || true + +if [ -z "$TENANCY_INFO" ] || [ "$TENANCY_INFO" = "ERROR" ]; then + emit_error 1 \ + "TENANCY_DISCOVERY_FAILED" \ + "Could not extract tenancy OCID and home region from OCI CLI output." \ + "Verify your ~/.oci/config has a valid tenancy OCID and region." \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliconfigure.htm" +fi + +HOME_REGION=$(echo "$TENANCY_INFO" | awk '{print $1}') +TENANCY_OCID=$(echo "$TENANCY_INFO" | awk '{print $2}') + +if [ -z "$TENANCY_OCID" ]; then + emit_error 1 \ + "TENANCY_OCID_MISSING" \ + "Tenancy OCID could not be determined from region-subscription output." \ + "Ensure your OCI CLI profile includes a valid tenancy_ocid in ~/.oci/config." \ + "https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliconfigure.htm" +fi + +# ── step 4: list subscribed regions ────────────────────────────────────────── + +REGIONS_JSON=$(python3 - "$TMP_SUBSCRIPTIONS_JSON" <<'PYEOF' +import json +import sys +from pathlib import Path +raw = Path(sys.argv[1]).read_text() or '{"data":[]}' +data = json.loads(raw).get("data", []) +regions = [{"region": r.get("region-name",""), "status": r.get("status","")} for r in data] +print(json.dumps(regions)) +PYEOF +) || REGIONS_JSON='[]' +printf '%s' "$REGIONS_JSON" > "$TMP_REGIONS_JSON" + +# ── step 5: list active compartments ───────────────────────────────────────── + +RAW_COMPARTMENTS=$(oci iam compartment list \ + --compartment-id "$TENANCY_OCID" \ + --compartment-id-in-subtree true \ + --all \ + --lifecycle-state ACTIVE \ + --output json 2>/dev/null) || RAW_COMPARTMENTS='{"data":[]}' +printf '%s' "$RAW_COMPARTMENTS" > "$TMP_COMPARTMENTS_JSON" + +COMPARTMENTS_JSON=$(python3 - "$TMP_COMPARTMENTS_JSON" "$TENANCY_OCID" <<'PYEOF' +import json +import sys +from pathlib import Path +raw = Path(sys.argv[1]).read_text() or '{"data":[]}' +data = json.loads(raw).get("data", []) +tenancy_ocid = sys.argv[2] +compartments = [{"name": c.get("name",""), "ocid": c.get("id",""), "parent": c.get("compartment-id","")} for c in data] +# prepend root compartment (the tenancy itself) +compartments.insert(0, {"name": "root (tenancy)", "ocid": tenancy_ocid, "parent": ""}) +print(json.dumps(compartments)) +PYEOF +) || COMPARTMENTS_JSON='[]' +printf '%s' "$COMPARTMENTS_JSON" > "$TMP_COMPARTMENTS_OUT_JSON" + +# ── step 6: emit structured output ─────────────────────────────────────────── + +python3 - "$TENANCY_OCID" "$HOME_REGION" "$TMP_REGIONS_JSON" "$TMP_COMPARTMENTS_OUT_JSON" <<'PYEOF' +import sys, json +from pathlib import Path +regions_raw = Path(sys.argv[3]).read_text() +compartments_raw = Path(sys.argv[4]).read_text() +result = { + "tenancy_ocid": sys.argv[1], + "home_region": sys.argv[2], + "regions": json.loads(regions_raw), + "compartments": json.loads(compartments_raw) +} +print(json.dumps(result, indent=2)) +PYEOF + +exit 0 diff --git a/oci/oke/scripts/validate-cidr.sh b/oci/oke/scripts/validate-cidr.sh new file mode 100755 index 0000000..c32dac3 --- /dev/null +++ b/oci/oke/scripts/validate-cidr.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# validate-cidr.sh +# Detect CIDR overlap between 2 or more CIDR blocks +# +# Usage: bash validate-cidr.sh [cidr3 ...] +# Exit: 0 = no overlaps found +# 1 = overlaps detected +# 2 = invalid input (bad CIDR format or insufficient arguments) +# +# Stderr: JSON {error_code, message, remediation, docs_url} (exit 2 only) +# Stdout: JSON {valid: bool, overlaps: [{cidr_a, cidr_b}], cidrs_checked: []} + +set -euo pipefail + +emit_error() { + local exit_code="$1" + local error_code="$2" + local message="$3" + local remediation="$4" + printf '{"error_code":"%s","message":"%s","remediation":"%s","docs_url":""}\n' \ + "$error_code" "$message" "$remediation" >&2 + exit "$exit_code" +} + +if [ "$#" -lt 2 ]; then + emit_error 2 \ + "INSUFFICIENT_ARGUMENTS" \ + "validate-cidr.sh requires at least 2 CIDR arguments." \ + "Usage: bash validate-cidr.sh 10.0.0.0/16 10.244.0.0/16 10.96.0.0/16" +fi + +# Delegate all parsing and overlap detection to Python +python3 - "$@" <<'PYEOF' +import sys, json, ipaddress + +cidrs_raw = sys.argv[1:] +cidrs = [] + +for raw in cidrs_raw: + try: + net = ipaddress.ip_network(raw, strict=False) + cidrs.append((raw, net)) + except ValueError as e: + error = { + "error_code": "INVALID_CIDR", + "message": f"Invalid CIDR block: {raw}. {e}", + "remediation": "Provide a valid CIDR in the form A.B.C.D/prefix (e.g. 10.0.0.0/16).", + "docs_url": "" + } + print(json.dumps(error), file=sys.stderr) + sys.exit(2) + +overlaps = [] +for i in range(len(cidrs)): + for j in range(i + 1, len(cidrs)): + raw_a, net_a = cidrs[i] + raw_b, net_b = cidrs[j] + if net_a.overlaps(net_b): + overlaps.append({"cidr_a": raw_a, "cidr_b": raw_b}) + +result = { + "valid": len(overlaps) == 0, + "overlaps": overlaps, + "cidrs_checked": cidrs_raw +} +print(json.dumps(result, indent=2)) +sys.exit(1 if overlaps else 0) +PYEOF diff --git a/oci/oke/shared/oci-resource-map.md b/oci/oke/shared/oci-resource-map.md new file mode 100644 index 0000000..0e8f630 --- /dev/null +++ b/oci/oke/shared/oci-resource-map.md @@ -0,0 +1,121 @@ +# OCI Resource Mapping Cheatsheet + +Use these command chains to relate Kubernetes objects to Oracle Cloud Infrastructure resources during troubleshooting. + +## Automated Correlation +Use the object correlator first when you have a namespace and one or more target selectors. It runs read-only `kubectl` and `oci` lookups and returns a normalized graph with Kubernetes nodes, OCI nodes, and edges between them. + +```bash +./scripts/oke-object-correlator.sh \ + --namespace \ + --cluster-id \ + --compartment-id \ + --region \ + [--pod ] \ + [--deployment ] \ + [--service ] \ + [--ingress ] \ + [--pvc ] \ + [--node ] +``` + +Expected graph edges include: +- `scheduled_on`: Pod to Kubernetes node. +- `runs_on_instance`: Kubernetes node to OCI Compute instance. +- `has_vnic`: OCI instance to VNIC attachment. +- `attached_to_subnet`: VNIC to subnet. +- `provisions`: Kubernetes Service or Ingress to OCI Load Balancer. +- `bound_to` and `backs_onto`: PVC to PV to OCI Block Volume. + +Use the manual command chains below when the correlator cannot resolve a link or when you need to inspect a specific object in more detail. + +## Pod → Node → Instance +1. Identify the node hosting the pod: + ```bash + kubectl get pod -n -o jsonpath='{.spec.nodeName}' + ``` +2. Fetch node annotations for the instance OCID: + ```bash + kubectl get node -o jsonpath='{.metadata.annotations."node\.oci\.oraclecloud\.com/instance-id"}' + ``` +3. Inspect the OCI instance: + ```bash + oci compute instance get --instance-id + ``` +4. List VNIC attachments for the instance: + ```bash + oci compute vnic-attachment list \ + --compartment-id \ + --instance-id \ + --all + ``` + +## Service / Ingress → Load Balancer +1. Obtain OCI load balancer OCID from annotations: + ```bash + kubectl get svc -n -o jsonpath='{.metadata.annotations."oci\.oraclecloud\.com/load-balancer-id"}' + ``` +2. Describe load balancer health: + ```bash + oci lb load-balancer get --load-balancer-id + ``` +3. Review backend set health: + ```bash + oci lb load-balancer-health get \ + --load-balancer-id + ``` + For a specific backend set: + ```bash + oci lb backend-set-health get \ + --load-balancer-id \ + --backend-set-name + ``` + +## PersistentVolumeClaim → Block Volume / File System +1. Retrieve PV name and volume handle: + ```bash + kubectl get pvc -n -o jsonpath='{.spec.volumeName} {.spec.volumeHandle}' + ``` +2. If Block Volume: + ```bash + oci bv volume get --volume-id + ``` + For attachment details: + ```bash + oci compute volume-attachment list --compartment-id --volume-id + ``` +3. If File Storage (FSS): + ```bash + oci fs file-system get --file-system-id + ``` + +## Namespace / Service Account → IAM Policies +1. Determine dynamic group mapping: + ```bash + oci iam dynamic-group list --query "data[].{name:name, matchingRule:matchingRule}" --all + ``` +2. Locate relevant IAM policy statements: + ```bash + oci iam policy list --compartment-id --all \ + --query "data[].{name:name, statements:statements}" + ``` +3. Cross-check Kubernetes service account annotations (workload identity): + ```bash + kubectl get serviceaccount -n -o yaml + ``` + +## Node Pool → Availability Domains +1. Identify node pool OCID: + ```bash + oci ce node-pool list --compartment-id --cluster-id + ``` +2. Fetch node pool details: + ```bash + oci ce node-pool get --node-pool-id + ``` +3. Map to AD and subnet IDs; confirm subnet health via: + ```bash + oci network subnet get --subnet-id + ``` + +Keep this document in sync with any future automation covering additional OCI resource types. diff --git a/oci/oke/skills/AGENTS.md b/oci/oke/skills/AGENTS.md new file mode 100644 index 0000000..bf3b5c5 --- /dev/null +++ b/oci/oke/skills/AGENTS.md @@ -0,0 +1,33 @@ +# Skills Directory Instructions + +## Skill Structure + +Every skill must have: + +- `SKILL.md` with YAML frontmatter containing `name` and `description`. +- A concise body that explains the core workflow and points to references or scripts only when needed. + +Optional files: + +- `references/` for longer notes, examples, constraints, and troubleshooting details. +- `scripts/` for deterministic helpers that are easier to test than prose-only workflows. +- `agents/openai.yaml` for Codex-facing skill UI metadata when useful. + +## Skill Authoring Rules + +- Keep `SKILL.md` focused on procedure, guardrails, and when to load extra resources. +- Put large examples, detailed troubleshooting notes, and edge-case explanations in `references/`. +- Prefer discovery helpers over asking users for raw OCIDs when the environment can enumerate choices. +- Make scripts safe to run offline where possible; destructive OCI or Kubernetes actions must require explicit user confirmation in the skill workflow. +- Public examples must use placeholders instead of real tenancy values, cluster OCIDs, private IPs, or customer names. +- When a skill relies on another skill, state the handoff clearly. For example, multihome pod validation should tell users to run the GVA deployer first if the node pool does not yet have GVA secondary VNIC profiles. + +## Validation + +After changing skill scripts, run: + +```bash +bash tests/scripts-smoke.sh +``` + +If adding a new OKE skill, update `oci/SKILL.md`, the root `README.md`, and the relevant `oci/oke/*.md` routing file. diff --git a/oci/oke/skills/oke-cluster-generator/SKILL.md b/oci/oke/skills/oke-cluster-generator/SKILL.md new file mode 100644 index 0000000..b10a1e5 --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/SKILL.md @@ -0,0 +1,809 @@ +--- +name: oke-cluster-generator +description: Use this skill when the user asks to build, generate, create, design, or scaffold an OKE (OCI Kubernetes Engine) Terraform stack, OCI Kubernetes infrastructure, ORM schema, or Resource Manager template. Trigger phrases include "build an OKE stack", "create OKE Terraform", "generate ORM schema", "deploy OKE cluster", "OKE infrastructure", "terraform-oci-oke", or any request to design OCI Kubernetes infrastructure with Terraform. Do not use it for active incident RCA, GVA secondary VNIC node-pool creation, or Multus pod manifest validation; use the troubleshooting, GVA, or multihome skills for those surfaces. +--- + +## OCI OKE Domain Context + +This operational skill is part of the OCI OKE domain. Route here from `oci/SKILL.md` or `oci/oke/cluster-design.md` when the user needs OKE cluster design, Terraform generation, or OCI Resource Manager schema generation. + +Supporting files (read before code generation): +- `reference.md` — terraform-oci-oke variable catalog: static fallback lists and D1–D6 variable mapping table +- `output-templates/terraform.md` — provider.tf, main.tf module call, and outputs.tf templates +- `output-templates/schema.md` — ORM schema.yaml structure, audience filtering rules, conditional visibility patterns, and validation regexes + +Utility scripts (invoke via Bash tool): +- `../../scripts/preflight-check.sh` — OCI CLI auth + tenancy + region + compartment discovery +- `../../scripts/validate-cidr.sh` — CIDR overlap detection for D2 networking phase + +--- + +## Arguments Pre-fill + +If `$ARGUMENTS` is non-empty, parse it **before** starting Pre-flight to extract +pre-filled answers. This lets users skip questions they already answered in the invocation. + +### Parsing Rules + +Split `$ARGUMENTS` on whitespace. Apply these rules in order: + +| Pattern | Matches when token... | Variable set | +|---------|-----------------------|--------------| +| `FAST_PATH` | matches (case-insensitive): `fast-path`, `fastpath`, `quick-start`, `quickstart`, `starter-stack`, `starter`, `minimal` | Set `FAST_PATH_MODE = true`; consume the token before cluster-name parsing | +| `WORKLOAD_TYPE` | matches (case-insensitive): `ai`, `ai/ml`, `aiml`, `ml`, `gpu`, `hpc`, `microservices`, `general` | Set `WORKLOAD_TYPE`; skip Domain 1 Q1 | +| `TARGET_REGION` | matches an OCI region pattern `--` (e.g. `us-ashburn-1`) | Set `TARGET_REGION`; skip Pre-flight Step 3 region selection | +| `CLUSTER_NAME` | any remaining token that doesn't match the above | Use as suggested `cluster_name`; confirm with user in Domain 1 | + +Canonical workload-type mappings: +- `ai`, `ai/ml`, `aiml`, `ml`, `gpu` → `"AI / ML"` +- `hpc`, `rdma` → `"HPC"` +- `microservices`, `micro` → `"Microservices"` +- anything else → `"General Purpose"` + +### Pre-fill Acknowledgment + +If any values were pre-filled from `$ARGUMENTS`, display this before Pre-flight begins: + +> "Detected from invocation: [list each pre-filled variable and its resolved value]. +> These answers are applied automatically — you can revise them at any domain summary." + +If `FAST_PATH_MODE = true`, do not treat the fast-path trigger token as `CLUSTER_NAME`. +Continue with the Fast Path workflow after pre-flight discovery. + +If `$ARGUMENTS` is empty, proceed with the full questionnaire. + +### Example Prompts + +| Prompt | Effect | +|------------|--------| +| `Use the OKE cluster generator` | Full questionnaire, no pre-fills | +| `Use the OKE cluster generator fast-path us-ashburn-1 demo-private-oke` | Fast path starter stack with region and name pre-filled | +| `Use the OKE cluster generator ai/ml` | WORKLOAD_TYPE = "AI / ML" pre-filled | +| `Use the OKE cluster generator hpc us-frankfurt-1` | WORKLOAD_TYPE = "HPC", TARGET_REGION pre-filled | +| `Use the OKE cluster generator general us-ashburn-1 prod-cluster` | All three pre-filled | + +--- + +## Fast Path Mode + +Use Fast Path Mode when the user says `fast path`, `quick start`, `starter stack`, +`minimal questions`, or asks to generate a basic OKE Terraform stack quickly. + +Fast Path Mode creates a production-friendly starter design and asks only for values +that cannot be safely inferred: + +Required inputs: +- `TENANCY_OCID` +- `COMPARTMENT_OCID` +- `TARGET_REGION` +- `CLUSTER_NAME` + +If any required value is missing after argument parsing and pre-flight discovery, ask +for that value directly. Do not run the full 7-domain questionnaire unless the user asks +to customize the design. + +Fast Path defaults: + +| Area | Default | +| --- | --- | +| Workload type | General Purpose | +| Kubernetes version | Latest GA from OCI CLI, static fallback from `reference.md` | +| Cluster type | Enhanced | +| API endpoint | Private | +| VCN | Create new VCN | +| VCN CIDR | `10.0.0.0/16` | +| Pod CIDR | `10.244.0.0/16` | +| Service CIDR | `10.96.0.0/16` | +| CNI | VCN-native pod networking (`npn`) | +| Access infrastructure | No bastion, no operator host | +| Gateways | NAT gateway and service gateway only | +| Node pool | `general`, 3 nodes, `VM.Standard.E5.Flex`, 2 OCPUs, 16 GB | +| Storage | Block Volume CSI enabled, FSS disabled | +| Security | Create IAM policies, Oracle-managed encryption | +| Workload Identity | Enabled | +| Add-ons | CoreDNS and kube-proxy managed add-ons | +| ORM audience | Expert | + +Fast Path workflow: +1. Announce that Fast Path defaults are being used. +2. Collect missing required inputs. +3. Present a compact architecture summary table with every default listed above. +4. Ask the user to confirm, revise, or switch to the full questionnaire. +5. On confirmation, proceed directly to Phase 3 code generation using the same + reference and template rules as the full workflow. +6. Mention `examples/transcripts/oke-cluster-generator.md` as an example of the + expected interaction and output. + +# OKE Cluster Generator + +You are an expert OCI Infrastructure Architect specializing in OCI Kubernetes Engine (OKE) +cluster design and Terraform automation. Guide the user through a structured, conversational +process to generate a production-ready Terraform stack and an OCI Resource Manager (ORM) +`schema.yaml` for deploying an OKE cluster on OCI. + +## Reference Modules + +Use these as the authoritative source for module structure, variable naming, and best practices: + +- **OCI OKE Terraform Module**: https://github.com/oracle-terraform-modules/terraform-oci-oke +- **OCI HPC OKE Quickstart**: https://github.com/oracle-quickstart/oci-hpc-oke +- **OCI Terraform Provider**: https://github.com/oracle/terraform-provider-oci +- **OKE Official Documentation**: https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm + +Always cite the relevant module variable or submodule by name when a user's choice maps to a +known pattern in these references. + +## OCI CLI Integration + +Use the `Bash` tool throughout the questionnaire to query the user's tenancy and populate +choice prompts with real data (K8s versions, compartments, VCNs, shapes, vault +keys, add-ons). Requires: OCI CLI installed, `~/.oci/config` configured, read access to +IAM, CE, Compute, Network, KMS, and Limits. For failures, apply the CLI Fallback Pattern +(see Behavioral Guidelines). + +## Web Research Constraints + +Only access these URLs and pages linked within them: the four Reference Modules listed +above plus `docs.oracle.com`. Do not perform general web searches. If information is not +found within 5 minutes of research, respond: "I was unable to find a definitive answer +within the approved reference sources." + +--- + +## Behavioral Guidelines + +- Explain *why* a configuration choice matters before asking the user to decide. +- Flag choices that may incur significant cost, require service-limit increases, or have known + OCI regional availability constraints. +- Default to production-grade configurations (HA control plane, private nodes, encrypted + volumes) unless the user explicitly requests otherwise. +- Never generate incomplete Terraform that would cause a `plan` or `apply` to fail. +- Present one domain at a time — summarize choices and confirm before moving to the next. +- **Use `AskUserQuestion` for fixed-choice questions when the runtime provides it.** + Present options as clickable choices with a short `label` and a `description` + explaining the trade-off. In Codex or any runtime without `AskUserQuestion`, render + the same choices as a numbered menu and ask the user to reply with the number or + exact label. Use multi-select only when multiple items may apply (e.g., gateways, + add-ons). Reserve free-text follow-up only for values that require user-specific + input such as CIDRs, OCIDs, cluster names, or node counts. +- You may batch up to **4 related fixed-choice questions** in a single prompt when they + are independent of each other. Split into separate calls when a later question depends + on the answer to an earlier one (e.g., ask VCN source first, then ask for CIDR only if + "New VCN" is chosen). +- After each domain, display a **summary table** of all answers and ask the user to confirm + (Yes / Revise) before moving to the next domain. +- **Use the `Bash` tool to run OCI CLI commands** whenever real tenancy data can improve a + question (e.g., list actual compartments, real K8s versions, available shapes). Always + run the CLI call *before* presenting the choice prompt so options reflect live data. +- **Parse CLI JSON output** with `--query` JMESPath expressions or `| python3 -c` to extract + only the fields needed (name, OCID, state). Never dump raw JSON to the user. + +### CLI Fallback Pattern + +Whenever a `Bash` CLI call is used to populate question options, apply this standard +procedure if the call fails (non-zero exit, empty output, or JSON parse error): + +1. Print informational text to the user (not an error): + > "Could not retrieve live [data-type] from your tenancy. Using the static list below." +2. Present the static fallback options defined in `reference.md` for that domain, OR + switch to a free-text prompt if no static list applies. +3. Set a session flag `CLI_[DOMAIN]_FALLBACK = true` (e.g. `CLI_K8S_FALLBACK`, + `CLI_VCN_FALLBACK`, `CLI_VAULT_FALLBACK`). +4. Continue the questionnaire without interruption. + +In the Phase 2 Architecture Summary, if any `CLI_*_FALLBACK` flag is set, add: +> "Note: Some values were entered manually because live tenancy discovery was unavailable." + +Never abort the skill because a single CLI call fails. + +### Session State Variables + +Track these variables across all domains. They are referenced in CLI calls, cross-domain +conditions, and Phase 3 code generation. + +| Variable | Set in | Used in | Notes | +|----------|--------|---------|-------| +| `WORKLOAD_TYPE` | $ARGUMENTS or D1 Q1 | D2 Extra NICs gate, D3 shape defaults, D6 GPU gate | "AI / ML", "HPC", "Microservices", "General Purpose" | +| `KUBERNETES_VERSION` | D1 Q2 | D6 add-on CLI call | e.g. `"v1.32.1"` | +| `CLUSTER_TYPE` | D1 Q4 | D6 add-on gate, D5 workload identity gate | "Enhanced" or "Basic" | +| `TARGET_REGION` | $ARGUMENTS or Pre-flight S3 | All CLI calls, `region` Terraform var | e.g. `"us-ashburn-1"` | +| `TENANCY_OCID` | Pre-flight S2 | Pre-flight S4, D3 limits CLI | Root compartment OCID | +| `HOME_REGION` | Pre-flight S2 | Display only | Tenancy home region | +| `COMPARTMENT_OCID` | Pre-flight S4 | All subsequent CLI calls | Target compartment OCID → `compartment_ocid` | +| `VCN_SOURCE` | D2 S1 Q1 | D2 S2 branching | `"new"` or `"existing"` | +| `EXISTING_VCN_OCID` | D2 S2 existing path | `vcn_id` Terraform var | Only set if VCN_SOURCE = "existing" | +| `CNI_TYPE` | D2 S1 Q2 | `cni_type` Terraform var | `"npn"` or `"flannel"` | +| `RDMA_ROCE_SELECTED` | D2 S3 Q2 | D3 GPU/RDMA validation | `true` if RDMA/RoCE chosen | +| `NODE_POOL_COUNT` | D3 S1 | D3 loop counter | Integer | +| `POOL_SHAPE_i` | D3 S3 per pool | D6 GPU observability gate | Shape name for pool i | +| `VAULT_MANAGEMENT_ENDPOINT` | D5 vault selection | D5 key list CLI | HTTPS management endpoint URL | +| `KMS_KEY_ID` | D5 key selection | `kms_key_id` Terraform var | OCID of chosen AES key | +| `WORKLOAD_IDENTITY_ENABLED` | D5 Q5 | `workload_identity_enabled` Terraform var | Enhanced clusters only | + +--- + +## Pre-flight: Tenancy Discovery + +Run the preflight script before starting Phase 1: + +```bash +bash ../../scripts/preflight-check.sh +``` + +Parse the JSON output to set session variables: +- `TENANCY_OCID` — from `.tenancy_ocid` +- `HOME_REGION` — from `.home_region` +- `REGIONS` — from `.regions[]` (use to populate Step 3 AskUserQuestion) +- `COMPARTMENTS` — from `.compartments[]` (use to populate Step 4 AskUserQuestion) + +If the script exits **2** (CLI not installed), tell the user: +> "The OCI CLI does not appear to be installed. Please install it and re-invoke the skill." +> Use `AskUserQuestion` — options: "Continue without CLI (enter OCIDs manually)", "Abort". + +If the script exits **1** (CLI not authenticated), tell the user: +> "The OCI CLI is installed but not authenticated. Please run `oci setup config`." +> Use `AskUserQuestion` — options: "Continue without CLI (enter OCIDs manually)", "Abort". + +If the script exits **0**, display: +> "Tenancy: `` | Home region: ``" + +### Step 3 — List available regions and ask which to deploy into + +Use the `REGIONS` array from the preflight script output. Use `AskUserQuestion` to let +the user pick their target deployment region from the subscribed regions. Map the selected +region name to the `region` Terraform variable. + +### Step 4 — List compartments and ask which to deploy into + +Use the `COMPARTMENTS` array from the preflight script output. Use `AskUserQuestion` to +present the compartment names as options. The root tenancy compartment is the first entry. +Store the selected OCID as `COMPARTMENT_OCID` for use in all subsequent CLI calls and as +the `compartment_ocid` Terraform variable. + +--- + +## Phase 1: Discovery (Guided Questionnaire) + +Work through the following domains **one at a time**. After each domain, summarize the user's +answers and ask for confirmation before proceeding. + +### Domain 1 — Cluster Fundamentals + +**CLI step** — Fetch the Kubernetes versions actually supported in the target region: + +```bash +oci ce cluster-options get --cluster-option-id all \ + --query 'data."kubernetes-versions"' \ + --output json +``` + +Use the returned list to populate the K8s version question. Mark the last (highest) version +as "Latest GA (Recommended)". Fall back to the static list in `reference.md` if the command fails. + +Then use a single `AskUserQuestion` call with all 4 questions (they are independent): + +``` +Question 1 — Workload type + header: "Workload" + options: + - label: "General Purpose" + description: "Balanced compute for web apps, APIs, and mixed workloads." + - label: "AI / ML" + description: "GPU-accelerated training and inference; drives GPU shape and DCGM add-on defaults." + - label: "HPC" + description: "Bare-metal, low-latency RDMA/RoCE networking for tightly-coupled simulations." + - label: "Microservices" + description: "High-density, smaller VM shapes optimised for containerised services." + +Question 2 — Kubernetes version + header: "K8s Version" + options: [populate from CLI output, up to 4 most recent versions; add "Other (specify)" as last option] + Static fallback: Read reference.md § Static Kubernetes Versions and use that list. + +Question 3 — Control plane visibility + header: "API Endpoint" + options: + - label: "Private (Recommended)" + description: "API server reachable only from within the VCN; requires bastion or operator for kubectl." + - label: "Public" + description: "API server has a public IP; simpler access but larger attack surface." + +Question 4 — Cluster type + header: "Cluster Type" + options: + - label: "Enhanced (Recommended)" + description: "Unlocks managed add-ons, virtual nodes, workload identity, and OCI Native Ingress." + - label: "Basic" + description: "Simpler setup; no managed add-ons or virtual nodes." +``` + +If the user selects **"Other (specify)"** for Kubernetes version, follow up with a free-text +prompt: "Enter the Kubernetes version string (e.g. `v1.29.1`)." + +### Domain 2 — Networking + +**Step 1** — Use `AskUserQuestion` with 3 independent questions: + +``` +Question 1 — VCN source + header: "VCN" + options: + - label: "Create new VCN" + description: "Terraform provisions a new VCN with subnets sized to your CIDRs." + - label: "Use existing VCN" + description: "Provide the OCID of an existing VCN; subnets will be created inside it." + +Question 2 — CNI plugin + header: "CNI" + options: + - label: "VCN-Native Pod Networking (Recommended)" + description: "Pods get real VCN IPs; enables OCI security lists and native Load Balancer integration. Sets cni_type = \"npn\"." + - label: "Flannel" + description: "Overlay network; simpler but pods are NAT'd — limits direct OCI service integration." + +Question 3 — Bastion / operator access + header: "Access" + options: + - label: "Bastion + Operator (Recommended for private)" + description: "Creates both a public bastion and a private operator instance for kubectl access." + - label: "Bastion only" + description: "Public bastion for SSH tunnelling; you manage kubectl from the bastion host." + - label: "Operator only" + description: "Private operator instance inside VCN; access via existing jump host." + - label: "None" + description: "No access infrastructure — suitable only for public API endpoints." +``` + +**Step 2** — Branch on the VCN source answer: + +**If "Create new VCN"**, ask via free text: +- "VCN CIDR? (default: `10.0.0.0/16`)" + +**If "Use existing VCN"**, run the CLI to list VCNs, then use `AskUserQuestion`: + +```bash +oci network vcn list \ + --compartment-id "$COMPARTMENT_OCID" \ + --lifecycle-state AVAILABLE \ + --query 'data[*].{Name:"display-name",OCID:id,CIDR:"cidr-block"}' \ + --output json +``` + +Present each VCN as an option: `label = Name`, `description = "CIDR: | OCID: "`. +Store the selected OCID as `EXISTING_VCN_OCID`. If the CLI returns no VCNs or fails, apply +the CLI Fallback Pattern (free-text mode): "Enter the existing VCN OCID." + +**Then, regardless of VCN source**, always ask via free text (these are Kubernetes-layer +addresses required by the OKE module for both new and existing VCN configurations): +- "Pod CIDR? (default: `10.244.0.0/16`) — mapped to `pods_cidr`. Must not overlap with the VCN CIDR." +- "Service CIDR? (default: `10.96.0.0/16`) — mapped to `services_cidr`. Must not overlap with VCN or Pod CIDR." + +After the user provides all three CIDRs (VCN CIDR or existing VCN CIDR, Pod CIDR, Service CIDR), +run the CIDR validation script to check for overlaps: + +```bash +bash ../../scripts/validate-cidr.sh "" "" "" +``` + +If the script exits **1** (overlaps detected), show the `.overlaps` array to the user and ask +them to revise the conflicting CIDRs before proceeding. If the script exits **2**, skip +validation and proceed (treat as if no overlap detected). + +**Step 3** — Use `AskUserQuestion` with 2 questions: + +``` +Question 1 — Gateways (multiSelect: true) + header: "Gateways" + options: + - label: "NAT Gateway" + description: "Required for private nodes to reach the internet for image pulls and OCI APIs." + - label: "Service Gateway" + description: "Recommended — free, private access to OCI services (Object Storage, Registry)." + - label: "Internet Gateway" + description: "Required only when subnets need direct inbound internet access." + +Question 2 — Additional interfaces (only ask if WORKLOAD_TYPE is "AI / ML" or "HPC") + header: "Extra NICs" + If "RDMA / RoCE" is selected, set session flag RDMA_ROCE_SELECTED = true. + options: + - label: "RDMA / RoCE" + description: "High-speed RDMA networking for GPU-to-GPU communication. Requires BM.GPU or BM.HPC shapes." + - label: "SR-IOV" + description: "Single Root I/O Virtualisation for near line-rate networking on supported VM shapes." + - label: "Multus multi-homed" + description: "Multiple network interfaces per pod via the Multus CNI meta-plugin." + - label: "None" + description: "Standard single-interface networking." +``` + +### Domain 3 — Node Pools + +**Step 1** — Ask via free text: "How many node pools do you need? (enter a number)" + +**CLI step** — Fetch all shapes available in the compartment, then filter by family for use +in Step 2. Run once before looping over node pools: + +```bash +# All available shapes +oci compute shape list \ + --compartment-id "$COMPARTMENT_OCID" \ + --query 'data[*].shape' \ + --output json + +# GPU shapes specifically (flag quota warnings) +oci limits value list \ + --compartment-id "$TENANCY_OCID" \ + --service-name compute \ + --all \ + --query 'data[?contains(name, `gpu`) || contains(name, `hpc`) || contains(name, `rdma`)]' \ + --output json +``` + +TODO(live validation): confirm the exact OCI Limits command and limit-name mapping for +GPU/HPC/RDMA shape families in the target tenancy before treating a zero or missing +value as conclusive. + +If any GPU/HPC quota value is `0`, warn the user: +> "Warning: Your tenancy shows 0 available service-limit value for [shape family]. You will need a service-limit +> increase before provisioning these nodes." + +For **each** node pool (1 through `NODE_POOL_COUNT`), repeat Steps 2–4. At the start of +each iteration display: + +> "Configuring node pool **[i] of [NODE_POOL_COUNT]**" + +Then ask via free text: "Name for this pool? (e.g. `workers`, `gpu-pool`, `system`)" +Store as `POOL_NAME_i`. After completing Steps 2–4, show a summary for this pool and +confirm before advancing to pool i+1. + +**Step 2** — Use `AskUserQuestion` with 2 questions: + +``` +Question 1 — Shape family + header: "Shape Family" + options: + - label: "VM Standard (E4/E5 Flex)" + description: "General-purpose AMD VMs; flexible OCPUs and memory. Best for microservices and web workloads." + - label: "VM GPU (A10, A100)" + description: "GPU-equipped VMs for inference and moderate training. Requires service limit approval." + - label: "BM GPU (H100, A100)" + description: "Bare-metal GPU nodes for large-scale AI/ML training. Requires RDMA and service limit increases." + - label: "BM HPC (HPC2.36)" + description: "Bare-metal HPC nodes with RDMA networking for tightly-coupled simulations." + +Question 2 — Scaling strategy + header: "Scaling" + options: + - label: "Fixed count" + description: "Static node_pool_size; predictable cost, no autoscaler required." + - label: "Autoscaling (min / max)" + description: "OKE Cluster Autoscaler adjusts the pool size. You specify min and max node counts." +``` + +**Step 3** — Ask via free text for the selected shape: +- "Enter the exact shape name (e.g. `VM.Standard.E4.Flex`, `BM.GPU.H100.8`)." +- If Flex shape: "OCPUs per node? Memory (GB) per node?" +- If Fixed count: "How many nodes?" +- If Autoscaling: "Min nodes? Max nodes?" + +Store as `POOL_SHAPE_i`. Then apply cross-domain validation: + +1. If `RDMA_ROCE_SELECTED = true` AND the shape does **not** start with `BM.GPU` or `BM.HPC`: + > "Warning: RDMA/RoCE (selected in Domain 2) requires a Bare Metal GPU or HPC shape. + > `[shape]` is incompatible." + Use `AskUserQuestion` — options: "Change the shape", "Remove RDMA/RoCE from Domain 2", "Proceed anyway (not recommended)". + +2. If the shape contains `GPU`, `H100`, or `A100`, and `WORKLOAD_TYPE` is not "AI / ML" or "HPC": + > "Note: GPU shapes are typically used with AI/ML or HPC workloads. Your workload type + > is [WORKLOAD_TYPE]. Proceeding — let me know if you'd like to revise." + +3. If the service-limit check from the CLI step showed no available value for this shape family, repeat + the service-limit warning. + +**Step 4** — Use `AskUserQuestion` with 3 questions: + +``` +Question 1 — Boot volume performance + header: "Boot Volume" + options: + - label: "Higher Performance (Recommended)" + description: "Higher IOPS and throughput; better container image pull times." + - label: "Balanced" + description: "Default OCI tier; lower cost, sufficient for light I/O workloads." + +Question 2 — OS image + header: "OS Image" + options: + - label: "OKE-optimised (Recommended)" + description: "Oracle-managed image with the correct kernel, containerd, and OKE agent pre-installed." + - label: "Custom image OCID" + description: "Bring your own image; you are responsible for OKE compatibility." + +Question 3 — Cloud-init / startup script + header: "Cloud-init" + options: + - label: "None" + description: "No custom startup script; use OKE defaults." + - label: "Inline script" + description: "Provide a short shell script to run on first boot (e.g. mount volumes, set kernel params)." + - label: "File path" + description: "Reference a local cloud-init YAML file to be embedded in the Terraform stack." +``` + +If "Custom image OCID" is selected, follow up with free text: "Enter the image OCID." +If "Inline script" or "File path" is selected, follow up with free text for the content or path. + +### Domain 4 — Storage + +Use `AskUserQuestion` with 2 questions: + +``` +Question 1 — Persistent storage backends (multiSelect: true) + header: "Storage" + options: + - label: "OCI Block Volume CSI" + description: "ReadWriteOnce PVCs backed by OCI Block Volumes; ideal for databases and stateful apps." + - label: "OCI File Storage (FSS)" + description: "ReadWriteMany PVCs backed by OCI FSS; required for shared-filesystem workloads." + - label: "Object Storage" + description: "S3-compatible access via rclone or s3fs; suitable for ML datasets and backups." + - label: "None" + description: "No persistent storage — stateless workloads only." + +Question 2 — Local NVMe (only ask if a DenseIO or BM.HPC shape was selected in Domain 3) + header: "Local NVMe" + options: + - label: "Yes — configure NVMe StorageClass" + description: "Provision a local-path StorageClass for the NVMe drives included with DenseIO shapes." + - label: "No" + description: "NVMe drives will not be configured as Kubernetes storage." +``` + +### Domain 5 — Security & Access + +Use `AskUserQuestion` with 4 questions (plus a 5th for Workload Identity if Enhanced): + +``` +Question 1 — Node identity + header: "Node Identity" + options: + - label: "Instance Principals (Recommended)" + description: "Nodes authenticate to OCI APIs via their instance identity — no credentials stored on disk." + - label: "User credentials" + description: "API key credentials stored as Kubernetes secrets; less secure, not recommended for production." + +Question 2 — IAM policies + header: "IAM Policies" + options: + - label: "Auto-generate (Recommended)" + description: "Terraform creates the required dynamic groups and policies for OKE, autoscaler, and add-ons." + - label: "Manual — I will create them" + description: "Skip IAM resource creation; you are responsible for pre-creating all required policies." + +Question 3 — Pod security + header: "Pod Security" + options: + - label: "Kubernetes Pod Security Admission (Recommended)" + description: "Enforce baseline or restricted pod security standards via the built-in K8s admission controller." + - label: "OCI Security Zones" + description: "OCI-level guardrails that prevent insecure resource configurations in the compartment." + - label: "Both" + description: "Apply both OCI Security Zones and Kubernetes PSA for defence-in-depth." + - label: "None" + description: "No additional pod security controls beyond Kubernetes RBAC." + +Question 4 — Volume encryption + header: "Encryption" + options: + - label: "Customer-managed key (BYOK)" + description: "Boot and block volumes encrypted with a key from OCI Vault. Sets kms_key_id and boot_volume_encryption_in_transit_enabled." + - label: "Oracle-managed key (Default)" + description: "OCI encrypts volumes automatically with Oracle-managed keys; simpler setup." +``` + +If "Customer-managed key (BYOK)" is selected, run the CLI to list vaults and keys, then +use `AskUserQuestion` instead of free text: + +```bash +# List vaults in the compartment +oci kms management vault list \ + --compartment-id "$COMPARTMENT_OCID" \ + --lifecycle-state ACTIVE \ + --query 'data[*].{Name:"display-name",OCID:id,Endpoint:"management-endpoint"}' \ + --output json +``` + +Use `AskUserQuestion` to let the user pick a vault. After the selection, **explicitly +capture `VAULT_MANAGEMENT_ENDPOINT`**: extract the `Endpoint` field from the CLI output +for the chosen vault and store it as `VAULT_MANAGEMENT_ENDPOINT`. If the vault list CLI +failed and the user entered a vault OCID manually, ask via free text: +"Enter the vault management endpoint URL (format: `https://-management.kms..oraclecloud.com`)." + +Then fetch keys from the selected vault: + +```bash +# List AES keys in the chosen vault (suitable for volume encryption) +oci kms management key list \ + --compartment-id "$COMPARTMENT_OCID" \ + --endpoint "$VAULT_MANAGEMENT_ENDPOINT" \ + --protection-mode HSM \ + --algorithm AES \ + --lifecycle-state ENABLED \ + --query 'data[*].{Name:"display-name",OCID:id}' \ + --output json +``` + +Present keys as `AskUserQuestion` options: `label = Name`, `description = "OCID: "`. +Store the selected OCID as `KMS_KEY_ID`. If the CLI call fails, apply the CLI Fallback +Pattern (free-text mode): "Enter the KMS key OCID." + +**Question 5 — Workload Identity** (only ask if `CLUSTER_TYPE = "Enhanced"`): + +``` +Question 5 — Workload Identity + header: "Workload Identity" + options: + - label: "Enable (Recommended for Enhanced)" + description: "Pods authenticate to OCI APIs using their Kubernetes service account identity. + No OCI credentials needed in pods. Sets workload_identity_enabled = true." + - label: "Disable" + description: "Pods use node instance principals or manually distributed OCI credentials." +``` + +Store as `WORKLOAD_IDENTITY_ENABLED`. + +### Domain 6 — Add-ons & Observability + +**Prerequisite check** — Before presenting any questions, check `CLUSTER_TYPE` from Domain 1: + +- If `CLUSTER_TYPE = "Basic"`: skip Question 1 (OKE managed add-ons) entirely and inform + the user: "Managed add-ons are only available on Enhanced clusters — skipping add-on + selection." Proceed directly to Question 2 (Observability). +- If `CLUSTER_TYPE = "Enhanced"`: present all questions below as written. + +**CLI step** — Fetch the add-ons available for the chosen Kubernetes version (Enhanced clusters +only). Run before presenting the add-ons question: + +```bash +oci ce addon-option list \ + --kubernetes-version "$KUBERNETES_VERSION" \ + --query 'data[*].{Name:name,Description:description}' \ + --output json +``` + +Use the returned add-on names and descriptions to populate the multiSelect options below. +Fall back to the static list in `reference.md § Static OKE Managed Add-ons` if the command fails. +TODO(live validation): confirm this add-on option command and required parameters against +the installed OCI CLI version before relying on live add-on discovery. + +Use `AskUserQuestion` with 2–3 questions: + +``` +Question 1 — OKE managed add-ons (multiSelect: true; only for Enhanced clusters) + header: "OKE Add-ons" + options: [populate from CLI output; static fallback: Read reference.md § Static OKE Managed Add-ons] + +Question 2 — Observability (multiSelect: true) + header: "Observability" + options: + - label: "OCI Logging" + description: "Stream container stdout/stderr logs to OCI Logging service." + - label: "OCI Monitoring" + description: "Emit cluster and node metrics to OCI Monitoring for alerting and dashboards." + - label: "Container Insights" + description: "OCI Container Insights for deep pod- and namespace-level metrics." + - label: "None" + description: "No OCI-managed observability; bring your own stack (Prometheus, Loki, etc.)." + +Question 3 — GPU observability (only ask if a GPU shape was selected in Domain 3) + header: "GPU Metrics" + options: + - label: "DCGM Exporter (Recommended for GPU)" + description: "Deploys NVIDIA DCGM exporter as a DaemonSet; Prometheus-compatible GPU metrics." + - label: "OCI GPU Scanner" + description: "OCI-native GPU health scanning and telemetry." + - label: "Both" + description: "Deploy both DCGM exporter and OCI GPU Scanner for full coverage." + - label: "None" + description: "No GPU-specific observability." +``` + +### Domain 7 — ORM Schema Preferences + +Use `AskUserQuestion` with 2 questions: + +``` +Question 1 — Target audience + header: "ORM Audience" + options: + - label: "Expert — expose all variables" + description: "Every Terraform variable is surfaced in the ORM UI; suitable for infrastructure engineers." + - label: "App team — simplified view" + description: "Hide networking and security details; expose only cluster name, node count, and shape." + - label: "Ops team — operational controls" + description: "Surface scaling, add-ons, and observability variables; hide core network settings." + - label: "Minimal — required inputs only" + description: "Only tenancy, compartment, and region are required; everything else uses safe defaults." + +Question 2 — Schema features (multiSelect: true) + header: "Schema Features" + options: + - label: "Variable groups" + description: "Organise variables into collapsible sections matching the 7 infrastructure domains." + - label: "Help text and descriptions" + description: "Add title, description, and tooltip text to every surfaced variable." + - label: "Input validation rules" + description: "CIDR format regex, Kubernetes version pattern checks, and allowed-values constraints." + - label: "Conditional visibility" + description: "GPU fields visible only when a GPU shape is chosen; bastion fields only for private clusters." +``` + +--- + +## Phase 2: Architecture Summary + +After all domains are confirmed, produce a concise summary: + +- **Cluster topology** — Control plane visibility, CNI, node pool layout, ADs/FDs. +- **Key design decisions** — Rationale for each major choice. +- **Cost / service-limit warnings** — Flag bare metal shapes, GPU/HPC service limits, cross-AD data transfer costs. +- **Known constraints** — Unsupported shape/CNI combinations, regional availability limits. + +Ask the user to confirm or revise before proceeding to code generation. + +--- + +## Phase 3: Code Generation + +Before generating any code: +1. Read `reference.md` — use the Variable Mapping table to map every user answer to the + exact `terraform-oci-oke` module variable name. +2. Read `output-templates/terraform.md` — use the `provider.tf`, `main.tf`, and `outputs.tf` + templates as the base structure. +3. Read `output-templates/schema.md` — use the schema.yaml structure and conditional + visibility patterns. + +**Never omit required blocks or leave placeholder values that would cause `terraform plan` +to fail. Remove any template lines that don't apply to this deployment.** + +### 1. Terraform Stack + +Generate these five files using the templates from `output-templates/terraform.md`: + +- **`provider.tf`** — Use the provider.tf template verbatim. +- **`variables.tf`** — Declare all input variables with `type`, `description`, and `default`. + Group with comments matching the 7 domain structure. +- **`main.tf`** — Use the module call template; populate all bindings from user answers + using `reference.md § Variable Mapping`. Remove commented-out lines that don't apply. +- **`outputs.tf`** — Use the outputs.tf template; omit `bastion_public_ip` and + `operator_private_ip` if neither was provisioned. +- **`terraform.tfvars.example`** — Populated example values for all declared variables, + with inline comments explaining each field. + +### 2. ORM Schema (`schema.yaml`) + +Use the structure, variable groups, conditional visibility patterns, and validation regex +from `output-templates/schema.md`. Apply the audience filter from Domain 7: + +| Audience | Expose | Hide | +|----------|--------|------| +| Expert | All 6 variable groups | Nothing | +| App team | Cluster Fundamentals only | Networking, Storage, Security, Add-ons | +| Ops team | Cluster Fundamentals + Add-ons & Observability | Networking, Storage, Security | +| Minimal | tenancy_ocid, compartment_ocid, region only | All others (set `required: false` with defaults) | + +--- + +## Phase 4: Iteration & Refinement + +After delivering the artifacts, proactively offer to: + +- Modify specific sections based on user feedback. +- Add advanced configurations: + - Cluster federation + - GitOps bootstrapping (Flux / ArgoCD Helm pre-install) + - Helm chart pre-installation via `null_resource` + `helm_release` + - RDMA / SR-IOV network device plugin DaemonSets +- Export a `README.md` explaining how to deploy via ORM Console, ORM CLI, or Terraform CLI. +- Generate a `Makefile` with common operational targets (`init`, `plan`, `apply`, `destroy`, + `kubeconfig`). diff --git a/oci/oke/skills/oke-cluster-generator/implementation.md b/oci/oke/skills/oke-cluster-generator/implementation.md new file mode 100644 index 0000000..ae1e7de --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/implementation.md @@ -0,0 +1,85 @@ +# Skill 1 Implementation Notes — OKE Cluster Generator + +## Overview +The OKE cluster generator walks operators through a structured, five-phase workflow to produce production-ready Terraform artifacts and an OCI Resource Manager (`schema.yaml`) bundle for OCI Kubernetes Engine (OKE). The skill mirrors the legacy **OKE Terraform Stack Builder** flow and is adapted to the OCI OKE domain with live tenancy discovery, argument pre-fill, and error-contract alignment. + +The skill definition lives in `skills/oke-cluster-generator/SKILL.md`, with supporting materials under the same directory. Shared scripts in `scripts/` (at repo root) provide tenancy discovery and CIDR validation. + +## Architecture Summary + +| Component | Purpose | Location | +|-----------|---------|----------| +| Skill spec | Conversational flow (Pre-flight → Domains → Summary → Generate) | `skills/oke-cluster-generator/SKILL.md` | +| Reference catalog | Static fallback lists + terraform-oci-oke variable map | `skills/oke-cluster-generator/reference.md` | +| Terraform templates | `main.tf`, `variables.tf`, `outputs.tf`, `provider.tf` scaffolding | `skills/oke-cluster-generator/output-templates/terraform.md` | +| ORM schema template | Base `schema.yaml` with conditional visibility rules | `skills/oke-cluster-generator/output-templates/schema.md` | +| Preflight script | Tenancy OCID, home region, compartment discovery | `scripts/preflight-check.sh` | +| CIDR validator | Detects overlap across VCN / Pod / Service CIDRs | `scripts/validate-cidr.sh` | + +## Recent Updates +- Removed non-standard frontmatter fields to align with skill validation rules. +- Added brief TOCs to reference/template files for faster navigation. + +## Execution Flow + +1. **Pre-flight Discovery** + - Parse invocation arguments for `WORKLOAD_TYPE`, `TARGET_REGION`, and `CLUSTER_NAME` (see `Arguments Pre-fill` section in the skill spec). + - Run `scripts/preflight-check.sh` to confirm OCI CLI auth, capture tenancy metadata, and prompt for a target compartment. + - Persist discovered values in session state for subsequent domains. + +2. **Domain Interviews (D1–D7)** + - Follow the sequence defined in the skill: Cluster Fundamentals, Networking, Node Pools, Storage, Security & Access, Add-ons & Observability, ORM Schema Preferences. + - For each domain, prefer live tenancy data via OCI CLI before falling back to `reference.md` lists. Record fallback flags when static options are used. + - Use structured choice prompts with descriptions and enforce branching logic (e.g., prompt for VCN CIDRs only when creating a new VCN). Use numbered menus when the runtime does not provide `AskUserQuestion`. + +3. **Architecture Summary (Phase 2)** + - Render a table of user selections per domain. + - Highlight any CLI fallbacks detected earlier. + - Allow the operator to accept the summary or re-open a domain for adjustments. + +4. **Artifact Generation (Phase 3)** + - Load templates from `output-templates/terraform.md` and `output-templates/schema.md`. + - Map session variables to module inputs using the table in `reference.md` (no ad-hoc naming). + - Populate the bundle with: + - `main.tf` (module invocation, networking extras, add-ons) + - `variables.tf` + - `outputs.tf` + - `provider.tf` + - `schema.yaml` + - Ensure CIDR consistency by re-running `scripts/validate-cidr.sh` if the operator changes networking answers late in the flow. + +5. **Handoff & Iteration (Phase 4)** + - Summarize generated files and next steps (Terraform `init`/`plan`, ORM import instructions). + - Offer to revisit domains for adjustments; regenerate artifacts if the user iterates. + +All phases follow the shared exit-code contract (`0` success, `1` expected issue, `2` unexpected failure) with structured JSON on stderr for errors. + +## Usage Guidelines + +### Prerequisites +- OCI CLI installed and authenticated (`oci setup config`). +- Operator permissions for IAM, Container Engine, Compute, Network, KMS, and Limits services. +- Access to compartment and VCN resources referenced during discovery. + +### Invocation Examples +```bash +Use the OKE cluster generator +Use the OKE cluster generator fast-path us-ashburn-1 demo-private-oke +Use the OKE cluster generator ai/ml us-ashburn-1 prod-cluster +Use the OKE cluster generator hpc us-frankfurt-1 +``` +Arguments pre-fill the workload type, region, and optional cluster name to accelerate Domain 1. + +### Expected Output +- Terraform bundle with validated variables and module wiring targeting `terraform-oci-oke`. +- ORM `schema.yaml` synchronized with generated variables (grouped for Resource Manager UI). +- Supplemental notes calling out manual inputs when CLI fallbacks were used. + +## Verification Checklist +- **Fresh tenancy, no pre-fill:** Run the skill without arguments; ensure Pre-flight discovers tenancy/compartment and domains present live options. +- **Pre-filled AI cluster:** Invoke with `ai/ml us-ashburn-1 prod-cluster`; verify GPU-friendly defaults (RDMA prompt, DCGM add-on availability) and correct mapping in generated Terraform. +- **Existing VCN path:** Choose "Use existing VCN" in Domain 2, supply an OCID, and confirm the generated `main.tf` disables VCN creation while wiring `vcn_id`. +- **CIDR validation:** Enter overlapping Pod/Service CIDRs to confirm `scripts/validate-cidr.sh` returns exit code `1` with structured remediation guidance. +- **Schema sync:** After any domain revisions, regenerate artifacts and inspect `schema.yaml` to ensure `variableGroups` align with the latest Terraform variables. + +Document verification outcomes in the project journal or future automated tests to maintain regression coverage. diff --git a/oci/oke/skills/oke-cluster-generator/output-templates/schema.md b/oci/oke/skills/oke-cluster-generator/output-templates/schema.md new file mode 100644 index 0000000..8156dd9 --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/output-templates/schema.md @@ -0,0 +1,211 @@ +# ORM Schema Templates + +Use these structures and patterns during Phase 3 when generating `schema.yaml`. +Apply audience filtering rules from Domain 7 to decide which variable groups to surface. + +## Contents +- schema.yaml Top-Level Structure +- Audience Filtering (from Domain 7) +- Conditional Visibility Patterns +- Input Validation Patterns +- Per-Variable Schema Entry Example + +--- + +## schema.yaml Top-Level Structure + +```yaml +title: "OKE Terraform Stack" +description: "Deploy an OKE cluster with customizable networking, node pools, and security." +schemaVersion: "1.1.0" +version: "1.0.0" +locale: "en" +logoUrl: "" + +variableGroups: + - title: "Cluster Fundamentals" + variables: + - tenancy_ocid + - compartment_ocid + - region + - cluster_name + - kubernetes_version + - cluster_type + - control_plane_is_public + + - title: "Networking" + variables: + - create_vcn + - vcn_cidrs + - existing_vcn_ocid + - pods_cidr + - services_cidr + - cni_type + - create_bastion + - create_operator + - create_nat_gateway + - create_service_gateway + - create_internet_gateway + + - title: "Node Pools" + variables: + - node_pools + + - title: "Storage" + variables: + - block_volume_csi_enabled + - fss_enabled + + - title: "Security & Access" + variables: + - create_policies + - kms_key_id + - workload_identity_enabled + + - title: "Add-ons & Observability" + variables: + - cluster_addons + - oke_logging_enabled + - oke_monitoring_enabled +``` + +--- + +## Audience Filtering (from Domain 7) + +| Audience | Expose these groups | Hide these groups | +|----------|--------------------|--------------------| +| Expert — all variables | All 6 groups | None | +| App team — simplified | Cluster Fundamentals only | Networking, Storage, Security, Add-ons | +| Ops team — operational | Cluster Fundamentals, Add-ons & Observability | Networking, Storage, Security | +| Minimal — required only | Cluster Fundamentals (tenancy_ocid, compartment_ocid, region only) | All others; set everything else to `required: false` with a sensible default | + +--- + +## Conditional Visibility Patterns + +### Show field only for private control plane: + +```yaml +visible: + and: + - eq: + - ${control_plane_is_public} + - false +``` + +### Show field only for Enhanced cluster: + +```yaml +visible: + and: + - eq: + - ${cluster_type} + - "enhanced" +``` + +### Show field only when creating a new VCN: + +```yaml +visible: + and: + - eq: + - ${create_vcn} + - true +``` + +### Show field only when using an existing VCN: + +```yaml +visible: + and: + - eq: + - ${create_vcn} + - false +``` + +### Show GPU-specific fields (any node pool shape contains "GPU"): + +```yaml +visible: + or: + - contains: + - ${node_pool_shape} + - "GPU" + - contains: + - ${node_pool_shape} + - "H100" + - contains: + - ${node_pool_shape} + - "A100" +``` + +### Show KMS key field only when BYOK is enabled: + +```yaml +visible: + and: + - eq: + - ${byok_enabled} + - true +``` + +--- + +## Input Validation Patterns + +### CIDR block validation: + +```yaml +pattern: "^([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}$" +``` + +### Kubernetes version validation: + +```yaml +pattern: "^v[0-9]+\.[0-9]+(\.[0-9]+)?$" +``` + +### OCI region identifier validation: + +```yaml +pattern: "^[a-z]{2}-[a-z]+-[0-9]$" +``` + +### OCID validation: + +```yaml +pattern: "^ocid1\.[a-z]+\.oc[0-9]+\.[a-z0-9]*\.[a-z0-9]+" +``` + +--- + +## Per-Variable Schema Entry Example + +```yaml +variables: + kubernetes_version: + title: "Kubernetes Version" + description: "OKE Kubernetes version to deploy. Use the latest GA release unless a specific version is required." + type: string + required: true + default: "v1.32" + pattern: "^v[0-9]+\.[0-9]+(\.[0-9]+)?$" + + cni_type: + title: "CNI Plugin" + description: "Container network interface plugin. VCN-Native (npn) is recommended for production." + type: enum + required: true + default: "npn" + enum: + - npn + - flannel + + control_plane_is_public: + title: "API Endpoint Visibility" + description: "Whether the Kubernetes API server endpoint is publicly accessible." + type: boolean + required: true + default: false +``` diff --git a/oci/oke/skills/oke-cluster-generator/output-templates/terraform.md b/oci/oke/skills/oke-cluster-generator/output-templates/terraform.md new file mode 100644 index 0000000..32c27a9 --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/output-templates/terraform.md @@ -0,0 +1,132 @@ +# Terraform Templates + +Use these as the base structure during Phase 3 code generation. Populate all variable +bindings from the user's answers using the mapping table in `reference.md`. +Never leave `# ...` placeholder comments or stub values that would cause `terraform plan` +to fail. + +## Contents +- provider.tf +- main.tf — Module Call +- outputs.tf + +--- + +## provider.tf + +```hcl +terraform { + required_version = ">= 1.3.0" + required_providers { + oci = { + source = "oracle/oci" + version = ">= 5.0.0" + } + } +} + +provider "oci" { + tenancy_ocid = var.tenancy_ocid + user_ocid = var.user_ocid + fingerprint = var.fingerprint + private_key_path = var.private_key_path + region = var.region +} +``` + +--- + +## main.tf — Module Call + +```hcl +module "oke" { + source = "oracle-terraform-modules/oke/oci" + version = "~> 5.0" + + # ── Cluster Fundamentals (Domain 1) ───────────────────────────────────── + tenancy_id = var.tenancy_ocid + compartment_id = var.compartment_ocid + region = var.region + cluster_name = var.cluster_name + kubernetes_version = var.kubernetes_version + cluster_type = var.cluster_type + control_plane_is_public = var.control_plane_is_public + + # ── Networking (Domain 2) ──────────────────────────────────────────────── + create_vcn = var.create_vcn + vcn_cidrs = var.vcn_cidrs # new VCN only + vcn_id = var.existing_vcn_ocid # existing VCN only + pods_cidr = var.pods_cidr + services_cidr = var.services_cidr + cni_type = var.cni_type + create_bastion = var.create_bastion + create_operator = var.create_operator + create_nat_gateway = var.create_nat_gateway + create_service_gateway = var.create_service_gateway + create_internet_gateway = var.create_internet_gateway + + # ── Node Pools (Domain 3) ──────────────────────────────────────────────── + # Expand this map for each pool defined during the questionnaire. + # See reference.md § Variable Mapping for per-pool variable structure. + node_pools = var.node_pools + + # ── Storage (Domain 4) ────────────────────────────────────────────────── + block_volume_csi_enabled = var.block_volume_csi_enabled + fss_enabled = var.fss_enabled + + # ── Security & Access (Domain 5) ──────────────────────────────────────── + create_policies = var.create_policies + kms_key_id = var.kms_key_id + workload_identity_enabled = var.workload_identity_enabled + + # ── Add-ons (Domain 6) ────────────────────────────────────────────────── + cluster_addons = var.cluster_addons +} +``` + +TODO(live validation): Before using this template for an operator-ready bundle, +confirm the current `terraform-oci-oke` module major version and variable names +against the approved module source. If a variable has changed, update +`reference.md` and this template together instead of guessing. + +Remove lines for variables not applicable to this deployment (e.g., remove `kms_key_id` +if Oracle-managed encryption was chosen, remove `vcn_id` if creating a new VCN, etc.). + +--- + +## outputs.tf + +```hcl +output "cluster_id" { + description = "OCID of the OKE cluster" + value = module.oke.cluster_id +} + +output "kubeconfig_cmd" { + description = "Command to generate kubeconfig for this cluster" + value = "oci ce cluster create-kubeconfig --cluster-id ${module.oke.cluster_id} --region ${var.region} --token-version 2.0.0" +} + +output "node_pool_ids" { + description = "Map of node pool name to OCID" + value = module.oke.node_pool_ids +} + +output "vcn_id" { + description = "OCID of the VCN (created or existing)" + value = module.oke.vcn_id +} + +output "bastion_public_ip" { + description = "Public IP of the bastion host (null if not created)" + value = try(module.oke.bastion_public_ip, null) +} + +output "operator_private_ip" { + description = "Private IP of the operator instance (null if not created)" + value = try(module.oke.operator_private_ip, null) +} +``` + +Omit `bastion_public_ip` and `operator_private_ip` if neither was created (both +`create_bastion` and `create_operator` are false). diff --git a/oci/oke/skills/oke-cluster-generator/reference.md b/oci/oke/skills/oke-cluster-generator/reference.md new file mode 100644 index 0000000..1fe11d9 --- /dev/null +++ b/oci/oke/skills/oke-cluster-generator/reference.md @@ -0,0 +1,100 @@ +# OKE Terraform Stack Builder — Reference Data + +This file contains static fallback lists and the authoritative variable mapping table. +Read this file during the questionnaire when a CLI call fails, and at the start of Phase 3 +code generation to ensure correct `terraform-oci-oke` module variable names. + +## Contents +- Static Kubernetes Versions +- Static OKE Managed Add-ons +- Variable Mapping: User Selection → terraform-oci-oke Module Variables + +--- + +## Static Kubernetes Versions + +Use this list when `oci ce cluster-options get` fails or returns empty output. +Mark the first entry as "Latest GA (Recommended)". + +| Label | Description | +|-------|-------------| +| `v1.32` (Latest GA) | Recommended — newest OKE-supported release with the longest support window. | +| `v1.31` | Previous GA release; stable and widely tested on OKE. | +| `v1.30` | Older GA release; choose only if a specific dependency requires it. | +| `Other (specify)` | Enter a custom version string in the follow-up free-text prompt. | + +--- + +## Static OKE Managed Add-ons + +Use this list when live add-on discovery fails or returns empty output. TODO(live validation): +confirm the exact `oci ce addon-option` command and required parameters for the installed +OCI CLI version before relying on live discovery. +These add-ons apply to Enhanced clusters only. + +| Name | Description | +|------|-------------| +| `CoreDNS` | Cluster DNS — always required; managed version receives automatic patch updates. | +| `Kube-proxy` | Network proxy — always required; managed version receives automatic patch updates. | +| `Cluster Autoscaler` | Automatically scales node pools based on pending pod demand. | +| `OCI Native Ingress Controller` | Provisions OCI Load Balancers directly from Ingress resources; no NGINX required. | +| `DCGM Exporter` | NVIDIA DCGM exporter DaemonSet for GPU metrics (AI/ML workloads). | +| `Multus` | Multiple network interfaces per pod via the Multus CNI meta-plugin. | + +--- + +## Variable Mapping: User Selection → terraform-oci-oke Module Variables + +Read this table during Phase 3 code generation. Use the exact variable names shown in the +"Terraform Variable" column — do not guess or infer variable names from the module README. +TODO(live validation): Reconcile this mapping with the currently approved +`terraform-oci-oke` module version before claiming generated Terraform has been +plan-verified. If live validation is unavailable, call out the mapping as a +module-version assumption in the generated summary. + +| Domain | User Selection / Answer | Terraform Variable | Type | Notes | +|--------|------------------------|-------------------|------|-------| +| D1 | Kubernetes version (e.g. `v1.32`) | `kubernetes_version` | string | Quote the string: `"v1.32.1"` | +| D1 | API visibility: Private | `control_plane_is_public = false` | bool | | +| D1 | API visibility: Public | `control_plane_is_public = true` | bool | | +| D1 | Cluster type: Enhanced | `cluster_type = "enhanced"` | string | | +| D1 | Cluster type: Basic | `cluster_type = "basic"` | string | | +| D2 | Create new VCN | `create_vcn = true` | bool | | +| D2 | Use existing VCN | `create_vcn = false`, `vcn_id = var.existing_vcn_ocid` | bool + string | | +| D2 | VCN CIDR | `vcn_cidrs = [""]` | list(string) | New VCN only | +| D2 | Pod CIDR | `pods_cidr = ""` | string | Always required | +| D2 | Service CIDR | `services_cidr = ""` | string | Always required | +| D2 | CNI: VCN-Native Pod Networking | `cni_type = "npn"` | string | | +| D2 | CNI: Flannel | `cni_type = "flannel"` | string | | +| D2 | Bastion + Operator | `create_bastion = true`, `create_operator = true` | bool | | +| D2 | Bastion only | `create_bastion = true`, `create_operator = false` | bool | | +| D2 | Operator only | `create_bastion = false`, `create_operator = true` | bool | | +| D2 | No access infra | `create_bastion = false`, `create_operator = false` | bool | | +| D2 | NAT Gateway | `create_nat_gateway = true` | bool | | +| D2 | Service Gateway | `create_service_gateway = true` | bool | | +| D2 | Internet Gateway | `create_internet_gateway = true` | bool | | +| D3 | Shape name (per pool) | `node_pools..shape` | string | e.g. `"VM.Standard.E4.Flex"` | +| D3 | Flex OCPUs (per pool) | `node_pools..ocpus` | number | Flex shapes only | +| D3 | Flex memory GB (per pool) | `node_pools..memory` | number | Flex shapes only | +| D3 | Fixed count (per pool) | `node_pools..node_pool_size` | number | | +| D3 | Autoscaling enabled | `node_pools..autoscale = true` | bool | | +| D3 | Autoscaling min | `node_pools..min_node_count` | number | | +| D3 | Autoscaling max | `node_pools..max_node_count` | number | | +| D3 | Boot: Higher Performance | `node_pools..boot_volume_vpus_per_gb = 20` | number | | +| D3 | Boot: Balanced | `node_pools..boot_volume_vpus_per_gb = 10` | number | | +| D3 | OS: OKE-optimised | (no override — use OKE default image) | — | | +| D3 | OS: Custom image OCID | `node_pools..image_id = ""` | string | | +| D4 | Block Volume CSI | `block_volume_csi_enabled = true` | bool | | +| D4 | OCI File Storage (FSS) | `fss_enabled = true` | bool | | +| D5 | Instance Principals | (default — no extra variable) | — | | +| D5 | Auto-generate IAM policies | `create_policies = true` | bool | | +| D5 | Manual IAM policies | `create_policies = false` | bool | | +| D5 | Kubernetes PSA | `psp_enabled = false` (use native K8s PSA) | bool | | +| D5 | BYOK encryption key | `kms_key_id = var.kms_key_id` | string | | +| D5 | Oracle-managed encryption | (no variable — OCI default) | — | | +| D5 | Workload Identity enabled | `workload_identity_enabled = true` | bool | Enhanced only | +| D6 | CoreDNS managed add-on | `cluster_addons = { coredns = { remove = false } }` | map | Enhanced only | +| D6 | Cluster Autoscaler add-on | `cluster_addons = { cluster-autoscaler = { remove = false } }` | map | Enhanced only | +| D6 | OCI Native Ingress add-on | `cluster_addons = { oci-native-ingress-controller = { remove = false } }` | map | Enhanced only | +| D6 | OCI Logging | `oke_logging_enabled = true` | bool | | +| D6 | OCI Monitoring | `oke_monitoring_enabled = true` | bool | | diff --git a/oci/oke/skills/oke-gva-deployer/SKILL.md b/oci/oke/skills/oke-gva-deployer/SKILL.md new file mode 100644 index 0000000..a086ceb --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/SKILL.md @@ -0,0 +1,276 @@ +--- +name: oke-gva-deployer +description: Use this skill when the user asks to enable, deploy, or configure Generic VNIC Attachment (GVA) for OCI Kubernetes Engine (OKE), create/update node pools with secondary VNIC profiles, map Application Resources to workloads, or explain GVA functionality, constraints, and scheduling behavior. Do not use it for general OKE incident RCA or for deploying Multus test pods after a GVA node pool already exists; use oke-troubleshooter or oke-multihome-deployer for those surfaces. +--- + +# OKE Generic VNIC Attachment (GVA) Deployer + +You are an OCI networking and OKE specialist. Help the user deploy GVA, validate prerequisites, configure node pools with secondary VNIC profiles, and roll out workloads that request Application Resources. Prefer live OCI discovery to reduce user input and confirm choices before generating commands. + +Hard constraint: +- Never query existing node pools for this workflow. Do not run `oci ce node-pool list` or `oci ce node-pool get`. +- Do not collect node-level information in this workflow. Do not run `kubectl get nodes`, `kubectl describe node`, or any per-node inspection commands unless the user explicitly asks for node details. +- Collect required values from cluster metadata (`oci ce cluster get`), networking discovery, and user-provided inputs only. +- Always use an interactive menu for node pool creation/update. Do not perform non-interactive create/update execution in this workflow. +- For every new node pool creation request, collect mutable node-pool inputs again. Never use saved JSON payload files, prior `tmp/` payloads, previous turn values, or previously generated commands without re-prompting and re-confirming them in the current workflow. +- Never start discovery on an implicit/default cluster. Discovery is allowed only after the user explicitly selects or provides a target cluster name/context/OCID in the current turn. +- Never use an OCI config default region for workflow execution. Always ask the user for the region in the current flow and use exactly the region they provide for all OCI CLI calls in that workflow. + +Supporting reference (load on demand): +- `references/gva.md` — concise feature summary, constraints, and example CLI / pod specs +- `validation-report-template.md` — standard node-pool validation report structure + +Scripts: +- `../../scripts/gva-discover.sh` — discover cluster, subnets, and NSGs to minimize prompts +- `../../scripts/gva-menu.sh` — guided interactive flow that consumes discovery data and prints CLI command + test manifest + +--- + +## Phase 0 — Intake +Flow requirements: +1) Confirm the target cluster first. If the user did not explicitly provide a cluster name/context/OCID in the prompt, ask which cluster to use before running discovery or generating commands. + - Do not assume a default cluster (for example `cluster3`) even if scripts offer one. + - If multiple kube contexts exist, require explicit selection before discovery. +2) Resolve **cluster OCID** from `~/.kube/config` when possible. +3) Resolve **tenancy defaults** from `~/.oci/config` only for non-region values if needed. + - Always ask the user for the region in the current flow. + - Use only the user-provided region for all OCI CLI calls in this workflow. +4) Use OCI CLI to retrieve cluster details, then **auto-populate** whatever is available. +5) Prompt only for missing information. + +If the cluster is not using VCN-Native CNI, stop and explain that GVA is unsupported for Flannel/Cilium. + +--- + +## Phase 1 — Fast Discovery (Mandatory) +For speed, use this sequence first before broader discovery: + +1) Resolve cluster OCID from kubeconfig. +2) Pull cluster details: + +```bash +oci ce cluster get --cluster-id --region +``` + +3) Pull VCNs only in the cluster compartment: + +```bash +oci network vcn list --compartment-id --region +``` + +4) Ask the user which VCN to use. +5) Pull subnets only for the selected VCN: + +```bash +oci network subnet list --compartment-id --vcn-id --region +``` + +6) Pull NSGs in the selected VCN (or compartment fallback if needed): + +```bash +oci network nsg list --compartment-id --vcn-id --region +``` + +Only if this flow fails should you fall back to the broader discovery helper below. + +## Phase 1b — Discovery Helper (Fallback) +When OCI CLI is available and authenticated, you may run: + +```bash +bash ../../scripts/gva-discover.sh --cluster [--region ] [--compartment-id ] [--profile ] [--timeout ] [--kubeconfig ] +``` + +Use the JSON output to populate: +- Cluster OCID, Kubernetes version, compartment OCID, region +- Subnet list (name, OCID, CIDR) +- NSG list (name, OCID) + +If any list is empty or the CLI call fails, fall back to manual prompts for that item. +Do not use node pool discovery as fallback. + +--- + +## Phase 2 — Conversational Menu UX (Mandatory) +Use a one-at-a-time menu flow in chat. Do not ask for multiple unrelated fields in a single prompt. + +Interaction rules: +- Ask exactly one configuration item per turn. +- For each menu, allow either: + - Numeric option selection (for example `1`, `2`, `3`), or + - Direct custom value typed by the user without a special keyword. +- Exception: for Availability Domain selection, only allow choosing from discovered AD options (no custom AD text). +- Prefer numeric menus consistently across all steps. +- Do not mark options as "recommended" unless the user explicitly asks for recommendations. +- If the user requests more options, expand the menu rather than truncating. +- Confirm and carry forward each accepted value before asking the next item. +- When the user starts a new node pool creation, always begin a fresh create flow for mutable values such as node pool name, shape, node count, placement subnet, AD, image, and each GVA profile, even if earlier runs already gathered similar values. +- Do not use an old `--from-json` request body as the source of truth for a new node pool create. + +Menu order: +1) VCN selection +2) `node_pool_name` +3) `node_shape` +4) shape config (OCPUs + memory) when shape is Flex +5) node count +6) Availability Domains (allow one, two, or all three; comma-separated) +7) primary node subnet +8) image selection (filter by Kubernetes version; validate shape architecture/family compatibility before finalizing) +9) Secondary VNIC profile fields: + - `applicationResource` label + - GVA secondary subnet + - NSG selection (`none` allowed) + - `ipCount` (1-256) +10) Ask: "Add another secondary VNIC profile?" and repeat step 9 until user says no. + +Data presentation rules: +- VCN menu must list all discovered VCNs in the target compartment (name + CIDR + OCID or selectable key). +- Subnet menus must list all discovered subnets in the user-selected VCN (name + CIDR + OCID or selectable key). +- Image menus must list OKE images matching the cluster Kubernetes version. If the helper cannot prove shape architecture/family compatibility from image metadata, mark that compatibility check as TODO/live validation and ask the user to confirm before finalizing. +- NSG menus must include all discovered NSGs and a "none" option. +- Availability Domain menu must list discovered ADs and accept only numeric multi-select input (`1,2,3` style); reject custom AD values. +- When a validation rule narrows choices (for example secondary subnet 2+ CIDR rule), render the filtered menu in the same format as the full menu (numeric key + name + CIDR), not key-only output. +- For every filtered menu, re-number options contiguously from `1..N` (do not reuse original indices like `4,10,11`). + +Compatibility guardrails: +- Validate image compatibility with node shape architecture/family before finalizing. +- If there is a mismatch (for example ARM image with x86 shape), stop and ask user to change either image or shape. +- Build one profile per secondary VNIC entry collected in step 9/10. +- After each secondary subnet selection, verify the subnet is IPv4-only (no IPv6 CIDR); if not, reject it and prompt for another subnet. +- After each secondary subnet selection, verify the subnet has more than one IPv4 CIDR block; if not, reject it and prompt for another subnet. + +Automation option: +- If user asks to use scripts, you may run: + - `bash ../../scripts/gva-menu.sh` + - `bash ../../scripts/gva-discover.sh ...` + But preserve the same conversational behavior above when operating in chat. + - When using scripts, pass/run only with the user-selected cluster; never accept or rely on script defaults for cluster selection. + +--- + +## Phase 3 — Design VNIC Profiles +Create a table of VNIC profiles with these fields: +- `applicationResource` (string label used by pods) +- `subnetId` (OCID) +- `ipCount` (integer, max 256) +- `nsgIds` (list, optional) +- `displayName` (optional) +- `assignPublicIp` (optional, default false) +- `tags` (optional) + +Validate: +- Each `applicationResource` is unique. +- Total IPs across VNICs fits expected pod count + buffer. +- Subnets align with intended traffic isolation. +- Each GVA secondary subnet is IPv4-only and has more than one IPv4 CIDR block. + +## Phase 3b — Required Variable Checklist (Always Collect) +Before generating create/update commands, collect and confirm: +- Cluster context/name +- Cluster OCID +- Region +- Compartment OCID +- VCN OCID +- Kubernetes version +- Node pool name (must be explicitly provided; do not auto-finalize a default name without user confirmation) +- Node shape +- OCPUs (if Flex) +- Memory in GB (if Flex) +- Node count +- Availability Domain(s) (one or more) +- Primary node subnet (placement subnet) +- Image OCID matching cluster Kubernetes version +- Secondary VNIC profile list (one or more) +- `applicationResource` label per profile +- GVA secondary subnet per profile +- NSG IDs per profile +- `ipCount` per profile (1-256) +- Secondary VNIC display name (recommended) +- Whether additional GVA profiles are required (explicit yes/no loop) +- Optional node pool parameters (tags, labels, boot volume, SSH key, etc.) +- Final explicit create confirmation in the current turn before generating a runnable command + +--- + +## Phase 4 — Create or Update Node Pool (CLI) +Execution constraint: +- Keep node pool create/update execution interactive (prompt/confirm driven). Command previews are allowed, but do not run non-interactive create/update commands directly. + +CLI runtime workflow (mandatory before create): +1) Check the installed OCI CLI for GA GVA node-pool flags: +```bash +oci --version +oci ce node-pool create --help | grep -E 'secondary-vnics|cni-type' +``` +2) If the flags are missing, stop before create and ask the operator to upgrade +the regular OCI CLI through their normal install path. A preview OCI CLI is no +longer required for this workflow. + +`../../scripts/gva-menu.sh` performs this check before executing a confirmed +create command. If the check fails, print the command only and ask the operator +to install an OCI CLI version with GVA flag support. + +If the user uses OCI CLI, generate a command using the prepared profiles. Use the template below and replace placeholders. + +```bash +oci ce node-pool create \ + --compartment-id "" \ + --cluster-id "" \ + --name "" \ + --kubernetes-version "" \ + --node-shape "" \ + --node-shape-config '{"ocpus":,"memoryInGBs":}' \ + --size \ + --cni-type OCI_VCN_IP_NATIVE \ + --placement-configs '[{"availabilityDomain":"","subnetId":""}]' \ + --node-source-details '{"sourceType":"IMAGE","imageId":""}' \ + --secondary-vnics '' +``` + +Use availability domains discovered from IAM for the tenant (for example `GrCh:US-ASHBURN-AD-1`), not guessed aliases. + +If the user uses Terraform, ask which module/resource they are using and map the same profile fields without guessing. + +--- + +## Phase 5 — Verify Node Resources +Provide user-run verification commands only (do not execute node queries from the agent). Instruct the user to confirm that GVA resources appear on nodes: + +```bash +kubectl describe node +``` + +Expected signals: +- Extended resources like `oke-application-resource.oci.oraclecloud.com/` +- Taint: `oci.oraclecloud.com/application-resource-only:NoSchedule` + +--- + +## Phase 6 — Deploy Workloads +Provide a pod/deployment snippet that: +- Requests exactly **1** unit of the chosen Application Resource +- Adds a toleration for the GVA taint + +Highlight validation rules: +- Exactly one Application Resource type per pod +- Resource count must be exactly 1 +- Pods without toleration will not schedule + +--- + +## Troubleshooting Quick Hits +- **Pods Pending**: No available IPs for requested resource, missing toleration, or wrong resource name +- **Validation webhook errors**: Pod requests multiple resources or incorrect count +- **Capacity issues**: Increase node pool size or rebalance `ipCount` across profiles + +--- + +## Output Expectations +Deliverables should include: +1. A short explanation of GVA functionality +2. A finalized VNIC profile table +3. A ready-to-run CLI command (or Terraform mapping notes) +4. A sample workload manifest +5. A verification checklist with pass/fail placeholders so the user can record node + resource validation after running `kubectl describe node ` +6. A final validation report using `validation-report-template.md` when post-create + validation evidence is available diff --git a/oci/oke/skills/oke-gva-deployer/USAGE.md b/oci/oke/skills/oke-gva-deployer/USAGE.md new file mode 100644 index 0000000..e2b9d66 --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/USAGE.md @@ -0,0 +1,87 @@ +# GVA Skill Usage + +This guide explains how to use the `oke-gva-deployer` skill to add a Generic VNIC Attachment (GVA) node pool to an OKE cluster. + +## Quick Start (Recommended) +Run the guided menu script: + +```bash +./scripts/gva-menu.sh +``` + +The menu: +- Uses only the **cluster name** you explicitly enter. There is no default cluster. +- Pulls cluster OCID from `~/.kube/config` if available. +- Always asks for the region and uses the value you reply with for the workflow. +- Auto-discovers cluster details (compartment, k8s version). +- Lists VCNs, then subnets, then OKE images for your k8s version. +- Prompts for fresh mutable node-pool inputs on each new create flow. +- Ignores old saved JSON payload files for new node-pool creation. +- Prints a ready-to-run OCI CLI command and a test Deployment manifest. +- Validates `ipCount` range and only lists discovered secondary subnets that are IPv4-only with more than one IPv4 CIDR block. + +## Manual Flow (No Menu) +If you can’t run the menu, follow this sequence: + +1. **Identify cluster OCID** + - From kubeconfig: `~/.kube/config` contains `--cluster-id ` in the user exec args. +2. **Fetch cluster details** + - `oci ce cluster get --cluster-id --region ` +3. **List VCNs** + - `oci network vcn list --compartment-id --region ` +4. **Select a VCN and list subnets** + - `oci network subnet list --compartment-id --vcn-id --region ` +5. **Pick OKE image for the cluster’s k8s version** + - `oci ce node-pool-options get --node-pool-option-id --region ` +6. **Create node pool with GVA** + - Use the `oci ce node-pool create` command generated by the skill. +7. **Verify** + - `kubectl describe node ` for GVA extended resources and taint. + +This is the preferred fast-path discovery because it scopes queries to the selected VCN and avoids broad compartment-wide subnet discovery. + +Constraint: +- Do not query existing node pools (`oci ce node-pool list/get`) as part of this skill flow. +- Build the new node pool from cluster metadata, VCN/subnet discovery, image options, and explicit user inputs. +- Never assume a cluster name. +- Always ask for the region and use the user-provided region for the current workflow. + +## OCI CLI Support +GVA node-pool flags are expected in the regular OCI CLI. A preview OCI CLI is no longer required for this workflow. + +```bash +oci --version +oci ce node-pool create --help | grep -E 'secondary-vnics|cni-type' +``` + +If the flags are missing, upgrade OCI CLI through your normal install path before running create. You can still print the generated command for review. + +## Inputs You Still Need to Provide +Even with discovery, you must choose: +- Node pool name +- Node shape (and OCPUs/memory for Flex) +- Node count +- Node placement subnet (from VCN list) +- GVA VNIC profiles (subnet, applicationResource, ipCount) +- Each GVA secondary subnet must be IPv4-only and have more than one IPv4 CIDR block +- Image (from the menu list) + +Always confirm additional GVA variables: +- NSG IDs per GVA profile +- Secondary VNIC display name (recommended) +- Whether additional profiles are needed +- Optional node-pool parameters (tags, labels, boot size, SSH key) + +## Test Deployment +The menu prints a test Deployment manifest. Replace `` with a real image and apply: + +```bash +kubectl apply -f gva-test.yaml +``` + +Ensure the pod requests **exactly one** Application Resource and includes the required toleration. + +## Troubleshooting +- **Pods Pending**: Check resource name, toleration, and IP capacity. +- **Validation errors**: Pods must request exactly 1 unit of a single Application Resource. +- **No extended resources on node**: GVA was not enabled on the node pool. diff --git a/oci/oke/skills/oke-gva-deployer/implementation.md b/oci/oke/skills/oke-gva-deployer/implementation.md new file mode 100644 index 0000000..de4d0a7 --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/implementation.md @@ -0,0 +1,58 @@ +# Implementation Notes — OKE GVA Deployer + +This document records how the `oke-gva-deployer` skill was derived, what has been added since the initial version, and how to extend it safely. + +## Source Material +- `gva-documentation_1_14.docx` (Generic VNIC Attachment for OKE, Document Version 1.0, Last Updated December 2025) + +## Scope +The skill focuses on: +- Explaining GVA functionality and constraints +- Gathering required inputs with minimal prompting +- Designing secondary VNIC profiles +- Producing OCI CLI guidance for node pool creation +- Providing pod scheduling manifests and verification steps + +It does not include Terraform module mappings because the source document is CLI‑oriented and does not specify a canonical Terraform schema. + +## Current Implementation +- **Skill spec**: `skills/oke-gva-deployer/SKILL.md` +- **Reference summary**: `skills/oke-gva-deployer/references/gva.md` +- **Usage guide**: `skills/oke-gva-deployer/USAGE.md` +- **Interactive menu**: `scripts/gva-menu.sh` +- **Discovery helper**: `scripts/gva-discover.sh` + +### Discovery Flow +1. User enters an explicit **cluster name**. No default cluster is allowed. +2. Cluster OCID is resolved from `~/.kube/config` when possible. +3. User enters the region explicitly, and the workflow uses that exact reply. +4. `oci ce cluster get --cluster-id ` retrieves compartment and K8s version. +5. VCN list → subnet list → image list are fetched via OCI CLI. +6. Mutable node-pool values are prompted again for each new create flow. + +### Menu Enhancements Added +- VCN selection **before** subnet selection. +- OKE image list filtered by cluster Kubernetes version. +- No default cluster name. +- Region is always asked explicitly and the user reply is used for OCI calls. +- Optional parameters section (only asks for values the user wants). +- End‑of‑flow action menu: **Run now / Print only / Exit**. +- `ipCount` validation enforces the documented range of `1..256`. +- Discovered secondary subnet choices are filtered to IPv4-only subnets with more than one IPv4 CIDR block. +- GVA node-pool create uses the regular OCI CLI. The workflow checks for `--secondary-vnics` and `--cni-type` before execution and no longer requires a preview OCI CLI. + +## Assumptions +- GVA is supported only with VCN‑Native CNI (`OCI_VCN_IP_NATIVE`). +- `ipCount` per secondary VNIC is capped at 256. +- Pods must request exactly one Application Resource and one unit. +- Nodes receive a taint that requires pod tolerations. + +## Known Gaps +- Terraform mapping and module-specific variables are not included. +- Minimum OCI CLI version for GA GVA flags is checked by help output, not by a pinned version string. +- Shape/image compatibility and VNIC attachment limits by shape are not programmatically validated; keep these as TODO/live validation checks unless OCI metadata proves them. + +## Follow-up Enhancements +- Add a script to generate `--secondary-vnics` JSON from YAML/JSON input. +- Add a Terraform mapping reference once a module or resource structure is confirmed. +- Add a shape/VNIC limit validation step. diff --git a/oci/oke/skills/oke-gva-deployer/references/gva.md b/oci/oke/skills/oke-gva-deployer/references/gva.md new file mode 100644 index 0000000..6ce12c2 --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/references/gva.md @@ -0,0 +1,85 @@ +# Generic VNIC Attachment (GVA) — Reference Summary + +Source: `gva-documentation_1_14.docx` (Document Version 1.0, Last Updated December 2025). + +## What GVA Does +- Provides per-secondary-VNIC configuration for OKE pods. +- Each secondary VNIC can have its own subnet, NSGs, tags, and IP allocation. +- Pods request a specific Application Resource label to select the VNIC type. + +## Why It Matters +- Enables workload isolation by network tier. +- Avoids identical secondary VNICs in standard VCN-Native CNI. +- Supports compliance and multi-tenant segmentation. + +## Key Concepts +- Primary VNIC: node management and control plane traffic. +- Secondary VNICs: pod networking. +- Application Resource: label that maps pod requests to a VNIC profile. +- Scheduler only places pods on nodes that have matching resources. + +## Prerequisites +- OKE cluster with **VCN-Native CNI** (GVA not supported on Flannel). +- Subnets and NSGs per workload tier. +- Each GVA secondary subnet must be IPv4-only and have more than one IPv4 CIDR block. +- Node pool permissions to create/manage VNICs. + +## Constraints and Limits +- `ipCount` per VNIC is capped at **256**. +- Pods can request **only one** Application Resource type. +- Pods must request **exactly 1** unit of that resource. +- Multi-homed pods are not supported. +- VNIC attachment limits depend on instance shape. + +## Node Behavior +- Nodes expose extended resources: + `oke-application-resource.oci.oraclecloud.com/` +- Nodes are tainted: + `oci.oraclecloud.com/application-resource-only:NoSchedule` + +## CLI Version Note +- GVA node-pool create is expected to use the regular OCI CLI. Confirm local support with `oci ce node-pool create --help | grep -E 'secondary-vnics|cni-type'`. + Confirm availability in the user's environment before relying on it. + +## Example Secondary VNIC JSON (template) +Use as `--secondary-vnics ''` in CLI. + +```json +[ + { + "createVnicDetails": { + "ipCount": 256, + "applicationResources": ["Frontend"], + "assignPublicIp": false, + "displayName": "vnic-frontend", + "nsgIds": ["ocid1.nsg..."], + "subnetId": "ocid1.subnet...", + "skipSourceDestCheck": false + }, + "displayName": "vnicattachment-frontend" + } +] +``` + +## Example Pod Snippet (template) + +```yaml +spec: + tolerations: + - key: "oci.oraclecloud.com/application-resource-only" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: app + image: + resources: + requests: + oke-application-resource.oci.oraclecloud.com/Frontend: "1" + limits: + oke-application-resource.oci.oraclecloud.com/Frontend: "1" +``` + +## max-pods Guidance +- GVA reduces per-interface pod capacity (256 IPs per VNIC). +- Set kubelet `max-pods` via cloud-init accordingly. +- HostNetwork pods still count toward max-pods. diff --git a/oci/oke/skills/oke-gva-deployer/validation-report-template.md b/oci/oke/skills/oke-gva-deployer/validation-report-template.md new file mode 100644 index 0000000..0b60430 --- /dev/null +++ b/oci/oke/skills/oke-gva-deployer/validation-report-template.md @@ -0,0 +1,50 @@ +# OKE GVA Node Pool Validation Report Template + +Use this structure after generating a GVA node-pool command or after the user reports +post-create validation results. + +## Summary + +State whether the GVA node-pool configuration is ready to create, created and +validated, or blocked by missing information. + +## Node Pool Configuration + +| Field | Value | +| --- | --- | +| Cluster | `` | +| Region | `` | +| Node pool | `` | +| Shape | `` | +| Node count | `` | +| Primary subnet | `` | +| Kubernetes version | `` | + +## Secondary VNIC Profiles + +| Application Resource | Subnet | `ipCount` | NSGs | Display name | +| --- | --- | ---: | --- | --- | +| `` | `` | `` | `` | `` | + +## Generated Artifacts + +| Artifact | Purpose | +| --- | --- | +| OCI CLI command | Creates the GVA-enabled node pool | +| Workload manifest | Requests one Application Resource and tolerates the GVA taint | +| Verification checklist | Confirms extended resources and scheduling constraints | + +## Validation Checklist + +| Check | Command | Result | +| --- | --- | --- | +| Extended resource present | `kubectl describe node ` | `` | +| GVA taint present | `kubectl describe node ` | `` | +| Test workload schedules | `kubectl -n get pods -o wide` | `` | +| Workload requests exactly one resource | `kubectl -n get pod -o yaml` | `` | + +## Remediation + +If validation fails, keep these causes separate: missing Application Resource on nodes, +wrong resource name in workload, missing toleration, invalid `ipCount`, subnet capacity, +or incompatible node-pool image/shape. diff --git a/oci/oke/skills/oke-multihome-deployer/SKILL.md b/oci/oke/skills/oke-multihome-deployer/SKILL.md new file mode 100644 index 0000000..4d9f7f4 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/SKILL.md @@ -0,0 +1,130 @@ +--- +name: oke-multihome-deployer +description: Deploy and troubleshoot multihome networking on OCI Kubernetes Engine using existing GVA secondary VNIC node pools, Multus thick plugin, OCI CNI/IPAM, NetworkAttachmentDefinitions, and test pods. Use when the user asks to auto-discover OKE cluster subnets or node pools for multihome, create OKE multihome pod manifests, configure Multus NADs for secondary VNIC interfaces, validate multihomed pod connectivity, inspect OCI CNI IPAM allocation on OKE nodes, or package repeatable OKE GVA multihome deployment steps. Do not use it to create or update GVA node pools. For broad incident RCA, DPDK/SR-IOV failures, or non-Multus pod symptoms, use `oke-troubleshooter` first. +--- + +# OKE Multihome Deployer + +## Scope + +Use this skill after an OKE node pool already has GVA secondary VNIC profiles attached. This skill deploys and verifies Multus-based pod multihoming; it does not create or update GVA node pools. + +If the user needs to create or update the node pool secondary VNIC profiles, use `oke-gva-deployer` first, then return here. + +Supporting references: +- `validation-report-template.md` — standard final validation report structure. + +## Core Workflow + +1. Confirm target cluster/context and auth. + - Prefer explicit `OCI_CLI_PROFILE` and `OCI_CLI_AUTH`; when using OCI security-token auth, pass `--auth security_token` or set `OCI_CLI_AUTH=security_token`. + - If `kubectl` fails with `exec: executable oci failed`, ask the user to refresh the OCI session before applying anything. +2. Auto-discover cluster, node pool, and subnet data before generating YAML. + - Use `scripts/discover-oke-multihome.py`. + - Prefer `--cluster-id` when the user provides it; otherwise try `--context`, then `--cluster-name` with `--compartment-id`. + - The discovery script queries OKE node pools and fetches all referenced placement and secondary VNIC subnets. + - Use the discovered node names for `--pod NAME=NODE_NAME` entries. +3. Verify prerequisites from Kubernetes and node-side checks. + - OKE CNI must be `OCI_VCN_IP_NATIVE`. + - Worker nodes must expose secondary VNIC interfaces. Treat `enp1s0` and `enp2s0` as common defaults, then confirm the actual interface names from discovery or node-side evidence before applying manifests. + - OCI CNI binaries should exist on nodes: `oci-ipam`, `oci-ipvlan`, `oci-ptp`. + - The standard `ipvlan` CNI binary is needed when using a plain `ipvlan` secondary NAD. +4. Install Multus thick plugin if absent. + - Upstream manifest used in this workflow: + `https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml` + - TODO(live validation): pin this manifest to a tested Multus release tag after validating it against the target OKE Kubernetes version. Keep using the repo's current manifest reference until that live validation is complete. +5. Generate or edit the deployment manifest. + - Use `scripts/generate-multihome-manifest.py` for repeatable YAML. + - Default network NAD: `kube-system/gva-default-network`, using `oci-ipvlan` on the first secondary VNIC path. + - Secondary NAD: workload namespace scoped, using `ipvlan` on the second secondary VNIC path with `oci-ipam`. +6. Apply and verify. + - Use fully qualified images such as `docker.io/nicolaka/netshoot:v0.13`; CRI-O may reject short image names. + - Check pod annotation `k8s.v1.cni.cncf.io/network-status`. + - Exec `ip -br addr` in each pod and ping the peer `net1` IP both directions. + - Use `validation-report-template.md` for the final validation report, whether + validation passed, failed, or remains incomplete. + +## Discovery + +Run discovery before manifest generation: + +```bash +OCI_CLI_PROFILE= OCI_CLI_AUTH= \ +python3 /scripts/discover-oke-multihome.py \ + --cluster-id \ + --region \ + --pretty > oke-multihome-discovery.json +``` + +If the current kubeconfig contains the cluster OCID in its exec user args, use: + +```bash +OCI_CLI_PROFILE= OCI_CLI_AUTH= \ +python3 /scripts/discover-oke-multihome.py \ + --context \ + --region \ + --pretty +``` + +Use the JSON output to pick: + +- Node pool with GVA `secondaryVnics`. +- Primary placement subnet(s), for node placement context. +- Secondary VNIC subnet IDs/CIDRs, for deciding which interface path maps to each pod network. +- Up to two ready node names for pinned smoke-test pods. + +Do not guess subnet CIDRs or node names when discovery can retrieve them. + +## Manifest Generation + +Generate a two-pod test manifest: + +```bash +python3 /scripts/generate-multihome-manifest.py \ + --namespace gva-multihome-test \ + --default-interface \ + --secondary-interface \ + --pod gva-multihome-a= \ + --pod gva-multihome-b= \ + > gva-multihome-pods.yaml +``` + +Use Kubernetes node names exactly as returned by `kubectl get nodes`. In OKE clusters configured like the handoff, node names may be node private IPs. + +## Verification Commands + +Use these checks after applying manifests: + +```bash +kubectl -n kube-system get pods -l name=multus -o wide +kubectl -n gva-multihome-test get pods -o wide +kubectl -n gva-multihome-test get pod gva-multihome-a gva-multihome-b \ + -o jsonpath='{range .items[*]}{.metadata.name}{" on "}{.spec.nodeName}{"\n"}{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}{"\n\n"}{end}' +kubectl -n gva-multihome-test exec gva-multihome-a -- ip -br addr +kubectl -n gva-multihome-test exec gva-multihome-b -- ip -br addr +``` + +Then ping `net1` IPs across pods: + +```bash +kubectl -n gva-multihome-test exec gva-multihome-a -- ping -c 3 -W 2 +kubectl -n gva-multihome-test exec gva-multihome-b -- ping -c 3 -W 2 +``` + +## Troubleshooting + +Read `references/oke-multihome-notes.md` when: + +- Multus pods are missing or failing. +- Pods are stuck creating sandbox/network. +- `ipvlan` is missing on the host. +- OCI IPAM capacity or allocation behavior is unclear. +- Floating IP/VIP behavior is being explored. + +Read `references/oke-dpdk-mlx5-notes.md` when: + +- The workload uses DPDK, SR-IOV device-plugin resources, Mellanox/NVIDIA VFs, `mlx5`, `vfio-pci`, hugepages, or `/dev/infiniband`. +- A pod requests Multus networks and device-plugin resources but `network-status` only shows `eth0`. +- You need to decide whether the issue is Multus/NAD attachment, resource allocation, driver binding, RDMA/verbs exposure, hugepages, or app PCI/interface mapping. + +For node-side inspection, prefer a privileged diagnostic DaemonSet that mounts `/` at `/host` and runs commands with `chroot /host`, rather than SSHing to each worker. diff --git a/oci/oke/skills/oke-multihome-deployer/agents/openai.yaml b/oci/oke/skills/oke-multihome-deployer/agents/openai.yaml new file mode 100644 index 0000000..4fd316c --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "OKE Multihome Deployer" + short_description: "Auto-discover OKE node pools and subnets, then deploy or troubleshoot Multus multihome networking." + default_prompt: "Auto-discover OKE cluster node pools and subnets, then deploy or troubleshoot multihome networking with GVA secondary VNICs, Multus, NetworkAttachmentDefinitions, and DPDK/SR-IOV mlx5 evidence checks." diff --git a/oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md b/oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md new file mode 100644 index 0000000..c7eb86b --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md @@ -0,0 +1,102 @@ +# OKE DPDK, Multus, and Mellanox mlx5 Notes + +Use this note when an OKE workload combines DPDK, Multus, SR-IOV device-plugin resources, Mellanox/NVIDIA VFs, `mlx5_core`, `vfio-pci`, hugepages, or RDMA/verbs devices. + +## Decision Model + +Do not collapse these mechanisms into one check: + +- Kubernetes device-plugin resource request: allocates a named resource to the pod. +- Multus annotation: asks for named network attachments. +- `NetworkAttachmentDefinition`: defines the requested network. +- `network-status`: proves what Multus actually attached. +- Driver binding: decides whether the VF is owned by Linux `mlx5_core`, `vfio-pci`, or another driver. +- DPDK app config: decides which PCI BDFs, ports, hugepage path, and PMD are used. + +A pod can be `Running` and still have no usable DPDK network path. + +## mlx5 Driver Rule + +For Mellanox/NVIDIA `mlx5` DPDK PMD, `vfio-pci` is not always the desired state. The bifurcated mlx5 model can require the VF to stay bound to Linux `mlx5_core` while DPDK uses mlx5 PMD libraries through the kernel/RDMA userspace stack. + +Treat `vfio-pci` as proven only when the node exposes numbered VFIO group devices such as `/dev/vfio/` and a controlled bind succeeds. If `/dev/vfio` contains only the control device and `/sys/kernel/iommu_groups` is empty, do not keep retrying disruptive binds on live nodes. + +## Read-Only Evidence Commands + +Collect the Kubernetes side first: + +```bash +kubectl get network-attachment-definitions -A | egrep '||noiommu|sriov|mlx' +kubectl -n get network-attachment-definition -o yaml +kubectl -n describe pod +kubectl -n get pod \ + -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' +``` + +Inside the pod: + +```bash +kubectl -n exec -- ip -br addr +kubectl -n exec -- ls -l /dev/infiniband +kubectl -n exec -- ibv_devices +kubectl -n exec -- lspci -nnk +kubectl -n exec -- dpdk-devbind.py -s +kubectl -n exec -- grep -i Huge /proc/meminfo +kubectl -n exec -- mount | grep -i huge +``` + +On the target node, after explicit approval for node-side inspection: + +```bash +ip -br addr +lspci -nnk | egrep -A3 'Mellanox|NVIDIA|Ethernet' +ls -l /dev/vfio +find /sys/kernel/iommu_groups -maxdepth 1 -mindepth 1 -type d | wc -l +dpdk-devbind.py -s +``` + +Capture DPDK startup logs: + +```bash +kubectl -n logs | egrep -i 'EAL|mlx5|ibv|verbs|huge|vfio|pci|rte' +``` + +## Expected Evidence + +Multus attachment is proven only when `network-status` lists the requested networks and interfaces, not just the default network: + +```text +eth0 ... +net1 ... +net2 ... +``` + +The mlx5 PMD path usually needs: + +```text +Kernel driver in use: mlx5_core +/dev/infiniband/uverbs* +ibv_devices shows mlx5 devices +HugePages_Total > 0 +``` + +The VFIO path usually needs: + +```text +/dev/vfio/vfio +/dev/vfio/ +non-empty /sys/kernel/iommu_groups +target PCI device bound to vfio-pci +``` + +## Common Failure Pattern + +If the pod requests networks such as `noiommu-0,noiommu-1` and device-plugin resources such as `mellanox.com/`, but `network-status` only shows `cbr0` or `eth0`, the immediate issue is not proven DPDK initialization. First verify: + +- The NADs exist in the correct namespace. +- The pod annotations reference the correct NAD names or namespace-qualified names. +- Each NAD maps to the intended device-plugin resource when using SR-IOV CNI. +- Pod events do not show Multus or CNI errors. +- The pod has the expected `net1`, `net2`, etc. interfaces. + +Only after this is correct should the investigation move to DPDK PMD, PCI allowlist, hugepage, or NUMA details. diff --git a/oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md b/oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md new file mode 100644 index 0000000..9952442 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/references/oke-multihome-notes.md @@ -0,0 +1,94 @@ +# OKE Multihome Notes + +## Known Working Pattern + +- OKE CNI: `OCI_VCN_IP_NATIVE` +- Multus: thick DaemonSet +- Default Multus network annotation: `v1.multus-cni.io/default-network` +- Additional network annotation: `k8s.v1.cni.cncf.io/networks` +- Default NAD uses `oci-ipvlan` plus `oci-ptp`. +- Secondary NAD can use standard `ipvlan` with OCI `oci-ipam`. +- Netshoot image should be fully qualified: `docker.io/nicolaka/netshoot:v0.13`. + +## Discovery Scope + +Use `scripts/discover-oke-multihome.py` to collect: + +- OKE cluster metadata. +- All node pools in the target cluster. +- Node pool placement subnets. +- GVA secondary VNIC profile subnets. +- Referenced subnet display names, CIDRs, VCN IDs, and availability domains. + +Discovery intentionally reads node pool details because secondary VNIC profile data lives on the node pool. It does not create or update any OCI resources. + +Example interface mapping from the observed OKE GVA nodes: + +```text +enp0s5 = primary node subnet +enp1s0 = first secondary VNIC path +enp2s0 = second secondary VNIC path +``` + +## CNI Binary Checks + +The OKE nodes should already have: + +```text +oci-ipam +oci-ipvlan +oci-ptp +``` + +The standard `ipvlan` binary may be missing. If the secondary NAD uses `"type": "ipvlan"`, install or copy the `ipvlan` CNI binary into host `/opt/cni/bin` on every target node. Avoid replacing the full CNI plugin bundle unless needed. + +## OCI IPAM State + +OCI private-IP API output may not show all addresses available to the node-local CNI pool. For allocation details, inspect node-side state: + +```text +/dev/shm/oci-cni/free +/dev/shm/oci-cni/used +``` + +Use a privileged diagnostic DaemonSet and `chroot /host` for repeated multi-node inspection. + +## Common Failures + +- `getting credentials: exec: executable oci failed`: refresh OCI CLI security token or set `OCI_CLI_PROFILE` and `OCI_CLI_AUTH` correctly. +- CRI-O rejects `nicolaka/netshoot:v0.13`: use `docker.io/nicolaka/netshoot:v0.13`. +- Pod sandbox creation fails with missing CNI plugin `ipvlan`: install the standard `ipvlan` CNI binary on the worker host. +- Pods do not receive the expected interfaces: check the exact NAD names, namespaces, and Multus annotations. +- Cross-pod `net1` ping fails: confirm both pods are on nodes with the secondary VNIC interface, check subnet security rules/NSGs, and verify each pod's `network-status` annotation. + +## DPDK / SR-IOV Boundary + +For DPDK workloads, keep these mechanisms separate: + +- `resources.requests` or `resources.limits` for a device-plugin resource only asks Kubernetes to allocate that resource. +- `k8s.v1.cni.cncf.io/networks` only asks Multus to attach named networks. +- A `NetworkAttachmentDefinition` defines what each named network means. +- `k8s.v1.cni.cncf.io/network-status` is the evidence that Multus actually attached the interfaces. + +If a pod requests SR-IOV or Mellanox resources but `network-status` only shows the default `eth0` network, scheduling may have succeeded while Multus attachment is still unproven or broken. First collect NAD YAML and pod `network-status` before debugging the DPDK application. + +For Mellanox/NVIDIA `mlx5` DPDK PMD, do not assume `vfio-pci` is the right driver. Some working mlx5 paths use the Linux `mlx5_core` driver plus RDMA/verbs devices. If the workload mentions `mlx5`, `/dev/infiniband`, `ibv_devices`, or DPDK mlx5 PMD libraries, read `oke-dpdk-mlx5-notes.md`. + +## Floating IP Note + +A VIP can be modeled as an OCI secondary private IP moved between matching secondary VNICs: + +```bash +oci network vnic assign-private-ip \ + --vnic-id \ + --ip-address \ + --unassign-if-already-assigned +``` + +Then add the /32 inside the active pod on the corresponding Multus interface: + +```bash +ip addr add /32 dev net1 +``` + +Reserve VIP ranges carefully. Confirm OCI IPAM allocation behavior first to avoid collisions with dynamically assigned pod addresses. diff --git a/oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py b/oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py new file mode 100755 index 0000000..2996412 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/scripts/discover-oke-multihome.py @@ -0,0 +1,358 @@ +#!/usr/bin/env python3 +"""Discover OKE cluster, node pool, and subnet data for Multus multihome setup.""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys +from typing import Any + + +CLUSTER_OCID_RE = re.compile(r"ocid1\.cluster\.[^\s\"']+") +SUBNET_OCID_RE = re.compile(r"ocid1\.subnet\.[^\s\"']+") + + +def emit_error(exit_code: int, error_code: str, message: str, remediation: str, docs_url: str = "") -> int: + print( + json.dumps( + { + "error_code": error_code, + "message": message, + "remediation": remediation, + "docs_url": docs_url, + } + ), + file=sys.stderr, + ) + return exit_code + + +class JsonArgumentParser(argparse.ArgumentParser): + def error(self, message: str) -> None: + raise SystemExit( + emit_error( + 2, + "INVALID_ARGUMENT", + message, + "Run with --help to view usage.", + ) + ) + + +def run_json(cmd: list[str], env: dict[str, str]) -> dict[str, Any]: + result = subprocess.run(cmd, env=env, text=True, capture_output=True, check=False) + if result.returncode != 0: + raise RuntimeError( + "command failed: {}\nstdout:\n{}\nstderr:\n{}".format( + " ".join(cmd), result.stdout.strip(), result.stderr.strip() + ) + ) + try: + return json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise RuntimeError(f"command did not return JSON: {' '.join(cmd)}") from exc + + +def run_text(cmd: list[str], env: dict[str, str]) -> str: + result = subprocess.run(cmd, env=env, text=True, capture_output=True, check=False) + if result.returncode != 0: + raise RuntimeError( + "command failed: {}\nstdout:\n{}\nstderr:\n{}".format( + " ".join(cmd), result.stdout.strip(), result.stderr.strip() + ) + ) + return result.stdout + + +def oci_base(args: argparse.Namespace) -> list[str]: + cmd = ["oci"] + if args.profile: + cmd.extend(["--profile", args.profile]) + if args.auth: + cmd.extend(["--auth", args.auth]) + return cmd + + +def with_region(cmd: list[str], region: str | None) -> list[str]: + if region: + return [*cmd, "--region", region] + return cmd + + +def first_ocid(value: Any, regex: re.Pattern[str]) -> str | None: + text = json.dumps(value) + match = regex.search(text) + return match.group(0) if match else None + + +def cluster_id_from_kubeconfig(context: str | None, env: dict[str, str]) -> str | None: + try: + config = run_json(["kubectl", "config", "view", "--raw", "-o", "json"], env) + except Exception: + return None + + selected = context or config.get("current-context") + if not selected: + return first_ocid(config, CLUSTER_OCID_RE) + + contexts = { + item.get("name"): item.get("context", {}) + for item in config.get("contexts", []) + if isinstance(item, dict) + } + ctx = contexts.get(selected, {}) + related_names = {selected, ctx.get("cluster"), ctx.get("user")} + related = [] + for section in ("contexts", "clusters", "users"): + for item in config.get(section, []): + if item.get("name") in related_names: + related.append(item) + return first_ocid(related or config, CLUSTER_OCID_RE) + + +def cluster_id_from_name(args: argparse.Namespace, env: dict[str, str]) -> str | None: + if not args.cluster_name or not args.compartment_id: + return None + cmd = with_region( + [ + *oci_base(args), + "ce", + "cluster", + "list", + "--compartment-id", + args.compartment_id, + "--name", + args.cluster_name, + ], + args.region, + ) + data = run_json(cmd, env).get("data", []) + if not data: + return None + if len(data) > 1: + raise RuntimeError(f"multiple clusters named {args.cluster_name!r}; pass --cluster-id") + return data[0].get("id") + + +def get_cluster(args: argparse.Namespace, cluster_id: str, env: dict[str, str]) -> dict[str, Any]: + cmd = with_region([*oci_base(args), "ce", "cluster", "get", "--cluster-id", cluster_id], args.region) + return run_json(cmd, env)["data"] + + +def list_node_pools(args: argparse.Namespace, cluster: dict[str, Any], env: dict[str, str]) -> list[dict[str, Any]]: + compartment_id = args.compartment_id or cluster.get("compartment-id") + if not compartment_id: + raise RuntimeError("missing compartment id; pass --compartment-id") + cmd = with_region( + [ + *oci_base(args), + "ce", + "node-pool", + "list", + "--compartment-id", + compartment_id, + "--cluster-id", + cluster["id"], + "--all", + ], + args.region, + ) + pools = run_json(cmd, env).get("data", []) + full = [] + for pool in pools: + pool_id = pool.get("id") + if not pool_id: + continue + get_cmd = with_region([*oci_base(args), "ce", "node-pool", "get", "--node-pool-id", pool_id], args.region) + full.append(run_json(get_cmd, env)["data"]) + return full + + +def collect_subnet_ids(cluster: dict[str, Any], pools: list[dict[str, Any]]) -> list[str]: + ids: set[str] = set() + for ocid in re.findall(SUBNET_OCID_RE, json.dumps(cluster)): + ids.add(ocid) + for pool in pools: + for ocid in re.findall(SUBNET_OCID_RE, json.dumps(pool)): + ids.add(ocid) + return sorted(ids) + + +def get_subnets(args: argparse.Namespace, subnet_ids: list[str], env: dict[str, str]) -> dict[str, dict[str, Any]]: + subnets = {} + for subnet_id in subnet_ids: + cmd = with_region([*oci_base(args), "network", "subnet", "get", "--subnet-id", subnet_id], args.region) + subnets[subnet_id] = run_json(cmd, env)["data"] + return subnets + + +def subnet_summary(subnet: dict[str, Any]) -> dict[str, Any]: + return { + "id": subnet.get("id"), + "displayName": subnet.get("display-name"), + "cidrBlock": subnet.get("cidr-block"), + "dnsLabel": subnet.get("dns-label"), + "vcnId": subnet.get("vcn-id"), + "availabilityDomain": subnet.get("availability-domain"), + "prohibitPublicIpOnVnic": subnet.get("prohibit-public-ip-on-vnic"), + } + + +def node_pool_summary(pool: dict[str, Any], subnets: dict[str, dict[str, Any]]) -> dict[str, Any]: + placement_configs = pool.get("placement-configs") or [] + secondary_vnics = pool.get("secondary-vnics") or [] + return { + "id": pool.get("id"), + "name": pool.get("name"), + "kubernetesVersion": pool.get("kubernetes-version"), + "nodeShape": pool.get("node-shape"), + "size": pool.get("size"), + "placementSubnets": [ + { + "availabilityDomain": pc.get("availability-domain"), + "subnetId": pc.get("subnet-id"), + "subnet": subnet_summary(subnets[pc["subnet-id"]]) if pc.get("subnet-id") in subnets else None, + } + for pc in placement_configs + ], + "secondaryVnics": [ + { + "displayName": item.get("display-name"), + "attachmentDisplayName": item.get("attachment-display-name"), + "subnetId": item.get("subnet-id"), + "subnet": subnet_summary(subnets[item["subnet-id"]]) if item.get("subnet-id") in subnets else None, + "ipCount": item.get("ip-count"), + "nicIndex": item.get("nic-index"), + "applicationResource": item.get("application-resource"), + "nsgIds": item.get("nsg-ids") or [], + } + for item in secondary_vnics + ], + } + + +def suggested_generator_args(pools: list[dict[str, Any]]) -> list[str]: + args = [ + "python3", + "/scripts/generate-multihome-manifest.py", + "--default-interface", + "enp1s0", + "--secondary-interface", + "enp2s0", + ] + node_names: list[str] = [] + for pool in pools: + for node in pool.get("nodes") or []: + name = node.get("name") or node.get("private-ip") + if name and name not in node_names: + node_names.append(name) + for index, node_name in enumerate(node_names[:2], start=1): + args.extend(["--pod", f"gva-multihome-{index}={node_name}"]) + return args + + +def main() -> int: + parser = JsonArgumentParser( + description="Discover OKE cluster node pools, placement subnets, and GVA secondary VNIC subnets." + ) + parser.add_argument("--cluster-id") + parser.add_argument("--cluster-name") + parser.add_argument("--context", help="Kubernetes context to inspect for cluster OCID") + parser.add_argument("--compartment-id") + parser.add_argument("--region") + parser.add_argument("--profile", default=os.environ.get("OCI_CLI_PROFILE")) + parser.add_argument("--auth", default=os.environ.get("OCI_CLI_AUTH")) + parser.add_argument("--pretty", action="store_true", help="Pretty-print JSON output") + args = parser.parse_args() + + env = os.environ.copy() + if args.profile: + env["OCI_CLI_PROFILE"] = args.profile + if args.auth: + env["OCI_CLI_AUTH"] = args.auth + + if shutil.which("oci") is None: + return emit_error( + 1, + "OCI_CLI_NOT_FOUND", + "OCI CLI is not installed or not on PATH.", + "Install OCI CLI, authenticate to the target tenancy, and rerun discovery.", + ) + + needs_kubeconfig_lookup = not args.cluster_id and not (args.cluster_name and args.compartment_id) + if needs_kubeconfig_lookup and shutil.which("kubectl") is None: + return emit_error( + 1, + "KUBECTL_NOT_FOUND", + "kubectl is not installed or not on PATH, and discovery needs kubeconfig to resolve the cluster OCID.", + "Install kubectl, pass --cluster-id, or pass --cluster-name with --compartment-id.", + ) + + cluster_id = ( + args.cluster_id + or cluster_id_from_kubeconfig(args.context, env) + or cluster_id_from_name(args, env) + ) + if not cluster_id: + return emit_error( + 1, + "CLUSTER_OCID_NOT_RESOLVED", + "Could not resolve cluster OCID.", + "Pass --cluster-id, or pass --cluster-name with --compartment-id, or use a kubeconfig context that contains --cluster-id.", + ) + + cluster = get_cluster(args, cluster_id, env) + if not args.region: + args.region = cluster.get("region") + + pools = list_node_pools(args, cluster, env) + subnet_ids = collect_subnet_ids(cluster, pools) + subnets = get_subnets(args, subnet_ids, env) + output = { + "cluster": { + "id": cluster.get("id"), + "name": cluster.get("name"), + "compartmentId": cluster.get("compartment-id"), + "kubernetesVersion": cluster.get("kubernetes-version"), + "vcnId": cluster.get("vcn-id"), + "type": cluster.get("type"), + "endpointConfig": cluster.get("endpoint-config"), + "options": cluster.get("options"), + }, + "subnets": [subnet_summary(subnets[subnet_id]) for subnet_id in sorted(subnets)], + "nodePools": [node_pool_summary(pool, subnets) for pool in pools], + "suggestedGeneratorArgs": suggested_generator_args(pools), + } + indent = 2 if args.pretty else None + print(json.dumps(output, indent=indent, sort_keys=bool(indent))) + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except BrokenPipeError: + raise SystemExit(1) + except RuntimeError as exc: + raise SystemExit( + emit_error( + 1, + "DISCOVERY_COMMAND_FAILED", + str(exc), + "Verify OCI CLI authentication, region, compartment, and kubeconfig context, then rerun discovery.", + ) + ) + except Exception as exc: + raise SystemExit( + emit_error( + 2, + "UNEXPECTED_ERROR", + str(exc), + "Inspect the input arguments and environment, then rerun discovery.", + ) + ) diff --git a/oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py b/oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py new file mode 100755 index 0000000..c628007 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/scripts/generate-multihome-manifest.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +"""Generate OKE GVA Multus multihome test manifests.""" + +from __future__ import annotations + +import argparse +import json +import sys +from typing import Iterable + + +def emit_error(exit_code: int, error_code: str, message: str, remediation: str, docs_url: str = "") -> int: + print( + json.dumps( + { + "error_code": error_code, + "message": message, + "remediation": remediation, + "docs_url": docs_url, + } + ), + file=sys.stderr, + ) + return exit_code + + +class JsonArgumentParser(argparse.ArgumentParser): + def error(self, message: str) -> None: + raise SystemExit( + emit_error( + 2, + "INVALID_ARGUMENT", + message, + "Run with --help to view usage.", + ) + ) + + +def q(value: str) -> str: + return json.dumps(value) + + +def block_json(value: dict) -> str: + text = json.dumps(value, indent=2) + return "\n".join(f" {line}" for line in text.splitlines()) + + +def parse_pod(value: str) -> tuple[str, str]: + if "=" not in value: + raise argparse.ArgumentTypeError("pod must use NAME=NODE_NAME") + name, node = value.split("=", 1) + if not name or not node: + raise argparse.ArgumentTypeError("pod must use NAME=NODE_NAME") + return name, node + + +def emit(lines: Iterable[str]) -> None: + sys.stdout.write("\n".join(lines)) + sys.stdout.write("\n") + + +def main() -> int: + parser = JsonArgumentParser( + description="Generate Multus NetworkAttachmentDefinitions and pinned test pods for OKE GVA multihome." + ) + parser.add_argument("--namespace", default="gva-multihome-test") + parser.add_argument("--default-nad-namespace", default="kube-system") + parser.add_argument("--default-nad-name", default="gva-default-network") + parser.add_argument("--secondary-nad-name", default="gva-secondary-network") + parser.add_argument("--default-interface", default="enp1s0") + parser.add_argument("--secondary-interface", default="enp2s0") + parser.add_argument("--image", default="docker.io/nicolaka/netshoot:v0.13") + parser.add_argument("--sleep", default="3600") + parser.add_argument( + "--pod", + action="append", + type=parse_pod, + required=True, + metavar="NAME=NODE_NAME", + help="Add a pinned test pod. Repeat for multiple pods.", + ) + args = parser.parse_args() + + default_config = { + "name": args.default_nad_name, + "cniVersion": "0.3.1", + "plugins": [ + { + "cniVersion": "0.3.1", + "type": "oci-ipvlan", + "mode": "l2", + "ipam": { + "type": "oci-ipam", + "deviceSelector": {"interfaceName": args.default_interface}, + }, + }, + { + "cniVersion": "0.3.1", + "type": "oci-ptp", + "containerInterface": "ptp-veth0", + "mtu": 9000, + }, + ], + } + + secondary_config = { + "cniVersion": "0.3.1", + "plugins": [ + { + "type": "ipvlan", + "mode": "l2", + "master": args.secondary_interface, + "ipam": { + "type": "oci-ipam", + "deviceSelector": {"interfaceName": args.secondary_interface}, + }, + } + ], + } + + lines: list[str] = [ + "apiVersion: v1", + "kind: Namespace", + "metadata:", + f" name: {args.namespace}", + "---", + "apiVersion: k8s.cni.cncf.io/v1", + "kind: NetworkAttachmentDefinition", + "metadata:", + f" name: {args.default_nad_name}", + f" namespace: {args.default_nad_namespace}", + "spec:", + " config: |", + block_json(default_config), + "---", + "apiVersion: k8s.cni.cncf.io/v1", + "kind: NetworkAttachmentDefinition", + "metadata:", + f" name: {args.secondary_nad_name}", + f" namespace: {args.namespace}", + "spec:", + " config: |", + block_json(secondary_config), + ] + + for pod_name, node_name in args.pod: + lines.extend( + [ + "---", + "apiVersion: v1", + "kind: Pod", + "metadata:", + f" name: {pod_name}", + f" namespace: {args.namespace}", + " labels:", + " app: gva-multihome", + " annotations:", + f" v1.multus-cni.io/default-network: {args.default_nad_namespace}/{args.default_nad_name}", + f" k8s.v1.cni.cncf.io/networks: {args.namespace}/{args.secondary_nad_name}", + "spec:", + f" nodeName: {q(node_name)}", + " containers:", + " - name: netshoot", + f" image: {q(args.image)}", + " command:", + " - sleep", + f" - {q(args.sleep)}", + ] + ) + + emit(lines) + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except BrokenPipeError: + raise SystemExit(1) + except Exception as exc: + raise SystemExit( + emit_error( + 2, + "UNEXPECTED_ERROR", + str(exc), + "Inspect the input arguments and rerun the manifest generator.", + ) + ) diff --git a/oci/oke/skills/oke-multihome-deployer/validation-report-template.md b/oci/oke/skills/oke-multihome-deployer/validation-report-template.md new file mode 100644 index 0000000..e143a18 --- /dev/null +++ b/oci/oke/skills/oke-multihome-deployer/validation-report-template.md @@ -0,0 +1,43 @@ +# OKE Multihome Validation Report Template + +Use this structure after generating or applying Multus/GVA validation manifests. + +## Summary + +State whether multihome validation passed, failed, or is incomplete. + +## Environment + +| Field | Value | +| --- | --- | +| Cluster | `` | +| Region | `` | +| Namespace | `` | +| Node pool | `` | +| Default interface | `` | +| Secondary interface | `` | + +## Generated Artifacts + +| Artifact | Purpose | +| --- | --- | +| Namespace | Test namespace for validation pods | +| Default NAD | Multus default network backed by OCI CNI/IPAM | +| Secondary NAD | Secondary `net1` network backed by OCI IPAM | +| Test pods | Pinned netshoot pods for interface and ping checks | + +## Validation Checks + +| Check | Command | Result | +| --- | --- | --- | +| Multus daemonset | `kubectl -n kube-system get pods -l name=multus -o wide` | `` | +| Pod placement | `kubectl -n get pods -o wide` | `` | +| Network status | `kubectl -n get pod -o jsonpath=...` | `` | +| Interfaces | `kubectl -n exec -- ip -br addr` | `` | +| Peer ping | `kubectl -n exec -- ping -c 3 ` | `` | + +## Remediation + +List the first failed check and the most likely fix. Keep Multus/NAD failures, +OCI CNI/IPAM allocation failures, node-interface issues, and DPDK/SR-IOV driver +issues as separate facts. diff --git a/oci/oke/skills/oke-troubleshooter/SKILL.md b/oci/oke/skills/oke-troubleshooter/SKILL.md new file mode 100644 index 0000000..84c3ddd --- /dev/null +++ b/oci/oke/skills/oke-troubleshooter/SKILL.md @@ -0,0 +1,386 @@ +--- +name: oke-troubleshooter +description: Use this skill when the user wants to diagnose or root-cause issues with an OCI Kubernetes Engine cluster or workload. Trigger phrases include "pods pending", "troubleshoot OKE", "service has no IP", "cluster unhealthy", DPDK/SR-IOV mlx5 pod failures, Multus network-status issues, or broad incident RCA across Kubernetes and OCI resources. Do not use it for greenfield Terraform generation, GVA node-pool creation/update, or routine Multus manifest deployment when no incident is being investigated; route those to oke-cluster-generator, oke-gva-deployer, or oke-multihome-deployer. +--- + +You are an experienced Site Reliability Engineer for OCI Kubernetes Engine. Guide the user through an evidence-driven investigation that spans Kubernetes signals and OCI infrastructure. + +Supporting references (load on demand): +- `symptom-triage.md` — initial mapping of symptom → diagnostic domains. +- `evidence-collectors.md` — command recipes for each domain. +- `final-report-template.md` — standard final report structure. +- `../../shared/oci-resource-map.md` — K8s-to-OCI mapping commands. +- `../oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md` — DPDK, Multus, Mellanox mlx5, `vfio-pci`, hugepage, and RDMA/verbs diagnostic rules. + +Optional accelerators (use only when the runtime supports delegation; never block on them): +- `../../agents/oke-evidence-collector.md` — agent for command execution and evidence normalization. +- `../../agents/oke-hypothesis-analyst.md` — agent for scoring hypotheses. +- `../../agents/oke-lb-log-collector.md` — agent for LB OCID resolution, logging-status checks, and LB log signal extraction. + +Scripts rely on the global error contract: exit 0 success, exit 1 expected issues, exit 2 unexpected. Emit JSON errors on stderr in failure scenarios. + +Helper scripts: +- `../../scripts/oke-discover.sh` — resolve cluster OCID from kubeconfig and fetch compartment/region via OCI CLI +- `../../scripts/oke-addon-health.sh` — collect kube-system add-on health signals +- `../../scripts/oke-pod-network-check.sh` — collect OCI CNI/IPAM, Multus, pod sandbox, and NAD signals +- `../../scripts/oke-autoscaler-check.sh` — collect Pending pod, cluster-autoscaler, and node-pool scaling signals +- `../../scripts/oke-dns-check.sh` — collect CoreDNS, Service, EndpointSlice, and pod DNS lookup signals +- `../../scripts/oke-ingress-check.sh` — collect OCI Native Ingress controller and Ingress object signals +- `../../scripts/oke-private-endpoint-check.sh` — collect private endpoint, kubeconfig, and API reachability signals +- `../../scripts/oke-ocir-image-pull-check.sh` — collect OCIR image pull, secret, service account, and repository signals +- `../../scripts/oke-workload-identity-check.sh` — collect service account, pod log, dynamic group, and IAM policy signals +- `../../scripts/oke-incident-timeline.sh` — merge Kubernetes events, rollout history, object descriptions, and OCI alarms into a timeline +- `../../scripts/oke-object-correlator.sh` — build a Kubernetes-to-OCI object graph for pods, nodes, services, ingress, PVCs, load balancers, instances, VNICs, volumes, and node pools + +--- + +## Execution Mode + +- Default to **local execution in the parent skill**. +- Use the optional agents above only as accelerators when the current runtime clearly supports agent delegation. +- If agents are unavailable, disabled, or return malformed output, continue locally with the same command list and payload shape. Do not stop the investigation solely because delegation is unavailable. +- Normalize local evidence to the same JSON shape documented in `evidence-collectors.md`. + +## Phase 0 — Input & Preflight +1. **Parse Arguments** + - `$ARGUMENTS` holds an optional symptom string. If empty, ask the user for a concise description (e.g., `"pods stuck Pending in prod namespace"`). + - Extract namespace hints (`-n`, `namespace:`) and resource names when present. +2. **Auto-Discover Cluster Context** + - Ask for **cluster name** if not provided. + - First list kubeconfig contexts to identify managed clusters and current context: + ```bash + kubectl config get-contexts + ``` + - Use this output to suggest likely cluster/context names before prompting for manual input. + - Derive `active_cluster_region` from the active kube context (`kubectl config view --minify`, user exec args, or cluster metadata tied to the current context) and treat it as authoritative. + - Resolve **cluster OCID** from `~/.kube/config` when possible. + - Use tenancy defaults from `~/.oci/config` only for auth/profile hints, not for region selection. + - Run: + ```bash + bash ../../scripts/oke-discover.sh --cluster [--region ] [--profile ] [--timeout ] [--kubeconfig ] [--deployment ] + ``` + - Always pass `--region ` to discovery and all OCI CLI calls in later phases. + - Never use implicit OCI CLI region or fallback/default region. + - Use the JSON output to auto-populate: `cluster_ocid`, `compartment_ocid`, `region`, `kubernetes_version`, and deployment namespace when available. + - If discovery reports a different region than `active_cluster_region`, flag the mismatch, keep `active_cluster_region` for all subsequent commands, and ask for confirmation only if the mismatch prevents resource resolution. + - Prompt only for fields that remain missing after discovery. + - **Single-cluster scope enforcement**: + - Treat the user-provided cluster (`name` or `ocid`) as the only in-scope target for the entire session. + - Do not run baseline checks, inventory commands, or evidence collection against any other cluster. + - If current `kubectl` context does not match the discovered cluster identity, stop and ask the user to switch context or provide the correct kubeconfig before continuing. + - If OCI lookup must be retried, retry only for the same specified cluster (for example with corrected `--region`/`--profile`), never by probing other clusters. +3. **Confirm Context** + - Ask only for missing essentials after discovery: namespace, target Deployment/Service name, desired time window (`15m`, `1h`, default `1h`), impact level (prod/non-prod). +4. **Tool Availability Checks** + - Run `kubectl version --client` and `oci --version`. + - Record `KUBECTL_AVAILABLE`/`OCI_AVAILABLE` booleans. If a CLI is missing, inform the user that evidence will be partial and continue with available tools. +5. **Session State** + - Initialize state structure: + ```json + { + "symptom": "...", + "namespace": "...", + "time_window": "1h", + "cluster_ocid": "...", + "compartment_ocid": "...", + "region": "...", + "domains": [], + "dependency_map": { + "entrypoint": "", + "hops": [], + "critical_path": [], + "latency_budget_ms": {} + }, + "fallbacks": {"kubectl": false, "oci": false}, + "evidence": [], + "node_doctor": { + "enabled": false, + "execution_mode": "ask_then_execute", + "image": "", + "targets": [], + "results": [] + } + } + ``` + +--- + +## Phase 1 — Symptom Triage +1. Load `symptom-triage.md` and identify candidate domains matching the symptom keywords (including application performance cases such as “deployment nginx is slow”). +2. Present the suggested domains to the user with brief rationales. Allow them to: + - Confirm the list. + - Add or remove domains. + - Provide additional focus (specific pod, service, node pool, PVC, IAM entity). +3. For application latency symptoms, model dependency context before evidence collection: + - Capture request entrypoint (Ingress/API/Job), target deployment, and downstream services (internal and external). + - Mark critical-path dependencies vs optional/background calls. + - Capture baseline latency and per-hop budget when known. +4. Capture clarifying answers (from the table's questions) and store them in session state (e.g., `POD_NAME`, `SERVICE_NAME`, `DEPLOYMENT_NAME`, `LABEL_SELECTOR`, `BASELINE_LATENCY`, `DEPENDENCY_MAP`). + +--- + +## Phase 2 — Dependency Path Modeling +1. Build a dependency map before running domain collectors when latency/throughput symptoms are present. +2. Dependency map structure: + ```json + { + "entrypoint": "ingress/payments", + "hops": [ + {"from": "ingress/payments", "to": "deployment/payments-api", "protocol": "HTTP"}, + {"from": "deployment/payments-api", "to": "svc/orders", "protocol": "gRPC"}, + {"from": "deployment/payments-api", "to": "svc/redis", "protocol": "TCP"} + ], + "critical_path": ["ingress/payments->deployment/payments-api", "deployment/payments-api->svc/orders"], + "latency_budget_ms": { + "end_to_end_p99": 500, + "ingress/payments->deployment/payments-api": 120, + "deployment/payments-api->svc/orders": 220 + } + } + ``` +3. If dependency data is incomplete, continue with a partial map and explicitly mark confidence reduction in later phases. + +--- + +## Phase 3 — Evidence Collection +1. Build the OCI object correlation graph before domain-specific collectors when enough selectors are known. + - Run the correlator with all discovered selectors, even if only one target object is known: + ```bash + bash ../../scripts/oke-object-correlator.sh \ + --namespace \ + --cluster-id \ + --compartment-id \ + --region \ + [--pod ] \ + [--deployment ] \ + [--service ] \ + [--ingress ] \ + [--pvc ] \ + [--node ] + ``` + - Treat the output as evidence with fields: `domain`, `graph.kubernetes`, `graph.oci`, `graph.edges`, `findings`, `anomalies`, `raw_snippets`, and `fallback_used`. + - Use the graph to narrow follow-on checks. Examples: + - If a Service maps to an OCI Load Balancer with unhealthy backend health, focus on backend set, node subnet, NSG, and endpoint readiness. + - If a Pod maps to a Node and Compute instance, prefer that node pool, AD, subnet, and VNIC path for node/network checks. + - If a PVC maps to a Block Volume, compare volume AD and attachment state before blaming CSI. + - If `fallback_used=true`, continue with domain-specific collectors and call out which object links could not be resolved. +2. For each selected domain: + - Look up required commands in `evidence-collectors.md`. + - Build command batches with placeholders filled (namespace, resource names, compartment OCID, time window, and dependency hop identifiers when present). + - **Auto-run read-only evidence commands without prompting** when tools are available. + - Only ask for confirmation before **potentially disruptive** actions (restarts, scaling, drains). + - Example command item: + ```json + { + "cmd": "kubectl describe pod trainer-0 -n ml-team", + "purpose": "Inspect scheduling events" + } + ``` + - For Networking/LB investigations, prefer the dedicated LB collector when delegation is available. Otherwise run the LB commands from `evidence-collectors.md` locally and normalize the same output fields in the parent skill. + - Use payload fields: `namespace`, `service`, `region`, `compartment_ocid`, `time_window`, and `enable_logging_mode`. + - Enablement interaction: + - Ask user only when collector reports `logging_status=disabled|unknown`: + - `No (report only)` + - `Yes (print command only)` + - `Yes (run now)` + - Map answer to `enable_logging_mode` and rerun collector if needed. + - Merge collector output into session evidence: + - `lb_ocid`, `logging_status`, `logging_status_source`, `log_findings`, `anomalies`, `fallback_used` + - If collector reports fallback/timeouts, continue with Kubernetes networking evidence and call out OCI visibility gap in the report. + - For OKE-specific domains, prefer the dedicated helper script before generic command batches: + - OKE Add-ons Health: + ```bash + bash ../../scripts/oke-addon-health.sh --namespace kube-system + ``` + - Pod Networking / OCI CNI / IPAM: + ```bash + bash ../../scripts/oke-pod-network-check.sh --namespace [--pod ] [--selector ] + ``` + - Cluster Autoscaler / Node Pool Scaling: + ```bash + bash ../../scripts/oke-autoscaler-check.sh --namespace [--deployment ] --cluster-id --compartment-id --region + ``` + - DNS / Service Discovery: + ```bash + bash ../../scripts/oke-dns-check.sh --namespace [--service ] [--pod ] [--lookup ] + ``` + - Ingress / OCI Native Ingress: + ```bash + bash ../../scripts/oke-ingress-check.sh --namespace --ingress [--region ] + ``` + - Private Cluster / API Endpoint Connectivity: + ```bash + bash ../../scripts/oke-private-endpoint-check.sh --cluster-id --region [--compartment-id ] + ``` + - OCIR / Image Pull: + ```bash + bash ../../scripts/oke-ocir-image-pull-check.sh --namespace --pod [--image ] [--compartment-id ] [--region ] + ``` + - Workload Identity / OCI API From Pods: + ```bash + bash ../../scripts/oke-workload-identity-check.sh --namespace --serviceaccount [--pod ] [--tenancy-id ] + ``` + - Incident Timeline: + ```bash + bash ../../scripts/oke-incident-timeline.sh --namespace [--pod ] [--deployment ] [--service ] [--compartment-id ] [--region ] + ``` + - Treat helper JSON output as evidence with fields: `domain`, `findings`, `anomalies`, `raw_snippets`, and `fallback_used`. + - For Node Health investigations, include optional Node Doctor diagnostics: + - Trigger when Node Health is selected and there are node readiness/kubelet/runtime signals, or when user explicitly asks. + - Scope starts with one candidate node first, then ask whether to continue to additional nodes. + - Default debug image to `docker.io/library/ubuntu` each run (`kubectl debug ... --image=`), and allow user override. Keep the selected image in session for additional nodes unless user changes it. + - Before execution, present the exact sequence and ask explicit confirmation per node: + 1) `bash ../../scripts/node-doctor-run.sh --node --image ` + 2) (script executes `kubectl debug` + `chroot /host` + `sudo /usr/local/bin/node-doctor.sh --check`) + - Options per node: + - `Execute now` + - `Print commands only` + - `Skip` + - Treat this flow as potentially disruptive/privileged. Never auto-run without confirmation. + - Capture normalized output fields in evidence: + - `node_doctor_attempted`, `node_doctor_executed`, `node_doctor_node`, `node_doctor_image` + - `node_doctor_result` (`pass` | `fail` | `unknown`) and `node_doctor_command_rc` + - `node_doctor_findings`, `node_doctor_raw_snippet`, `node_doctor_fallback_reason` + - If the helper script reports failure (debug blocked, image pull, chroot/sudo/script missing), set fallback details and continue Node Health evidence collection. +3. Assemble collector input payload: + ```json + { + "symptom": "...", + "domains": ["Pod Scheduling"], + "namespace": "...", + "time_window": "...", + "selectors": {"pod": "...", "service": "...", "deployment": "...", "label": "..."}, + "dependency_map": { + "entrypoint": "...", + "hops": [], + "critical_path": [], + "latency_budget_ms": {} + }, + "object_graph": {...}, + "fallbacks": {"kubectl": false, "oci": true}, + "compartment_ocid": "..." + } + ``` +4. Execute the prepared command list. + - If delegation is available, you may hand the payload to `oke-evidence-collector`. + - Otherwise run the commands locally in the parent skill and normalize them to the documented evidence JSON shape (`domain`, `findings`, `raw_snippets`, `anomalies`, `fallback_used`). + - If delegated collection fails or returns malformed output, fall back to local execution immediately. +5. After all domains processed, summarize key findings to the user before analysis. Note any `fallback_used` signals or missing data. + +--- + +## Phase 4 — Hypothesis Ranking +1. Construct analyst payload containing: + ```json + { + "symptom": "...", + "domains": [...], + "dependency_map": {...}, + "object_graph": {...}, + "evidence": [...], + "fallbacks": {"kubectl": false, "oci": true} + } + ``` +2. Rank hypotheses. + - If delegation is available, you may use `oke-hypothesis-analyst`. + - Otherwise rank hypotheses locally using this rubric: + - `9-10`: direct, converging evidence for one root cause + - `6-8`: strong multi-signal correlation with limited ambiguity + - `3-5`: plausible but missing a decisive signal + - `1-2`: weak signal or mostly evidence-gap guidance + - Ensure each hypothesis includes score, bottleneck hop attribution when relevant, evidence bullets, remediation commands, and prevention guidance. + - Prefer hypotheses that are supported by explicit graph edges across Kubernetes and OCI resources over hypotheses supported only by isolated symptoms. + - If delegated analysis fails or returns malformed output, fall back to local ranking immediately. +3. Validate that evidence quotes reference actual snippets collected. If not, request clarification from the analyst or adjust evidence payload. + +--- + +## Phase 5 — Report & Next Steps +1. Load `final-report-template.md` and present a structured report using that shape: + - Table of top hypotheses with scores. + - Highlight confidence level (e.g., `High`, `Medium`, `Low` based on score thresholds). + - For latency incidents, include a hop-by-hop budget table: `hop`, `expected_p99_ms`, `observed_p99_ms`, `delta_ms`, `confidence`. + - Remediation commands rendered in fenced code blocks, prefixed with comments where necessary. + - Prevention recommendations as concise bullet points. +2. Call out any limitations: missing tooling, commands that failed, domains not yet explored, and missing dependency telemetry. +3. Offer next actions: + - Rerun for another namespace/resource. + - Deep-dive into IAM, capacity, or service-limit evidence when the collected data points there. + - Export findings to a file (future enhancement). +4. Thank the user and remind them to redact sensitive data if sharing the report. + +--- + +## Error Handling +- Missing CLI: Continue with available evidence, set fallback flags, warn the user. +- Permission denied or forbidden: include remediation (e.g., "ensure tenancy OCID has access to compartment"). +- Delegation unavailable or subagent failure: continue locally; do not abort the incident flow. +- Unexpected script errors: emit JSON error per contract and stop the current phase while keeping collected data. + +## Security & Logging +- Do not echo secret values or service account tokens. Redact with `***`. +- Reference the audit logging guidance: avoid storing credentials in outputs or state. +- Encourage the user to review any local runtime audit log if their agent environment records one. + +--- + +## Invocation Examples +- `Use the OKE troubleshooter for pods stuck Pending in prod namespace` +- `Use the OKE troubleshooter for lb service has no IP us-phoenix-1` +- `Use the OKE troubleshooter for cluster api timing out` +- `Use the OKE troubleshooter for customer is indicating poor performance for deployment` +- `Use the OKE troubleshooter for CoreDNS timeouts in prod` +- `Use the OKE troubleshooter for cluster autoscaler is not adding nodes` +- `Use the OKE troubleshooter for pods fail sandbox creation with OCI CNI IPAM errors` +- `Use the OKE troubleshooter for OCIR ImagePullBackOff unauthorized` +- `/oke-troubleshooter "workload identity pod gets NotAuthorized"` +- `/oke-troubleshooter "private OKE API endpoint unreachable"` +- `/oke-troubleshooter "OCI native ingress TLS backend errors"` + +## Latency Walkthrough (Dependency-Aware) +Use this pattern when the incident is "deployment is slow" and the deployment depends on other services. + +1. **Input Example** + - Symptom: `"payments API p99 jumped from 350ms to 1.8s"` + - Namespace: `prod` + - Deployment: `payments-api` + - Time window: `1h` +2. **Dependency Map Example** + ```json + { + "entrypoint": "ingress/payments", + "hops": [ + {"from": "ingress/payments", "to": "deployment/payments-api", "protocol": "HTTP"}, + {"from": "deployment/payments-api", "to": "svc/orders", "protocol": "gRPC"}, + {"from": "deployment/payments-api", "to": "svc/redis", "protocol": "TCP"} + ], + "critical_path": [ + "ingress/payments->deployment/payments-api", + "deployment/payments-api->svc/orders" + ], + "latency_budget_ms": { + "end_to_end_p99": 500, + "ingress/payments->deployment/payments-api": 120, + "deployment/payments-api->svc/orders": 220, + "deployment/payments-api->svc/redis": 80 + } + } + ``` +3. **Expected Evidence Interpretation** + - Compare observed hop p99 to budget and compute delta. + - Identify the largest over-budget hop on critical path first. + - Validate with both client-side and server-side evidence when possible. +4. **Expected Report Snippet** + - Hypothesis: `"Orders dependency latency spike is primary bottleneck"` + - Confidence: `High` when both sides of hop agree. + - Budget table: + + | Hop | Expected p99 (ms) | Observed p99 (ms) | Delta (ms) | Confidence | + |-----|-------------------|-------------------|------------|------------| + | ingress/payments->payments-api | 120 | 140 | +20 | Medium | + | payments-api->orders | 220 | 980 | +760 | High | + | payments-api->redis | 80 | 95 | +15 | Medium | + + - Remediation should target `payments-api->orders` first, then re-measure end-to-end p99. + +The skill should deliver actionable insight even when only partial data is available. diff --git a/oci/oke/skills/oke-troubleshooter/USAGE.md b/oci/oke/skills/oke-troubleshooter/USAGE.md new file mode 100644 index 0000000..de86d0a --- /dev/null +++ b/oci/oke/skills/oke-troubleshooter/USAGE.md @@ -0,0 +1,186 @@ +# OKE Troubleshooter Usage Guide + +Use the `oke-troubleshooter` skill when you need to diagnose an OCI Kubernetes Engine incident from both the Kubernetes and OCI sides. The skill is designed to collect read-only evidence, correlate related objects, rank likely root causes, and produce remediation steps. + +## Before You Start + +Verify your local tools point at the right cluster: + +```bash +kubectl config current-context +kubectl version --client +oci --version +``` + +For OCI-layer evidence, the OCI CLI profile must be authenticated and authorized to read OKE, Compute, Network, Load Balancer, Block Volume, IAM, Logging, Monitoring, and Limits data in the target compartment. + +If your kubeconfig uses OCI security-token auth and `kubectl` fails with an OCI exec error, renew the OCI session first. This repo does not assume a default cluster or default region during troubleshooting. + +## Basic Skill Usage + +Start with the symptom in plain English: + +```bash +Use the OKE troubleshooter for pods stuck Pending in prod namespace +Use the OKE troubleshooter for service web has no external IP +Use the OKE troubleshooter for private OKE API endpoint unreachable +Use the OKE troubleshooter for OCIR ImagePullBackOff unauthorized +Use the OKE troubleshooter for workload identity pod gets NotAuthorized +``` + +The skill will ask for missing context such as: + +- Cluster name or OCID. +- Region. +- Namespace. +- Pod, Deployment, Service, Ingress, PVC, or Node name. +- Time window, such as `15m` or `1h`. +- Impact level, such as production or non-production. + +## How To Pick The Right Object + +Start with the object where the symptom appears. Add related objects when you know them. + +| Symptom | Provide | +|---|---| +| Pod Pending, CrashLoopBackOff, ImagePullBackOff | Pod and namespace | +| Deployment is slow or unavailable | Deployment, namespace, and optionally a pod | +| Service has no IP or LB is unhealthy | Service and namespace; add one backend pod if known | +| Ingress route or TLS is broken | Ingress and namespace; add backend Service if known | +| PVC is Pending or volume attach fails | PVC and namespace; add the consuming pod if known | +| Node is NotReady or unstable | Node; add one affected pod if known | +| Multus, GVA, or pod sandbox failure | Pod, namespace, and any related NetworkAttachmentDefinition details | + +## What The Skill Does + +1. **Preflight and discovery** + - Confirms the active kube context. + - Resolves the OKE cluster OCID from kubeconfig when possible. + - Uses the explicit region for all OCI CLI calls. + - Enforces single-cluster scope. + +2. **Symptom triage** + - Maps the symptom to diagnostic domains such as scheduling, node health, DNS, CNI/IPAM, autoscaler, Load Balancer, Ingress, OCIR, Workload Identity, storage, or control plane. + +3. **OCI object correlation** + - Builds a read-only graph connecting Kubernetes objects to OCI resources. + - Example graph paths: + +```text +pod/default/web-0 -> node/node-a -> instance/ocid1.instance... +service/default/web -> loadbalancer/ocid1.loadbalancer... +pvc/default/data -> pv/pvc-123 -> volume/ocid1.volume... +``` + +4. **Evidence collection** + - Runs focused read-only `kubectl` and `oci` commands. + - Uses helper scripts for OKE add-ons, pod networking, autoscaler, DNS, Ingress, private endpoint, OCIR image pulls, Workload Identity, incident timeline, and object correlation. + +5. **Hypothesis ranking** + - Ranks likely root causes with evidence. + - Prefers causes supported by explicit Kubernetes-to-OCI graph edges. + +6. **Report** + - Summarizes findings, anomalies, confidence, remediation commands, and evidence gaps. + +## Direct Object Graph Usage + +Use the object correlator directly when you want a graph without a full troubleshooting session. + +```bash +./scripts/oke-object-correlator.sh \ + --namespace \ + --cluster-id \ + --compartment-id \ + --region \ + --pod \ + --service +``` + +For a storage issue: + +```bash +./scripts/oke-object-correlator.sh \ + --namespace \ + --cluster-id \ + --compartment-id \ + --region \ + --pvc \ + --pod +``` + +The output includes: + +- `graph.kubernetes`: Kubernetes objects found. +- `graph.oci`: OCI resources found. +- `graph.edges`: relationships between objects. +- `findings`: useful correlations. +- `anomalies`: broken or suspicious links. +- `fallback_used`: whether any command failed or had partial coverage. + +## Reading The Graph + +A healthy Service-to-LB path might look like: + +```text +Service/neuvector-service-webui + -> OCI Load Balancer ACTIVE + -> subnet ocid1.subnet... + +Pod/neuvector-manager + -> Node 10.0.5.128 + -> OCI Compute instance +``` + +Use the graph to decide where to investigate next: + +- If the Service has no endpoints, focus on selectors, pod readiness, and EndpointSlices. +- If the Load Balancer is not healthy, focus on backend health, node ports, NSGs, and subnets. +- If a pod maps to a NotReady node, focus on node health, instance lifecycle, kubelet, and CNI. +- If a PVC maps to a volume with an attachment issue, focus on CSI and Block Volume attachment state. + +## Safety Rules + +- The default troubleshooting flow is read-only. +- The skill may auto-run read-only evidence commands. +- It must ask before potentially disruptive actions such as restarts, scaling, drains, or privileged node diagnostics. +- Node Doctor diagnostics use `kubectl debug` and require explicit confirmation. + +## Common Examples + +Pod scheduling: + +```bash +Use the OKE troubleshooter for pods are Pending in namespace payments +``` + +Service or Load Balancer: + +```bash +Use the OKE troubleshooter for service neuvector-service-webui load balancer is unhealthy +``` + +DNS: + +```bash +Use the OKE troubleshooter for pods cannot resolve web.default.svc.cluster.local +``` + +Multus or GVA: + +```bash +Use the OKE troubleshooter for FailedCreatePodSandBox with Multus and ipvlan +``` + +Workload Identity: + +```bash +Use the OKE troubleshooter for pod gets NotAuthorized when calling OCI API +``` + +## Troubleshooting The Troubleshooter + +- If `kubectl` fails with an OCI exec error, renew your OCI security-token session. +- If OCI evidence is missing, verify the profile, region, compartment permissions, and `--auth security_token` when applicable. +- If the object graph is sparse, rerun with more selectors, such as both `--service` and `--pod`. +- If only Kubernetes evidence is available, the skill can still rank Kubernetes-side causes but should mark OCI visibility as partial. diff --git a/oci/oke/skills/oke-troubleshooter/evidence-collectors.md b/oci/oke/skills/oke-troubleshooter/evidence-collectors.md new file mode 100644 index 0000000..c666eb3 --- /dev/null +++ b/oci/oke/skills/oke-troubleshooter/evidence-collectors.md @@ -0,0 +1,346 @@ +# `/oke-troubleshooter` — Evidence Collection Recipes + +For each diagnostic domain, gather the following evidence. Prefer JSON output (`-o json`) when available. Summaries returned to the parent skill should follow the structure: + +```json +{ + "domain": "", + "findings": ["Short bullet summary"], + "raw_snippets": ["Trimmed command output"], + "anomalies": ["Detected warnings/errors"], + "fallback_used": false +} +``` + +When a command fails, set `fallback_used` to `true`, capture stderr (sanitized), and continue with other evidence. + +## Pod Scheduling +- **Kubernetes** + - `kubectl get pods -n -o wide` + - `kubectl describe pod -n ` + - `kubectl get events -n --field-selector involvedObject.name= --sort-by=.lastTimestamp` +- **OCI** + - `oci ce node-pool list --compartment-id ` (ensure node pool capacity) + - `oci limits resource-availability get --compartment-id --service-name compute --limit-name --availability-domain ` + - TODO(live validation): resolve the exact compute `limit-name` for the target shape from `oci limits value list` before treating remaining capacity as conclusive. +- **Normalization tips**: Highlight scheduling failure reasons (`0/3 nodes available`, taints), summarize resource requests vs. node capacity, include current node pool size. + +## Pod Runtime +- **Kubernetes** + - `kubectl describe pod -n ` + - `kubectl logs -n --previous` (when restart count > 0) + - `kubectl get events -n --field-selector type=Warning` +- **OCI** + - `oci logging search --time-start --time-end --search-query "search where podName = ''"` +- **Normalization tips**: Capture container state (`Waiting`, `CrashLoopBackOff`), include last log lines causing failure, flag missing secrets or configmaps. + +## Node Health +- **Kubernetes** + - `kubectl get nodes -o wide` + - `kubectl describe node ` + - `kubectl top node ` (requires metrics server) +- **OCI** + - `oci ce node-pool get --node-pool-id ` + - `oci compute instance get --instance-id ` + - `oci health-check probe-result get --probe-configuration-id ` (if using health checks) +- **Normalization tips**: Surface conditions not `True`, kubelet versions, OCI lifecycle state, recent maintenance events. +- **Advanced: Node Doctor (OKE node deep diagnostics)** + - Use when Node Health symptoms indicate node readiness/runtime faults, or when user explicitly requests deep node checks. + - Potentially disruptive/privileged; require explicit confirmation per node before execution. + - Start with one affected node first, then ask whether to continue on more nodes. + - Use default debug image `docker.io/library/ubuntu` each run (allow user override) and execute via helper script: + - `bash ../../scripts/node-doctor-run.sh --node --image docker.io/library/ubuntu [--namespace ]` + - Under the hood this runs: + 1) `kubectl -n debug node/ --image=` + 2) `chroot /host` + 3) `sudo /usr/local/bin/node-doctor.sh --check` + - If execution is not approved, print commands only and continue other Node Health evidence. + - Failure handling: + - capture and continue when `kubectl debug` is blocked, image pull fails, `chroot` fails, `sudo` missing, or `/usr/local/bin/node-doctor.sh` not found. + - Script output is normalized JSON, including: + - `node_doctor_attempted`, `node_doctor_executed`, `node_doctor_node`, `node_doctor_image` + - `node_doctor_result` (`pass` | `fail` | `unknown`) and `node_doctor_command_rc` + - `node_doctor_findings`, `node_doctor_raw_snippet`, `node_doctor_fallback_reason` + - `node_doctor_counts` (`pass`/`fail`/`warn`/`skip`) + +## OKE Add-ons Health +- **Helper script** + - `bash ../../scripts/oke-addon-health.sh --namespace kube-system` +- **Kubernetes** + - `kubectl -n kube-system get pods -o wide` + - `kubectl -n kube-system get deploy -o wide` + - `kubectl -n kube-system get ds -o wide` + - `kubectl -n kube-system get events --field-selector type=Warning --sort-by=.lastTimestamp` + - `kubectl -n kube-system logs deployment/coredns --tail=200` when DNS symptoms are present +- **Normalization tips**: Flag kube-system pods that are not Running/Ready, add-on deployments with unavailable replicas, daemonsets missing scheduled pods, and warning events after cluster upgrades or node pool changes. + +## Pod Networking / OCI CNI / IPAM +- **Helper script** + - `bash ../../scripts/oke-pod-network-check.sh --namespace [--pod ] [--selector ]` +- **Kubernetes** + - `kubectl -n describe pod ` + - `kubectl -n get pod -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'` + - `kubectl -n get events --field-selector type=Warning --sort-by=.lastTimestamp` + - `kubectl -n kube-system get pods -l app=oci-cni -o wide` + - `kubectl -n kube-system get pods -l name=multus -o wide` + - `kubectl get network-attachment-definitions -A` +- **Node-side checks** + - Use a privileged diagnostic pod or `kubectl debug node/` only after confirmation. + - Inspect `/opt/cni/bin` for required binaries such as `oci-ipam`, `oci-ipvlan`, `oci-ptp`, and `ipvlan`. + - Inspect `/dev/shm/oci-cni/free` and `/dev/shm/oci-cni/used` when OCI IPAM allocation state is in question. +- **Normalization tips**: Separate OCI CNI primary-pod-network failures from Multus secondary-network failures. Flag `FailedCreatePodSandBox`, missing CNI binaries, subnet/IP exhaustion, stale or missing `network-status`, and NAD namespace/name mismatches. + +## DPDK / SR-IOV / Mellanox mlx5 +- **When to use** + - User mentions DPDK, SR-IOV, Mellanox/NVIDIA, `mlx5`, `vfio-pci`, `/dev/infiniband`, `ibv_devices`, `dpdk-devbind.py`, hugepages, or a pod that requests device-plugin resources but lacks expected Multus interfaces. + - Also load `../oke-multihome-deployer/references/oke-dpdk-mlx5-notes.md`. +- **Kubernetes** + - `kubectl -n describe pod ` + - `kubectl -n get pod -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'` + - `kubectl get network-attachment-definitions -A | egrep '|noiommu|sriov|mlx|dpdk'` + - `kubectl -n get network-attachment-definition -o yaml` + - `kubectl -n get events --field-selector involvedObject.name= --sort-by=.lastTimestamp` +- **Pod exec checks** + - `kubectl -n exec -- ip -br addr` + - `kubectl -n exec -- ls -l /dev/infiniband` + - `kubectl -n exec -- ibv_devices` + - `kubectl -n exec -- lspci -nnk` + - `kubectl -n exec -- dpdk-devbind.py -s` + - `kubectl -n exec -- grep -i Huge /proc/meminfo` + - `kubectl -n exec -- mount | grep -i huge` + - `kubectl -n logs | egrep -i 'EAL|mlx5|ibv|verbs|huge|vfio|pci|rte'` +- **Node-side checks** + - Use a privileged diagnostic pod, `kubectl debug node/`, or SSH only after confirmation. + - `ip -br addr` + - `lspci -nnk | egrep -A3 'Mellanox|NVIDIA|Ethernet'` + - `ls -l /dev/vfio` + - `find /sys/kernel/iommu_groups -maxdepth 1 -mindepth 1 -type d | wc -l` + - `dpdk-devbind.py -s` +- **Normalization tips**: Keep resource allocation, Multus attachment, driver binding, RDMA/verbs exposure, hugepage visibility, and application PCI/interface mapping as separate facts. If `network-status` only shows `eth0`, do not claim SR-IOV/Multus attachment is working just because the pod is `Running`. For Mellanox `mlx5` PMD, do not assume `vfio-pci` is correct; `mlx5_core` plus RDMA/verbs may be the intended model. + +## Cluster Autoscaler / Node Pool Scaling +- **Helper script** + - `bash ../../scripts/oke-autoscaler-check.sh --namespace [--deployment ] --cluster-id --compartment-id --region ` +- **Kubernetes** + - `kubectl -n get pods --field-selector=status.phase=Pending -o wide` + - `kubectl -n get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp` + - `kubectl -n kube-system get deploy cluster-autoscaler -o wide` + - `kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200` + - `kubectl -n describe deployment ` +- **OCI** + - `oci ce node-pool list --compartment-id --cluster-id --region --all --output json` + - `oci ce node-pool get --node-pool-id --region ` + - `oci limits resource-availability list --compartment-id --service-name compute --region ` +- **Normalization tips**: Distinguish Kubernetes scheduling constraints from autoscaler refusal. Flag max node pool size reached, no matching node group, shape capacity/limit errors, subnet IP exhaustion, taints/tolerations mismatch, and missing autoscaler deployment. + +## Networking / CNI / Load Balancer +- **Kubernetes** + - `kubectl get svc -n -o yaml` + - `kubectl get svc -n -o jsonpath='{.status.loadBalancer.ingress[0].ip}'` (capture LB public IP when type is `LoadBalancer`) + - `kubectl get ingress -n -o yaml` + - `kubectl describe networkpolicy -n ` + - `kubectl get pods -n kube-system -l k8s-app=cilium-agent` (or corresponding CNI) +- **OCI** + - `oci lb load-balancer list --compartment-id --region --all --output json | jq -r '.data[] | select((."ip-addresses" // []) | any(."ip-address"=="")) | .id'` (resolve LB OCID from service external IP) + - `oci lb load-balancer get --load-balancer-id --region ` + - `oci lb load-balancer get --load-balancer-id --region --query 'data."access-log"' --output json` (check whether LB access logging is enabled) + - `oci logging log-group list --compartment-id --region --all --output json` (list candidate log groups) + - `oci logging log list --log-group-id --all --query "data[?configuration.source.resource=='' && configuration.source.service=='loadbalancer'].[\"display-name\",id,\"is-enabled\",configuration]" --output json` (second check for logging objects tied to LB OCID) + - `oci logging search search-logs --region --search-query "search \"/\" | where data.loadBalancerId = '' | sort by datetime desc" --time-start --time-end ` (when LB logs are enabled) + - `oci nlb network-load-balancer list --compartment-id --region --all --output json | jq -r '.data[] | select((."ip-addresses" // []) | any(."ip-address"=="")) | .id'` (fallback if classic LB lookup is empty) + - `oci network nsg list --compartment-id ` + - `oci network subnet get --subnet-id ` +- **Subagent preference** + - Prefer delegating LB-specific discovery/log retrieval to `oke-lb-log-collector`: + - resolves LB/NLB OCID from Service IP + - checks access-log status + - optionally enables logging (with explicit user approval path) + - extracts log issue signals for ranking +- **Normalization tips**: Note load balancer lifecycle (`PROVISIONING`, `FAILED`), security list/NSG rules, CNI pod status, service annotations impacting provisioning. Explicitly record LB logging status as `enabled`, `disabled`, or `unknown`, and include `logging_status_source` showing which check(s) succeeded. +- **TODO(live validation)**: Confirm the exact OCI CLI fields and Logging Search syntax for LB access-log discovery in the target OCI CLI version before treating log absence as conclusive. +- **If LB logs are disabled or unknown**: recommend enabling access logs before closing the incident so future RCA has request-level evidence. + - Offer operator action: + - `No (report only)` + - `Yes (print enable command)` + - `Yes (execute enable command now)` + - Enable command template: + ```bash + oci lb load-balancer update \ + --load-balancer-id \ + --region \ + --access-log '{"isEnabled":true,"logGroupId":"","logId":""}' + ``` + - Post-check: + ```bash + oci lb load-balancer get \ + --load-balancer-id \ + --region \ + --query 'data."access-log"' \ + --output json + ``` +- **If LB logs are enabled**: summarize concrete issue signals from log lines: + - 5xx rate and top failing paths/backends + - timeout/reset/error signatures + - highest observed latency fields in the selected window + +## DNS / Service Discovery +- **Helper script** + - `bash ../../scripts/oke-dns-check.sh --namespace [--service ] [--pod ] [--lookup ]` +- **Kubernetes** + - `kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide` + - `kubectl -n kube-system get deploy coredns -o wide` + - `kubectl -n kube-system get configmap coredns -o yaml` + - `kubectl -n kube-system logs deployment/coredns --tail=200` + - `kubectl -n get svc -o yaml` + - `kubectl -n get endpoints -o yaml` + - `kubectl -n get endpointslices -l kubernetes.io/service-name= -o yaml` + - `kubectl -n exec -- nslookup ` +- **Normalization tips**: Separate DNS server health from service object problems. Flag CoreDNS unavailable, ConfigMap rewrite/stub-domain issues, Service without endpoints, EndpointSlice readiness issues, `NXDOMAIN`, `SERVFAIL`, timeout, and pod-local resolver failures. + +## Ingress / OCI Native Ingress +- **Helper script** + - `bash ../../scripts/oke-ingress-check.sh --namespace --ingress [--region ]` +- **Kubernetes** + - `kubectl get ingressclass -o yaml` + - `kubectl -n get ingress -o yaml` + - `kubectl -n describe ingress ` + - `kubectl -n kube-system get pods -l app.kubernetes.io/name=oci-native-ingress-controller -o wide` + - `kubectl -n kube-system logs -l app.kubernetes.io/name=oci-native-ingress-controller --tail=200` + - `kubectl -n get secret -o yaml` +- **OCI** + - `oci lb load-balancer get --load-balancer-id --region ` + - `oci lb listener list --load-balancer-id --region ` + - `oci lb backend-set-health get --load-balancer-id --backend-set-name --region ` +- **Normalization tips**: Flag missing ingress class, controller reconciliation errors, TLS secret/certificate mismatch, listener/backend-set mismatch, backend health failures, and annotation drift. + +## Private Cluster / API Endpoint Connectivity +- **Helper script** + - `bash ../../scripts/oke-private-endpoint-check.sh --cluster-id --region [--compartment-id ]` +- **Kubernetes / workstation** + - `kubectl config current-context` + - `kubectl cluster-info` + - `kubectl get --raw=/readyz?verbose` + - `kubectl version` +- **OCI** + - `oci ce cluster get --cluster-id --region ` + - `oci network subnet get --subnet-id --region ` + - `oci network nsg list --compartment-id --region ` +- **Normalization tips**: Separate kubeconfig exec/auth failures from network reachability. Flag expired OCI security tokens, private endpoint DNS/routing problems, NSG/security-list blocks, and public/private endpoint expectation mismatch. + +## OCIR / Image Pull +- **Helper script** + - `bash ../../scripts/oke-ocir-image-pull-check.sh --namespace --pod [--image ] [--compartment-id ] [--region ]` +- **Kubernetes** + - `kubectl -n describe pod ` + - `kubectl -n get secret -o yaml` + - `kubectl -n get serviceaccount -o yaml` + - `kubectl -n get events --field-selector involvedObject.name= --sort-by=.lastTimestamp` +- **OCI** + - `oci artifacts container repository list --compartment-id --region ` + - `oci artifacts container image list --compartment-id --repository-name --region ` +- **Normalization tips**: Flag image region mismatch, missing namespace-local pull secret, expired auth token, wrong tenancy namespace, repository not found, and node egress/DNS failures to OCIR. + +## Workload Identity / OCI API From Pods +- **Helper script** + - `bash ../../scripts/oke-workload-identity-check.sh --namespace --serviceaccount [--pod ] [--tenancy-id ]` +- **Kubernetes** + - `kubectl -n get serviceaccount -o yaml` + - `kubectl -n describe pod ` + - `kubectl -n logs --tail=200 | egrep -i "notauthorized|forbidden|401|403|principal|workload|token|oci"` +- **OCI** + - `oci iam dynamic-group list --compartment-id --all` + - `oci iam policy list --compartment-id --all` +- **Normalization tips**: Flag missing service account annotations, dynamic group rule mismatch, policy statement gaps, token projection failures, and pod code using the wrong OCI auth provider. + +## Incident Timeline +- **Helper script** + - `bash ../../scripts/oke-incident-timeline.sh --namespace [--pod ] [--deployment ] [--service ] [--compartment-id ] [--region ]` +- **Purpose** + - Merge event timing from Kubernetes objects and OCI alarms so the final report can explain what changed first. +- **Kubernetes** + - `kubectl -n get events --sort-by=.lastTimestamp` + - `kubectl -n describe pod ` + - `kubectl -n rollout history deployment/` + - `kubectl -n describe deployment ` + - `kubectl -n describe service ` +- **OCI** + - `oci monitoring alarm-status list-alarms-status --compartment-id --status FIRING --region --all` +- **Normalization tips**: Prefer chronology over volume. Keep warning/failure events, rollout changes, object readiness changes, and firing alarms. Use the timeline to avoid blaming symptoms that occurred after the first failing signal. + +## Application Performance +- **Kubernetes** + - `kubectl get deployment -n -o yaml` + - `kubectl describe deployment -n ` + - `kubectl rollout history deployment/ -n ` + - `kubectl top pods -n -l app=