From bcba55cbcbc3301d1b63f12da2623cca8c07378e Mon Sep 17 00:00:00 2001 From: elementalsouls Date: Mon, 15 Jun 2026 10:00:43 -0400 Subject: [PATCH] Add run-claude-osint skill (driver + man page) This repo is a Claude skills package, not an app. The skill validates the SKILL.md frontmatter Claude loads and exercises the two Python helpers the way CI does, via a one-shot smoke.sh driver: - py_compile secret_scan.py and h1_reference.py - secret_scan.py stdin detection (AWS key + JWT canaries) and directory scan - SKILL.md frontmatter validation (name/description/version/triggers, >=5) - sync-skill-content.sh --check - h1_reference.py live HackerOne fetch (non-fatal if offline) SKILL.md documents the driver as the primary path plus verified individual commands and the repo's non-obvious gotchas (sync is a no-op without docs/full-skills/, secret_scan self-filters noise dirs). Co-Authored-By: Claude Opus 4.8 --- .claude/skills/run-claude-osint/SKILL.md | 111 +++++++++++++++++++++++ .claude/skills/run-claude-osint/smoke.sh | 89 ++++++++++++++++++ 2 files changed, 200 insertions(+) create mode 100644 .claude/skills/run-claude-osint/SKILL.md create mode 100755 .claude/skills/run-claude-osint/smoke.sh diff --git a/.claude/skills/run-claude-osint/SKILL.md b/.claude/skills/run-claude-osint/SKILL.md new file mode 100644 index 0000000..300a223 --- /dev/null +++ b/.claude/skills/run-claude-osint/SKILL.md @@ -0,0 +1,111 @@ +--- +name: run-claude-osint +description: Build, validate, and run the claude-osint skills repo — check SKILL.md frontmatter, run the secret_scan.py and h1_reference.py helpers, run sync-skill-content.sh, run the smoke test. Use when asked to run, build, test, validate, or smoke-test claude-osint or its OSINT skills/scripts. +version: 1.0.0 +triggers: + - run claude-osint + - build claude-osint + - test claude-osint + - validate claude-osint + - smoke test claude-osint + - run secret_scan + - run h1_reference + - validate SKILL.md frontmatter + - sync skill content +--- + +# Run: claude-osint + +`claude-osint` is **not an app** — it's a Claude *skills package*. Its product +is two `SKILL.md` files (`skills/offensive-osint/`, `skills/osint-methodology/`) +plus two runnable Python helpers under `skills/offensive-osint/scripts/`. There +is no GUI, server, or TUI. "Running it" means: the SKILL.md frontmatter parses +and is complete (that's what Claude loads), and the helper scripts work. + +The driver is **`.claude/skills/run-claude-osint/smoke.sh`** — it does all of +that in one shot. **All paths below are relative to the repo root.** + +## Prerequisites + +Python 3 with PyYAML (already present on this container). If `import yaml` fails: + +```bash +pip install pyyaml # or: apt-get install -y python3-yaml +``` + +The helpers are stdlib-only otherwise. `h1_reference.py` needs outbound HTTPS to +`hackerone.com`. No build step, no `npm install` — nothing to compile. + +## Run (agent path) — the driver + +From the repo root: + +```bash +.claude/skills/run-claude-osint/smoke.sh # full smoke (exits 0 on pass) +.claude/skills/run-claude-osint/smoke.sh --no-net # skip the HackerOne live check +``` + +It runs six checks and prints a `==> PASS` / `==> FAIL` line (exit 0 / 1): + +1. `py_compile` both helpers. +2. `secret_scan.py` detects AWS key + JWT from stdin (the CI canaries). +3. `secret_scan.py` recursively scans `skills/` and emits JSONL (~11 findings — example tokens in the SKILL.md docs). +4. Every `skills/*/SKILL.md` has valid YAML frontmatter with `name`/`description`/`version`/`triggers` and ≥5 triggers. +5. `sync-skill-content.sh --check` exits 0. +6. `h1_reference.py` fetches one live disclosed report (non-fatal if offline). + +## Run individual pieces + +Verified this session: + +```bash +# Secret scanner — stdin +printf 'AKIAIOSFODNN7EXAMPLE\n' | python3 skills/offensive-osint/scripts/secret_scan.py +# -> {"pattern": "AWS_ACCESS_KEY", "severity": "critical", ...} + +# Secret scanner — scan a directory tree (JSONL, one finding per line) +python3 skills/offensive-osint/scripts/secret_scan.py skills/ + +# HackerOne reference helper (needs network) +python3 skills/offensive-osint/scripts/h1_reference.py --top-voted --limit 3 +python3 skills/offensive-osint/scripts/h1_reference.py --top-bounty --limit 3 +python3 skills/offensive-osint/scripts/h1_reference.py --top-voted --query "XSS" --pages 3 + +# Install the skills for Claude to load +cp -r skills/* ~/.claude/skills/ +``` + +## Test + +CI (`.github/workflows/lint.yml`) runs four jobs: markdown lint +(`markdownlint-cli2`), the frontmatter check, the `secret_scan.py` smoke, and +`shellcheck ./scripts`. `smoke.sh` covers the latter two plus the helpers +directly. `markdownlint-cli2` and `shellcheck` are **not installed** on this +container; install with `npm i -g markdownlint-cli2` / `apt-get install -y +shellcheck` if you need to reproduce those jobs locally. + +## Gotchas + +- **`sync-skill-content.sh` is a no-op in a fresh clone.** It copies from + `docs/full-skills/`, which **is not in the repo** — the script prints + "⚠ Source missing … Skipping" and exits 0. The `skills/*/SKILL.md` files are + *already* the full inline content (offensive-osint ≈ 4,200 lines), so you do + **not** need to run sync after cloning. Don't be alarmed by the warnings. +- **`secret_scan.py` skips its own findings as noise filters.** It excludes + `.git`, `node_modules`, `__pycache__`, `.venv`, `dist`, `build`, `.cache` + and any file >10 MB. Scanning `skills/` returns the example/doc tokens (~11), + not real secrets. +- **`h1_reference.py` hits an undocumented public GraphQL endpoint.** It works + unauthenticated but is sensitive to HackerOne-side changes and rate limits; + treat a failure there as environmental, not a repo bug (hence non-fatal in + the driver). It caps at 50 results/page — use `--pages N` for breadth. +- **No screenshot / no window** — this repo has no visual surface. If a request + asks to "see it run," that means the `smoke.sh` PASS output, not a GUI. + +## Troubleshooting + +| Symptom | Fix | +|---|---| +| `ModuleNotFoundError: No module named 'yaml'` in check [4] | `pip install pyyaml` (only the frontmatter check needs it). | +| Check [6] shows `~ no network / HackerOne unreachable` | Expected when egress is blocked; run with `--no-net` to silence. Driver still passes. | +| `sync --check` shows "Source missing" | Expected — see Gotchas. Not a failure; exit code is 0. | diff --git a/.claude/skills/run-claude-osint/smoke.sh b/.claude/skills/run-claude-osint/smoke.sh new file mode 100755 index 0000000..b27cc61 --- /dev/null +++ b/.claude/skills/run-claude-osint/smoke.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# smoke.sh — driver for the claude-osint skills repo. +# +# This repo is a *skills package*, not an app: its product is two SKILL.md +# files plus two runnable Python helpers. "Running" it means validating the +# SKILL.md frontmatter (what Claude loads) and exercising the helpers the +# way CI does. This driver does exactly that, in one shot. +# +# Usage (run from the repo root): +# .claude/skills/run-claude-osint/smoke.sh # full smoke (h1 net check is non-fatal) +# .claude/skills/run-claude-osint/smoke.sh --no-net # skip the HackerOne live-network check +# +# Exit codes: 0 = all required checks passed; 1 = a required check failed. +set -uo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +cd "$REPO_ROOT" + +NO_NET=false +[ "${1:-}" = "--no-net" ] && NO_NET=true + +SECRET_SCAN="skills/offensive-osint/scripts/secret_scan.py" +H1="skills/offensive-osint/scripts/h1_reference.py" +fail=0 +pass() { printf ' \033[32m✓\033[0m %s\n' "$1"; } +bad() { printf ' \033[31m✗ %s\033[0m\n' "$1"; fail=1; } + +echo "==> claude-osint smoke (repo: $REPO_ROOT)" + +# 1. Both Python helpers compile. +echo "[1] py_compile helpers" +python3 -m py_compile "$SECRET_SCAN" && pass "secret_scan.py compiles" || bad "secret_scan.py syntax error" +python3 -m py_compile "$H1" && pass "h1_reference.py compiles" || bad "h1_reference.py syntax error" + +# 2. secret_scan.py detects known tokens from stdin (AWS + JWT are the CI canaries). +echo "[2] secret_scan.py stdin detection" +out=$(printf 'AKIAIOSFODNN7EXAMPLE\neyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c\n' | python3 "$SECRET_SCAN") +echo "$out" | grep -q '"AWS_ACCESS_KEY"' && pass "detects AWS_ACCESS_KEY" || bad "missed AWS_ACCESS_KEY" +echo "$out" | grep -q '"JWT"' && pass "detects JWT" || bad "missed JWT" + +# 3. secret_scan.py recursively scans a directory and emits JSONL. +echo "[3] secret_scan.py directory scan" +n=$(python3 "$SECRET_SCAN" skills/ | wc -l | tr -d ' ') +[ "$n" -gt 0 ] && pass "directory scan emitted $n JSONL findings" || bad "directory scan emitted nothing" + +# 4. SKILL.md frontmatter is valid + complete (this is what Claude loads). +echo "[4] SKILL.md frontmatter validation" +for skill in skills/*/SKILL.md; do + python3 - "$skill" <<'PY' +import sys, yaml +skill = sys.argv[1] +content = open(skill).read() +if not content.startswith("---"): + print(f"FAIL no-frontmatter {skill}"); sys.exit(1) +data = yaml.safe_load(content[3:content.index("---", 3)]) +for r in ("name", "description", "version", "triggers"): + if r not in data: + print(f"FAIL missing-{r} {skill}"); sys.exit(1) +if not isinstance(data["triggers"], list) or len(data["triggers"]) < 5: + print(f"FAIL <5-triggers {skill}"); sys.exit(1) +print(f"OK {data['name']} v{data['version']} triggers={len(data['triggers'])}") +PY + # shellcheck disable=SC2181 + if [ $? -eq 0 ]; then pass "$skill"; else bad "$skill frontmatter"; fi +done + +# 5. sync-skill-content.sh --check runs cleanly (no-op without docs/full-skills/). +echo "[5] sync-skill-content.sh --check" +if ./scripts/sync-skill-content.sh --check >/dev/null 2>&1; then + pass "sync --check exited 0" +else + bad "sync --check failed" +fi + +# 6. h1_reference.py live network (non-fatal — sandbox may block egress). +echo "[6] h1_reference.py live network (HackerOne GraphQL)" +if [ "$NO_NET" = true ]; then + printf ' \033[33m~\033[0m skipped (--no-net)\n' +else + if timeout 40 python3 "$H1" --top-voted --limit 1 2>/dev/null | grep -q 'hackerone.com/reports/'; then + pass "fetched a live disclosed report" + else + printf ' \033[33m~\033[0m no network / HackerOne unreachable (non-fatal)\n' + fi +fi + +echo +if [ "$fail" -eq 0 ]; then echo "==> PASS"; else echo "==> FAIL"; fi +exit "$fail"