[VascoSch92]

Summary

Based on an analysis of 72 real conversations that invoked this skill, cross-referenced with production run data (54k runs) from dev-oss-automation-vm. The skill's guidance was sound but agents routinely skipped the explicit user-facing checkpoints it mandates.

• Interview-style creation flow: Automation Creation Process is now four explicit phases — Discovery (trigger, behavior, LLM-vs-deterministic call stated out loud, access checks, reachability, stakes) → Plan Presentation (shows the plan, ends the turn — never deploys in the same turn it's presented) → Deploy & Verify (dispatches one test run and polls to a terminal state before declaring success) → Report.
• OPENHANDS_HOST moved to the very first thing the agent reads, ahead of everything else.
• Sub-5-minute cron upgraded from a soft suggestion to a hard default to push back on, backed by real production numbers (every-minute automations failed 16–79% of the time vs. 1–31% for 5-minute-or-looser schedules).
• Timeout-vs-cron-interval overlap note added (guards against the concurrent-run races seen in production).
• Retry-with-backoff helper wired into the no-LLM script boilerplate (network fragility is the dominant real-world failure mode).
• Tarball-integrity check added before upload (a corrupted tarball previously uploaded and created the automation fine, then failed silently on the first run).
• Pre-flight health-check pattern added as a recommended custom-script pattern.
• High-stakes-automation confirmation and secret-hygiene-in-prompts guidance folded into the existing Security Considerations section from docs: add Security Considerations section to openhands-automation skill #429 (money-moving/credential-spending/irreversible actions need a restated, specific confirmation — not a bare "yes" — and secrets must never be embedded verbatim in a persisted prompt or script).

Test plan

• npm run build:skills — regenerated skills/index.js to match the updated SKILL.md content
• uv run -p 3.12 --group test pytest tests/test_skills_catalog.py tests/test_sync_extensions.py tests/test_skills_have_readme.py tests/test_skill_plugin_loading.py tests/test_catalog_schema.py tests/test_catalogs.py — 236 passed
• Manual read-through of both edited files for internal consistency (done locally; worth a second pass in review)

[enyst]

I'm an AI agent (Claude Code, based on Opus 5.5) helping Engel Nyst (@enyst) with project work.

Leaving a comment: this is a draft, two months stale, and it conflicts with main in SKILL.md, references/custom-automation.md and skills/index.js. Several ideas here are worth keeping: deploy only after a separate confirming turn, verify a run, watch for timeout/interval overlap, confirm high-stakes automations explicitly, and keep secrets out of persisted prompts. But main has changed this skill a lot since (#535 host selection, #709/#712 Cloud secrets and templates, #686 framing of presets vs scripts), so the rebase will take real reconciliation. Issues to fix along the way:

1. The host snippet contradicts main and breaks in zsh. SKILL.md:29 uses ${HOST:-https://app.all-hands.dev}, but <HOST> is a system-prompt value, not an environment variable. zsh sets $HOST to the machine hostname, so there it resolves to the hostname. Falling back to Cloud is also what main's "Determining the API Host" now forbids on a local Agent Canvas. Drop the snippet or link to that section.
2. get_secret is the pre-#709 version. SKILL.md:140 and custom-automation.md only use AGENT_SERVER_URL. Main also handles Cloud and Enterprise through OPENHANDS_CLOUD_API_URL/SANDBOX_ID, so taking this side in the conflict would break secrets on Cloud.
3. fetch_with_retry retries every error. SKILL.md:129 retries on any Exception, including 401/403/404, and sets no urlopen timeout. Retry only on URLError, timeouts, 5xx and 429, and pass timeout=.
4. The tarball check misses truncation. custom-automation.md:61: tarfile.open() only reads the first header, and in a local test it accepted a half-truncated .tar.gz. tar -tzf automation.tar.gz >/dev/null or .getmembers() reads the whole archive.
5. The automatic test dispatch can cause real side effects. SKILL.md:68 dispatches a run right after every creation, including the irreversible automations this PR flags. Make the test run part of the confirmed plan.

Overlap with #378: both change the timeout docs. Main still says 1-600 s, but the service now accepts up to 1800 s (max_run_duration).