feat: add curated llms.txt with validation CI

[omarespejel]

Summary

Add a production-ready, curated llms.txt for docs.starknet.io, and enforce it with a lightweight validation check.

Changes

1. Add root llms.txt with high-signal canonical docs links and concise descriptions.
2. Add scripts/validate_llms_txt.py to validate format, link shape, duplicates, and local path mapping.
3. Add .github/workflows/llms-check.yml to run validation on PRs and on main updates touching llms assets.

Why

• Curated llms.txt improves retrieval quality compared with broad auto-generated lists.
• Mintlify supports custom llms.txt override while preserving llms-full.txt for exhaustive context.
• Search-oriented AI crawlers use explicit bot controls and robots policy (OpenAI/Anthropic/Perplexity).

References:

• https://llmstxt.org/
• https://www.mintlify.com/docs/ai/llmstxt
• https://developers.openai.com/api/docs/bots
• https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• https://docs.perplexity.ai/docs/resources/perplexity-crawlers

Validation

• python scripts/validate_llms_txt.py
• Result: llms.txt validation passed: 45 links, 6 sections, no format/link errors.
• Spot-check: all linked URLs currently return HTTP 200.

Scope

• No robots.txt changes in this PR.
• Focused only on llms.txt quality and guardrails.

[omarespejel]

Superseded by #1751, opened from the clean branch name .

[omarespejel]

Superseded by #1751, opened from the clean branch name feat/llms-txt-validation.