Skip to content

AGENTS.md

Latest commit

 

History

History
402 lines (291 loc) · 37.2 KB

AGENTS.md

File metadata and controls

402 lines (291 loc) · 37.2 KB
Raw

AGENTS.md

Guidance for future Codex sessions working in this repo.

Overview

pupila is a config-driven, forkable daily job aggregator. It fetches listings from 13 public sources (3 ATS APIs — Ashby, Greenhouse, Lever — plus RSS feeds, JSON job boards, Hacker News, HTML scrapers, an Aave Next.js scraper, and ashby-private — a config-driven fetcher for orgs hosted on Ashby with the public posting-API disabled), normalizes them, applies hard exclusion filters, computes a per-job fitScore, deduplicates, and writes data/jobs.json, an RSS feed at data/feed.xml, and an auto-regenerated JOBS.md table. The hand-written README.md is the project doc and is not rewritten by the pipeline. No external services. No DB. Output lives in this repo.

The repo ships a neutral template in config/profile.json. After onboarding (CV upload → brief generation), /api/profile-generate shells out to the local LLM CLI to fill in the personal keyword lists + weights based on the brief. Re-runnable from Settings → Scoring profile → Regenerate. config/slugs.json ships with the full ~50-company tier-S list (all public ATS URLs — non-personal data, edit by hand to add/remove companies).

First-run UX: a forker generates their config/candidate-brief.md by running pnpm run setup-brief --file ~/cv.pdf (or via the UI's Profile tab → drop a PDF/DOCX/MD CV). That CLI shells out to whichever local LLM CLI is installed (Codex, codex, gemini, opencode — auto-detected, override via PUPILA_LLM=<provider>).

Stack

  • Node 22 LTS, ESM, TypeScript 5.9 (NodeNext)
  • Biome 2.4 (lint + format, single config in biome.json)
  • pnpm 10
  • Vitest 3 (tests in tests/, run via pnpm test — 120 cases)
  • simple-git-hooks (pre-commit lint && typecheck)
  • Single runtime dep: fast-xml-parser. Native fetch only.

Run locally

pnpm install                # also installs the pre-commit hook
pnpm run dev                # tsx, no build step (this is what the launchd/cron aggregate agent runs)
pnpm start                  # built output: requires pnpm run build first
pnpm run typecheck          # tsc --noEmit on src/, then on src/+tests/ via tsconfig.test.json
pnpm run lint               # biome check
pnpm run lint:fix            # biome check --write
pnpm test                   # vitest run (120 unit tests)
pnpm run test:watch         # vitest watch mode
pnpm run ui                 # local-only UI: Vite dev server on http://127.0.0.1:5173, reads data/jobs.json
pnpm run ai-review          # local-only: shells out to `Codex -p` to write data/ai-reviews.json
pnpm run ai-review --top=50 # raise the per-run cap (default 20 highest-fitScore unreviewed)
pnpm run ai-review --force  # re-review entries that already exist
pnpm run ai-review --ids=a,b # review specific job ids only
pnpm run setup-brief --file ~/cv.pdf  # generate config/candidate-brief.md from a CV (PDF/DOCX/MD/TXT)
pnpm run daily              # convenience: pnpm run dev && pnpm run ai-review (morning routine)

The pipeline writes to data/jobs.json (slim — body field stripped), data/feed.xml (RSS 2.0 of new jobs), JOBS.md, optionally data/archive/<YYYY-MM>.json on day 1 of the month, and per-source raw dumps in data/raw/<source>-<YYYY-MM-DD>.json (gitignored). README.md is hand-maintained — never overwrite it from code.

Pre-commit runs lint && typecheck automatically. Bypass with SKIP_SIMPLE_GIT_HOOKS=1 git commit ... for emergency commits.

Repo layout

src/
  index.ts          # orchestrator
  types.ts          # Job, Source, Category, ApplicationStatus, AppliedEntry, FetcherResult, Raw* shapes
  utils.ts          # fetchWithTimeout (1-retry), isSafeUrl, sha1, stripHtml, readJsonOrNull, ...
  rss.ts            # shared fast-xml-parser wrapper
  normalize.ts      # one normalize<Source> per source -> Job (uses withSalary() for parsed salary fields)
  salary.ts         # parseSalary(): raw string -> { min, max, currency } (annual integer, ISO code)
  filters.ts        # createFilters(profile) factory + applyFilters default. Hard excludes + scoring + category + _signals.
  applied.ts        # loads config/applied.json, attaches AppliedEntry to Job by URL hash
  dedup.ts          # 2-pass dedup, priority-aware tiebreak
  render.ts         # JOBS.md markdown (applied, ✨ new, 🗑 removed, 🚨 source-health, status/salary in title)
  feed.ts           # RSS 2.0 generator -> data/feed.xml (top 50 ✨ new jobs)
  setup-brief.ts    # CLI: parse CV (pdf/docx/md/txt) → LLM CLI → write config/candidate-brief.md
  lib/
    llm.ts            # detectLlmCli + runLlm — provider-agnostic shell-out (Codex/codex/gemini/opencode)
    cv-parser.ts      # parseCvBuffer + parseCvFile (pdfjs-dist for PDF, mammoth for DOCX, raw text otherwise)
    brief-template.ts # readBriefBody / writeBriefBody — preserve preamble + markers in candidate-brief.md
  fetchers/
    _shared.ts                                              # fetchMultiSlug orchestration helper
    ashby.ts            greenhouse.ts       lever.ts        # ATS APIs (public, multi-slug)
    ashby-private.ts                                        # multi-slug GraphQL for hidden Ashby orgs
    aave.ts                                                 # first-party scraper (Next.js __NEXT_DATA__)
    aijobsnet.ts        cryptojobslist.ts   web3career.ts   # boards / scrapers
    hn-hiring.ts        hn-jobs.ts                          # Hacker News
    remoteok.ts         remotive.ts         weworkremotely.ts

config/
  slugs.json        # tier-S Ashby/Greenhouse/Lever slug arrays — neutral defaults; tune for your interests
  profile.json      # scoring weights + keyword lists (non-code tuning surface) — neutral defaults
  candidate-brief.md # natural-language candidate description (template until generated by setup-brief)
  applied.json      # list of jobs you've applied to (UI writes here via /api/applied)

scripts/
  install-launchd.sh # macOS local scheduler — wraps `pnpm run daily` in a launchd agent
  install-cron.sh    # Linux local scheduler — appends a crontab entry for `pnpm run daily`

tests/
  fixtures/
    test-profile.json     # frozen tuned profile used by filters.test.ts so scoring assertions don't shift when config/profile.json is genericized
  filters.test.ts          # 33 cases — hard drops, droppedByRule, scoring, plurals, frontendBody, boilerplate, tiered weighting; uses createFilters(testProfile)
  dedup.test.ts            # 10 cases — id/title collapse, priority, compareJobs salary/postedAt/id chain
  applied.test.ts          # 4 cases  — STATUS_EMOJI map + summarizeApplied grouping/ordering
  salary.test.ts           # 15 cases — K/M suffix, currency detection, hourly→annual, free-text fallback
  feed.test.ts             # 6 cases  — RSS skeleton, escaping, sort, 50-item cap
  aave.test.ts             # 7 cases  — __NEXT_DATA__ extraction + normalizer
  ashby-private.test.ts    # 9 cases  — GraphQL list/detail parsers + slug-to-company derivation
  ai-review-parse.test.ts  # 9 cases  — markdown-fence stripping, invalid verdicts, missing fields, dirty arrays
  normalize-hn.test.ts     # 7 cases  — hn-hiring header parsing, plausible-company guard, role-pattern fallback
  utils.test.ts            # 20 cases — URL safety, stripHtml, time math, human date formatter

ui/                 # local-only browser dashboard (Vite + React)
  index.html        # Vite entry
  vite.config.ts    # explicit `root` set via fileURLToPath so `pnpm run ui` works from repo root
  tsconfig.json     # bundler module resolution + DOM lib + JSX
  src/
    main.tsx        # ReactDOM.createRoot
    App.tsx         # single-component MVP: filter + sort + table over data/jobs.json
    types.ts        # local copy of Job/Source/Category/AppliedEntry (slim — no body, no _signals)
    styles.css      # CSS variables + dark mode via prefers-color-scheme
    vite-env.d.ts   # /// <reference types="vite/client" /> for CSS-import typing

tsconfig.json       # rootDir=src/, strict NodeNext
tsconfig.test.json  # extends above with rootDir=. so tests/ typecheck without leaking into the build

.github/
  workflows/
    check.yml       # PR/push: biome + typecheck + tests + build + audit (only remaining workflow)

CodeQL workflow removed. Code Scanning isn't available on private repos without GitHub Advanced Security. If the repo ever goes public, restore .github/workflows/codeql.yml from commit 7397117.

How to add a new fetcher

  1. Add a Raw shape to src/types.ts (e.g. RawFooBoard).
  2. Create src/fetchers/<name>.ts exporting fetch<Name>(): Promise<FetcherResult<Raw>> — i.e. returning { items: Raw[]; errors: string[] }. Never throw. Catch all errors internally, push them onto the errors array, and return.
    • Use fetchWithTimeout / fetchJson / fetchText from utils.ts (30s timeout, 1 retry on 5xx/network).
    • Pass JSON_HEADERS or RSS_HEADERS so the request looks like a real browser.
    • For multi-slug/multi-page fetchers, use fetchMultiSlug from src/fetchers/_shared.ts — it handles the Promise.all + per-slug try/catch + console.error logging + flatMap-aggregate scaffolding. The fetcher only owns the per-slug extraction (URL construction + response shape mapping). See ashby.ts, greenhouse.ts, lever.ts for the canonical pattern.
  3. Add the literal source name to the Source union in src/types.ts.
  4. Add a normalizer to src/normalize.ts: normalize<Name>(items, fetchedAt): Job[].
  5. Wire it into src/index.ts:
    • Import fetch<Name> and normalize<Name>.
    • Add a line to the Promise.all block: processFetcher('<source>', fetch<Name>, normalize<Name>, fetchedAt, today).
  6. Add the new source to SOURCE_PRIORITY in src/dedup.ts and to the SOURCES list in src/render.ts.
  7. Add at least one test in tests/ covering the parser if it's an HTML scraper.

Smoke-test locally before wiring it into index.ts:

npx tsx -e "import('./src/fetchers/<name>.ts').then(async m => { const r = await m.fetch<Name>(); console.log('count:', r.items.length, 'errors:', r.errors, 'first:', r.items[0]); })"

How to add a tier-S company

All three slug arrays live in config/slugs.json — adding a company is a non-code edit. Pick the right ATS based on where the company hosts:

ATS JSON key URL pattern (find slug here)
Ashby ashby jobs.ashbyhq.com/<slug>
Greenhouse greenhouse boards.greenhouse.io/<slug>
Lever lever jobs.lever.co/<slug>

The TIER_S_ASHBY_SLUGS / TIER_S_SLUGS / TIER_S_LEVER_SLUGS exports in each fetcher file are now thin re-exports of the JSON config.

Probe before adding (each ATS exposes a public board endpoint that returns 200 + JSON if the slug is live):

curl -sI "https://api.ashbyhq.com/posting-api/job-board/<slug>?includeCompensation=true"
curl -sI "https://boards-api.greenhouse.io/v1/boards/<slug>/jobs"
curl -sI "https://api.lever.co/v0/postings/<slug>?mode=json"

Slugs that 404 are logged and skipped silently, so it's safe to leave a known-bad slug in the list while waiting for upstream to restore it.

If a target company isn't on any of the three big ATSes, it might still be on Ashby's hosted-board GraphQL even when the public posting-API returns 404 — try https://jobs.ashbyhq.com/<slug> in a browser; if that loads, just append the slug to config/slugs.json#ashbyPrivate (the ashby-private fetcher in src/fetchers/ashby-private.ts will pick it up — no new code needed). Genuine custom ATSes (Webflow CMS, Next.js careers pages, etc.) need their own per-company HTML scraper — src/fetchers/aave.ts is the canonical example for Next.js __NEXT_DATA__ extraction.

Filter rules

All in src/filters.ts. Weights and keyword lists are loaded from config/profile.json at startup — compileKw() joins each keyword array with | and wraps in \b...\b/i. Adjusting a weight or adding a keyword is a non-code change.

The hard-drop chain is a named-rule list (HARD_RULES in filters.ts) — applyFilters runs Array.find over the rules, increments droppedHard, and tallies the rule name in droppedByRule. The breakdown surfaces in JOBS.md next to the hard-drop count (e.g. (missing_senior_req=812, title_non_eng_role=64, ...)) so you can see at a glance which rule does the heavy lifting. To add a new hard-drop check, append a { name, test } entry to HARD_RULES — no other plumbing needed.

Applied in this order:

  1. Hard excludes (drop entirely)

    • URL is not http/https (security gate via isSafeUrl)
    • Title contains junior/jr/intern/entry-level/associate/graduate/trainee/apprentice
    • Title does NOT contain a senior_req keyword (senior/sr/staff/principal/lead/head/director/engineer(s)/developer(s)/architect(s))
    • Body matches a hard US-only/onsite pattern (uses full body, not the truncated scoring body — onsite language at the bottom of a posting must still count)
    • Title or body matches a non-engineering pattern AND title lacks an engineering keyword
    • Title matches TITLE_NON_ENG_COMPOUND: customer support/success engineer, sales engineer, solutions engineer, developer relations/advocate/experience, devrel, field engineering/operations, business/sales/people operations, partner(ships) engineer, technical sourcer/recruiter, forward deployed/implementation/onboarding engineer, gtm, go-to-market. These titles contain the word "Engineer" but aren't real engineering.
    • Title matches TITLE_NON_ENG_LEADERSHIP: VP/Vice President, CMO, CRO, CFO, COO.
    • Title matches TITLE_NON_FRONTEND_ENG: product security / data / devops / sre / infrastructure / platform / qa / network / firmware / embedded engineer (the user is a frontend engineer; these specialties are out of scope).
    • Title matches TITLE_NON_ENG_ROLE: lead/manager roles for client/account/customer/business/product/operations/regional/country (real "engineering lead" roles still pass via the seniorReq engineering keyword).
    • Title matches TITLE_NON_TECH_ROLE: analyst, trader, scientist, researcher.

  2. Body preparation for scoring. preparedScoringBody() strips known company boilerplate (EEO, privacy notice, accommodations, "About us") and truncates the remainder to scoringBodyMaxChars (default 1500). All keyword scoring runs against this prepared body — prevents footer text like "we use Anthropic Codex internally" from landing a +20 AI signal on a backend role. Hard-drop checks above still see the full body.

  3. Soft scoring (additive, capped at maxScore = 100):

    • Web3 signals (+20 title/body, +20 stack) — binary
    • AI signals (+20 title/body, +20 stack) — binary
    • Stack signals (+10 React/Next/TS, +5 RN/Expo, +5 GraphQL/Tailwind/Vite) — tiered (see below)
    • Seniority (+15 lead/staff/principal/head, +10 senior/sr) — binary
    • Frontend title (+10 if title contains frontend/fullstack/web/mobile) — binary
    • Frontend body (+10 if body contains role-specific frontend phrases like "design system" / "ship components" / "accessibility") — tiered
    • Location (+10 remote/EMEA/CET/Spain/anywhere) — binary
    • Freshness (+10 within 7 days, +5 within 14 days) — binary

    Tiered weighting applies to the four signals marked tiered via tieredWeight(count, baseWeight) in filters.ts: 0 mentions = 0, 1 = floor(base * 0.5) (half), 2–3 = base, 4+ = floor(base * 1.5) (1.5× boost). Implemented with global-flag regexes (STACK_PRIMARY_G, STACK_RN_G, STACK_OTHER_G, BODY_FRONTEND_KW_G) + countMatches. The other signals stay binary because they're inherently low-cardinality and gaming them with repetition isn't a real concern.

  4. Negative: -10 if body hints US-centric without remote-worldwide language. Applied after capping.

  5. Drop anything with fitScore < minScoreToKeep (default 30).

  6. Category: web3+ai if both web3 and AI signals fired, else web3, ai, or general.

To adjust a weight, edit config/profile.json#weights.<field>. To add a keyword to an existing list, edit config/profile.json#keywords.<list>. For new signal types or new hard-drop branches, edit applyFilters in filters.ts directly.

Adding a new positive signal. Append the field to the JobSignals interface in types.ts, add its weight to config/profile.json#weights, and append one line to the positives object literal in applyFilters. The sum is computed via Object.values(positives).reduce(...) so you don't need to remember to update an addition chain — it auto-includes any new field.

Debugging fitScore via _signals

Every kept job in data/jobs.json has a _signals object showing exactly which scoring rules fired:

"_signals": {
  "web3TitleBody": 0,    "web3Stack": 20,
  "aiTitleBody": 20,     "aiStack": 20,
  "stackPrimary": 10,    "stackRn": 0,    "stackOther": 0,
  "leadTitle": 0,        "seniorTitle": 10,
  "frontendTitle": 10,   "frontendBody": 10,
  "locationRemote": 10,
  "freshness7d": 10,     "freshness14d": 0,
  "usCentricPenalty": 0,
  "rawTotal": 110,       "capped": true
}

When tuning regexes, run pnpm run dev, then jq '.[0]._signals' data/jobs.json (or read it directly) to see the breakdown of the top job. rawTotal is the un-capped positive sum; capped: true means positives summed > 100 before clamping; usCentricPenalty is applied after capping.

Application tracking

config/applied.json is the source of truth for jobs you've applied to. It can be hand-edited, but day-to-day it's written by the local UI via the dev-server middleware (see "Local UI"). Schema in src/types.ts:

type ApplicationStatus = 'applied' | 'interview' | 'offer' | 'rejected' | 'withdrawn';
interface AppliedEntry { url: string; status: ApplicationStatus; date: string; notes?: string }

src/applied.ts exports:

  • STATUS_EMOJI — the emoji prefix shown in JOBS.md titles (📝 / 💬 / 🎯 / ❌ / ⏸).
  • loadAppliedMap(path?) — reads the JSON, hashes each url with sha1(normalizeUrl(...)) (the same identity used for Job.id), and returns a Map<idHash, AppliedEntry>.
  • summarizeApplied(entries) — produces the one-line summary header (e.g. 🎯 1 offer · 💬 2 interview · 📝 5 applied).

Wired in src/index.ts after dedup+sort: every kept job gets job.applied = appliedMap.get(job.id) when matched. render.ts then:

  1. Renders a "📋 Application status" section at the top of JOBS.md (above "✨ New since last run") if any applied entries exist, sorted by date desc.
  2. Prefixes the title cell with the status emoji in every category section so already-applied jobs are still visible (deliberate — you may want to follow up or compare against a duplicate posting).

Don't filter applied jobs out of the main list. The user explicitly asked for them to remain visible.

Persistence model. Edits made via the UI go straight to disk (config/applied.json). Future pnpm run dev runs (whether triggered by you or by the launchd/cron agent) re-read it. There's no auto-commit anymore — the project is local-first. If you want to keep applied edits across machines, commit config/applied.json manually (or symlink it in from a private dotfiles repo).

Dedup

In src/dedup.ts:

  1. By id (sha1 of normalized URL).
  2. By sha1(normalize(company) + '|' + normalize(title)).

Tiebreak: highest fitScore wins; on ties, the source with higher SOURCE_PRIORITY wins. Order: aave = ashby-private > ashby > lever > greenhouse > cryptojobslist > web3career > aijobsnet > hn-hiring > hn-jobs > remotive > weworkremotely > remoteok.

Final sort (compareJobs)

src/dedup.ts exports a compareJobs(a, b) comparator that the orchestrator uses for the post-dedup sort. Order:

  1. fitScore desc (primary)
  2. salaryMax desc (transparent-comp companies float up among score-tied roles; null is treated as 0 so unstated comp sinks below stated comp)
  3. postedAt desc (newest first)
  4. id asc (deterministic tiebreak so day-over-day diffs are stable when everything else ties)

The comparator is exported (not just inlined) so it can be unit-tested directly in tests/dedup.test.ts.

New-since / removed-since diff

Right before writing the new data/jobs.json, the orchestrator reads the previous committed copy via readJsonOrNull, builds two sets (previous IDs, current IDs), and computes:

  • newJobs = current − previous → "✨ New since last run" section (top 20 by fitScore desc, also drives data/feed.xml).
  • removedJobs = previous − current → "🗑 Removed since last run" section (top 10 by previous fitScore).

Both lists become counts in RenderStats.newCount / removedCount and are passed to renderReadme(jobs, stats, newJobs, removedJobs).

On the very first run (no previous file or unparseable file) previous === null and both diffs are treated as empty (sections are omitted entirely). Don't change this — it prevents the first run from declaring "all 900 jobs are new" when there's no baseline.

The read happens after filter+dedup+sort but before writeJson('data/jobs.json', ...), so the new file overwrites the old one cleanly.

RSS feed

src/feed.ts emits a hand-rolled RSS 2.0 XML to data/feed.xml containing the top 50 newJobs by fitScore. The XML is hand-built (not via fast-xml-parser) because we control the content shape — escapeXml covers the five entity classes. The feed metadata (title, description, link) is overridable via PUPILA_FEED_TITLE / PUPILA_FEED_DESC / PUPILA_FEED_LINK env vars. To subscribe locally, point your RSS reader at the file:// path of data/feed.xml. (No remote URL anymore — the project is local-first.)

Salary parsing

src/salary.ts parseSalary(raw) returns { min, max, currency }, normalizing to annual integers (USD/EUR/etc., minor units stripped). It handles $120K-$180K, €80,000 - €110,000, 100K-150K USD, hourly via 2080 annualization, M-suffixed ($1M-$2M), single-value salaries, currency code or symbol detection, and rejects sub-$1000 amounts as noise. Returns { null, null, null } for free-text like "competitive". Hooked into src/normalize.ts via a withSalary() spread so adding a future salary-emitting source is one line.

Job.salary (raw string for display) and Job.salaryMin / salaryMax / salaryCurrency (parsed) are both populated. The display column in JOBS.md keeps using salary; future filtering/sorting can use the numeric fields.

Source-health alarms

src/render.ts flags fetchers whose fetched === 0 OR errors > 0 in the current run: a 🚨 banner appears above the Stats section, and the offending sources get a 🚨 prefix in the by-source list. Single-run signal only — no historical tracking. Catches silent breakage (web3career and aijobsnet markup changes have hit us before). If a normally-quiet source legitimately returns 0 (e.g. cryptojobslist while upstream is down), it'll alarm — that's still useful surfacing.

GitHub Actions

One workflow only — the project moved to local-first scheduling.

  • .github/workflows/check.yml — every push to main and every PR. Six gates: Biome lint, typecheck (3 tsconfigs), Vitest, tsc build, Vite UI build, pnpm audit. permissions: contents: read.

The previously included jobs.yml (daily aggregator cron) and keepalive.yml (cron-keepalive) workflows were removed. Daily aggregation now runs locally via scripts/install-launchd.sh (macOS) or scripts/install-cron.sh (Linux), which install two agents: one for pnpm run dev, one for pnpm run ai-review. See README "Schedule the daily run" for usage.

Pinning. All third-party actions are referenced by full 40-char commit SHA, not a floating @v4 / @v5 tag, with the version in a trailing comment. When updating an action, replace both the SHA and the comment manually.

Tests

Vitest, 120 cases across 10 files in tests/ with *.test.ts glob. Run via pnpm test (CI) or pnpm run test:watch (interactive).

  • tests/utils.test.ts (20): isSafeUrl allowlist, normalizeUrl (utm strip, scheme reject), stripHtml, normalizeText, sha1Hex, relativeTime, withinDays, formatDateTimeUTC (human-readable "DD Month YYYY, HH:MM UTC").
  • tests/normalize-hn.test.ts (7): normalizeHnHiring header extraction with | and em-dash separators, plausible-company guard (rejects whole-body leak, sentence-like candidates, >60 char strings), role-pattern fallback when no header is present.
  • tests/filters.test.ts (33): every hard-drop branch (junior, senior_req, US-only, compound non-eng, non-frontend eng, non-eng role, non-tech role, exec, URL scheme), droppedByRule rule attribution, every score signal, category derivation, score capping with _signals.capped, US-centric penalty, plural title acceptance, frontendTitle/frontendBody bonuses, boilerplate stripping (AI keywords in EEO footer must NOT score), tiered keyword weighting (1 mention = half-weight, 4+ = 1.5× boost across stackPrimary and frontendBody).
  • tests/dedup.test.ts (10): id collapse, normalized company+title collapse, fitScore tiebreak, source priority tiebreak (ashby > greenhouse > remoteok, lever > web3career), empty input, plus 5 cases for compareJobs covering the four-key chain (fitScore desc → salaryMax desc with null-as-zero → postedAt desc → id asc).
  • tests/applied.test.ts (4): STATUS_EMOJI map presence, summarizeApplied empty input, grouping/counting, ordering (offer → interview → applied → withdrawn → rejected).
  • tests/salary.test.ts (15): K/M suffix parsing, comma-grouped numbers, currency symbol vs code detection (USD/EUR/GBP/CAD), hourly→annual conversion via 2080 hours, single-value salaries, sub-$1000 rejection, free-text fallback, range inversion handling.
  • tests/feed.test.ts (6): RSS 2.0 skeleton + channel metadata, item title with fit score prefix, XML escaping (&, <, >), fitScore desc ordering, 50-item cap, salary surfaced in description.
  • tests/aave.test.ts (7): __NEXT_DATA__ extraction from raw HTML and normalizeAave mapping (URL build, body strip, postedAt, location/remote inference).
  • tests/ashby-private.test.ts (9): GraphQL parseListResponse / parseDetailResponse shape handling and normalizeAshbyPrivate slug-to-company derivation (chainlink-labs → "Chainlink Labs", matter-labs → "Matter Labs"), brief-only fallback when detail is null, stable id derivation.

When tuning a filter regex or scoring weight (in config/profile.json or filters.ts), update the test in the same commit. The check.yml workflow runs the full suite on every PR.

tsconfig.test.json extends tsconfig.json with rootDir: "." and include: ["src/**/*", "tests/**/*"]. The pnpm typecheck script runs both: first tsc --noEmit against tsconfig.json (the production build config) to catch issues that would block compilation, then tsc --noEmit -p tsconfig.test.json to typecheck the tests without leaking them into the build's rootDir.

Local UI (pnpm run ui)

UI rules live in ui/AGENTS.md (nested context, auto-loaded by Codex/Cursor when working in ui/). Two hard rules summarized: (1) every class comes from a co-located *.module.css import — never write a class name as a string literal; (2) every server call goes through ui/src/lib/api/ — never write fetch('/api/...') at a call site. Both are also enforced by pnpm run lint + the pre-commit hook.

A Vite + React 19 dashboard at ui/ that reads data/jobs.json and data/ai-reviews.json directly via JSON import. Local-only — no auth, no hosting, intentionally not exposed beyond 127.0.0.1:5173 (the user explicitly chose this over public Pages because a public dashboard surfacing their applied-job statuses could be Google-indexed and visible to recruiters). Don't add a pnpm run ui:deploy or wire it into a workflow without explicit instruction.

Single-component MVP (no router, no state-management lib): filter chips for category/source/applied, search box, sortable columns (score / salaryMax / postedAt), dark-mode via prefers-color-scheme. Score cells are tier-colored (green ≥80, gold 50-79, muted <50). Long company/title cells are clamped to 2 lines via display: -webkit-box (the clamp is on a <span> wrapper inside the <td> — applying it directly on the <td> breaks table-cell layout).

Expandable rows. Clicking a row opens a 3-column detail panel with: the LLM "AI take" (summary, verdict reason, wants/offers/red-flags) when data/ai-reviews.json has an entry for that job; the _signals score breakdown showing which scoring rules fired and what they contributed; meta (location, tags, posted date, id). The Apply link in the row uses e.stopPropagation() so it doesn't trigger expand. Verdict badge (strong-match / match / weak-match / skip) appears next to the title when an AI review exists.

Group by company (default on). When the Group by company checkbox is on, jobs fold by lower-cased company field. Single-job "groups" render flat (no header noise) — only multi-job companies get a collapsible header row showing top score + role count + a one-line preview of the top role. Click the header to reveal the rest. The active sort key (fitScore / salaryMax / postedAt) drives both the within-group order and the inter-group order (using each group's top job as the comparator). Toggle off for a flat view.

URL-encoded state. All filter / sort / group / expand state syncs to window.location.search via history.replaceState (no back-button spam). Keys: q (search), cat, src, applied=1, sort, dir=asc, group=0 (only when off, since on is the default), expanded=<jobId>, co=<lowercased-company>. Defaults are omitted from the URL to keep it short. On load the App reads the URL once via a useMemo lazy initializer; afterwards a single useEffect writes back any state change. Don't use pushState — every keystroke would create a history entry.

Mark-as-applied from the UI. The expanded detail panel has an "Applied" bar with status pills (📝 applied / 💬 interview / 🎯 offer / ❌ rejected / ⏸ withdrawn) plus a notes input. Clicking a pill toggles the status; clicking the active pill again clears it. Notes save on blur or Enter. All edits go through a small Vite middleware (appliedApiPlugin in ui/vite.config.ts) exposing GET / POST / DELETE /api/applied, which reads/writes config/applied.json directly. The UI uses optimistic updates (the pill flips immediately; a failure rolls back and surfaces a banner). On mount, the App fetches /api/applied and reconciles with the baked-in j.applied field from jobs.json, so hand-edits or changes from a prior session show up correctly. The middleware only runs under pnpm run ui — a static pnpm run ui:build preview won't have it (the UI silently falls back to the baked-in state). config/applied.json is the same file loadAppliedMap reads in pnpm run dev, so subsequent pipeline runs pick up UI edits with no extra plumbing. The user must git add config/applied.json && git commit to keep edits across machines / a fresh checkout — the project is local-first now, so nothing is auto-pushed anywhere.

ui/src/types.ts is a deliberate copy of the relevant subset of src/types.ts (Job, JobSignals, AppliedEntry, AiReview, AiReviews). The pipeline strips body from the persisted data/jobs.json, but _signals is kept so the UI can render it without re-running scoring. Don't rewrite ui/src/types.ts to import from src/types.ts — that pulls in with { type: 'json' } config imports that don't resolve in the browser context.

The HTML has <meta name="robots" content="noindex,nofollow"> as belt-and-suspenders even though it's local-only.

pnpm run typecheck runs all three TS configs: tsconfig.json, tsconfig.test.json, and ui/tsconfig.json.

AI per-job review (pnpm run ai-review)

src/ai-review.ts is a local-only companion to the daily pipeline that augments selected jobs with an LLM review. It shells out via src/lib/llm.ts (auto-detects Codex / codex / gemini / opencode on PATH, override with PUPILA_LLM=<provider>). The CLI uses the user's local subscription (e.g. Codex Max) — not an API key, so there are no per-token charges. With the new local-first scheduling, the launchd/cron review agent runs this every day at 07:15 by default. If you don't have an LLM CLI installed, run scripts/install-launchd.sh --no-review (or the cron equivalent) to skip the review step.

Inputs:

  • data/jobs.json — the slim list (committed)
  • data/jobs-bodies.json — sidecar with full bodies (gitignored, regenerated by pnpm run dev so the AI step has the body to review)
  • data/ai-reviews.json — existing reviews (committed; starts as {})
  • config/candidate-brief.md — natural-language description of who the candidate is and what they're avoiding. Hand-edited. The prompt embeds it verbatim, so this is the main lever for tuning what "match" / "skip" actually mean.

Output: data/ai-reviews.json, a Record<jobId, AiReview> map. Each AiReview carries a one-sentence summary, 3 bullets each for wants / offers / redFlags, a verdict (strong-match | match | weak-match | skip), and a one-sentence reason explaining the verdict (especially when the LLM disagrees with the rule-based fitScore). The script writes after every successful review so a Ctrl-C or rate-limit kill leaves a partial-but-valid file.

Selection logic. Default: top 20 by fitScore that aren't already in ai-reviews.json. Reviews for jobs no longer in jobs.json are pruned automatically each run. CLI flags: --top=N, --force (re-review existing), --ids=a,b,c (specific ids).

Prompt → JSON parsing. The LLM occasionally wraps its JSON in markdown fences despite explicit instructions; src/ai-review-parse.ts strips them and falls back to safe defaults for missing/invalid fields rather than throwing — partial reviews are still useful. Tests in tests/ai-review-parse.test.ts (9 cases) cover fenced JSON, invalid verdicts, missing fields, dirty arrays, and malformed input.

Daily workflow:

pnpm run daily       # = `pnpm run dev && pnpm run ai-review` — fetch + write data/jobs.json + ai-reviews.json
pnpm run ui          # browse with verdicts + score breakdowns inline
git add data/jobs.json data/feed.xml data/ai-reviews.json JOBS.md
git commit -m "chore: daily run + ai reviews"

config/candidate-brief.md ships with a starter that should be edited once and then left alone. It's the only natural-language config in the repo (everything else is JSON / TS).

Pre-commit

simple-git-hooks is registered via the prepare lifecycle script. The hook runs:

pnpm run lint && pnpm run typecheck

Bypass with SKIP_SIMPLE_GIT_HOOKS=1 git commit ... when you really need to land a WIP. Don't make a habit of it.

Security checklist for new fetchers / parsers

  • Use fetchWithTimeout from utils.ts (timeout + retry + abort built-in).
  • Do not embed user-controllable strings in HTML attributes; use escapeHtmlAttr in render.ts.
  • All scraped URLs flow through the filter's isSafeUrl gate — don't bypass it.
  • All scraped bodies flow through stripHtml before any regex/scoring.
  • New external HTTP endpoints get added to a tier-S slug list when applicable; ad-hoc URLs in code should be reviewed.

Conventional commits

Use the conventional commits style:

  • feat: new fetcher / filter rule / output section
  • fix(fetchers): upstream URL change, parser bug
  • chore: housekeeping, scaffolding
  • ci: workflow changes
  • docs: README/AGENTS.md changes

Known upstream issues (as of 2026-04)

  • cryptojobslist.com is fully Cloudflare-challenged for HTML and the api.cryptojobslist.com/jobs.rss endpoint currently returns an empty channel. The fetcher gracefully returns []; will pick up jobs again if upstream restores the feed.
  • All 5 web3 holdouts from the original spec are now covered. morpho, magiceden, li.fi were on the public Ashby posting-API after all (added to config/slugs.json#ashby). aave is scraped via the Next.js __NEXT_DATA__ blob on aave.com/careers (src/fetchers/aave.ts). chainlink-labs is scraped via Ashby's private non-user-graphql endpoint at jobs.ashbyhq.com/api/non-user-graphql (src/fetchers/ashby-private.ts, slug list in config/slugs.json#ashbyPrivate). The Greenhouse stale-slug list was reduced from 14 → 8 by removing the always-404 entries (aave, chainlink, morpho, lifi, magiceden, ledger). A 100-candidate sweep across web3 / AI / dev-tools tier-S companies turned up no other orgs using the Ashby-private pattern — chainlink-labs appears unique. The fetcher is config-driven anyway, so adding a future hit is a single-line edit.
  • web3.career and aijobs.net (formerly ai-jobs.net) removed RSS — both are scraped from HTML via small inline regex parsers, which means a markup change upstream will silently break them. If a fetcher returns 0 for several days, eyeball the HTML for new selectors.
  • aijobs.net is dominated by spam-aggregator listings (one posting cloned to 50 cities). The fetcher dedups by base ID via the -idNNNNN- slug pattern, which collapses an entire page down to 2–5 distinct postings. Don't be alarmed at the low kept count.
  • hn-jobs routinely keeps 0–2 entries because YC company posts rarely match the senior+stack signal threshold. Filtering working as intended, not a bug.