Automated, distributed open job index and labor-market dataset across 68 enterprise ATS platforms and 24 job feeds.
RoleBase continuously aggregates, enriches, and archives open job vacancies directly from over 112,600+ employer career portals across 68 applicant tracking systems (Workday, Greenhouse, Lever, Ashby, SmartRecruiters, ADP, UKG Pro, Dayforce, Paylocity, Oracle HCM, Hireology, Feishu/Lark, Moka HR, Beisen i-Talent, Baidu, Tencent, ByteDance, NetEase, JD.com, Unitree, and more) plus 24 curated job feeds and governmental portals (German Federal Employment Agency Arbeitsagentur, Singapore WSG MyCareersFuture, FreeHire, etc.).
Every posting is parsed and normalized with seniority level, professional field, required years of experience, yearly salary ranges, and technical/domain skills. Historical job closures are tracked to detect stale listings and ghost jobs.
- π Live Web Search: chandranshb.github.io/roleBase
- π Kaggle Relational Dataset: kaggle.com/datasets/chandranshbinjola/job-listings
- π Starter Analysis Notebook: Job Listings Explorer
- π€ AI Agent & Developer Guide: AGENTS.md Β· ARCHITECTURE.md
- Reverse-Engineered ATS Protocol Support: Direct native APIs and decrypted endpoints across 68 ATS vendors, including dynamic AES-128-CBC payload decryption for Moka HR, automated path probing for Feishu/Lark Hire, NFKC Unicode normalization for Kangxi radicals, and direct enterprise career feeds.
- 10M+ Active Listings: Deep index refreshed every 6 hours across 16 parallel runner shards; not restricted to tech (covers healthcare, manufacturing, finance, retail, logistics, education, and legal).
- Early-Career ("Fresher-Friendly") Filter: Isolates roles genuinely accessible to candidates with 0β2 years of experience; automatically flags and filters out "junior" titles quietly asking for 3+ years.
- Ghost Job Risk Scoring: Transparent heuristic audit flagging perpetual talent pools, undisclosed staffing agencies, serial reposts, and listings open >6β12 months.
- Longitudinal Labor Market Metrics: Daily snapshots tracking global open job volume, early-career hiring share, and daily demand shares across 80+ technical and non-technical skills.
- 100% Serverless & Open: Fully automated on GitHub Actions; persists state in Git, publishes static indices to GitHub Pages, and updates Kaggle daily.
| File | Description | Key Fields |
|---|---|---|
jobs.csv |
Primary job database (open & closed <90 days) | url, company, title, level, category, years_experience, skills, pay_min, pay_max, pay_currency, remote, ghost_score, first_seen, closed_at |
jobs_by_company.csv |
Early-career hiring index by employer (5+ roles) | company, open_roles, fresher_roles, fresher_share, junior_asking_3plus, years_stated_share |
open_jobs_daily.csv |
Daily active and fresher job counts | date, open_jobs, fresher_friendly_jobs |
skill_demand_daily.csv |
Skill frequency time series | date, skill, jobs, tagged, share_pct |
One compact, sorted JSONL file per employer board. Empty and derivable values are omitted to minimize storage and diff size:
{
"id": "greenhouse:stripe:12345",
"company": "Stripe",
"title": "Software Engineer, Infrastructure",
"location": "Seattle, WA",
"country": "United States",
"remote": true,
"level": "senior",
"category": "engineering",
"employment": "full_time",
"years": 5,
"skills": ["python", "aws", "kubernetes"],
"pay": [180000, 240000, "USD"],
"posted": "2026-11-01",
"first_seen": "2026-11-01",
"closed_at": null,
"reposts": 0
}flowchart LR
A["68 ATS Platforms & 24 Feeds<br/>(Workday, Feishu, Moka, etc.)"] --> B["16 Parallel Shards<br/>(scrape.yml matrix 0..15)"]
B --> C["Consolidation Engine<br/>(scraper.py --consolidate)"]
C --> D[("data/jobs/*.jsonl<br/>data/pulse.csv<br/>data/skills.csv")]
D --> E["Site & Pages Generator<br/>(build_site.py, build_pages.py)"]
D --> F["Dataset Exporter<br/>(export_kaggle.py)"]
E --> G["GitHub Pages<br/>chandranshb.github.io/roleBase"]
F --> H["Kaggle Datasets & Notebooks"]
- Crawl Pipeline: 16 parallel runner shards partition the 112,600+ employer boards with per-provider rate limiting and lane concurrency (
LANES). - Consolidation: Downloads shard artifacts, prunes jobs closed >90 days into
data/archive.csv, and precomputes hiring analytics intodata/charts_cache.json. - Zero-RAM Bottleneck: Processing loops stream files using chunked iterators to operate strictly within standard GitHub Actions runner memory limits (<7 GB).
- Python 3.10+
- Git
git clone https://github.com/chandranshB/roleBase.git
cd roleBase
pip install -r requirements-dev.txt# Run unit test suite
python -m unittest discover tests -v
# Run legacy integration checks
python test_scraper.py
# Run linter
ruff check rolebase tests# Run single-shard crawl test
python scraper.py --shard 0 --total-shards 16
# Consolidate artifacts & precalculate analytics
python scraper.py --consolidate
# Generate static search indices (_site/)
python build_site.py
# Fetch company icons (cached in logo-cache/)
python build_logos.py
# Generate static SEO HTML pages
python build_pages.py
# Export Kaggle relational CSVs (kaggle/)
python export_kaggle.py- Find the employer's ATS identifier:
- Greenhouse / Lever / Ashby / SmartRecruiters: board slug from career page URL (e.g.
boards.greenhouse.io/<slug>). - Workday:
<tenant>.<pod>.myworkdayjobs.com/<site>(e.g.adobe.wd5.myworkdayjobs.com/external_experienced).
- Greenhouse / Lever / Ashby / SmartRecruiters: board slug from career page URL (e.g.
- Add
"slug": "Company Name"under the respective ATS key incompanies.json. - (Optional) Add the company's domain to
domains.jsonfor high-resolution logo resolution.
- Official First-Party Sources Only: Postings are fetched exclusively from public company ATS feeds and open APIs. Aggregators that restrict scraping (LinkedIn, Indeed, Naukri) are omitted on purpose.
- No Wall Gardens: Every search result links directly back to the employer's official job portal.
- Polite Crawling: Per-provider concurrency limits (
LANES), exponential backoff on HTTP 429/5xx, and descriptive User-Agent headers.
This project is licensed under the GNU Affero General Public License v3.0. Company names, logos, and trademarks belong to their respective employers and are used strictly for identification purposes.