Data Pipeline Demo

A lightweight demonstration of a job aggregation & enrichment pipeline. Built to mirror how modern data companies collect, structure, and serve real-time market intelligence.

What it does

• Scrapes live job postings
• Auto-detects mentioned tech stacks (Python, FastAPI, LangChain, etc.)
• Stores structured data in SQLite
• Exposes a FastAPI endpoint with search + tech filtering

Stack

• Python 3.14
• FastAPI + Uvicorn (API server)
• BeautifulSoup + requests (scraping)
• SQLite (database)

How to run

# 1. Activate virtual environment
.\venv\Scripts\activate

# 2. Initialize database
python database.py

# 3. Run scraper
python scraper.py

# 4. Start API server
python main.py

# 5. Test endpoints
curl http://127.0.0.1:8000/health
curl "http://127.0.0.1:8000/jobs?tech=Python"