BD News Scraper

BD News Scraper is a full-stack application that scrapes the day's featured or most-read news from major Bangladeshi news sites and displays them in a clean, modern web interface.

Features

• Multiple Sources: Scrapes news from Prothom Alo, The Daily Star, Amar Desh, and more.
• Dynamic Scraping: Uses BeautifulSoup for static sites and Playwright for JavaScript-heavy sites.
• React Frontend: A responsive and user-friendly interface built with Vite, React, and TypeScript.
• Easy to Run: Simple CLI commands to run the scraper and the frontend.
• Extensible: Easily add new parsers for other news sites.

Tech Stack

• Backend: Python, BeautifulSoup, Playwright
• Frontend: React, TypeScript, Vite
• Styling: CSS Modules

Setup and Usage

Backend (Scraper)

1. Set up a virtual environment:

   cd ~/Projects/bd-news-scraper
   python3 -m venv .venv
   source .venv/bin/activate

2. Install dependencies:

   pip install -r requirements.txt

3. For amar_desh parser, install Playwright browsers:

   pip install playwright
   python -m playwright install chromium

4. Run the scraper:

   # Scrape all sites and save to output/news.json
   python -m scraper
   
   # Scrape a specific site
   python -m scraper --site prothom_alo

Frontend (React App)

1. Navigate to the frontend directory:

   cd frontend

2. Install dependencies:

   npm install

3. Sync the scraped data: This command copies the scraper's output to the frontend's public directory.

   npm run sync-data

4. Run the development server:

   npm run dev

   The application will be available at http://localhost:5173.

Project Structure

• scraper/: Contains the Python web scraping logic.
  • parsers/: Individual parsers for each news site.
  • models.py: Data models for articles.
  • __main__.py: CLI entrypoint.
• frontend/: The Vite + React frontend application.
  • src/: Frontend source code.
  • public/: Static assets, including the news.json data file.
• output/: Default output directory for the scraper.

Contributing

Contributions are welcome! Please see the CONTRIBUTING.md file for guidelines on how to contribute to this project.

License

This project is licensed under the MIT License. See the LICENSE file for details.