Skip to content

refactor: convert bsoup into an importable package with public API, full test coverage, and CI#1

Merged
lipnelz merged 4 commits into
mainfrom
copilot/review-architecture-improvements
Mar 14, 2026
Merged

refactor: convert bsoup into an importable package with public API, full test coverage, and CI#1
lipnelz merged 4 commits into
mainfrom
copilot/review-architecture-improvements

Conversation

Copilot AI commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Single-file bsoup.py had no package structure, no importable API, global mutable state, and zero tests. This PR restructures it as a proper Python package, exposes a clean async library interface, and adds a GitHub Actions CI pipeline.

Package structure

Path Role
bsoup/scraper.py Core logic — Scraper class + ScrapeResult dataclass
bsoup/cli.py CLI entry-point (main(), get_output_path())
bsoup/__init__.py Public exports
bsoup/__main__.py python -m bsoup support

Architecture changes

  • Removed module-level semaphore — created per-batch inside Scraper.scrape() instead
  • parse_page was async with no awaits — renamed to synchronous parse_html
  • All tuning knobs (max_connections, decimal_sep, request_timeout, overall_timeout, retries) are now Scraper constructor params

Public library API

import asyncio
from bsoup import Scraper, ScrapeResult

scraper = Scraper(decimal_sep=',')
results: list[ScrapeResult] = asyncio.run(scraper.scrape([
    ("https://www.boursorama.com/cours/historique/1rPEN", "BOUYGUES", 1),
]))
# r.name, r.daily_value, r.max_value, r.max_date, r.min_value, r.min_date
# r.to_csv_line(decimal_sep=',')

Tests — 100 % coverage (40 tests)

File Scope
tests/test_parser.py parse_html edge cases, ScrapeResult.to_csv_line
tests/test_fetcher.py Retry logic, semaphore, HTTP errors
tests/test_scraper.py scrape/scrape_to_csv, timeout cancellation, task exceptions
tests/test_cli.py All CLI paths, error handling, cross-platform Desktop resolution

pyproject.toml enforces fail_under = 90.

CI — GitHub Actions

Added .github/workflows/ci.yml which runs automatically on every push and pull request targeting main:

  • Matrix across Python 3.10, 3.11, and 3.12
  • Installs dependencies from requirements.txt
  • Runs pytest --cov=bsoup — fails the build if coverage drops below 90 %
  • Uploads coverage.xml as a workflow artifact (retained 7 days)
Original prompt
  • Review code, propose architecture improvements
  • split commits with conventional commits style.
  • Think of a public interface to be plugged in an external python program.
    Update readme
  • Add code coverage of at least 90%

💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

Copilot AI and others added 2 commits March 14, 2026 10:41
…ts, and updated docs

Co-authored-by: lipnelz <174376617+lipnelz@users.noreply.github.com>
…tefacts

Co-authored-by: lipnelz <174376617+lipnelz@users.noreply.github.com>
Copilot AI changed the title [WIP] Propose architecture improvements and update readme refactor: convert bsoup into an importable package with public API and full test coverage Mar 14, 2026
Copilot AI requested a review from lipnelz March 14, 2026 10:43
@lipnelz lipnelz marked this pull request as ready for review March 14, 2026 11:58
@lipnelz

lipnelz commented Mar 14, 2026

Copy link
Copy Markdown
Owner

@copilot add test automatically run in pull request CI

Co-authored-by: lipnelz <174376617+lipnelz@users.noreply.github.com>

Copilot AI commented Mar 14, 2026

Copy link
Copy Markdown
Contributor Author

@copilot add test automatically run in pull request CI

Added in commit c723596.github/workflows/ci.yml runs pytest --cov=bsoup on every push and pull request targeting main, across Python 3.10, 3.11, and 3.12. The coverage XML report is uploaded as a workflow artifact (retained 7 days).

Copilot AI changed the title refactor: convert bsoup into an importable package with public API and full test coverage refactor: convert bsoup into an importable package with public API, full test coverage, and CI Mar 14, 2026
@lipnelz lipnelz merged commit de5edd7 into main Mar 14, 2026
3 checks passed
@lipnelz lipnelz deleted the copilot/review-architecture-improvements branch March 14, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants