Scrape websites from .env file#1
Conversation
This commit modifies the backend to scrape a list of websites mentioned in the .env file instead of performing a broad web search. The key changes include: - A new `WEBSITES` variable in the `.env` file to specify the list of websites to be scraped. - The `web_search` agent is updated to use the `WEBSITES` variable. - A new service is added to save the scraped product data to a JSON file with a unique timestamped filename. - The `README.md` file is updated with a sample `curl` command to test the end-to-end flow. - The configuration handling is made more robust by using `pydantic-settings` and a reliable path to the `.env` file.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. New to Jules? Learn more at jules.google/docs. |
This commit refactors the backend to use the `crawl4ai` package for web scraping and data extraction, replacing the previous implementation that used `requests` and `playwright`. This change simplifies the codebase by consolidating the scraping and extraction logic into a single, more efficient agent. Key changes: - Replaced `requests` and `playwright` with `crawl4ai` for web scraping. - Created a unified scraper agent that handles both scraping and LLM-based data extraction. - Updated the orchestrator to use the new unified scraper agent. - Removed the old, redundant `web_scraper.py` and `product_extractor.py` files. - Updated the `Product` schema with descriptive comments to improve the accuracy of the LLM-based data extraction. - Updated the `requirements.txt` file to include `crawl4ai` and remove `playwright` and `selectolax`. - Addressed a nitpick from a previous code review to use `model_dump()` instead of the deprecated `dict()` method.
This commit updates the README.md file to reflect the latest changes to the project, including the new architecture and installation instructions. It also removes the b2c-journey-prototype directory, which is no longer needed.
This change modifies the backend to scrape a list of websites mentioned in the .env file instead of performing a broad web search. It also includes updating the README.md file with sample curl commands to test the end to end flow.
PR created automatically by Jules for task 10200566754231567707 started by @git-hari-j