Agrolead

🌱 Discover and score European agricultural importers, wholesalers, and distributors automatically.

Agrolead is a production-ready Python application that:

• Discovers companies from B2B directories and websites
• Enriches data with contact information and social media links
• Scores leads based on relevance to agricultural imports
• Exports high-quality CRM-ready databases

Features

Web Crawling

• Multi-source B2B directory adapters (Europages, Kompass, etc.)
• Direct website crawling with JavaScript support
• Async concurrent crawling for performance
• Intelligent rate limiting and retry logic
• URL caching to avoid revisits

Data Enrichment

• Automatic email and phone extraction
• Social media link discovery
• Language detection
• Contact page and about page detection
• Certification identification (GlobalG.A.P., ISO, HACCP, etc.)

Lead Scoring

• Configurable scoring weights
• Multi-factor relevance assessment
• Confidence scoring
• Quality thresholds
• Keyword matching for products

Data Management

• SQLite development database
• PostgreSQL production ready
• Deduplication by website, email, name
• Activity logging and tracking
• URL caching system

Export Functionality

• CSV export (CRM-ready)
• Excel with formatting
• JSON structured data
• SQLite database export
• Batch export with chunking

CLI Interface

• Intuitive command-line interface
• Real-time progress indicators
• Rich table output
• Configuration management

Installation

Prerequisites

• Python 3.12+
• pip or poetry

Setup

1. Clone the repository:

cd Agrolead

2. Install dependencies:

pip install -r requirements.txt

3. Install Playwright browsers:

playwright install chromium

4. Initialize the application:

python -m agrolead.cli.main init

Configuration

Create a .env file in the project root:

# Database
DB_TYPE=sqlite
DB_SQLITE_PATH=data/agrolead.db
DB_ECHO=false

# Crawler
CRAWLER_MAX_CONCURRENT_TASKS=5
CRAWLER_REQUEST_TIMEOUT=30
CRAWLER_RATE_LIMIT_DELAY=1.0
CRAWLER_MAX_PAGES_PER_DOMAIN=50

# Enrichment
ENRICHMENT_ENABLED=true
ENRICHMENT_MAX_PAGES_TO_VISIT=5

# Scoring
SCORING_SCORE_IMPORTER=25
SCORING_SCORE_FRESH_PRODUCE=20
SCORING_SCORE_TOMATO_MENTION=20
SCORING_MINIMUM_SCORE=20

# Export
EXPORT_FORMATS=csv,excel,json
EXPORT_MAX_ROWS_PER_FILE=10000

# Logging
LOG_LEVEL=INFO

Usage

Search for companies

python -m agrolead.cli.main search "tomato importer" --directory europages --country FR --limit 50

Crawl companies

python -m agrolead.cli.main crawl --directory europages --max-pages 100

Enrich company data

python -m agrolead.cli.main enrich --max-companies 50

Score leads

python -m agrolead.cli.main score

Export data

python -m agrolead.cli.main export --format excel
python -m agrolead.cli.main export --format csv
python -m agrolead.cli.main export --format json

View statistics

python -m agrolead.cli.main stats

List available sources

python -m agrolead.cli.main list-sources

Project Structure

agrolead/
├── agrolead/
│   ├── app/                 # Application entry points
│   ├── config/
│   │   └── settings.py      # Configuration management
│   ├── crawler/
│   │   ├── base_adapter.py  # Base adapter class
│   │   └── sources/
│   │       └── adapters.py  # Directory-specific adapters
│   ├── database/
│   │   └── models.py        # SQLAlchemy models
│   ├── enrichment/
│   │   └── enricher.py      # Website enrichment
│   ├── scoring/
│   │   └── scorer.py        # Lead scoring
│   ├── export/
│   │   └── exporter.py      # Data export
│   ├── models/
│   │   └── company.py       # Pydantic models
│   ├── cli/
│   │   └── main.py          # CLI interface
│   └── utils/
│       └── helpers.py       # Utility functions
├── tests/                   # Unit tests
├── data/                    # Database files
├── logs/                    # Application logs
├── requirements.txt
└── README.md

Creating Custom Directory Adapters

Create a new adapter by extending BaseAdapter:

from Agrolead.crawler.base_adapter import AdapterConfig, HTTPAdapter

class MyDirectoryAdapter(HTTPAdapter):
    def __init__(self):
        config = AdapterConfig(
            name="my_directory",
            base_url="https://example.com",
            rate_limit_delay=2.0,
        )
        super().__init__(config)
    
    async def search(self, query, country=None, limit=50):
        # Implement search logic
        pass
    
    async def crawl(self, start_urls, max_pages=None):
        # Implement crawl logic
        pass
    
    async def parse_company(self, url):
        # Implement parsing logic
        pass

Register your adapter in agrolead/crawler/sources/adapters.py:

ADAPTERS = {
    "my_directory": MyDirectoryAdapter,
}

Scoring System

Companies receive scores 0-100 based on:

Factor	Points	Condition
Importer	25	Listed as importer
Fresh Produce	20	Handles fresh produce
Tomato Mention	20	Specifically mentions tomatoes
Public Email	10	Email address available
Contact Page	10	Has dedicated contact page
Export Mention	10	Mentions exports/imports
European	5	Located in target country
Certifications	15	GlobalG.A.P., ISO, HACCP, etc.

Data Models

Company Model

class Company(BaseModel):
    name: str                           # Company name
    website: Optional[str]              # Primary website
    country: CountryEnum               # Country of operation
    city: Optional[str]                # City
    industries: List[IndustryEnum]    # Business categories
    products: ProductInfo              # Product details
    contact: ContactInfo               # Contact information
    final_score: float                 # Lead score 0-100
    source_directory: str              # Source adapter name
    source_url: str                    # Direct source URL
    date_crawled: datetime             # Crawl timestamp

Performance Considerations

• Async Concurrency: Supports up to 10 concurrent crawl tasks
• Rate Limiting: Configurable delays between requests
• Caching: Visited URLs are cached to prevent duplicates
• Database Indexing: Optimized queries on score, country, date
• Batching: Efficient batch operations for scoring and export

Production Deployment

PostgreSQL Setup

pip install psycopg2-binary

Set environment variables:

DB_TYPE=postgresql
DB_POSTGRESQL_URL=postgresql://user:password@localhost/Agrolead

Monitoring

Enable SQL logging:

DB_ECHO=true

Scheduling

Use cron or similar for regular crawling:

# Daily crawl at 2 AM
0 2 * * * cd /path/to/agrolead && python -m agrolead.cli.main crawl

Ethical Considerations

Agrolead respects:

• ✅ robots.txt rules
• ✅ Rate limiting guidelines
• ✅ Terms of service
• ✅ Public data only (no LinkedIn scraping)
• ✅ No authentication bypass

Testing

Run tests:

pytest tests/
pytest tests/ --cov

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Submit a pull request

License

MIT License - See LICENSE file for details

Support

For issues and questions:

• Open an issue on GitHub
• Check existing documentation
• Review example configurations

Roadmap

• Machine learning company classification
• API interface (FastAPI)
• Web dashboard (Streamlit)
• Email verification
• Company size detection
• Export market analysis
• Email campaign integration
• Slack notifications

Changelog

v1.0.0 (Initial Release)

• Core crawling infrastructure
• Europages and Kompass adapters
• Website enrichment system
• Lead scoring system
• CSV/Excel/JSON export
• CLI interface
• SQLite and PostgreSQL support

---

Developer

m223rx – 2026

© 2026 m223rx. All rights reserved.

---

Made with 🌱 for agricultural businesses by m223rx