PDF Text Extraction and Arabic Term Definition Library

This library provides a complete workflow for extracting text from PDF documents, processing Arabic content, and creating term-definition datasets. It's designed for researchers working with Arabic texts and PDF documents.

📋 Table of Contents

• Overview
• Files Description
• Installation
• Usage
• Workflow
• Requirements
• API Keys Setup

🎯 Overview

This toolkit consists of four Python scripts that work together to:

1. Extract text from PDF files (both text-based and scanned)
2. Process Arabic text to extract terms and definitions using AI
3. Combine multiple CSV files into a single shuffled dataset

📁 Files Description

1. pdf_extractor.py

Purpose: Extracts text from text-based PDF files using PyPDF2.

What it does:

• Reads PDF files that contain selectable text
• Extracts text from each page
• Saves the extracted pages as a pickle file for further processing

When to use: Use this for PDFs that already contain text (not scanned images).

Usage:

python pdf_extractor.py <PDF_PATH>

Example:

python pdf_extractor.py document.pdf

Output: Creates a pickle file named <filename>_pages.pkl containing a list of page texts.

---

2. ocr_extractor.py

Purpose: Extracts text from scanned PDF files or image-based PDFs using Mistral AI OCR.

What it does:

• Uploads PDF files to Mistral AI OCR service
• Performs Optical Character Recognition (OCR) on scanned documents
• Extracts text from each page
• Saves the extracted pages as a pickle file

When to use: Use this for scanned PDFs, image-based PDFs, or documents where pdf_extractor.py fails to extract text.

Usage:

python ocr_extractor.py <PDF_PATH>

Example:

python ocr_extractor.py scanned_document.pdf

Output: Creates a pickle file named <filename>_texts.pkl containing a list of page texts.

Note: Requires a Mistral API key (see API Keys Setup).

---

3. create_defs.py

Purpose: Extracts Arabic terms and their definitions from text using OpenAI's GPT models.

What it does:

• Reads a pickle file containing page texts
• Sends each page to OpenAI API for Arabic term extraction
• Extracts terms and definitions in JSON format
• Saves results to a CSV file (extracted_terms.csv)

When to use: Use this after extracting text from PDFs to create a dataset of Arabic terms and definitions.

Usage:

python create_defs.py <PICKLE_FILE>

Example:

python create_defs.py document_pages.pkl

Output: Creates/updates extracted_terms.csv with columns: word, definition.

Features:

• Automatic retry on API errors
• Progress tracking
• Summary statistics
• Appends to existing CSV (doesn't overwrite)

Note: Requires an OpenAI API key (see API Keys Setup).

---

4. csv_combinor.py

Purpose: Combines multiple CSV files containing word-definition pairs into a single shuffled dataset.

What it does:

• Reads all CSV files from a specified folder
• Validates that files contain word and definition columns
• Removes duplicates
• Shuffles the combined data
• Saves to a single output CSV file

When to use: Use this to merge multiple term-definition CSV files into one dataset for training or analysis.

Usage:

python csv_combinor.py <FOLDER_PATH> <OUTPUT_FILE>

Example:

python csv_combinor.py ./csv_files combined_dataset.csv

Output: Creates a shuffled CSV file with all unique word-definition pairs.

Features:

• Automatic duplicate removal
• Empty row filtering
• Progress reporting
• UTF-8 encoding support for Arabic text

---

🚀 Installation

1. Clone or download this repository

2. Install Python dependencies:

pip install -r requirements.txt

3. Set up API keys (see API Keys Setup)

📖 Usage

Complete Workflow Example

Here's a typical workflow for processing PDF documents:

Step 1: Extract text from PDF

# For text-based PDFs
python pdf_extractor.py my_document.pdf

# OR for scanned PDFs
python ocr_extractor.py scanned_document.pdf

This creates a pickle file (e.g., my_document_pages.pkl).

Step 2: Extract terms and definitions

python create_defs.py my_document_pages.pkl

This creates extracted_terms.csv with Arabic terms and definitions.

Step 3: Combine multiple CSV files

python csv_combinor.py ./csv_folder combined_output.csv

This creates a single shuffled dataset from all CSV files in the folder.

---

🔄 Workflow Diagram

PDF Document
    │
    ├─→ pdf_extractor.py (text-based PDFs)
    │   └─→ document_pages.pkl
    │
    └─→ ocr_extractor.py (scanned PDFs)
        └─→ document_texts.pkl
            │
            └─→ create_defs.py
                └─→ extracted_terms.csv
                    │
                    └─→ csv_combinor.py (combine multiple CSVs)
                        └─→ combined_dataset.csv

---

📦 Requirements

Python Version

• Python 3.7 or higher

Dependencies

See requirements.txt for the complete list. Main dependencies include:

• openai - For GPT-based term extraction
• pandas - For CSV processing
• mistralai - For OCR functionality
• PyPDF2 - For PDF text extraction

---

🔑 API Keys Setup

OpenAI API Key (for create_defs.py)

Option 1: Environment Variable (Recommended)

# Windows (PowerShell)
$env:OPENAI_API_KEY="your-api-key-here"

# Windows (CMD)
set OPENAI_API_KEY=your-api-key-here

# Linux/Mac
export OPENAI_API_KEY="your-api-key-here"

Option 2: Set in script (not recommended for security) Edit create_defs.py and set the API key directly (not recommended).

Mistral API Key (for ocr_extractor.py)

Option 1: Environment Variable (Recommended)

# Windows (PowerShell)
$env:MISTRAL_API_KEY="your-api-key-here"

# Windows (CMD)
set MISTRAL_API_KEY=your-api-key-here

# Linux/Mac
export MISTRAL_API_KEY="your-api-key-here"

Get API Keys:

• OpenAI: https://platform.openai.com/api-keys
• Mistral: https://console.mistral.ai/

---

⚙️ Configuration

create_defs.py Configuration

You can modify these constants in the script:

• MODEL_NAME: OpenAI model to use (default: "gpt-4o-mini")
• OUTPUT_CSV: Output CSV filename (default: "extracted_terms.csv")
• MAX_RETRIES: Maximum retry attempts (default: 5)
• RETRY_DELAY: Delay between retries in seconds (default: 10)

---

🐛 Troubleshooting

Common Issues

1. "API key not set" error

   • Make sure you've set the environment variable correctly
   • Restart your terminal after setting the variable

2. "File not found" error

   • Check that the file path is correct
   • Use absolute paths if relative paths don't work

3. Empty extraction results

   • For scanned PDFs, use ocr_extractor.py instead of pdf_extractor.py
   • Check that the PDF contains readable text

4. CSV encoding issues

   • The scripts use UTF-8 encoding for Arabic text
   • Open CSV files with a UTF-8 compatible editor (e.g., Excel, VS Code)

5. API rate limits

   • The scripts include automatic retry logic
   • If you hit rate limits, wait and try again

---

📝 Notes

• All scripts support Arabic text and UTF-8 encoding
• Pickle files are Python-specific binary formats
• CSV files are saved with UTF-8-SIG encoding for Excel compatibility
• The create_defs.py script appends to existing CSV files (doesn't overwrite)
• Progress indicators (✅, ⚠️, ❌) help track script execution

---

🤝 Contributing

Feel free to improve the code, add features, or fix bugs. Some ideas:

• Add support for other OCR services
• Implement batch processing for multiple PDFs
• Add progress bars for long operations
• Support for other languages

---

📄 License

This project is provided as-is for research and educational purposes.

---

🙏 Acknowledgments

• OpenAI for GPT models
• Mistral AI for OCR capabilities
• PyPDF2 for PDF text extraction
• pandas for data processing