Docling GPU Configuration Guide

This guide explains how to configure Docling for GPU acceleration and save parsing results for inspection.

Overview

Docling is the default open-source document parser in Flexible GraphRAG. It supports GPU acceleration for significantly faster processing of complex documents, especially those with tables and images.

Recommended Package Manager: This guide uses uv, a fast Python package manager. All pip commands can be replaced with uv pip for better performance.

GPU vs CPU Performance

GPU Benefits:

5-10x faster PDF processing with complex layouts
Better performance on documents with many tables (like technical manuals)
Faster OCR processing for scanned documents
Recommended for production use with large document volumes

When to use CPU:

GPU not available
Small, simple documents (< 5 pages)
Debugging GPU-related issues
Development/testing with limited resources

Configuration Options

Environment File Setup

Before configuring options, create your .env file from the template:

# In the flexible-graphrag directory
cp env-sample.txt .env  # On Windows use: copy env-sample.txt .env

The env-sample.txt file contains all available configuration options with descriptions. Copy it to .env and modify as needed.

1. Device Selection (`DOCLING_DEVICE`)

Control which device Docling uses for processing using Docling's official AcceleratorDevice API:

# Auto-detect (default) - uses GPU if available, falls back to CPU
DOCLING_DEVICE=auto

# Force CPU-only (slower but always works)
DOCLING_DEVICE=cpu

# Force CUDA/GPU (requires CUDA-capable GPU and PyTorch with CUDA)
DOCLING_DEVICE=cuda

# Force Apple Metal (Mac with Apple Silicon only)
DOCLING_DEVICE=mps

Recommendation: Use auto for most cases. Docling will automatically detect and use GPU if available.

2. Save Parsing Results (`SAVE_PARSING_OUTPUT`)

Enable saving of intermediate parsing results to disk for inspection (works for both parsers):

# Enable saving parsing results
SAVE_PARSING_OUTPUT=true

# Disable saving (default)
SAVE_PARSING_OUTPUT=false

When enabled, the system will save parsing outputs to ./parsing_output/:

For Docling:

{filename}_docling_markdown.md - Markdown format with table structures preserved
{filename}_docling_plaintext.txt - Plain text extraction for better entity recognition
{filename}_docling_metadata.json - Processing metadata (file info, lengths, table detection, format used)

For LlamaParse:

{filename}_llamaparse_output.md - Parsed markdown output (multiple chunks automatically combined)
{filename}_llamaparse_output.txt - Plaintext version with markdown formatting stripped
{filename}_llamaparse_metadata.json - Processing metadata (file info, chunk count, character count)

Example output location:

./parsing_output/
  ├── document1_docling_markdown.md
  ├── document1_docling_plaintext.txt
  ├── document1_docling_metadata.json
  ├── document2_llamaparse_output.md
  ├── document2_llamaparse_output.txt
  ├── document2_llamaparse_metadata.json
  └── technical_spec_docling_markdown.md

GPU Setup Requirements

Step 1: Check Your CUDA Version (NVIDIA GPUs Only)

If you have an NVIDIA GPU, first check your CUDA driver version:

Windows/Linux:

nvidia-smi

Look for the CUDA Version in the output header. Example:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 591.74       Driver Version: 591.74       CUDA Version: 13.1   |
+-----------------------------------------------------------------------------+

This shows your maximum supported CUDA version (13.1 in this example). You can install PyTorch for CUDA 13.0 or lower.

What nvidia-smi shows:

Driver Version: Your NVIDIA driver version (e.g., 591.74)
CUDA Version: Maximum CUDA toolkit version your driver supports
GPU Name: Your GPU model (e.g., NVIDIA GeForce RTX 5090)
GPU Util: Current GPU utilization percentage
Memory-Usage: GPU memory in use / total available

Step 2: Install PyTorch with GPU Support

Visit PyTorch.org Get Started and select your configuration:

Configuration Steps:

PyTorch Build: Select "Stable (2.7.0)" or latest version
Your OS: Select Windows, Linux, or Mac
- Selecting your OS will display platform-specific instructions (e.g., "Installing on Windows")
Package: Select "Pip"
Language: Select "Python"
Compute Platform: Select your CUDA version (e.g., CUDA 11.8, 12.6, 12.8, 13.0) or CPU
- Note: Mac users will see "CUDA is not available on macOS, please use default package"

The website will generate an install command. Prefix it with uv for faster installation.

CUDA (NVIDIA GPUs)

On PyTorch.org, select your OS (Windows or Linux) to see platform-specific instructions. The page will display "Installing on Windows" or "Installing on Linux".

Current PyTorch offerings (as of January 2026): CUDA 12.6, 12.8, and 13.0

For CUDA 12.6:

# Uninstall existing versions first (recommended)
uv pip uninstall torch torchvision

# PyTorch.org command
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126

# With uv (faster)
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

For CUDA 12.8:

uv pip uninstall torch torchvision
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

For CUDA 13.0 (Latest - Windows/Linux):

# First uninstall existing versions
uv pip uninstall torch torchvision

# Install CUDA 13.0 build
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130

# Windows only: Install Triton for Windows (enables full PyTorch compiler support)
uv pip install triton-windows

Note: The triton-windows package is only needed on Windows for full PyTorch compiler support.

For older CUDA 11.8 (if needed):

uv pip uninstall torch torchvision
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

Triton for Windows (Optional but Recommended)

PyTorch's Torch Inductor compiler requires Triton. On Windows, install the Windows-specific package:

uv pip install triton-windows

If you get a Triton error:

torch._inductor.exc.TritonMissing: Cannot find a working triton installation.

Solutions (choose one):

Install triton-windows (recommended):
```
uv pip install triton-windows
```
Disable Torch Inductor (workaround): Add to your .env file:
```
TORCH_COMPILE_DISABLE=1
```
This has minimal performance impact on Docling GPU processing.

Apple Silicon (Mac M1/M2/M3/M4)

CUDA is not available on macOS. When you select "Mac" in the OS selector on PyTorch.org, the page displays "Installing on macOS" and shows:

# PyTorch.org command (Mac - CPU/MPS only)
pip3 install torch torchvision

Use uv for faster installation:

# Install PyTorch (MPS support included)
uv pip install torch torchvision

# Set device to use Metal in .env
DOCLING_DEVICE=mps

CPU Only (No GPU)

If you don't have a GPU or prefer CPU processing:

# Uninstall GPU versions first
uv pip uninstall torch torchvision

# Install CPU-only PyTorch (smaller download, no CUDA)
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

# Force CPU device in .env
DOCLING_DEVICE=cpu

Step 3: Verify Installation

Check installed PyTorch version:

uv pip list | grep torch

Expected output for CUDA 13.0:

torch           2.7.0+cu130
torchvision     0.22.0+cu130

Verify GPU detection:

import torch
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Check CUDA
if torch.cuda.is_available():
    logger.info(f"✓ CUDA available: {torch.cuda.get_device_name(0)}")
    logger.info(f"  CUDA version: {torch.version.cuda}")
    logger.info(f"  GPU count: {torch.cuda.device_count()}")
    logger.info(f"  Total memory: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.2f} GB")
else:
    logger.info("✗ CUDA not available - will use CPU")

# Check MPS (Mac)
if hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
    logger.info("✓ MPS (Apple Metal) available")

Expected output for RTX 5090:

✓ CUDA available: NVIDIA GeForce RTX 5090
  CUDA version: 13.0
  GPU count: 1
  Total memory: 34.19 GB

2. Test Docling Processing

Process a PDF with GPU acceleration:

# Enable GPU and save results in .env
DOCLING_DEVICE=cuda
SAVE_PARSING_OUTPUT=true
DOCUMENT_PARSER=docling

# Process the document
# Use your UI or REST API to upload and process a PDF

3. Compare CPU vs GPU Performance

You can benchmark the difference:

# Test with CPU
DOCLING_DEVICE=cpu
# (Process document and note the time)

# Test with GPU
DOCLING_DEVICE=cuda
# (Process same document and compare time)

Troubleshooting

"CUDA not available" Error

Cause: PyTorch installed without CUDA support

Solution: Reinstall PyTorch with CUDA:

# Uninstall existing versions
uv pip uninstall torch torchvision

# Reinstall with CUDA (replace cu130 with your CUDA version)
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130

"NVIDIA GeForce RTX XXXX with CUDA capability sm_XXX is not compatible"

Cause: Your GPU architecture is newer than what your PyTorch build supports.

Example Error:

UserWarning: NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible 
with the current PyTorch installation. The current PyTorch install supports CUDA 
capabilities sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.

Solution: Upgrade to a newer PyTorch build that supports your GPU:

# Check your CUDA version
nvidia-smi

# For RTX 5090 (Blackwell/sm_120), use CUDA 13.0
uv pip uninstall torch torchvision
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130

"Cannot find a working triton installation" (Windows)

Cause: Triton for Windows not installed.

Solutions (choose one):

Install triton-windows (recommended):
```
uv pip install triton-windows
```
Disable Torch Inductor (workaround): Add to your .env file:
```
TORCH_COMPILE_DISABLE=1
```
This disables Torch Inductor compilation without affecting Docling GPU performance.

"Out of memory" Error

Cause: GPU doesn't have enough memory for large documents

Solutions:

Force CPU processing: DOCLING_DEVICE=cpu
Process smaller batches
Close other GPU-intensive applications
Upgrade GPU memory

Slow Processing Even with GPU

Possible causes:

GPU not actually being used (check logs)
Document is very large (100+ pages)
CUDA drivers outdated
Thermal throttling (GPU overheating)

Debugging:

# Check if GPU is being utilized
nvidia-smi  # On Linux/Windows

# Enable verbose logging
# Check logs for "CUDA available" message

Files Not Saved to parsing_output

Cause: SAVE_PARSING_OUTPUT not set or permission issues

Solutions:

Verify setting in .env: SAVE_PARSING_OUTPUT=true
Check write permissions for ./parsing_output/
Check logs for error messages

Configuration Examples

Production Setup (GPU, No Saving)

DOCUMENT_PARSER=docling
DOCLING_DEVICE=auto
DOCLING_TIMEOUT=600
SAVE_PARSING_OUTPUT=false

Development/Debugging Setup (CPU, Save Results)

DOCUMENT_PARSER=docling
DOCLING_DEVICE=cpu
DOCLING_TIMEOUT=300
SAVE_PARSING_OUTPUT=true

High-Performance Setup (Force GPU, Extended Timeout)

First, create .env from template if you haven't already:

cp env-sample.txt .env  # On Windows use: copy env-sample.txt .env

Then add to .env:

DOCUMENT_PARSER=docling
DOCLING_DEVICE=cuda
DOCLING_TIMEOUT=900
SAVE_PARSING_OUTPUT=false

# Optional: Disable Torch Inductor if triton-windows not installed
# TORCH_COMPILE_DISABLE=1

LlamaParse with Saved Output (for comparison)

DOCUMENT_PARSER=llamaparse
LLAMAPARSE_API_KEY=your-api-key
LLAMAPARSE_MODE=parse_page_with_llm
SAVE_PARSING_OUTPUT=true

Performance Tips

Package Manager: Use uv pip instead of pip for 10-100x faster package installation
GPU Selection: Use cuda for NVIDIA GPUs, mps for Apple Silicon
Batch Processing: Process multiple documents in one session for better GPU utilization
Timeout Settings: Increase DOCLING_TIMEOUT for very large documents (100+ pages)
Memory Management: Close other GPU applications during processing
Driver Updates: Keep CUDA drivers and PyTorch updated for best performance
Triton on Windows: Install triton-windows for full PyTorch compiler support, or use TORCH_COMPILE_DISABLE=1 if you encounter Triton errors

Quick Reference: Installation Commands

Check Your Setup

# Check CUDA version (NVIDIA GPUs)
nvidia-smi

# Check installed PyTorch
uv pip list | grep torch

# Verify GPU in Python
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}')"

Install PyTorch with UV

NVIDIA GPU (CUDA 13.0 - Latest, Windows/Linux):

uv pip uninstall torch torchvision
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130

# Windows only: For full PyTorch compiler support
uv pip install triton-windows

NVIDIA GPU (CUDA 12.8):

uv pip uninstall torch torchvision
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

NVIDIA GPU (CUDA 12.6):

uv pip uninstall torch torchvision
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

Mac (Apple Silicon):

uv pip install torch torchvision

CPU Only:

uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

Environment Configuration

If you don't have a .env file yet, copy env-sample.txt to .env:

# In the flexible-graphrag directory
cp env-sample.txt .env  # On Windows use: copy env-sample.txt .env

Then add these settings to your .env file:

# GPU Configuration
DOCLING_DEVICE=auto  # or cuda, mps, cpu

# Optional: Disable Torch Inductor if triton-windows not installed
# TORCH_COMPILE_DISABLE=1

# Optional: Save parsing results
SAVE_PARSING_OUTPUT=true

OCR Configuration

OCR is disabled by default. Enable it for scanned PDFs and image-only documents.

DOCLING_OCR=true
DOCLING_OCR_ENGINE=auto   # auto | rapidocr | easyocr | tesseract_cli | tesserocr | ocrmac

Engine	Install	Notes
`auto`	nothing extra	Docling picks best installed engine at runtime
`rapidocr`	nothing extra	Included in `docling-slim[standard]`
`easyocr`	`uv pip install -e ".[docling-ocr-easyocr]"`	CPU-based; GPU not used
`tesseract_cli`	system Tesseract on PATH	Recommended on Windows — no native build needed
`tesserocr`	`uv pip install -e ".[docling-ocr-tesserocr]"`	Compiles C++ extension; fragile on Windows (prefer `tesseract_cli`)
`ocrmac`	`uv pip install -e ".[docling-ocr-ocrmac]"`	macOS only

Windows — Tesseract CLI setup

Install Tesseract: choco install tesseract (or from UB Mannheim builds)
Verify: where tesseract → should show C:\Program Files\Tesseract-OCR\tesseract.exe
Set in .env: DOCLING_OCR=true and DOCLING_OCR_ENGINE=tesseract_cli

Log messages

When OCR is configured, the startup log shows:

Docling OCR config (app): enabled=true requested_engine='tesseract_cli' pipeline_ocr_options=TesseractCliOcrOptions

Docling's own runtime message (Auto OCR model selected ...) shows the effective engine chosen when using auto.

Language Support

Docling supports multi-language documents including:

English, German, French, Spanish, Italian
Czech, Polish, Slovak
Russian, Chinese, Japanese
And many more...

Multi-language technical documents with tables and diagrams process well with Docling on GPU.

FilesExpand file tree

DOCLING-GPU-CONFIGURATION.md

Latest commit

History