Skip to content

Home

Jump to bottom
arn edited this page May 7, 2026 · 4 revisions

CrawlLama Wiki

CrawlLama is a high-performance, local AI Research Agent engineered for advanced Open Source Intelligence (OSINT) and complex multi-hop reasoning. It utilizes local Large Language Models (LLMs) via Ollama to provide a secure, private, and highly extensible research environment.

Table of Contents

  1. Introduction
  2. Architecture
  3. Key Features
  4. Interfaces and Tooling
  5. Getting Started
  6. Developer Resources

Introduction

CrawlLama is designed for researchers, developers, and security professionals who require a local-first agent capable of executing deep-dive queries. Unlike standard conversational AI, CrawlLama is built to:

  • Execute Multi-Step Reasoning: Deconstruct complex tasks into sequential research phases.
  • Perform Specialized OSINT: Utilize dedicated modules for email, phone, IP, and social media analysis.
  • Maintain Privacy: Process all LLM logic locally while maintaining high performance.
  • Scale Effort Dynamically: Adjust reasoning depth based on task complexity.

Architecture

The system utilizes a modular, tool-centric architecture:

  • Core Engine: Manages state, session persistence, and LLM communication.
  • Tool Registry: Orchestrates external integrations including web search providers (DuckDuckGo, Brave, Serper), Wikipedia, and OSINT modules.
  • Agent Orchestrator: Powered by LangGraph, managing the iterative flow of Search, Analysis, Critique, and Synthesis.
  • Plugin System: Supports dynamic loading of custom functionality without core modification.

Key Features

Adaptive Intelligence

CrawlLama features an Adaptive Agent Hopping system. It automatically analyzes query intent to select the optimal agent level (Low, Mid, or High complexity). This ensures resource efficiency for simple tasks while providing maximum depth for complex investigations.

Multi-Hop Reasoning

Utilizing LangGraph, the agent performs iterative research cycles:

  1. Router: Path determination and initial strategy.
  2. Search: Data acquisition from multiple sources.
  3. Analyze: Interpretation of findings and entity extraction.
  4. Follow-Up: Gap identification and secondary research cycles.
  5. Critique: Self-evaluation for factual accuracy and hallucination prevention.

OSINT Intelligence Suite

CrawlLama provides a robust suite of tools for Open Source Intelligence:

  • Email Intelligence: Validation, MX record analysis, and breach detection.
  • Phone Intelligence: Carrier identification, normalization, and country detection.
  • IP Intelligence: Geolocation, ISP verification, and reputation scoring.
  • Social Intelligence: Automated profile discovery across 12+ platforms.
  • Advanced Operators: Support for site:, inurl:, filetype:, and other advanced search parameters.

RAG and Search

  • Hybrid Search: Combines semantic vector search with traditional keyword matching.
  • Intelligent Caching: Implements TTL-based caching to optimize performance and reduce API latency.
  • Extended Context: Support for 16k+ tokens (hardware dependent) for deep history maintenance.

Interfaces and Tooling

Interactive CLI

A professional terminal interface featuring Markdown rendering, real-time token monitoring, and a comprehensive interactive settings menu.

FastAPI REST API

Integrate CrawlLama into existing workflows via a robust RESTful interface.

  • Key Endpoints: /query, /osint/query, /memory, and /health.
  • Security: Integrated API key authentication and configurable rate limiting.

Health Monitoring Dashboard

A live monitoring interface for system-wide visibility:

  • System Metrics: Real-time tracking of CPU, RAM, and Disk utilization.
  • Component Status: Health checks for Ollama, RAG, Cache, and Tools.
  • Performance Logs: Response latency tracking and error diagnostics.

Getting Started

  1. Install Ollama: ollama.com
  2. Clone and Setup: 
    git clone https://github.com/arn-c0de/Crawllama.git
cd Crawllama
./setup.sh  # or setup.bat on Windows
  3. Run: 
    ./run.sh    # or run.bat

Developer Resources

Comprehensive guides for extending and integrating CrawlLama:

Maintenance

Cloning this Wiki locally

You can clone this wiki to your local machine for offline editing or version control:

git clone https://github.com/arn-c0de/Crawllama.wiki.git

For comprehensive documentation, refer to the Documentation Overview.

CrawlLama Research Agent | GitHub Repository | Project Website Privacy-focused, local AI intelligence for the modern researcher.

Navigation

Developer Resources

Documentation Overview

Clone this wiki locally