EFK Stack (Elasticsearch, Fluentd, Kibana)

Centralized logging and monitoring solution used to collect, process, store, and visualize log data from various sources.

Use Case

This project provides a centralized logging and monitoring architecture designed for modern infrastructure. Organizations and teams can leverage this EFK stack setup to automatically aggregate logs from diverse sources (applications, containers, infrastructure), store and index them efficiently, and visualize metrics dynamically. Rather than manually parsing isolated log files across multiple servers, this automated logging pipeline improves overall observability, speeds up debugging, and simplifies DevSecOps monitoring and compliance requirements.

Minimum Requirements

To deploy and execute this project, the following minimum requirements must be met:

• Containerization: Docker and Docker Compose (for local containerized setup) or Kubernetes (for a cloud-native setup).
• System Resources: At least 4GB RAM and 2 CPUs for efficient testing and execution.
• Network / Open Ports:
  • 9200 (Elasticsearch API)
  • 5601 (Kibana UI)
  • 24224 (Fluentd, if using TCP input stream)

Architecture

flowchart TD
    subgraph Log Sources ["Log Sources"]
        AppLogs[Application / Web Logs]
        ContainerLogs[Kubernetes / Container Logs]
        SystemLogs[Infrastructure / OS Logs]
    end

    subgraph EFK Stack ["EFK Stack Storage & Processing"]
        Fluentd["Fluentd\n(Log Collector & Processor)"]
        Elasticsearch[("Elasticsearch\n(Search & Analytics Engine)")]
        Kibana["Kibana\n(Visualization Dashboard)"]
    end

    Admin([DevOps / Administrator])

    AppLogs -->|Generates| Fluentd
    ContainerLogs -->|Generates| Fluentd
    SystemLogs -->|Generates| Fluentd

    Fluentd -->|Parses, Filters & Forwards| Elasticsearch
    Elasticsearch -->|Indexes Data| Kibana
    Admin -->|Explores Data & Dashboards| Kibana

Loading

Tech Stack

• Search & Analytics: Elasticsearch
• Log Processor: Fluentd / Fluent Bit
• Data Visualization: Kibana
• Deployment & Orchestration: Docker, Kubernetes
• Languages & Integrations: Python (for event generation/web apps), NGINX

Features

• Automated Log Aggregation: Effortlessly collect logs from multiple disparate sources including NGINX, Python apps, and Kubernetes.
• Fast Search & Retrieval: Powerful querying capabilities via Elasticsearch Query DSL.
• Dynamic Visualization: Kibana provides interactive dashboards, charts, and graphs for deep log analysis.
• Container-Native: Full support for standalone Docker setups or scalable Kubernetes clusters.
• Customizable Ingestion Rules: Fluentd plugins for log transformation, filtering, and tag-based routing.

Project Structure

.
├── 1.elasticsearch-readme/    # Elasticsearch standalone setup & docs
├── 2.kibana-readme/           # Kibana setup & docs
├── 3.Fluent_bit_readme/       # Fluent Bit configuration & alternatives
├── 4.elasticsearch-kibana/    # Integrated ES & Kibana guides
├── 5.event-generator/         # Log simulation scripts for practice
├── 6.ngnix/                   # NGINX reverse proxy & log collection
├── 7.k8-monitoring/           # Kubernetes cluster monitoring manifests
├── 8.webapp/                  # Web app logging examples
├── 9.python-simple/           # Simple Python scripts log aggregation
├── 10.python-webapp/          # Python web app specific setups
├── 11.elastic_cloud/          # Elastic Cloud SaaS configuration
├── ecommerce_logs.csv         # Sample e-commerce static dataset
├── website_visit_data.csv     # Sample website visit dataset
└── readme.md                  # Main overview documentation

Implementation Plan & Execution Guide

Follow these instructions to deploy the application on your environment.

Method 1: Local Docker Compose Deployment

1. Navigate to the relevant sub-folder or create a docker-compose.yml defining Elasticsearch, Kibana, and Fluentd services.
2. Provide a standard fluentd.conf inside the volume directory:

   <source>
     @type tail
     path /var/log/*.log
     pos_file /var/log/fluentd.pos
     tag app.logs
     format none
   </source>
   
   <match app.logs>
     @type elasticsearch
     host elasticsearch
     port 9200
     logstash_format true
   </match>

3. Run the stack in the background:

   docker-compose up -d

4. Verify all components are up seamlessly by reviewing the logs:

   docker-compose logs -f

Method 2: Kubernetes Deployment

1. Apply the EFK namespace and definitions natively using kubectl:

   kubectl apply -f efk-deployment.yaml

2. Verify that your deployments and pods are healthy:

   kubectl get pods -n efk-stack

Usage

1. Open Kibana at http://localhost:5601.
2. Configure an initial Index Pattern matching your logs (e.g., logstash-* or fluentd-*).
3. Leverage the Discover tab to explore real-time log data.
4. Establish custom visualizations and Dashboards for tracking KPIs.

Security Considerations

• Access Control: Enable authentication and RBAC limits directly in Elasticsearch.
• Encryption: Utilize HTTPS/TLS for secure inner-cluster communication.
• Resource Management: Strictly define CPU and Memory requests/limits to prevent node resource exhaustion.

Testing & Troubleshooting

• Ensure Fluentd is correctly mapped to target paths and has exact read/write bounds.
• Use cluster health queries to identify status checks across Elasticsearch:

  curl http://localhost:9200/_cluster/health?pretty

---

Manish Pandey — Senior DevOps/Platform Engineer

🛠️ Technology Stack

☁️ Cloud & Platforms

⚙️ Platform & DevOps

🔐 Security & Ops

🧑‍💻 Programming

💾 Database

Connect With Me

• GitHub: @mpandey95
• LinkedIn: manish-pandey95
• Email: mnshkmrpnd@gmail.com

License

See LICENSE | Support: GitHub • LinkedIn