Datalake MISP Integration

Objective

This repository provides a connector that continuously retrieves threat intelligence from the Datalake platform and imports it into a MISP instance. The connector automates data ingestion, ensuring that the latest threat indicators are available for analysis and correlation within MISP.

Prerequisites

Before setting up the connector, ensure you have the following:

• An operational MISP instance with the necessary privileges for reading and writing events.
• An active Datalake subscription. To authenticate, you can:
  • Generate a Datalake Long-Term Token
  • Or use your Datalake account credentials (username and password)

Getting started

The connector is provided as a docker image. Here is a step by step process to download and run the image:

1. Pull the Docker Image

Retrieve this connector's Docker image from Docker Hub by running:

docker pull ocddev/datalake2misp:$VERSION

Replace $VERSION with the latest version available.

2. Configure Environment Variables

Copy the environment template and modify it with your credentials:

cp template.env .env

Edit the .env file to include:

• Your Datalake authentication details (either Username & Password OR a Long-Term Token)
• Your MISP instance host and API key

3. Configure Queries

Copy the query template file and customize it to define your Datalake queries:

cp template_queries.json queries.json

Modify queries.json to add valid Datalake query hashes following the template's structure. For details on creating custom query hashes, refer to the Datalake documentation. The frequency key determines how often queries are executed. The first query execution doesn't trigger when the connector starts, so to test ingestion quickly, set frequency to a small value.

4. Run the Container

Start the connector using the following command:

docker run --env-file .env -v ./queries.json:/code/queries.json ocddev/datalake2misp:$VERSION

Replace $VERSION with the version you previously pulled.

Stopping the container

To stop the container gracefully, allowing the events to be fully inserted, use:

docker stop -t 120 <container_name>

Testing the image with a local misp

For development and testing, you can set up a local MISP instance by following the instruction of this repo

Once your MISP is running at https://localhost, set the env variable OCD_DTL_MISP_HOST to localhost and get the MISP API key here: https://localhost/users/view/me for OCD_DTL_MISP_API_KEY. Also, you may set the OCD_DTL_MISP_USE_SSL env variriable to false when working with a local MISP instance.

Then build and run the image with:

make start-connector

When you're done using the connector, take it down with:

make clean

Testing the code

You can test the code with pytest using this command:

make test

Events in MISP

Event description

Event correlation graph

Event history