Experiment Reproduction Guide for the paper "Exploiting GenAI for Plan Generation in BDI Agents"

Accepted at ECAI 2025 main track

Authors:

• Giovanni Ciatto (giovanni.ciatto@unibo.it),
• Gianluca Aguzzi (gianluca.aguzzi@unibo.it),
• Riccardo Battistini (riccardo.battistini2@studio.unibo.it),
• Martina Baiardi (m.baiardi@unibo.it),
• Samuele Burattini (samuele.burattini@unibo.it) and
• Alessandro Ricci (a.ricci@unibo.it)

Affiliation: Department of Computer Science and Engineering
Alma Mater Studiorum---University of Bologna, Italy

Prerequisites

Before starting, make sure you have JDK 21 or newer installed.

Evaluation

You can rerun experiments using previously recorded LLM responses and generate new execution traces.

To run an experiment with an existing LLM response and see the execution trace in your console:

./gradlew replayExperiment --args="--legacy --exp-dir experiments/anthropic/claude-3.7-sonnet-thinking/gifted_bell"

To run an experiment and save the new execution trace as a separate experiment:

./gradlew replayExperiment --args="--legacy --log-to-file --exp-dir experiments/anthropic/claude-3.7-sonnet-thinking/gifted_bell"

The execution traces generated by the human-provided plans can be observed by running this task:

./gradlew runBaseline

Analyzing the results

The results referenced in the paper are stored in the ./jakta-playground/experiments directory.

We tested four models. For each of them we performed five repetitions, for a total of twenty experiments.

The results are organized in a hierarchical directory structure where:

1. the first level contains the provider name (e.g., "openai");
2. the second level contains the name of the model (e.g., "gpt-4.1-preview");
3. the third level contains the names of the individual experiment runs;

For each experiment run:

1. the first level contains:
   • a directory for each agent in the multi-agent system;
   • a mas.log and a mas.jsonl file where all external actions and environmental logs are stored.
2. in the directory of each agent there is:
   • a .log and .jsonl file where all internal actions, intentions and events logs are stored;
   • a chat directory containing a .jsonl and a .log file for each PGP attempt, where are stored:
     • the prompt sent to the LLM;
     • the response of the LLM;
     • a textual representation of the plans parsed by JaKtA from the response.

In addition, the human.baseline.txt file at the root of the experiments directory stores the reference human plans.

Below, a graphical overview of the directory structure is provided for reference.

.
├── human.baseline.txt
├── anthropic
│   └── claude-3.7-sonnet-thinking
│       ├── elastic_goldwasser
│       │   ├── ExplorerBot
│       │   │   ├── chat
│       │   │   ├── ExplorerBot.jsonl
│       │   │   └── ExplorerBot.log
│       │   ├── mas.jsonl
│       │   └── mas.log
│       ├── eloquent_bhaskara
│       ├── gifted_bell
│       ├── xenodochial_pike
│       └── zealous_engelbart
├── deepseek
│   └── deepseek-V3-0324
├── google
│   └── gemini-pro-2.5-preview
└── openai
    └── gpt-4.1-preview

Computing metrics

The metrics computed in the paper are:

• Plan Count (PC): Total number of generated plans.
• Context Complexity (CC): Average number of beliefs per plan context.
• Plan Body Complexity (PBC): Average number of operations per plan body.
• Generalization Count (GC): Number of generated plans that use variables (general) rather than constants (specific).
• Redundancy Amount (RA): Number of generated plans which are useless (e.g., not executable at runtime or subsumed by more general plans).
• Novel Goal Count (NGC): Number of newly invented goals.
• Novel Belief Count (NBC): Number of newly invented beliefs.
• Goal Semantic Misalignment (GSM): Number of semantically misaligned uses of admissible goals.
• Belief Semantic Misalignment (BSM): Number of semantically misaligned uses of admissible beliefs.
• Task Success Rate (TSR): Percentage of experiments where the agent successfully achieves the !reach(home) goal.

How were these metrics computed?

Taking as reference the experiment openai/gpt-4.1-preview/interesting_hoover we can see there are three messages in the chat/20873d04-53ad-4681-9efc-0be8c89d96c5.log file:

• the first contains the prompt for the GPT 4.1 model;
• the second contains the response;
• the third contains the textual representation of the plans, as parsed by JaKTA.

In particular, the response is made of two YAML code blocks.

The first one holds the sequence of plans generated by the LLM in the requested format.

EVENT: achieve reach(Object)
CONDITIONS:
  - object(Object)
  - ¬there_is(Object, here)
OPERATIONS:
  - execute getDirectionToMove(Direction)
  - execute move(Direction)
  - achieve reach(Object)
---
EVENT: achieve reach(Object)
CONDITIONS:
  - object(Object)
  - there_is(Object, here)
OPERATIONS:
  - <none>

We can see the PGP invocation generated two plans, and that both of them use a variable in the event trigger. This means that PC is two and GC too. There are a total of four beliefs, two for each plan context. This means that CC is two. There are three operations in total, so the PBC is one and a half. The RR is zero, since both plans are executable at runtime and none of them subsumes the other.

The second block of the response holds the admissible goals and beliefs invented by the LLM.

- goal: `reach(Object)`
  purpose: Achieve a situation where `Object` is at the agent's current position (i.e., there_is(Object, here))

We can see that both the NGC and the GSA are one, since the LLM invented a goal, but it was already provided as admissible by the programmer. Both the NBC and the BSA are zero since no new beliefs were invented.

In general, semantic misalignment of a goal or belief is counted each time appears a goal or a belief which is:

• syntactically incorrect;
• invented but never used;
• used but not admissible;
• already admissible in the prompt given by the programmer.

The last message shows that both the plans were correctly parsed.

Plan(
  parentGenerationGoal = Generate(reach(home)),
  trigger = AchievementGoalInvocation(value=reach(Object_6), purpose=null),
  guard = object(source(self), Object3) & ~ there_is(source(self), Object3, here),
  goals = [
    TrackGoalExecution(execute getDirectionToMove(Direction)),
    TrackGoalExecution(execute move(Direction)),
    TrackGoalExecution(achieve reach(Object)),
  ]
)
Plan(
  parentGenerationGoal = Generate(reach(home)),
  trigger = AchievementGoalInvocation(value=reach(Object_6), purpose=null),
  guard = object(source(self), Object3) & there_is(source(self), Object3, here),
  goals = [
    TrackGoalExecution(<none>),
  ]
)

The TSR is computed by searching for occurrences of the string "Reached object home" in the mas.log file. If that line appears, it means the environment successfully registered the presence of the explorer agent in the position of the object home. interesting hoover is such an experiment, as we can see in its mas.log file:

...
INFO mas - Key-values {directionToMove=east} have been added to the environment
INFO mas - Key-values {directionToMove=east} have been added to the environment
INFO mas - Reached object home

Generate Plots

Here we describe the procedure to generate plots used in the paper and in the Supplementary Material.

Requirements

• Python 3.X
• Bash terminal

Procedure

0. Locate the root project folder

cd experiment-2025-ECAI-generative-bdi-agents/

1. Create a Python virtual environment

python -m venv venv

2. Load virtual environment in current shell

source venv/bin/activate

3. Install python requirements

pip install -r requirements.txt

4. Execute plots generation script

python data_extraction_and_plotting.py

5. Inspect the generated plots, located in the root folder: geval_results_bar_plot.png and plan_metrics_comparison.pdf.

Experiments' execution time

The execution time of each experiment where home is reached is reported in the following table.

Model Name	Experiment name	Time in seconds
anthropic/claude-3.7-sonnet:thinking	gifted_bell	1.12
anthropic/claude-3.7-sonnet:thinking	xenodochial_pike	4.40
anthropic/claude-3.7-sonnet:thinking	zealous_engelbart	2.90
deepseek/deepseek-V3-0324	charming_cori	1.22
deepseek/deepseek-V3-0324	suspicious_cerf	4.29
google/gemini-pro-2.5-preview	fervent_colden	1.31
openai/gpt-4.1-preview	adoring_shtern	2.27
openai/gpt-4.1-preview	determined_gould	2.41
openai/gpt-4.1-preview	interesting_hoover	1.21
openai/gpt-4.1-preview	sleepy_wiles	2.56
openai/gpt-4.1-preview	vigilant_agnesi	3.09

Note that the execution time is computed by subtracting the timestamp of the last log event from the timestamp of the first log event of the ExplorerAgent's .jsonl file.

For the other experiments, the execution time is undefined
since either the agents keep running with no intentions and without having reached home or are stuck in a recursive loop.