MoodTrackAI — Audio Journaling MVP

Final Grade: 18.5 / 20.0

MoodTrackAI is a Portuguese-first audio journaling platform for daily emotional self-monitoring. Users record short voice entries about their day; the system transcribes the audio, analyses the emotional content of the text, extracts acoustic/prosodic signals from the voice, combines both modalities through late fusion, and presents emotional trends and personalized self-regulation recommendations.

The project was developed as a local Docker-based MVP and is composed of a web frontend, a Node.js API, a Python analysis engine, background workers, object storage, queue infrastructure, relational persistence, and a local LLM runtime.

---

Main Features

• Record daily voice journal entries directly in the browser.
• Upload audio to local object storage through the API backend.
• Transcribe speech using configurable Whisper ASR models.
• Analyse text emotions with a local LLM running through Ollama.
• Analyse vocal/prosodic emotion with WavLM embeddings and an SVC/RBF classifier.
• Fuse semantic and prosodic scores using fixed-weight late fusion.
• Store journal metadata, transcription, emotional scores and status.
• Display journal history, daily trends and emotional evolution.
• Generate and manage personalized wellbeing recommendations.
• Run the full system locally with Docker Compose.

---

Emotional Model

MoodTrackAI uses seven canonical emotional classes:

joy, sadness, surprise, anger, disgust, fear, neutral

The six primary classes follow the basic-emotion perspective used in the project, while neutral is included to avoid forcing factual or routine journal entries into an expressive emotion when no clear affective signal is present.

The final emotion vector is produced by combining two modalities:

final_score = 0.7 × semantic_score + 0.3 × prosody_score

This makes the textual interpretation the dominant signal while still allowing the vocal delivery to modulate the final emotional distribution.

---

Architecture

Service Topology

┌─────────────────────────────────────────────────────────────┐
│                    Frontend Web App                         │
│        Audio capture, journal timeline, charts, UI          │
└───────────────────────────┬─────────────────────────────────┘
                            │ HTTP/REST
┌───────────────────────────▼─────────────────────────────────┐
│                    Node.js REST API                         │
│       Journal CRUD, uploads, callbacks, aggregations        │
└───────────────┬──────────────────────────────┬──────────────┘
                │                              │
                │ Bull/Redis jobs              │ Prisma ORM
                │                              │
┌───────────────▼──────────────┐       ┌───────▼──────────────┐
│        API Worker            │       │      PostgreSQL      │
│  Background analysis jobs    │       │ Metadata + results   │
└───────────────┬──────────────┘       └──────────────────────┘
                │
                │ Analysis request + callback
┌───────────────▼────────────────────────────────────────────────┐
│              Python FastAPI Analysis Engine                    │
│  Audio fetch → transcription → prosody → text emotion → fusion │
└───────┬──────────────────────┬─────────────────────────────────┘
        │                      │                       │
        │                      │                       │
┌───────▼───────┐      ┌───────▼────────┐      ┌───────▼───────┐
│     MinIO     │      │     Ollama     │      │     Redis     │
│ Audio storage │      │ Local LLM API  │      │ Queue/cache   │
└───────────────┘      └────────────────┘      └───────────────┘

---

Stack

Frontend

• React / Next.js application.
• Browser audio recording through MediaRecorder.
• Hooks for recording/uploading, polling analysis status, timeline loading, daily trends and recommendations.
• Charts and dashboard components for emotional evolution.

API Backend

• Node.js 20 LTS.
• Express + TypeScript.
• Prisma ORM.
• Journal creation and retrieval.
• File upload orchestration.
• Callback endpoint for analysis completion.
• Recommendation endpoints.
• Integration with Redis/Bull, PostgreSQL and MinIO.

Analysis Engine

• Python 3.11.
• FastAPI.
• Whisper ASR through Hugging Face transformers.
• Ollama-based local text emotion classification.
• WavLM-based prosody emotion recognition.
• Late fusion between text and audio emotion distributions.

Infrastructure

• PostgreSQL 16.
• Redis 7.
• MinIO object storage.
• Ollama local LLM runtime.
• pgAdmin.
• Docker Compose.

---

Analysis Pipeline

The end-to-end analysis pipeline works as follows:

1. The user records an audio journal entry in the browser.
2. The frontend sends the audio blob to the Node.js API.
3. The API stores the audio in MinIO and creates a journal record in PostgreSQL.
4. A background job is queued through Redis/Bull.
5. The Python analysis service downloads the audio from MinIO.
6. The audio is transcribed using the selected Whisper model.
7. Acoustic/prosodic features and WavLM embeddings are extracted.
8. The prosody classifier produces an audio-based emotion distribution.
9. The transcription is sent to the local LLM through Ollama.
10. The text model produces a semantic emotion distribution.
11. The two distributions are combined through late fusion.
12. The Python service calls back the Node.js API with the result.
13. The API persists transcription, emotion scores and status.
14. The frontend polls the journal status and refreshes the timeline and charts.

---

Transcription

The analysis service supports several Whisper ASR models:

Key	Model ID	Notes
whisper_tiny	openai/whisper-tiny	Very light and fast, lower accuracy.
whisper_base	openai/whisper-base	Baseline model.
whisper_small	openai/whisper-small	Good speed/quality compromise.
whisper_medium	openai/whisper-medium	Heavier, usually more accurate.
whisper_large_v3	openai/whisper-large-v3	Full large model.
whisper_large_v3_turbo	openai/whisper-large-v3-turbo	Final candidate for better Portuguese transcription quality.

The selected model can be controlled with:

ASR_MODEL_KEY=whisper_large_v3_turbo

If model initialization fails, the service has a lightweight fallback transcription mode so that the pipeline can still fail gracefully in constrained environments.

---

Text Emotion Model

The semantic emotion classifier runs locally through Ollama.

Default configuration:

OLLAMA_HOST=http://ollama:11434
OLLAMA_TEXT_MODEL=qwen2.5:3b-instruct
OLLAMA_TEXT_STRATEGY=Light_Ensemble
OLLAMA_TIMEOUT=120

The strategy combines two complementary prompts:

• a criteria-based prompt with few-shot examples in Portuguese;
• a structured prompt that forces internal reasoning about event, valence and dominant emotion.

The final text distribution is normalized over:

joy, sadness, surprise, anger, disgust, fear, neutral

A conservative post-processing layer is also applied to reduce common ambiguity between close negative emotions, especially:

• sadness vs anger;
• sadness vs fear;
• surprise vs fear;
• anger vs disgust.

If Ollama is unavailable, the text component returns a safe fallback distribution rather than crashing the whole analysis pipeline.

---

Prosody Model

The prosody component uses a trained artifact based on:

audio → load/trim → chunk → WavLM-base embeddings → SVC/RBF classifier → emotion distribution

Main characteristics:

• Audio is loaded as mono and resampled to 16 kHz.
• Long audio is split into chunks.
• microsoft/wavlm-base is used as a frozen SSL feature extractor.
• For each chunk, mean and standard deviation of hidden states are concatenated.
• Each audio sample is represented by a 1536-dimensional embedding.
• A trained SVC/RBF classifier with probability output maps embeddings to the seven canonical emotions.
• The classifier output is normalized into a probability distribution.

Expected artifact path inside the container:

/app/prosody_artifact/prosody_final_artifact_v5_SSL_WAVLM_RBF_RECOVERED.joblib

The path can be overridden with:

PROSODY_ARTIFACT_PATH=/custom/path/to/artifact.joblib

If the artifact is missing, the service logs a warning and falls back safely.

---

Data Storage

The system stores different types of data in different services:

Component	Responsibility
PostgreSQL	Journal metadata, status, transcription, emotion scores, recommendations.
MinIO	Raw uploaded audio files.
Redis	Queue/cache backend for background processing.
Local model cache	Whisper, WavLM and Ollama model artifacts.

---

MVP Scope

This repository is an MVP focused on local experimentation and academic demonstration.

Included:

• Local Docker Compose deployment.
• Single-user usage flow.
• Audio recording and upload.
• Automatic transcription.
• Text-based emotion analysis.
• Prosody-based emotion analysis.
• Late fusion.
• Journal timeline.
• Daily emotional trends.
• Recommendation generation and feedback.
• Local-first infrastructure.

Not included / outside the MVP scope:

• Multi-user authentication and authorization.
• Production-grade HTTPS configuration.
• Cloud deployment.
• Audio retention policies.
• Clinical diagnosis or medical decision-making.
• Human therapist replacement.
• Full privacy/compliance hardening.

---

Requirements

Before running the project, make sure you have:

• Docker.
• Docker Compose.
• Git.
• At least 8 GB RAM recommended.
• Enough free disk space for Docker images and local model downloads.
• Free local ports:

5173, 3000, 8000, 11434, 5432, 6379, 9000, 9001, 5051

Note: port 11434 is commonly used by a local Ollama installation. If Ollama is already running on the host machine, stop it before starting the Docker stack.

---

Quick Start

1. Clone the repository

git clone <repository-url>
cd CA-25_26-main

If you already have the repository locally:

cd ~/Transferências/CA-25_26-main

2. Create the environment file

cp .env.example .env

Review .env and adjust values if needed.

3. Stop local Ollama if needed

If Ollama is installed directly on your machine, it may already be using port 11434.

sudo systemctl stop ollama

4. Start the full stack

sudo docker compose up --build

If your user belongs to the Docker group, sudo may not be necessary:

docker compose up --build

5. Open the application

Services should be available at:

Service	URL
Frontend	http://localhost:5173
API	http://localhost:3000/api
Analysis service	http://localhost:8000
Ollama	http://localhost:11434
PostgreSQL	localhost:5432
Redis	localhost:6379
MinIO API	http://localhost:9000
MinIO Console	http://localhost:9001
pgAdmin	http://localhost:5051

Default local credentials, if unchanged:

MinIO:
  user: minioadmin
  password: minioadmin

pgAdmin:
  user: admin@example.com
  password: admin

---

Docker Compose Services

The Docker Compose stack includes:

Service	Container	Description	Port
frontend	journaling-app-frontend	Web UI	5173
api	journaling-app-api	Node.js REST API	3000
api-worker	journaling-app-api-worker	Background worker	internal
analysis	journaling-app-analysis	Python FastAPI analysis engine	8000
ollama	journaling-app-ollama	Local LLM runtime	11434
postgres	journaling-app-postgres	Relational database	5432
redis	journaling-app-redis	Queue/cache backend	6379
minio	journaling-app-minio	Object storage	9000, 9001
pgadmin	journaling-app-pgadmin	Database administration UI	5051

---

Useful Commands

Check running services

sudo docker compose ps

View logs

sudo docker compose logs -f frontend
sudo docker compose logs -f api
sudo docker compose logs -f api-worker
sudo docker compose logs -f analysis
sudo docker compose logs -f ollama
sudo docker compose logs -f postgres
sudo docker compose logs -f redis
sudo docker compose logs -f minio

Restart after code changes

sudo docker compose up --build

If no dependencies or Dockerfiles changed:

sudo docker compose up

Stop services

sudo docker compose down

Stop and remove volumes

Use this only when you want a full local reset.

sudo docker compose down -v

---

Troubleshooting

Container name already in use

If Docker reports an error like:

The container name "/journaling-app-redis" is already in use

remove old containers from a previous run:

sudo docker rm -f $(sudo docker ps -aq --filter "name=journaling-app")

Then restart:

sudo docker compose up --build

Port 11434 already in use

If Docker reports:

failed to bind host port 0.0.0.0:11434/tcp: address already in use

check what is using the port:

sudo lsof -i :11434

If it is local Ollama, stop it:

sudo systemctl stop ollama

Then restart the stack:

sudo docker compose up --build

Port already in use by another container

Check containers using the port:

sudo docker ps | grep 11434

Remove the conflicting container:

sudo docker rm -f <container-name-or-id>

Docker is using too much disk space

Check Docker disk usage:

sudo docker system df

Clean unused Docker resources:

sudo docker system prune -a

To also remove unused volumes:

sudo docker system prune -a --volumes

Warning: removing volumes can delete local databases, uploaded files and model caches.

Ollama model is not available

Check Ollama logs:

sudo docker compose logs -f ollama

If the project uses an initialization script, confirm that the required model has been pulled inside the Ollama container.

Analysis fails after upload

Check the analysis and worker logs:

sudo docker compose logs -f analysis
sudo docker compose logs -f api-worker

Common causes:

• Missing prosody artifact.
• Ollama not reachable.
• Whisper/WavLM model download failed.
• MinIO object not found.
• Insufficient RAM or disk space.

Browser cannot record audio

Make sure:

• You are using a browser with MediaRecorder support.
• Microphone permission was granted.
• The app is opened through localhost.
• The microphone is not being used by another application.

---

Project Structure

.
├── frontend/                  # Web application
│   ├── app/                   # App routes/layout/styles
│   ├── src/
│   │   ├── components/        # UI components
│   │   ├── hooks/             # Recording, timeline, trends, recommendations
│   │   ├── lib/               # Frontend utilities
│   │   └── types/             # Frontend types
│   ├── package.json
│   ├── Dockerfile
│   └── next.config.ts
│
├── api-node/                  # Node.js API backend
│   ├── src/
│   │   ├── routes/            # REST endpoints
│   │   ├── workers/           # Queue consumers
│   │   ├── lib/               # Prisma, Redis, MinIO, shared helpers
│   │   ├── scripts/           # Utility scripts
│   │   └── index.ts           # API entrypoint
│   ├── prisma/
│   │   └── schema.prisma      # Database schema
│   ├── package.json
│   ├── Dockerfile
│   └── tsconfig.json
│
├── analysis-python/           # Python FastAPI analysis service
│   ├── app/
│   │   ├── models/            # Pydantic schemas and task store
│   │   ├── services/          # Transcription, prosody, text emotion, storage, callbacks
│   │   └── main.py            # FastAPI entrypoint
│   ├── notebooks/             # Experiments and model development
│   ├── scripts/               # Benchmark/support scripts
│   ├── requirements.txt
│   ├── Dockerfile
│   └── .dockerignore
│
├── infra/                     # Infrastructure bootstrap scripts
│   ├── ollama-init.sh         # Pull/init Ollama model
│   └── postgres-init.sql      # Database initialization SQL
│
├── docs/                      # Documentation
│   ├── contracts/             # API contracts
│   ├── images/                # Architecture/report images
│   └── report/                # Academic report files
│
├── tests/                     # Experiments and model artifacts
│   ├── final/
│   ├── prosody/
│   ├── text/
│   └── whisper/
│
├── docker-compose.yml
├── .env.example
└── README.md

---

Main API Flow

The frontend mainly interacts with the Node.js API through endpoints conceptually equivalent to:

POST   /api/journals
GET    /api/journals
GET    /api/journals/:id
GET    /api/journals/:id/status
GET    /api/recommendations
POST   /api/recommendations/generate-daily
POST   /api/recommendations/:id/complete
POST   /api/recommendations/:id/feedback

The Python analysis service receives analysis jobs and returns the result through a callback URL provided by the Node.js API.

---

Development Notes

Adding or changing files

After modifying source files:

sudo docker compose up --build

Checking whether analysis completed

Use the frontend timeline/status UI, or inspect logs:

sudo docker compose logs -f api-worker
sudo docker compose logs -f analysis

Keeping model downloads persistent

Whisper, WavLM and Ollama models can be large. If the Compose file uses volumes for model caches, avoid deleting volumes unless a full reset is needed.

Updating the prosody model

If a new prosody artifact is trained, replace the artifact file or set:

PROSODY_ARTIFACT_PATH=/path/to/new/artifact.joblib

Then rebuild/restart the analysis service.

---

Known Limitations

• The system is an MVP and is not intended for clinical diagnosis.
• The prosody classifier depends on the quality and domain of the training corpora.
• Emotional speech datasets often contain acted or controlled speech, which may differ from spontaneous daily journaling.
• Portuguese-first usage is supported, but some underlying datasets and models are multilingual or English-heavy.
• Local inference can be slow on machines without GPU acceleration.
• Large models may require significant disk space and RAM.
• The current setup is designed for local development rather than production deployment.
• Authentication and multi-user isolation are outside the current MVP scope.

---

Recommended Demo Flow

1. Start the stack with Docker Compose.
2. Open the frontend at http://localhost:5173.
3. Record a short daily voice entry.
4. Submit the recording.
5. Wait for analysis completion.
6. Open the journal timeline and inspect the transcription/emotion scores.
7. Check the daily trends chart.
8. Generate daily recommendations.
9. Mark a recommendation as completed or submit feedback.

---

License

This project is licensed under the MIT License. See the LICENSE file for details.

---

Authors

• Tomás Melo (PG60018)
• Rodrigo Ferreira (PG60392)
• Nuno Araújo (PG61218)