Conversational SHL Assessment Recommender

Overview

This project is a conversational recommendation agent that helps recruiters and hiring managers identify suitable SHL assessments through dialogue.

The system:

• clarifies vague hiring needs,
• recommends SHL assessments,
• supports refinement,
• compares assessments,
• refuses out-of-scope requests,
• and stays grounded to the SHL catalog.

---

Architecture

Backend

• FastAPI

Retrieval

• Hybrid retrieval:
  • BM25 lexical search
  • SentenceTransformer semantic embeddings

Vector Store

• ChromaDB

Ranking

• Hybrid weighted scoring
• Diversity reranking

Conversation Design

• Stateless orchestration
• State reconstruction from conversation history
• Clarification-before-recommendation policy

---

Features

Clarification

The agent asks targeted questions before recommending assessments when the user query is vague.

Recommendation

Returns grounded SHL assessments with:

• names,
• URLs,
• metadata.

Refinement

Conversation history is reconstructed on every request, enabling iterative refinement.

Comparison

Compares SHL assessments using catalog-grounded metadata.

Refusal

Refuses:

• non-SHL requests,
• prompt injection attempts,
• unrelated hiring/legal advice.

---

API Endpoints

Health Check

GET /health

Response:

{
  "status": "ok"
}

---

Chat Endpoint

POST /chat

Request:

{
  "messages": [
    {
      "role": "user",
      "content": "Hiring Java developers with analytical reasoning skills"
    }
  ]
}

Response:

{
  "reply": "For Software Developer hiring needs, these SHL assessments provide strong coverage across technical and cognitive evaluation.",
  "recommendations": [
    {
      "name": "Java 8 (New)",
      "url": "https://www.shl.com/...",
      "test_type": "Technical",
      "duration": "40 minutes"
    }
  ],
  "end_of_conversation": true
}

---

Running Locally

Install Dependencies

pip install -r requirements.txt

Build Vector Store

python build_vectorstore.py

Run Server

uvicorn app.main:app --reload

Swagger Docs

Open:

http://127.0.0.1:8000/docs

---

Design Decisions

Stateless Architecture

The evaluator provides the full conversation history in every request. The system reconstructs state from messages rather than storing server-side memory.

Hybrid Retrieval

Combines:

• semantic similarity,
• lexical matching,
• metadata-aware reranking.

This improves Recall@10 performance.

Grounding

Recommendations and comparisons are strictly grounded to the SHL catalog.

---

Limitations

• Current catalog is a representative subset of SHL assessments.
• Retrieval quality improves further with larger catalog ingestion.
• Rule-based state extraction may miss nuanced enterprise requirements.

---

Future Improvements

• Full SHL catalog ingestion
• Metadata filtering
• Advanced LLM-based JD parsing
• Automated evaluation harness
• Better comparison explanationsZ