Gap Detection Methodology

This document explains the coordination gap detection system, algorithms, and best practices for detecting and resolving organizational coordination failures.

Note: Currently, only duplicate work detection is fully implemented. Other gap types (missing context, stale docs, knowledge silos) are planned for future milestones.

Overview
Detection Pipeline
Duplicate Work Detection
Confidence Scoring
Impact Assessment
Tuning Parameters
Best Practices
Troubleshooting
Future Gap Types

Overview

What is a Coordination Gap?

A coordination gap occurs when organizational information flows break down, causing:

Duplicate Work: Teams independently solving the same problem ✅ Implemented
Missing Context: Decisions made without critical stakeholder input (planned)
Stale Documentation: Documentation contradicting current implementation (planned)
Knowledge Silos: Critical knowledge trapped within individual teams (planned)

How Gap Detection Works

The system uses a multi-stage AI pipeline combining:

Semantic Analysis: Clustering similar discussions using embeddings
Entity Extraction: Identifying teams, people, and projects
Temporal Analysis: Detecting work happening simultaneously
LLM Reasoning: Using Claude to verify actual coordination failures
Impact Scoring: Quantifying organizational cost

Detection Pipeline

Stage 1: Data Retrieval

# Retrieve messages from specified timeframe
messages = await retrieve_messages(
    timeframe_days=30,
    sources=["slack"],
    channels=["#engineering", "#platform"]
)

What happens:

Fetch messages from PostgreSQL database
Filter by timeframe, sources, channels
Load embeddings from vector store

Stage 2: Semantic Clustering

# Group similar technical discussions
clusters = semantic_clusterer.cluster(
    messages,
    similarity_threshold=0.85,  # 85% similarity required
    time_window_days=30,        # Within 30-day window
    min_cluster_size=2          # At least 2 messages
)

What happens:

Compute cosine similarity between message embeddings
Apply DBSCAN clustering algorithm
Group messages with >85% semantic similarity
Filter clusters within time window

Key Insight: Similar discussions about "OAuth" or "API design" get grouped together, even if in different channels.

Stage 3: Entity Extraction

# Identify teams, people, projects in each cluster
for cluster in clusters:
    entities = entity_extractor.extract(cluster.messages)

    # entities contains:
    # {
    #   "teams": ["platform-team", "auth-team"],
    #   "people": ["alice@company.com", "bob@company.com"],
    #   "projects": ["OAuth", "authentication"],
    #   "topics": ["implementation", "security"]
    # }

What happens:

Extract @mentions (teams and people)
Identify email addresses
Detect channel-based teams (#platform → platform-team)
Find project/feature mentions
Extract technical terms

Key Insight: If multiple teams are found in a cluster, it's a potential coordination gap.

Stage 4: Temporal Overlap Analysis

# Check if teams are working simultaneously
def has_temporal_overlap(cluster, teams, min_overlap_days=3):
    team_timelines = {}

    for msg in cluster.messages:
        team = msg.metadata.get("team")
        if team not in team_timelines:
            team_timelines[team] = []
        team_timelines[team].append(msg.timestamp)

    # Calculate overlap between teams
    overlap = compute_timeline_overlap(team_timelines)
    return overlap >= min_overlap_days

What happens:

Build timeline for each team's messages
Calculate overlapping time periods
Require minimum 3 days of simultaneous work

Key Insight: Teams working 2 months apart = knowledge sharing (good). Teams working same 2 weeks = potential duplication (investigate).

Stage 5: LLM Verification (Optional)

# Use Claude to verify actual duplication
verification = await claude_client.verify_duplicate_work(
    messages=cluster.messages,
    teams=entities["teams"],
    topic=cluster.topic,
    context=cluster.summary
)

# verification contains:
# {
#   "is_duplicate": true,
#   "confidence": 0.87,
#   "reasoning": "Both teams independently implementing OAuth2...",
#   "evidence": ["Quote 1", "Quote 2"],
#   "recommendation": "Connect alice@ and bob@ immediately",
#   "overlap_ratio": 0.8  # 80% of work is duplicated
# }

What happens:

Send cluster context to Claude API (if enabled)
LLM analyzes if work is truly duplicated or intentionally coordinated
Returns structured verification with reasoning
Provides action recommendations

Key Insight: LLM can distinguish between:

✅ Duplication: "Starting OAuth implementation" (team A) + "Building OAuth support" (team B)
❌ Collaboration: "Working with @team-b on OAuth" + "@team-a is handling auth flow"

Stage 6: Gap Creation & Response

if verification.is_duplicate and verification.confidence > 0.7:
    gap = CoordinationGap(
        id=generate_gap_id(),
        type="DUPLICATE_WORK",
        topic=cluster.topic,
        teams_involved=entities["teams"],
        evidence=collect_evidence(cluster, verification),
        impact_score=calculate_impact(cluster, entities),
        confidence=verification.confidence,
        verification=verification,
        recommendation=verification.recommendation,
        detected_at=datetime.utcnow()
    )

    # Return in API response (not persisted to DB)
    gaps.append(gap)

What happens:

Only gaps with >70% confidence are included
Evidence is collected and ranked by relevance
Impact is scored (0-1 scale)
Gap returned in API response

Note: Gaps are currently returned in the detection response only. Persistent storage is planned for a future milestone.

Duplicate Work Detection

Detection Criteria

A cluster must meet ALL of these to be flagged as duplicate work:

✅ Semantic Similarity > 0.85: Messages discuss similar topics
✅ Multiple Teams (≥2): At least 2 different teams involved
✅ Temporal Overlap (≥3 days): Teams working simultaneously
✅ LLM Verification (confidence >0.7): Claude confirms duplication (if enabled)

Exclusion Rules

The following scenarios are NOT flagged as duplicate work:

❌ Explicit Collaboration: Cross-references present (@team mentions)
❌ Different Scopes: LLM identifies distinct goals (e.g., user auth vs service auth)
❌ Sequential Work: No temporal overlap (Team B starts after Team A completes)
❌ Mentor/Mentee Pattern: One team helping another learn
❌ Intentional Redundancy: Backup systems, A/B tests, disaster recovery

Example: OAuth Duplication (Detected)

Platform Team (#platform channel):
  Dec 1, 09:00 - "Starting OAuth2 implementation for API gateway"
  Dec 3, 10:15 - "Finished OAuth flow research, going with authorization code"
  Dec 5, 14:20 - "Got token validation working"

Auth Team (#auth-team channel):
  Dec 1, 14:20 - "We're building OAuth support for auth service"
  Dec 3, 15:30 - "Decided on authorization code flow"
  Dec 5, 16:45 - "Token endpoint complete"

Detection Result:
✅ Semantic Similarity: 0.92 (both discussing OAuth2 implementation)
✅ Multiple Teams: 2 teams (platform-team, auth-team)
✅ Temporal Overlap: 5 days of parallel work
✅ LLM Confidence: 0.89
✅ No Cross-References: Teams not aware of each other

→ DUPLICATE WORK DETECTED (High Impact)

Example: Collaboration (NOT Detected)

Platform Team:
  "Working with @auth-team on OAuth integration. We're handling the client flow."

Auth Team:
  "@platform-team is building the client, we're doing the server. Coordinating via sync meetings."

Detection Result:
✅ Semantic Similarity: 0.88
✅ Multiple Teams: 2 teams
❌ Collaboration Detected: Cross-references present
❌ LLM Verdict: "Teams are intentionally collaborating with clear division of labor"

→ NOT DUPLICATE WORK (Collaboration)

Confidence Scoring

Confidence Formula

confidence = (
    0.3 * semantic_similarity_score +   # How similar are discussions?
    0.2 * team_separation_score +       # How isolated are teams?
    0.3 * temporal_overlap_score +      # How much simultaneous work?
    0.2 * llm_confidence                # LLM verification strength
)

Confidence Tiers

Score	Tier	Meaning	Action
0.9-1.0	Very High	Extremely confident this is duplicate work	Immediate intervention
0.8-0.9	High	Strong evidence of duplication	Intervene within 24h
0.7-0.8	Medium-High	Likely duplicate, needs review	Flag for review
0.6-0.7	Medium	Possible duplicate, monitor	Monitor situation
<0.6	Low	Insufficient evidence	No action (filtered out)

Calibration

Confidence scores are calibrated against:

Known duplicate work scenarios (ground truth)
False positive examples (collaboration, different scopes)
Historical detection accuracy

Target Performance:

Precision: >80% (low false positives)
Recall: >70% (catch most duplicates)
F1 Score: >75%

Impact Assessment

Impact Score Formula

impact_score = (
    0.25 * team_size_score +            # How many people affected?
    0.25 * time_investment_score +      # How much time wasted?
    0.20 * project_criticality_score +  # How important is this?
    0.15 * velocity_impact_score +      # What's blocked?
    0.15 * duplicate_effort_score       # How much actual duplication?
)

Range: [0, 1] where 1 = catastrophic waste

Impact Tiers

Critical (0.8 - 1.0) 🔴

Multiple large teams (10+ people)
100+ engineering hours wasted
Roadmap/OKR-critical work
Action: Immediate intervention required

High (0.6 - 0.8) 🟠

5-10 people affected
40-100 hours wasted
Important projects
Action: Address within 1 week

Medium (0.4 - 0.6) 🟡

2-5 people affected
10-40 hours wasted
Moderate importance
Action: Monitor and advise teams

Low (0.0 - 0.4) 🟢

Small scope (<10 hours)
Low criticality
Action: FYI notification only

Cost Estimation

# Organizational waste cost (not Claude API cost)
avg_hourly_rate = 100  # $100/hour loaded engineer cost
estimated_hours = calculate_time_investment(gap)
estimated_cost = estimated_hours * avg_hourly_rate

# Example: 60 hours × $100 = $6,000 organizational waste
# Claude API cost is separate: ~$0.01-0.05 per gap verification

Tuning Parameters

Similarity Threshold

similarity_threshold = 0.85  # Default: 85% similarity

Increase (0.90) if:

Too many false positives
Need higher precision
Want only extremely similar work

Decrease (0.80) if:

Missing obvious duplicates
Teams use different terminology
Want higher recall

Temporal Overlap Minimum

min_overlap_days = 3  # Default: 3 days

Increase (5-7 days) if:

Short overlaps causing false positives
Want sustained parallel effort only

Decrease (1-2 days) if:

Fast-moving teams (sprints)
Short project cycles

LLM Confidence Threshold

min_llm_confidence = 0.7  # Default: 70%

Increase (0.8) if:

Too many low-confidence gaps
Want only clear-cut cases

Decrease (0.6) if:

Missing edge cases
Want to catch uncertain situations

Impact Score Filter

min_impact_score = 0.6  # Default: High+ impact only

Increase (0.8) if:

Only want critical gaps
High volume of gaps

Decrease (0.4) if:

Want to see all gaps
Learning mode

Best Practices

1. Start Conservative

# Recommended initial settings
{
    "similarity_threshold": 0.90,      # High precision
    "min_overlap_days": 5,             # Sustained effort
    "min_llm_confidence": 0.8,         # Clear cases only
    "min_impact_score": 0.7            # High+ impact
}

Gradually relax thresholds as you gain confidence.

2. Regular Monitoring

Run detection weekly
Review false positives/negatives
Adjust thresholds based on results
Track precision/recall over time

3. Act on High-Impact Gaps Immediately

Critical gaps (0.8+): Intervene within 24h
High gaps (0.6-0.8): Address within 1 week
Document resolution outcomes
Use for future calibration

4. Provide Context to Teams

When notifying teams about gaps:

Share full evidence (messages, timestamps)
Explain detection reasoning
Provide recommendation
Ask for feedback on accuracy

5. Iterate on Prompts

LLM verification prompts can be tuned:

Add domain-specific context
Include organizational patterns
Refine verification questions
Test prompt variations

Troubleshooting

Issue: No Gaps Detected (Expected Some)

Possible Causes:

Similarity threshold too high → Lower to 0.80-0.85
Temporal overlap too strict → Reduce min_overlap_days to 2-3
Impact filter excluding results → Lower min_impact_score to 0.4
LLM confidence too high → Lower to 0.6-0.7

Debugging:

# Check detection output
curl -X POST http://localhost:8000/api/v1/gaps/detect \
  -H "Content-Type: application/json" \
  -d '{"timeframe_days": 30, "min_impact_score": 0.0}'

# Review logs for cluster statistics
docker compose logs api | grep "clusters_found"

Issue: Too Many False Positives

Possible Causes:

Similarity threshold too low → Increase to 0.90-0.92
Collaboration not being detected → Review LLM prompt
Different scopes confused → Add scope detection to prompt

Resolution:

Review false positives to find patterns
Adjust exclusion rules
Refine LLM verification prompt
Add domain-specific heuristics

Issue: Slow Detection Performance

Performance Targets:

Entity extraction: <20ms per message
Clustering: <500ms for 1000 messages
LLM verification: <2s per gap
Full detection: <5s for 30 days

Optimization Steps:

Enable embedding caching (24h TTL)
Batch LLM calls (verify multiple gaps together)
Reduce timeframe_days
Limit sources/channels
Use faster Claude model (Haiku for verification)

Issue: LLM API Errors

Common Errors:

Rate Limit (429): Exponential backoff, reduce request rate
Invalid API Key (401): Check ANTHROPIC_API_KEY in .env
Timeout (408): Increase timeout, use shorter prompts
Quota Exceeded (429): Wait for quota reset, implement daily limits

Monitoring:

# Check API usage
docker compose logs api | grep "LLM_API"

# View retry attempts
docker compose logs api | grep "retry"

Future Gap Types

The following gap types are planned for future milestones:

Missing Context Detection (Planned)

Detect when important decisions are made without key stakeholders:

Identify decision points in discussions
Check for required stakeholder participation
Flag missing perspectives

Stale Documentation Detection (Planned)

Detect when documentation contradicts current implementation:

Compare docs with recent code changes
Detect semantic drift over time
Flag outdated references

Knowledge Silo Detection (Planned)

Detect when critical knowledge is trapped in one team:

Build knowledge graph (who knows what)
Identify single points of failure
Measure cross-team information flow

Summary

Key Takeaways:

Multi-Stage Pipeline: Clustering → Entity Extraction → Temporal Analysis → LLM Verification
High Precision: Default thresholds favor low false positives
Tunable: Adjust parameters based on your organization
Evidence-Based: Every gap includes detailed evidence and reasoning
Actionable: Recommendations help teams resolve issues

Current Status:

✅ Duplicate work detection: Fully implemented
⏳ Gap persistence: Planned for future milestone
⏳ Other gap types: Planned for future milestones

Next Steps:

Last Updated: January 2025 Status: Duplicate work detection implemented; gap persistence and other gap types planned for future milestones

FilesExpand file tree

GAP_DETECTION.md

Latest commit

History

GAP_DETECTION.md

File metadata and controls

Gap Detection Methodology

Table of Contents

Overview

What is a Coordination Gap?

How Gap Detection Works

Detection Pipeline

Stage 1: Data Retrieval

Stage 2: Semantic Clustering

Stage 3: Entity Extraction

Stage 4: Temporal Overlap Analysis

Stage 5: LLM Verification (Optional)

Stage 6: Gap Creation & Response

Duplicate Work Detection

Detection Criteria

Exclusion Rules

Example: OAuth Duplication (Detected)

Example: Collaboration (NOT Detected)

Confidence Scoring

Confidence Formula

Confidence Tiers

Calibration

Impact Assessment

Impact Score Formula

Impact Tiers

Critical (0.8 - 1.0) 🔴

High (0.6 - 0.8) 🟠

Medium (0.4 - 0.6) 🟡

Low (0.0 - 0.4) 🟢

Cost Estimation

Tuning Parameters

Similarity Threshold

Temporal Overlap Minimum

LLM Confidence Threshold

Impact Score Filter

Best Practices

1. Start Conservative

2. Regular Monitoring

3. Act on High-Impact Gaps Immediately

4. Provide Context to Teams

5. Iterate on Prompts

Troubleshooting

Issue: No Gaps Detected (Expected Some)

Issue: Too Many False Positives

Issue: Slow Detection Performance

Issue: LLM API Errors

Future Gap Types

Missing Context Detection (Planned)

Stale Documentation Detection (Planned)

Knowledge Silo Detection (Planned)

Summary