Skip to content

README.md

Latest commit

 

History

History
268 lines (186 loc) · 7.68 KB

README.md

File metadata and controls

268 lines (186 loc) · 7.68 KB
Raw
title ASD Multi-Agent Web Game
emoji 🧠
colorFrom blue
colorTo purple
sdk docker
app_port 8501
tags
streamlit
ai
healthcare
autism
pinned false
short_description Multi-agent system for autism children

ASD Multi-Agent Web Game

Project Introduction

This is an agent-assisted system designed specifically for children with autism, providing personalized interactive experiences for children with autism through a multi-agent framework combined with eye-tracking technology.

Project Architecture

Core Components

  1. Multi-Agent Framework (multi_agent_framework.py)

    • Coordinates perception, decision, and action agents
    • Processes user input and generates responses

  2. Perception Agent (perception_agent.py)

    • Analyzes user input and context
    • Extracts key information and emotional states

  3. Decision Agent (decision_agent.py)

    • Makes decisions based on perception results
    • Selects the most suitable interaction strategy

  4. Action Agent (action_agent.py)

    • Executes decisions and generates responses
    • Creates HTML content and interactive elements

  5. Eye-Tracking Module (gaze_tracking/)

    • Real-time eye movement tracking
    • Provides attention analysis functionality

  6. Web Interface (app.py)

    • Streamlit-based user interface
    • Provides intuitive interactive experience

Features

Main Features

  • Intelligent Dialogue: Multi-turn dialogue system based on LangChain
  • Eye Tracking: Real-time monitoring of user attention status
  • Personalized Response: Adjusts interaction strategies based on user status
  • Web Interface: User-friendly interface design

Technical Features

  • Multi-Agent Collaboration: Three-layer architecture of perception-decision-action
  • Real-time Processing: Supports real-time eye movement data analysis and response
  • Scalability: Modular design for easy feature expansion

Installation and Usage

Requirements

  • Python 3.8+
  • Camera device (for eye tracking)
  • Stable internet connection

Deployment Methods

Method 1: Hugging Face Spaces (Recommended)

  1. Access Application: Directly visit [Hugging Face Spaces Link]
  2. No Installation Required: Application is already running in the cloud, ready to use
  3. Environment Variables: Configure OPENAI_API_KEY in Spaces settings

Method 2: Local Deployment

  1. Clone Project

    git clone [project address]
cd ASD_agent

  2. Install Dependencies

    pip install -r requirements.txt

  3. Configure Environment Variables Copy environment variable template and configure:

    cp env.example .env

    Then edit the .env file and enter your actual API key:

    OPENAI_API_KEY=your OpenAI API key

  4. Run Application

    streamlit run app.py

Method 3: Docker Deployment

  1. Build Image

    docker build -t asd-agent .

  2. Run Container

    docker run -p 8501:8501 -e OPENAI_API_KEY=your_key asd-agent

Usage Instructions

  1. Start Application: After running the above commands, the browser will automatically open the application interface
  2. Camera Authorization: First-time use requires allowing camera access permissions
  3. Start Interaction: Enter content in the text box, and the system will generate responses through the multi-agent framework
  4. Eye Tracking: The system will automatically analyze the user's attention status

Dependency Library Description

Core Dependencies

  • numpy (>=1.22.0): Numerical computation and array operations
  • opencv-python (>=4.2.0.32): Computer vision and image processing
  • dlib (>=19.16.0): Face detection and feature point recognition

Web Framework

  • streamlit (>=1.28.0): Web application development framework

AI and Language Processing

  • langchain (>=0.1.0): Large language model application framework
  • langchain-openai (>=0.0.5): OpenAI model integration

Utility Libraries

  • python-dotenv (>=1.0.0): Environment variable management
  • websockets (>=11.0.0): WebSocket communication

Project Structure

ASD_agent/
├── app.py                      # Main application entry (Streamlit app)
├── multi_agent_framework.py    # Multi-agent framework
├── perception_agent.py         # Perception agent
├── decision_agent.py           # Decision agent
├── action_agent.py             # Action agent
├── example.py                  # Example code
├── test.py                     # Test code
├── requirements.txt            # Dependency library list
├── Dockerfile                  # Docker configuration file
├── gaze_tracking/              # Eye-tracking module
│   ├── __init__.py
│   ├── gaze_tracking.py        # Main tracking class
│   ├── eye.py                  # Eye detection
│   ├── pupil.py                # Pupil detection
│   ├── calibration.py          # Calibration functionality
│   └── trained_models/         # Pre-trained models
└── README.md                   # Project description

Development Notes

Code Standards

  • All Python files contain detailed comments
  • Use type hints to improve code readability
  • Follow PEP 8 coding standards

Extension Development

  • Add new agents: Implement agent logic in corresponding files
  • Modify interface: Edit Streamlit components in app.py
  • Optimize eye tracking: Adjust parameters in the gaze_tracking/ module

Hugging Face Spaces Deployment

Spaces Configuration Description

This project is configured to run directly on Hugging Face Spaces:

  • SDK: Docker
  • Port: 8501
  • Framework: Streamlit
  • Tags: streamlit, ai, healthcare, autism

Environment Variable Configuration

In Hugging Face Spaces, you need to configure the following environment variables on the Settings page:

  • OPENAI_API_KEY: Your OpenAI API key

Important Notes

  • Ensure the API key is valid and has sufficient quota
  • Camera functionality may be limited in cloud environments
  • It's recommended to test eye-tracking functionality in a local environment

Security Reminders

⚠️ Important Security Notice:

  • Never commit .env files containing real API keys to Git repositories
  • Ensure .env files are added to .gitignore
  • In Hugging Face Spaces, configure environment variables through the Settings page
  • If API keys are accidentally leaked, regenerate them immediately

Troubleshooting

Common Issues

  1. Camera Cannot Access

    • Check browser permission settings
    • Ensure camera is not occupied by other applications

  2. Dependency Installation Failed

    • Ensure Python version compatibility
    • Try using a virtual environment

  3. API Call Failed

    • Check if OpenAI API key is correct
    • Confirm network connection is normal

Performance Optimization

  • Adjust eye-tracking parameters to improve accuracy
  • Optimize multi-agent response speed
  • Adjust image processing parameters based on hardware configuration

Contributing Guidelines

Welcome to submit Issues and Pull Requests to improve the project. Before submitting code, please ensure:

  • Code passes all tests
  • Add necessary documentation and comments
  • Follow project coding standards

License

This project is licensed under the MIT License. See LICENSE file for details.

Contact Information

For questions or suggestions, please contact through:

  • Submit GitHub Issue
  • Send email to project maintainer

Note: This project is designed specifically for children with autism. Please ensure it is used under professional guidance and adjust interaction strategies according to specific needs.