feat: initial web_platform offering#28
Open
AlankritVerma01 wants to merge 135 commits into
Open
Conversation
…ading process - Changed ArcPlusClassifier's device parameter to auto-detect the best available device. - Added background loading capability for tools in the API, allowing for asynchronous loading of large models. - Introduced a new loading status for tools and updated related logging for better tracking of tool loading processes.
- Added new tools: langchain-google-genai, langgraph, and langgraph-checkpoint to requirements. - Improved chat response streaming by integrating MedRAX agent and error handling for tool loading. - Enhanced logging for better error tracking during message processing and tool execution. - Implemented auto-polling for tool status updates in the settings interface. - Updated date formatting functions to handle null or invalid inputs gracefully.
- Deleted the test file `test_agent_integration.py` which contained comprehensive tests for agent functionality, memory/context persistence, tool execution, and error handling. - This removal may be part of a refactor or restructuring of the testing framework.
… handling - Refactored the ChatProcessor to improve result parsing, including JSON and tuple handling for tool outputs. - Added functionality to extract and store generated image paths from tool results. - Updated Message component to enhance the display of attached scans with improved styling. - Modified Sidebar and PatientCard components for better layout and user experience. - Improved ToolHistoryDrawer and ToolResultCard components to handle tool execution data more effectively, including renaming fields for consistency. - Updated API response handling in patients and tool history services to align with new data structures.
- Updated chat response to include last message timestamp, message count, and scan count for better context. - Refactored message listing to enrich tool execution data with computed fields such as execution time and display names. - Improved API response handling in chat and message services to align with new data structures. - Adjusted frontend API calls to accommodate changes in chat response format and ensure consistent data mapping.
- Add device detection with CUDA/CPU fallback - Support environment variables (CUDA, FORCE_CPU, DEVICE) - Utility functions for PyTorch and HuggingFace models - Proper error handling and logging - GPU availability checking and reporting
- Update ArcPlus and TorchXRayVision classifiers - Replace manual device detection with centralized utility - Add proper logging for initialization and warnings - Update type hints for Pydantic V2 compatibility - Add CPU fallback warnings
- Update LLaVA-Med and CheXagent VQA tools - Add MedGemma API client support with configurable URL - Replace manual device detection with centralized utility - Add proper logging throughout - Update type hints for Pydantic V2 - Add CPU mode support with warnings
- Update MedSAM2 and ChestXRay segmentation tools - Replace manual device detection with centralized utility - Add proper logging and error handling - Update type hints for Pydantic V2 - Improve cache directory handling
- Update radiology report generator and X-ray generator - Replace manual device detection with centralized utility - Add proper logging and dtype handling - Update type hints for Pydantic V2 - Improve device-based dtype selection
- Patch BaseImageProcessor for transformers compatibility - Patch LlavaProcessor to accept MAIRA-2 parameters - Fix rope_scaling validation bug in config loading - Load config with base classes to bypass validation - Add comprehensive logging for debugging - Migrate to Pydantic V2 and centralized device management This fixes multiple upstream compatibility issues with MAIRA-2 model on HuggingFace, enabling it to load successfully.
- Fix return types to Tuple[Dict, Dict] (LangChain contract) - Update API calls for duckduckgo-search v4.4.3 - Remove context manager pattern - Add better null checking for results - Simplify get_search_summary to handle tuples correctly - Remove excessive comments and clean up code
- Simplify tool result parsing for Tuple[Dict, Dict] returns - Remove emojis from status messages for professionalism - Add eager loading to avoid N+1 database queries - Add API configuration for MedGemma and Google services - Remove excessive comments and clean up code - Improve error handling and logging
- Fix chat scrolling behavior - Improve message rendering and formatting - Auto-create Initial Consultation chat for new patients - Remove excessive comments - Better tool output display - Layout adjustments for responsive behavior
- Update pinecone-client to pinecone (package renamed) - Enforce Python 3.11 in startup script - Improve venv activation and dependency installation - Add asyncio event loop configuration
- Add DuckDuckGo search tool tests - Add comprehensive tests for all tools - Test tool loading/unloading functionality - Verify return types and structures - Test error handling
- Add SSE endpoint for real-time progress streaming - Zero polling - server pushes updates to client - Add SSE-compatible authentication (token as query param) - Monitor tool loading with 2-second intervals - Send keepalive events to prevent connection timeout - 30-minute timeout protection - Graceful error handling and connection closure Performance: 40x fewer API calls during tool loading
- Add useToolLoadingSSE hook for real-time updates - EventSource-based streaming (no polling) - Automatic connection management and cleanup - Progress callbacks for UI updates - Add ToolLoadingProgress component with progress bar - Real-time percentage and message display - Error state handling Eliminates hundreds of polling requests during tool loading
CRITICAL FIX: Remove polling that was hammering backend Before: - Frontend polled GET /api/tools every 3 seconds - 400+ requests during 20-minute tool load - Backend logs filled with repeated API calls After: - Single EventSource connection per tool load - Real-time progress updates via SSE - Progress bar shows live status (0-100%) - Connection closes when complete - Zero polling Impact: Eliminates 99% of tool status API calls during loading
… loads CRITICAL FIXES after end-to-end review: 1. Memory Leak Fix (CRITICAL): - Add eventSourceRef to store EventSource instance - Cleanup on component unmount - Prevents dangling connections if user navigates away 2. API Config Consistency: - Use centralized API_CONFIG.baseURL - Remove hardcoded process.env check - Consistent URL configuration 3. Multiple Load Protection: - Disable all Load buttons when any tool is loading - Clear existing connection before starting new one - Prevents UI confusion and state conflicts 4. Better State Management: - Clear eventSourceRef on connection close - Consistent ref and state cleanup - Proper connection lifecycle These fixes ensure production-ready SSE implementation.
CRITICAL AUTH BUG FIX:
Issue: Users got 'Authentication required' even when logged in
Root Cause: Token stored as 'medrax_auth_token' but code looked for 'token'
Fixes:
1. ToolsSettings: Use authStore to get token directly
2. useToolLoadingSSE: Use AUTH_CONFIG.tokenKey instead of hardcoded 'token'
Before:
localStorage.getItem('token') // ❌ Wrong key
After:
- ToolsSettings: const { token } = useAuthStore() // ✅ From store
- Hook: localStorage.getItem(AUTH_CONFIG.tokenKey) // ✅ Correct key
This ensures SSE connections use the correct authentication token.
CRITICAL FIX for tool initialization errors:
Issue: Some tools require parameters but were instantiated with tool_class()
- RAGTool: needs RAGConfig parameter
- MedGemmaAPIClientTool: needs api_url parameter
Changes:
1. Added conditional initialization logic in _load_tool_instance()
2. RAGTool: Create default RAGConfig with proper defaults
3. MedGemmaAPIClientTool: Use MEDGEMMA_API_URL from settings or default
Before:
return tool_class() # ❌ Fails for tools needing params
After:
if tool.tool_class == 'RAGTool':
config = RAGConfig() # ✅ Proper config
return tool_class(config)
elif tool.tool_class == 'MedGemmaAPIClientTool':
api_url = settings.MEDGEMMA_API_URL # ✅ Proper API URL
return tool_class(api_url=api_url)
else:
return tool_class() # ✅ Works for other tools
Fixes errors:
- 'RAGTool.__init__() missing 1 required positional argument: config'
- 'MedGemmaAPIClientTool.__init__() missing 1 required positional argument: api_url'
CRITICAL FIX for RAG tool initialization: Issue: Cohere SDK requires CO_API_KEY environment variable - RAGTool uses Cohere for chat, embeddings, and reranking - RAGTool uses Pinecone for vector storage - Environment variables weren't set before tool initialization Changes: 1. Set CO_API_KEY from settings.COHERE_API_KEY before RAGTool init 2. Set COHERE_API_KEY as fallback 3. Set PINECONE_API_KEY from settings 4. Added logging for API key configuration Before: config = RAGConfig() # ❌ Cohere not configured return tool_class(config) After: os.environ['CO_API_KEY'] = settings.COHERE_API_KEY # ✅ os.environ['PINECONE_API_KEY'] = settings.PINECONE_API_KEY # ✅ config = RAGConfig() return tool_class(config) Fixes error: 'The client must be instantiated be either passing in token or setting CO_API_KEY' User needs to add to .env: COHERE_API_KEY=<key> PINECONE_API_KEY=<key>
- Introduced a new endpoint for bulk loading tools, allowing users to load multiple tools at once. - Added request and response schemas for bulk loading operations. - Updated the tool management API to support bulk loading, enhancing efficiency in tool management. - Improved the frontend to include bulk load options in the ToolsSettings component, allowing users to load all or selected tools easily. This feature streamlines the tool loading process, making it more user-friendly and efficient.
- Changed logger level from info to debug for unsupported LlavaProcessor kwargs in XRayPhraseGroundingTool. - Enhanced device handling in CheXagentXRayVQATool to set dtype based on the accelerator (CUDA, MPS, or CPU). - Updated device detection logic in get_device to include MPS for Apple Silicon. - Improved backend startup script to prioritize conda environments and streamline virtual environment setup. - Added checks for existing environments and clarified installation instructions in README. - Implemented background loading management in ToolManager to limit concurrent loads and ensure clean shutdown. - Enhanced error handling during tool loading and shutdown processes.
AlankritVerma01
force-pushed
the
feat/web-platform
branch
from
d49871e to
017b00e
Compare
- Added support for loading environment variables from a .env file in the backend startup script. - Updated backend configuration to require SECRET_KEY and API_SECRET_KEY from the .env file for improved security. - Implemented API secret validation middleware to protect sensitive endpoints and prevent unauthorized access. - Enhanced logging for request handling, including suspicious activity detection. - Updated frontend to manage API secret in localStorage, ensuring it is included in requests. - Improved API client configuration to handle API secret and provide clearer error messages for authentication issues.
- Removed unused import from ToolsSettings component to streamline code. - Updated useToolLoadingSSE hook to enforce type safety for tool data, ensuring proper handling of tool loading progress and completion. - Corrected API base URL reference in useToolLoadingSSE for consistency across the application. These changes enhance code clarity and maintainability while ensuring accurate tool loading behavior.
…bulk loading capabilities - Updated ToolsSettings component to utilize Server-Sent Events (SSE) for real-time tool loading progress, replacing previous polling methods. - Implemented bulk loading functionality, allowing users to load all or selected tools simultaneously with live progress updates. - Improved state management for loading tools, ensuring accurate UI representation during loading operations. - Enhanced error handling and user feedback for tool loading processes, including real-time status updates and connection management. These changes significantly improve the user experience and efficiency of tool management in the application.
- Upgraded huggingface-hub from version 0.33.2 to 0.34.0 across pyproject.toml, environment.yml, and requirements.txt for improved functionality. - Updated tokenizers from version 0.20.3 to 0.21.0 in the same files to ensure compatibility with the latest features.
- Included the python-jose[cryptography] dependency in pyproject.toml, environment.yml, and requirements.txt to support JSON Web Token (JWT) handling in the backend. - Ensured consistency across all configuration files by adding the new dependency in the same locations.
- Added a check to ensure that the model continues with default-initialized tokens if loading embed token weights fails. - Refactored the logic for assigning embed token weights to input embeddings, ensuring proper shape validation and error handling. - Enhanced code readability and maintainability by restructuring the weight assignment logic.
- Changed the parameter name from `dtype` to `torch_dtype` for clarity and consistency in the model loading function. - Ensured that the correct data type is used based on the device type (CUDA or CPU).
VictorLJZ
force-pushed
the
feat/web-platform
branch
from
44e2851 to
3ce68fc
Compare
AlankritVerma01
force-pushed
the
feat/web-platform
branch
2 times, most recently
from
44e2851 to
dacd5de
Compare
- Updated the ToolResultCard component to improve parsing of raw tool output, ensuring better handling of structured data and metadata. - Introduced utility functions to convert Python-like output into JSON-compatible format. - Enhanced the rendering logic to sort pathology predictions for clearer presentation. - Adjusted the backend startup script to disable eager loading of tools for improved performance during development.
- Updated the scan retrieval logic to handle cases where scan_ids is explicitly provided as an empty list, preventing unnecessary fallback to chat history. - Improved logging to clarify when no scans are retrieved due to empty scan_ids, enhancing debugging and user feedback.
- Added logic to update the chat store with the latest chat details, ensuring the sidebar and chat list remain in sync with message counts and chat names. - Implemented a condition to clear the selected chat ID when the associated patient chat IDs change, improving user experience and state management.
- Added methods to extract, scale, and visualize bounding boxes from model responses, improving the interpretability of findings. - Implemented a temporary directory for storing generated visualizations, ensuring organized output management. - Updated response structure to include findings and visualization paths, enhancing the output data for better user experience.
- Refactored the MedGemma VQA tool to load the model and processor directly instead of using a pipeline, enhancing control over image token handling. - Added support for an optional list of image paths to accommodate legacy input formats. - Improved image path resolution logic to handle both single and multiple image inputs. - Updated logging messages for clarity regarding model loading and image processing. - Adjusted backend startup script to pin GPU usage to a specific device for consistency.
…image handling - Upgraded Next.js from version 15.5.6 to 15.5.7 in package.json and package-lock.json for improved performance and security. - Enhanced the MarkdownRenderer component to normalize backend image paths using a new utility function, improving image loading reliability and error handling.
- Introduced methods to resolve image paths and infer MIME types for better image encoding in the ChatProcessor. - Added functionality to convert DICOM files to PNG format for improved display handling. - Updated image processing logic to utilize the new utilities, ensuring compatibility with both DICOM and standard image formats. - Enhanced logging for image processing to provide clearer insights into the encoding process and potential errors.
- Added .env1 to the .gitignore file to prevent accidental inclusion of environment files. - Modified the build script in package.json to remove the turbopack flag for compatibility. - Enhanced vercel.json by adding an outputDirectory property for better deployment configuration. - Improved cleanup logic in ToolsSettings component to ensure proper closure of SSE connections. - Updated ToolResultCard component to enhance type handling and improve data normalization. - Added eslint disable comments for image handling in ToolOutputsSidebar and MarkdownRenderer components to address Next.js image element warnings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.