Repository navigation
Conversation
- Add Prometheus alert rules with SLO-aligned metrics - Add Alertmanager configuration for alert routing - Add prometheus.yml for monitoring setup - Update observability documentation with alert details - Add release notes and release PR template - Bump version to 0.2.0 Includes: - Environment-aware alerting rules - Multi-environment deployment guidance - Release workflow documentation - Complete observability stack documentation
The Dockerfile was copying only pyproject.toml and uv.lock, but pyproject.toml references Readme.md, causing the build to fail with: FileNotFoundError: Readme path `/build/Readme.md` does not exist This fix includes: - Copy Readme.md in the build stage - Copy src/ directory to build context Docker build now succeeds.
## Changes ### CD Workflow Improvements - **Simplified cd.yml**: Removed problematic manifest update jobs that were causing token authentication errors - Removed persist-credentials configuration that was interfering with GitHub token - Kept build-and-push and security-scan jobs which work reliably - Clear and focused workflow for core CD tasks ### Semantic Release Setup - **Added release.yml**: New workflow for automatic semantic versioning - Uses python-semantic-release for intelligent version bumping - Automatically creates git tags and GitHub releases - Triggered on main branch commits - Follows semantic versioning: major.minor.patch based on commit messages ### Configuration - **Added [tool.semantic_release] section to pyproject.toml** - Configures version detection from pyproject.toml - Sets changelog file to docs/RELEASE_NOTES.md - Defines branch-specific prerelease rules (main=stable, dev=prerelease) ### Cleanup - **Removed sonar-scan.yml.disabled**: Cleaned up disabled SonarQube workflow - No longer needed, reducing clutter in workflow definitions ## Why These Changes 1. **CD workflow was failing**: Permission and token issues with create-pull-request action made manifest updates unreliable. Simplified to focus on proven reliable tasks. 2. **Manual tagging is error-prone**: Semantic-release automates version detection, tagging, and release creation based on conventional commit messages. 3. **Better developer experience**: Removing disabled workflows reduces confusion about what's actually being used. ## Next Steps - Merge to dev branch first for testing - Verify CI/CD pipeline still works - Merge to main for production use
- Ruff lint: 222 → 0 errors - mypy strict: 168 → 0 errors - Test coverage: 62% → 94% (122 tests) - Pre-commit hooks (ruff, mypy, standard hooks) - Dockerfile hardening (non-root user, healthcheck) - CI: mypy step, fetch-based Slack notifications - Poe tasks: typecheck, security, pre-commit, check-all - Integration tests: API contracts, middleware, error handling - Docs: ARCHITECTURE.md, DEVELOPMENT.md updated for v0.3.0 - Security: pip-audit 0 vulnerabilities
* fix: infrastructure alignment and workflow fixes - Replace axios with fetch() in cd.yml, release.yml, security.yml (actions/github-script@v7 does not bundle axios) - Add dev branch to security.yml triggers - Fix duplicate syncPolicy in ArgoCD PR preview ApplicationSet - Add AlertManager service to docker-compose - Add Grafana auto-provisioning (datasources + dashboards) - Add healthcheck to API service in docker-compose - Mount Prometheus alert rules volume - Fix test-cov to cover all packages (dataenginex, careerdex, weatherdex) - Add coverage.xml output for SonarCloud integration - Remove unused black dependency (ruff handles formatting) - Add coverage and version badges to README - Fix Grafana README import path typo - Update roadmap CSV to reflect v0.2.0 completion * feat: enrich Slack failure notifications with detailed context All failure notifications now include structured Slack blocks with: - Workflow name, branch, commit SHA (linked) - Triggered-by actor, event type - Direct link to failed run - Contextual hint about what likely failed --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
feat: v0.3.2 — DEX Module Epics & Project-Wide Improvements (#79)\n\nImplements all 6 DEX module epics (#37-#42) plus 9 project-wide improvements.\n77 files changed, 4640 insertions, 143 deletions.\n233 tests passing, 0 lint/mypy errors.\n\nCloses #37, closes #38, closes #39, closes #40, closes #41, closes #42
Bug: notify-deployment and notify-security jobs used success()/failure() at step level which checks current job steps — not upstream dependent jobs. This meant success notifications always fired and failure notifications never fired. Fix: Replace with explicit needs.*.result checks so notifications correctly reflect the actual build/scan outcomes. Changes: - cd.yml: notify-deployment now checks needs.build-and-push.result and needs.security-scan.result; enriched success message with branch, run link, and context footer; failure message now lists which specific jobs failed - security.yml: notify-security now checks needs.semgrep.result and needs.codeql.result; enriched with per-tool status fields - Bump version 0.3.2 → 0.3.3 (pyproject.toml, sonar-project.properties)
…hrough an exception Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…ep, poe install uses uv
# Conflicts: # poe_tasks.toml # pyproject.toml # uv.lock
…check fix (#99) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @AbstractMethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @AbstractMethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
#101) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @AbstractMethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
…on (#102) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @AbstractMethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD * chore: add image pull/list step before Trivy scan for GHCR verification --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
…tion to 0.34.1 (#103) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @AbstractMethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD * chore: add image pull/list step before Trivy scan for GHCR verification * fix: update image tag format and upgrade download-artifact action version --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
Updated Trivy image reference to use only the image tag output.
Auto-merge: GitOps dev image update (sha-eca077ab)
Auto-merge: GitOps dev image update (sha-de539bec)
… scheduler, docstrings (#128) * feat: add PyPI badge and update CHANGELOG (#87) * feat: storage abstraction — add list_objects, exists, get_storage factory (#89) - Add list_objects(prefix) and exists(path) abstract methods to StorageBackend ABC - Implement in all 6 backends: LocalParquetStorage, BigQueryStorage, JsonStorage, ParquetStorage, S3Storage, GCSStorage - Add get_storage(uri) factory: file:// → JsonStorage, s3:// → S3Storage, gs:// → GCSStorage - Export get_storage from lakehouse package - Add py.typed and update StorageBackend docstring - 12 new unit tests (264 total passed) * feat: generic model serving API endpoints (#92) - Add ML router with POST /predict, GET /models, GET /models/{name} - Add ML-specific Prometheus metrics (latency, throughput, drift) - Add PredictRequestBody/PredictResponseBody Pydantic models - Add set_model_server/get_model_server configuration helpers - Add 9 unit tests for ML router endpoints * test: storage integration tests with Docker emulators - Add endpoint_url param to S3Storage for LocalStack support - Add api_endpoint param to GCSStorage for fake-gcs-server support - Add docker-compose.test.yml (LocalStack:4566 + fake-gcs-server:4443) - Add LocalStack init script for test bucket creation - Add Terraform module for real cloud test buckets (AWS S3 + GCS) - Add 25 integration tests with emulator auto-detection - Fix unit tests to pass endpoint kwargs preventing auth failures * feat: drift monitoring scheduler with alerting (#93) - Add DriftScheduler with background thread for periodic checks - Add DriftMonitorConfig / DriftCheckResult dataclasses - Publish PSI scores to model_drift_psi Prometheus gauge - Increment model_drift_alerts_total counter on drift detection - Add Prometheus alert rules (moderate, severe, spike, stale) - Add mypy overrides for optional cloud SDKs (boto3, google) - Add 18 unit tests for scheduler lifecycle and metrics * docs: add docstrings to all public APIs (#88) - Add Google-style docstrings to 55+ methods and 9 Pydantic models - Cover core/schemas, api/, data/, lakehouse/, ml/, warehouse/ - Fix storage.py ParquetStorage read/delete try-block structure - All 40 source files pass mypy strict, 299 tests pass * chore: bump dataenginex to 0.5.0, fix docs workflow & Dockerfile - Bump dataenginex version from 0.4.11 to 0.5.0 - Fix docs-pages.yml: use uv sync --frozen instead of uv lock + uv sync - Fix Dockerfile: add missing COPY --from=builder /build/packages /app/packages - Update CHANGELOG with [0.5.0] - 2026-03-01 release - Update RELEASE_NOTES, CI_CD docs, roadmap CSV - Clean up mypy type-ignore annotations for cloud SDK imports * fix: resolve CI mypy unused-ignore and Trivy S3 encryption (AWS-0132) - Add per-module mypy override for storage module (warn_unused_ignores=false) handles env asymmetry: pre-commit has google-cloud-storage, CI does not - Keep # type: ignore[no-untyped-call] on AnonymousCredentials for pre-commit - Switch S3 test bucket from AES256 (SSE-S3) to SSE-KMS with CMK (aws_kms_key with rotation + bucket_key_enabled) Fixes CI workflow (mypy) and Security workflow (Trivy AWS-0132) * chore: add system setup script, consolidate READMEs to GUIDE.md - Add scripts/setup-system.sh: one-command system deps installer (Ubuntu/Debian, Fedora/RHEL, Arch, macOS) - Add poe task 'setup-system' for convenience - Update docs/DEVELOPMENT.md with system dependencies table - Update README.md Quick Start with setup-system.sh step - Rename 7 subdirectory README.md files to GUIDE.md (examples, scripts, packages, learning, src/careerdex, infra/monitoring/grafana, infra/terraform/test-storage) - Fix 3 cross-references to renamed files - Keep only root README.md and packages/dataenginex README.md --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
Auto-merge: GitOps dev image update (sha-0b003e0e)
Auto-merge: GitOps dev image update (sha-b2ebaa36)
…, docs overhaul feat: CareerDEX v0.5.0 — domain extraction, plugin system, ML modules, and docs overhaul (#138)
* feat: mount ml_router in CareerDEX and add OpenAPI tags (#68) - Import and include ml_router from dataenginex.api.routers.ml - Add v1 and ml tags to custom OpenAPI schema - Enables ML model serving endpoints in CareerDEX API * fix: add pre-commit to dev dependencies and re-sync venv - pre-commit was used by git hooks but not listed as dependency - venv had stale shebangs from old directory path (DEX → dex) - Re-synced venv fixes poe and pre-commit resolution * refactor: extract domain code from dataenginex to careerdex - Move routers (ml.py, v1.py) from dataenginex to careerdex/api/routers/ - Move pipeline_config.py from dataenginex to careerdex/core/ - Add careerdex-specific schemas (JobPosting, UserProfile, JobSourceEnum) - Add careerdex validators (SchemaValidator, QualityScorer, DataHash) - Add careerdex settings, exceptions, and config modules - Clean up dataenginex framework: remove domain-specific code - Update all phase imports to use careerdex.core instead of dataenginex - dataenginex now contains only generic framework code * feat: add vectorstore, LLM adapter, plugin system, and dashboard - RAG vectorstore: VectorStoreBackend ABC, InMemoryBackend, ChromaDBBackend - LLM adapter: LLMProvider ABC, OllamaProvider (REST), MockProvider - Plugin system: DataEngineXPlugin ABC, PluginRegistry, discover() via entry_points - Dashboard: DashboardConfig, BaseDashboard, 4 Streamlit panels - All modules exported from dataenginex.ml, dataenginex.plugins, dataenginex.dashboard * feat: add local file connectors, embeddings wiring, and CareerDEX plugin - JsonFileConnector and CsvFileConnector for local job data ingestion - JobSourceEnum extended with LOCAL_JSON and LOCAL_CSV - Phase 3 embeddings: sentence-transformers integration with fallback - CareerDEX plugin: health_check, get_metrics, register_routes - Plugin auto-discovered via entry_points group 'dataenginex.plugins' * test: add tests for domain extraction, plugins, connectors, and ML modules - 441 tests passing, 28 skipped (optional deps) - New test suites: plugins, vectorstore, LLM, careerdex phases - Updated existing tests for domain extraction import changes - Load testing with Locust for API performance benchmarking - Test fixtures: sample_jobs.json, sample_jobs.csv * chore: coverage gate, dependency updates, and build config - Add --cov-fail-under=80 to test-cov-core poe task - Add sentence-transformers to ml dependency group - Add streamlit to dashboard dependency group with uv conflict config - Add dataenginex.plugins entry_points for CareerDEX plugin discovery - Add mypy overrides for streamlit stubs - Sync uv.lock with all dependency changes * docs: update documentation, examples, and changelogs - README: add plugin system and dashboard sections, update tech stack - TODO: mark Streamlit dashboard as complete - Architecture docs: reflect domain extraction changes - New examples: RAG demo, LLM quickstart, dashboard runner - CHANGELOG: plugin system, dashboard, coverage gate entries - Add quickstart guide and security scanning docs - Update tasks with findings and lessons learned * chore: update CI/CD workflows, tooling, and project configs - Update CI/CD workflows: ci.yml, cd.yml, security.yml, release pipelines - Add Claude Code commands and settings - Update pre-commit hooks and gitignore patterns - Add gitleaks config for secret scanning - Update Dockerfile for new dependency groups - Update SonarQube and MCP server configs - Update agent/instruction/prompt files for new modules * chore: fix docs formatting, version refs, and add missing docs - Add API reference docs: plugins.md, dashboard.md - Update mkdocs nav with new module pages - Add Bruno API collection for all 8 CareerDEX endpoints - Update ADR medallion version reference * fix: align example 04_ml_training.py with actual API signatures - SklearnTrainer: use model_name/version/estimator constructor args - ModelRegistry: use persist_path, register(ModelArtifact) - DriftDetector: use check_dataset() instead of nonexistent detect() - Add proper sklearn import and evaluate() call * feat: boost test coverage to 83%+ via ML endpoint integration tests - Initialize ModelServer and ModelRegistry in app lifespan for ML endpoint support - Add comprehensive integration tests for ML endpoints (GET /models, GET /models/{name}, POST /predict) - Exclude dead code from coverage (dataenginex ml router, dashboard Streamlit modules) - Coverage improved from 79.45% to 83.62%, exceeding 80% gate - All tests passing (442 passed, 28 skipped) - Lint and typecheck passing --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
- Fix import ordering in examples (isort) - Fix line length in 04_ml_training.py - Add blank line after imports in test_plugins.py - Add autoUpdatesChannel to claude settings - Fix missing newline in .gitleaks.toml Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…deep clean - Remove src/careerdex/ and src/weatherdex/ entirely - Promote packages/dataenginex/ to repo root (src/dataenginex/) - Rewrite pyproject.toml as dataenginex 0.6.0 (was careerdex-app 0.5.0) - Convert weatherdex to numbered examples (07_api_ingestion, 08_spark_ml, 09_feature_engineering, 10_model_analysis) - Fix examples/02_api_quickstart.py (remove careerdex router, fix HealthChecker API) - Deep clean all stale careerdex/weatherdex/packages refs across docs, .github agents/prompts/instructions/workflows, CLAUDE.md, README.md, copilot-instructions.md, CHECKLISTS.md, GUIDELINES.md - Fix mypy sklearn type-ignore inconsistency (env-dependent installed/not-found) - Regenerate uv.lock; lint, typecheck, tests all pass (265 passed) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Deleted api-collection/ (Bruno HTTP collections — not appropriate for a pure library repo) - Deleted infra/ (Kubernetes manifests, Terraform, LocalStack config) - Relocated infra/monitoring/ → monitoring/ (kept docker-compose.yml functional) - Relocated infra/localstack/create-buckets.sh → scripts/localstack/ (kept docker-compose.test.yml functional) - Removed .github/workflows/cd.yml (broken: referenced infra/argocd/overlays/ paths via set -euo pipefail + sed; pure PyPI package repos don't need K8s CD) - Removed scripts/get-tags.sh (read from infra/argocd/overlays/ which no longer exists) - Simplified scripts/promote.sh: removed --image-tag mode that wrote to kustomization.yaml - Removed k8s-deploy/k8s-delete poe tasks - Updated all docs and configs to remove infra/ path references: - docker-compose.yml, docker-compose.test.yml: corrected volume mount paths - docs/CI_CD.md: stripped K8s/ArgoCD CD sections; CI + PyPI release only - docs/DEPLOY_RUNBOOK.md: repurposed as PyPI release runbook; removed K8s deploy steps - docs/OBSERVABILITY.md, docs/LOCAL_K8S_SETUP.md, docs/SECURITY_SCANNING.md: path corrections - .github/workflows/WORKFLOWS_GUIDE.md: removed cd.yml from diagram - .github/instructions/infrastructure.instructions.md: updated applyTo glob + paths - .github/agents/devops-engineer.agent.md: removed stale K8s/infra paths - CHANGELOG.md, docs/RELEASE_NOTES.md: corrected historical path references Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove copilot instructions, dependabot config, learning/ scripts, monitoring stack (moved to infradex), docker-compose, and unused CI workflows (docs-pages, pypi-publish, security, package-validation, project-automation, release). Slim CI to lint→typecheck→test only. Update CLAUDE.md, README, docs, mkdocs nav, and slash commands to reflect current lean monorepo layout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Accidentally dropped in the repo flatten. Matches the pattern used across all other workspace repos (datadex, agentdex, careerdex, dex-studio, infradex). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* merge: dev to main (lint fixes + merge main) (#142)
* feat: add v0.2.0 Production Hardening features (#56)
- Add Prometheus alert rules with SLO-aligned metrics
- Add Alertmanager configuration for alert routing
- Add prometheus.yml for monitoring setup
- Update observability documentation with alert details
- Add release notes and release PR template
- Bump version to 0.2.0
Includes:
- Environment-aware alerting rules
- Multi-environment deployment guidance
- Release workflow documentation
- Complete observability stack documentation
* fix: add missing Readme.md and src/ to Docker build context
The Dockerfile was copying only pyproject.toml and uv.lock, but
pyproject.toml references Readme.md, causing the build to fail with:
FileNotFoundError: Readme path `/build/Readme.md` does not exist
This fix includes:
- Copy Readme.md in the build stage
- Copy src/ directory to build context
Docker build now succeeds.
* refactor: fix CD workflow and add semantic versioning (#59)
## Changes
### CD Workflow Improvements
- **Simplified cd.yml**: Removed problematic manifest update jobs that were causing token authentication errors
- Removed persist-credentials configuration that was interfering with GitHub token
- Kept build-and-push and security-scan jobs which work reliably
- Clear and focused workflow for core CD tasks
### Semantic Release Setup
- **Added release.yml**: New workflow for automatic semantic versioning
- Uses python-semantic-release for intelligent version bumping
- Automatically creates git tags and GitHub releases
- Triggered on main branch commits
- Follows semantic versioning: major.minor.patch based on commit messages
### Configuration
- **Added [tool.semantic_release] section to pyproject.toml**
- Configures version detection from pyproject.toml
- Sets changelog file to docs/RELEASE_NOTES.md
- Defines branch-specific prerelease rules (main=stable, dev=prerelease)
### Cleanup
- **Removed sonar-scan.yml.disabled**: Cleaned up disabled SonarQube workflow
- No longer needed, reducing clutter in workflow definitions
## Why These Changes
1. **CD workflow was failing**: Permission and token issues with create-pull-request action made manifest updates unreliable. Simplified to focus on proven reliable tasks.
2. **Manual tagging is error-prone**: Semantic-release automates version detection, tagging, and release creation based on conventional commit messages.
3. **Better developer experience**: Removing disabled workflows reduces confusion about what's actually being used.
## Next Steps
- Merge to dev branch first for testing
- Verify CI/CD pipeline still works
- Merge to main for production use
* feat: v0.3.0 production hardening (#76)
- Ruff lint: 222 → 0 errors
- mypy strict: 168 → 0 errors
- Test coverage: 62% → 94% (122 tests)
- Pre-commit hooks (ruff, mypy, standard hooks)
- Dockerfile hardening (non-root user, healthcheck)
- CI: mypy step, fetch-based Slack notifications
- Poe tasks: typecheck, security, pre-commit, check-all
- Integration tests: API contracts, middleware, error handling
- Docs: ARCHITECTURE.md, DEVELOPMENT.md updated for v0.3.0
- Security: pip-audit 0 vulnerabilities
* fix: v0.3.1 Infrastructure Alignment (#78)
* fix: infrastructure alignment and workflow fixes
- Replace axios with fetch() in cd.yml, release.yml, security.yml
(actions/github-script@v7 does not bundle axios)
- Add dev branch to security.yml triggers
- Fix duplicate syncPolicy in ArgoCD PR preview ApplicationSet
- Add AlertManager service to docker-compose
- Add Grafana auto-provisioning (datasources + dashboards)
- Add healthcheck to API service in docker-compose
- Mount Prometheus alert rules volume
- Fix test-cov to cover all packages (dataenginex, careerdex, weatherdex)
- Add coverage.xml output for SonarCloud integration
- Remove unused black dependency (ruff handles formatting)
- Add coverage and version badges to README
- Fix Grafana README import path typo
- Update roadmap CSV to reflect v0.2.0 completion
* feat: enrich Slack failure notifications with detailed context
All failure notifications now include structured Slack blocks with:
- Workflow name, branch, commit SHA (linked)
- Triggered-by actor, event type
- Direct link to failed run
- Contextual hint about what likely failed
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* feat: v0.4.0 — DEX Module Epics & Project-Wide Improvements (#79)
feat: v0.3.2 — DEX Module Epics & Project-Wide Improvements (#79)\n\nImplements all 6 DEX module epics (#37-#42) plus 9 project-wide improvements.\n77 files changed, 4640 insertions, 143 deletions.\n233 tests passing, 0 lint/mypy errors.\n\nCloses #37, closes #38, closes #39, closes #40, closes #41, closes #42
* fix(v0.3.3): CD & Security Slack notification conditions
Bug: notify-deployment and notify-security jobs used success()/failure()
at step level which checks current job steps — not upstream dependent
jobs. This meant success notifications always fired and failure
notifications never fired.
Fix: Replace with explicit needs.*.result checks so notifications
correctly reflect the actual build/scan outcomes.
Changes:
- cd.yml: notify-deployment now checks needs.build-and-push.result
and needs.security-scan.result; enriched success message with
branch, run link, and context footer; failure message now lists
which specific jobs failed
- security.yml: notify-security now checks needs.semgrep.result
and needs.codeql.result; enriched with per-tool status fields
- Bump version 0.3.2 → 0.3.3 (pyproject.toml, sonar-project.properties)
* Potential fix for code scanning alert no. 132: Information exposure through an exception
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix: remove unused exception variable in auth middleware (F841)
* refactor: single version source in pyproject.toml, fix release.yml grep, poe install uses uv
* fix: copy LICENSE into Docker build context for Poetry Core
* chore: bump app and dataenginex package version to 0.3.4
* fix: apply Ruff import-order fixes for CI lint
* chore: enforce import-order check in local and CI gates
* fix: trigger pypi publish on release published events
* fix: skip trusted publish on manual workflow dispatch
* fix: use vars-based environments for pypi trusted publishing
* chore: bump app and dataenginex package version to 0.3.5
* chore: audit & governance setup + release automation matrix + health check fix (#99)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* fix: cd.yml - correct Trivy image reference construction (#100)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* fix: always push both latest and sha-<short_sha> tags for Docker image (#101)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* chore: add image pull/list step before Trivy scan for GHCR verification (#102)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
* chore: add image pull/list step before Trivy scan for GHCR verification
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* chore: bump actions/download-artifact to v7 and aquasecurity/trivy-action to 0.34.1 (#103)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
* chore: add image pull/list step before Trivy scan for GHCR verification
* fix: update image tag format and upgrade download-artifact action version
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* Update image reference for Trivy vulnerability scanner
* Simplify Trivy image reference in workflow
Updated Trivy image reference to use only the image tag output.
* Change IMAGE_NAME to 'thedataenginex/dex'
Updated IMAGE_NAME to comply with Docker image reference spec.
* Update IMAGE_NAME to use repository owner dynamically
* Refactor Docker image tagging in CD workflow
* Fix repository owner extraction in CD workflow
Updated the extraction of the repository owner for Docker compliance.
* fix: compute lowercase image name for GitHub Container Registry
* fix: add login step for GitHub Container Registry in CD workflow
* fix: improve Slack notification error handling and ensure webhook is set
* fix: harden CI/CD security and deployment workflows (#109)
* CD Workflow fix (#105)
fix: compute lowercase image name for GitHub Container Registry
* CD Bug Fix (#106)
fix: CD Bug Fix
* fix: harden workflows and add poe actionlint task
- add poe task for actionlint\n- upgrade actions versions and validate workflows\n- improve CD semver/GitOps verification/rollback flow\n- add preview deploy workflow\n- remove deprecated release workflow
* fix: allow preview workflow to comment on PRs
Grant pull-requests/issues write permissions for github-script comment step.
* chore: bump version to 0.3.6 and update dependencies
* fix: upgrade semgrep to v2 and add nosemgrep for safe workflow_run checkouts
* fix: revert semgrep-action to v1 (v2 does not exist)
* refactor: replace semgrep action with trivy repo security scan
* fix: make Trivy SARIF report non-blocking and gate on misconfig findings
* fix: harden k8s deployment security contexts for Trivy gate
* fix: add base deployment securityContext for Trivy misconfig policy
* fix: harden CI/CD workflows and add poe actionlint task (#107)
* feat: add v0.2.0 Production Hardening features (#56)
- Add Prometheus alert rules with SLO-aligned metrics
- Add Alertmanager configuration for alert routing
- Add prometheus.yml for monitoring setup
- Update observability documentation with alert details
- Add release notes and release PR template
- Bump version to 0.2.0
Includes:
- Environment-aware alerting rules
- Multi-environment deployment guidance
- Release workflow documentation
- Complete observability stack documentation
* fix: add missing Readme.md and src/ to Docker build context
The Dockerfile was copying only pyproject.toml and uv.lock, but
pyproject.toml references Readme.md, causing the build to fail with:
FileNotFoundError: Readme path `/build/Readme.md` does not exist
This fix includes:
- Copy Readme.md in the build stage
- Copy src/ directory to build context
Docker build now succeeds.
* refactor: fix CD workflow and add semantic versioning (#59)
## Changes
### CD Workflow Improvements
- **Simplified cd.yml**: Removed problematic manifest update jobs that were causing token authentication errors
- Removed persist-credentials configuration that was interfering with GitHub token
- Kept build-and-push and security-scan jobs which work reliably
- Clear and focused workflow for core CD tasks
### Semantic Release Setup
- **Added release.yml**: New workflow for automatic semantic versioning
- Uses python-semantic-release for intelligent version bumping
- Automatically creates git tags and GitHub releases
- Triggered on main branch commits
- Follows semantic versioning: major.minor.patch based on commit messages
### Configuration
- **Added [tool.semantic_release] section to pyproject.toml**
- Configures version detection from pyproject.toml
- Sets changelog file to docs/RELEASE_NOTES.md
- Defines branch-specific prerelease rules (main=stable, dev=prerelease)
### Cleanup
- **Removed sonar-scan.yml.disabled**: Cleaned up disabled SonarQube workflow
- No longer needed, reducing clutter in workflow definitions
## Why These Changes
1. **CD workflow was failing**: Permission and token issues with create-pull-request action made manifest updates unreliable. Simplified to focus on proven reliable tasks.
2. **Manual tagging is error-prone**: Semantic-release automates version detection, tagging, and release creation based on conventional commit messages.
3. **Better developer experience**: Removing disabled workflows reduces confusion about what's actually being used.
## Next Steps
- Merge to dev branch first for testing
- Verify CI/CD pipeline still works
- Merge to main for production use
* feat: v0.3.0 production hardening (#76)
- Ruff lint: 222 → 0 errors
- mypy strict: 168 → 0 errors
- Test coverage: 62% → 94% (122 tests)
- Pre-commit hooks (ruff, mypy, standard hooks)
- Dockerfile hardening (non-root user, healthcheck)
- CI: mypy step, fetch-based Slack notifications
- Poe tasks: typecheck, security, pre-commit, check-all
- Integration tests: API contracts, middleware, error handling
- Docs: ARCHITECTURE.md, DEVELOPMENT.md updated for v0.3.0
- Security: pip-audit 0 vulnerabilities
* fix: v0.3.1 Infrastructure Alignment (#78)
* fix: infrastructure alignment and workflow fixes
- Replace axios with fetch() in cd.yml, release.yml, security.yml
(actions/github-script@v7 does not bundle axios)
- Add dev branch to security.yml triggers
- Fix duplicate syncPolicy in ArgoCD PR preview ApplicationSet
- Add AlertManager service to docker-compose
- Add Grafana auto-provisioning (datasources + dashboards)
- Add healthcheck to API service in docker-compose
- Mount Prometheus alert rules volume
- Fix test-cov to cover all packages (dataenginex, careerdex, weatherdex)
- Add coverage.xml output for SonarCloud integration
- Remove unused black dependency (ruff handles formatting)
- Add coverage and version badges to README
- Fix Grafana README import path typo
- Update roadmap CSV to reflect v0.2.0 completion
* feat: enrich Slack failure notifications with detailed context
All failure notifications now include structured Slack blocks with:
- Workflow name, branch, commit SHA (linked)
- Triggered-by actor, event type
- Direct link to failed run
- Contextual hint about what likely failed
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* feat: v0.4.0 — DEX Module Epics & Project-Wide Improvements (#79)
feat: v0.3.2 — DEX Module Epics & Project-Wide Improvements (#79)\n\nImplements all 6 DEX module epics (#37-#42) plus 9 project-wide improvements.\n77 files changed, 4640 insertions, 143 deletions.\n233 tests passing, 0 lint/mypy errors.\n\nCloses #37, closes #38, closes #39, closes #40, closes #41, closes #42
* fix(v0.3.3): CD & Security Slack notification conditions
Bug: notify-deployment and notify-security jobs used success()/failure()
at step level which checks current job steps — not upstream dependent
jobs. This meant success notifications always fired and failure
notifications never fired.
Fix: Replace with explicit needs.*.result checks so notifications
correctly reflect the actual build/scan outcomes.
Changes:
- cd.yml: notify-deployment now checks needs.build-and-push.result
and needs.security-scan.result; enriched success message with
branch, run link, and context footer; failure message now lists
which specific jobs failed
- security.yml: notify-security now checks needs.semgrep.result
and needs.codeql.result; enriched with per-tool status fields
- Bump version 0.3.2 → 0.3.3 (pyproject.toml, sonar-project.properties)
* Potential fix for code scanning alert no. 132: Information exposure through an exception
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix: remove unused exception variable in auth middleware (F841)
* refactor: single version source in pyproject.toml, fix release.yml grep, poe install uses uv
* fix: copy LICENSE into Docker build context for Poetry Core
* chore: bump app and dataenginex package version to 0.3.4
* fix: apply Ruff import-order fixes for CI lint
* chore: enforce import-order check in local and CI gates
* fix: trigger pypi publish on release published events
* fix: skip trusted publish on manual workflow dispatch
* fix: use vars-based environments for pypi trusted publishing
* chore: bump app and dataenginex package version to 0.3.5
* chore: audit & governance setup + release automation matrix + health check fix (#99)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* fix: cd.yml - correct Trivy image reference construction (#100)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* fix: always push both latest and sha-<short_sha> tags for Docker image (#101)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* chore: add image pull/list step before Trivy scan for GHCR verification (#102)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
* chore: add image pull/list step before Trivy scan for GHCR verification
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* chore: bump actions/download-artifact to v7 and aquasecurity/trivy-action to 0.34.1 (#103)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
* chore: add image pull/list step before Trivy scan for GHCR verification
* fix: update image tag format and upgrade download-artifact action version
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* Update image reference for Trivy vulnerability scanner
* Simplify Trivy image reference in workflow
Updated Trivy image reference to use only the image tag output.
* Change IMAGE_NAME to 'thedataenginex/dex'
Updated IMAGE_NAME to comply with Docker image reference spec.
* Update IMAGE_NAME to use repository owner dynamically
* Refactor Docker image tagging in CD workflow
* Fix repository owner extraction in CD workflow
Updated the extraction of the repository owner for Docker compliance.
* fix: compute lowercase image name for GitHub Container Registry
* fix: add login step for GitHub Container Registry in CD workflow
* fix: improve Slack notification error handling and ensure webhook is set
* fix: harden workflows and add poe actionlint task
- add poe task for actionlint\n- upgrade actions versions and validate workflows\n- improve CD semver/GitOps verification/rollback flow\n- add preview deploy workflow\n- remove deprecated release workflow
* fix: allow preview workflow to comment on PRs
Grant pull-requests/issues write permissions for github-script comment step.
* chore: bump version to 0.3.6 and update dependencies
* fix: upgrade semgrep to v2 and add nosemgrep for safe workflow_run checkouts
* fix: revert semgrep-action to v1 (v2 does not exist)
* refactor: replace semgrep action with trivy repo security scan
* fix: make Trivy SARIF report non-blocking and gate on misconfig findings
* fix: harden k8s deployment security contexts for Trivy gate
* fix: add base deployment securityContext for Trivy misconfig policy
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix: fallback to PR when GitOps push hits protected branch
* fix: ensure GitHub Pages configuration is enabled in workflow
* fix: harden CD dependency gating and branch-safe GitOps fallback
* fix: harden CD gating and protected-branch manifest flow
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* refactor: simplify to 2-env branch-based deployment (dev/prod)
* CD Workflow fix (#105)
fix: compute lowercase image name for GitHub Container Registry
* CD Bug Fix (#106)
fix: CD Bug Fix
* fix: harden workflows and add poe actionlint task
- add poe task for actionlint\n- upgrade actions versions and validate workflows\n- improve CD semver/GitOps verification/rollback flow\n- add preview deploy workflow\n- remove deprecated release workflow
* fix: allow preview workflow to comment on PRs
Grant pull-requests/issues write permissions for github-script comment step.
* chore: bump version to 0.3.6 and update dependencies
* fix: upgrade semgrep to v2 and add nosemgrep for safe workflow_run checkouts
* fix: revert semgrep-action to v1 (v2 does not exist)
* refactor: replace semgrep action with trivy repo security scan
* fix: make Trivy SARIF report non-blocking and gate on misconfig findings
* fix: harden k8s deployment security contexts for Trivy gate
* fix: add base deployment securityContext for Trivy misconfig policy
* fix: harden CI/CD workflows and add poe actionlint task (#107)
* feat: add v0.2.0 Production Hardening features (#56)
- Add Prometheus alert rules with SLO-aligned metrics
- Add Alertmanager configuration for alert routing
- Add prometheus.yml for monitoring setup
- Update observability documentation with alert details
- Add release notes and release PR template
- Bump version to 0.2.0
Includes:
- Environment-aware alerting rules
- Multi-environment deployment guidance
- Release workflow documentation
- Complete observability stack documentation
* fix: add missing Readme.md and src/ to Docker build context
The Dockerfile was copying only pyproject.toml and uv.lock, but
pyproject.toml references Readme.md, causing the build to fail with:
FileNotFoundError: Readme path `/build/Readme.md` does not exist
This fix includes:
- Copy Readme.md in the build stage
- Copy src/ directory to build context
Docker build now succeeds.
* refactor: fix CD workflow and add semantic versioning (#59)
## Changes
### CD Workflow Improvements
- **Simplified cd.yml**: Removed problematic manifest update jobs that were causing token authentication errors
- Removed persist-credentials configuration that was interfering with GitHub token
- Kept build-and-push and security-scan jobs which work reliably
- Clear and focused workflow for core CD tasks
### Semantic Release Setup
- **Added release.yml**: New workflow for automatic semantic versioning
- Uses python-semantic-release for intelligent version bumping
- Automatically creates git tags and GitHub releases
- Triggered on main branch commits
- Follows semantic versioning: major.minor.patch based on commit messages
### Configuration
- **Added [tool.semantic_release] section to pyproject.toml**
- Configures version detection from pyproject.toml
- Sets changelog file to docs/RELEASE_NOTES.md
- Defines branch-specific prerelease rules (main=stable, dev=prerelease)
### Cleanup
- **Removed sonar-scan.yml.disabled**: Cleaned up disabled SonarQube workflow
- No longer needed, reducing clutter in workflow definitions
## Why These Changes
1. **CD workflow was failing**: Permission and token issues with create-pull-request action made manifest updates unreliable. Simplified to focus on proven reliable tasks.
2. **Manual tagging is error-prone**: Semantic-release automates version detection, tagging, and release creation based on conventional commit messages.
3. **Better developer experience**: Removing disabled workflows reduces confusion about what's actually being used.
## Next Steps
- Merge to dev branch first for testing
- Verify CI/CD pipeline still works
- Merge to main for production use
* feat: v0.3.0 production hardening (#76)
- Ruff lint: 222 → 0 errors
- mypy strict: 168 → 0 errors
- Test coverage: 62% → 94% (122 tests)
- Pre-commit hooks (ruff, mypy, standard hooks)
- Dockerfile hardening (non-root user, healthcheck)
- CI: mypy step, fetch-based Slack notifications
- Poe tasks: typecheck, security, pre-commit, check-all
- Integration tests: API contracts, middleware, error handling
- Docs: ARCHITECTURE.md, DEVELOPMENT.md updated for v0.3.0
- Security: pip-audit 0 vulnerabilities
* fix: v0.3.1 Infrastructure Alignment (#78)
* fix: infrastructure alignment and workflow fixes
- Replace axios with fetch() in cd.yml, release.yml, security.yml
(actions/github-script@v7 does not bundle axios)
- Add dev branch to security.yml triggers
- Fix duplicate syncPolicy in ArgoCD PR preview ApplicationSet
- Add AlertManager service to docker-compose
- Add Grafana auto-provisioning (datasources + dashboards)
- Add healthcheck to API service in docker-compose
- Mount Prometheus alert rules volume
- Fix test-cov to cover all packages (dataenginex, careerdex, weatherdex)
- Add coverage.xml output for SonarCloud integration
- Remove unused black dependency (ruff handles formatting)
- Add coverage and version badges to README
- Fix Grafana README import path typo
- Update roadmap CSV to reflect v0.2.0 completion
* feat: enrich Slack failure notifications with detailed context
All failure notifications now include structured Slack blocks with:
- Workflow name, branch, commit SHA (linked)
- Triggered-by actor, event type
- Direct link to failed run
- Contextual hint about what likely failed
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* feat: v0.4.0 — DEX Module Epics & Project-Wide Improvements (#79)
feat: v0.3.2 — DEX Module Epics & Project-Wide Improvements (#79)\n\nImplements all 6 DEX module epics (#37-#42) plus 9 project-wide improvements.\n77 files changed, 4640 insertions, 143 deletions.\n233 tests passing, 0 lint/mypy errors.\n\nCloses #37, closes #38, closes #39, closes #40, closes #41, closes #42
* fix(v0.3.3): CD & Security Slack notification conditions
Bug: notify-deployment and notify-security jobs used success()/failure()
at step level which checks current job steps — not upstream dependent
jobs. This meant success notifications always fired and failure
notifications never fired.
Fix: Replace with explicit needs.*.result checks so notifications
correctly reflect the actual build/scan outcomes.
Changes:
- cd.yml: notify-deployment now checks needs.build-and-push.result
and needs.security-scan.result; enriched success message with
branch, run link, and context footer; failure message now lists
which specific jobs failed
- security.yml: notify-security now checks needs.semgrep.result
and needs.codeql.result; enriched with per-tool status fields
- Bump version 0.3.2 → 0.3.3 (pyproject.toml, sonar-project.properties)
* Potential fix for code scanning alert no. 132: Information exposure through an exception
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix: remove unused exception variable in auth middleware (F841)
* refactor: single version source in pyproject.toml, fix release.yml grep, poe install uses uv
* fix: copy LICENSE into Docker build context for Poetry Core
* chore: bump app and dataenginex package version to 0.3.4
* fix: apply Ruff import-order fixes for CI lint
* chore: enforce import-order check in local and CI gates
* fix: trigger pypi publish on release published events
* fix: skip trusted publish on manual workflow dispatch
* fix: use vars-based environments for pypi trusted publishing
* chore: bump app and dataenginex package version to 0.3.5
* chore: audit & governance setup + release automation matrix + health check fix (#99)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* fix: cd.yml - correct Trivy image reference construction (#100)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* fix: always push both latest and sha-<short_sha> tags for Docker image (#101)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* chore: add image pull/list step before Trivy scan for GHCR verification (#102)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
* chore: add image pull/list step before Trivy scan for GHCR verification
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* chore: bump actions/download-artifact to v7 and aquasecurity/trivy-action to 0.34.1 (#103)
* feat: clean public API surface & stabilize exports (v0.4.0) (#86)
- Define __all__ in every subpackage __init__.py
- Add from __future__ import annotations to all __init__.py
- Export all public classes in organized __all__ lists
- Add comprehensive module docstrings with usage examples
- Bump version to 0.4.0
- Mark clear public API boundaries per submodule
Closes #86
* feat: CHANGELOG.md & release notes automation (v0.4.1) (#87)
* docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88)
* feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89)
- Refactored StorageBackend to proper ABC with @abstractmethod
- Added S3Storage backend (requires boto3)
- Added GCSStorage backend (requires google-cloud-storage)
- Re-exported StorageBackend from dataenginex.lakehouse
- Added optional dependency extras: s3, gcs, cloud, parquet, all
- Added mkdocs-material and mkdocstrings[python] to dev deps
- Updated root dataenginex dep constraint to >=0.4.0
* feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33)
* test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90)
* docs: examples & quickstart scripts (v0.4.10) (#91)
* chore: audit & governance setup + release automation matrix + health check fix
* feat: add comprehensive README with quick start, architecture, and documentation links
* fix: cd.yml - correct Trivy image reference construction
- Extract full image tag (registry/image:tag) in vars step
- Add build step ID to capture digest output
- Update Trivy to use image_tag from build-and-push outputs
- Fixes: 'failed to parse the image name' error in security scan
* fix: always push both latest and sha-<short_sha> tags for Docker image
- Ensures Trivy scan matches pushed tag
- Improves reproducibility and traceability in CI/CD
* chore: add image pull/list step before Trivy scan for GHCR verification
* fix: update image tag format and upgrade download-artifact action version
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* Update image reference for Trivy vulnerability scanner
* Simplify Trivy image reference in workflow
Updated Trivy image reference to use only the image tag output.
* Change IMAGE_NAME to 'thedataenginex/dex'
Updated IMAGE_NAME to comply with Docker image reference spec.
* Update IMAGE_NAME to use repository owner dynamically
* Refactor Docker image tagging in CD workflow
* Fix repository owner extraction in CD workflow
Updated the extraction of the repository owner for Docker compliance.
* fix: compute lowercase image name for GitHub Container Registry
* fix: add login step for GitHub Container Registry in CD workflow
* fix: improve Slack notification error handling and ensure webhook is set
* fix: harden workflows and add poe actionlint task
- add poe task for actionlint\n- upgrade actions versions and validate workflows\n- improve CD semver/GitOps verification/rollback flow\n- add preview deploy workflow\n- remove deprecated release workflow
* fix: allow preview workflow to comment on PRs
Grant pull-requests/issues write permissions for github-script comment step.
* chore: bump version to 0.3.6 and update dependencies
* fix: upgrade semgrep to v2 and add nosemgrep for safe workflow_run checkouts
* fix: revert semgrep-action to v1 (v2 does not exist)
* refactor: replace semgrep action with trivy repo security scan
* fix: make Trivy SARIF report non-blocking and gate on misconfig findings
* fix: harden k8s deployment security contexts for Trivy gate
* fix: add base deployment securityContext for Trivy misconfig policy
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix: fallback to PR when GitOps push hits protected branch
* fix: ensure GitHub Pages configuration is enabled in workflow
* fix: harden CD dependency gating and branch-safe GitOps fallback
* fix: harden CD gating and protected-branch manifest flow
* refactor: simplify to 2-env branch-based deployment (dev/prod)
- Remove stage and preview overlays and workflows
- Simplify ArgoCD ApplicationSet to 2 generators (dev→dex-dev, main→dex)
- Update AppProject destinations to dex and dex-dev namespaces only
- Simplify CD workflow update-gitops-manifests for dev/main branches
- Update all docs (ARCHITECTURE, CI_CD, DEPLOY_RUNBOOK, LOCAL_K8S_SETUP)
- Simplify promote.sh, get-tags.sh, alerting rules
- Remove preview-deploy.yml workflow
Closes the branch-based deployment re-architecture.
Refs: #64
* fix: remove preview-deploy.yml (no preview overlay exists in 2-env model)
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* chore: update dev image to sha-77f6984b [skip ci]
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix: remove preview-deploy.yml and fix docs-pages enablement
- Delete preview-deploy.yml (2-env model has no preview environments)
- Remove enablement: true from configure-pages (requires admin perms)
- Pages must be enabled once via repo Settings > Pages > Source: GitHub Actions
* chore: update dev image to sha-eca077ab [skip ci] (#116)
Auto-merge: GitOps dev image update (sha-eca077ab)
* feat: clean public API surface & stabilize exports (#86) (#126)
* chore: update dev image to sha-de539bec [skip ci] (#127)
Auto-merge: GitOps dev image update (sha-de539bec)
* feat: P1-high issues #87 #88 #89 #92 #93 — storage, ML serving, drift scheduler, docstrings (#128)
* feat: add PyPI badge and update CHANGELOG (#87)
* feat: storage abstraction — add list_objects, exists, get_storage factory (#89)
- Add list_objects(prefix) and exists(path) abstract methods to StorageBackend ABC
- Implement in all 6 backends: LocalParquetStorage, BigQueryStorage, JsonStorage,
ParquetStorage, S3Storage, GCSStorage
- Add get_storage(uri) factory: file:// → JsonStorage, s3:// → S3Storage, gs:// → GCSStorage
- Export get_storage from lakehouse package
- Add py.typed and update StorageBackend docstring
- 12 new unit tests (264 total passed)
* feat: generic model serving API endpoints (#92)
- Add ML router with POST /predict, GET /models, GET /models/{name}
- Add ML-specific Prometheus metrics (latency, throughput, drift)
- Add PredictRequestBody/PredictResponseBody Pydantic models
- Add set_model_server/get_model_server configuration helpers
- Add 9 unit tests for ML router endpoints
* test: storage integration tests with Docker emulators
- Add endpoint_url param to S3Storage for LocalStack support
- Add api_endpoint param to GCSStorage for fake-gcs-server support
- Add docker-compose.test.yml (LocalStack:4566 + fake-gcs-server:4443)
- Add LocalStack init script for test bucket creation
- Add Terraform module for real cloud test buckets (AWS S3 + GCS)
- Add 25 integration tests with emulator auto-detection
- Fix unit tests to pass endpoint kwargs preventing auth failures
* feat: drift monitoring scheduler with alerting (#93)
- Add DriftScheduler with background thread for periodic checks
- Add DriftMonitorConfig / DriftCheckResult dataclasses
- Publish PSI scores to model_drift_psi Prometheus gauge
- Increment model_drift_alerts_total counter on drift detection
- Add Prometheus alert rules (moderate, severe, spike, stale)
- Add mypy overrides for optional cloud SDKs (boto3, google)
- Add 18 unit tests for scheduler lifecycle and metrics
* docs: add docstrings to all public APIs (#88)
- Add Google-style docstrings to 55+ methods and 9 Pydantic models
- Cover core/schemas, api/, data/, lakehouse/, ml/, warehouse/
- Fix storage.py ParquetStorage read/delete try-block structure
- All 40 source files pass mypy strict, 299 tests pass
* chore: bump dataenginex to 0.5.0, fix docs workflow & Dockerfile
- Bump dataenginex version from 0.4.11 to 0.5.0
- Fix docs-pages.yml: use uv sync --frozen instead of uv lock + uv sync
- Fix Dockerfile: add missing COPY --from=builder /build/packages /app/packages
- Update CHANGELOG with [0.5.0] - 2026-03-01 release
- Update RELEASE_NOTES, CI_CD docs, roadmap CSV
- Clean up mypy type-ignore annotations for cloud SDK imports
* fix: resolve CI mypy unused-ignore and Trivy S3 encryption (AWS-0132)
- Add per-module mypy override for storage module (warn_unused_ignores=false)
handles env asymmetry: pre-commit has google-cloud-storage, CI does not
- Keep # type: ignore[no-untyped-call] on AnonymousCredentials for pre-commit
- Switch S3 test bucket from AES256 (SSE-S3) to SSE-KMS with CMK
(aws_kms_key with rotation + bucket_key_enabled)
Fixes CI workflow (mypy) and Security workflow (Trivy AWS-0132)
* chore: add system setup script, consolidate READMEs to GUIDE.md
- Add scripts/setup-system.sh: one-command system deps installer
(Ubuntu/Debian, Fedora/RHEL, Arch, macOS)
- Add poe task 'setup-system' for convenience
- Update docs/DEVELOPMENT.md with system dependencies table
- Update README.md Quick Start with setup-system.sh step
- Rename 7 subdirectory README.md files to GUIDE.md
(examples, scripts, packages, learning, src/careerdex,
infra/monitoring/grafana, infra/terraform/test-storage)
- Fix 3 cross-references to renamed files
- Keep only root README.md and packages/dataenginex README.md
---------
Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com>
* chore: update dev image to sha-0b003e0e [skip ci] (#129)
Auto-merge: GitOps dev image update (sha-0b003e0e)
* chore: update dev image to sha-b2ebaa36 [skip ci] (#131)
Auto-merge: GitOps dev image update (sha-b2ebaa36)
* feat: CareerDEX v0.5.0 — domain extraction, plugin system, ML modules, docs overhaul
feat: CareerDEX v0.5.0 — domain extraction, plugin system, ML modules, and docs overhaul (#138)
* fix: boost test coverage to 83%+ (feature → dev) (#140)
* feat: mount ml_router in CareerDEX and add OpenAPI tags (#68)
- Import and include ml_router from dataenginex.api.routers.ml
- Add v1 and ml tags to custom OpenAPI schema
- Enables ML model serving endpoints in CareerDEX API
* fix: add pre-commit to dev dependencies and re-sync venv
- pre-commit was used by git hooks but not listed as dependency
- venv had stale shebangs from old directory path (DEX → dex)
- Re-synced venv fixes poe and pre-commit resolution
* refactor: extract domain code from dataenginex to careerdex
- Move routers (ml.py, v1.py) from dataenginex to careerdex/api/routers/
- Move pipeline_config.py from dataenginex to careerdex/core/
- Add careerdex-specific schemas (JobPosting, UserProfile, JobSourceEnum)
- Add careerdex validators (SchemaValidator, QualityScorer, DataHash)
- Add careerdex settings, exceptions, and config modules
- Clean up dataenginex framework: remove domain-specific code
- Update all phase imports to use careerdex.core instead of dataenginex
- dataenginex now contains only generic framework code
* feat: add vectorstore, LLM adapter, plugin system, and dashboard
- RAG vectorstore: VectorStoreBackend ABC, InMemoryBackend, ChromaDBBackend
- LLM adapter: LLMProvider ABC, OllamaProvider (REST), MockProvider
- Plugin system: DataEngineXPlugin ABC, PluginRegistry, discover() via entry_points
- Dashboard: DashboardConfig, BaseDashboard, 4 Streamlit panels
- All modules exported from dataenginex.ml, dataenginex.plugins, dataenginex.dashboard
* feat: add local file connectors, embeddings wiring, and CareerDEX plugin
- JsonFileConnector and CsvFileConnector for local job data ingestion
- JobSourceEnum extended with LOCAL_JSON and LOCAL_CSV
- Phase 3 embeddings: sentence-transformers integration with fallback
- CareerDEX plugin: health_check, get_metrics, register_routes
- Plugin auto-discovered via entry_points group 'dataenginex.plugins'
* test: add tests for domain extraction, plugins, connectors, and ML modules
- 441 tests passing, 28 skipped (optional deps)
- New test suites: plugins, vectorstore, LLM, careerdex phases
- Updated existing tests for domain extraction import changes
- Load testing with Locust for API performance benchmarking
- Test fixtures: sample_jobs.json, sample_jobs.csv
* chore: coverage gate, dependency updates, and build config
- Add --cov-fail-under=80 to test-cov-core poe task
- Add sentence-transformers to ml dependency group
- Add streamlit to dashboard dependency group with uv conflict config
- Add dataenginex.plugins entry_points for CareerDEX plugin discovery
- Add mypy overrides for streamlit stubs
- Sync uv.lock with all dependency changes
* docs: update documentation, examples, and changelogs
- README: add plugin system and dashboard sections, update tech stack
- TODO: mark Streamlit dashboard as complete
- Architecture docs: reflect domain extraction changes
- New examples: RAG demo, LLM quickstart, dashboard runner
- CHANGELOG: plugin system, dashboard, coverage gate entries
- Add quickstart guide and security scanning docs
- Update tasks with findings and lessons learned
* chore: update CI/CD workflows, tooling, and project configs
- Update CI/CD workflows: ci.yml, cd.yml, security.yml, release pipelines
- Add Claude Code commands and settings
- Update pre-commit hooks and gitignore patterns
- Add gitleaks config for secret scanning
- Update Dockerfile for new dependency groups
- Update SonarQube and MCP server configs
- Update agent/instruction/prompt files for new modules
* chore: fix docs formatting, version refs, and add missing docs
- Add API reference docs: plugins.md, dashboard.md
- Update mkdocs nav with new module pages
- Add Bruno API collection for all 8 CareerDEX endpoints
- Update ADR medallion version reference
* fix: align example 04_ml_training.py with actual API signatures
- SklearnTrainer: use model_name/version/estimator constructor args
- ModelRegistry: use persist_path, register(ModelArtifact)
- DriftDetector: use check_dataset() instead of nonexistent detect()
- Add proper sklearn import and evaluate() call
* feat: boost test coverage to 83%+ via ML endpoint integration tests
- Initialize ModelServer and ModelRegistry in app lifespan for ML endpoint support
- Add comprehensive integration tests for ML endpoints (GET /models, GET /models/{name}, POST …
…railing newline (#152) * merge: dev to main (lint fixes + merge main) (#142) * feat: add v0.2.0 Production Hardening features (#56) - Add Prometheus alert rules with SLO-aligned metrics - Add Alertmanager configuration for alert routing - Add prometheus.yml for monitoring setup - Update observability documentation with alert details - Add release notes and release PR template - Bump version to 0.2.0 Includes: - Environment-aware alerting rules - Multi-environment deployment guidance - Release workflow documentation - Complete observability stack documentation * fix: add missing Readme.md and src/ to Docker build context The Dockerfile was copying only pyproject.toml and uv.lock, but pyproject.toml references Readme.md, causing the build to fail with: FileNotFoundError: Readme path `/build/Readme.md` does not exist This fix includes: - Copy Readme.md in the build stage - Copy src/ directory to build context Docker build now succeeds. * refactor: fix CD workflow and add semantic versioning (#59) ## Changes ### CD Workflow Improvements - **Simplified cd.yml**: Removed problematic manifest update jobs that were causing token authentication errors - Removed persist-credentials configuration that was interfering with GitHub token - Kept build-and-push and security-scan jobs which work reliably - Clear and focused workflow for core CD tasks ### Semantic Release Setup - **Added release.yml**: New workflow for automatic semantic versioning - Uses python-semantic-release for intelligent version bumping - Automatically creates git tags and GitHub releases - Triggered on main branch commits - Follows semantic versioning: major.minor.patch based on commit messages ### Configuration - **Added [tool.semantic_release] section to pyproject.toml** - Configures version detection from pyproject.toml - Sets changelog file to docs/RELEASE_NOTES.md - Defines branch-specific prerelease rules (main=stable, dev=prerelease) ### Cleanup - **Removed sonar-scan.yml.disabled**: Cleaned up disabled SonarQube workflow - No longer needed, reducing clutter in workflow definitions ## Why These Changes 1. **CD workflow was failing**: Permission and token issues with create-pull-request action made manifest updates unreliable. Simplified to focus on proven reliable tasks. 2. **Manual tagging is error-prone**: Semantic-release automates version detection, tagging, and release creation based on conventional commit messages. 3. **Better developer experience**: Removing disabled workflows reduces confusion about what's actually being used. ## Next Steps - Merge to dev branch first for testing - Verify CI/CD pipeline still works - Merge to main for production use * feat: v0.3.0 production hardening (#76) - Ruff lint: 222 → 0 errors - mypy strict: 168 → 0 errors - Test coverage: 62% → 94% (122 tests) - Pre-commit hooks (ruff, mypy, standard hooks) - Dockerfile hardening (non-root user, healthcheck) - CI: mypy step, fetch-based Slack notifications - Poe tasks: typecheck, security, pre-commit, check-all - Integration tests: API contracts, middleware, error handling - Docs: ARCHITECTURE.md, DEVELOPMENT.md updated for v0.3.0 - Security: pip-audit 0 vulnerabilities * fix: v0.3.1 Infrastructure Alignment (#78) * fix: infrastructure alignment and workflow fixes - Replace axios with fetch() in cd.yml, release.yml, security.yml (actions/github-script@v7 does not bundle axios) - Add dev branch to security.yml triggers - Fix duplicate syncPolicy in ArgoCD PR preview ApplicationSet - Add AlertManager service to docker-compose - Add Grafana auto-provisioning (datasources + dashboards) - Add healthcheck to API service in docker-compose - Mount Prometheus alert rules volume - Fix test-cov to cover all packages (dataenginex, careerdex, weatherdex) - Add coverage.xml output for SonarCloud integration - Remove unused black dependency (ruff handles formatting) - Add coverage and version badges to README - Fix Grafana README import path typo - Update roadmap CSV to reflect v0.2.0 completion * feat: enrich Slack failure notifications with detailed context All failure notifications now include structured Slack blocks with: - Workflow name, branch, commit SHA (linked) - Triggered-by actor, event type - Direct link to failed run - Contextual hint about what likely failed --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * feat: v0.4.0 — DEX Module Epics & Project-Wide Improvements (#79) feat: v0.3.2 — DEX Module Epics & Project-Wide Improvements (#79)\n\nImplements all 6 DEX module epics (#37-#42) plus 9 project-wide improvements.\n77 files changed, 4640 insertions, 143 deletions.\n233 tests passing, 0 lint/mypy errors.\n\nCloses #37, closes #38, closes #39, closes #40, closes #41, closes #42 * fix(v0.3.3): CD & Security Slack notification conditions Bug: notify-deployment and notify-security jobs used success()/failure() at step level which checks current job steps — not upstream dependent jobs. This meant success notifications always fired and failure notifications never fired. Fix: Replace with explicit needs.*.result checks so notifications correctly reflect the actual build/scan outcomes. Changes: - cd.yml: notify-deployment now checks needs.build-and-push.result and needs.security-scan.result; enriched success message with branch, run link, and context footer; failure message now lists which specific jobs failed - security.yml: notify-security now checks needs.semgrep.result and needs.codeql.result; enriched with per-tool status fields - Bump version 0.3.2 → 0.3.3 (pyproject.toml, sonar-project.properties) * Potential fix for code scanning alert no. 132: Information exposure through an exception Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: remove unused exception variable in auth middleware (F841) * refactor: single version source in pyproject.toml, fix release.yml grep, poe install uses uv * fix: copy LICENSE into Docker build context for Poetry Core * chore: bump app and dataenginex package version to 0.3.4 * fix: apply Ruff import-order fixes for CI lint * chore: enforce import-order check in local and CI gates * fix: trigger pypi publish on release published events * fix: skip trusted publish on manual workflow dispatch * fix: use vars-based environments for pypi trusted publishing * chore: bump app and dataenginex package version to 0.3.5 * chore: audit & governance setup + release automation matrix + health check fix (#99) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * fix: cd.yml - correct Trivy image reference construction (#100) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * fix: always push both latest and sha-<short_sha> tags for Docker image (#101) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * chore: add image pull/list step before Trivy scan for GHCR verification (#102) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD * chore: add image pull/list step before Trivy scan for GHCR verification --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * chore: bump actions/download-artifact to v7 and aquasecurity/trivy-action to 0.34.1 (#103) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD * chore: add image pull/list step before Trivy scan for GHCR verification * fix: update image tag format and upgrade download-artifact action version --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * Update image reference for Trivy vulnerability scanner * Simplify Trivy image reference in workflow Updated Trivy image reference to use only the image tag output. * Change IMAGE_NAME to 'thedataenginex/dex' Updated IMAGE_NAME to comply with Docker image reference spec. * Update IMAGE_NAME to use repository owner dynamically * Refactor Docker image tagging in CD workflow * Fix repository owner extraction in CD workflow Updated the extraction of the repository owner for Docker compliance. * fix: compute lowercase image name for GitHub Container Registry * fix: add login step for GitHub Container Registry in CD workflow * fix: improve Slack notification error handling and ensure webhook is set * fix: harden CI/CD security and deployment workflows (#109) * CD Workflow fix (#105) fix: compute lowercase image name for GitHub Container Registry * CD Bug Fix (#106) fix: CD Bug Fix * fix: harden workflows and add poe actionlint task - add poe task for actionlint\n- upgrade actions versions and validate workflows\n- improve CD semver/GitOps verification/rollback flow\n- add preview deploy workflow\n- remove deprecated release workflow * fix: allow preview workflow to comment on PRs Grant pull-requests/issues write permissions for github-script comment step. * chore: bump version to 0.3.6 and update dependencies * fix: upgrade semgrep to v2 and add nosemgrep for safe workflow_run checkouts * fix: revert semgrep-action to v1 (v2 does not exist) * refactor: replace semgrep action with trivy repo security scan * fix: make Trivy SARIF report non-blocking and gate on misconfig findings * fix: harden k8s deployment security contexts for Trivy gate * fix: add base deployment securityContext for Trivy misconfig policy * fix: harden CI/CD workflows and add poe actionlint task (#107) * feat: add v0.2.0 Production Hardening features (#56) - Add Prometheus alert rules with SLO-aligned metrics - Add Alertmanager configuration for alert routing - Add prometheus.yml for monitoring setup - Update observability documentation with alert details - Add release notes and release PR template - Bump version to 0.2.0 Includes: - Environment-aware alerting rules - Multi-environment deployment guidance - Release workflow documentation - Complete observability stack documentation * fix: add missing Readme.md and src/ to Docker build context The Dockerfile was copying only pyproject.toml and uv.lock, but pyproject.toml references Readme.md, causing the build to fail with: FileNotFoundError: Readme path `/build/Readme.md` does not exist This fix includes: - Copy Readme.md in the build stage - Copy src/ directory to build context Docker build now succeeds. * refactor: fix CD workflow and add semantic versioning (#59) ## Changes ### CD Workflow Improvements - **Simplified cd.yml**: Removed problematic manifest update jobs that were causing token authentication errors - Removed persist-credentials configuration that was interfering with GitHub token - Kept build-and-push and security-scan jobs which work reliably - Clear and focused workflow for core CD tasks ### Semantic Release Setup - **Added release.yml**: New workflow for automatic semantic versioning - Uses python-semantic-release for intelligent version bumping - Automatically creates git tags and GitHub releases - Triggered on main branch commits - Follows semantic versioning: major.minor.patch based on commit messages ### Configuration - **Added [tool.semantic_release] section to pyproject.toml** - Configures version detection from pyproject.toml - Sets changelog file to docs/RELEASE_NOTES.md - Defines branch-specific prerelease rules (main=stable, dev=prerelease) ### Cleanup - **Removed sonar-scan.yml.disabled**: Cleaned up disabled SonarQube workflow - No longer needed, reducing clutter in workflow definitions ## Why These Changes 1. **CD workflow was failing**: Permission and token issues with create-pull-request action made manifest updates unreliable. Simplified to focus on proven reliable tasks. 2. **Manual tagging is error-prone**: Semantic-release automates version detection, tagging, and release creation based on conventional commit messages. 3. **Better developer experience**: Removing disabled workflows reduces confusion about what's actually being used. ## Next Steps - Merge to dev branch first for testing - Verify CI/CD pipeline still works - Merge to main for production use * feat: v0.3.0 production hardening (#76) - Ruff lint: 222 → 0 errors - mypy strict: 168 → 0 errors - Test coverage: 62% → 94% (122 tests) - Pre-commit hooks (ruff, mypy, standard hooks) - Dockerfile hardening (non-root user, healthcheck) - CI: mypy step, fetch-based Slack notifications - Poe tasks: typecheck, security, pre-commit, check-all - Integration tests: API contracts, middleware, error handling - Docs: ARCHITECTURE.md, DEVELOPMENT.md updated for v0.3.0 - Security: pip-audit 0 vulnerabilities * fix: v0.3.1 Infrastructure Alignment (#78) * fix: infrastructure alignment and workflow fixes - Replace axios with fetch() in cd.yml, release.yml, security.yml (actions/github-script@v7 does not bundle axios) - Add dev branch to security.yml triggers - Fix duplicate syncPolicy in ArgoCD PR preview ApplicationSet - Add AlertManager service to docker-compose - Add Grafana auto-provisioning (datasources + dashboards) - Add healthcheck to API service in docker-compose - Mount Prometheus alert rules volume - Fix test-cov to cover all packages (dataenginex, careerdex, weatherdex) - Add coverage.xml output for SonarCloud integration - Remove unused black dependency (ruff handles formatting) - Add coverage and version badges to README - Fix Grafana README import path typo - Update roadmap CSV to reflect v0.2.0 completion * feat: enrich Slack failure notifications with detailed context All failure notifications now include structured Slack blocks with: - Workflow name, branch, commit SHA (linked) - Triggered-by actor, event type - Direct link to failed run - Contextual hint about what likely failed --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * feat: v0.4.0 — DEX Module Epics & Project-Wide Improvements (#79) feat: v0.3.2 — DEX Module Epics & Project-Wide Improvements (#79)\n\nImplements all 6 DEX module epics (#37-#42) plus 9 project-wide improvements.\n77 files changed, 4640 insertions, 143 deletions.\n233 tests passing, 0 lint/mypy errors.\n\nCloses #37, closes #38, closes #39, closes #40, closes #41, closes #42 * fix(v0.3.3): CD & Security Slack notification conditions Bug: notify-deployment and notify-security jobs used success()/failure() at step level which checks current job steps — not upstream dependent jobs. This meant success notifications always fired and failure notifications never fired. Fix: Replace with explicit needs.*.result checks so notifications correctly reflect the actual build/scan outcomes. Changes: - cd.yml: notify-deployment now checks needs.build-and-push.result and needs.security-scan.result; enriched success message with branch, run link, and context footer; failure message now lists which specific jobs failed - security.yml: notify-security now checks needs.semgrep.result and needs.codeql.result; enriched with per-tool status fields - Bump version 0.3.2 → 0.3.3 (pyproject.toml, sonar-project.properties) * Potential fix for code scanning alert no. 132: Information exposure through an exception Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: remove unused exception variable in auth middleware (F841) * refactor: single version source in pyproject.toml, fix release.yml grep, poe install uses uv * fix: copy LICENSE into Docker build context for Poetry Core * chore: bump app and dataenginex package version to 0.3.4 * fix: apply Ruff import-order fixes for CI lint * chore: enforce import-order check in local and CI gates * fix: trigger pypi publish on release published events * fix: skip trusted publish on manual workflow dispatch * fix: use vars-based environments for pypi trusted publishing * chore: bump app and dataenginex package version to 0.3.5 * chore: audit & governance setup + release automation matrix + health check fix (#99) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * fix: cd.yml - correct Trivy image reference construction (#100) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * fix: always push both latest and sha-<short_sha> tags for Docker image (#101) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * chore: add image pull/list step before Trivy scan for GHCR verification (#102) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD * chore: add image pull/list step before Trivy scan for GHCR verification --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * chore: bump actions/download-artifact to v7 and aquasecurity/trivy-action to 0.34.1 (#103) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD * chore: add image pull/list step before Trivy scan for GHCR verification * fix: update image tag format and upgrade download-artifact action version --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * Update image reference for Trivy vulnerability scanner * Simplify Trivy image reference in workflow Updated Trivy image reference to use only the image tag output. * Change IMAGE_NAME to 'thedataenginex/dex' Updated IMAGE_NAME to comply with Docker image reference spec. * Update IMAGE_NAME to use repository owner dynamically * Refactor Docker image tagging in CD workflow * Fix repository owner extraction in CD workflow Updated the extraction of the repository owner for Docker compliance. * fix: compute lowercase image name for GitHub Container Registry * fix: add login step for GitHub Container Registry in CD workflow * fix: improve Slack notification error handling and ensure webhook is set * fix: harden workflows and add poe actionlint task - add poe task for actionlint\n- upgrade actions versions and validate workflows\n- improve CD semver/GitOps verification/rollback flow\n- add preview deploy workflow\n- remove deprecated release workflow * fix: allow preview workflow to comment on PRs Grant pull-requests/issues write permissions for github-script comment step. * chore: bump version to 0.3.6 and update dependencies * fix: upgrade semgrep to v2 and add nosemgrep for safe workflow_run checkouts * fix: revert semgrep-action to v1 (v2 does not exist) * refactor: replace semgrep action with trivy repo security scan * fix: make Trivy SARIF report non-blocking and gate on misconfig findings * fix: harden k8s deployment security contexts for Trivy gate * fix: add base deployment securityContext for Trivy misconfig policy --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: fallback to PR when GitOps push hits protected branch * fix: ensure GitHub Pages configuration is enabled in workflow * fix: harden CD dependency gating and branch-safe GitOps fallback * fix: harden CD gating and protected-branch manifest flow --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * refactor: simplify to 2-env branch-based deployment (dev/prod) * CD Workflow fix (#105) fix: compute lowercase image name for GitHub Container Registry * CD Bug Fix (#106) fix: CD Bug Fix * fix: harden workflows and add poe actionlint task - add poe task for actionlint\n- upgrade actions versions and validate workflows\n- improve CD semver/GitOps verification/rollback flow\n- add preview deploy workflow\n- remove deprecated release workflow * fix: allow preview workflow to comment on PRs Grant pull-requests/issues write permissions for github-script comment step. * chore: bump version to 0.3.6 and update dependencies * fix: upgrade semgrep to v2 and add nosemgrep for safe workflow_run checkouts * fix: revert semgrep-action to v1 (v2 does not exist) * refactor: replace semgrep action with trivy repo security scan * fix: make Trivy SARIF report non-blocking and gate on misconfig findings * fix: harden k8s deployment security contexts for Trivy gate * fix: add base deployment securityContext for Trivy misconfig policy * fix: harden CI/CD workflows and add poe actionlint task (#107) * feat: add v0.2.0 Production Hardening features (#56) - Add Prometheus alert rules with SLO-aligned metrics - Add Alertmanager configuration for alert routing - Add prometheus.yml for monitoring setup - Update observability documentation with alert details - Add release notes and release PR template - Bump version to 0.2.0 Includes: - Environment-aware alerting rules - Multi-environment deployment guidance - Release workflow documentation - Complete observability stack documentation * fix: add missing Readme.md and src/ to Docker build context The Dockerfile was copying only pyproject.toml and uv.lock, but pyproject.toml references Readme.md, causing the build to fail with: FileNotFoundError: Readme path `/build/Readme.md` does not exist This fix includes: - Copy Readme.md in the build stage - Copy src/ directory to build context Docker build now succeeds. * refactor: fix CD workflow and add semantic versioning (#59) ## Changes ### CD Workflow Improvements - **Simplified cd.yml**: Removed problematic manifest update jobs that were causing token authentication errors - Removed persist-credentials configuration that was interfering with GitHub token - Kept build-and-push and security-scan jobs which work reliably - Clear and focused workflow for core CD tasks ### Semantic Release Setup - **Added release.yml**: New workflow for automatic semantic versioning - Uses python-semantic-release for intelligent version bumping - Automatically creates git tags and GitHub releases - Triggered on main branch commits - Follows semantic versioning: major.minor.patch based on commit messages ### Configuration - **Added [tool.semantic_release] section to pyproject.toml** - Configures version detection from pyproject.toml - Sets changelog file to docs/RELEASE_NOTES.md - Defines branch-specific prerelease rules (main=stable, dev=prerelease) ### Cleanup - **Removed sonar-scan.yml.disabled**: Cleaned up disabled SonarQube workflow - No longer needed, reducing clutter in workflow definitions ## Why These Changes 1. **CD workflow was failing**: Permission and token issues with create-pull-request action made manifest updates unreliable. Simplified to focus on proven reliable tasks. 2. **Manual tagging is error-prone**: Semantic-release automates version detection, tagging, and release creation based on conventional commit messages. 3. **Better developer experience**: Removing disabled workflows reduces confusion about what's actually being used. ## Next Steps - Merge to dev branch first for testing - Verify CI/CD pipeline still works - Merge to main for production use * feat: v0.3.0 production hardening (#76) - Ruff lint: 222 → 0 errors - mypy strict: 168 → 0 errors - Test coverage: 62% → 94% (122 tests) - Pre-commit hooks (ruff, mypy, standard hooks) - Dockerfile hardening (non-root user, healthcheck) - CI: mypy step, fetch-based Slack notifications - Poe tasks: typecheck, security, pre-commit, check-all - Integration tests: API contracts, middleware, error handling - Docs: ARCHITECTURE.md, DEVELOPMENT.md updated for v0.3.0 - Security: pip-audit 0 vulnerabilities * fix: v0.3.1 Infrastructure Alignment (#78) * fix: infrastructure alignment and workflow fixes - Replace axios with fetch() in cd.yml, release.yml, security.yml (actions/github-script@v7 does not bundle axios) - Add dev branch to security.yml triggers - Fix duplicate syncPolicy in ArgoCD PR preview ApplicationSet - Add AlertManager service to docker-compose - Add Grafana auto-provisioning (datasources + dashboards) - Add healthcheck to API service in docker-compose - Mount Prometheus alert rules volume - Fix test-cov to cover all packages (dataenginex, careerdex, weatherdex) - Add coverage.xml output for SonarCloud integration - Remove unused black dependency (ruff handles formatting) - Add coverage and version badges to README - Fix Grafana README import path typo - Update roadmap CSV to reflect v0.2.0 completion * feat: enrich Slack failure notifications with detailed context All failure notifications now include structured Slack blocks with: - Workflow name, branch, commit SHA (linked) - Triggered-by actor, event type - Direct link to failed run - Contextual hint about what likely failed --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * feat: v0.4.0 — DEX Module Epics & Project-Wide Improvements (#79) feat: v0.3.2 — DEX Module Epics & Project-Wide Improvements (#79)\n\nImplements all 6 DEX module epics (#37-#42) plus 9 project-wide improvements.\n77 files changed, 4640 insertions, 143 deletions.\n233 tests passing, 0 lint/mypy errors.\n\nCloses #37, closes #38, closes #39, closes #40, closes #41, closes #42 * fix(v0.3.3): CD & Security Slack notification conditions Bug: notify-deployment and notify-security jobs used success()/failure() at step level which checks current job steps — not upstream dependent jobs. This meant success notifications always fired and failure notifications never fired. Fix: Replace with explicit needs.*.result checks so notifications correctly reflect the actual build/scan outcomes. Changes: - cd.yml: notify-deployment now checks needs.build-and-push.result and needs.security-scan.result; enriched success message with branch, run link, and context footer; failure message now lists which specific jobs failed - security.yml: notify-security now checks needs.semgrep.result and needs.codeql.result; enriched with per-tool status fields - Bump version 0.3.2 → 0.3.3 (pyproject.toml, sonar-project.properties) * Potential fix for code scanning alert no. 132: Information exposure through an exception Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: remove unused exception variable in auth middleware (F841) * refactor: single version source in pyproject.toml, fix release.yml grep, poe install uses uv * fix: copy LICENSE into Docker build context for Poetry Core * chore: bump app and dataenginex package version to 0.3.4 * fix: apply Ruff import-order fixes for CI lint * chore: enforce import-order check in local and CI gates * fix: trigger pypi publish on release published events * fix: skip trusted publish on manual workflow dispatch * fix: use vars-based environments for pypi trusted publishing * chore: bump app and dataenginex package version to 0.3.5 * chore: audit & governance setup + release automation matrix + health check fix (#99) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * fix: cd.yml - correct Trivy image reference construction (#100) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * fix: always push both latest and sha-<short_sha> tags for Docker image (#101) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * chore: add image pull/list step before Trivy scan for GHCR verification (#102) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD * chore: add image pull/list step before Trivy scan for GHCR verification --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * chore: bump actions/download-artifact to v7 and aquasecurity/trivy-action to 0.34.1 (#103) * feat: clean public API surface & stabilize exports (v0.4.0) (#86) - Define __all__ in every subpackage __init__.py - Add from __future__ import annotations to all __init__.py - Export all public classes in organized __all__ lists - Add comprehensive module docstrings with usage examples - Bump version to 0.4.0 - Mark clear public API boundaries per submodule Closes #86 * feat: CHANGELOG.md & release notes automation (v0.4.1) (#87) * docs: comprehensive docstrings & mkdocs API reference (v0.4.3) (#88) * feat: storage abstraction layer with S3/GCS backends (v0.4.5) (#89) - Refactored StorageBackend to proper ABC with @abstractmethod - Added S3Storage backend (requires boto3) - Added GCSStorage backend (requires google-cloud-storage) - Re-exported StorageBackend from dataenginex.lakehouse - Added optional dependency extras: s3, gcs, cloud, parquet, all - Added mkdocs-material and mkdocstrings[python] to dev deps - Updated root dataenginex dep constraint to >=0.4.0 * feat: data quality framework with QualityGate & QualityStore (v0.4.6) (#33) * test: PySpark local-mode fixtures & sample data helpers (v0.4.8) (#90) * docs: examples & quickstart scripts (v0.4.10) (#91) * chore: audit & governance setup + release automation matrix + health check fix * feat: add comprehensive README with quick start, architecture, and documentation links * fix: cd.yml - correct Trivy image reference construction - Extract full image tag (registry/image:tag) in vars step - Add build step ID to capture digest output - Update Trivy to use image_tag from build-and-push outputs - Fixes: 'failed to parse the image name' error in security scan * fix: always push both latest and sha-<short_sha> tags for Docker image - Ensures Trivy scan matches pushed tag - Improves reproducibility and traceability in CI/CD * chore: add image pull/list step before Trivy scan for GHCR verification * fix: update image tag format and upgrade download-artifact action version --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * Update image reference for Trivy vulnerability scanner * Simplify Trivy image reference in workflow Updated Trivy image reference to use only the image tag output. * Change IMAGE_NAME to 'thedataenginex/dex' Updated IMAGE_NAME to comply with Docker image reference spec. * Update IMAGE_NAME to use repository owner dynamically * Refactor Docker image tagging in CD workflow * Fix repository owner extraction in CD workflow Updated the extraction of the repository owner for Docker compliance. * fix: compute lowercase image name for GitHub Container Registry * fix: add login step for GitHub Container Registry in CD workflow * fix: improve Slack notification error handling and ensure webhook is set * fix: harden workflows and add poe actionlint task - add poe task for actionlint\n- upgrade actions versions and validate workflows\n- improve CD semver/GitOps verification/rollback flow\n- add preview deploy workflow\n- remove deprecated release workflow * fix: allow preview workflow to comment on PRs Grant pull-requests/issues write permissions for github-script comment step. * chore: bump version to 0.3.6 and update dependencies * fix: upgrade semgrep to v2 and add nosemgrep for safe workflow_run checkouts * fix: revert semgrep-action to v1 (v2 does not exist) * refactor: replace semgrep action with trivy repo security scan * fix: make Trivy SARIF report non-blocking and gate on misconfig findings * fix: harden k8s deployment security contexts for Trivy gate * fix: add base deployment securityContext for Trivy misconfig policy --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: fallback to PR when GitOps push hits protected branch * fix: ensure GitHub Pages configuration is enabled in workflow * fix: harden CD dependency gating and branch-safe GitOps fallback * fix: harden CD gating and protected-branch manifest flow * refactor: simplify to 2-env branch-based deployment (dev/prod) - Remove stage and preview overlays and workflows - Simplify ArgoCD ApplicationSet to 2 generators (dev→dex-dev, main→dex) - Update AppProject destinations to dex and dex-dev namespaces only - Simplify CD workflow update-gitops-manifests for dev/main branches - Update all docs (ARCHITECTURE, CI_CD, DEPLOY_RUNBOOK, LOCAL_K8S_SETUP) - Simplify promote.sh, get-tags.sh, alerting rules - Remove preview-deploy.yml workflow Closes the branch-based deployment re-architecture. Refs: #64 * fix: remove preview-deploy.yml (no preview overlay exists in 2-env model) --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * chore: update dev image to sha-77f6984b [skip ci] Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix: remove preview-deploy.yml and fix docs-pages enablement - Delete preview-deploy.yml (2-env model has no preview environments) - Remove enablement: true from configure-pages (requires admin perms) - Pages must be enabled once via repo Settings > Pages > Source: GitHub Actions * chore: update dev image to sha-eca077ab [skip ci] (#116) Auto-merge: GitOps dev image update (sha-eca077ab) * feat: clean public API surface & stabilize exports (#86) (#126) * chore: update dev image to sha-de539bec [skip ci] (#127) Auto-merge: GitOps dev image update (sha-de539bec) * feat: P1-high issues #87 #88 #89 #92 #93 — storage, ML serving, drift scheduler, docstrings (#128) * feat: add PyPI badge and update CHANGELOG (#87) * feat: storage abstraction — add list_objects, exists, get_storage factory (#89) - Add list_objects(prefix) and exists(path) abstract methods to StorageBackend ABC - Implement in all 6 backends: LocalParquetStorage, BigQueryStorage, JsonStorage, ParquetStorage, S3Storage, GCSStorage - Add get_storage(uri) factory: file:// → JsonStorage, s3:// → S3Storage, gs:// → GCSStorage - Export get_storage from lakehouse package - Add py.typed and update StorageBackend docstring - 12 new unit tests (264 total passed) * feat: generic model serving API endpoints (#92) - Add ML router with POST /predict, GET /models, GET /models/{name} - Add ML-specific Prometheus metrics (latency, throughput, drift) - Add PredictRequestBody/PredictResponseBody Pydantic models - Add set_model_server/get_model_server configuration helpers - Add 9 unit tests for ML router endpoints * test: storage integration tests with Docker emulators - Add endpoint_url param to S3Storage for LocalStack support - Add api_endpoint param to GCSStorage for fake-gcs-server support - Add docker-compose.test.yml (LocalStack:4566 + fake-gcs-server:4443) - Add LocalStack init script for test bucket creation - Add Terraform module for real cloud test buckets (AWS S3 + GCS) - Add 25 integration tests with emulator auto-detection - Fix unit tests to pass endpoint kwargs preventing auth failures * feat: drift monitoring scheduler with alerting (#93) - Add DriftScheduler with background thread for periodic checks - Add DriftMonitorConfig / DriftCheckResult dataclasses - Publish PSI scores to model_drift_psi Prometheus gauge - Increment model_drift_alerts_total counter on drift detection - Add Prometheus alert rules (moderate, severe, spike, stale) - Add mypy overrides for optional cloud SDKs (boto3, google) - Add 18 unit tests for scheduler lifecycle and metrics * docs: add docstrings to all public APIs (#88) - Add Google-style docstrings to 55+ methods and 9 Pydantic models - Cover core/schemas, api/, data/, lakehouse/, ml/, warehouse/ - Fix storage.py ParquetStorage read/delete try-block structure - All 40 source files pass mypy strict, 299 tests pass * chore: bump dataenginex to 0.5.0, fix docs workflow & Dockerfile - Bump dataenginex version from 0.4.11 to 0.5.0 - Fix docs-pages.yml: use uv sync --frozen instead of uv lock + uv sync - Fix Dockerfile: add missing COPY --from=builder /build/packages /app/packages - Update CHANGELOG with [0.5.0] - 2026-03-01 release - Update RELEASE_NOTES, CI_CD docs, roadmap CSV - Clean up mypy type-ignore annotations for cloud SDK imports * fix: resolve CI mypy unused-ignore and Trivy S3 encryption (AWS-0132) - Add per-module mypy override for storage module (warn_unused_ignores=false) handles env asymmetry: pre-commit has google-cloud-storage, CI does not - Keep # type: ignore[no-untyped-call] on AnonymousCredentials for pre-commit - Switch S3 test bucket from AES256 (SSE-S3) to SSE-KMS with CMK (aws_kms_key with rotation + bucket_key_enabled) Fixes CI workflow (mypy) and Security workflow (Trivy AWS-0132) * chore: add system setup script, consolidate READMEs to GUIDE.md - Add scripts/setup-system.sh: one-command system deps installer (Ubuntu/Debian, Fedora/RHEL, Arch, macOS) - Add poe task 'setup-system' for convenience - Update docs/DEVELOPMENT.md with system dependencies table - Update README.md Quick Start with setup-system.sh step - Rename 7 subdirectory README.md files to GUIDE.md (examples, scripts, packages, learning, src/careerdex, infra/monitoring/grafana, infra/terraform/test-storage) - Fix 3 cross-references to renamed files - Keep only root README.md and packages/dataenginex README.md --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> * chore: update dev image to sha-0b003e0e [skip ci] (#129) Auto-merge: GitOps dev image update (sha-0b003e0e) * chore: update dev image to sha-b2ebaa36 [skip ci] (#131) Auto-merge: GitOps dev image update (sha-b2ebaa36) * feat: CareerDEX v0.5.0 — domain extraction, plugin system, ML modules, docs overhaul feat: CareerDEX v0.5.0 — domain extraction, plugin system, ML modules, and docs overhaul (#138) * fix: boost test coverage to 83%+ (feature → dev) (#140) * feat: mount ml_router in CareerDEX and add OpenAPI tags (#68) - Import and include ml_router from dataenginex.api.routers.ml - Add v1 and ml tags to custom OpenAPI schema - Enables ML model serving endpoints in CareerDEX API * fix: add pre-commit to dev dependencies and re-sync venv - pre-commit was used by git hooks but not listed as dependency - venv had stale shebangs from old directory path (DEX → dex) - Re-synced venv fixes poe and pre-commit resolution * refactor: extract domain code from dataenginex to careerdex - Move routers (ml.py, v1.py) from dataenginex to careerdex/api/routers/ - Move pipeline_config.py from dataenginex to careerdex/core/ - Add careerdex-specific schemas (JobPosting, UserProfile, JobSourceEnum) - Add careerdex validators (SchemaValidator, QualityScorer, DataHash) - Add careerdex settings, exceptions, and config modules - Clean up dataenginex framework: remove domain-specific code - Update all phase imports to use careerdex.core instead of dataenginex - dataenginex now contains only generic framework code * feat: add vectorstore, LLM adapter, plugin system, and dashboard - RAG vectorstore: VectorStoreBackend ABC, InMemoryBackend, ChromaDBBackend - LLM adapter: LLMProvider ABC, OllamaProvider (REST), MockProvider - Plugin system: DataEngineXPlugin ABC, PluginRegistry, discover() via entry_points - Dashboard: DashboardConfig, BaseDashboard, 4 Streamlit panels - All modules exported from dataenginex.ml, dataenginex.plugins, dataenginex.dashboard * feat: add local file connectors, embeddings wiring, and CareerDEX plugin - JsonFileConnector and CsvFileConnector for local job data ingestion - JobSourceEnum extended with LOCAL_JSON and LOCAL_CSV - Phase 3 embeddings: sentence-transformers integration with fallback - CareerDEX plugin: health_check, get_metrics, register_routes - Plugin auto-discovered via entry_points group 'dataenginex.plugins' * test: add tests for domain extraction, plugins, connectors, and ML modules - 441 tests passing, 28 skipped (optional deps) - New test suites: plugins, vectorstore, LLM, careerdex phases - Updated existing tests for domain extraction import changes - Load testing with Locust for API performance benchmarking - Test fixtures: sample_jobs.json, sample_jobs.csv * chore: coverage gate, dependency updates, and build config - Add --cov-fail-under=80 to test-cov-core poe task - Add sentence-transformers to ml dependency group - Add streamlit to dashboard dependency group with uv conflict config - Add dataenginex.plugins entry_points for CareerDEX plugin discovery - Add mypy overrides for streamlit stubs - Sync uv.lock with all dependency changes * docs: update documentation, examples, and changelogs - README: add plugin system and dashboard sections, update tech stack - TODO: mark Streamlit dashboard as complete - Architecture docs: reflect domain extraction changes - New examples: RAG demo, LLM quickstart, dashboard runner - CHANGELOG: plugin system, dashboard, coverage gate entries - Add quickstart guide and security scanning docs - Update tasks with findings and lessons learned * chore: update CI/CD workflows, tooling, and project configs - Update CI/CD workflows: ci.yml, cd.yml, security.yml, release pipelines - Add Claude Code commands and settings - Update pre-commit hooks and gitignore patterns - Add gitleaks config for secret scanning - Update Dockerfile for new dependency groups - Update SonarQube and MCP server configs - Update agent/instruction/prompt files for new modules * chore: fix docs formatting, version refs, and add missing docs - Add API reference docs: plugins.md, dashboard.md - Update mkdocs nav with new module pages - Add Bruno API collection for all 8 CareerDEX endpoints - Update ADR medallion version reference * fix: align example 04_ml_training.py with actual API signatures - SklearnTrainer: use model_name/version/estimator constructor args - ModelRegistry: use persist_path, register(ModelArtifact) - DriftDetector: use check_dataset() instead of nonexistent detect() - Add proper sklearn import and evaluate() call * feat: boost test coverage to 83%+ via ML endpoint integration tests - Initialize ModelServer and ModelRegistry in app lifespan for ML endpoint support - Add comprehensive integration tests for ML endpoints (G…
Main was missing the pre-commit>=4.5.1 dev dependency added in #152. Keep dev's pyproject.toml and uv.lock which include it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stripped by the flatten-workspace op (6271d3a) — moved to org .github repo where they don't execute. Restored to dex/.github/workflows/ so release automation, PyPI publishing, and security scans run again. Restored: - release-dataenginex.yml — tags + GitHub release on pyproject.toml bump - pypi-publish.yml — TestPyPI/PyPI publish triggered by release - security.yml — Trivy + CodeQL on push/PR to main/dev Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: bump GitHub Actions to Node.js 24-compatible versions Applies all 4 Dependabot upgrade PRs (#147-#150) in one pass across ci.yml, pypi-publish.yml, release-dataenginex.yml, and security.yml. Closes #147, #148, #149, #150. - actions/checkout v4 → v6 - actions/setup-python v5 → v6 - codecov/codecov-action v4 → v5 - astral-sh/setup-uv v5 → v7 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: wire RAG→LLM — SentenceTransformerEmbedder + RAGPipeline.answer() Closes the gap between the vector store and LLM provider layers: - Add SentenceTransformerEmbedder: callable embed_fn backed by sentence-transformers (optional dep group: ml). Falls back gracefully with a clear ImportError if not installed. - Add RAGPipeline.answer(question, llm): full retrieve→augment→generate loop in one call. Delegates to LLMProvider.generate_with_context(). - Update examples/05_rag_demo.py: CLI flags --embed/--llm show hash fallback, sentence-transformer, mock, and Ollama paths. - Add mypy overrides for sentence_transformers (ignore_missing_imports) and vectorstore module (warn_unused_ignores=false) so type: ignore comments survive both envs (with and without optional ml deps). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Main had old action versions (v4/v5) from the restore-cicd-workflows PR. Dev has updated versions (v6/v7) from the bump-github-actions PR. Taking dev's workflow files throughout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…sh (#158) PyPI 0.6.0 was built from pre-PR#157 code (manual dispatch ran before the RAG/LLM wiring merge). Bump to 0.6.1 so the pipeline publishes SentenceTransformerEmbedder and RAGPipeline.answer() correctly. Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…#160) All 8 CVEs (CVE-2025-62402, CVE-2025-62503, CVE-2025-66388, CVE-2025-65995, CVE-2025-68438, CVE-2025-68675, CVE-2026-22922, CVE-2026-24098) affect apache-airflow>=3.0.0 only. airflow 3.x also conflicts with fastapi>=0.128.4 (requires <0.118.0). Pin to >=2.8.0,<3.0.0 — resolves to 2.11.2 which is clean. Co-authored-by: data-literate <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
apache-airflow>=2.8.0,<3.0.0to eliminate all 8 Dependabot CVEs (all in airflow 3.x). Resolves to2.11.2. Airflow 3.x also conflicts with ourfastapi>=0.128.4requirement.Test plan
dataenginex 0.6.1now on PyPI (published via separate release event)