A production-style market data ingestion and validation framework for historical OHLCV bars (Daily + 1-Minute) using Alpaca market data. The pipeline writes curated, partitioned Parquet datasets and provides a structured QA layer with artifact-based observability and optional strict enforcement suitable for CI gating and trading research workflows.
For pip-install and notebook usage, including Colab-ready fintech-init-project --colab-profile setup, the Colab data-session profile for local runtime roots, mounted Drive persistence roots, archive roots, date windows, timeframes, symbols, and StratLake handoff via CURATED_ROOT, see docs/notebook_pip_install_guide.md. For a complete fintech-side lifecycle from fresh Colab runtime to StratLake handoff metadata, see docs/m11_colab_fintech_to_stratlake_workflow.md and docs/m11_release_readiness.md. That guide also includes fintech-notebook-doctor, a read-only readiness command that checks workspace roots, optional mounted Drive/archive paths, likely restored datasets, and optional secret presence (SET/NOT SET only). For portable project-session boundaries and mounted-path persistence, see docs/project_sessions.md, docs/google_drive_persistence.md, examples/project_session_quickstart.py, and docs/m9_release_readiness.md. For the M10 archive backup pack manifest contract, see docs/archive_backup_pack_manifest.md; for the Colab restore-to-local workflow with mounted Google Drive archival storage, see docs/colab_archive_backup_restore.md; for focused M10 validation, see docs/m10_release_readiness.md. For cross-platform contributor setup and validation, see docs/cross_platform_contributor_validation.md. For setup, packaging, linting, build validation, and future publishing boundaries, see docs/packaging_pypi_readiness.md. For the focused M7 release checklist, see docs/m7_release_readiness.md. For the focused M5 release checklist, see docs/m5_release_readiness.md. For the focused M4 validation checklist, see docs/m4_release_readiness.md.
Version 0.11.0 finalizes M11 Colab notebook ergonomics and fintech-side
handoff readiness workflows for StratLake consumers while preserving local
Parquet canonicality.
- Reusable Colab data-session profile and end-to-end fintech lifecycle docs.
fintech-init-project --colab-profilefor Colab runtime workspace bootstrap.- Read-only
fintech-notebook-doctorchecks for pre/post restore readiness. - Post-restore QA recipe for local curated datasets.
- Derived/non-canonical
fintech-stratlake-handoff-reportCLI and API. - Deterministic handoff-report default metadata (
generated_at_utc = null). - Deterministic lexical QA-run selection note for handoff artifact references.
- M11 release-readiness and merge-readiness documentation.
Boundary preserved: local partitioned Parquet remains canonical working data, Google Drive remains mounted filesystem persistence/archive storage, and archive packs plus handoff metadata remain derived/non-canonical artifacts.
Version 0.10.0 finalizes M10 archive backup packs and Colab restore workflows while preserving local Parquet canonicality.
- Deterministic archive backup pack manifest contract for local partitioned Parquet inventories.
- Sharded ZIP backup pack writer with dry-run planning and stable archive/member ordering.
- Local restore workflow with staged extraction and explicit
fail,replace, andmergeoverwrite policies. - Read-only validation and inspection APIs for notebook-friendly pack checks.
fintech-backup-dataCLI commands for pack, restore, validate, and inspect workflows.- Colab restore-to-local-first documentation and deterministic round-trip validation.
Boundary preserved: local partitioned Parquet remains canonical working data, archive backup packs are derived/non-canonical transfer artifacts, and mounted Google Drive is treated only as ordinary filesystem archival storage.
Alpaca API
↓
Alpaca Client (retry + pagination)
↓
Normalization Layer (UTC + dedupe)
↓
Partitioned Parquet (symbol/date)
↓
DuckDB Analytics + QA Observability
-
Auth via
.env -
Supports:
symbol,start,end,timeframe(1Day,1Min),feed(default:iex) -
Automatic pagination
-
Exponential backoff for:
- HTTP 429 (rate limiting)
- HTTP 5xx (server errors)
- network timeouts
-
Returns results as a pandas DataFrame
-
Iterates over a 50-ticker universe
-
Fetches daily bars in monthly windows
-
Normalizes into a canonical schema (UTC enforced)
-
Deterministic dedupe on primary key:
(symbol, ts_utc, timeframe) -
Writes partitioned Parquet:
data/curated/bars_daily/symbol=XYZ/date=YYYY-MM-DD/ -
Safe to re-run (idempotent)
-
Day-windowed ingestion for intraday OHLCV bars
-
Failure isolation per (symbol, date)
-
Restartable + rate-limit safe
-
Writes partitioned Parquet:
data/curated/bars_1m/symbol=XYZ/date=YYYY-MM-DD/
- Implemented via
pyarrow.write_to_dataset - Uses
delete_matchingto safely overwrite partitions - DuckDB-compatible queries over glob paths
-
Duplicate primary key detection
-
OHLC integrity checks
-
Timestamp sanity / ordering checks
-
Gap detection (daily + intraday)
-
Coverage metrics (calendar-aware)
-
Deterministic artifacts per QA run:
artifacts/qa/<run_id>/ qa_summary_by_symbol.csv qa_summary_global.csv qa_coverage_by_symbol.csv -
Optional strict enforcement mode (CI-friendly)
M3 adds a separate Alpaca corporate-actions ingestion path for dividend events. Corporate actions are event-based issuer/security records, not OHLCV bars, trades, or quotes.
Supported dividend event types:
cash_dividend
stock_dividend
The corporate-actions client supports both flat Alpaca responses and nested
dividend groups under corporate_actions.cash_dividends and
corporate_actions.stock_dividends. Top-level cash_dividends and
stock_dividends groups are also accepted. Missing Alpaca currency on
cash-dividend rows defaults the normalized currency field to USD, with the
defaulting count recorded in dividend metadata.
Patch release 0.5.1 includes this nested Alpaca dividend response support,
the explicit cash-dividend missing-currency policy, metadata counters for
nested responses and currency defaulting, and preserved flat/list response
compatibility.
Live CLI usage:
fintech-ingest-corporate-actions \
--symbols AAPL MSFT SPY \
--start 2024-01-01 \
--end 2024-12-31 \
--types cash_dividend stock_dividend \
--output-root data/curated/corporate_actions/dividendsThe existing module invocation remains supported:
python -m src.cli.ingest_corporate_actions \
--symbols AAPL MSFT SPY \
--start 2024-01-01 \
--end 2024-12-31 \
--types cash_dividend stock_dividend \
--output-root data/curated/corporate_actions/dividendsExpected outputs:
data/curated/corporate_actions/dividends/dividends.parquet
data/curated/corporate_actions/dividends/metadata.json
Derived research mart output is also available for symbol and year analysis:
data/research/corporate_actions/dividends/
Build the derived mart from an existing curated dividend snapshot with:
python -m src.cli.build_dividend_research_mart \
--snapshot-root data/curated/corporate_actions/dividends \
--research-root data/research/corporate_actions/dividendsInstalled console script:
fintech-build-dividend-research-mart \
--snapshot-root data/curated/corporate_actions/dividends \
--research-root data/research/corporate_actions/dividendsValidate and inspect the derived mart without mutating it:
python -m src.cli.validate_dividend_research_mart \
--research-root data/research/corporate_actions/dividendsInstalled console script:
fintech-validate-dividend-research-mart \
--research-root data/research/corporate_actions/dividendsThe validation command is read-only and reports validation status in its JSON
valid field.
Join local dividend events to local bar data as a derived research output:
python -m src.cli.join_dividend_event_windows \
--dividend-source research-mart \
--research-root data/research/corporate_actions/dividends \
--bars-path data/local/synthetic_bars.parquet \
--pre-window-days 2 \
--post-window-days 2 \
--output-path data/research/corporate_actions/dividend_event_windows/event_windows.parquetInstalled console script:
fintech-join-dividend-event-windows \
--dividend-source research-mart \
--research-root data/research/corporate_actions/dividends \
--bars-path data/local/synthetic_bars.parquet \
--pre-window-days 2 \
--post-window-days 2 \
--output-path data/research/corporate_actions/dividend_event_windows/event_windows.parquetThe join command is local, credential-free, and does not mutate canonical dividend snapshots, research marts, or bar inputs.
For a contract-style derived output with metadata:
python -m src.cli.join_dividend_event_windows \
--dividend-source research-mart \
--research-root data/research/corporate_actions/dividends \
--bars-path data/local/synthetic_bars.parquet \
--pre-window-days 2 \
--post-window-days 2 \
--output-root data/research/corporate_actions/dividend_event_windows/example_runThat writes event_windows.parquet and metadata.json under the output root.
The metadata records dividend input path, bar input path, event-window config,
row counts, and schema. Event-window outputs are derived research artifacts, not
canonical data.
Notebook-style quickstart with synthetic data only:
python examples/dividend_research_mart_quickstart.py \
--output-root artifacts/examples/dividend_research_mart_quickstartThe quickstart is plain Python and safe for CI. It demonstrates snapshot -> research mart -> validation -> event-window join -> derived output metadata without calling Alpaca or mutating canonical repository data.
Pipeline-style synthetic workflow:
python examples/dividend_research_pipeline_workflow.py \
--output-root artifacts/examples/dividend_research_pipeline_workflowThe workflow example is scheduler-free and uses small stage functions. It writes only under the configured output root and demonstrates snapshot -> research mart -> validation -> event-window join -> derived output metadata -> workflow summary.
The deterministic curated snapshot remains the canonical ingestion output.
For the full schema, metadata contract, CLI arguments, and a CI-safe sample-data example, see docs/corporate_actions_dividends.md.
Dividend records are event evidence. See docs/corporate_actions_dividends.md for point-in-time research semantics and adjusted-return boundaries.
Derived dividend-to-bars event-window joins are available for research workflows via src.ingestion.dividend_event_window.
M7 makes the dividend research workflow easy to build, inspect, validate, and
compose from CLI commands, Python APIs, notebook-style scripts, and
pipeline-style scripts. The boundary is explicit: the curated dividend snapshot
under data/curated/corporate_actions/dividends/ remains the canonical
ingestion artifact; research marts and event-window outputs under
data/research/ are derived research artifacts.
Build the derived dividend research mart:
fintech-build-dividend-research-mart \
--snapshot-root data/curated/corporate_actions/dividends \
--research-root data/research/corporate_actions/dividends \
--summary-output artifacts/examples/dividend_research_mart_summary.jsonValidate and inspect the mart without mutating it:
fintech-validate-dividend-research-mart \
--research-root data/research/corporate_actions/dividends \
--summary-output artifacts/examples/dividend_research_mart_validation.jsonJoin dividend events to local bar data and write direct CSV or Parquet output:
fintech-join-dividend-event-windows \
--dividend-source research-mart \
--research-root data/research/corporate_actions/dividends \
--bars-path data/local/synthetic_bars.parquet \
--pre-window-days 2 \
--post-window-days 2 \
--output-path data/research/corporate_actions/dividend_event_windows/event_windows.parquetWrite contract-style derived event-window output with metadata:
fintech-join-dividend-event-windows \
--dividend-source research-mart \
--research-root data/research/corporate_actions/dividends \
--bars-path data/local/synthetic_bars.parquet \
--pre-window-days 2 \
--post-window-days 2 \
--output-root data/research/corporate_actions/dividend_event_windows/example_runModule invocations are also supported for the three CLIs:
python -m src.cli.build_dividend_research_mart --help
python -m src.cli.validate_dividend_research_mart --help
python -m src.cli.join_dividend_event_windows --helpRun the CI-safe synthetic examples:
python examples/dividend_research_mart_quickstart.py \
--output-root artifacts/examples/dividend_research_mart_quickstart
python examples/dividend_research_pipeline_workflow.py \
--output-root artifacts/examples/dividend_research_pipeline_workflowThe examples use fixed local synthetic data, require no Alpaca credentials, do not call Alpaca, and write only under the configured output root. M7 does not implement adjusted prices, total-return reconstruction, dividend reinvestment, or backtest cash-flow behavior. See docs/corporate_actions_dividends.md for the detailed metadata contract, safety boundaries, API examples, and output layouts.
The preferred contributor workflow is editable install with the development dependency group:
python -m pip install -e ".[dev]"That setup supports the installed console script and the existing module invocation:
fintech-ingest-corporate-actions --help
python -m src.cli.ingest_corporate_actions --helpRun the cross-platform validation wrapper before opening a pull request:
python scripts/validate.pyThe wrapper includes repository hygiene checks for LF line endings, generated outputs accidentally tracked by Git, local absolute paths, and credential files. You can run those checks directly with:
python scripts/check_repo_hygiene.pyTo smoke-test the built wheel locally without publishing:
python -m build
python scripts/smoke_test_wheel.pyFor a full cross-platform contributor workflow, see docs/cross_platform_contributor_validation.md. For a packaging and PyPI-readiness walkthrough, including validation commands, build inspection, and cleanup, see docs/packaging_pypi_readiness.md. The M7 release-readiness checklist lives in docs/m7_release_readiness.md, the M5 release-readiness checklist lives in docs/m5_release_readiness.md, and the M4 deterministic release-readiness checklist lives in docs/m4_release_readiness.md.
If you need a frozen environment for reproduction, requirements.txt is available as an optional path, but it is not the preferred development workflow for contributors.
The repository includes a framework-level QA export command that computes integrity metrics and writes deterministic artifacts per run.
Artifacts are written to:
artifacts/qa/<run_id>/
qa_summary_by_symbol.csv
qa_summary_global.csv
qa_coverage_by_symbol.csv
Exports QA metrics but does not fail the pipeline.
python -m src.ingestion.qa_export \
--timeframe all \
--start 2025-11-01 \
--end 2025-12-01You may also run for a single timeframe:
python -m src.ingestion.qa_export \
--timeframe 1D \
--start 2025-11-01 \
--end 2025-12-01python -m src.ingestion.qa_export \
--timeframe 1Min \
--start 2025-11-01 \
--end 2025-12-01Fails the pipeline if integrity thresholds are exceeded.
python -m src.ingestion.qa_export \
--timeframe all \
--start 2025-11-01 \
--end 2025-12-01 \
--strict \
--max-duplicate-keys 0 \
--max-ohlc-violations 0| Condition | Exit Code |
|---|---|
| PASS (within thresholds) | 0 |
| WARN (coverage/gap thresholds exceeded) | 0 |
| FAIL (strict integrity violation) | 1 |
Strict mode ensures:
- Duplicate primary keys fail the pipeline
- OHLC violations fail the pipeline
- QA artifacts are still written for debugging
- CI jobs automatically fail when data integrity is compromised
- name: Strict QA
run: |
python -m src.ingestion.qa_export \
--timeframe all \
--start 2025-11-01 \
--end 2025-12-01 \
--strict \
--max-duplicate-keys 0 \
--max-ohlc-violations 0If integrity rules are violated, the CI job exits with code 1.
In trading and backtesting systems:
- Duplicate bars distort indicators (EMA, RSI, VWAP, etc.)
- OHLC violations corrupt return calculations
- Silent ingestion errors invalidate research
- Coverage gaps can bias signal evaluation
The qa_export CLI formalizes dataset observability and enables production-grade guardrails before any modeling or execution layer consumes the data.
After local restore, local backfill, or post-restore QA, generate a deterministic local handoff report that summarizes curated dataset presence, symbols, date coverage, timeframe coverage, and QA artifact references.
python -m src.cli.stratlake_handoff_report \
--root . \
--curated-root data/curated \
--qa-root artifacts/qa \
--output artifacts/handoff/stratlake_marketlake_handoff.jsonInstalled console script:
fintech-stratlake-handoff-report \
--root . \
--curated-root data/curated \
--qa-root artifacts/qa \
--output artifacts/handoff/stratlake_marketlake_handoff.jsonUse stratlake_marketlake_root from the generated JSON as StratLake
MARKETLAKE_ROOT or --marketlake-root.
By default, generated_at_utc is null, which keeps the JSON deterministic
for unchanged local filesystem inputs. Pass --generated-at-utc only when you
want explicit timestamp metadata in the report.
If QA artifacts are present, the selected QA run is chosen by deterministic lexical ordering of QA run directory names.
Boundary: the report is derived/non-canonical diagnostics. It does not run ingestion, QA, archive restore/pack, Google API/OAuth, network operations, or StratLake execution workflows.
The project operates on a curated universe (currently 50 tickers) optimized for stable historical availability.
Document:
- Selection logic (liquidity + large-cap bias)
- Exclusion rules
- Intended expansion strategy
Tip: include this as
docs/universe.mdso it’s easy to audit and evolve.
Reusable Alpaca historical market data client:
src/ingestion/alpaca_client.py
from src.ingestion.alpaca_client import AlpacaMarketDataClient
client = AlpacaMarketDataClient.from_env()
df = client.fetch_bars(
symbol="AAPL",
start="2025-01-01T00:00:00Z",
end="2025-02-01T00:00:00Z",
timeframe="1Day"
)
print(df.head())python -m venv venv
.\venv\Scripts\activateFor local development, install the project in editable mode with the optional development dependencies:
python -m pip install -e ".[dev]"This keeps the current src.* import layout available while preparing the
project for standard Python build tooling.
requirements.txt records a fuller pinned working environment, including
notebook and exploratory tooling. Use it when you need to reproduce that exact
environment:
python -m pip install -r requirements.txtIf you install new packages and want to pin exact versions:
pip freeze > requirements.txtRuff provides the baseline lint and formatting checks for M4 quality gates:
ruff check src tests examples
ruff format --check src tests examplesTo apply Ruff formatting locally:
ruff format src tests examplesBuild the source distribution and wheel locally before any release preparation:
python -m buildBuild outputs are written under dist/. Inspect artifact contents before any
future TestPyPI or PyPI publishing step:
tar tf dist/*.tar.gz
python -m zipfile --list dist/*.whlPowerShell inspection alternatives:
Get-ChildItem dist\*.tar.gz | ForEach-Object { tar tf $_.FullName }
Get-ChildItem dist\*.whl | ForEach-Object { python -m zipfile --list $_.FullName }The package artifacts must not include credentials, .env, generated datasets,
local artifacts, reports, caches, virtual environments, or build outputs.
Optional wheel smoke test:
python -m pip install dist/*.whl
python -m src.cli.ingest_corporate_actions --helpClean generated build artifacts when finished:
rm -rf dist build *.egg-infoPowerShell equivalent:
Remove-Item -Recurse -Force dist, build -ErrorAction SilentlyContinue
Remove-Item -Recurse -Force *.egg-info -ErrorAction SilentlyContinueCreate a .env file in the project root:
ALPACA_API_KEY_ID=your_key_here
ALPACA_API_SECRET_KEY=your_secret_here
ALPACA_DATA_BASE_URL=https://data.alpaca.markets
ALPACA_FEED=iexdata/
curated/
bars_daily/
bars_1m/
artifacts/
qa/
reports/
src/
ingestion/
qa/
utils/
notebooks/
python -m src.ingestion.backfill_daily --start 2023-01-01 --end 2026-01-01data/curated/bars_daily/symbol=XYZ/date=YYYY-MM-DD/part-*.parquet
Windowed ingestion of 1-minute OHLCV bars (IEX-compatible) into curated Parquet partitions.
Designed for:
- Idempotent re-runs
- Restartable backfills
- Rate-limit safe API usage
- Failure isolation
- DuckDB/Pandas/backtesting compatibility
| Column | Type | Description |
|---|---|---|
| symbol | string | Ticker symbol |
| ts_utc | timestamp (UTC) | Bar open time in UTC |
| open | float | Open price |
| high | float | High price |
| low | float | Low price |
| close | float | Close price |
| volume | float/int | Trade volume |
| source | string | e.g., alpaca_iex |
| timeframe | string | "1Min" |
| date | string | YYYY-MM-DD derived from ts_utc |
Primary key (idempotency):
(symbol, ts_utc, timeframe)
data/curated/bars_1m/
symbol=XYZ/
date=YYYY-MM-DD/
part-*.parquet
python -m src.ingestion.backfill_1m \
--start 2024-01-01 \
--end 2024-04-01Optional flags:
--symbols configs/tickers_50.txt
--max-symbols 5
--sleep-ms 100
--no-progressFailed (symbol, date) requests are recorded:
reports/backfill_1m_failures_YYYYMMDD_HHMMSS.csv
Backfill continues even when individual symbol/day fetches fail.
Example: validate per-symbol ranges and row counts directly over Parquet:
import duckdb
con = duckdb.connect()
df = con.execute("""
SELECT symbol,
MIN(ts_utc) AS min_ts,
MAX(ts_utc) AS max_ts,
COUNT(*) AS row_count
FROM 'data/curated/bars_daily/**/*.parquet'
GROUP BY symbol
ORDER BY symbol
""").df()
print(df.head())Validates curated market data after ingestion runs (daily and 1-minute).
Ensures:
- Primary key uniqueness
- OHLC integrity
- Timestamp ordering
- Coverage / gap detection
- Repeatable reporting
- Optional strict enforcement mode
| Timeframe | Path |
|---|---|
| Daily (1D) | data/curated/bars_daily/**/*.parquet |
| 1-Minute | data/curated/bars_1m/**/*.parquet |
Expected columns:
symbol, ts_utc, open, high, low, close, volume, source, timeframe
Primary key:
(symbol, ts_utc, timeframe)
Detects duplicates on:
(symbol, ts_utc, timeframe)
- Reports per symbol
- Strict mode fails if duplicates exist
Validates:
high >= max(open, close)low <= min(open, close)high >= lowopen, high, low, close > 0volume >= 0- Required fields non-null
ts_utcnon-null- Out-of-order detection per symbol
- Calendar-aware expected sessions (XNYS default)
Gap metrics computed within (symbol, session_date) using timestamp deltas:
gap_countgap_segmentsmax_gap_len
This prevents overnight/weekend closures from being counted as gaps.
Note on Alpaca IEX feed: missing minute bars may reflect “no trades” rather than corruption. Tune thresholds accordingly.
QA exports are written to:
artifacts/qa/<run_id>/
qa_summary_by_symbol.csv
qa_summary_global.csv
qa_coverage_by_symbol.csv
Global summary includes:
- total_rows / unique_rows / duplicate_rows
- total_gap_count
- total_ohlc_violation_count
- pct_symbols_below_coverage_threshold
- overall_status (PASS / WARN / FAIL)
Run IDs are deterministic from:
dataset_name + interval + start_ts + end_ts + calendar
When strict mode is enabled:
- Duplicate primary keys → FAIL (exit code 1)
- OHLC violations over threshold → FAIL (exit code 1)
- Artifacts still written for debugging
- CI-compatible gating
python -m src.ingestion.qa_export \
--timeframe all \
--start 2025-11-01 \
--end 2025-12-01 \
--strict \
--max-duplicate-keys 0 \
--max-ohlc-violations 0Run unit tests:
pytest -qTests cover:
- Normalization logic (canonical schema + UTC + sorting)
- Duplicate key detection
- OHLC violation detection
- Strict enforcement path
Notebook:
notebooks/01_duckdb_sanity_queries.ipynb
Validates:
- Schema presence and timestamp parsing
- Coverage + completeness
- Duplicate primary key contract
- Basic validity constraints
- Example analytics with window functions
Exports:
reports/duckdb_sanity_summary.csv
- Fully session-aware expected-minute index comparison
- Extended-hours handling
- Per-run historical QA trend table
- Machine-readable JSON summary export
- CI auto-fail policies based on
qa_summary_global.csv
