Skip to content

Commit bbea40c

Browse files
nsheffnsheff
committed
Add comprehensive data loading documentation
1 parent ce53bed commit bbea40c

3 files changed

Lines changed: 149 additions & 50 deletions

File tree

‎docs/bedbase/bedbase-data-loading.md‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# BEDbase Data Loading
2+
3+
This document describes how BEDbase stays up to date with new BED files from public repositories.
4+
5+
## Data Loading Steps
6+
7+
1. **GEO publishes new data** — Researchers deposit BED files (ChIP-seq peaks, ATAC-seq regions, etc.) to NCBI GEO
8+
2. **geopephub scans GEO** — Every 2 days, [pepkit/geopephub](https://github.com/pepkit/geopephub) GitHub Actions scans GEO for new projects containing BED-like files
9+
3. **geopephub uploads to PEPhub** — Matching projects are uploaded as PEPs to the [`bedbase` namespace](https://pephub.databio.org/bedbase) in PEPhub
10+
4. **bedbase-loader queries PEPhub** — Daily, [databio/bedbase-loader](https://github.com/databio/bedbase-loader) GitHub Actions queries PEPhub for recently updated projects
11+
5. **Light processing runs** — Files are downloaded, validated, and inserted into PostgreSQL/S3 (fast, no heavy computation)
12+
6. **Heavy processing runs** — AWS Fargate runs statistics, generates embeddings, and indexes to Qdrant (slow, compute-intensive)
13+
14+
## Architecture Diagram
15+
16+
```mermaid
17+
flowchart TB
18+
subgraph Sources
19+
GEO[("1. GEO (NCBI)<br/>Source Data")]
20+
end
21+
22+
subgraph PEPhubSystem["PEPhub System (pepkit)"]
23+
geopephub["2. geopephub<br/>(GitHub Actions)"]
24+
PEPhub[("3. PEPhub DB<br/>bedbase namespace")]
25+
end
26+
27+
subgraph BEDbaseAutomation["BEDbase Automation (databio)"]
28+
Loader["4. bedbase-loader<br/>(GitHub Actions)"]
29+
end
30+
31+
subgraph Infrastructure["BEDbase Infrastructure"]
32+
direction LR
33+
Postgres[("PostgreSQL")]
34+
Qdrant[("Qdrant")]
35+
S3[("S3")]
36+
end
37+
38+
subgraph HeavyProcessing
39+
Fargate["6. AWS Fargate"]
40+
end
41+
42+
GEO -->|"scan for BED files"| geopephub
43+
geopephub -->|"upload PEPs"| PEPhub
44+
PEPhub -->|"query by date"| Loader
45+
Loader -->|"5. Light processing"| Infrastructure
46+
Fargate -->|"Heavy processing"| Infrastructure
47+
```
48+
49+
## Data Sources
50+
51+
### GEO (Gene Expression Omnibus)
52+
53+
GEO is the primary source of BED files. NCBI's GEO database contains genomic datasets including ChIP-seq peaks (narrowPeak, broadPeak), ATAC-seq accessibility regions, DNase-seq hypersensitivity sites, and other genomic interval data.
54+
55+
### PEPhub and the `bedbase` Namespace
56+
57+
PEPhub (https://pephub.databio.org) serves as a metadata intermediary between GEO and BEDbase. The key component is the **`bedbase` namespace** — a curated subset of GEO containing only projects with BED-like files.
58+
59+
#### How the `bedbase` Namespace is Populated
60+
61+
The `bedbase` namespace is populated by **[geopephub](https://github.com/pepkit/geopephub)**, a tool maintained by the pepkit organization (separate from bedbase). This runs as a GitHub Actions workflow in the geopephub repository:
62+
63+
**Workflow:** `bedbase_uploader.yml` (runs every 2 days at 10:00 UTC)
64+
65+
```bash
66+
geopephub run-queuer --target bedbase --period 2
67+
geopephub run-uploader --target bedbase
68+
```
69+
70+
**What geopephub does:**
71+
72+
1. **Queuer** — Scans GEO for new projects containing BED-like files (narrowPeak, broadPeak, BED) using geofetch's Finder module
73+
2. **Uploader** — Downloads metadata via GEOfetch and uploads PEPs to PEPhub under the `bedbase` namespace
74+
3. **Checker** — Verifies upload success and retries failures
75+
76+
This means:
77+
- The `bedbase` namespace is **not** managed by the bedbase project itself
78+
- It's managed by the pepkit organization via geopephub
79+
- BEDbase (via bedbase-loader) is a **consumer** of this namespace, not the producer
80+
81+
#### PEPhub Namespaces
82+
83+
| Namespace | URL | Contents | Updated by |
84+
|-----------|-----|----------|------------|
85+
| `geo` | https://pephub.databio.org/geo | All GEO projects (~99% of GEO) | geofetch (weekly) |
86+
| `bedbase` | https://pephub.databio.org/bedbase | BED-file projects only | geopephub (every 2 days) |
87+
88+
## bedbase-loader Repository
89+
90+
The [databio/bedbase-loader](https://github.com/databio/bedbase-loader) repository contains the GitHub Actions workflows that pull data from the `bedbase` namespace and load it into BEDbase infrastructure.
91+
92+
| Workflow | Schedule | Purpose |
93+
|----------|----------|---------|
94+
| `upload_cron.yml` | Daily 18:00 UTC | Fetch new GEO samples from last 3 days |
95+
| `upload_cron_series.yml` | Daily 18:00 UTC | Fetch new GEO series from last 3 days |
96+
| `update_genomes.yml` | Weekly (Tuesdays) | Sync genome info from Refgenie |
97+
| `update_umap.yml` | Manual | Regenerate UMAP visualizations |
98+
99+
Configuration: See [`config.yaml`](https://github.com/databio/bedbase-loader/blob/master/config.yaml) in the bedbase-loader repo.
100+
101+
## Phase 1: Light Processing (GitHub Actions)
102+
103+
The daily workflows run `bedboss geo upload-all` with the `--lite` flag:
104+
105+
```bash
106+
bedboss geo upload-all \
107+
--start-date <3 days ago> --end-date <today> \
108+
--geo-tag samples --lite \
109+
--bedbase-config config.yaml --outfolder /tmp/out
110+
```
111+
112+
**What light processing does:**
113+
114+
1. Query PEPhub for GSE projects updated in the time window
115+
2. Skip already-processed projects (via database flags)
116+
3. Download metadata and BED files
117+
4. Validate file format and genome
118+
5. Insert records to PostgreSQL and upload files to S3
119+
6. Flag project as ready for heavy processing
120+
121+
## Phase 2: Heavy Processing (AWS Fargate)
122+
123+
After light processing, heavy processing runs on AWS Fargate via a scheduled task using the [bedboss Docker image](https://github.com/databio/bedboss/blob/main/Dockerfile):
124+
125+
```bash
126+
bedboss reprocess-all --bedbase-config config.yaml --outfolder /tmp/out --limit 100
127+
```
128+
129+
**What heavy processing does:**
130+
131+
1. Quality control (bedqc) — validate file integrity, region counts, region widths
132+
2. Statistics (bedstat) — calculate GC content, TSS distances, genomic feature percentages
133+
3. Embeddings (region2vec) — generate vector embeddings for semantic search
134+
4. Qdrant indexing — upload embeddings to vector database
135+
5. Flag file as fully processed
136+
137+
## Infrastructure
138+
139+
BEDbase uses: PostgreSQL (metadata, processing status), Qdrant (vector embeddings), and S3-compatible storage (files). See [BEDbase Configuration](./how-to-configure.md) for details.
140+
141+
## Related Documentation
142+
143+
- [databio/bedbase-loader](https://github.com/databio/bedbase-loader) — Automation repo with workflows and config
144+
- [databio/bedboss](https://github.com/databio/bedboss) — Processing pipeline CLI
145+
- [pepkit/geopephub](https://github.com/pepkit/geopephub) — Populates the `bedbase` namespace in PEPhub
146+
- [BEDboss CLI Reference](../bedboss/usage.md) — Full command documentation
147+
- [How to Upload GEO Data](../bedboss/how-to/upload-geo-data.md) — Manual upload guide
148+
- [BEDbase Configuration](./how-to-configure.md) — Config file format

‎docs/bedbase/bedbase-loader.md‎

Lines changed: 0 additions & 49 deletions
This file was deleted.

‎mkdocs.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ nav:
6868
- Guide: bbconf/bbc_api.md
6969
- Changelog: bbconf/changelog.md
7070
- 🧰 BEDBoss processing pipeline: bedboss/README.md
71-
- 🔄 BEDbase auto uploader: bedbase/bedbase-loader.md
71+
- 🔄 Data Loading: bedbase/bedbase-data-loading.md
7272
- 📜 Configuration file: bedbase/how-to-configure.md
7373
- Reference:
7474
- How to cite: citations.md

0 commit comments

Comments
 (0)