Skip to content

Commit f398cd0

Browse files
ybaah-biotechclaude
andcommitted
docs: interactive notebooks, LinkedIn series, overhauled README
notebooks/ 01_BLAST_and_Species_Parsing.ipynb — 21 cells; eDNA context, BLAST algorithm, live species parsing on 6 title formats, LCA demo, diversity metrics, 3 exercises. Runs fully offline. 02_Local_BLAST_and_Databases.ipynb — 16 cells; web vs local comparison, BLAST+ file formats, caching demo, database selection guide, db_version.json audit trail, 2 exercises. README.md — how to open in Codespaces or locally docs/linkedin/ post_01_project_intro.md — series launch, builder journey angle post_02_phase1_local_blast.md — rate-limit problem + version stamping post_03_phase2_pdf_reports.md — CSV-is-not-a-report, regulatory PDF README.md — full overhaul: 10-phase roadmap with statuses, updated project tree, all new CLI flags, output files table, offline demo section, non-technical intro for regulators and clients Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 5ac939e commit f398cd0

7 files changed

Lines changed: 1537 additions & 122 deletions

README.md

Lines changed: 158 additions & 122 deletions
Large diffs are not rendered by default.
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# LinkedIn Post 01 — Project Introduction
2+
3+
*Angle: Builder journey — biologist teaching himself bioinformatics*
4+
5+
---
6+
7+
Biology degrees teach you to pipette. They don't teach you to write a pipeline.
8+
9+
I spent years working with environmental DNA in the lab — extracting, amplifying, sequencing. But when I started trying to make sense of the data at scale, I hit a wall. Processing results from a single eDNA water survey meant hours of manual BLAST searches, copying hits into spreadsheets, and trying to remember which database version I'd used six months ago. There had to be a better way.
10+
11+
So I built one.
12+
13+
Over the past few months I've been writing a Python pipeline that automates species identification from environmental DNA samples. You give it a FASTA file of sequences from a water sample; it runs BLAST, parses the results, resolves taxonomy using Lowest Common Ancestor logic, and outputs a clean species table with confidence scores. With the local BLAST mode I added in Phase 1, 100 sequences that used to take hours over the web API now finish in about 8 minutes offline.
14+
15+
Where it stands now:
16+
- 38 unit tests passing
17+
- 10 phases planned — 2 complete, 1 in progress
18+
- Runs entirely offline once a database is downloaded
19+
- Generates a regulatory PDF report that an environmental consultant or EA officer can actually read
20+
21+
I'm an MSc Biotechnology graduate (NTU, High Commendation) building commercial tools for eDNA work. This series documents the build phase by phase — the decisions, the mistakes, and what I learn along the way.
22+
23+
Follow along if you work in ecology, environmental consulting, or bioinformatics — or if you're also figuring out the gap between a biology degree and the computational world.
24+
25+
#eDNA #Bioinformatics #Python #EnvironmentalDNA #OpenSource
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# LinkedIn Post 02 — Phase 1: Local BLAST+
2+
3+
*Angle: Solved a real problem — the rate-limiting bottleneck of web BLAST*
4+
5+
---
6+
7+
The NCBI web BLAST API caps you at 3 requests per second. Each sequence takes 30 to 120 seconds to come back. Run 100 sequences and you're waiting 3 hours — if nothing times out.
8+
9+
That's not a workflow. That's babysitting.
10+
11+
Phase 1 of my eDNA pipeline replaces the web API with a local BLAST+ installation running on your own machine. Same science, same database, no rate limits. With `--threads 4` on an SSD, those same 100 sequences run in about 8 minutes.
12+
13+
The core of it is `src/local_blast.py` — a subprocess wrapper that fires BLAST jobs in parallel using Python's `ThreadPoolExecutor`. Each query writes its own XML file, so nothing blocks and caching still works. If you stop halfway through and restart, it picks up from where it left off.
14+
15+
The part I didn't expect to care about: database version stamping.
16+
17+
NCBI updates its nucleotide database daily. A sequence that maps to *Betula pendula* today might resolve to a different accession or a revised species name in six months, because taxonomists update records continuously. If you can't say exactly which version of the database you used, your results are technically not reproducible — and for environmental compliance work, that matters.
18+
19+
Every local run now writes a `db_version.json` file alongside the results. It records the database path, the version string from `blastdbcmd`, and the timestamp. The PDF report embeds this in the methodology section automatically.
20+
21+
The learning: I didn't expect version control for biological databases to be a scientific credibility issue — but it is.
22+
23+
Code: github.com/ybaah-biotech/eDNA-sequence-analysis
24+
25+
#eDNA #BLAST #Bioinformatics #Python #ReproducibleScience
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# LinkedIn Post 03 — Phase 2: Regulatory PDF Reports
2+
3+
*Angle: Bridging science and communication — CSVs aren't reports*
4+
5+
---
6+
7+
A CSV file is not a report. It's the raw material for one.
8+
9+
Most bioinformatics tools stop at the data file. You get a table of species names and identity percentages, and then it's up to you to turn that into something a client or regulator can read and act on. For a one-off analysis, fine. For a commercial eDNA service that needs to be traceable, auditable, and consistent, that gap is a real problem.
10+
11+
Phase 2 of my pipeline closes it.
12+
13+
`src/report.py` generates a four-section PDF directly from the pipeline outputs:
14+
15+
1. **Cover page** — site name, sample date, analyst, report date
16+
2. **Executive summary** — five key metric tiles (species richness, Shannon H', Pielou J', sequences analysed, unclassified count) plus a plain-English interpretation of the community structure
17+
3. **Species identification table** — one row per sequence, with the LCA-resolved species name, identity %, e-value, accession, and a colour-coded confidence flag
18+
4. **Methodology** — database version embedded, LCA thresholds explained, full sample metadata
19+
20+
The confidence flag behaviour is the part I'm most pleased with. Every row is automatically flagged HIGH, MEDIUM, or LOW based on identity threshold. LOW rows render with a red background — no analyst has to remember to go back and flag uncertain results before sending to a client. It's built into the output.
21+
22+
The report is designed around Environment Agency and Natural England eDNA guidance, so the structure and language should be familiar to anyone who reviews these surveys.
23+
24+
The hardest part wasn't the code. It was working out what a regulator actually needs to see — and what they don't.
25+
26+
Code: github.com/ybaah-biotech/eDNA-sequence-analysis
27+
28+
#eDNA #EnvironmentalConsulting #Bioinformatics #Python #EnvironmentAgency

0 commit comments

Comments
 (0)