Skip to content

Commit 2afbb35

Browse files
author
ma111e
committed
docs: add Quickstart guide and fix README accuracy
- Add a step-by-step Quickstart (build, config, demo feeds, apply, refresh, generate, view) with the feed apply/reconcile mechanism. - Add Mistral to the supported providers list. - Fix the feed scraping field (top-level 'scraping', not a 'dynamic_scraping' scraper key) and the DOWNLINK_* env prefix.
1 parent cdcef60 commit 2afbb35

1 file changed

Lines changed: 113 additions & 54 deletions

File tree

README.md

Lines changed: 113 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,42 +1,38 @@
1-
# DOWNLINK
1+
# Downlink
22

33
[![CI](https://github.com/ma111e/downlink/actions/workflows/ci.yml/badge.svg)](https://github.com/ma111e/downlink/actions/workflows/ci.yml)
44
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
55
[![Go Version](https://img.shields.io/badge/go-1.25-00ADD8.svg)](go.mod)
6+
[![Docker Image](https://img.shields.io/github/v/tag/ma111e/downlink?logo=docker&logoColor=white&label=ghcr.io%2Fma111e%2Fdownlink&color=2496ED)](https://github.com/ma111e/downlink/pkgs/container/downlink)
67

7-
A feed aggregator and content-analysis platform for security news. DOWNLINK
8+
A feed aggregator and content-analysis platform for security news. Downlink
89
collects RSS/Atom feeds, scrapes full article content, runs each article through
910
an LLM for rubric-based scoring and summarization, and assembles ranked digests
1011
that it can publish to GitHub Pages and Discord.
1112

12-
```
13-
feeds → scrape → LLM analysis & scoring → digest → publish (GitHub Pages / Discord)
14-
```
15-
1613
## Features
1714

18-
- **Multi-provider LLMs** — Anthropic Claude, OpenAI, vLLM, Ollama, and llama.cpp
19-
(any OpenAI-compatible endpoint). Configure several and pick one per run.
20-
- **Rubric scoring** articles are scored on six dimensions (specificity, severity,
15+
- **Multi-provider LLMs** Claude, ChatGPT Codex, Mistral, vLLM, Ollama, and llama.cpp
16+
(any OpenAI-compatible endpoint)
17+
- **Scoring** articles are either vibe scored by the LLM or based on six dimensions (specificity, severity,
2118
breadth, novelty, actionability, credibility) rather than a single opaque number.
2219
See [`pkg/scoring`](pkg/scoring).
23-
- **Flexible scraping** plain RSS, dynamic scraping (Lightpanda), and full-browser
24-
scraping (Solimen) with per-feed CSS selectors, triggers, blacklists, and headers.
25-
- **Digest generation** categorized, importance-ranked digests with deduplication.
26-
- **Publishing** push static digests to GitHub Pages and notify a Discord webhook.
27-
- **Interfaces** gRPC API (default `:50051`), an Atom feed export (`:65261`), and the
20+
- **Flexible scraping** plain RSS, dynamic scraping (Lightpanda), and full-browser
21+
scraping ([Solimen](https://github.com/ma111e/solimen)) with per-feed CSS selectors, triggers, blacklists, and headers.
22+
- **Digest generation** categorized, importance-ranked digests with deduplication.
23+
- **Publishing** push static digests to GitHub Pages and notify a Discord webhook.
24+
- **Interfaces** gRPC API (default `:50051`), an Atom feed export (`:65261`), and the
2825
`dlk` command-line client.
29-
- **Storage** — embedded SQLite via GORM; no external database required.
3026

3127
## Architecture
3228

3329
| Component | Path | Description |
3430
| ----------------- | ------------- | ------------------------------------------------------------------- |
35-
| Server | `cmd/server` | gRPC services + feed manager + Atom feed export. The core daemon. |
36-
| CLI (`dlk`) | `cmd/dlk` | gRPC client for articles, feeds, analysis, digests, config, queue. |
31+
| Server | `cmd/server` | gRPC services + feed manager + Atom feed export. |
32+
| CLI (`dlk`) | `cmd/dlk` | gRPC client for articles, feeds, analysis, digests, config, queue management. |
3733
| Shared packages | `pkg/` | Scoring, LLM gateway/providers, codex auth, models, protobufs. |
3834

39-
The server exposes gRPC services for articles, analysis, feeds, digests, queue,
35+
The server exposes gRPC services for articles, analysis, feeds, digests, queue management,
4036
config, and auth. The Atom feed server publishes analyzed articles at
4137
`http://localhost:65261`.
4238

@@ -45,16 +41,16 @@ config, and auth. The Atom feed server publishes analyzed articles at
4541
With Go 1.25+:
4642

4743
```sh
48-
go install github.com/ma111e/downlink/cmd/dlk@latest # CLI
49-
go install github.com/ma111e/downlink/cmd/server@latest # server
44+
go install github.com/ma111e/downlink/cmd/dlk@latest # CLI
45+
go install github.com/ma111e/downlink/cmd/server@latest # server
5046
```
5147

5248
Or build from source:
5349

5450
```sh
55-
make all # builds server + cli
56-
make server # build just the server (embeds the git commit)
57-
make cli # build just dlk
51+
make all # builds server + cli
52+
make server # build just the server
53+
make cli # build just dlk
5854
```
5955

6056
Or with Docker:
@@ -70,59 +66,122 @@ docker run --rm -p 50051:50051 -p 65261:65261 \
7066
A `docker-compose.yml` is provided that also wires up the optional Solimen
7167
full-browser scraper.
7268

73-
## Configuration
69+
## Quickstart
7470

75-
DOWNLINK reads configuration from three sources. Copy the bundled examples and
76-
fill in your values:
71+
**1. Build the binaries** (see [Install](#install)):
7772

7873
```sh
79-
cp config.example.json config.json # LLM providers, analysis, notifications
80-
cp feeds.example.yml feeds.yml # feed sources and per-feed scraping rules
81-
cp .env.example .env # runtime/env overrides
74+
make all # produces ./server and ./dlk
8275
```
8376

84-
- **`config.json`** — LLM provider definitions (name, type, model, base URL, API key),
85-
the active analysis provider and persona/writing style, and notification settings
86-
(Discord webhook, GitHub Pages repo/token/branch). See
87-
[config.example.json](config.example.json).
88-
- **`feeds.yml`** — the list of feeds with per-feed scraping strategy, CSS selectors,
89-
triggers, blacklists, and custom HTTP headers. See [feeds.example.yml](feeds.example.yml).
90-
- **`.env` / environment** — every server flag has a `DOWNLINK_*` env var equivalent
91-
(loaded via Viper). See [.env.example](.env.example). Precedence: CLI flag → env/`.env`
92-
`config.json` → default.
77+
**2. Create your config** and enable at least one LLM provider (analysis needs one;
78+
fetching feeds does not):
79+
80+
```sh
81+
cp config.example.json config.json
82+
$EDITOR config.json # enable a provider + fill in its api_key
83+
```
9384

94-
> **Secrets:** `config.json`, `.env`, and `feeds.yml` are gitignored. Never commit
95-
> real API keys, tokens, or webhook URLs.
85+
**3. Write a demo `feeds.yml`** with a few public security feeds:
86+
87+
```yaml
88+
feeds:
89+
- url: https://cert.gov.ua/api/articles/rss
90+
title: Cert-UA
91+
type: rss
92+
enabled: true
93+
scraping: dynamic # "dynamic" (Lightpanda) or "full_browser" (Solimen); omit for static RSS
94+
scraper:
95+
article: div.article-item__content # CSS selector for the article body
96+
- url: https://www.bleepingcomputer.com/feed/
97+
title: Bleeping Computer
98+
type: rss
99+
enabled: true
100+
scraper:
101+
triggers:
102+
loaded:
103+
- article .article_section
104+
scraping: full_browser
105+
selectors:
106+
article: div.articleBody
107+
- url: https://feeds.feedburner.com/TheHackersNews
108+
title: The Hacker News
109+
type: rss
110+
enabled: true
111+
scraper:
112+
triggers:
113+
loaded:
114+
- div.main-box
115+
scraping: full_browser
116+
selectors:
117+
article: '#articlebody'
118+
cutoff: .stophere
96119

97-
## Running
120+
```
121+
122+
> The quickstart mixes scraping modes: `scraping: dynamic` needs **Lightpanda** and
123+
> `scraping: full_browser` needs **Solimen**. The `--auto-start-*` flags in step 4
124+
> launch both in Docker. For a no-dependencies first run, drop the
125+
> `scraping`/`scraper`/`selectors` keys to use plain RSS.
98126
99-
Start the server:
127+
**4. Start the server** (in one terminal). The quickstart feeds use both scrapers, so start
128+
them too (requires Docker):
100129

101130
```sh
102-
./server --refresh # fetch all feeds on startup
103-
./server --port 50051 --log-level info
131+
./server --auto-start-lightpanda --auto-start-solimen
104132
```
105133

106-
Use the CLI against a running server:
134+
**5. Apply the feeds.** `dlk feed apply` reconciles the database to match the file —
135+
feeds in the file are created or updated, and feeds no longer listed are disabled
136+
(their articles are kept). Preview first with `--dry-run`:
107137

108138
```sh
109-
dlk feed list
110-
dlk article list
111-
dlk analysis run
112-
dlk digest generate
113-
dlk --help
139+
./dlk feed apply -f feeds.yml --dry-run # show what would change
140+
./dlk feed apply -f feeds.yml # apply it
114141
```
115142

116-
Check versions:
143+
**6. Fetch and generate a digest.** `digest generate` analyzes any not-yet-scored
144+
articles with your LLM provider, then assembles the ranked digest:
117145

118146
```sh
119-
./server --version
120-
dlk --version
147+
./dlk feed refresh all # pull the latest articles
148+
./dlk digest generate # analyze + assemble the ranked digest
121149
```
122150

151+
**7. View the result:**
152+
153+
```sh
154+
./dlk digest list # list generated digests
155+
./dlk digest get # pick one and view it (add --markdown for prose)
156+
```
157+
158+
`./dlk feed export -f feeds.yml` does the reverse of step 5: it writes the feeds
159+
currently in the database back out to a YAML file.
160+
161+
## Configuration
162+
163+
Downlink reads configuration from three sources. Copy the bundled examples and
164+
fill in your values:
165+
166+
```sh
167+
cp config.example.json config.json # LLM providers, analysis, notifications
168+
cp feeds.example.yml feeds.yml # feed sources and per-feed scraping rules
169+
cp .env.example .env # runtime/env overrides
170+
```
171+
172+
- **`config.json`** LLM provider definitions (name, type, model, base URL, API key),
173+
the active analysis provider and persona/writing style, and notification settings
174+
(Discord webhook, GitHub Pages repo/token/branch). See
175+
[config.example.json](config.example.json).
176+
- **`feeds.yml`** the list of feeds with per-feed scraping strategy, CSS selectors,
177+
triggers, blacklists, and custom HTTP headers. See [feeds.example.yml](feeds.example.yml).
178+
- **`.env`** every server flag has a `DOWNLINK_*` env var equivalent
179+
(loaded automatically via Viper). See [.env.example](.env.example). Precedence: CLI flag -> env variables/`.env`
180+
-> `config.json` -> default.
181+
123182
## Publishing digests
124183

125-
DOWNLINK can publish generated digests to GitHub Pages and announce them on Discord.
184+
Downlink can publish generated digests to GitHub Pages and announce them on Discord.
126185
See [docs/github-pages.md](docs/github-pages.md) for the full setup, including
127186
`--init-gh-pages` and the required token scopes.
128187

0 commit comments

Comments
 (0)