1- # DOWNLINK
1+ # Downlink
22
33[ ![ CI] ( https://github.com/ma111e/downlink/actions/workflows/ci.yml/badge.svg )] ( https://github.com/ma111e/downlink/actions/workflows/ci.yml )
44[ ![ License: MIT] ( https://img.shields.io/badge/License-MIT-yellow.svg )] ( LICENSE )
55[ ![ Go Version] ( https://img.shields.io/badge/go-1.25-00ADD8.svg )] ( go.mod )
6+ [ ![ Docker Image] ( https://img.shields.io/github/v/tag/ma111e/downlink?logo=docker&logoColor=white&label=ghcr.io%2Fma111e%2Fdownlink&color=2496ED )] ( https://github.com/ma111e/downlink/pkgs/container/downlink )
67
7- A feed aggregator and content-analysis platform for security news. DOWNLINK
8+ A feed aggregator and content-analysis platform for security news. Downlink
89collects RSS/Atom feeds, scrapes full article content, runs each article through
910an LLM for rubric-based scoring and summarization, and assembles ranked digests
1011that it can publish to GitHub Pages and Discord.
1112
12- ```
13- feeds → scrape → LLM analysis & scoring → digest → publish (GitHub Pages / Discord)
14- ```
15-
1613## Features
1714
18- - ** Multi-provider LLMs** — Anthropic Claude, OpenAI , vLLM, Ollama, and llama.cpp
19- (any OpenAI-compatible endpoint). Configure several and pick one per run.
20- - ** Rubric scoring ** — articles are scored on six dimensions (specificity, severity,
15+ - ** Multi-provider LLMs** Claude, ChatGPT Codex, Mistral , vLLM, Ollama, and llama.cpp
16+ (any OpenAI-compatible endpoint)
17+ - ** Scoring ** articles are either vibe scored by the LLM or based on six dimensions (specificity, severity,
2118 breadth, novelty, actionability, credibility) rather than a single opaque number.
2219 See [ ` pkg/scoring ` ] ( pkg/scoring ) .
23- - ** Flexible scraping** — plain RSS, dynamic scraping (Lightpanda), and full-browser
24- scraping (Solimen) with per-feed CSS selectors, triggers, blacklists, and headers.
25- - ** Digest generation** — categorized, importance-ranked digests with deduplication.
26- - ** Publishing** — push static digests to GitHub Pages and notify a Discord webhook.
27- - ** Interfaces** — gRPC API (default ` :50051 ` ), an Atom feed export (` :65261 ` ), and the
20+ - ** Flexible scraping** plain RSS, dynamic scraping (Lightpanda), and full-browser
21+ scraping ([ Solimen] ( https://github.com/ma111e/solimen ) ) with per-feed CSS selectors, triggers, blacklists, and headers.
22+ - ** Digest generation** categorized, importance-ranked digests with deduplication.
23+ - ** Publishing** push static digests to GitHub Pages and notify a Discord webhook.
24+ - ** Interfaces** gRPC API (default ` :50051 ` ), an Atom feed export (` :65261 ` ), and the
2825 ` dlk ` command-line client.
29- - ** Storage** — embedded SQLite via GORM; no external database required.
3026
3127## Architecture
3228
3329| Component | Path | Description |
3430| ----------------- | ------------- | ------------------------------------------------------------------- |
35- | Server | ` cmd/server ` | gRPC services + feed manager + Atom feed export. The core daemon. |
36- | CLI (` dlk ` ) | ` cmd/dlk ` | gRPC client for articles, feeds, analysis, digests, config, queue. |
31+ | Server | ` cmd/server ` | gRPC services + feed manager + Atom feed export. |
32+ | CLI (` dlk ` ) | ` cmd/dlk ` | gRPC client for articles, feeds, analysis, digests, config, queue management . |
3733| Shared packages | ` pkg/ ` | Scoring, LLM gateway/providers, codex auth, models, protobufs. |
3834
39- The server exposes gRPC services for articles, analysis, feeds, digests, queue,
35+ The server exposes gRPC services for articles, analysis, feeds, digests, queue management ,
4036config, and auth. The Atom feed server publishes analyzed articles at
4137` http://localhost:65261 ` .
4238
@@ -45,16 +41,16 @@ config, and auth. The Atom feed server publishes analyzed articles at
4541With Go 1.25+:
4642
4743``` sh
48- go install github.com/ma111e/downlink/cmd/dlk@latest # CLI
49- go install github.com/ma111e/downlink/cmd/server@latest # server
44+ go install github.com/ma111e/downlink/cmd/dlk@latest # CLI
45+ go install github.com/ma111e/downlink/cmd/server@latest # server
5046```
5147
5248Or build from source:
5349
5450``` sh
55- make all # builds server + cli
56- make server # build just the server (embeds the git commit)
57- make cli # build just dlk
51+ make all # builds server + cli
52+ make server # build just the server
53+ make cli # build just dlk
5854```
5955
6056Or with Docker:
@@ -70,59 +66,122 @@ docker run --rm -p 50051:50051 -p 65261:65261 \
7066A ` docker-compose.yml ` is provided that also wires up the optional Solimen
7167full-browser scraper.
7268
73- ## Configuration
69+ ## Quickstart
7470
75- DOWNLINK reads configuration from three sources. Copy the bundled examples and
76- fill in your values:
71+ ** 1. Build the binaries** (see [ Install] ( #install ) ):
7772
7873``` sh
79- cp config.example.json config.json # LLM providers, analysis, notifications
80- cp feeds.example.yml feeds.yml # feed sources and per-feed scraping rules
81- cp .env.example .env # runtime/env overrides
74+ make all # produces ./server and ./dlk
8275```
8376
84- - ** ` config.json ` ** — LLM provider definitions (name, type, model, base URL, API key),
85- the active analysis provider and persona/writing style, and notification settings
86- (Discord webhook, GitHub Pages repo/token/branch). See
87- [ config.example.json] ( config.example.json ) .
88- - ** ` feeds.yml ` ** — the list of feeds with per-feed scraping strategy, CSS selectors,
89- triggers, blacklists, and custom HTTP headers. See [ feeds.example.yml] ( feeds.example.yml ) .
90- - ** ` .env ` / environment** — every server flag has a ` DOWNLINK_* ` env var equivalent
91- (loaded via Viper). See [ .env.example] ( .env.example ) . Precedence: CLI flag → env/` .env `
92- → ` config.json ` → default.
77+ ** 2. Create your config** and enable at least one LLM provider (analysis needs one;
78+ fetching feeds does not):
79+
80+ ``` sh
81+ cp config.example.json config.json
82+ $EDITOR config.json # enable a provider + fill in its api_key
83+ ```
9384
94- > ** Secrets:** ` config.json ` , ` .env ` , and ` feeds.yml ` are gitignored. Never commit
95- > real API keys, tokens, or webhook URLs.
85+ ** 3. Write a demo ` feeds.yml ` ** with a few public security feeds:
86+
87+ ``` yaml
88+ feeds :
89+ - url : https://cert.gov.ua/api/articles/rss
90+ title : Cert-UA
91+ type : rss
92+ enabled : true
93+ scraping : dynamic # "dynamic" (Lightpanda) or "full_browser" (Solimen); omit for static RSS
94+ scraper :
95+ article : div.article-item__content # CSS selector for the article body
96+ - url : https://www.bleepingcomputer.com/feed/
97+ title : Bleeping Computer
98+ type : rss
99+ enabled : true
100+ scraper :
101+ triggers :
102+ loaded :
103+ - article .article_section
104+ scraping : full_browser
105+ selectors :
106+ article : div.articleBody
107+ - url : https://feeds.feedburner.com/TheHackersNews
108+ title : The Hacker News
109+ type : rss
110+ enabled : true
111+ scraper :
112+ triggers :
113+ loaded :
114+ - div.main-box
115+ scraping : full_browser
116+ selectors :
117+ article : ' #articlebody'
118+ cutoff : .stophere
96119
97- ## Running
120+ ```
121+
122+ > The quickstart mixes scraping modes: ` scraping: dynamic ` needs ** Lightpanda** and
123+ > ` scraping: full_browser ` needs ** Solimen** . The ` --auto-start-* ` flags in step 4
124+ > launch both in Docker. For a no-dependencies first run, drop the
125+ > ` scraping ` /` scraper ` /` selectors ` keys to use plain RSS.
98126
99- Start the server:
127+ ** 4. Start the server** (in one terminal). The quickstart feeds use both scrapers, so start
128+ them too (requires Docker):
100129
101130``` sh
102- ./server --refresh # fetch all feeds on startup
103- ./server --port 50051 --log-level info
131+ ./server --auto-start-lightpanda --auto-start-solimen
104132```
105133
106- Use the CLI against a running server:
134+ ** 5. Apply the feeds.** ` dlk feed apply ` reconciles the database to match the file —
135+ feeds in the file are created or updated, and feeds no longer listed are disabled
136+ (their articles are kept). Preview first with ` --dry-run ` :
107137
108138``` sh
109- dlk feed list
110- dlk article list
111- dlk analysis run
112- dlk digest generate
113- dlk --help
139+ ./dlk feed apply -f feeds.yml --dry-run # show what would change
140+ ./dlk feed apply -f feeds.yml # apply it
114141```
115142
116- Check versions:
143+ ** 6. Fetch and generate a digest.** ` digest generate ` analyzes any not-yet-scored
144+ articles with your LLM provider, then assembles the ranked digest:
117145
118146``` sh
119- ./server --version
120- dlk --version
147+ ./dlk feed refresh all # pull the latest articles
148+ ./ dlk digest generate # analyze + assemble the ranked digest
121149```
122150
151+ ** 7. View the result:**
152+
153+ ``` sh
154+ ./dlk digest list # list generated digests
155+ ./dlk digest get # pick one and view it (add --markdown for prose)
156+ ```
157+
158+ ` ./dlk feed export -f feeds.yml ` does the reverse of step 5: it writes the feeds
159+ currently in the database back out to a YAML file.
160+
161+ ## Configuration
162+
163+ Downlink reads configuration from three sources. Copy the bundled examples and
164+ fill in your values:
165+
166+ ``` sh
167+ cp config.example.json config.json # LLM providers, analysis, notifications
168+ cp feeds.example.yml feeds.yml # feed sources and per-feed scraping rules
169+ cp .env.example .env # runtime/env overrides
170+ ```
171+
172+ - ** ` config.json ` ** LLM provider definitions (name, type, model, base URL, API key),
173+ the active analysis provider and persona/writing style, and notification settings
174+ (Discord webhook, GitHub Pages repo/token/branch). See
175+ [ config.example.json] ( config.example.json ) .
176+ - ** ` feeds.yml ` ** the list of feeds with per-feed scraping strategy, CSS selectors,
177+ triggers, blacklists, and custom HTTP headers. See [ feeds.example.yml] ( feeds.example.yml ) .
178+ - ** ` .env ` ** every server flag has a ` DOWNLINK_* ` env var equivalent
179+ (loaded automatically via Viper). See [ .env.example] ( .env.example ) . Precedence: CLI flag -> env variables/` .env `
180+ -> ` config.json ` -> default.
181+
123182## Publishing digests
124183
125- DOWNLINK can publish generated digests to GitHub Pages and announce them on Discord.
184+ Downlink can publish generated digests to GitHub Pages and announce them on Discord.
126185See [ docs/github-pages.md] ( docs/github-pages.md ) for the full setup, including
127186` --init-gh-pages ` and the required token scopes.
128187
0 commit comments