Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
23e1115
feat(data): introduce safe incremental evidence-based stream probes
SamoTech Nov 19, 2026
fcd2f61
fix(data): quarantine large incomplete rebuilds and sample verified a…
SamoTech Nov 19, 2026
f5aad81
test(data): cover audio evidence, redirects, SSRF, retries and catalo…
SamoTech Nov 19, 2026
11712f6
docs(data): record source audit baseline and evidence limitations
SamoTech Nov 19, 2026
88ef24f
docs(data): synchronize roadmap and readme with source-health safeguards
SamoTech Nov 19, 2026
4659077
docs(data): synchronize roadmap and readme with source-health safeguards
SamoTech Nov 19, 2026
e0a8426
feat(data): add read-only worldwide coverage and integrity audit
SamoTech Nov 19, 2026
be0f7da
fix(data): use atomic JSON snapshot replacement to keep last good files
SamoTech Nov 19, 2026
576f01e
fix(data): preserve stream-health last known good report atomically
SamoTech Nov 19, 2026
dc0beac
test(data): measure entire global catalog and atomic rollback in CI
SamoTech Nov 19, 2026
b7ec928
fix(data): keep fallback catalog, head and delta manifest in sync
SamoTech Nov 19, 2026
f6bbf8a
test(data): acknowledge existing count drift and verify fallback snap…
SamoTech Nov 19, 2026
caa3c2a
docs(data): disclose real snapshot count discrepancy from full CI audit
SamoTech Nov 19, 2026
56c7880
docs(data): disclose real snapshot count discrepancy from full CI audit
SamoTech Nov 19, 2026
a723eb1
docs(data): disclose real snapshot count discrepancy from full CI audit
SamoTech Nov 19, 2026
c96a6eb
feat(data): back off failed stream probes exponentially across daily …
SamoTech Nov 19, 2026
19d1895
test(data): check exponential probe backoff across daily refreshes
SamoTech Nov 19, 2026
d8ba135
docs(data): capture retry backoff and synchronized snapshot publication
SamoTech Nov 19, 2026
bde96ed
docs(data): capture retry backoff and synchronized snapshot publication
SamoTech Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@

### 📻 محطات البث

- فحص جودة جديد تدريجي: data/stream_quality.json (48 رابطًا في دورة تحديث المحطات)، يفرق بين دليل صوت حقيقي وبين رد HTTP ناجح فقط أو ملف HLS. فحوص stream_checks.json القديمة ليست تأكيد تشغيل بالمتصفح. تُعزل اللقطات الناقصة جدًا من دون حذف محطات. [تقرير تدقيق المصادر](docs/RADIO_SOURCES_AUDIT_2026-11-19.md).


- كتالوج كامل من Radio-Browser (كل المحطات غير المكسورة (أكثر من 50,000)، ترتيب: سمعية/عربية/IPRD + الكتالوج الكامل) + في البحث المباشر في FM tab أيضا
- بحث فوري في الكتالوج الكامل (~13MB) بعد التحميل الأول — مع ملف إقلاع خفيف stations_head (~530KB) يعرض أول 2500 محطة فورًا، وزياراتك التالية تُحدَّث تفاضليًا بكيلوبايتات عبر IndexedDB
- فلترة حسب الدولة/اللغة/الفئة (FACET_OPTIONS + filterFacetOptions(kind) — يحتفظ بالخيار المحدد أثناء التصفية)
Expand Down
2 changes: 2 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

---

> **صيانة المصادر العالمية — 2026-11-19:** خط الأساس المعلن في manifest v34 هو 60,732 محطة، بينما صفوف stations.json الفعلية 60,741 (انحراف سابق +9)؛ آخر سجل قديم فحص 1,185 رابطًا وصنّف 1,071 alive (مش ضمان تشغيل صوت). أضفنا فاحص دليل الصوت التدريجي (48 رابطًا في دورة المحطات)، وحماية تمنع فقد أكثر من 20% من المحطات بسبب جلب ناقص. لا حذف للمحطات ولا تغيير لمعرّفاتها. التفاصيل والتقييم في [تقرير تدقيق المصادر](docs/RADIO_SOURCES_AUDIT_2026-11-19.md). اكتمال التفعيل معلق على CI والنشر ودورة صيانة فعلية.

## 1) الوضع الحالي — نوفمبر 2026

### ✅ منجز (سجل مختصر — السجل الكامل في PRs المستودع)
Expand Down
82 changes: 82 additions & 0 deletions docs/RADIO_SOURCES_AUDIT_2026-11-19.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Radio Atlas — Global radio source audit (2026-11-19)

Baseline from repository tracked assets at 17:35:19 UTC.

| Metric | Before | Meaning |
| --- | ---: | --- |
| Catalog declared count | 60,732 | data/stations.manifest.json v34 AND stations.json count field |
| Actual station rows in stations.json | 60,741 | full JSON array parsed in CI 2026-11-19; mismatch of +9 rows, existing before PR |
| Historical stream checks | 1,185 | data/stream_checks.json |
| Historical results classified alive | 1,071 (90.4%) | HTTP probe, NOT verified browser playback |
| Historical results not classified alive | 114 (9.6%) | Includes blocks, no proof these are broken |
| Official manual catalog | 26 | scripts/official_stations.json |
| Arabic manual catalog | 106 | scripts/arabic_stations.json |
| Manual catalogs with missing language | 0 | 26 + 106 entries |
| Manual catalog duplicate URLs (within each file) | 0 | Case-insensitive literal URL comparison |
| Manual catalog HTTP-only URLs | 0 | 26 + 106 entries |
| Manual catalogs without explicit logo field | 132 | UI fallback exists |
| Global verified playable stations | NOT MEASURED | Historic alive is not audio proof |
| Global country/language/genre completeness | NOT MEASURED | Requires full catalog aggregation |
| Global duplicate broadcaster identities | NOT MEASURED | Same name does not mean same broadcast |
| Before/after real playback success | NOT MEASURED | Needs browser trials |

## Existing station source registry

- Official broadcaster endpoints in the hand-curated official registry.
- Arabic curated stations and IPRD Arab-country country playlists.
- Radio Browser API with five server mirrors, pagination, source failover.
- IPRD country playlists globally (excluding forbidden country).
- Community playlists by junguler (unstructured stream discovery).
- Broadcaster portal scraper with seed inventory; Maspero official video pages
use Dailymotion and cannot be interpreted as direct audio.
- eyalev/verified-radio-streams community directory.
- Icecast YP public index.
- Live Radio Browser search (client side; separate from daily snapshot).

These are existing providers. This PR does not add an unvalidated provider.

## Root causes found

- Historic HTTP-based checks can classify 200 and certain access errors
as alive without actual sound bytes.
- 1,185 historic checks are just a small curated/head subset.
- The stored count fields differ from the actual stations array by 9 rows (60,732 vs 60,741) at this baseline. Both normal refresh and fallback now use one snapshot publication function that recalculates count and synchronizes head and delta manifest; the persisted historical data remains unchanged until the next successful station refresh.
- The old absolute minimum of 10,000 catalog records would still permit a
severe incomplete rebuild from 60k to 40k.
- Station identity is URL-based; collapsing records by broadcaster name would
break favorites, history and deep links, and could merge unrelated stations.
- Regional/datacenter IP blocking requires uncertain statuses and repeated
observations; one failed network check is not evidence of a broken stream.
- Official/Arabic curated stations have no explicit logo field, but UI
supplies flags and fallback artwork.

## Improvements in this PR

- Safe sampled stream-evidence verification, maximum 48 distinct URLs per
station refresh with curated/failure priorities plus rotating world discovery.
- Bounded 2 KiB sample, 6-second timeout, safe DNS and redirects, one playlist hop.
- Signatures for MP3, AAC, OGG, WAV, FLAC, audio MP4; HLS manifest structural
checks alone are labelled DEGRADED (not verified browser playback).
- Separate 0-100 score and five internal statuses: VERIFIED, DEGRADED,
UNVERIFIED, TEMPORARILY_UNAVAILABLE, BROKEN.
- Exponential retry backoff for failed streams (1, 2, 4, max 7 days).
- No delete-on-failure; 404/410 needs three consecutive checked failures to
reach BROKEN; historical last success preserved.
- Observational sample stored separately in data/stream_quality.json.
- Atomically stage individual JSON files; normal and fallback station refreshes synchronize station, head, delta and manifest counts.
- Detect major incomplete source crawls: a new catalog over 20% smaller than
previous is quarantined before writing the new stations.json/head/deltas.
- Old URL-based station IDs and player schema remain unchanged.

## Limits and follow-up

- Sampling is incremental, not a random population sample. Do not generalize
sample success percentages to 60,732 stations.
- Existing stream_checks.json alive flag is unchanged and NOT the VERIFIED flag.
- HLS segment decode and real mobile/browser playback remain to test.
- URL DNS validation with stdlib has a DNS rebinding TOCTOU limitation.
- GitHub Actions CI and deployment must pass before release is called complete.
- Before/after country and language coverage must be computed from full
catalog during a separate audit job; do not invent coverage figures.
- No alternative stream is accepted without proof it belongs to the same
broadcaster and independent playback evidence.
114 changes: 114 additions & 0 deletions scripts/catalog_audit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""Offline, read-only catalog audit. No external requests or station deletion."""
import collections
import json
import re
import sys
import unicodedata
import urllib.parse
from pathlib import Path


def url_key(url):
try:
parts = urllib.parse.urlsplit(str(url or "").strip())
if parts.scheme.lower() not in ("http", "https") or not parts.hostname:
return ""
if parts.username or parts.password:
return ""
port = parts.port
host = parts.hostname.lower()
if port and port not in (80 if parts.scheme == "http" else 443,):
host += ":" + str(port)
return urllib.parse.urlunsplit((parts.scheme.lower(), host, parts.path or "/",
parts.query, ""))
except (TypeError, ValueError):
return ""


def normalize_name(name):
text = unicodedata.normalize("NFKC", str(name or "")).casefold()
return re.sub(r"\s+", " ", text).strip()


def audit_catalog(stations):
"""Counts metadata completeness, URL duplicates and POSSIBLE name collisions.

Different streams with the same name are NOT merged: country/name is only
a manual review candidate, not evidence of a duplicate station.
"""
countries = collections.Counter()
languages = collections.Counter()
tags = collections.Counter()
codec = collections.Counter()
names = collections.defaultdict(set)
known_urls = set()
duplicate_urls = 0
missing_country = missing_language = missing_logo = missing_name = http_only = 0
invalid_url = 0
for station in stations:
if not isinstance(station, dict):
continue
cc = str(station.get("cc") or "").strip().upper()
country = cc if len(cc) == 2 else "(missing)"
countries[country] += 1
if country == "(missing)":
missing_country += 1
langs = [part.strip().casefold() for part in
re.split(r"[,;|]", str(station.get("language") or "")) if part.strip()]
if not langs:
missing_language += 1
languages.update(set(langs))
tags.update(set(t.strip().casefold() for t in
str(station.get("tags") or "").split(",") if t.strip()))
codec[str(station.get("codec") or "").strip().upper() or "(missing)"] += 1
if not station.get("favicon") and not station.get("logo"):
missing_logo += 1
name = normalize_name(station.get("name"))
if not name or name == "—":
missing_name += 1
key = url_key(station.get("url"))
if not key:
invalid_url += 1
elif key in known_urls:
duplicate_urls += 1
known_urls.add(key)
if str(station.get("url") or "").lower().startswith("http://"):
http_only += 1
if name and name != "—" and key:
names[(country, name)].add(key)
potential_same_names = sum(1 for urls in names.values() if len(urls) > 1)
result = {
"total": len(stations),
"countries": len([c for c in countries if c != "(missing)"]),
"languages": len(languages),
"missing_country": missing_country,
"missing_language": missing_language,
"missing_logo_field": missing_logo,
"missing_name": missing_name,
"http_only": http_only,
"invalid_url": invalid_url,
"duplicate_stream_urls": duplicate_urls,
"potential_same_country_name_groups_not_merged": potential_same_names,
"country_counts": dict(countries.most_common()),
"language_counts": dict(languages.most_common()),
"codec_counts": dict(codec.most_common()),
"top_tags": dict(tags.most_common(40)),
}
return result


def main():
source = Path(sys.argv[1]) if len(sys.argv) > 1 else (
Path(__file__).resolve().parents[1] / "data" / "stations.json")
with source.open(encoding="utf-8") as handle:
payload = json.load(handle)
stations = payload.get("stations") if isinstance(payload, dict) else payload
if not isinstance(stations, list):
raise SystemExit("invalid catalog shape")
result = audit_catalog(stations)
print(json.dumps(result, indent=2, ensure_ascii=False))


if __name__ == "__main__":
main()
Loading
Loading