Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CURRENT_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,12 @@ Complete the latest GitHub API ranking in all five languages, including the unre
- [x] Reuse the existing project title image and keep the facts table readable on narrow screens.
- [x] Check generated site consistency in the repository verifier.
- [ ] Publish an independent draft PR for review; do not merge automatically.

## Star-farm filter (2026-11-14)

- [x] Find how the `<account>/ai--manus` flood reached the AI and Web new-repo boards (API samples, history replay).
- [x] Drop clone clusters, fake-watcher and zero-fork farms before ranking; flag borderline signals; blocklist on every board (`data/anti_abuse.json`).
- [x] Log dropped repos in `stats.json`, never in a README; tests in `tests/test_antiabuse.py`.
- [x] Replay the rules on every committed board and on a live discovery run.
- [ ] Publish through a pull request; merge only on the owner's word. The next daily run regenerates the boards.

3 changes: 2 additions & 1 deletion TODO_BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
## Now

- [x] Verified delayed automatic update and same-day duplicate skip on 2026-11-08 (runs 42487291739 and 42490659376); another real update succeeded on 2026-11-09 (42659639717)
- [ ] Developer Tools new-repo board: 15 of 20 entries were flagged ⚠ in the last run because star-farmed repos use `cli` / `developer-tools`; decide whether to add a quality signal (owner call)
- [x] Star-farm filter for new-repo boards (`tools/antiabuse.py`, 2026-11-14): clone floods / fake watchers / zero-fork farms are dropped
- [ ] Owner call: also drop (not just flag) single cracked / pirated repos
- [ ] Design and Data new-repo boards are thin (0-2 entries): widen with specific topics only (no generic ones; `tests/test_spam.py` guards that)

## Next
Expand Down
45 changes: 45 additions & 0 deletions data/anti_abuse.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
{
"_comment": "Anti star-farming rules for the new-repo boards (tools/antiabuse.py, docs/ANTI_ABUSE.md). Thresholds were checked against every new-repo board in rankings/ (history replay). 'blocklist' applies to every board.",
"clone_clusters": {
"enabled": true,
"_doc": "Repos from different owners with the same normalized repo name, or the same normalized description, form a cluster. Same description (>= min_desc_chars) across >= desc_min_owners owners, or same name across >= name_min_owners owners, is a clone cluster: every member is dropped, except the earliest member whose owner is credible (when one exists).",
"name_min_owners": 3,
"desc_min_owners": 2,
"name_and_desc_min_owners": 2,
"min_desc_chars": 24,
"keep_check_earliest": 3
},
"credible_owner": {
"_doc": "Only used to keep the original of a clone cluster.",
"min_account_days": 365,
"min_followers": 50
},
"new_owner": {
"_doc": "A user account younger than max_account_days when the repo was created, with fewer than max_followers followers. Alone: flagged and ranked last. Together with low_fork: dropped.",
"max_account_days": 60,
"max_followers": 5
},
"low_fork": {
"_doc": "stars / (forks + 1). >= flag_ratio (and stars >= min_stars): flagged and ranked last. >= drop_ratio: dropped.",
"min_stars": 150,
"flag_ratio": 100,
"drop_ratio": 500
},
"watch_farm": {
"_doc": "subscribers_count (watchers) >= share x stars: dropped. Real new repos stay under ~10 %.",
"min_stars": 100,
"share": 0.4
},
"fork_farm": {
"_doc": "forks >= share x stars: flagged and ranked last.",
"min_stars": 100,
"share": 0.9
},
"owner_lookup_budget": 150,
"blocklist": {
"_doc": "Case-insensitive. repos: owner/name. owners: account logins. names: repo-name regexes (full match). Applies to every board.",
"repos": [],
"owners": [],
"names": ["ai--manus"]
}
}
57 changes: 57 additions & 0 deletions docs/ANTI_ABUSE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Star-farm filter

The "hottest new repos (7 days)" boards are found by GitHub search, so they are the boards a star farm can
reach. On 2026-11-14 the AI and Web boards were flooded by one farm: hundreds of throwaway accounts each
published the same repo, `<account>/ai--manus` ("AI Manus is a general-purpose AI Agent system ..."), all
created within minutes on 2026-11-13, ~270 stars each, 0 forks, owner accounts one month old with 0 followers.
GitHub search returned 547 copies.

Code: `tools/antiabuse.py`. Thresholds: `data/anti_abuse.json` (edit there, not in code).
Tests: `tests/test_antiabuse.py`. Only fields the GitHub API already returns are used
(search, `GET /repos/{r}`, `GET /users/{u}`); nothing is estimated and nothing is sent anywhere.

## Rules

Screened **before** the `GET /repos` refresh budget is spent, so a flood cannot push real repos out:

| Rule | Condition (defaults) | Result |
|---|---|---|
| Blocklist | `blocklist.repos` / `owners` / `names` (regex, full match, case-insensitive) | dropped, on **every** board |
| Clone cluster | same normalized description (>= 24 letters) under >= 2 owners, or same name **and** description under >= 2 owners, or same normalized name under >= 3 owners | every member dropped, except the earliest member whose owner is credible (account >= 365 days and >= 50 followers); none credible -> all dropped |
| Stars nobody forks (extreme) | stars >= 150 and stars / (forks + 1) >= 500 | dropped |

After the refresh (needs `subscribers_count` and the owner account):

| Rule | Condition (defaults) | Result |
|---|---|---|
| Fake watchers | watchers (`subscribers_count`) >= 40 % of stars, stars >= 100 | dropped |
| Fresh throwaway owner + low forks | user account < 60 days old at repo creation, < 5 followers, and stars / (forks + 1) >= 100 | dropped |
| Low forks | stars >= 150 and stars / (forks + 1) >= 100 | ⚠ kept, ranked last, never a leader |
| Fresh throwaway owner | user account < 60 days old at repo creation, < 5 followers | ⚠ kept, ranked last, never a leader |
| Fork farm | forks >= 90 % of stars, stars >= 100 | ⚠ kept, ranked last, never a leader |

- A cluster or a dropped repo seen in one category is dropped from the later categories too.
- Owner lookups are cached and capped (`owner_lookup_budget`, default 150 per run). An owner that cannot be
read never triggers an owner rule (fail open), and a missing `forks_count` never triggers a fork rule.
- Tracked projects (week / month / year / total boards) are hand-picked in `data/projects.json`; only the
blocklist applies to them (`blocked` in `stats.json`, never ranked, still fetched).

## Report

Dropped repos never reach a README, `latest/` or a ranking page. Each run records them in
`data/stats.json` (and the history snapshot) under `newcomers.categories.<id>.anti_abuse`:
`dropped_count`, `by_reason`, `clusters` (key, size, owners, kept), `bulk_sample` (20 names from clusters and
the blocklist) and `dropped` (every other dropped repo with its reason, up to 200). `newcomers.anti_abuse` has the run totals and the number of owner lookups. The update log prints the
same summary.

## Calibration

The thresholds were replayed against every new-repo board committed under `rankings/` (408 repos, re-read
from the API on 2026-11-14): 74 repos would have been dropped (38 `ai--manus` copies, 26 repos with ~1,100 or
~300 stars and 0-1 forks, 4 fake-watcher repos, ...), none of them a repo with normal fork / watcher numbers.
8 repos would only be flagged ⚠, see the pull request for the list. Real hot newcomers (for example
`FrigadeHQ/yogurt`, `opencoredev/islandcn`, `alexeyw/lot-map`, `HolyArrs/HolyArrs`) have forks at 5-15 % of
stars and watchers under 10 %, far from every threshold.

To silence a farm by hand, add it to `blocklist` in `data/anti_abuse.json`. To relax a rule, change its
threshold there and run `python3 tools/verify.py`.
4 changes: 3 additions & 1 deletion docs/CATEGORIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ Rules (`tools/spam.py`, `suspicious_reason`), matched on repo name, description
4. A commercial software name in the repo name (Adobe, Photoshop, Canva, Autodesk, AutoCAD, Cinema 4D, Microsoft Office, Wondershare, ...) unless the owner is the vendor.
5. A `-Pro` / `-Premium` / year-suffixed name with no description and no language.

This is a heuristic, not a verdict: a flagged repo is only moved down, never hidden. `tests/test_spam.py` pins the rules, the ordering, the leader exclusion and that the scheduled entry points use the same code.
This is a heuristic, not a verdict: a flagged repo is only moved down, never hidden.
Clear star farms (clone floods such as hundreds of `<account>/ai--manus`, fake watchers, stars nobody forks) are
a different case: they are **dropped** by the star-farm filter, see [ANTI_ABUSE.md](ANTI_ABUSE.md). `tests/test_spam.py` pins the rules, the ordering, the leader exclusion and that the scheduled entry points use the same code.

## Per-category boards

Expand Down
9 changes: 6 additions & 3 deletions docs/NEW_ARRIVALS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,12 @@ Suspected cracked / pirated / activator repos are listed last and never lead a c

1. Created within the last 7 days (exact timestamp check on `created_at`).
2. At least 100 stars, not a fork, not archived, not private.
3. The best 30 candidates by search star count (per category) are re-read with `GET /repos/{owner}/{repo}`.
4. **Hotness = stars ÷ max(age in days, 1).**
5. Ranked by hotness (ties: more stars, then name), top 20 shown per category, with suspected cracked / pirated repos after all normal ones.
3. Star farms are dropped before and after the refresh: clone clusters, blocklist, stars nobody forks, fake
watchers, fresh throwaway owners ([ANTI_ABUSE.md](ANTI_ABUSE.md)). Dropped repos are logged under
`newcomers.categories.<id>.anti_abuse` and never shown.
4. The best 30 remaining candidates by search star count (per category) are re-read with `GET /repos/{owner}/{repo}`.
5. **Hotness = stars ÷ max(age in days, 1).**
6. Ranked by hotness (ties: more stars, then name), top 20 shown per category, with suspected cracked / pirated repos after all normal ones.

If every search query for a category fails, that category's new-repo board is shown as missing. If only some
queries fail, the README says so. Search is index-based and can lag by minutes.
Expand Down
5 changes: 5 additions & 0 deletions docs/project-governance/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# CHANGELOG.md

## 2026-11-14 (star-farm filter)

- New-repo boards drop star farms: clone floods (the same repo under many throwaway accounts, e.g. `<account>/ai--manus`), fake watchers, stars nobody forks, fresh throwaway owners. Borderline cases are kept with ⚠ and ranked last. Rules: `docs/ANTI_ABUSE.md`, thresholds `data/anti_abuse.json`.
- A blocklist in `data/anti_abuse.json` applies to every board. Dropped repos are logged in `stats.json`, never shown.

## 2026-11-07 (public first screen)

- README first screen now states the value, the live-board URL, and three quick-use steps before the category leaders.
Expand Down
9 changes: 9 additions & 0 deletions docs/project-governance/DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,12 @@
- The raw-data refresh stays fail-safe and independent of translation availability. The local translation job is complete only when both its cache-aware description and hook audits report zero gaps across current categories.
- Valid partial output is retained and published, but remaining gaps return status 4 and are retried on the next existing scheduled invocation. Audit failures cannot be interpreted as zero work. One invocation makes one bounded generation pass; a push race rechecks the new ranking without opening an unbounded model loop.
- Existing detailed descriptions, translations and valid hooks are preserved. This completion also fills the previous 98 missing hook fields for reuse without rewriting past numeric snapshots. No model/backend, credentials, installed schedule or ranking policy changes are part of this fix.

## 2026-11-14

### Star farms are dropped, not just flagged

- Owner request: the boards must not be gamed. Clear farms (clone clusters, fake watchers, >= 500 stars per fork, fresh throwaway owner + low forks, blocklist) are removed from the new-repo boards and logged in `stats.json`; borderline signals (low forks alone, fresh owner alone, fork farm) keep the existing ⚠ / rank-last treatment so a real repo is never hidden by one weak signal.
- Single cracked / pirated repos keep the 2026-10-30 decision (flag, rank last) unless a farm rule drops them.
- Thresholds live in `data/anti_abuse.json`, calibrated by replaying every committed board (`docs/ANTI_ABUSE.md`).

7 changes: 7 additions & 0 deletions docs/project-governance/WORKLOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# WORKLOG.md

## 2026-11-14

- The 2026-11-14 AI board had 19 of 20 entries and the Web board 19 of 20 from one farm: `<account>/ai--manus`, same description, created minutes apart on 2026-11-13, ~270 stars, 0 forks, owners ~30 days old with 0 followers (search finds 547 copies). Root cause: discovery ranked any non-fork repo with >= 100 stars by stars/day and had no rule for identical repos under different owners; the flood also used the whole refresh budget.
- Added `tools/antiabuse.py` + `data/anti_abuse.json` (clone clusters, blocklist, low-fork, fake watchers, fresh owners), wired into `newcomers.discover_category` before and after the refresh, report under `anti_abuse`; `risers_core.rank` skips `blocked` tracked projects.
- Replay over every committed new-repo board (408 repos re-read from the API): 74 dropped, all farm-shaped; 8 flagged only.
- Verification: `python3 tools/verify.py` green; `tests/test_antiabuse.py` added.

## 2026-11-06

- PLAN-005: production cadence is once daily at America/Los_Angeles 00:00. Dropped the Beijing cron (`0 16 * * *`) and the Beijing launchd agent. Scheduled Actions runs always use slot `pt`.
Expand Down
Loading
Loading