fix(delphi): ns includes PASS votes (Clojure parity, pre-D10)

Bug
---
`compute_group_comment_stats_df` in
`delphi/polismath/pca_kmeans_rep/repness.py` computed `ns = na + nd` and
`total_votes = total_agree + total_disagree`, silently dropping PASS (0)
votes from every vote count.

Clojure reference: `math/src/polismath/math/repness.clj:56-61, :70`:

    (defn- count-votes [votes & [vote]]
      (let [filt-fn (if vote #(= vote %) identity)]
        (count (filter filt-fn votes))))
    ...
    :ns (fnk [votes] (count-votes votes))

`count-votes` with no `vote` argument uses `identity` as the filter
predicate. In Clojure 0 is truthy, so `(filter identity ...)` keeps every
non-nil entry — including PASS. Therefore Clojure
`ns = na + nd + np` (PASS count).

Every downstream metric that consumes `ns` — `pa`, `pd`, `pat`, `pdt`,
`ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, and the
upcoming D11 `consensus_stats_df` — was off whenever PASS votes existed.

Fix
---
Both `total_counts` and `group_counts` now compute the count column via
`('vote', 'size')` directly in the pandas groupby aggregation. The frame
is `dropna(subset=['vote'])`-filtered upstream, so `size()` counts
exactly the non-NaN rows — including PASS (0). This is the Clojure
`(count (filter identity votes))` recipe verbatim. Both sites carry a
comment citing the Clojure reference.

Why D5 BlobInjection didn't catch this
--------------------------------------
D5's blob-injection tests pull `(n-success, n-trials)` directly from the
Clojure blob's `repness` entries and feed them to `prop_test_vectorized`.
They bypass `compute_group_comment_stats_df` entirely, so anything
downstream of `ns = na + nd` was invisible. The same gap will recur for
every fix whose stage-inputs are built by Python code. Pure-formula
tests over a tiny vote matrix are the only RED path.

Tests added
-----------
`TestNsIncludesPassVotes` in `tests/test_repness_unit.py`:

- `test_ns_includes_pass_votes` — single comment, 2 agree / 1 disagree /
  2 pass → `na=2, nd=1, ns=5`.
- `test_ns_all_pass_column` — all-PASS column → `na=0, nd=0, ns=3`.
- `test_ns_mixed_with_nan_only_explicit_votes_count` — NaN never counts;
  PASS always does.
- `test_other_votes_includes_other_group_pass` — `other_votes` (the
  complement of group `ns`) also includes out-of-group PASS.

Suite delta
-----------
Pre: 295 passed, 12 skipped, 58 xfailed (PR 14a baseline).
Post: 299 passed, 12 skipped, 58 xfailed. Delta = +4 (the new tests).
No pre-existing test broke — the existing `TestVectorizedFunctions`
fixtures used only AGREE/DISAGREE/NaN and were never sensitive.

Cascade to D11
--------------
D11's `consensus_stats_df(vote_matrix_df)` (whole-conversation
counterpart) will mirror the same recipe at the conversation level. This
fix lands BEFORE D10 in the stack so D11's implementation, when it
arrives, starts from the corrected ns semantics. D11 should follow the
same pure-formula test pattern.

Goldens
-------
DEFERRED. Re-recording is gated on sklearn-KMeans-seeding consensus; no
golden values shift at the goldens commit until D10/D11/D12 land.

commit-id:0761e6c3

Author: jucor

delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md

@@ -1347,3 +1347,64 @@ PR 14a unblocks (in stack order):
    readability reference. Research agent produced a clean split proposal:
    `_build_group_comment_index` (plumbing) + `_compute_per_group_stats`
    (math) + 5-line orchestrator.
+
+## Session: ns-PASS fix (2026-06-11)
+
+**Context.** While preparing the D10/D11/D12 rework on top of PR 14a, a
+Clojure re-read surfaced a latent bug in `compute_group_comment_stats_df`:
+`ns` (and `total_votes`) were computed as `na + nd`, silently dropping PASS
+votes. Clojure's `:ns` is `(count-votes votes)` (math/repness.clj:56-61,
+:70), which calls `(filter identity votes)`. In Clojure 0 is truthy, so
+PASS (0) counts; only `nil` is filtered out. Therefore Clojure
+`ns = na + nd + np`, and every downstream metric (`pa`, `pd`, `pat`,
+`pdt`, `ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, plus
+D11's `consensus_stats_df` which will mirror the same recipe at the
+whole-conversation level) was off whenever PASS votes existed.
+
+**Why D5 BlobInjection didn't catch it.** D5's blob-injection tests pull
+`(n-success, n-trials)` straight from the Clojure blob's `repness`
+entries and feed them to `prop_test_vectorized`. They bypass
+`compute_group_comment_stats_df` entirely, so the bug downstream of
+`ns = na + nd` was invisible. The same gap will recur for D11 / D12 until
+we ship pure-formula tests that build a tiny vote matrix and assert on the
+counts. Lesson: blob-injection is necessary but not sufficient — every
+formula whose inputs are themselves computed by Python needs at least one
+pure-formula unit test that exercises the input-building code.
+
+**TDD cycle.**
+- **BASELINE** — full suite at PR 14a parent: 295 passed, 12 skipped,
+  58 xfailed.
+- **RED** — added `TestNsIncludesPassVotes` in `tests/test_repness_unit.py`
+  with four pure-formula tests: single-comment mixed AGREE/DISAGREE/PASS,
+  all-PASS column, NaN-vs-PASS distinction, two-group `other_votes`
+  including out-group PASS. All four failed on the buggy code (4 fails).
+- **GREEN** — in `polismath/pca_kmeans_rep/repness.py`:
+  - `total_counts` now computes `total_votes=('vote', 'size')` directly in
+    the groupby agg, instead of `total_agree + total_disagree`.
+  - `group_counts` now computes `ns=('vote', 'size')` directly, instead of
+    `na + nd`.
+  - `'size'` on the already-`dropna(subset=['vote'])`-filtered frame counts
+    exactly the non-NaN entries — including PASS (0). This is the
+    Clojure `(count (filter identity votes))` recipe verbatim.
+  - Both sites carry a comment citing repness.clj:56-61, :70 and the
+    truthy-0 reasoning.
+  - Docstring updated: `ns` now documented as `agree + disagree + PASS`.
+- **FULL SUITE** — 299 passed, 12 skipped, 58 xfailed. Delta = +4
+  (exactly the new ns-PASS tests). No existing pre-PR-14a or PR-14a test
+  broke. The pre-existing `TestVectorizedFunctions` fixtures use only
+  AGREE/DISAGREE/NaN (no PASS), so they were never sensitive to the bug.
+
+**Cascade to D11.** D11's plan introduces a `consensus_stats_df(vote_matrix_df)`
+function computing whole-conversation stats (the `:mod-out` Clojure path).
+That function will inevitably mirror the same `na + nd + np` recipe — so
+the ns-PASS fix lands BEFORE D10 in the stack to keep D11's implementation
+clean. D11 should follow the same pure-formula test pattern: build a vote
+matrix with mixed PASS and assert `ns == count of non-NaN cells`.
+
+**Goldens.** Stays DEFERRED. Re-recording is gated on
+sklearn-KMeans-seeding consensus (see scratch/COPILOT_MATH_QUESTIONS.md);
+no values shift at the goldens commit until D10/D11/D12 land.
+
+**Stack position.** New commit inserted between PR 14a (#2564) and D10
+(#2566). D10, D11, D12, goldens rebased cleanly on top — no conflict
+markers in `jj log -r 'edge..@+++++'`.

delphi/docs/PLAN_DISCREPANCY_FIXES.md

@@ -29,6 +29,7 @@ This plan's "PR N" labels map to actual GitHub PRs as follows:
 | PR 12 (D15) | #2523 | Stack 16/17 | Fix D15: moderation handling |
 | (K-inv) | #2524 | Stack 17/17 | Fix K-means k divergence: preserve row order |
 | PR 14a (scalar deletion) | — (in flight) | — | Delete dead scalar paths in `repness.py`; migrate blob injection tests to vectorized |
+| PR ns-PASS fix | — (planned) | — | Fix `ns` to include PASS votes in `compute_group_comment_stats_df` (Clojure `count-votes` parity) |
 | PR 8 (D10) | — (WIP) | — | Fix D10: rep comment selection — **NEEDS REWORK** |
 | PR 9 (D11) | — (WIP) | — | Fix D11: consensus selection — **NEEDS REWORK** |
 | PR 10 (D3) | — (WIP) | — | Fix D3: k-smoother buffer — **NEEDS REWORK** |

delphi/polismath/pca_kmeans_rep/repness.py

@@ -192,7 +192,8 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame,
         DataFrame indexed by (group_id, comment) with columns:
             - na: number of agrees
             - nd: number of disagrees
-            - ns: number of votes (agrees + disagrees)
+            - ns: number of votes (agrees + disagrees + PASS, Clojure parity;
+                  see repness.clj:56-61, :70)
             - pa: probability of agree (with pseudocount smoothing)
             - pd: probability of disagree (with pseudocount smoothing)
             - pat: proportion test z-score for agree
@@ -221,11 +222,17 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame,
     # Compute total counts per comment BEFORE filtering to group members
     # This matches the old behavior where "other" included ALL participants
     # not in the current group (even those not in any cluster)
+    #
+    # total_votes counts agree + disagree + PASS, matching Clojure's
+    # `count-votes` (math/src/polismath/math/repness.clj:56-61, :70).
+    # `count-votes` called with no `vote` arg uses `identity` as the filter
+    # predicate; in Clojure 0 is truthy, so PASS (0) votes are kept. NaN
+    # entries are already dropped above. Use size() to count non-NaN rows.
     total_counts = votes_only.groupby('comment').agg(
         total_agree=('vote', lambda x: (x == AGREE).sum()),
         total_disagree=('vote', lambda x: (x == DISAGREE).sum()),
+        total_votes=('vote', 'size'),
     )
-    total_counts['total_votes'] = total_counts['total_agree'] + total_counts['total_disagree']
 
     # Now add group column and filter to only group members
     votes_with_group = votes_only.copy()
@@ -244,12 +251,18 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame,
     # Get all group IDs
     all_group_ids = [group['id'] for group in group_clusters]
 
-    # Compute vote counts per (group, comment) for votes from group members
+    # Compute vote counts per (group, comment) for votes from group members.
+    #
+    # ns counts agree + disagree + PASS, matching Clojure's `count-votes`
+    # (math/src/polismath/math/repness.clj:56-61, :70). `count-votes` with
+    # no `vote` arg uses `identity` as filter; in Clojure 0 is truthy, so
+    # PASS (0) votes count. NaN entries were already dropped above. Use
+    # size() to count non-NaN rows.
     group_counts = votes_in_groups.groupby(['group_id', 'comment']).agg(
         na=('vote', lambda x: (x == AGREE).sum()),
         nd=('vote', lambda x: (x == DISAGREE).sum()),
+        ns=('vote', 'size'),
     )
-    group_counts['ns'] = group_counts['na'] + group_counts['nd']
 
     # Create full index with all (group, comment) combinations to match old behavior
     # Old implementation: for each group, iterate over ALL comments (that have any votes)

delphi/tests/test_repness_unit.py

@@ -17,6 +17,7 @@
     # DataFrame-native vectorized functions
     prop_test_vectorized, two_prop_test_vectorized, compute_group_comment_stats_df,
 )
+from polismath.utils.general import AGREE, DISAGREE, PASS
 from polismath.conversation.conversation import Conversation
 
 
@@ -334,4 +335,125 @@ def test_compute_group_comment_stats_consistency_with_conv_repness(self):
             df_row = stats_df.loc[(gid, tid)]
 
             assert np.isclose(entry['pa'], df_row['pa'], atol=1e-10)
-            assert np.isclose(entry['pd'], df_row['pd'], atol=1e-10)
+            assert np.isclose(entry['pd'], df_row['pd'], atol=1e-10)
+
+
+class TestNsIncludesPassVotes:
+    """ns / total_votes must count agree + disagree + PASS (Clojure parity).
+
+    Clojure (math/src/polismath/math/repness.clj:56-61, :70):
+        (defn- count-votes [votes & [vote]]
+          (let [filt-fn (if vote #(= vote %) identity)]
+            (count (filter filt-fn votes))))
+        ...
+        :ns (fnk [votes] (count-votes votes))
+
+    `count-votes` is called with no `vote` arg → `filt-fn = identity`. In
+    Clojure, 0 is truthy, so `(filter identity ...)` keeps every non-nil
+    entry — including PASS (0). Therefore ns = na + nd + np (PASS count).
+
+    Python had ns = na + nd, silently dropping PASS. Every downstream metric
+    (pa, pd, pat, pdt, ra, rd, rat, rdt, agree_metric, disagree_metric,
+    consensus stats) was off whenever PASS votes existed. D5 BlobInjection
+    tests bypassed `compute_group_comment_stats_df` entirely (they feed a
+    pre-baked stats blob), so the bug was invisible there — pure-formula
+    tests are the only way to RED it.
+    """
+
+    def test_ns_includes_pass_votes(self):
+        """ns counts AGREE + DISAGREE + PASS, not just AGREE + DISAGREE."""
+        # 5 ptpts, 1 comment, mixed votes: 2 agree, 1 disagree, 2 pass.
+        # Clojure ns = count of all non-nil = 5.
+        # Buggy Python ns = na + nd = 3.
+        votes_long = pd.DataFrame({
+            'participant': ['p1', 'p2', 'p3', 'p4', 'p5'],
+            'comment': ['c1'] * 5,
+            'vote': [AGREE, AGREE, DISAGREE, PASS, PASS],
+        })
+        group_clusters = [{'id': 0, 'members': ['p1', 'p2', 'p3', 'p4', 'p5']}]
+
+        stats_df = compute_group_comment_stats_df(votes_long, group_clusters)
+        row = stats_df.loc[(0, 'c1')]
+
+        assert row['na'] == 2
+        assert row['nd'] == 1
+        assert row['ns'] == 5, (
+            f"ns should include PASS (Clojure parity); got {row['ns']}"
+        )
+
+    def test_ns_all_pass_column(self):
+        """All-PASS column: na=0, nd=0, ns=3 (not 0)."""
+        votes_long = pd.DataFrame({
+            'participant': ['p1', 'p2', 'p3'],
+            'comment': ['c1'] * 3,
+            'vote': [PASS, PASS, PASS],
+        })
+        group_clusters = [{'id': 0, 'members': ['p1', 'p2', 'p3']}]
+
+        stats_df = compute_group_comment_stats_df(votes_long, group_clusters)
+        row = stats_df.loc[(0, 'c1')]
+
+        assert row['na'] == 0
+        assert row['nd'] == 0
+        assert row['ns'] == 3, (
+            f"All-PASS column should still have ns=3 (Clojure parity); "
+            f"got {row['ns']}"
+        )
+
+    def test_ns_mixed_with_nan_only_explicit_votes_count(self):
+        """NaN (unvoted) must NOT count; only explicit AGREE/DISAGREE/PASS do."""
+        # 6 ptpts on c1: 1 agree, 1 disagree, 2 pass, 2 unvoted (NaN).
+        # Clojure parity: ns = 4 (the 4 explicit votes). NaN never counts.
+        votes_long = pd.DataFrame({
+            'participant': ['p1', 'p2', 'p3', 'p4', 'p5', 'p6'],
+            'comment': ['c1'] * 6,
+            'vote': [AGREE, DISAGREE, PASS, PASS, np.nan, np.nan],
+        })
+        group_clusters = [{'id': 0, 'members': ['p1', 'p2', 'p3', 'p4', 'p5', 'p6']}]
+
+        stats_df = compute_group_comment_stats_df(votes_long, group_clusters)
+        row = stats_df.loc[(0, 'c1')]
+
+        assert row['na'] == 1
+        assert row['nd'] == 1
+        assert row['ns'] == 4, (
+            f"ns must include PASS but exclude NaN; got {row['ns']}"
+        )
+
+    def test_other_votes_includes_other_group_pass(self):
+        """`other_votes` = total_votes - ns must include PASS in BOTH halves.
+
+        Two groups, one comment. Group 0 votes [AGREE, PASS], group 1 votes
+        [DISAGREE, PASS]. Total na=1, nd=1, total_votes (Clojure) = 4.
+        Group 0: na=1, nd=0, ns=2 → other_votes=2 (the group-1 disagree + pass).
+        Group 1: na=0, nd=1, ns=2 → other_votes=2 (the group-0 agree + pass).
+        """
+        votes_long = pd.DataFrame({
+            'participant': ['p1', 'p2', 'p3', 'p4'],
+            'comment': ['c1'] * 4,
+            'vote': [AGREE, PASS, DISAGREE, PASS],
+        })
+        group_clusters = [
+            {'id': 0, 'members': ['p1', 'p2']},
+            {'id': 1, 'members': ['p3', 'p4']},
+        ]
+
+        stats_df = compute_group_comment_stats_df(votes_long, group_clusters)
+
+        g0 = stats_df.loc[(0, 'c1')]
+        assert g0['na'] == 1
+        assert g0['nd'] == 0
+        assert g0['ns'] == 2, f"group 0 ns should include its PASS; got {g0['ns']}"
+        assert g0['other_votes'] == 2, (
+            f"group 0 other_votes should include group-1 PASS; "
+            f"got {g0['other_votes']}"
+        )
+
+        g1 = stats_df.loc[(1, 'c1')]
+        assert g1['na'] == 0
+        assert g1['nd'] == 1
+        assert g1['ns'] == 2, f"group 1 ns should include its PASS; got {g1['ns']}"
+        assert g1['other_votes'] == 2, (
+            f"group 1 other_votes should include group-0 PASS; "
+            f"got {g1['other_votes']}"
+        )