fix: group-informed-consensus should not be high for statements with extremely low engagement

[nicobao]

Current example in polis

• https://pol.is/42ssceevsw
• https://pol.is/report/r2msmbb2bm7nmjxtftayt

See:

Expected behavior

The statements highlighted in red above should not appear so high in the ranking of group-informed-consensus, because the number of people who interacted with these statements is negligible compared to the total amount of participants (8 participants voted on the first statement, and 2 participants voted on the second statement, out of 1165 total amount of participants).

Detailed changes required

p-success should decrease with the # of participants who actually expressed their vote. When a negligible amount of participants actually voted, p-success should be close to 0, even if every participants who expressed their vote agreed/disagreed.

As a result, group-aware-consensus value (which is calculated by multiplying the p-success across each cluster) should be close to 0 for statements with very low participation such as the ones highlighted in red in the screenshot above. Those statements would therefore appear much lower in this ranking, reaching the expected behavior.

[patcon]

Going to speak very plainly for expedience, but open to being wrong. (I'm just trying to capture some of my sense of certainty on certain things.)

p-success should decrease with the # of participants who actually expressed their vote. When a negligible amount of participants actually voted, p-success should be close to 0, even if every participants who expressed their vote agreed/disagreed.

This would break a very foundational assumption of group representative calculations. The whole point of designing the polis pipeline to be "greedy" (accepting a small or large number of votes from participants, and so having sparse sets of votes per statement) is to allow such calculations where a high percentage supports, but with very incomplete proportion of votes. (E.g., if only 30 of 200 in a group respond to a statement, but all 30 agree, Polis is supposed to treat that as comparable, and use stats to qualify it, ensuring that it's still statistically significant)

I take your request as a separate concern, more like the other select_*() functions -- the cut-off is arbitrary. It should definitely NOT calculate p-success based on the full participant count of a group, and p-success being in proportion to those participants in a group who've seen/voted on a statement is definitely the right choice for the internal calculations that rank things.

I think what we actually want (which is the root of this request), is an equivalent of select_representative_statements() (for group-representative statements) and selection_consensus_statements() (for "majority opinion" statements, for lack of better word), but for select_group_informed_consensus_statements().

These functions each have an informal algorithm that selects (and also filters out) statements based on somewhat arbitrary (but important) concerns, that make the statements more relevant to a reasonable end-user who wants to learn something. Right now, group-informed-consensus is an important internal metric (for comment routing), and sorting by it can seem a little zany in edge-cases, as you're seeing. We need to write a selection algorithm, which we'd likely want to use in place of select_consensus_statements()

Does the above align with your desire here, or am I off-base?

[nicobao]

Interesting. You're not off-base! I am thinking about it and will come back to you

[nicobao]

This would break a very foundational assumption of group representative calculations. The whole point of designing the polis pipeline to be "greedy" (accepting a small or large number of votes from participants, and so having sparse sets of votes per statement) is to allow such calculations where a high percentage supports, but with very incomplete proportion of votes. (E.g., if only 30 of 200 in a group respond to a statement, but all 30 agree, Polis is supposed to treat that as comparable, and use stats to qualify it, ensuring that it's still statistically significant)

=> That makes sense — and I may be off base here — but I’m wondering if there’s a way to apply a slightly more nuanced approach. Specifically, what if we use an exponential scaling function to account for the "incompleteness" of participation?

The idea would be: when very few participants vote (e.g., 2 out of 200,000), the impact of those votes should be minimal. But as participation increases, the influence on p-success should increase rapidly — not linearly, but exponentially — until it plateaus at a full weighting (i.e., multiplying by 1).

Just to clarify: I’m not suggesting that p-success itself should increase exponentially with more votes. Instead, I’m proposing that we multiply p-success by a confidence factor, which itself grows exponentially with the vote-to-participant ratio, starting near 0 and asymptotically approaching 1.

Illustrative function proposed by AI from the above idea:

def adjusted_p_success(p_success, num_votes, total_participants, growth_rate=15):
    """
    Apply an exponential weighting factor to p_success based on vote completeness.
    
    - p_success: The raw p-success value (e.g., from agreement metrics).
    - num_votes: Number of participants who voted on this statement.
    - total_participants: Total participants in the group.
    - growth_rate: Controls steepness of the curve. Higher = faster approach to 1.
    """
    if total_participants == 0:
        return 0  # Avoid division by zero

    ratio = num_votes / total_participants
    factor = 1 - math.exp(-growth_rate * ratio)  # Exponential curve approaching 1
    return p_success * factor

This function ensures:

• Very low participation → very low impact.
• Moderate participation → rapidly increasing influence.
• High participation → close to full influence (p_success * 1).