Fix skewed search replica request distribution when write replicas exist

[mars2cbh]

Description

IndexShardRoutingTable#filterAndOrderShards (used by searchReplicaActiveInitializingShardIt() / the _search_replica preference path) rotated the full replicas list before applying the search-only filter. RotationShardShuffler is a pure round-robin rotation, so rotation offsets that land on a filtered-out write replica collapse onto the same head shard as the adjacent offset. In a reader/writer separated cluster with one write replica and n search replicas, the search replica that precedes the write replica in list order becomes the iterator head for two of the n+1 rotation offsets, permanently receiving a 2/(n+1) share of search traffic (exactly 2x the others). With replicas = [W, S1, S2, S3]:

rotation offset	rotated list	iterator head
0	W, S1, S2, S3	S3
1	S1, S2, S3, W	S3
2	S2, S3, W, S1	S1
3	S3, W, S1, S2	S2

This change applies the filter first and rotates only the filtered active list, restoring a uniform round-robin. Active shards still come before initializing ones. replicaActiveInitializingShardIt() (filter accepts everything, so no offset collapse) and the zero-write-replica case keep their previous uniform behavior.

Added tests:

• testSearchReplicaShardIteratorEvenDistribution — randomizes 0–3 write replicas, 2–5 search replicas, and list order, then asserts each search replica is the iterator head an exactly equal number of times. Fails on the old implementation whenever a write replica is present.
• testSearchReplicaShardIteratorReturnsActiveBeforeInitializing — asserts only search replicas are returned, active before initializing.

Related Issues

Resolves #22527

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 0243728)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

Persistent review updated to latest commit 6925ae6

[github-actions]

❌ Gradle check result for 6925ae6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit bea0a45

[github-actions]

✅ Gradle check result for bea0a45: SUCCESS

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.44%. Comparing base (0213338) to head (bea0a45).

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #22528      +/-   ##
============================================
- Coverage     73.48%   73.44%   -0.04%     
+ Complexity    76460    76451       -9     
============================================
  Files          6104     6104              
  Lines        346582   346586       +4     
  Branches      49879    49879              
============================================
- Hits         254675   254543     -132     
- Misses        71664    71782     +118     
- Partials      20243    20261      +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Persistent review updated to latest commit 0243728

[github-actions]

✅ Gradle check result for 0243728: SUCCESS