feat: "newest courses first" Algolia sort — config-gated replica registry + fail-safe

[macdiesel]

Summary

Backend support for a "newest courses first" sort on the enterprise Learner Portal search page, via a recency-sorted Algolia replica. Builds and configures the replica; the waffle flag (edx-enterprise) and the MFE point search at it (separate PRs).

This PR also generalizes replica handling into one config-gated registry and makes replica configuration fail-safe. (The fail-safe was briefly split into a separate PR #119 — now folded back in, so the replica feature and its safe configuration ship together.)

What's here

• Recency replica — recently_published_timestamp (earliest published course-run start) added to course records; a replica that leads customRanking with desc(recently_published_timestamp). Undated courses get 0 so they sort last (not the far-future ALGOLIA_DEFAULT_TIMESTAMP).
• Unified replica registry — ALGOLIA_REPLICA_CONFIG_KEYS (in api_client/constants) is the single source of truth for all sort replicas (base duration replica + recency). _build_algolia_replicas, configure_algolia_index, and the secured-key restrictIndices all loop it. A replica is declared/configured/restricted only when its settings.ALGOLIA index-name key is set — inert by default (no virtual(None)), so it's safe to deploy ahead of ops enabling it. Adding a future sort is a registry entry + the field its ranking sorts on.
• Unified set_index_settings — folded the old primary_index boolean / separate replica method into one name-based set_index_settings(index_settings, index_name=None) routing through _get_index.
• Fail-safe configuration — each replica's set_index_settings is wrapped so an AlgoliaException is logged and skipped; one replica failing never aborts the reindex (primary + other replicas stay configured). Degraded state is always the safe base sort.
• Docs — ADR 0014 records the design (registry, config-vs-flag layering, the missing-index open question + operational mitigation); REPLICA_INDEX_NAME documented as the duration replica that powers MFE video search.
• Fixed incremental_reindex_algolia for the unified set_index_settings signature.

Test

• Full test_algolia_utils.py / test_algolia.py / command tests green (only the 3 pre-existing time-sensitive test_get_course_run* failures remain, which fail on master too). isort + pylint clean.

Rollout (see ADR 0014)

Deploy + reindex_algolia here before ops sets the MFE env var / enables the enterprise.search_default_sort_newest waffle flag.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.96%. Comparing base (3975505) to head (672f1e4).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #118      +/-   ##
==========================================
+ Coverage   85.67%   85.96%   +0.29%     
==========================================
  Files         109      109              
  Lines        6700     6777      +77     
  Branches      827      840      +13     
==========================================
+ Hits         5740     5826      +86     
+ Misses        830      821       -9     
  Partials      130      130              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds backend support for a “newest courses first” Algolia sort by introducing a recency-based replica index, while also consolidating replica handling behind a config-gated registry and making replica configuration resilient to partial failures.

Changes:

• Adds recently_released_timestamp to course Algolia records and defines a recency-sorted replica (desc(recently_released_timestamp)).
• Introduces a shared, config-gated replica registry (ALGOLIA_REPLICA_CONFIG_KEYS) and updates index configuration + secured-key restrictions to loop over configured replicas only.
• Makes replica configuration fail-safe (replica failures are logged and skipped so primary configuration continues) and updates incremental reindex tooling/tests for the unified set_index_settings API.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
enterprise_catalog/settings/base.py	Documents and adds the new RECENTLY_RELEASED_REPLICA_INDEX_NAME setting key.
enterprise_catalog/apps/search/management/commands/incremental_reindex_algolia.py	Updates incremental reindex flow to configure replica settings via index_name=.
enterprise_catalog/apps/search/management/commands/tests/test_incremental_reindex_algolia.py	Adjusts tests for lazy replica initialization via name-based settings calls.
enterprise_catalog/apps/catalog/algolia_utils.py	Adds replica registry plumbing, recency replica settings, fail-safe per-replica configuration, and recently_released_timestamp computation.
enterprise_catalog/apps/catalog/tests/test_algolia_utils.py	Adds coverage for recency timestamp, configured-replica gating, and safe-fail configuration behavior.
enterprise_catalog/apps/api_client/constants.py	Defines ALGOLIA_REPLICA_CONFIG_KEYS as the shared replica registry.
enterprise_catalog/apps/api_client/algolia.py	Implements name-based set_index_settings routing and secured-key restriction over all configured replicas.
enterprise_catalog/apps/api_client/tests/test_algolia.py	Adds tests for set_index_settings(index_name=...) and restrictIndices behavior with/without the recency replica.
docs/how_to/add_an_algolia_sort_replica.rst	Adds a step-by-step guide for adding future Algolia sort replicas via the registry pattern.
docs/decisions/0014-newest-courses-sort-replica.rst	Adds ADR documenting the design, rollout strategy, and operational tradeoffs.

[pwnage101]

Logic LGTM! Just have some nits and strong desires to rename things.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

[pwnage101]

Selecting virtual replicas would be an Alternative Considered, not part of the Context section. Please move this below.

Also I don't think I agree with this as a foundation for rejecting virtual replicas. Virtual replicas simply don't support fully customizable sort orders, since there's always a built-in "relevance" factor as the top priority sort order. We'd use virtual replicas to save money and record usage, but the tradeoff is less precise and non-deterministic sorting.

[pwnage101]

AI docs do this a lot where they explain low-level details, rather than what actually matters and what the end goals are. Very few people will read this in the first place, let alone understand it and have the correct takeaway, even if it is factually true.

My version is simpler:

Suggested change

	* ``ALGOLIA`` is *replaced* (not merged) from the deployment YAML, so a replica
	is only live once ops sets its index-name key in ``edx-internal`` and a
	``reindex_algolia`` run declares it on the primary.
	* The fundamental design of the enterprise-catalog logic assumes we will only provision one replica, so there is only one setting (``ALGOLIA['REPLICA_INDEX_NAME']``) and one python function to provision that one replica.

[pwnage101]

There is one missing consequence: cost. Another full replica (non-virtual) doubles our record count.

[iloveagent57]

the amount of misdirection here is tripping me up. can we make it more directly settings-driven, something like:

# settings/base.py
ALGOLIA = {
    'INDEX_NAME': '',
    # Base replica: the long-standing sort the learner-portal MFE points its video search
    # (SEARCH_INDEX_IDS.VIDEOS) at -- the MFE's ALGOLIA_REPLICA_INDEX_NAME env var. Its
    # customRanking lives in ALGOLIA_REPLICA_INDEX_SETTINGS.
    'REPLICA_INDEX_NAME': '',
    'ADDITIONAL_REPLICA_INDEX_SETTINGS': {'newest_courses_index_repl': 'ALGOLIA_RECENTLY_RELEASED_REPLICA_INDEX_SETTINGS'}, 
    'APPLICATION_ID': '',
    'API_KEY': '',
}

this defines both the additional replica index names that are enabled and which algolia index setting to use.

[iloveagent57]

but actually, that's still a lot of indirection, which is a code smell (with the original settings/algolia_utils.py relationship, not with your code). can we just yoink the settings values from algolia_utils into settings?

ALGOLIA_RECENTLY_RELEASED_REPLICA_INDEX_SETTINGS = {
    'customRanking': [
        'desc(recently_released_timestamp)',
        'asc(metadata_language)',
        'asc(visible_via_association)',
        'asc(created)',
        'desc(course_bayesian_average)',
        'desc(recent_enrollment_count)',
    ],
}
ALGOLIA = {
    'INDEX_NAME': '',
    # Base replica: the long-standing sort the learner-portal MFE points its video search
    # (SEARCH_INDEX_IDS.VIDEOS) at -- the MFE's ALGOLIA_REPLICA_INDEX_NAME env var. Its
    # customRanking lives in ALGOLIA_REPLICA_INDEX_SETTINGS.
    'REPLICA_INDEX_NAME': '',
    'ADDITIONAL_REPLICA_INDEX_SETTINGS': {'newest_courses_index_repl': ALGOLIA_RECENTLY_RELEASED_REPLICA_INDEX_SETTINGS}, 
    'APPLICATION_ID': '',
    'API_KEY': '',
}

in a later PR, we could clean up the rest of the algolia settings later to follow a similar pattern. I think that will be much more straightforward everywhere.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.