Repository navigation
feat(flags): report unresolved local flags on evaluate_flags snapshots - #319
Conversation
Implements the "Snapshot unresolved flag reporting" and "Warning for flags local evaluation never resolves" requirements of the evaluate-flags spec (PostHog/sdk-specs#85). - Add `FeatureFlagEvaluations#unresolved_flags`, with a `PostHog::UnresolvedFlagReason` for each flag and the spec's fixed precedence. - Report `local_evaluation_inconclusive` instead of `flag_missing` when a snapshot read hits an unresolved flag. - Do not stop a condition at the first inconclusive filter, so a filter that does not match resolves the condition locally. - Check `active` before experience continuity, also for dependencies. - Log one warning per flag that local evaluation can never resolve, repeated only when its definition changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 39793b31-809e-4689-a5d8-b4359cbb22c1
posthog-ruby-sync Compliance ReportDate: 2026-10-29T17:20:04.739245+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 11ms |
| Format Validation.Event Has Uuid | ✅ | 6ms |
| Format Validation.Event Has Lib Properties | ✅ | 9ms |
| Format Validation.Distinct Id Is String | ✅ | 5ms |
| Format Validation.Token Is Present | ✅ | 6ms |
| Format Validation.Custom Properties Preserved | ✅ | 5ms |
| Format Validation.Event Has Timestamp | ✅ | 8ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 7ms |
| Retry Behavior.Retries On 503 | ✅ | 5326ms |
| Retry Behavior.Does Not Retry On 400 | ❌ | 10ms |
| Retry Behavior.Does Not Retry On 401 | ❌ | 6ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8016ms |
| Retry Behavior.Implements Backoff | ❌ | 351ms |
| Retry Behavior.Retries On 500 | ✅ | 5117ms |
| Retry Behavior.Retries On 502 | ✅ | 5118ms |
| Retry Behavior.Retries On 504 | ✅ | 5120ms |
| Retry Behavior.Max Retries Respected | ❌ | 542ms |
| Deduplication.Generates Unique Uuids | ✅ | 18ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5115ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10308ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5128ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 20ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 9ms |
| Compression.Sends Gzip When Enabled | ✅ | 5ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 6ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 3ms |
| Batch Format.Multiple Events Batched Together | ❌ | 16ms |
| Error Handling.Does Not Retry On 403 | ❌ | 5ms |
| Error Handling.Does Not Retry On 413 | ❌ | 5ms |
| Error Handling.Retries On 408 | ✅ | 5112ms |
Failures
retry_behavior.does_not_retry_on_400
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.does_not_retry_on_401
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.implements_backoff
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.max_retries_respected
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
batch_format.multiple_events_batched_together
Expected 1 requests, got 5
error_handling.does_not_retry_on_403
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
error_handling.does_not_retry_on_413
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 10ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 7ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 7ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 7ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 9ms |
| Request Payload.Groups Round Trip | ✅ | 8ms |
| Request Payload.Groups Default To Empty Object | ✅ | 8ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 6ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 6ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 6ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 2ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 6ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 11ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 6ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 147ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 139ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 9ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
posthog-ruby-async Compliance ReportDate: 2026-10-29T17:20:38.170955+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 10ms |
| Format Validation.Event Has Uuid | ✅ | 106ms |
| Format Validation.Event Has Lib Properties | ✅ | 110ms |
| Format Validation.Distinct Id Is String | ✅ | 107ms |
| Format Validation.Token Is Present | ✅ | 107ms |
| Format Validation.Custom Properties Preserved | ✅ | 111ms |
| Format Validation.Event Has Timestamp | ✅ | 106ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 8ms |
| Retry Behavior.Retries On 503 | ✅ | 5309ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2109ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2110ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8013ms |
| Retry Behavior.Implements Backoff | ✅ | 15509ms |
| Retry Behavior.Retries On 500 | ✅ | 5213ms |
| Retry Behavior.Retries On 502 | ✅ | 5212ms |
| Retry Behavior.Retries On 504 | ✅ | 5209ms |
| Retry Behavior.Max Retries Respected | ✅ | 15623ms |
| Deduplication.Generates Unique Uuids | ✅ | 112ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5213ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10412ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5214ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 113ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 108ms |
| Compression.Sends Gzip When Enabled | ✅ | 106ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 107ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 5ms |
| Batch Format.Multiple Events Batched Together | ✅ | 111ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2109ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2109ms |
| Error Handling.Retries On 408 | ✅ | 5213ms |
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 108ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 108ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 108ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 109ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 109ms |
| Request Payload.Groups Round Trip | ✅ | 108ms |
| Request Payload.Groups Default To Empty Object | ✅ | 108ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 107ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 108ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 107ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 4ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 106ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 111ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 107ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 210ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 249ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 109ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
dustinbyrne
left a comment
There was a problem hiding this comment.
The unresolved-flag diagnostics and supporting coverage look sound overall. One compatibility issue remains: returned failed flags now lose their documented false snapshot value. Please preserve that behavior or explicitly resolve the contract and release change described inline.
AI-assisted review.
dustinbyrne
left a comment
There was a problem hiding this comment.
The returned-failed-record compatibility issue is fixed, including caller defaults and capture enrichment. No required findings remain; one optional release-note clarification is inline.
AI-assisted review.
💡 Motivation and Context
Why: sdk-specs #85 defines how a server SDK reports flags that local evaluation could not resolve. Today such a flag is absent from the
evaluate_flagssnapshot, so it looks the same as a flag that does not exist, and$feature_flag_calledreportsflag_missingfor both. Withonly_evaluate_locally: truethis is permanent. posthog-node already implements this (PostHog/posthog-js#5234); posthog-python (#1133) and posthog-server (#896) are in review.Changes:
FeatureFlagEvaluations#unresolved_flagsmaps each unresolved flag to aPostHog::UnresolvedFlagReason(experience_continuity,unsupported_definition,unresolved_dependency,missing_context). When several causes apply, the spec's fixed precedence picks one.only/only_accessedsnapshots have no entries, and reading the hash is not an access.local_evaluation_inconclusiveinstead offlag_missing.InconclusiveMatchErrorandRequiresServerEvaluationtake an optionalreason:, set at every raise site. A bad person value ismissing_context; a bad filter value isunsupported_definition.Behavior changes (all in the changeset):
falselocally.Divergences from the spec or posthog-node:
false, as in posthog-node.unsupported_definition, even when the person value causes the deferral. Patterns that always need the server are in the load-time warning; posthog-node does not warn about regexes.💚 How did you test it?
New RSpec examples mirror every scenario tagged
@unresolved_flags_capableand@unresolvable_flag_warning_capableinacceptance/public/evaluate-flags.feature, plus precedence, the condition loop, dependencies, warning dedup, failed fallback and unexpected errors. Run locally:bundle exec rspec(2007 examples, 0 failures, 40 pending for the optional OpenTelemetry bundle)bundle exec rubocopbundle exec rake public_api:checkNo manual testing in an app.
📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset file🤖 Agent context
Autonomy: Human-driven (agent-assisted)
PostHog::Utils.convert_to_datetime(@api private) now takes a requiredreason:so date parsing can tell a bad person value from a bad filter value.Created with PostHog Desktop
🤖 Generated with Claude Code