Skip to content

fix: normalize non-finite numbers in data-only event payloads - #276

Open
marandaneto wants to merge 17 commits into
mainfrom
fix/non-finite-event-properties-20261022
Open

marandaneto wants to merge 17 commits into
mainfrom
fix/non-finite-event-properties-20261022

Conversation

@marandaneto

@marandaneto marandaneto commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

💡 Motivation and Context

NaN and positive/negative infinity can make JSON encoding discard an event or an entire batch. Following the design checkpoint and explicit maintainer approval, this revision narrows the original proposal to bounded, data-only payloads instead of reproducing PHP object serialization.

  • Repair non-finite numbers to JSON null only when the whole payload contains scalar values/arrays, no objects/resources/retained array-element references, valid strings, at most 10,000 values, 64 array levels, and 1 MiB of raw string/key bytes. Counts include the complete envelope and SDK metadata.
  • Otherwise pass the original whole payload unchanged to native JSON. Exhausting an inspection budget is not a new rejection: finite payloads still encode natively. Objects, enums, serializer results and reference-cell updates are not normalized.
  • This deliberately does not isolate events in a mixed batch. An opaque value anywhere can leave a non-finite neighbor subject to native encoding failure. Ineligible File events retain existing failure handling, including empty-line output. The encoded batch-size guard is unchanged and separate from the repair budget.
  • Remove the custom sequential writer, object-handler emulation, property-layout reconstruction, active-reference tracking and synthetic JSON error restoration. Eligibility is read-only; copying starts only after full qualification. Native JSON remains authoritative for ordering, redaction, callback counts, recursion, exceptions and errors. There are no serializer retries, partial-output flags or token replacement.

Existing SDK methods/options are unchanged; capture callers can use the existing before_send hook for their own JSON-safe conversion policy. The encoder lives in PostHog\Internal and that namespace is excluded from public API snapshots. The API snapshot and README match main; no public API is added. The serialization/design docs were removed at maintainer request, with the agreed limits and fallback policy retained in this description. A patch change intent is included. The capture spec has no NaN/infinity policy. Related Python work: PostHog/posthog-python#1081.

💚 How did you test it?

  • Retained every concrete reported reproduction. The five newest cases failed on 577a8b7 and now pass without changing their expectations: abstract-property redaction order, stateless/stateful active-array serializer re-entry, hooked DateInterval output/caller state, and a 12 MiB string under 32 MiB before normal batch-size rejection.
  • The new data-only policy had six expected failures against the previous writer before replacement. Boundary tests now cover exact value/byte/depth limits, successful finite fallback beyond each bound, opaque/reference whole-payload fallback, caller immutability and no serializer retry.
  • Original object/non-finite examples remain as native-failure cases under the approved narrowed contract. Added finite companions retain successful wire-format/privacy coverage rather than dropping the examples. Added supported data-only NaN/infinity integration cases for both batch and File consumers, plus 32 MiB dense-array and escaped-string repair probes against independent native-null fixtures.
  • Follow-up retained-capacity repros were red on d16a8b1: a million-slot pruned root, 16 repeated pruned children, and a sparse high key each exhausted 32 MiB while native-error/native-null baselines completed normally. Key-preserving compaction now passes all three under the same limit; a numeric-string symbol-table regression also protects object-vs-list JSON shape. Unchanged prefixes, sparse keys and caller values remain covered.
  • 165 focused tests / 1,220 assertions and the complete 1,595-test / 8,349-assertion suite with Xdebug 3.5.3 coverage pass locally on PHP 8.5.10. Existing notices remain 27 deprecations and 42 PHPUnit deprecations. All 16 adapter tests pass.
  • An API snapshot regression was red before excluding the internal namespace and now passes, while retaining existing public/legacy symbols. All encoding regressions remain unchanged apart from imports and fully qualified helper references.
  • CI runs PHPUnit on PHP 8.2, 8.3, 8.4 and 8.5. Changed-file PHP_CodeSniffer, public API snapshot checks, strict Composer validation and whitespace checks pass. Tests use offline fakes/loopback servers, not live PostHog endpoints.

Compliance assertions remain advisory: lib_curl 46/47, socket 45/47 and fork_curl 36/47 on d16a8b1, matching the previous 577a8b7 reports exactly. Inventory-completeness checks pass; unrelated transport gaps are not changed here.

Design invariants were reviewed separately from test counts: no copying before full eligibility; no executable user values/reference cells in a repairable graph; original graphs preserved for native fallback; primitive string validation does not overwrite callback-visible JSON error state. The initial assumption that live entry counts bound copy-on-write capacity was disproved; normalization now compacts live entries before mapping values and reuses unchanged subtrees without writing into caller-seeded tables. Earlier autoreview covered an older revision, not this replacement; human review remains required.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm change to generate a change intent file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Human DRI: @marandaneto. Implemented with Pi using bash and file tools. Earlier graph/writer approaches were rejected after compatibility and ownership regressions; the maintainer approved conservative whole-payload native delegation with narrowly bounded data-only repair. No new hook or SDK option was introduced. No public session link is available. Human review is required.

@marandaneto marandaneto self-assigned this Oct 22, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed a3fd115: 1 Must fix, 1 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

posthog-php-socket Compliance Report

Date: 2026-10-23T13:00:30.216518+00:00
Duration: 114637ms

⚠️ Some Tests Failed

45/47 tests passed, 2 failed


Capture Tests

⚠️ 28/30 tests passed, 2 failed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 25ms
Format Validation.Event Has Uuid ✅ 520ms
Format Validation.Event Has Lib Properties ✅ 524ms
Format Validation.Distinct Id Is String ✅ 520ms
Format Validation.Token Is Present ✅ 522ms
Format Validation.Custom Properties Preserved ✅ 520ms
Format Validation.Event Has Timestamp ✅ 520ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 522ms
Retry Behavior.Retries On 503 ✅ 5832ms
Retry Behavior.Does Not Retry On 400 ✅ 2524ms
Retry Behavior.Does Not Retry On 401 ✅ 2523ms
Retry Behavior.Respects Retry After Header ❌ 5631ms
Retry Behavior.Implements Backoff ✅ 16238ms
Retry Behavior.Retries On 500 ✅ 5137ms
Retry Behavior.Retries On 502 ✅ 5628ms
Retry Behavior.Retries On 504 ✅ 5632ms
Retry Behavior.Max Retries Respected ✅ 16246ms
Deduplication.Generates Unique Uuids ✅ 527ms
Deduplication.Preserves Uuid On Retry ✅ 5629ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10838ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5633ms
Deduplication.No Duplicate Events In Batch ✅ 527ms
Deduplication.Different Events Have Different Uuids ✅ 523ms
Compression.Sends Gzip When Enabled ✅ 522ms
Batch Format.Uses Proper Batch Structure ✅ 522ms
Batch Format.Flush With No Events Sends Nothing ✅ 518ms
Batch Format.Multiple Events Batched Together ✅ 511ms
Error Handling.Does Not Retry On 403 ✅ 2524ms
Error Handling.Does Not Retry On 413 ✅ 2524ms
Error Handling.Retries On 408 ❌ 5529ms

Failures

retry_behavior.respects_retry_after_header

Retry delay too short: 103ms < 2500ms

error_handling.retries_on_408

Expected at least 2 requests, got 1

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 520ms
Request Payload.Flags Request Uses V2 Query Param ✅ 524ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 518ms
Request Payload.Flags Request Omits Authorization Header ✅ 519ms
Request Payload.Token In Flags Body Matches Init ✅ 519ms
Request Payload.Groups Round Trip ✅ 524ms
Request Payload.Groups Default To Empty Object ✅ 520ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 520ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 519ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 520ms
Request Lifecycle.No Flags Request On Init Alone ✅ 516ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 508ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 527ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 522ms
Retry Behavior.Retries Flags On 502 ✅ 621ms
Retry Behavior.Retries Flags On 504 ✅ 621ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 523ms

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

posthog-php-fork_curl Compliance Report

Date: 2026-10-23T13:00:19.534237+00:00
Duration: 112407ms

⚠️ Some Tests Failed

36/47 tests passed, 11 failed


Capture Tests

⚠️ 19/30 tests passed, 11 failed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 25ms
Format Validation.Event Has Uuid ✅ 520ms
Format Validation.Event Has Lib Properties ✅ 522ms
Format Validation.Distinct Id Is String ✅ 522ms
Format Validation.Token Is Present ✅ 521ms
Format Validation.Custom Properties Preserved ✅ 522ms
Format Validation.Event Has Timestamp ✅ 523ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 524ms
Retry Behavior.Retries On 503 ❌ 5524ms
Retry Behavior.Does Not Retry On 400 ✅ 2528ms
Retry Behavior.Does Not Retry On 401 ✅ 2524ms
Retry Behavior.Respects Retry After Header ❌ 5527ms
Retry Behavior.Implements Backoff ❌ 15538ms
Retry Behavior.Retries On 500 ❌ 5526ms
Retry Behavior.Retries On 502 ❌ 5534ms
Retry Behavior.Retries On 504 ❌ 5526ms
Retry Behavior.Max Retries Respected ❌ 15538ms
Deduplication.Generates Unique Uuids ✅ 527ms
Deduplication.Preserves Uuid On Retry ❌ 5527ms
Deduplication.Preserves Uuid And Timestamp On Retry ❌ 10533ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ❌ 5529ms
Deduplication.No Duplicate Events In Batch ✅ 528ms
Deduplication.Different Events Have Different Uuids ✅ 523ms
Compression.Sends Gzip When Enabled ✅ 526ms
Batch Format.Uses Proper Batch Structure ✅ 523ms
Batch Format.Flush With No Events Sends Nothing ✅ 514ms
Batch Format.Multiple Events Batched Together ✅ 516ms
Error Handling.Does Not Retry On 403 ✅ 2525ms
Error Handling.Does Not Retry On 413 ✅ 2525ms
Error Handling.Retries On 408 ❌ 5528ms

Failures

retry_behavior.retries_on_503

Expected at least 3 requests, got 1

retry_behavior.respects_retry_after_header

Expected at least 2 requests, got 1

retry_behavior.implements_backoff

Expected at least 3 requests, got 1

retry_behavior.retries_on_500

Expected at least 2 requests, got 1

retry_behavior.retries_on_502

Expected at least 2 requests, got 1

retry_behavior.retries_on_504

Expected at least 2 requests, got 1

retry_behavior.max_retries_respected

Expected 4 requests, got 1

deduplication.preserves_uuid_on_retry

Need at least 2 requests to check retry

deduplication.preserves_uuid_and_timestamp_on_retry

Expected at least 3 requests, got 1

deduplication.preserves_uuid_and_timestamp_on_batch_retry

Expected at least 2 requests, got 1

error_handling.retries_on_408

Expected at least 2 requests, got 1

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 518ms
Request Payload.Flags Request Uses V2 Query Param ✅ 516ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 517ms
Request Payload.Flags Request Omits Authorization Header ✅ 517ms
Request Payload.Token In Flags Body Matches Init ✅ 516ms
Request Payload.Groups Round Trip ✅ 517ms
Request Payload.Groups Default To Empty Object ✅ 516ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 516ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 517ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 515ms
Request Lifecycle.No Flags Request On Init Alone ✅ 513ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 512ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 518ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 517ms
Retry Behavior.Retries Flags On 502 ✅ 619ms
Retry Behavior.Retries Flags On 504 ✅ 618ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 523ms

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

posthog-php-lib_curl Compliance Report

Date: 2026-10-23T13:00:21.621508+00:00
Duration: 116659ms

⚠️ Some Tests Failed

46/47 tests passed, 1 failed


Capture Tests

⚠️ 29/30 tests passed, 1 failed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 28ms
Format Validation.Event Has Uuid ✅ 523ms
Format Validation.Event Has Lib Properties ✅ 524ms
Format Validation.Distinct Id Is String ✅ 526ms
Format Validation.Token Is Present ✅ 525ms
Format Validation.Custom Properties Preserved ✅ 525ms
Format Validation.Event Has Timestamp ✅ 526ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 525ms
Retry Behavior.Retries On 503 ✅ 5835ms
Retry Behavior.Does Not Retry On 400 ✅ 2527ms
Retry Behavior.Does Not Retry On 401 ✅ 2529ms
Retry Behavior.Respects Retry After Header ❌ 6335ms
Retry Behavior.Implements Backoff ✅ 16248ms
Retry Behavior.Retries On 500 ✅ 5634ms
Retry Behavior.Retries On 502 ✅ 5635ms
Retry Behavior.Retries On 504 ✅ 5633ms
Retry Behavior.Max Retries Respected ✅ 17050ms
Deduplication.Generates Unique Uuids ✅ 232ms
Deduplication.Preserves Uuid On Retry ✅ 5631ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10842ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5634ms
Deduplication.No Duplicate Events In Batch ✅ 533ms
Deduplication.Different Events Have Different Uuids ✅ 527ms
Compression.Sends Gzip When Enabled ✅ 525ms
Batch Format.Uses Proper Batch Structure ✅ 524ms
Batch Format.Flush With No Events Sends Nothing ✅ 521ms
Batch Format.Multiple Events Batched Together ✅ 513ms
Error Handling.Does Not Retry On 403 ✅ 2528ms
Error Handling.Does Not Retry On 413 ✅ 2526ms
Error Handling.Retries On 408 ✅ 5633ms

Failures

retry_behavior.respects_retry_after_header

Retry delay too short: 804ms < 2500ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 525ms
Request Payload.Flags Request Uses V2 Query Param ✅ 522ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 523ms
Request Payload.Flags Request Omits Authorization Header ✅ 523ms
Request Payload.Token In Flags Body Matches Init ✅ 523ms
Request Payload.Groups Round Trip ✅ 522ms
Request Payload.Groups Default To Empty Object ✅ 523ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 522ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 523ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 523ms
Request Lifecycle.No Flags Request On Init Alone ✅ 517ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 510ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 527ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 524ms
Retry Behavior.Retries Flags On 502 ✅ 626ms
Retry Behavior.Retries Flags On 504 ✅ 627ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 525ms

@marandaneto
marandaneto marked this pull request as ready for review October 22, 2026 16:27
@marandaneto
marandaneto requested a review from a team as a code owner October 22, 2026 16:27
Comment thread lib/JsonEncoder.php Outdated
@veria-ai

veria-ai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 3 · PR risk: 0/10

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 2155f5a: 2 Should fix.

Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed e414589: 1 Must fix, 1 Should fix.

This is review round 3 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 7e232ad: 1 Must fix.

This is review round 4 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed c5d55ce: 2 Should fix.

This is review round 5 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

This Standard review started from the posthog-review label. Comment @posthog review --deep for a Deep review, or @posthog help to see all commands.

Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated
@posthog posthog Bot removed the reviewhog label Oct 23, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 5a4a540: 1 Must fix, 2 Should fix.

This is review round 6 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 577a8b7: 1 Must fix, 3 Should fix.

This is review round 7 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated
Comment thread lib/JsonEncoder.php Outdated

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed d16a8b1: 1 Must fix.

This is review round 8 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread lib/JsonEncoder.php Outdated
@marandaneto marandaneto changed the title fix: serialize non-finite event properties as null fix: normalize non-finite numbers in data-only event payloads Oct 23, 2026
Comment thread lib/JsonEncoder.php Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants