Summary
After the per-org rollout of the workflow-engine read flags for issue/metric alert endpoints (organizations:workflow-engine-metric-alert-endpoints-get and the related per-method flags from #112827, #111230, #111330), WorkflowEngineRuleSerializer does not round-trip several fields that the corresponding PUT/POST endpoints accept and persist. The values are stored correctly on the server, but the GET response that follows the write does not include them.
For automation that compares the post-apply response to the planned write (notably the jianyuan/terraform-provider-sentry Terraform provider), this surfaces as Provider produced inconsistent result after apply errors and prevents Terraform applies from succeeding, even though the underlying alert configuration on Sentry's side is unchanged.
Affected fields (observed)
For sentry_issue_alert resources (actions_v2, conditions_v2, filters_v2):
-
actions_v2[*].notify_email.fallthrough_type for target_type = "IssueOwners"
- PUT body includes
fallthroughType: "ActiveMembers" (or "AllMembers" / "NoOne").
- GET response omits
fallthroughType entirely for IssueOwners targets.
- Code reference:
WorkflowEngineRuleSerializer only synthesizes a default fallthroughType for Member/Team targets, see src/sentry/api/serializers/models/rule.py around the # XXX: add default fallthroughType for email Team/Member actions block. There is no equivalent path that re-emits a stored fallthroughType for IssueOwners when the underlying Action.data["fallthrough_type"] is empty.
-
filters_v2[*].tagged_event.value
- PUT body includes
value: "<some string>".
- GET response omits
value from the serialized tagged_event filter.
- Code reference:
create_tagged_event_data_condition in src/sentry/workflow_engine/migration_helpers/issue_alert_conditions.py only writes value when present in the source blob, and the read-side equivalent does not always re-emit it.
-
conditions_v2[*].first_seen_event / regression_event / event_frequency / reappeared_event
- PUT body includes the populated condition object (with
name, and for event_frequency the value/interval/comparison_type fields).
- GET response either drops the entire object or returns it in a different position than it was written, depending on the rule. The provider sees this as
was <object>, but now null (or was null, but now <object>).
For sentry_metric_alert resources, separately (this one is arguably "by design" but is the same class of round-trip mismatch):
resolve_threshold is now always returned with the auto-calculated value derived from the lowest-severity trigger's alert_threshold, where the legacy serializer returned null if the user never set it explicitly. This also produces drift in the Terraform provider, though it is not a hard apply failure since PUT-with-null is accepted and re-derived. Code reference: WorkflowEngineDataConditionSerializer.resolve_threshold in src/sentry/incidents/endpoints/serializers/workflow_engine_data_condition.py.
Reproduction
Note: the following is reproducible with any org that has been flipped onto the workflow-engine read path. The provider version below is the one most likely to be in use.
- On an organization with the workflow-engine read flags enabled, create an issue alert via the API or UI with the following minimal shape:
{
"actions": [
{
"id": "sentry.mail.actions.NotifyEmailAction",
"targetType": "IssueOwners",
"fallthroughType": "ActiveMembers"
}
],
"conditions": [
{"id": "sentry.rules.conditions.first_seen_event.FirstSeenEventCondition"}
],
"filters": [
{
"id": "sentry.rules.filters.tagged_event.TaggedEventFilter",
"match": "eq",
"key": "<some-tag-key>",
"value": "<some-value>"
}
],
"actionMatch": "any",
"filterMatch": "all",
"frequency": 30,
"name": "<rule-name>"
}
- Immediately
GET the same rule via /api/0/projects/<org>/<project>/rules/<id>/ (or the project alert-rule endpoint).
Observed:
actions[0].fallthroughType is missing from the response.filters[0].value is missing from the response.- The condition object shape may differ from the request.
Expected:
- The response echoes back the exact
fallthroughType, value, and condition object shape that was written, identical to the legacy RuleSerializer behavior for the same rule shape.
- Same flow via the
jianyuan/terraform-provider-sentry provider (any 0.14.x) using sentry_issue_alert with actions_v2.notify_email, filters_v2.tagged_event, and conditions_v2.{first_seen_event,event_frequency,regression_event,reappeared_event} produces:
Error: Provider produced inconsistent result after apply
When applying changes to sentry_issue_alert.<resource>, provider
"provider[\"registry.terraform.io/jianyuan/sentry\"]" produced an
unexpected new value: .actions_v2[N].notify_email.fallthrough_type: was
cty.StringVal("ActiveMembers"), but now null.
Analogous errors are produced for filters_v2[N].tagged_event.value and conditions_v2[N].*.
Impact
- Terraform applies against affected orgs fail per-resource and cannot complete cleanly. Customers managing their alerting via Terraform are blocked on any change once their org is flipped onto the workflow-engine read path.
- Because so many resources are flagged as drifting at once, the cascade of PUT requests increases Slack channel-validation calls and pushes some orgs into Slack-side rate limiting (separate from this issue, but downstream of it).
- The data on Sentry's side is correct. The regression is purely in the GET response shape.
Suggested direction
The intent of the workflow-engine read serializers is to be a drop-in replacement for the legacy RuleSerializer / AlertRuleSerializer. To honor that, the serializers should round-trip every field the corresponding write endpoints accept, including:
- Always emit
fallthroughType for email actions, with the same defaulting behavior the legacy serializer applies to all targetType values (not just Member/Team).
- Always emit
value on tagged_event filters when the stored data condition has it, and avoid silently dropping it during the legacy-rule-to-data-condition migration.
- Match the shape and presence of
conditions_v2 / filters_v2 / actions_v2 sub-objects exactly, including the name fields, so that providers comparing pre/post can converge.
If a fully drop-in fix isn't feasible quickly, please consider rolling the per-method workflow-engine read flags back for affected orgs while patching, since the impact is a hard block on Terraform-managed alerting.
Versions
- Sentry SaaS, US region.
- jianyuan/sentry Terraform provider
0.14.13 (also reproduced on earlier 0.14.x).
- Issue first observed when the org's workflow-engine read flag flipped on.
Related
Summary
After the per-org rollout of the workflow-engine read flags for issue/metric alert endpoints (
organizations:workflow-engine-metric-alert-endpoints-getand the related per-method flags from #112827, #111230, #111330),WorkflowEngineRuleSerializerdoes not round-trip several fields that the corresponding PUT/POST endpoints accept and persist. The values are stored correctly on the server, but the GET response that follows the write does not include them.For automation that compares the post-apply response to the planned write (notably the
jianyuan/terraform-provider-sentryTerraform provider), this surfaces asProvider produced inconsistent result after applyerrors and prevents Terraform applies from succeeding, even though the underlying alert configuration on Sentry's side is unchanged.Affected fields (observed)
For
sentry_issue_alertresources (actions_v2,conditions_v2,filters_v2):actions_v2[*].notify_email.fallthrough_typefortarget_type = "IssueOwners"fallthroughType: "ActiveMembers"(or"AllMembers"/"NoOne").fallthroughTypeentirely forIssueOwnerstargets.WorkflowEngineRuleSerializeronly synthesizes a defaultfallthroughTypeforMember/Teamtargets, seesrc/sentry/api/serializers/models/rule.pyaround the# XXX: add default fallthroughType for email Team/Member actionsblock. There is no equivalent path that re-emits a storedfallthroughTypeforIssueOwnerswhen the underlyingAction.data["fallthrough_type"]is empty.filters_v2[*].tagged_event.valuevalue: "<some string>".valuefrom the serializedtagged_eventfilter.create_tagged_event_data_conditioninsrc/sentry/workflow_engine/migration_helpers/issue_alert_conditions.pyonly writesvaluewhen present in the source blob, and the read-side equivalent does not always re-emit it.conditions_v2[*].first_seen_event/regression_event/event_frequency/reappeared_eventname, and forevent_frequencythevalue/interval/comparison_typefields).was <object>, but now null(orwas null, but now <object>).For
sentry_metric_alertresources, separately (this one is arguably "by design" but is the same class of round-trip mismatch):resolve_thresholdis now always returned with the auto-calculated value derived from the lowest-severity trigger'salert_threshold, where the legacy serializer returnednullif the user never set it explicitly. This also produces drift in the Terraform provider, though it is not a hard apply failure since PUT-with-null is accepted and re-derived. Code reference:WorkflowEngineDataConditionSerializer.resolve_thresholdinsrc/sentry/incidents/endpoints/serializers/workflow_engine_data_condition.py.Reproduction
Note: the following is reproducible with any org that has been flipped onto the workflow-engine read path. The provider version below is the one most likely to be in use.
{ "actions": [ { "id": "sentry.mail.actions.NotifyEmailAction", "targetType": "IssueOwners", "fallthroughType": "ActiveMembers" } ], "conditions": [ {"id": "sentry.rules.conditions.first_seen_event.FirstSeenEventCondition"} ], "filters": [ { "id": "sentry.rules.filters.tagged_event.TaggedEventFilter", "match": "eq", "key": "<some-tag-key>", "value": "<some-value>" } ], "actionMatch": "any", "filterMatch": "all", "frequency": 30, "name": "<rule-name>" }GETthe same rule via/api/0/projects/<org>/<project>/rules/<id>/(or the project alert-rule endpoint).Observed:
actions[0].fallthroughTypeis missing from the response.filters[0].valueis missing from the response.Expected:
fallthroughType,value, and condition object shape that was written, identical to the legacyRuleSerializerbehavior for the same rule shape.jianyuan/terraform-provider-sentryprovider (any 0.14.x) usingsentry_issue_alertwithactions_v2.notify_email,filters_v2.tagged_event, andconditions_v2.{first_seen_event,event_frequency,regression_event,reappeared_event}produces:Analogous errors are produced for
filters_v2[N].tagged_event.valueandconditions_v2[N].*.Impact
Suggested direction
The intent of the workflow-engine read serializers is to be a drop-in replacement for the legacy
RuleSerializer/AlertRuleSerializer. To honor that, the serializers should round-trip every field the corresponding write endpoints accept, including:fallthroughTypefor email actions, with the same defaulting behavior the legacy serializer applies to alltargetTypevalues (not justMember/Team).valueontagged_eventfilters when the stored data condition has it, and avoid silently dropping it during the legacy-rule-to-data-condition migration.conditions_v2/filters_v2/actions_v2sub-objects exactly, including thenamefields, so that providers comparing pre/post can converge.If a fully drop-in fix isn't feasible quickly, please consider rolling the per-method workflow-engine read flags back for affected orgs while patching, since the impact is a hard block on Terraform-managed alerting.
Versions
0.14.13(also reproduced on earlier 0.14.x).Related
fallthroughTypemigration and fixes)notify_email.fallthrough_typefield)