[BUG] Dashboard changes on cloud not syncing to edge.

[jrkubic]

Describe the bug
I have tenant admin level dashboards created on our PE Cloud environment. We have an edge instance we're trying to test the full work flow of. Right now, I have everything syncing initially when the group is assigned, but entity updates that should be syncing asynchronously .

Thingsboard PE 4.3.1.1
Edge 4.3.1
Gateway 3.8.3

• own setup
  • Deployment: Single thingsboard-pe container running all services (core, rule-engine, etc.)
  • Deployment type: docker compose

To Reproduce
Steps to reproduce the behavior:
Set up edge instance
Configure platform for multi-tenant dashboard views. Not dashboard for each customer.
Create and share permissions of group intended to be functional on Edge.
Find that minor changes only sync on manual edge sync or destruction and re-assignment of group. Not occurring on change as planned

Expected behavior
A minor changes and entity updates should trigger syncs asynchronously.
Not finding the event hit edge on shared groups, only entities assigned directly to the edge, not shared groups as expected.

Additional context
For some reason, our cloud events log on edge is also not working.

Edit Found Edge to Cloud syncs work as expected. Must be a cloud outbound config.

---

Disclaimer

We appreciate your contribution whether it is a bug report, feature request, or pull request with improvement (hopefully). Please comply with the Community ethics policy, and do not expect us to answer your requests immediately. Also, do not treat GitHub issues as a support channel.

[jrkubic]

I believe the main problem is that asynchronous updates to entities like the dashboard ("ENTITY_UPDATED") only propagate down to entities that are assigned directly to it, not shared groups**it should be working with shared groups. Looking into it now. Cloud -> Edge issue as Edge -> cloud works fine.

So, Edge can sync to cloud no problem but Cloud syncing to Edge isn't working because the dash isn't assigned directly to the edge, it's just in the shared group.

[volodymyr-babak]

Hi @jrkubic, thanks for the detailed report and the follow-up narrowing it down to cloud -> edge.

Before I dig further, I want to make sure I'm reproducing the exact same setup you have. Could you confirm a few things about your environment?

Setup details we'd need to reproduce

1. Group setup on the cloud

   • Is the dashboard group an actual Dashboard group (Dashboard Groups section), or a different group type?
   • Is this group assigned to the edge (Edge management -> your edge -> Manage dashboard groups -> Assign), or only shared with users/roles via Group Permissions / Roles?
     • These are two different operations, and only the first one makes the group's contents sync to the edge.
   • Could you share a screenshot of the edge's "Dashboard Groups" tab so we can see which groups are actually assigned to the edge?

2. Where the dashboard lives

   • Is the dashboard explicitly added to that custom group, or is it only in the tenant-level "All" group?
   • Is the dashboard owned by the tenant, or by a customer? (You mentioned "tenant admin level dashboards" - just want to confirm ownerId == tenantId.)

3. Reproduction sequence

   • Exact step-by-step of what you do: create dashboard -> add to group -> assign group to edge -> update dashboard -> observe missing update? Or a different order?
   • Roughly how long after the group is assigned do you start updating the dashboard? (Helps us tell whether it's a race or a steady-state issue.)

4. Deployment

   • You mentioned microservices on Docker Compose - how many tb-core / tb-rule-engine instances?
   • Are you using Redis caching in front of tb-core? If yes, standalone, sentinel, or cluster?

5. Logs from a failing case
   If you can grab cloud-side TRACE logs around the time of a failing dashboard update for these classes:

   • org.thingsboard.server.service.edge.EdgeEventSourcingListener
   • org.thingsboard.server.service.edge.rpc.processor.BaseEdgeProcessor
   • org.thingsboard.server.dao.edge.BaseRelatedEdgesService

   …that would tell us immediately whether the event is being produced on the cloud and just not reaching the edge, or never being produced at all.

Why I'm asking

Reading the description again, the phrasing "create and share permissions of group intended to be functional on Edge" is a little ambiguous. In ThingsBoard PE there are two distinct operations:

• Sharing a group (Group Permissions / Roles) - grants other users/roles access to the group. This does not make the group's contents sync to an edge.
• Assigning a group to an edge (Edge -> Manage dashboard groups -> Assign group) - this is what causes the group's contents to sync to the edge.

You also wrote "everything syncing initially when the group is assigned," which sounds like the group is assigned to the edge - but I'd like to confirm that explicitly before I commit to a root cause, because the fix is different depending on which case it actually is.

Once I have those details I can either reproduce locally or point at the specific code path that's misbehaving.

About the empty "Cloud Events" log on the edge

This is expected with your deployment. The Cloud Events page only shows data when the edge uses the in-memory queue (events are persisted as rows in the PostgreSQL DB and read back for the UI). On a Kafka-backed queue the events flow through Kafka topics instead of the DB table, so the UI has nothing to render. It is unrelated to the dashboard sync issue.

[jrkubic]

Setup details we'd need to reproduce

1. Group setup on the cloud

   • Is the dashboard group an actual Dashboard group (Dashboard Groups section), or a different group type?

   • Is this group assigned to the edge (Edge management -> your edge -> Manage dashboard groups -> Assign), or only shared with users/roles via Group Permissions / Roles?

     • These are two different operations, and only the first one makes the group's contents sync to the edge.

   • Could you share a screenshot of the edge's "Dashboard Groups" tab so we can see which groups are actually assigned to the edge?

Both. Dashboard group is an actual group and is assigned to the edge instance.

2. Where the dashboard lives

   • Is the dashboard explicitly added to that custom group, or is it only in the tenant-level "All" group?
   • Is the dashboard owned by the tenant, or by a customer? (You mentioned "tenant admin level dashboards" - just want to confirm ownerId == tenantId.)

Tenant owned dashboard we're trying to share with different customers. I have put it into that custom group.

3. Reproduction sequence

   • Exact step-by-step of what you do: create dashboard -> add to group -> assign group to edge -> update dashboard -> observe missing update? Or a different order?
   • Roughly how long after the group is assigned do you start updating the dashboard? (Helps us tell whether it's a race or a steady-state issue.)

Those steps are about right. Waiting a decent amount of time. Tested after hours-days.

4. Deployment

   • You mentioned microservices on Docker Compose - how many tb-core / tb-rule-engine instances?
   • Are you using Redis caching in front of tb-core? If yes, standalone, sentinel, or cluster?

Sorry, miscommunication, we're running a monolithic deployment, not microservices.
A single thingsboard-pe container running all services (core, rule engine, transport) bundled.

Redis caching: Not configured. No Redis service in either compose file, no CACHE_TYPE or REDIS_* env vars. Using TB's default in-memory caffeine cache.

5. Logs from a failing case
   If you can grab cloud-side TRACE logs around the time of a failing dashboard update for these classes:

   • org.thingsboard.server.service.edge.EdgeEventSourcingListener
   • org.thingsboard.server.service.edge.rpc.processor.BaseEdgeProcessor
   • org.thingsboard.server.dao.edge.BaseRelatedEdgesService

Will come back and add as soon as I can..

Why I'm asking

Reading the description again, the phrasing "create and share permissions of group intended to be functional on Edge" is a little ambiguous. In ThingsBoard PE there are two distinct operations:

• Sharing a group (Group Permissions / Roles) - grants other users/roles access to the group. This does not make the group's contents sync to an edge.
• Assigning a group to an edge (Edge -> Manage dashboard groups -> Assign group) - this is what causes the group's contents to sync to the edge.

Understood.

You also wrote "everything syncing initially when the group is assigned," which sounds like the group is assigned to the edge - but I'd like to confirm that explicitly before I commit to a root cause, because the fix is different depending on which case it actually is.

Confirmed. When unassigning the group from the edge and assigning it back, I can see the event log fire and dashboard changes sync in seconds. But not on changes or saves as planned.

[jrkubic]

Cloud Role

Edge Role that doesnt sync....

Healthy links.

Update: confirmed related symptom — owner-match enforcement blocks tenant-shared groups on edge
Continuing to debug from my last comment, I found a second symptom that appears to be the same Cloud → Edge sync gap surfacing differently.
Setup:

TB PE 4.3.1.1, Edge 4.3.1.1, Gateway 3.8.3
Tenant-owned dashboard in a tenant-owned dashboard group (** CUSTOMER ** - Site Power User Shared (Dashboard Group))
Group is assigned to the edge (not just shared via permissions)
Group + dashboard arrive on edge correctly on initial assign, confirmed visible in Edge UI as tenant admin

Symptom A — role permission edits to default groups don't sync Cloud → Edge:

On Cloud, on a sub-customer's Customer Users group, I added a Group permission referencing role ** CUSTOMER ** Site User Read Only targeting the tenant-owned dashboard group above.
On Edge, the same sub-customer's Customer Users group still shows only the autogenerated Customer User (Generic) role. The added Group permission never appears.
DL Lag is 0, no failed events, no errors in edge logs.

Symptom B — Share entity group on tenant-owned dashboard groups doesn't propagate to edge:

On Cloud, used Share entity group → sub-customer → All users → Read on the tenant-owned dashboard group.
Customer users on edge still get Access Forbidden for dashboards in that group.
No event reflecting the share appears on the edge side.

Symptom C — owner-match rule blocks the standard Cloud workaround:

Attempting to add a sub-customer-owned user directly to a tenant-owned user group (the pattern that works for shared dashboards in pure Cloud TB PE) returns:

POST /api/entityGroup/{groupId}/addEntities 400 Bad Request
"Unable to add entity with other owner than group. Entity id: {userId}"

This is correct enforcement, but it means the only Cloud-side path to grant customer users access to tenant-owned shared dashboards in TB PE Edge is via role permissions on default groups (Symptom A) or Share entity group (Symptom B) — both of which don't sync to edge.

Net effect:
The tenant-owned-dashboard-shared-with-customer-users pattern that works in pure Cloud TB PE is functionally broken in 4.3.1.1 Edge. Initial assignment syncs the group/dashboard, but no permission-grant mechanism reaches the edge afterward, leaving customer users on the edge with no path to access the synced dashboards. Per-customer-owned dashboards appear to be the only working pattern in this version.
Will add the requested TRACE logs from EdgeEventSourcingListener, BaseEdgeProcessor, and BaseRelatedEdgesService as soon as I can.

[jrkubic]

Just appears to be specifically Cloud side UI dashboard changes. Custom widgets that are served get synced cloud -> edge instantly.

Would this behavior reflect what would be happening with a dashboard of size ~18MB? I know there is an issue out there explaining 16MB dashboard syncs break.
But I know initial syncs work fine and same with custom extension changes?

[jrkubic]

@volodymyr-babak

Update — TRACE logs captured, isolated to Cloud→Edge delivery layer

I was able to capture the requested TRACE logs after working through some logback config issues (PE 4.3.1.1 docker monolith, no conf/ bind-mount in the compose, so changes required a container restart). Final config used:

<logger name="org.thingsboard.server.service.edge.EdgeEventSourcingListener" level="TRACE"/>
<logger name="org.thingsboard.server.service.edge.rpc.processor.BaseEdgeProcessor" level="TRACE"/>
<logger name="org.thingsboard.server.dao.edge.BaseRelatedEdgesService" level="TRACE"/>

Test setup

• Cloud: TB PE 4.3.1.1, single-container docker monolith, no Redis (Caffeine cache), Kafka-backed queue
• Edge: TB Edge PE 4.3.1.1, separate stack (postgres + cassandra + kafka + mytbedge), version-matched
• Dashboard: tenant-owned, ~18 MB, customerId = 13814000-1dd2-11b2-8080-808080808080 (NULL_UUID), shared via tenant-owned dashboard group, group assigned to edge
• Edge UUID: 58071a60-3783-11f1-ba79-932e36e4903b
• Test action: trivial save on the dashboard via Cloud UI at 2026-05-22 20:21:11 UTC

Cloud side — event IS generated and pushed

Full sequence captured for entityId c40fe9b0-da88-11f0-8df5-4dce13e974e8:

20:21:11,552 [http-nio-0.0.0.0-8080-exec-41] TRACE EdgeEventSourcingListener
  SaveEntityEvent called: SaveEntityEvent(tenantId=46f09b60-..., 
  entity=Dashboard [customerId=13814000-1dd2-11b2-8080-808080808080, ...], 
  oldEntity=null, entityId=c40fe9b0-..., created=false, broadcastEvent=null)

20:21:11,565 [tb-edge-consumer-70-thread-4] TRACE BaseRelatedEdgesService 
  Executing findEdgeIdsByTenantIdAndGroupEntityId, tenantId [46f09b60-...], 
  entityId [c40fe9b0-...], pageLink [PageLink(pageSize=1000, page=0, ...)]

20:21:11,579 [DbCallbackExecutorService-0-2] DEBUG BaseEdgeProcessor 
  Pushing event to edge queue. tenantId [46f09b60-...], 
  edgeId [58071a60-3783-11f1-ba79-932e36e4903b], 
  type[DASHBOARD], action [UPDATED], entityId [c40fe9b0-...], 
  body [null], entityGroupId [null]

The cloud-side pipeline works end to end: SaveEntityEvent fires, the related-edges lookup resolves to the correct edge, and BaseEdgeProcessor pushes a DASHBOARD/UPDATED event to the edge queue for that edge. No errors, warnings, or stack traces in the surrounding window.

Edge side — event never arrives

On the Edge container in the same window:

• gRPC channel is healthy — uplinks every ~25 seconds via GrpcCloudEventUplinkSender
• TbCoreConsumerStats.cloudNfs = 0 consistently across every 1-minute sample for 10+ minutes spanning the save
• No reference to c40fe9b0-... in Edge logs after 2026-05-07 (initial assignment)
• No "DASHBOARD" or "UPDATED" entries appear post-save

So the channel is bidirectional and uplink works, but no Cloud→Edge notification is being received for this UPDATED event.

Possibly-related historical artifact

The 13 occurrences of c40fe9b0-... in Edge logs are all from initial assignment on 2026-05-07. One of those entries may be relevant:

2026-05-07 00:36:12,152 [grpc-default-executor-154] WARN BaseRelationProcessor 
  [46f09b60-...] Skipping relating update msg because from/to entity 
  doesn't exists on edge, entity: {
    "from":{"entityType":"CUSTOMER","id":"ae42a5f0-2520-11f0-94ae-1d22e18ebf65"},
    "to":{"entityType":"DASHBOARD","id":"c40fe9b0-..."},
    "type":"Contains","typeGroup":"COMMON","version":524047
  }

The Edge is rejecting the Customer→Dashboard Contains relation because Customer ae42a5f0-2520-11f0-94ae-1d22e18ebf65 doesn't exist on the Edge. This may be related to symptoms A/B from my earlier comment — if the customer that "contains" the dashboard via the entity group never propagates to the Edge, the relation chain stays incomplete.

Summary

Stage	Result
Cloud emits SaveEntityEvent on dashboard save	✓
Cloud's BaseRelatedEdgesService identifies target edge	✓
Cloud's BaseEdgeProcessor pushes DASHBOARD/UPDATED to queue	✓
Edge gRPC channel healthy, uplinks flowing	✓
Edge receives cloud notification (cloudNfs)	✗ (stays at 0)
Edge logs reference to dashboard UUID for the save	✗

The cloud-side code paths fire correctly. The gap is between the BaseEdgeProcessor queue push and the Edge's TbCoreConsumerStats.cloudNfs counter — i.e., either the cloud isn't actually delivering the queued event over gRPC, or the edge isn't consuming/counting it.

Happy to dig further with TRACE on the cloud-side gRPC delivery classes or on the edge-side cloud-event consumer — let me know what would be most useful and which class names to target.

[volodymyr-babak]

Hi @jrkubic, thanks for the very thorough TRACE capture — that pinpointed the exact handoff to look at. I'm fairly confident this is a gRPC message-size limit issue on the cloud→edge side, not a sourcing/relation bug. Let me walk through what's happening.

Root cause

The cloud-side pipeline you traced is doing everything right: SaveEntityEvent fires, BaseRelatedEdgesService resolves the edge, BaseEdgeProcessor builds and queues the DASHBOARD/UPDATED downlink. The handoff that's failing is one layer further down, inside EdgeGrpcSession, which has a pre-send guard that drops downlinks larger than the edge's advertised max inbound message size.

The edge advertises this limit during the gRPC handshake (EdgeGrpcClient#connect → ConnectRequestMsg.maxInboundMessageSize). The cloud stores it as clientMaxInboundMessageSize and enforces it here, in EdgeGrpcSession#scheduleDownlinkMsgsPackSend:

for (DownlinkMsg downlinkMsg : copy) {
    if (clientMaxInboundMessageSize != 0
            && downlinkMsg.getSerializedSize() > clientMaxInboundMessageSize) {
        String message = String.format(
            "Downlink msg size %s exceeds client max inbound message size %s. " +
            "Please increase value of CLOUD_RPC_MAX_INBOUND_MESSAGE_SIZE env variable on the edge and restart it.",
            downlinkMsg.getSerializedSize(), clientMaxInboundMessageSize);
        log.error("[{}][{}][{}] {} Message {}", tenantId, edge.getId(), sessionId, message, downlinkMsg);
        // fires EdgeCommunicationFailureTrigger notification
        // records DOWNLINK_MSGS_PERMANENTLY_FAILED stat
        sessionState.getPendingMsgsMap().remove(downlinkMsg.getDownlinkMsgId()); // dropped
    } else {
        sendDownlinkMsg(ResponseMsg.newBuilder().setDownlinkMsg(downlinkMsg).build());
    }
}

Default on both sides:

Side	Property	Default
Edge	CLOUD_RPC_MAX_INBOUND_MESSAGE_SIZE	4194304 (4 MB)
Cloud	EDGES_RPC_MAX_INBOUND_MESSAGE_SIZE	4194304 (4 MB)

Your dashboard is ~18 MB, so the serialized DownlinkMsg for a DASHBOARD/UPDATED event sits well over 4 MB and gets dropped before sendDownlinkMsg is ever called. That's exactly why your edge's cloudNfs counter stays at 0 — the message never leaves the cloud.

Why this didn't show up in your TRACE logs

The log.error above is on org.thingsboard.server.service.edge.rpc.EdgeGrpcSession, not on any of the three classes you turned up to TRACE. Could you check two more things on the cloud side:

1. ERROR-level lines from org.thingsboard.server.service.edge.rpc.EdgeGrpcSession around your test save at 20:21:11. If my theory is right you'll see something like Downlink msg size 18xxxxxx exceeds client max inbound message size 4194304…
2. Cloud UI → Notification center → look for Edge communication failure notifications. The drop path fires EdgeCommunicationFailureTrigger, which produces a user-visible notification.

Why the initial assignment worked but updates don't

The initial-sync path (EdgeGrpcSession#doSync) and the event-driven path (EdgeGrpcSession#processEdgeEvents) both funnel through the same sendDownlinkMsgsPack → size-check code. So they behave identically with respect to this 4 MB ceiling.

The simplest explanation is that on 2026-05-07 when you first assigned the group, the dashboard was smaller, and you've been adding widgets/HTML/assets since. A quick way to confirm: unassign the dashboard group from the edge, then re-assign it now. If my theory is right, the initial-sync attempt will also fail this time, with the same error in the cloud log — which is the opposite of what you saw on 2026-05-04 when re-assignment did push the dashboard down.

Fix

On the edge container, set:

CLOUD_RPC_MAX_INBOUND_MESSAGE_SIZE=33554432   # 32 MB

…and restart the edge. To stay symmetric and avoid the inverse problem on uplinks involving large entities, also set on the cloud container:

EDGES_RPC_MAX_INBOUND_MESSAGE_SIZE=33554432   # 32 MB

…and restart the cloud. 32 MB is comfortable headroom for an 18 MB payload; pick any round value above ~2× the size of your largest dashboard.

After the restart, do another trivial save on the dashboard. You should see the DASHBOARD/UPDATED event flow all the way through to the edge and the dashboard refresh in the edge UI within seconds.

A note on symptoms A and B (role permission edits / Share entity group)

These are real, but they are a separate issue from the dashboard sync — different code path, different root cause. Tenant-owned group permission changes (Group Permissions on default Customer Users groups + the "Share entity group" action) are not currently emitted as edge events for groups assigned to the edge. We'll need a separate ticket to extend EdgeEventSourcingListener / role-permission processing on the cloud side to cover that case. Let's verify the size fix resolves the dashboard sync first, and once that's confirmed I'll either reuse this issue or open a follow-up specifically for symptoms A/B.

On the empty Cloud Events log

Just to re-confirm from my earlier reply: that's expected for a Kafka-backed queue. The Cloud Events page only renders data when the edge runs with the in-memory queue (events are persisted as rows in the local DB and read back for the UI). On Kafka the events flow through topics instead, so the UI is empty by design. Unrelated to the sync issue.

---

Could you give the env-var change a try and share back the result, plus those two extra log/notification checks if you don't mind? That'll let us close out the dashboard-update part of this issue cleanly and move on to symptoms A/B.

[jrkubic]

Update — applied the fix, dashboard sync still broken. Additional diagnostics below.

Thanks @volodymyr-babak for the detailed root-cause analysis. I confirmed your diagnosis in the cloud UI on Friday — I had Edge communication failure notifications with the exact Downlink msg size XXXXXXX exceeds client max inbound message size 4194304 error you predicted. Strong evidence the size guard was being hit.

I then applied the recommended fix on both sides, but the sync remains broken. Full details of what I tried and what I'm now seeing:

Configuration changes applied

Edge (docker-compose.yml, then docker compose up -d mytbedge to force container recreate):

CLOUD_RPC_MAX_INBOUND_MESSAGE_SIZE: 33554432

Cloud (docker-compose.yml, then docker compose up -d thingsboard-pe):

EDGES_RPC_MAX_INBOUND_MESSAGE_SIZE: 33554432

Both verified live inside the running containers:

$ docker exec thingsboard-mytbedge-1 env | grep -i "CLOUD_RPC_MAX"
CLOUD_RPC_MAX_INBOUND_MESSAGE_SIZE=33554432

$ docker exec ubuntu-thingsboard-pe-1 env | grep -i "EDGES_RPC_MAX"
EDGES_RPC_MAX_INBOUND_MESSAGE_SIZE=33554432

I also additionally tried CLOUD_RPC_GRPC_MAX_INBOUND_MESSAGE_SIZE on edge as belt-and-suspenders. No change.

Verification — Edge UI

Same dashboard, trivial save in Cloud UI, wait 30+ seconds, refresh Edge UI: change does not appear. Identical behavior to pre-fix.

Cloud TRACE — pipeline still fires correctly

Re-enabled the same three TRACE loggers from my previous capture. The exact same sequence as last time still fires on every save:

TRACE EdgeEventSourcingListener
  SaveEntityEvent called: SaveEntityEvent(tenantId=46f09b60-..., 
  entity=Dashboard [tenantId=..., customerId=13814000-1dd2-11b2-8080-808080808080], 
  entityId=c40fe9b0-da88-11f0-8df5-4dce13e974e8, created=false)

TRACE BaseRelatedEdgesService
  Executing findEdgeIdsByTenantIdAndGroupEntityId, tenantId [46f09b60-...], 
  entityId [c40fe9b0-...]

DEBUG BaseEdgeProcessor
  Pushing event to edge queue. tenantId [46f09b60-...], 
  edgeId [58071a60-3783-11f1-ba79-932e36e4903b], 
  type[DASHBOARD], action [UPDATED], entityId [c40fe9b0-...]

So source-event sourcing, edge resolution, and queue push are all firing. No exceeds error at ERROR or any other level after the env var change.

No new Edge communication failure notifications

Cloud UI → Notification center: no new failure notifications since Friday, which is when the size-guard errors stopped appearing. So the size guard is no longer being hit.

Edge side — cloudNfs still 0

On every save, the edge logs show normal gRPC channel health (uplinks flow every ~1-2 seconds, no errors, cloud-manager-uplink thread active). But TbCoreConsumerStats.cloudNfs stays at 0:

INFO TbCoreConsumerStats - Core Stats: totalMsgs = [1] ... cloudNfs = [0] ...
INFO GrpcCloudEventUplinkSender - Sending of [1] uplink msg(s) took 31 ms.

No reference to dashboard UUID c40fe9b0-... in Edge logs at any point after the cloud-side queue push. Channel is bidirectional and uplink-healthy, but cloud→edge notifications for this entity are not arriving.

Also tested — HAProxy in TCP mode

In case HAProxy in mode http with alpn h2,http/1.1 was enforcing its own HTTP/2 frame buffer limits on the 18 MB downlink (separate from TB's size guard), I switched both listen edges-rpc-in and backend grpc_backend to mode tcp, keeping TLS termination at HAProxy with alpn h2 advertised at the TLS layer only (no HTTP/2 parsing). Validated config, reloaded HAProxy, edge reconnected cleanly through the new TCP-mode listener (no more ALPN-negotiation failures).

Behavior is identical — same cloudNfs = 0 after every save, same trace-log signature on cloud, same absence of any reference to the dashboard UUID in edge logs.

What I think is happening

Either:

1. The size guard fix is working but a different downstream silent-drop is occurring between BaseEdgeProcessor.pushEvent and the actual gRPC sendDownlinkMsg call
2. The gRPC sendDownlinkMsg call succeeds but the edge receiver silently drops the message
3. There's an additional size/frame limit somewhere I haven't found (gRPC's maxInboundMessageSize at the channel level, separately from TB's EDGES_RPC_MAX_INBOUND_MESSAGE_SIZE?)

Given the cloud pipeline fires identically to a working flow and no errors surface at any level, the silent-drop behavior strongly suggests a path that isn't logged.

Environment summary

• TB PE 4.3.1.1, Edge PE 4.3.1.1EDGEPE, Gateway 3.8.3
• Cloud: monolithic docker compose, Kafka-backed queue, no Redis (Caffeine cache), HAProxy fronting :7070
• Edge: separate stack on M.2 SSD host, postgres+cassandra+kafka+mytbedge, Kafka-backed queue
• Dashboard: tenant-owned, ~18 MB, in tenant-owned dashboard group assigned to edge
• Edge UUID: 58071a60-3783-11f1-ba79-932e36e4903b
• Dashboard entityId: c40fe9b0-da88-11f0-8df5-4dce13e974e8

What I'd like guidance on

1. Are there any additional TRACE loggers further down the gRPC send path (EdgeGrpcSession#scheduleDownlinkMsgsPackSend, EdgeGrpcSession#sendDownlinkMsg, or the message-pack consumer thread) that would surface a silent drop after the BaseEdgeProcessor.pushEvent queue write?
2. Is there a second size limit I should be aware of (e.g. gRPC channel-level maxInboundMessageSize separate from EDGES_RPC_MAX_INBOUND_MESSAGE_SIZE)?
3. Anything in your testing of the size-guard fix that you'd recommend I verify on my end?

Happy to run any additional TRACE captures or instrumentation. Just need to know which classes/methods to target.

[jrkubic]

Seems to be widget configuration/depth of custom widgets.
When I change the dashboard title, I do successfully get full dashboard syncs.
But when changing something very small (ie. html/markdown code in widgets), those changes don't ever sync.

[volodymyr-babak]

Hi @jrkubic, thanks for the additional data point — the title-saves-work / widget-saves-don't observation is the key clue that narrows this down. Let me share what I found re-reading the code at v4.3.1.1 and a sharper plan.

What I now believe is happening

There is no separate code path for "save title" vs "save widget HTML" — they both go through POST /api/dashboard → DashboardServiceImpl.doSaveDashboard → SaveEntityEvent, and the cloud→edge converter (DashboardEdgeProcessor.convertEdgeEventToDownlink) always serializes the whole dashboard via JacksonUtil.toString(dashboard). There is no diff/patch downlink. So the proto shape is identical for both operations.

The only mechanism that produces divergent sync behavior for the same entity is the payload size crossing the clientMaxInboundMessageSize ceiling — i.e. the 32 MB you set. Title saves and widget-HTML saves both serialize the full dashboard, but the dashboard's current configuration may be larger now than it was on Friday when you verified the size-guard errors had stopped. Saves that build a > 32 MB proto get silently dropped at EdgeGrpcSession.java:558, exactly the same drop path as before, just hitting a higher ceiling.

Two things make it hard to spot:

1. The error log line lives on org.thingsboard.server.service.edge.rpc.EdgeGrpcSession, not on any of the three TRACE loggers you have enabled. So your TRACE captures look identical for "size guard tripped" vs "successfully sent" — the guard fires after BaseEdgeProcessor pushes to queue.
2. The Edge communication failure notifications in the UI can be missed (throttled, marked read, or simply scrolled past). The cloud log file is the authoritative source.

Three checks that will pin this down in ~5 minutes

1. Grep the cloud log directly for the size-guard error, not the UI:

docker exec ubuntu-thingsboard-pe-1 \
  grep "exceeds client max inbound message size" \
  /var/log/thingsboard/thingsboard.log

If anything shows up dated after the 32 MB env var change, the guard is still tripping.

2. Measure the actual dashboard configuration size on disk:

SELECT length(configuration::text) AS bytes,
       round(length(configuration::text)::numeric / 1048576, 2) AS mb
FROM dashboard
WHERE id = 'c40fe9b0-da88-11f0-8df5-4dce13e974e8';

A DashboardUpdateMsg proto built from this is the configuration size plus title/name/customerId/assignedCustomers/resources/version metadata — call it configuration_size + a few KB of overhead. If mb is above ~30, you are at the edge of the 32 MB guard and a widget-HTML edit that adds even a few hundred KB will push you over.

3. Add the two TRACE loggers that cover the post-queue path:

<logger name="org.thingsboard.server.service.edge.rpc.EdgeGrpcSession" level="TRACE"/>
<logger name="org.thingsboard.server.service.edge.rpc.KafkaEdgeGrpcSession" level="TRACE"/>

After enabling, do one title save and one widget-HTML save. You want to see — for both — these lines on the cloud side around the save timestamp:

• [KafkaEdgeGrpcSession] starting processing edge events ← per-edge Kafka consumer picked up the event
• [EdgeGrpcSession] converting edge event to downlink msg [...] ← DashboardEdgeProcessor was invoked
• [EdgeGrpcSession] entity message processed [DownlinkMsg{...}] ← proto built
• [EdgeGrpcSession] downlink msg(s) are going to be send ← scheduler picked them up

If the widget-HTML save reaches entity message processed but never reaches downlink msg(s) are going to be send, or if you see Downlink msg size N exceeds client max inbound message size 33554432 at ERROR, the size guard is still the cause — just at the higher ceiling.

If size is confirmed as the cause

Bump both env vars to 64 MB and restart:

# edge
CLOUD_RPC_MAX_INBOUND_MESSAGE_SIZE: 67108864
# cloud
EDGES_RPC_MAX_INBOUND_MESSAGE_SIZE: 67108864

This is a stopgap. The underlying design limitation is real: a single DashboardUpdateMsg always carries the full dashboard JSON, so any dashboard at the high end of the size envelope is a fragile sync target. We track this on our side as a chunked-downlink improvement; for now keeping the ceiling well above 2× your largest dashboard is the practical workaround.

Symptoms A and B (role-permission edits / Share entity group)

Confirmed at the source: in EdgeEventSourcingListener the only ENTITY_GROUP save events emitted are entity-group updates themselves; there is no propagation hook for GroupPermission save events on a group that is assigned to an edge, nor for the "Share entity group" action targeting a tenant-owned dashboard group. These would need their own edge-event emission, which is a separate code change.

I'd like to keep this issue focused on the dashboard sync (which we can close once the three checks above resolve it) and open a follow-up specifically for the permission-propagation gap. Happy to file that follow-up myself once we confirm the dashboard side is settled — just want to make sure we are not bundling two unrelated bugs into one fix.

On the "Skipping relating update msg" WARN from 2026-05-07

That log line is from the edge receiving a Customer→Dashboard Contains relation for Customer ae42a5f0-2520-11f0-94ae-1d22e18ebf65 which doesn't exist on the edge. It is a downstream symptom of the same permission-propagation gap (symptoms A/B): the customer that "owns" the share never propagates to the edge, so any relation involving that customer arrives orphaned and is correctly rejected by BaseRelationProcessor. Same root cause as A/B, not related to the dashboard-size issue.

[MazurenkoNick]

Hi @jrkubic , thanks for the detailed report. We’ve been able to reproduce the issue, and a fix is included in the next LTS release.
The problem was that we compared two Dashboards (old & new) and saved it only if the change was detected (BaseDashboardProcessor.java). The bug rised because Dashboard had configuration marked with transient. That's why it wasn't included in equals() and hashCode() methods.

We will also introduce the fix for dashboard synchronisation during edge -> cloud change.

[jrkubic]

Should I work on getting the requested information for @volodymyr-babak? Or, do you guys have all the information you need?
Let me know if you need anything else and thank you for the help!

[MazurenkoNick]

@jrkubic no other information is required for now. You can try to pull new thingsboard/tb-edge:4.3.1.2EDGE image and tell us if it resolves your issue.

The upgrade steps are identical to the 4.3.1.1 version. We will add separate flow for 4.3.1.2 to the portal soon!