Document how HTTP/file-transfer streaming chunk size is handled in thin-edge.io

[dtimotijevic]

Is your documentation request related to a problem? Please describe.

It is currently unclear from the thin-edge.io documentation how HTTP/file-transfer streaming chunk sizes are handled, and whether they are configurable.

The configuration documentation exposes settings such as http.bind.*, http.client.*, and mapper MQTT payload limits such as c8y.mapper.mqtt.max_payload_size, but there does not appear to be a documented setting for HTTP upload/download stream chunk size. This can be confusing when investigating memory usage, slow networks, proxy limitations, or large file transfers.

This would be especially helpful to understand for very resource-constrained devices, where RAM, CPU, disk I/O, and network throughput are limited and streaming/buffering behavior can directly affect stability.

Describe the improvement you'd like

Documentation on how streaming chunk size is handled for file transfer and upload/download paths.

Specifically, it would be helpful to clarify:

• Whether HTTP/file-transfer streaming chunk size is configurable via tedge.toml or tedge config
• If it is not configurable, which defaults are used
• Which components have which chunking behaviour (C8Y proxy, file transfer, tedge uploads)

A short note in the File Transfer Service / C8Y Proxy documentation and/or configuration reference would be enough, for example:

thin-edge.io does not currently expose a configuration option for HTTP streaming chunk size. File transfer streaming uses the defaults of the underlying Rust HTTP/streaming libraries. MQTT mapper payload-size settings do not affect HTTP file-transfer chunking.

[reubenmiller]

Thanks @dtimotijevic for the ticket. Can you give us an idea of the rough dimensions of the very resource-constrained devices? Are we talking <32MB RAM here?

[didier-wenzek]

There is indeed no documentation about the size of the chunks used during download. We can always improve the docs, but there is no much to say: the size of these chunks is no configurable and even not constant during a download or across downloads.

• During a file transfer the file is not materialized in memory but read chunk by chunk - each chunk being persisted on disk before reading the next chunk. See https://github.com/reubenmiller/thin-edge.io/blob/ae54b5c4fdbb6bfb1f6e97293d87e60a2fd2ff34/crates/common/download/src/download.rs#L398
• The actual size of these chunks is controlled by reqwest so a single network read makes a chunk.

[didier-wenzek]

That said, I wonder if you have a more critical issue behind this documentation request. Can you tell more about the behavior or issues your observed and would like to improve or solve with a better understanding of the download logic?

[dtimotijevic]

@reubenmiller @didier-wenzek
Thanks both.

The device class we are looking at has less than 50 MB RAM available during runtime, so our concern is not whether the full file is materialized in memory, but whether the download path has a predictable upper bound for transient memory usage while streaming.

From Didier's explanation and the linked reqwest discussion, it sounds like thin-edge.io writes the response incrementally to disk, but the size of each in-memory chunk is effectively determined by reqwest/the underlying transport/TLS/HTTP framing and is therefore not configurable and not guaranteed to be constant.

I tested this and measured the RAM usage during downloads. In my tests, I did not see RAM usage grow by more than around 5 MB, which is encouraging. But I am not sure whether this is a guaranteed property of the current implementation or just a coincidence based on the server, protocol, TLS behavior, file size, network conditions, etc.

So the main question is whether thin-edge.io can guarantee, or be configured to guarantee, a maximum buffer size during downloads - for example by limiting reads/writes to a fixed-size buffer such as 8 KB, 16 KB, etc. If not, then documenting that the chunk size is delegated to reqwest and is not bounded/configurable by thin-edge.io would already be useful.

We have seen OOM issues in our own software caused by uploads and in all places that we control we have explicitly bounded the upper limit of memory usage. We are trying to understand whether downloads / uploads in thin-edge.io can provide a similar predictable bound.

[didier-wenzek]

Thin-edge does not guarantee any maximum allocated memory size - for the process as a whole nor for specific operations. We are doing our best to reduce memory pressure. For instance, I would have flagged as a bug "a download materializing the complete file in memory". But we provide no guarantees.

In this precise case, even if thin-edge doesn't have predictable memory-usage bounds during file transfer nor provide a way to configure a maximum buffer size - such bounds exist underneath (e.g hyper max_frame_size is inherited by reqwest and thin-edge).

About OOM during file transfer, the major source of issue is trying to materialize files in memory. I would not be worried too much here. However if you experienced OOMs with thin-edge, please, create a ticket and we will investigate.