Skip to content

refactor: replace the collectd mapper with local flows #4136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
didier-wenzek wants to merge 11 commits into
base: main
Choose a base branch
from
Open

refactor: replace the collectd mapper with local flows #4136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
931caef
refactor: remove dead code
didier-wenzek Apr 16, 2026
4e1bc90
The batcher flow transformer batch messages over time windows
didier-wenzek Apr 16, 2026
81a77df
breaking change! deprecate tedge-mapper-collectd
didier-wenzek Apr 21, 2026
f0b950c
breaking change! deprecate collectd_ext
didier-wenzek Apr 21, 2026
c3a1c5c
breaking change! deprecate tedge-mapper-collectd packaging
didier-wenzek Apr 21, 2026
d4e73b4
Remove batcher dependency on actors
didier-wenzek Apr 21, 2026
55d5bae
remove collectd from tedge diag plugins
didier-wenzek Apr 21, 2026
509369a
docs: remove references to tedge-mapper-collectd
didier-wenzek Apr 21, 2026
414d387
fully deprecate common batcher
didier-wenzek May 26, 2026
121eaf1
Docs: removing sections related to the deprecated collectd mapper
didier-wenzek May 28, 2026
24ad17c
support deprecating a service from maintainer scripts
reubenmiller Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 0 additions & 31 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 0 additions & 2 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ repository = "https://github.com/thin-edge/thin-edge.io"
aws_mapper_ext = { path = "crates/extensions/aws_mapper_ext" }
axum_tls = { path = "crates/common/axum_tls" }
az_mapper_ext = { path = "crates/extensions/az_mapper_ext" }
batcher = { path = "crates/common/batcher" }
c8y-firmware-plugin = { path = "plugins/c8y_firmware_plugin" }
c8y-remote-access-plugin = { path = "plugins/c8y_remote_access_plugin" }
c8y_api = { path = "crates/core/c8y_api" }
Expand All @@ -37,7 +36,6 @@ c8y_http_proxy = { path = "crates/extensions/c8y_http_proxy" }
c8y_mapper_ext = { path = "crates/extensions/c8y_mapper_ext" }
certificate = { path = "crates/common/certificate" }
clock = { path = "crates/common/clock" }
collectd_ext = { path = "crates/extensions/collectd_ext" }
download = { path = "crates/common/download" }
flockfile = { path = "crates/common/flockfile" }
json-writer = { path = "crates/common/json_writer" }
Expand Down
14 changes: 0 additions & 14 deletions configuration/init/systemd/tedge-mapper-collectd.service
View file
Open in desktop

This file was deleted.

13 changes: 1 addition & 12 deletions configuration/package_manifests/nfpm.tedge-mapper.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,17 +132,6 @@ contents:
mode: 0644
packager: rpm

- src: ./configuration/init/systemd/tedge-mapper-collectd.service
dst: /lib/systemd/system/tedge-mapper-collectd.service
file_info:
mode: 0644
packager: deb
- src: ./configuration/init/systemd/tedge-mapper-collectd.service
dst: /lib/systemd/system/tedge-mapper-collectd.service
file_info:
mode: 0644
packager: rpm

- src: ./configuration/init/systemd/tedge-mapper-local.service
dst: /lib/systemd/system/tedge-mapper-local.service
file_info:
Expand Down Expand Up @@ -264,4 +253,4 @@ overrides:
preinstall: configuration/package_scripts/_generated/tedge-mapper/deb/preinst
postinstall: configuration/package_scripts/_generated/tedge-mapper/deb/postinst
preremove: configuration/package_scripts/_generated/tedge-mapper/deb/prerm
postremove: configuration/package_scripts/_generated/tedge-mapper/deb/postrm
postremove: configuration/package_scripts/_generated/tedge-mapper/deb/postrm
6 changes: 0 additions & 6 deletions configuration/package_scripts/_generated/tedge-mapper/apk/postinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,6 @@ if command -v systemctl >/dev/null; then
if [ -f "/etc/tedge/mosquitto-conf/aws-bridge.conf" ]; then
enable_start_service tedge-mapper-aws.service
fi
if [ -d /run/systemd/system ]; then
### Enable the service if the collectd is running on the device
if systemctl is-active --quiet collectd.service; then
enable_start_service tedge-mapper-collectd.service
fi
fi
fi

if [ -f /var/lib/dpkg/info/tedge_mapper.postrm ]; then
Expand Down
3 changes: 1 addition & 2 deletions configuration/package_scripts/_generated/tedge-mapper/apk/postrm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ purge_mapper_lock() {
/run/lock/tedge-mapper-c8y.lock \
/run/lock/tedge-mapper-az.lock \
/run/lock/tedge-mapper-aws.lock \
/run/lock/tedge-mapper-local.lock \
/run/lock/tedge-mapper-collectd.lock
/run/lock/tedge-mapper-local.lock
}

case "$1" in
Expand Down
10 changes: 2 additions & 8 deletions configuration/package_scripts/_generated/tedge-mapper/deb/postinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,9 +230,9 @@ if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-decon
systemctl --system daemon-reload >/dev/null || true
if [ -n "$2" ]; then
if command -v deb-systemd-invoke >/dev/null 2>&1; then
deb-systemd-invoke try-restart tedge-mapper-aws.service tedge-mapper-az.service tedge-mapper-c8y.service tedge-mapper-local.service tedge-mapper-collectd.service >/dev/null || true
deb-systemd-invoke try-restart tedge-mapper-aws.service tedge-mapper-az.service tedge-mapper-c8y.service tedge-mapper-local.service >/dev/null || true
else
systemctl try-restart tedge-mapper-aws.service tedge-mapper-az.service tedge-mapper-c8y.service tedge-mapper-local.service tedge-mapper-collectd.service >/dev/null || true
systemctl try-restart tedge-mapper-aws.service tedge-mapper-az.service tedge-mapper-c8y.service tedge-mapper-local.service >/dev/null || true
fi
fi
fi
Expand Down Expand Up @@ -273,12 +273,6 @@ if command -v systemctl >/dev/null; then
if [ -f "/etc/tedge/mosquitto-conf/aws-bridge.conf" ]; then
enable_start_service tedge-mapper-aws.service
fi
if [ -d /run/systemd/system ]; then
### Enable the service if the collectd is running on the device
if systemctl is-active --quiet collectd.service; then
enable_start_service tedge-mapper-collectd.service
fi
fi
fi

if [ -f /var/lib/dpkg/info/tedge_mapper.postrm ]; then
Expand Down
3 changes: 1 addition & 2 deletions configuration/package_scripts/_generated/tedge-mapper/deb/postrm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ purge_mapper_lock() {
/run/lock/tedge-mapper-c8y.lock \
/run/lock/tedge-mapper-az.lock \
/run/lock/tedge-mapper-aws.lock \
/run/lock/tedge-mapper-local.lock \
/run/lock/tedge-mapper-collectd.lock
/run/lock/tedge-mapper-local.lock
}

case "$1" in
Expand Down
8 changes: 1 addition & 7 deletions configuration/package_scripts/_generated/tedge-mapper/rpm/postinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ fi
if [ $1 -eq 2 ]; then
if [ -d /run/systemd/system ]; then
systemctl --system daemon-reload >/dev/null || true
systemctl restart tedge-mapper-aws.service tedge-mapper-az.service tedge-mapper-c8y.service tedge-mapper-local.service tedge-mapper-collectd.service >/dev/null || true
systemctl restart tedge-mapper-aws.service tedge-mapper-az.service tedge-mapper-c8y.service tedge-mapper-local.service >/dev/null || true
fi
fi
# End automatically added section
Expand Down Expand Up @@ -117,12 +117,6 @@ if command -v systemctl >/dev/null; then
if [ -f "/etc/tedge/mosquitto-conf/aws-bridge.conf" ]; then
enable_start_service tedge-mapper-aws.service
fi
if [ -d /run/systemd/system ]; then
### Enable the service if the collectd is running on the device
if systemctl is-active --quiet collectd.service; then
enable_start_service tedge-mapper-collectd.service
fi
fi
fi

if [ -f /var/lib/dpkg/info/tedge_mapper.postrm ]; then
Expand Down
3 changes: 1 addition & 2 deletions configuration/package_scripts/_generated/tedge-mapper/rpm/postrm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ purge_mapper_lock() {
/run/lock/tedge-mapper-c8y.lock \
/run/lock/tedge-mapper-az.lock \
/run/lock/tedge-mapper-aws.lock \
/run/lock/tedge-mapper-local.lock \
/run/lock/tedge-mapper-collectd.lock
/run/lock/tedge-mapper-local.lock
}

case "$1" in
Expand Down
4 changes: 0 additions & 4 deletions configuration/package_scripts/_generated/tedge/apk/preinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,4 @@ if user_exists "$TEDGE_USER" && group_exists "$TEDGE_GROUP"; then
chown "$TEDGE_USER:$TEDGE_GROUP" /run/lock/tedge-mapper-aws.lock
fi

if [ -f "/run/lock/tedge-mapper-collectd.lock" ]; then
chown "$TEDGE_USER:$TEDGE_GROUP" /run/lock/tedge-mapper-collectd.lock
fi

fi # end: user_exists && group_exists
4 changes: 0 additions & 4 deletions configuration/package_scripts/_generated/tedge/deb/preinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,4 @@ if user_exists "$TEDGE_USER" && group_exists "$TEDGE_GROUP"; then
chown "$TEDGE_USER:$TEDGE_GROUP" /run/lock/tedge-mapper-aws.lock
fi

if [ -f "/run/lock/tedge-mapper-collectd.lock" ]; then
chown "$TEDGE_USER:$TEDGE_GROUP" /run/lock/tedge-mapper-collectd.lock
fi

fi # end: user_exists && group_exists
4 changes: 0 additions & 4 deletions configuration/package_scripts/_generated/tedge/rpm/preinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,4 @@ if user_exists "$TEDGE_USER" && group_exists "$TEDGE_GROUP"; then
chown "$TEDGE_USER:$TEDGE_GROUP" /run/lock/tedge-mapper-aws.lock
fi

if [ -f "/run/lock/tedge-mapper-collectd.lock" ]; then
chown "$TEDGE_USER:$TEDGE_GROUP" /run/lock/tedge-mapper-collectd.lock
fi

fi # end: user_exists && group_exists
19 changes: 13 additions & 6 deletions configuration/package_scripts/generate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ class Service:
stop_on_upgrade: bool = True
restart_after_upgrade: bool = True

# a deprecated service should not be automatically enabled or started,
# but it should still be removed on upgrade.
# Eventually the service may be completely removed from the services list in the packages.json file
deprecated: bool = False


def get_template(name, default=""):
file = Path(name)
Expand Down Expand Up @@ -200,20 +205,21 @@ def append_matching_services(
# Special case for rpm packages:
# By default rpm maintainer scripts restart mark a service to be restarted in the postrm script
# unlike debian which does this in the postinst.
# Note: if the service is deprecated, it should still be removed
if package_type == "deb":
append_matching_services(
postrm, "postrm-systemd", lambda x: True, variables
)
elif package_type == "rpm":
append_matching_services(
postrm, "postrm-systemd", lambda x: service.stop_on_upgrade, variables
postrm, "postrm-systemd", lambda x: service and service.stop_on_upgrade, variables
)

## postinst: restart after upgrade and start
append_matching_services(
postinst,
"postinst-systemd-restart",
lambda x: x.restart_after_upgrade and x.start,
lambda x: x.restart_after_upgrade and x.start and not x.deprecated,
{
**variables,
"RESTART_ACTION": "restart",
Expand All @@ -224,7 +230,7 @@ def append_matching_services(
append_matching_services(
postinst,
"postinst-systemd-restartnostart",
lambda x: x.restart_after_upgrade and not x.start,
lambda x: x.restart_after_upgrade and not x.start and not x.deprecated,
{
**variables,
"RESTART_ACTION": "try-restart",
Expand All @@ -235,15 +241,16 @@ def append_matching_services(
append_matching_services(
postinst,
"postinst-systemd-start",
lambda x: not x.restart_after_upgrade and x.start,
lambda x: not x.restart_after_upgrade and x.start and not x.deprecated,
variables,
)

# prerm: stop_on_upgrade=false or restart_after_upgrade=true
# Note: if the service is deprecated, it should still be removed
append_matching_services(
prerm,
"prerm-systemd-restart",
lambda x: not x.stop_on_upgrade or x.restart_after_upgrade,
lambda x: (not x.stop_on_upgrade or x.restart_after_upgrade),
variables,
)

Expand All @@ -252,7 +259,7 @@ def append_matching_services(
prerm,
"prerm-systemd",
lambda x: not (not x.stop_on_upgrade or x.restart_after_upgrade)
and x.start,
and x.start and not x.deprecated,
variables,
)

Expand Down
2 changes: 1 addition & 1 deletion configuration/package_scripts/packages.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
{"name": "tedge-mapper-az", "enable": false, "start": false, "restart_after_upgrade": true, "stop_on_upgrade": true},
{"name": "tedge-mapper-c8y", "enable": false, "start": false, "restart_after_upgrade": true, "stop_on_upgrade": true},
{"name": "tedge-mapper-local", "enable": true, "start": false, "restart_after_upgrade": true, "stop_on_upgrade": true},
{"name": "tedge-mapper-collectd", "enable": false, "start": false, "restart_after_upgrade": true, "stop_on_upgrade": true},
{"name": "tedge-mapper-collectd", "enable": false, "start": false, "restart_after_upgrade": true, "stop_on_upgrade": true, "deprecated": true},
{"name": "tedge-mapper-aws.target", "enable": true, "start": true, "restart_after_upgrade": true, "stop_on_upgrade": true},
{"name": "tedge-mapper-az.target", "enable": true, "start": true, "restart_after_upgrade": true, "stop_on_upgrade": true},
{"name": "tedge-mapper-c8y.target", "enable": true, "start": true, "restart_after_upgrade": true, "stop_on_upgrade": true}
Expand Down
6 changes: 0 additions & 6 deletions configuration/package_scripts/tedge-mapper/postinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,6 @@ if command -v systemctl >/dev/null; then
if [ -f "/etc/tedge/mosquitto-conf/aws-bridge.conf" ]; then
enable_start_service tedge-mapper-aws.service
fi
if [ -d /run/systemd/system ]; then
### Enable the service if the collectd is running on the device
if systemctl is-active --quiet collectd.service; then
enable_start_service tedge-mapper-collectd.service
fi
fi
fi

if [ -f /var/lib/dpkg/info/tedge_mapper.postrm ]; then
Expand Down
3 changes: 1 addition & 2 deletions configuration/package_scripts/tedge-mapper/postrm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ purge_mapper_lock() {
/run/lock/tedge-mapper-c8y.lock \
/run/lock/tedge-mapper-az.lock \
/run/lock/tedge-mapper-aws.lock \
/run/lock/tedge-mapper-local.lock \
/run/lock/tedge-mapper-collectd.lock
/run/lock/tedge-mapper-local.lock
}

case "$1" in
Expand Down
4 changes: 0 additions & 4 deletions configuration/package_scripts/tedge/preinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,4 @@ if user_exists "$TEDGE_USER" && group_exists "$TEDGE_GROUP"; then
chown "$TEDGE_USER:$TEDGE_GROUP" /run/lock/tedge-mapper-aws.lock
fi

if [ -f "/run/lock/tedge-mapper-collectd.lock" ]; then
chown "$TEDGE_USER:$TEDGE_GROUP" /run/lock/tedge-mapper-collectd.lock
fi

fi # end: user_exists && group_exists
22 changes: 0 additions & 22 deletions crates/common/batcher/Cargo.toml
View file
Open in desktop

This file was deleted.

Loading
Loading