unifi-network-list-sync

Bash scripts that keep UniFi Network firewall address-groups ("network lists" in the UI) in sync with public IP blocklists/allowlists, plus a few small device-management helpers. Talks to the UniFi Network controller's local REST API directly - no cloud dependency, no Python, just bash and standard Unix tools.

Warning

This project does not create or modify any firewall rules. It only fills address groups - creating the rules that use those groups is up to you, in the UniFi Network application.

Tested against UniFi Network version 10. Older or future major versions may have a different local API.

Features

• API key auth only - no username/password, no Ubiquiti cloud account.
• Reads plain-text feeds (one IP/CIDR per line) and JSON feeds (AWS, GCP).
• Handles single IPs and CIDR ranges, IPv4 and IPv6.
• Lossless compression: adjacent/overlapping entries merge into the minimal CIDR blocks covering exactly the same addresses.
• Deduplication and normalization of every list before upload.
• Large lists split across multiple firewall groups automatically.
• Efficient re-syncs: each list is fetched and processed at most once per run (not once per site), a source that reports no change (HTTP Last-Modified/ETag) is skipped entirely, and a source that changed but still compresses to the exact same address set as before is never re-pushed to the router. No list content is kept on disk between runs - only a small per-list timestamp/hash survives (see config/.lists_state.json below).
• Multi-site: one run syncs every configured UniFi site.
• Runs under bash or zsh; Linux, macOS, WSL, or a UniFi OS console.
• Safety rails: config validation, run lock, and a broken/empty feed or a failed push never wipes your existing groups.

Disclaimer

This project is intended for advanced users who are comfortable reading bash scripts before pointing them at their own network equipment.

Caution

Use at your own risk. A misconfigured list or firewall rule can lock you out of your own controller, block legitimate traffic, or degrade gateway performance. Test on a non-critical site first and keep out-of-band access to your gateway. This software is provided under the MIT License "AS IS", without warranty of any kind - the authors are not responsible for any damage, outage, or data loss caused by running it. See LICENSE.

The example configs point at lists published by third parties. This repository only ships their URLs - the data is downloaded by you, at run time, and is subject to each provider's own terms. Review them and keep only the entries you want.

Requirements

You need bash or zsh, curl, jq, and awk on any Linux/macOS/WSL host (or UniFi OS console) that can reach your controller. Every script checks for these on startup and names anything missing.

Install commands for common distributions

• Debian / Ubuntu / WSL: sudo apt-get update && sudo apt-get install -y curl jq
• Fedora / RHEL / CentOS 8+: sudo dnf install -y jq
• RHEL / CentOS 7 (EOL, upgrade if you can): sudo yum install -y epel-release && sudo yum install -y jq - jq was never in the base/extras repos on 7, only in EPEL
• Arch Linux: sudo pacman -S --needed jq
• Alpine Linux: sudo apk add bash curl jq (Alpine's default shell is ash, not bash or zsh, so bash does need installing here)
• openSUSE: sudo zypper install -y jq findutils (its minimal image is missing find)
• macOS (Homebrew): brew install jq

On a stock UniFi OS console (confirmed with a Dream Machine), bash, curl, awk, and jq are all already present - no extra installs needed.

Warning

Running these scripts directly on a UniFi OS console requires enabling SSH, which might void your warranty - see Ubiquiti's SSH guide (enable steps included) and Ubiquiti's warranty terms. Consider running it instead from a Raspberry Pi, a VM, WSL, or any other spare Linux/macOS machine to avoid warranty issues.

Setup

1. Get the code - a shallow clone (--depth 1) keeps the download small:

   git clone --depth 1 https://github.com/secu-tools/unifi-network-list-sync.git
   cd unifi-network-list-sync
   

   For a branch other than main, add --branch <name>.

   [!TIP] To upgrade later (stays shallow), from inside the repo directory:

   git fetch --depth 1 origin main
   git checkout main
   git reset --hard origin/main
   

   This discards local edits to the scripts but never touches config/ (git-ignored) - your sites/lists/state survive upgrades.

2. Create an API key in the UniFi Network application: open the Integrations panel (plug icon, bottom of the left sidebar), click "Create New API Key", and copy it immediately - it is only shown once. This is a local controller key; a Ubiquiti/UI.com cloud login will not work here.

   

3. Copy the site template and fill in name, host, and api_key (see Configuration):

   cp docs/examples/site.example.json config/sites/my-site.json
   

4. Pick the list registry that matches your use case (see Lists) and copy it:

   cp docs/examples/lists.outgoing.example.json config/lists.json
   

5. Check the config: ./validate_config.sh

6. Check connectivity: ./get_firewall_groups.sh my-site should print your controller's current firewall groups as JSON.

7. Run it: ./sync_lists.sh (or ./sync_lists.sh my-site for one site). Groups appear under Settings -> Firewall & Security -> Firewall Groups.

8. Schedule it (see Scheduling), then create the firewall rules that use these groups.

Tip

Adding another site later? Run ./sync_lists.sh my-new-site --force once, so the new site is populated immediately - normal runs skip sources that haven't changed since the last run (see What is config/.lists_state.json?).

Configuration

The whole config/ folder is real, per-installation data and is git-ignored (site files hold live API keys). Templates live in docs/examples/.

Sites (config/sites/*.json)

One file per UniFi site/controller, copied from docs/examples/site.example.json:

• name - a label you choose; unique across your site files, and what you pass on the command line.
• host - controller base URL including scheme, e.g. https://192.168.1.1.
• api_key - the API key from Setup step 2.
• verify_cert - see Certificate verification.
• group_max_members - max members per firewall group, default 5000. Lowering it is safe; raising it is not recommended - UniFi gateways can become slow or unstable with much larger groups.
• auto_expand_groups - optional, default false. Lets a category create new groups once its source outgrows the ones it already has. Requires both this and that category's own auto_expand_groups in config/lists.json to be true (see Lists) - one flag alone does nothing. When either is false, overflow entries are dropped instead ([WARN] logs the count). Doesn't affect a category's first sync - its initial min_groups is always created.

Warning

Do not hand-edit the groups field or the unifi_site field. Both are managed automatically: groups records every firewall group the sync creates (start with "groups": {} and let it fill in), and unifi_site is resolved from the controller on first use. If unifi_site was ever entered wrong, delete the field and it will be re-resolved.

Already created the firewall groups yourself (by hand, or with another tool) before adopting this project? Run ./sync_groups.sh my-site instead of starting from "groups": {} - it finds groups on the controller whose name and type match what a category in config/lists.json expects (e.g. "Firehol Level 1-1", "Firehol Level 1-2", ...) and registers them into groups for you, so sync_lists.sh reuses them instead of creating duplicates alongside them. See sync_groups.sh.

Certificate verification

verify_cert (site or list entry) takes one of:

• true / "system" / omitted (default) - normal verification against your system's CA store. Only works if the certificate is CA-trusted.
• false - no TLS verification. Only allowed when host/url is http:// - config validation refuses false paired with https://, and refuses anything but false paired with http://.
• one or more "sha256//<base64>" pins, ;-separated - recommended for a self-signed controller. Pins the public key (curl's --pinnedpubkey), which works with a self-signed certificate but still catches a swapped or expired one. Use two pins during a rotation.

Get the current pin with ./get_cert_pin.sh https://192.168.1.1 (or ./get_cert_pin.sh my-site for an already-configured site).

Every HTTPS connection (controller and list downloads) requires TLS 1.2 or newer.

Lists (config/lists.json)

The registry of lists to sync, shared by all sites. Two ready-to-use templates are included - pick the one matching your use case, or merge them (they intentionally overlap; in the end, choose what fits your usage):

• docs/examples/lists.outgoing.example.json - home use. Blocks outbound traffic (LAN to WAN) to known-bad destinations: phishing sites, botnet command-and-control servers, malware infrastructure.
• docs/examples/lists.incoming.example.json - for hosting servers (web, game, mail, ...). Blocks inbound traffic (WAN to server) from known attack sources: scanners, brute-forcers, honeypot-flagged IPs. Also includes Cloudflare/AWS/GCP ranges, useful as allowlists.

Hosting something publicly and also want outbound protection for the rest of your LAN? Merge both files' lists arrays into one config/lists.json (drop duplicate category entries - a few overlap on purpose) and create two firewall rules, one per direction.

Each entry:

{
  "category": "firehol_level1",
  "display_name": "Firehol Level 1",
  "name_template": "Firehol Level 1-%d",
  "group_type": "address-group",
  "min_groups": 2,
  "url": "https://iplists.firehol.org/files/firehol_level1.netset",
  "preprocess": "cut -d' ' -f1"
}

• category - internal key, unique across the file.
• display_name - shown in log output.
• name_template - the UniFi group name; %d numbers multiple groups (e.g. "Firehol Level 1-%d" -> "Firehol Level 1-1", "-2", ...). A template with no %d (e.g. "Firehol Webserver") is used as-is for the first group, with any overflow auto-numbered "Firehol Webserver 2", "3", etc. Growth after the first sync is governed by auto_expand_groups (below).
• group_type - "address-group" (IPv4) or "ipv6-address-group" (IPv6).
• min_groups - always keep at least this many groups registered, even if the list currently fits in fewer.
• auto_expand_groups - optional, default false. This category's half of the double opt-in from Sites - also needs the site's own auto_expand_groups set to true for this category to grow past its existing groups.
• url - http(s) URL to download the list from.
• verify_cert - optional; see Certificate verification.
• preprocess - optional text filter for plain-text feeds, e.g. cut -d' ' -f1 to keep the first field. Not a free-form shell command: only cat, cut, grep, tr, head, tail (chained with |) are allowed, and shell metacharacters are rejected outright. Tab-delimited feeds (e.g. DShield's block.txt, see below) can join two columns into one ip/prefix value with cut and tr, writing the tab as a literal JSON \t escape: "cut -f1,3 | tr '\t' '/'".
• json_path - for JSON feeds instead of preprocess (the two are mutually exclusive): dot-separated field names telling the sync where the addresses are nested, e.g. "prefixes.ip_prefix" for AWS's ip-ranges.json. Arrays are traversed automatically at every level, and entries missing the field are skipped, so one path works whether a level is an object, an array of objects, or an array of strings.

See Example Lists for what each entry in the two templates actually is (source, size, recommended direction).

Logging (config/logging.json, optional)

Copy docs/examples/logging.example.json to config/logging.json and set "enabled": true to also write each run's output to a timestamped log file. Fields: log_dir (default logs), log_file_prefix (default update), and keep_runs (how many old run logs to keep; older ones are pruned automatically). Without this file, output only goes to stdout/stderr.

Example Lists

What is each example list?

Recommendation: Server = block on your inbound rule (WAN to server); Outgoing = block on your outbound rule (LAN to WAN); Both = safe for either, appears in both example files; Allowlist = permit-rule material, not something to block; (optional) = lower-priority (broader/ noisier by design, or not actual attack evidence, e.g. Tor, bogons) - treat as an extra layer, not a must-have. "Size" is the approximate number of CIDR blocks after this project's own dedup/merge.

Caution: the Cloudflare, AWS, and GCP entries (Allowlist below) are address space of major cloud/CDN providers, not attacker IPs. Blocking them outright will break access to many unrelated, legitimate services hosted behind the same providers.

Warning: a few entries are very large (tens of thousands of addresses), firehol_level2/firehol_level3, blocklist_de_all, and cins_army in particular. Large groups can slow down a UniFi gateway - test on a non-critical site first and keep a backup of your controller config.

Category	Recommendation	Description	Size
emerging_threats_compromised	Both	Hosts confirmed compromised or otherwise malicious	small
firehol_level1	Both	Highest-confidence aggregation (includes Spamhaus DROP, DShield, and others)	~small
ipsum_level5	Both	Aggregated from many public blocklists; flagged by at least 5 distinct sources	small
spamhaus_drop	Both	Netblocks entirely under criminal control ("Don't Route Or Peer"); near-zero false positives	small
spamhaus_dropv6	Both	IPv6 counterpart of Spamhaus DROP	tiny
binarydefense_artillery	Server	Threat-intel banlist from a honeypot/sensor network	small
blocklist_de_all	Server	Real-time abuse reports (SSH, mail, web, FTP, IRC, and more), fail2ban-style	large
cins_army	Server	IPs with a poor trust score not yet flagged elsewhere	large
dshield_top_attackers	Server	Top 20 attacking /24 subnets over the last 3 days	tiny
firehol_level2	Server	Broader aggregation than Level 1; more sources, still curated	medium
firehol_level3	Server (optional)	Broadest aggregation tier; large, lower-confidence sources - see the warning above	large
firehol_webserver	Server	IPs seen specifically attacking web servers	small
greensnow	Server	Community honeypot network reports for brute-forcing/scanning	small
team_cymru_bogons_ipv4	Server (optional)	Unallocated/reserved IPv4 space that should never be a legitimate source	small
tor_exit_nodes	Server (optional)	Active Tor exit nodes - anonymized traffic, not inherently malicious	small
feodo_tracker	Outgoing	Active botnet command-and-control servers	tiny
firehol_webclient	Outgoing	IPs seen specifically attacking client software (browsers and similar)	tiny
phishing_database	Outgoing	Active phishing-hosting IPs, continuously re-verified	medium
aws_ipv4 / aws_ipv6	Allowlist	Amazon's own published IP ranges	small
cloudflare_ipv4 / cloudflare_ipv6	Allowlist	Cloudflare's own published edge IP ranges	tiny
gcp_ipv4 / gcp_ipv6	Allowlist	Google's own published IP ranges	small

Scripts

• sync_lists.sh [site_name] [--force] - sync every list against every configured site, or just one site if named. This is the one you schedule. Normal runs skip re-fetching a source that hasn't changed and skip re-pushing to the router when nothing actually changed; --force bypasses both and re-syncs everything unconditionally (see What is config/.lists_state.json?).
• sync_groups.sh <site_name> - match already-existing UniFi firewall groups to config/lists.json categories by name and type, and register them into that site's groups field. For admins adopting this project on a controller that already has the groups set up - see Sites. Safe to re-run any time; never creates, renames, or deletes a group itself.
• get_firewall_groups.sh <site_name> - dump a site's firewall groups as JSON.
• get_device_list.sh <site_name> - dump a site's device inventory as JSON.
• set_device_disabled.sh <site_name> <device_id> <true|false> - enable or disable a UniFi device.
• get_cert_pin.sh <https://host[:port]|site_name> - print a controller's current TLS certificate pin (plus subject/issuer/expiry), ready to paste into that site's verify_cert - see Certificate verification.
• validate_config.sh - validate every config file with specific error messages.

Every script above that takes site_name requires it, except sync_lists.sh, which syncs every configured site if it's omitted (or just one, if named) - each site is different, so nothing ever silently picks one for you or acts on all of them without being asked.

Scheduling

Point cron (Linux/macOS/WSL) or Task Scheduler (Windows) at sync_lists.sh with no arguments, e.g. a daily run at 03:00:

0 3 * * * /path/to/sync_lists.sh

Once a day is plenty - these lists do not change minute to minute, and some providers rate-limit aggressive fetchers. Overlapping runs are prevented by a lock, and the script exits non-zero if anything failed, so your scheduler can alert on bad runs.

FAQ

Group creation fails with a name conflict

UniFi does not allow two firewall groups on the same site to share a name. If a group this project wants to create (e.g. "Firehol Level 1-2") already exists on the controller - created manually, or left over from another setup - the run prints:

[ERROR] Could not create firewall group 'Firehol Level 1-2': a group with this name already exists on the UniFi controller (api.err.FirewallGroupExisted)

In the UniFi Network application, go to Settings -> Firewall & Security -> Firewall Groups and rename or delete the conflicting group, then re-run sync_lists.sh - it picks up where it left off.

UniFi shows "Gateway Configuration Failed" after a sync

This should not happen with the default group_max_members of 5000 - especially since lists are CIDR-compressed before upload. If you see it, you most likely raised group_max_members or your gateway is under memory pressure: set group_max_members back to 5000 (or lower), disable the largest lists, and re-run.

My group has fewer entries than the source list

Expected. Entries are deduped and losslessly merged into CIDR blocks before upload (e.g. two adjacent /25 networks become one /24). The group covers exactly the same addresses as the source - no more, no less - it is just a more compact representation that keeps the gateway fast.

A run says entries were dropped for a category

That category's source now needs more groups than it already has, and auto_expand_groups isn't true at both the site level and that category's own entry in config/lists.json (it defaults to false at both, so this is the out-of-the-box behavior - see Configuration). Rather than create more groups, the run keeps whichever entries already fit in the existing groups and drops the rest, logging how many. Set auto_expand_groups: true in both places to let it grow instead, then re-run.

My Ubiquiti account login does not work as api_key

The api_key is a local controller API key created in the UniFi Network application's Integrations panel (see Setup step 2). Ubiquiti cloud (UI.com / SSO) credentials are a different thing and will not work.

Why do I see file-permission warnings?

config/ holds live API keys, so every script warns (without stopping) when project files are wider open than they should be: config/ readable by other local users, scripts writable by them, an unexpected setuid/setgid bit, or writable .git metadata (which would let another local user plant a git hook). Fix with:

chmod -R go-rwx config
chmod 644 lib/*.sh tests/*.sh && chmod go-w lib tests
chmod 755 *.sh

plus chmod go-w on any other path the warning names. The check is skipped on filesystems that don't enforce POSIX permissions (e.g. Windows/NTFS), where it would be meaningless.

Why do I see a "Local files modified" or "not found on git remote" warning?

When the project directory is a git clone, every script starts with a quick advisory integrity check of its own checkout: uncommitted changes to tracked files, untracked .sh files, or a HEAD commit that the origin remote does not have each print a [WARN] - a heads-up that the code about to run is not what was published. Anything under config/ and docs/ is never flagged - those are yours to edit. If a flagged change is yours (you edited a script, or you are developing), ignore the warning or commit; each warning names the exact git command to inspect with. It never blocks a run: an unreachable remote is reported as exactly that, and the check skips itself when git or an origin remote is missing.

What is config/.lists_state.json?

Per-list bookkeeping (HTTP Last-Modified/ETag plus a content checksum) so a re-run can skip a source that has not changed, and skip re-pushing to the router when the result is identical to last time. No actual list content is ever kept, only these small validators/hashes.

Safe to delete any time - the next run rebuilds it and re-syncs everything once, same as --force (see Scripts). That is also how a newly added site, or one that missed an update, catches up immediately. It is git-ignored, same as the rest of config/.

Can I sync more often than daily?

Some providers rate-limit or block aggressive fetchers (Spamhaus, for example, requires at least an hour between automated downloads). Daily is enough for these feeds; check each provider's terms before shortening the interval.

Development

The test suite is plain bash - no framework beyond the tools already required. Run it from the project root:

bash tests/run_all.sh

or a single file: bash tests/run_all.sh test_cidr_merge.sh

To run the same suite under zsh instead of bash: TEST_SHELL=zsh bash tests/run_all.sh

Tests run against local fixtures and fake API calls - no real network or controller is touched. .github/workflows/ci.yml runs the suite plus shellcheck on Ubuntu and macOS for every push and pull request; the macOS job also re-runs the suite under zsh.

License

MIT License - see LICENSE.

Copyright (c) 2026 Jack L. (Cpt-JackL) (https://jack-l.com)

GitHub Repository: https://github.com/secu-tools/unifi-network-list-sync