pdns-etcd3

A PowerDNS remote backend with ETCD v3 cluster as storage. It uses the official client to get the data from the cluster. Responses are authoritative for each zone found in the data. Only the DNS class IN is supported, but that's because of the limitation of PowerDNS.

There is no stable release yet, even no beta. The latest release is v0.4.0+0.2.0, the fourth development release, considered alpha quality. Any testing is appreciated.

Features

• Automatic serial for SOA records (based on the cluster revision).
• Replication is handled by the ETCD cluster, no additional configuration is needed for using multiple authoritative PowerDNS servers.
  • DNS responses are nearly instantly up-to-date (on every server instance!) after data changes by using a watcher into ETCD (multi-master)
• Multiple syntax possibilities for (values of) object-supported records
  • JSON5 or YAML
  • different representations of values (e.g. an IPv4 as "192.0.2.1" or [192, 0, 2, 1], a duration as 2h, and more...)
• Short syntax for single-value objects
  • or for the last value left when using defaults (e.g. target in SRV)
• Default prefix for IP addresses
  • overrideable per entry
• Support for custom records (types), like those supported by PowerDNS but unimplemented in pdns-etcd3
• Support for automatically appending zone name to unqualified domain names
• Override of domain name appended to unqualified names (instead of zone name)
  • useful for PTR records in reverse zones
• Support for defaults and zone appending in most plain-string records (only supported ones)
  • e.g. in an SRV entry: 20 5 _ server1, the port will be searched for in default values, the name server1 will be appended with the zone name
  • same entry in JSON5 syntax: {priority: 20, weight: 5, target: "server1"} (this is longer but clearer)
• ALIAS support
• Multi-level defaults and options, overridable
• Domain metadata
  • can also be read and modified with the command line tool pdnsutil (pdnssec in v3.4)
• Upgrade data structure (if needed for new program version) without interrupting service
• Run standalone for usage as a Unix or HTTP connector
  • This could be needed for big data sets, because the initialization from PowerDNS is done lazily (at least as of v4) on first request (which possibly could time out on "big data"…) :-(

Planned

• Reduce redundancy in the data by automatically deriving corresponding data
  • A ⇒ PTR (in-addr.arpa)
  • AAAA ⇒ PTR (ip6.arpa)
  • …
• "Collect record", automatically combining A and/or AAAA records from "server records"
  • e.g. etcd.example.com based on etcd-1.example.com, etcd-2.example.com, …
• "Labels" for selectively applying defaults and/or options to record entries
  • sth. like com/example/-options-ptr → {"auto-ptr": true} and com/example/www/-options-collect → {"collect": …} for com/example/www-1/A+ptr+collect without global options
  • precedence betweeen QTYPE and id (id > label > QTYPE)
• DNSSEC support (PowerDNS DNSSEC-specific calls)
• Search support

Optional

• Support more encodings for values
  • EDN by go-edn
  • TOML by pelletier/go-toml or BurntSushi/toml
  • …
• DNS update support
• Prometheus exporter
• ZeroMQ connector
• Redirecting (or duplicating) log output to something else than stderr

Overview over the support of optional PDNS features in a remote backend:

• Primary and (Auto)Secondary: no
  • AXFR support: not yet
• DNSSEC (live-signing): not yet (planned feature)
  • Metadata: yes
• Search (web API): not yet (planned feature)
• API lookup (for web): not yet
• Disabled zones/domains: no
• Zone caching: yes

Installation

git clone https://github.com/nixn/pdns-etcd3.git
cd pdns-etcd3
make

NOTE: A plain go build will also work, but you will get a dynamically linked executable and incomplete version information in the binary. The build command in Makefile produces a static build with setting the version string properly. For convenience, it is repeated here:

export CGO_ENABLED=0
go build -o pdns-etcd3 -a -ldflags="-extldflags=-static -X main.gitVersion=$(git describe --always --dirty)"

Usage

Of course, you need an up and running ETCD v3 cluster and a PowerDNS installation.

You have to decide in which mode you want to use the backend: either the pipe mode or a standalone mode.

Pipe mode

In pipe mode the backend is launched by PowerDNS dynamically and communicates with it via standard input and output. All the configuration options must be given in the PowerDNS configuration file. But since PowerDNS (at least as of v4) initiates the backend lazily, the 'initialize' call occurs with the first (client) request and the backend has to be fast enough to connect to ETCD, read all data, and reply to this first request. This can be too long, if there is much data to read.

Example PowerDNS configuration file:

launch=remote
remote-connection-string=pipe:command=/path/to/pdns-etcd3[,pdns-version=3|4|5][,<config>][,prefix=<string>][,timeout=<integer>][,dial-keep-alive-time=<duration>][,dial-keep-alive-timeout=<duration>][,auto-sync-interval=<duration>][,permit-without-stream=<bool>][,log-level=[<component>[.<subcomponent>]...=]<level>[;...]]
# since in pipe mode every instance connects to ETCD and loads the data for itself (uses memory), possibly do this:
distributor-threads=1

<config> is one of config-file=... or endpoints=... (see "Parameters" below for details on the value). When using config-file, all other ETCD configuration flags are ignored.

Standalone mode(s)

All other modes are so-called "standalone" modes: the backend must be launched outside of PowerDNS (manually, e.g. as a system service). The standalone mode creates a listening socket and waits for connections (from PowerDNS). It takes the ETCD related parameters from the command line and connects to it right after starting up. Then it accepts connections on the socket and serves them. If the standalone mode begins with an 'initialize' call, only the non-ETCD parameters are available to it.

The data is loaded only once (uses memory only once). The data is loaded before accepting connections from PowerDNS, so it is available directly after a PowerDNS instance has connected. It is okay to have parallel accesses to the instance, the data access is protected by mutexes (including updates).

A standalone mode is started by passing the -standalone=<connector-url> flag to pdns-etcd3. The <connector-url> must be a valid URL, specific for each mode.

Unix

The unix mode uses a UNIX domain socket, thus it can only run on the same system as PowerDNS. The <connector-url> looks like:

unix:///path/to/pdns-etcd3-socket[?relative=<bool>]

It gives the path to the socket file (which is then used in the PowerDNS configuration, see below). relative is false by default, so the path is taken as an absolute path. When set to true, the leading slash is ignored and the path is taken as a relative path.

The unix mode takes an 'initialize' call, so one can pass parameters to it, which are defined in the PowerDNS configuration.

Example PowerDNS configuration file:

launch=remote
remote-connection-string=unix:path=/path/to/pdns-etcd3-socket[,pdns-version=3|4|5]
distributor-threads=3

HTTP

The HTTP mode uses an HTTP listening socket, thus can serve PowerDNS instances from virtually everywhere, based on the listening address. The <connector-url> looks like:

http://<address>:<port>

One has to give both, <address> and <port>. To listen on all interfaces, the URL could look like: http://0.0.0.0:8053.

The HTTP mode does not take an 'initialize' call.

In the PowerDNS configuration, the parameters post and post_json must be both set to a truthy value (e.g. yes). The requests would be rejected otherwise.

Example PowerDNS configuration file:

launch=remote
remote-connection-string=http:url=http://localhost:8053[/client-id=xyz][/pdns-version=3],post=yes,post_json=yes

Because there is no 'initialize' call, the version of a connecting PowerDNS must be given as a parameter, if it differs from the default PDNS version. The default PDNS version can be changed via the -pdns-version option (see below). Other parameters could be set the same way, the URL path replaces the 'initialize' call for the HTTP connector.

Parameters

All parameter keys must be given exactly as denoted here (no case modifications). The ETCD related parameters in standalone mode are given as command line "options", starting with a -: e.g. -config-file=....

The parameters in detail (the parameters, which have to be passed as command line argument in standalone mode, are tagged by #STANDALONE; pipe mode means the PDNS setting remote-connection-string=pipe:...):

• config-file=/path/to/etcd.conf #STANDALONE
  The path to an ETCD (client) configuration file, as accepted by the official client (see etcd/client/v3/config.go, TODO find documentation)
  TLS and authentication is only possible when using such a configuration file.
  When used, disables (ignores) all other ETCD configuration flags. Defaults to not set.
• endpoints=<IP:Port>[|<IP:Port>|...] #STANDALONE
  For a simple connection use the endpoints given here. endpoints accepts hostnames too (instead of IP), but be sure they are resolvable before PowerDNS has started.
  Defaults to [::1]:2379|127.0.0.1:2379.
• prefix=<string> #STANDALONE
  Every entry in ETCD will be prefixed with that. It is not interpreted or changed in any way, also the data watcher uses it, so any other keys under another prefix do not affect DNS data.
  Tip: Let the prefix start and end with /, so you can use etcdkeeper for easier web-based data management.
  There is no default (= empty).
• timeout=<duration> #STANDALONE or
  timeout=<integer> pipe mode (in milliseconds, e.g. 1500 for 1.5 seconds)
  An optional parameter which sets the dial timeout to ETCD. Must be a positive value (>= 1ms).
  Defaults to 2 seconds.
• dial-keep-alive-time=<duration> #STANDALONE
  Interval at which the client sends keep-alive pings to ETCD on the underlying gRPC (HTTP/2) connection. These pings allow the client to detect a dead endpoint (e.g. a host that vanished without sending a TCP RST/FIN) and rotate to another endpoint instead of waiting for the kernel TCP timeout (~13–15 minutes). Set to 0 to disable keep-alive pings.
  Defaults to 10 seconds.
• dial-keep-alive-timeout=<duration> #STANDALONE
  Time the client waits for an acknowledgement after sending a keep-alive ping. If no ack arrives within this timeout, the connection is considered dead and the client reconnects (to another endpoint if available).
  Defaults to 5 seconds.
• auto-sync-interval=<duration> #STANDALONE
  Interval at which the client refreshes its view of the ETCD cluster member list. This makes new endpoints (e.g. cluster members added or rotated in after the client connected) reachable without restarting the backend. Set to 0 to disable.
  Defaults to 1 minute.
• permit-without-stream=<bool> #STANDALONE
  When true, the client sends keep-alive pings even when no RPC stream is active on the connection. This is needed to detect a dead endpoint while the backend is idle (e.g. between watch events on a low-traffic deployment).
  Defaults to true.
• pdns-version=3|4|5
  The (major) PowerDNS version. Version 3 and 4 have incompatible protocols with the backend, so one must use the proper one. Version 5 is accepted, but works currently the same as 4 (no relevant API changes yet).
  Defaults to 4.
• client-id
  // TODO describe
• log-level=[<component>[.<subcomponent>]...=]<level>[;...] #STANDALONE and pipe mode
  Sets the logging level(s) for the given (sub)component <component>[.<subcomponent>]... to <level> (see below for values). Leaving out the component names means to set the root level. Can be repeated for other (sub)components by using the ; separator. The order is not important for different (sub)components: -log-level=4;data.values=2 results in the same effect as -log-level=data.values=2;4 (root level 4 = currently all messages, but omit messages of levels 3+ in data.values).
  Currently only numbers are allowed for the <level> (names should be supported in a future version).
  Example: log-level=-1;conn=4;data.values=3
  Defaults to 0 (info) for the root with no overrides (= all components).

One can see all available command-line (standalone) parameters with a short description, when running pdns-etcd3 -help.

ETCD structure

See ETCD structure. The structure lies beneath the prefix parameter (see above).

Compatibility

pdns-etcd3 is tested on different PowerDNS versions (3.y.z, 4.y.z, and 5.y.z) and uses an ETCD v3 cluster (API 3.0 or higher). It's only one version of each minor (.y), but most likely all (later and earlier) "patch" versions (.z) are compatible. Therefore, each release shall state which exact versions were used for testing, so one can be sure to have a working combination for deploying, when using those (tested) versions.

TODO describe (recommended) cache settings

Testing / Debugging

There is much logging in the program for being able to test and debug it properly. It is structured and leveled.

The structure consists of different components, namely main, etcd, pdns, conn and data; all components (can) have subcomponents, every (sub)component can have its own level, which makes it easier to debug only things of interest. The components and subcomponents in detail:

• main - The main thread / loop of the program, e.g. setting up logging, creating data objects, processing signals and events, etc.
  • init - Setting up things (reading configuration, creating main objects, ...).
  • done - Program shutdown (waiting for coroutines, ...).
  • {signal} - Coroutine for handling OS signals.
• etcd - The communication with ETCD, e.g. real queries against it, connection issues, watcher, etc.
  • watch - The watcher stuff (events and (re-)connects).
• pdns - The communication with PowerDNS, e.g. incoming requests and sending results.
  • init - Mostly the initialize method handling.
• conn - The standalone mode connector(s), setting them up, shutting them down, etc.
  • unix, http - The connector internal work (sockets, ...).
• data - Everything concerning the values (records, ...), parsing data from ETCD, searching records for requests etc.
  • values - Messages concerning the concrete values.
  • locking - Messages for concurrency control (level 4).

TODO list and describe all components

The levels are as follows:

• FTL (fatal, -4) - Errors which prevent the program to continue service. After a fatal error the program exits. (Mostly in main component.)
• ERR (error, -3) - Errors which don't prevent the program to continue service. Different meanings for different components.
• WRN (warning, -2) - Not errors, but situations where it could be done better. An admin should take care of those.
• IMP (important, -1) - Important information on the program, something like "initialized / ready for service".
• INF (info, 0) - Useful information on the program, something like "initializing / connecting / starting". This is the default level for each component.
• D+1 (debug 1, 1) - "Coarse debug" ("big steps"), like "Reloading zones", "Handling watch events"
• D+2 (debug 2, 2) - "Fine debug", like "reloading zone X", "handling event 1 - PUT ..."
• D+3 (debug 3, 3) - "Coarse trace", like "parsing entry key Y"
• D+4 (debug 4, 4) - "Fine trace", like "value for X found in Y", coroutine tracing (begin, end, status)

There is no logical limit in the debug levels (and not a real technical limit, being an int), but currently only the nine described levels are used (three exceptional, two informational, four debug). Fatal errors cannot be suppressed; (simple) errors can, but that is not recommended (especially in non-data components). The level for the root log can be set, which takes effect in every component, unless overridden.

For output, the logrus library is used, with its output handling. Later (optional feature) it could be used to send (selected) output into a file or even a log server.

License

Copyright © 2016-2026 nix https://keybase.io/nixn

Distributed under the Apache 2.0 license, available in the file LICENSE.

Donations

If you like pdns-etcd3, please consider donating to support the further development. Thank you!

Bitcoin (BTC): 1pdns4U2r4JqkzsJRpTEYNirTFLtuWee9
Monero (XMR): 4CjXUfpdcba5G5z1LXAx3ngoDtAHoFGdpJWvCayULXeaEhA4QvJEHdR7Xi3ptsbhSfGcSpdBHbK4CgyC6Qcwy5Rt2GGDfQCM7PcTgfEQ5Q
Ethereum (ETH): 0x003D87efb7069e875a8a1226c9DadaC03dE1f779

These addresses are dedicated to pdns-etcd3 development. For my general development, other projects and personal donation addresses see my profile or my web page.