PowerDNS Training May 2026

• Authoritative DNS server, such as the PowerDNS server, can be deployed in various different setups

• For security reasons, it sometimes not desirable to have the zone data write-able on a server that is exposed to the Internet
  • In such situations, a hidden primary setup can be useful
  • In an hidden primary setup, the server hosting the primary zone is located inside a protected network (often behind a firewall)
  • The public secondaries are exposed to the Internet, but as secondaries the zone data is read-only.
  • In a hidden primary setup, the delegation and in-zone NS-records do only list the publicly reachable secondaries
    • As an option, the domain name of the hidden primary can be listed in the SOA record MNAME field to support synamic updates to the zone

  

• For DNSSEC signed zones, the hidden master could also act as a DNSSEC signer, adding DNSSEC data to the zones
  • The precious DNSSEC keys are not exposed to the Internet
  • Non-DNSSEC aware DNS management software can be augmented with DNSSEC capabilities using such a hidden signer (also known as a bump-in-the-wire signer.

  

• In the hands-on lab sessions for this training, we will be installing the PowerDNS server from the Debain 13 repositories

apt-get install pdns-server

• PowerDNS supports a range of different backend (SQL Database, BIND 9, LDAP, Lua, Pipe etc). To list the available PowerDNS backends (on Debian 13):

# apt search pdns-backend
Sorting... Done
Full Text Search... Done
pdns-backend-bind/stable,now 4.4.1-1 amd64 [installed,automatic]
  BIND backend for PowerDNS

pdns-backend-geoip/stable 4.4.1-1 amd64
  GeoIP backend for PowerDNS

pdns-backend-ldap/stable 4.4.1-1 amd64
  LDAP backend for PowerDNS

pdns-backend-lmdb/stable 4.4.1-1 amd64
  LMDB backend for PowerDNS

pdns-backend-lua2/stable 4.4.1-1 amd64
  Lua2 backend for PowerDNS

pdns-backend-mysql/stable 4.4.1-1 amd64
  MySQL backend for PowerDNS

pdns-backend-odbc/stable 4.4.1-1 amd64
  UnixODBC backend for PowerDNS

pdns-backend-pgsql/stable 4.4.1-1 amd64
  PostgreSQL backend for PowerDNS

pdns-backend-pipe/stable 4.4.1-1 amd64
  pipe/coprocess backend for PowerDNS

pdns-backend-remote/stable 4.4.1-1 amd64
  remote backend for PowerDNS

pdns-backend-sqlite3/stable 4.4.1-1 amd64
  sqlite 3 backend for PowerDNS

pdns-backend-tinydns/stable 4.4.1-1 amd64
  tinydns compatibility backend for PowerDNS

• Select and install once or more of the available backends. In this example we're installing the MySQL/MariaDB backend:

% apt install pdns-backend-mysql

• After installation, the PowerDNS server should be up and running with a default configuration. To check the status of the PowerDNS service, we use systemctl status pdns:

# systemctl status pdns.service
● pdns.service - PowerDNS Authoritative Server
     Loaded: loaded (/lib/systemd/system/pdns.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-10-29 06:38:25 UTC; 14s ago
       Docs: man:pdns_server(1)
	     man:pdns_control(1)
	     https://doc.powerdns.com
   Main PID: 22789 (pdns_server)
      Tasks: 8 (limit: 1132)
     Memory: 43.6M
	CPU: 139ms
     CGroup: /system.slice/pdns.service
	     └─22789 /usr/sbin/pdns_server --guardian=no --daemon=no --disable-syslog --log-timestamp=no --write-pid=no

Oct 29 06:38:25 pdns01 pdns_server[22789]: TCP server bound to [::]:53
Oct 29 06:38:25 pdns01 pdns_server[22789]: PowerDNS Authoritative Server 4.4.1 (C) 2001-2020 PowerDNS.COM BV
Oct 29 06:38:25 pdns01 pdns_server[22789]: Using 64-bits mode. Built using gcc 10.2.1 20210110.
Oct 29 06:38:25 pdns01 pdns_server[22789]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms >
Oct 29 06:38:25 pdns01 pdns_server[22789]: Creating backend connection for TCP
Oct 29 06:38:25 pdns01 pdns_server[22789]: [bindbackend] Parsing 0 domain(s), will report when done
Oct 29 06:38:25 pdns01 pdns_server[22789]: [bindbackend] Done parsing domains, 0 rejected, 0 new, 0 removed
Oct 29 06:38:25 pdns01 systemd[1]: Started PowerDNS Authoritative Server.
Oct 29 06:38:25 pdns01 pdns_server[22789]: About to create 3 backend threads for UDP
Oct 29 06:38:25 pdns01 pdns_server[22789]: Done launching threads, ready to distribute questions

• The default security settings of the PowerDNS service is already quite good, as can be seen with systemd-analyze:

% systemd-analyze security pdns

• However some additional hardening steps are possible (optional):
  • MemoryDenyWriteExecute - enables a function in the memory management unit (and modern CPUs) that makes memory pages either write-able (data) or executable (program code), but never both. This prevents certain attacks, such as buffer overflow attacks
  • umask - Files created by the PowerDNS processes can only be read or written by the PowerDNS user or root, but not by any other user
  • MemoryMax - Restrict the memory consumption of the PowerDNS process to 2 GB. This is usually more than enough for an authoritative server even with large zone files and prevents system instability in case of a memory leak
  • TasksMax - restricts the number of threads (tasks) of the PowerDNS service to 20. This prevents DoS attacks through a security vulnerability that would allow and external attacker to start new threads. This setting should be monitored and adjusted to the real world scenario where PowerDNS is deloyed (number of CPU cores, network interfaces etc).

# Additional hardening for the PowerDNS systemd unit
MemoryDenyWriteExecute=true
UMask=077
MemoryMax=2G
TasksMax=20

• opcode: query, notify, update, DSO … see IANA registry for DNS OpCodes
• status/rcode:
  • NOERROR - the operation was successful
  • NXDOMAIN - the domain name requested does not exist (or is not delegated)
  • SERVFAIL - some remote DNS server failure or DNSSEC validation failure
  • FORMERR - the query was not correct DNS
  • REFUSED - this server has an access control list that forbids the answer to this client
  • NOTIMPL - a feature used/requested that this server does not implement
  • BADCOOKIE - Bad/missing Server Cookie

  … see IANA registry for DNS RCODEs

• queryid: 16bit value to sort DNS answers to DNS queries
• flags: information on the query and the answer (see IANA registry for DNS Header Flags)
  • AA authoritative answer - answer is coming directly from an authoritative server
  • TC truncated - answer does not fit into the advertised UDP packet size, please re-query over TCP
  • RD recursion desired - this is a query from a client machine, please provide a full complete answer (no referral please)
  • RA recursion available - this answer comes from a DNS resolver that is willing to accept queries with RD flag set
• DNSSEC flags: AD- and CD-Flag
  • AD authentic data - the DNS resolver sending this answer has performed a successful DNSSEC validation on the data. If we trust the resolver, we can trust the data
  • CD checking disabled - a client asking a DNSSEC validating DNS resolver to not perform DNSSEC validation but to pass all DNSSEC data unaltered (even if the data is invalid). Used for troubleshooting DNSSEC issues.
• QUERY: number of query resource records (usually one)
• ANSWER: count of DNS resource records in the answer. Can be more than one. Can be zero if no data is available for the query.
• AUTHORITY: number of authority records in the answer. Can be a SOA-Record (for negative answers) or NS-Records for referrals or positive answers. Many modern DNS server only fill the authority section if required by the protocol to keep answer packets small
• ADDITIONAL: additional resource records that not have been requested but might help with the name resolution, and the EDNS (Extended DNS) OPT-Record (see RFC 6891 Extension Mechanisms for DNS (EDNS(0))

• EDNS Version 0 is the current version
  • new Versions are being discussed in the IETF
• additional EDNS flags. DO - DNSSEC OK, this DNS client supports DNSSEC records
• maximum UDP answer packet size in byte as negotiated between dig and the DNS server/resolver. Current default is 4096, must be between 512 and 4096. The default changed to 1232 on DNS flag day 2020

• The answer section contains zero (if there is no data available), one or more DNS resource records that match the query

• if present, the authority section contains the authoritative name server that hosts the content delivered in the answer. For negative (NXDOMAIN/NOERROR-NODATA) answers, the authority section contains the SOA record of the zone that is authoritative for the negative answer.

• if present, the additional section contains additional DNS records that have not been explicitly requested, but might help in the name resolution process. Because the additional section can be misused in attacks, modern DNS server software minimizes the additional section data.
• If EDNS data is available, it is also in the additional section, but dig displays this data in the OPT pseudo-section

• the footer contains the size (in bytes) of the answer packet, the DNS server that has send the answer, the time it took to receive the answer and the time and date of the DNS communication

• usually dig sends the query to the DNS resolver configured in the operating system (file /etc/resolv.conf on Unix/Linux)
• with the @ syntax the query can be sent to other DNS servers (resolver or authoritative server)

• command line switches can alter the query sent by dig.
  • +multi formats the output in human readable form (wrapped for 80 column terminal)
  • +norec sends a query without RD flag (non-recursive or iterative query)

• the switch +dnssec requests DNSSEC data from the upstream server
• the switch +cd sends the CD (checking disabled) flag to disable upstream DNSSEC validation (if the target of the query is a DNS resolver)

• dig can initiate a zone transfer from an authoritative server that contains the zone database (primary or secondary server)
• the data is printed in the Master Zone Format and can be used to start a new zone database file
• Example: list top level domains with DNSSEC delegation:

  $ dig @f.root-servers.net AXFR . +noidnout | \
    grep DS | \
    grep -v RRSIG | \
    cut -d "." -f 1 | \
    uniq
  

• pdns_server is the actual PowerDNS server binary. Besides starting the PowerDNS service, this command can be used to check the configuration file

% pdns_server --config=check
Apr 28 08:31:46 Loading '/usr/lib/x86_64-linux-gnu/pdns/libgmysqlbackend.so'

• The parameter --version shows some help information, but also the compile time configuration of the PowerDNS server binary. This is helpful to recompile a newer PowerDNS version with the same configuration, or to check for the existence of a compile-time feature

% pdns_server --version
Apr 28 08:32:29 PowerDNS Authoritative Server 4.9.7 (C) PowerDNS.COM BV
Apr 28 08:32:29 Using 64-bits mode. Built using gcc 14.2.0.
Apr 28 08:32:29 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Apr 28 08:32:29 Features: libcrypto-ecdsa libcrypto-ed25519 libcrypto-ed448 libcrypto-eddsa libgeoip libmaxminddb lua lua-records PKCS#11 protobuf sodium curl scrypt 
Apr 28 08:32:29 Built-in modules: 
Apr 28 08:32:29 Loading '/usr/lib/x86_64-linux-gnu/pdns/libbindbackend.so'
Apr 28 08:32:29 Loading '/usr/lib/x86_64-linux-gnu/pdns/libgmysqlbackend.so'
Apr 28 08:32:29 Loaded modules: bind gmysql
Apr 28 08:32:29 Configured with: " '--build=x86_64-linux-gnu'
                   '--prefix=/usr'
		   '--includedir=${prefix}/include'
		   '--mandir=${prefix}/share/man'
		   '--infodir=${prefix}/share/info'
		   '--sysconfdir=/etc'
		   '--localstatedir=/var'
		   '--disable-option-checking'
		   '--libdir=${prefix}/lib/x86_64-linux-gnu'
		   '--runstatedir=/run'
		   '--disable-maintainer-mode'
		   '--disable-dependency-tracking'
		   '--sysconfdir=/etc/powerdns'
		   '--enable-systemd'
		   '--with-systemd=/usr/lib/systemd/system'
		   '--with-dynmodules=bind ldap lmdb lua2 pipe gmysql godbc gpgsql gsqlite3 geoip remote tinydns'
		   '--with-modules='
		   '--enable-ixfrdist'
		   '--enable-tools'
		   '--with-protobuf'
		   '--enable-unit-tests'
		   '--enable-lua-records'
		   '--enable-experimental-pkcs11'
		   '--enable-reproducible'
		   '--disable-silent-rules'
		   '--with-libcrypto=/usr'
		   'build_alias=x86_64-linux-gnu'
		   'CFLAGS=-g -O2 -Werror=implicit-function-declaration -ffile-prefix-map=/build/reproducible-path/pdns-4.9.7=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/pdns-4.9.7=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -DPACKAGEVERSION='\''"4.9.7-1.Debian"'\'''"

• pdns_control is the older of the two PowerDNS command line utilities. It contains many functions to maintain the running PowerDNS server

• pdnsutil is the newer command line management tools. It contains functions to work with DNSSEC and TSIG security protocols (which will be covered on day 3 of this training).
  • It also contains functions to manage zones and zone content

• In the default configuration PowerDNS will write log information to standard-out (stdout), which will be read by systemd and send to the systemd journal. This is the command that is executed by systemd to start the PowerDNS process:

/usr/sbin/pdns_server --guardian=no --daemon=no --disable-syslog --log-timestamp=no --write-pid=no

• The PowerDNS unit name is pdns. The command tool journalctl can be used to show and filter the PowerDNS log content. Here the log is shown in follow mode where journalctl will block and display new log messages as they arrive:

# journalctl -fu pdns
-- Journal begins at Fri 2021-10-29 06:10:29 UTC. --
Oct 29 07:34:23 pdns01 pdns_server[26503]: TCP server bound to 0.0.0.0:53
Oct 29 07:34:23 pdns01 pdns_server[26503]: TCP server bound to [::]:53
Oct 29 07:34:23 pdns01 pdns_server[26503]: PowerDNS Authoritative Server 4.4.1 (C) 2001-2020 PowerDNS.COM BV
Oct 29 07:34:23 pdns01 pdns_server[26503]: Using 64-bits mode. Built using gcc 10.2.1 20210110.
Oct 29 07:34:23 pdns01 pdns_server[26503]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Oct 29 07:34:23 pdns01 pdns_server[26503]: Not validating response for security status update, this is a non-release version
Oct 29 07:34:23 pdns01 pdns_server[26503]: Creating backend connection for TCP
Oct 29 07:34:23 pdns01 systemd[1]: Started PowerDNS Authoritative Server.
Oct 29 07:34:23 pdns01 pdns_server[26503]: About to create 3 backend threads for UDP
Oct 29 07:34:24 pdns01 pdns_server[26503]: Done launching threads, ready to distribute questions
Oct 29 08:04:23 pdns01 pdns_server[26503]: Not validating response for security status update, this is a non-release version
Oct 29 08:11:28 pdns01 pdns_server[26503]: Cache clear request for 'zone001.dnslab.org' received from operator
Oct 29 08:11:37 pdns01 pdns_server[26503]: Cache clear request received from operator
Oct 29 08:13:11 pdns01 pdns_server[26503]: Rediscovery was requested
Oct 29 08:29:42 pdns01 pdns_server[26503]: Cache clear request received from operator
Oct 29 08:30:05 pdns01 pdns_server[26503]: AXFR-out zone 'zone001.dnslab.org', client '127.0.0.1:48973', transfer initiated
Oct 29 08:34:23 pdns01 pdns_server[26503]: Not validating response for security status update, this is a non-release version

• You might see the message below in the log and might be worried. The release versions of PowerDNS as distributed by PowerDNS B.V. will phone home to check for new security vulnerabilities. This does not work on non-release version, such as the version distributed in Debian 13.

  Not validating response for security status update, this is a non-release version
  

• This function can be disabled by setting the security-poll-suffix to an empty string in the PowerDNS configuration:

Zone name from which to query security update notifications. Setting this to an empty string disables secpoll.

• PowerDNS can log every incoming DNS query. This DNS query-logging is CPU intensive and can slow down the DNS server operation. It is recommended to only enable query-logging in a debugging situation for a limited time and under direct control of the administrator. Configuration in pdns.conf

# Query Logging
log-dns-queries=yes
loglevel=6

• Example DNS query-log output

Oct 30 02:01:24 pdns01 pdns_server[28629]: Remote 127.0.0.1 wants 'zone001.dnslab.org|SOA', do = 0, bufsize = 1232 (4096): packetcache MISS
Oct 30 02:04:44 pdns01 pdns_server[28629]: Remote 127.0.0.1 wants 'zone001.dnslab.org|SOA', do = 0, bufsize = 1232 (4096): packetcache MISS
Oct 30 02:04:55 pdns01 pdns_server[28629]: Remote 127.0.0.1 wants 'zone001.dnslab.org|A', do = 0, bufsize = 1232 (4096): packetcache MISS
Oct 30 02:05:08 pdns01 pdns_server[28629]: Remote 127.0.0.1 wants 'zone001.dnslab.org|A', do = 0, bufsize = 1232 (4096): packetcache HIT
Oct 30 02:05:37 pdns01 pdns_server[28629]: Remote 127.0.0.1 wants 'zone001.dnslab.org|AAAA', do = 1, bufsize = 1232 (4096): packetcache MISS

• Log more DNS details. In the PowerDNS configuration file

log-dns-details=yes

• In addition to DNS queries, PowerDNS can also log the database or backend queries. In the PowerDNS configuration file add

query-logging=yes

• As SQL query-logging is also CPU intensive and can hurt the DNS servers performance, the SQL query-logging function can be controlled at run-time from pdns_control

% pdns_control set query-logging yes
done

• Example of SQL query-logging output

Oct 30 02:29:50 pdns01 pdns_server[28849]: Remote 127.0.0.1 wants 'zone001.dnslab.org|AAAA', do = 1, bufsize = 1232 (4096): packetcache MISS
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614378640: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614378640: 709 usec to execute
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614378640: 780 total usec to last row
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614420096: select kind,content from domains, domainmetadata where domainmetadata.domain_id=domains.id and name=?
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614420096: 1108 usec to execute
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614420096: 1141 total usec to last row
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614379408: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name=? and domain_id=?
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614379408: 1108 usec to execute
Oct 30 02:29:50 pdns01 pdns_server[28849]: Query 140584614379408: 1181 total usec to last row

• DNSIND is shorthand for a group of extensions to the original DNS protocol.
  • I is for Incremental Zone Transfer
  • N is for NOTIFY
  • D is for Dynamic Update
• Together, the DNSIND extensions allow DNS to handle dynamic networks.
• A dynamic update allows a node to change the contents of a zone. DNS dynamic update is the RFC standard API to change DNS content over the network. As it is a RFC standard, it works across different DNS server products. Use cases:
  • DHCP servers, for example, can add A, AAAA and PTR records to reflect the leases they give out.
  • Microsoft Active Directory can update the SRV records required for service discovery via dynamic updates
  • DNS management systems such as VinylDNS (https://www.vinyldns.io) or Men & Mice Micetro (https://menandmice.com) can manage zone content through dynamic updates
  • ACME compatible x509 certificate management systems (e.g. Let's encrypt, RFC 8555 https://datatracker.ietf.org/doc/html/rfc8555) can authenticate the domain owner with the help of dynamic DNS updates, allowing centralized automated certificate management
• A dynamic update can:
  • Add resource records to a zone
  • Delete records from a zone:
    • One record
    • All records of a certain type with one domain name (an RRSet)
    • All records with one domain name
• A server can be configured to only accept updates from senders with the proper PSK (Pre-Shared TSIG Key).
• A sender can define update prerequisites, such as the existence/non-existence of:
  • A particular record
  • Resource Records with a specific domain name and type
• A sender first tries to transmit an update to a zone's primary master.
  • Most compare the SOA's mname field to the NS RRSet.
  • If the mname matches an NS RR, the update is sent there.
  • Otherwise the update is sent to one of the domain names in the NS RRSet (from the sender's perspective, a secondary).

    

• PowerDNS supports dynamic updates on a server that hosts primary (master) zones. Dynamic updates are disabled and can be enabled in the configuration file (global)

dnsupdate=yes

• In addition to the global configuration, dynamic updates need to be enabled on a zone

% pdnsutil set-meta example.org ALLOW-DNSUPDATE-FROM 127.0.0.0/8
Oct 30 08:03:57 gmysql Connection successful. Connected to database 'powerdns' on '/run/mysqld/mysqld.sock'.
Oct 30 08:03:57 gmysql Connection successful. Connected to database 'powerdns' on '/run/mysqld/mysqld.sock'.
Set 'example.org' meta ALLOW-DNSUPDATE-FROM = 127.0.0.0/8

• The tool nsupdate (Debian packet dnsutils) is the standard tool to send dynamic update messages. Here an example on how to add a new record to a zone:

$ nsupdate
> server 127.0.0.1
> ttl 60
> add ddns.zoneNNN.dnslab.org in TXT "This is a dynamically added DNS record"
> send
> quit

• The dynamic update in the PowerDNS logfile

Oct 30 08:05:45 pdns01 pdns_server[30874]: Remote 127.0.0.1 wants 'ddns.zoneNNN.dnslab.org|SOA', do = 0, bufsize = 512: packetcache MISS
Oct 30 08:05:45 pdns01 pdns_server[30874]: Remote 127.0.0.1 wants 'zoneNNN.dnslab.org|SOA', do = 0, bufsize = 512: packetcache MISS
Oct 30 08:05:45 pdns01 pdns_server[30874]: UPDATE (9824) from 127.0.0.1 for zoneNNN.dnslab.org: Processing started.
Oct 30 08:05:45 pdns01 pdns_server[30874]: UPDATE (9824) from 127.0.0.1 for zoneNNN.dnslab.org: starting transaction.
Oct 30 08:05:45 pdns01 pdns_server[30874]: UPDATE (9824) from 127.0.0.1 for zoneNNN.dnslab.org: Adding record ddns.zoneNNN.dnslab.org|TXT
Oct 30 08:05:45 pdns01 pdns_server[30874]: UPDATE (9824) from 127.0.0.1 for zoneNNN.dnslab.org: Increasing SOA serial (1111 -> 2021103001)
Oct 30 08:05:45 pdns01 pdns_server[30874]: UPDATE (9824) from 127.0.0.1 for zoneNNN.dnslab.org: Update completed, 2 changed records committed.
Oct 30 08:06:26 pdns01 pdns_server[30874]: 1 domain for which we are master needs notifications
Oct 30 08:06:26 pdns01 pdns_server[30874]: Queued notification of domain 'zoneNNN.dnslab.org' to 165.22.233.102:53

• The record will be published immediately and can be queried over DNS:

% dig @localhost ddns.zone001.dnslab.org txt

; <<>> DiG 9.16.22-Debian <<>> @localhost ddns.zone001.dnslab.org txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2483
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;ddns.zone001.dnslab.org.       IN      TXT

;; ANSWER SECTION:
ddns.zone001.dnslab.org. 60     IN      TXT     "This is a dynamically added DNS record"

;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Oct 30 08:06:28 UTC 2021
;; MSG SIZE  rcvd: 103

• In the default configuration PowerDNS will wait 60 seconds after a dynamic update before sending out NOTIFY messages to the secondary. This configuration changes PowerDNS to send a notification to all slave servers after every update. This will speed up the propagation of changes and is very useful for ACME TLS certificate verification

% pdnsutil set-meta example.org NOTIFY-DNSUPDATE 1

• Each dynamic update will increment the SOA serial number. PowerDNS has a SOA serial number edit policy (SOA-EDIT-DNSUPDATE) that defines how the SOA serial number should be incremented
  • DEFAULT: Generate a soa serial of YYYYMMDDNN. If the current serial is lower than the generated serial, use the generated serial. If the current serial is higher or equal to the generated serial, increase the current serial by 1.
  • INCREASE: Increase (increment) the current serial by 1.
  • EPOCH: Change the serial to the number of seconds since the EPOCH (seconds since 1.1.1970, also known as unixtime)
  • SOA-EDIT: Change the serial to whatever SOA-EDIT would provide. Used for DNSSEC zones to ensure freshness of signatures on secondary (slave) server
  • SOA-EDIT-INCREASE: Change the serial to whatever SOA-EDIT would provide. If what SOA-EDIT provides is lower than the current serial, increase the current serial by 1. Exception: with SOA-EDIT=INCEPTION-EPOCH, the serial is bumped to at least the current EPOCH time.

% pdnsutil set-meta example.org SOA-EDIT-DNSUPDATE INCREASE

• nsupdate is a BIND application for sending dynamic changes to a server.
  • It works both interactively and non-interactively.
  • In both modes, commands are one-per-line.
• The three main commands are:
  • update add
  • update delete
  • send
• update prepares RRs changes, send executes them.
• Note: ENTER by itself is the same as send
• Non-interactively, commands come from a file or STDIN.

$ nsupdate nsupdate.input.file

$ nsupdate < nsupdate.input.file

• Interactive mode:

$ nsupdate
>

• The interactive command show allows changes to be reviewed before being sent.
  • Here no updates have been prepared.

$ nsupdate
> show
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
>

• update add prepares a RR to be added to a zone.
  • All fields, except the Class must be provided.
  • TTL can be provided in a separate statement.

> update add a.example.com IN A 192.0.2.1
ttl 'IN': not a valid number
> update add a.example.com 3600 IN A 192.0.2.1
> ttl 3600
> update add a.example.com AAAA 2001:db8:0:deaf::1
> show
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; UPDATE SECTION:
a.example.com.		3600	IN	A	192.0.2.1
a.example.com.		3600	IN	AAAA	2001:db8:0:deaf::1
> send

• update delete prepares RRs to be deleted.
  • Class is optional, and TTL, if provided, is ignored.
  • Providing all fields except TTL and Class deletes one RR.

> update delete b.example.com. 1800 A 192.0.2.99
> update delete example.net. MX 15 mailserver.example.net.
> show
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; UPDATE SECTION:
b.example.com.		0	NONE	A	192.0.2.99
example.net.		    0	NONE	MX	15 mailserver.example.net.

• update delete Providing the RTYPE without RDATA, deletes a RRSet.

> update delete b.example.com. AAAA
> update delete c.example.com IN SRV
> show
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; UPDATE SECTION:
b.example.com.		0	NONE	A	192.0.2.99
example.net.		  0	NONE	MX	15 mailserver.example.net.
b.example.com.		0	ANY	AAAA
c.example.com.		0	ANY	SRV
>

• update delete Omitting RTYPE and RDATA deletes all RRSets.

> update delete d.example.com.
> show
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; UPDATE SECTION:
b.example.com.		0	NONE	A	192.0.2.99
example.net.	    0	NONE	MX	15 mailserver.example.net.
b.example.com.		0	ANY	AAAA
c.example.com.		0	ANY	SRV
d.example.com.		0	ANY	ANY

• Since BIND 9.9.0, the key word update is optional.

> delete  www.example.com. A
> add www.example.com. 600 A 192.0.2.80
> add www.example.com. 600 A 192.0.2.88

• send - Transmit the prepared updates to the authoritative server.
  • WARNING - hitting ENTER is the same as send!

> send

• answer - shows the results received from the server after a send.

> send
update failed: REFUSED
> answer
Answer:
;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id:  13117
;; flags: qr; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
>

• prereq nxdomain <domain name> - requires that "domain name" doesn't exist (no RRs of any type).

> prereq nxdomain www.example.com
> update add www.example.com 3600 CNAME example.com.
> send

• prereq yxdomain <domain name> - requires that "domain name" does exist.

> prereq yxdomain www.example.com
> update add web.example.com 3600 CNAME www.example.com.
> send

• server <servername or IP> [port] - specifies the authoritative server that will receive the update.
  • If not set, nsupdate uses MNAME from the SOA RR and port 53.
  • Generally, updates should be sent to the primary authoritative server.

> server ns0.example.com

• nsupdate -l sends all updates to localhost.
  • The MNAME is not checked, and server commands are ignored.
  • This is easy use, and is recommended if you are on the primary server
• local <IP address> [port] - specifies the outgoing IP address. This is useful for multi-homed machines, especially when IPv6 is used, and when the server limits updates to specific IPs.

> local 2001:db8:100:0:ff:aa01:42:12:feaa

• zone <zonename> - specify the zone to be changed.
  • This is necessary when the update is an NS or glue RR to be made in the parent zone.

$ nsupdate
> zone example.com.
> update add ns1.subdomain.example.com. 3600 IN A 192.0.2.53
> update add ns2.subdomain.example.com. 3600 IN A 203.0.113.53
> send

• The hardware used to host the DNS server can have a big impact on DNS server performance
  • Turn off symmetric multi-threading (aka Hyper-Threading) and run the DNS server on real CPU cores
  • 4-8 Cores are usually enough for an authoritative DNS server. More CPU cores can kill CPU cache performance.
    • Pin the PowerDNS processes and the Database processes to their own CPU cores
    • Check for IRQ balancing of the network cards. Sometimes all IRQs are routed towards one CPU core, which becomes a bottleneck for DNS style IP traffic (many small packets)
  • When selecting hardware, use server with less number of cores but high performance of each core
  • Configure the PowerDNS receiver-threads to be one per core
  • Memory bandwidth between CPU, cache-memory und main-memory is important
  • Select quality network cards with good driver support for you operating system
  • Avoid system virtualization (VMWare, QEMU, VirtualBox etc). If virtualization is required, use PCI pass-through for the network cards
  • The Spectre and Meltdown type of CPU vulnerabilities usually don't affect dedicated non-virtualized DNS server. Consider switching off the kernel mitigation for these security issue to gain more performance. Consult with your security department first.
  • Possibly adjust the memory allocation scheme of the operating system LIBC library (see MALLOC_ARENA_MAX https://doc.powerdns.com/authoritative/performance.html)
    • Alternative memory allocation libraries can help performance (jmalloc https://github.com/jemalloc/jemalloc, mimalloc https://github.com/microsoft/mimalloc, tcmalloc https://gperftools.github.io/gperftools/tcmalloc.html, scudo https://expertmiami.blogspot.com/2019/05/what-is-scudo-hardened-allocator_10.html)

• Logging can hurt the DNS server performance. Generating text base log message is CPU intensive, and can slow down the PowerDNS server under high query load. Disable query- and detail-logging.

• The DNS protocol has a build in load-distribution function
  • DNS query load will be spread over all authoritative server (primary and secondary)
  • Deploy DNS server in many different networks and geographic regions
  • Select data-center with fast Internet access that are near potential DNS user
• DNS load-balancer in front of authoritative server can hurt the performance and increase the query latency

• PowerDNS configuration options that have an impact on the network performance
  • If the operating system support the reuseport socket option, configure PowerDNS to use this option to be able to have multiple processes/threads listening on the same IP address and port. This helps distributing the load over multiple CPU cores
  • For DNS server with many TCP connections (primary with many secondaries, large Resource-Records set that demand TCP connections, DNS-over-TLS/HTTPS) tune the TCP network stack of your operating system and the PowerDNS TCP configuration:
    • Enable tcp-fast-open when available in the operating system
    • Adjust the max-tcp-connections and max-tcp-connections-per-client to match your PowerDNS server workload. Monitor the PowerDNS server to detect if these threshold
    • Limit the time a TCP connection can be held open from a client (see max-tcp-connection-duration)
    • Limit the number of transactions that can be send through one TCP connection (see max-tcp-transactions-per-conn)

• For a PowerDNS server with database backend, the packet and query caches are an important performance factor. The packet cache holds DNS responses that have been fetched from the database. The query cache holds results from additional database queries required to find DNS data in the database. Carefully tune the PowerDNS caches:
  • max-cache-entries - Max entries in the query cache
  • max-packet-cache-entries - Max entries in the packet cache
  • query-cache-ttl - How long are database queries resulting in positive answers be stored in the query cache
  • negquery-cache-ttl - How long are database queries resulting in negative answers be stored in the query cache
  • cache-ttl - How long are full DNS answer packets are stored in the packet cache

• Database performance can be the bottleneck with a PowerDNS instance using a database backend. PowerDNS stores DNS queries in a queue waiting to the send to the database. Settings that effects the performance of the database query queue
  • max-queue-length length of the backend query queue. If this threshold is reached, close the database connection and restart operation (to work around database connection issues)
  • overload-queue-length defines a threshold size for the database query queue. Default unlimited. Once reached, DNS requests are only answered from the packet cache
  • queue-limit - Maximum time (milliseconds) that a query can be waiting in the backend queue

• The PowerDNS command line tool pdnsutil contains a build-in benchmarking function for database connections

% pdnsutil bench-db

• Many websites offer domain name checking.
• These are most useful for your authoritative zones.
• Some checkers are broken, outdated, or simply bad.
• Two are excellent and recommended:
  • https://zonemaster.net (Alternative: https://zonemaster.iis.se)
  • https://dnsviz.net

• PowerDNS provides an optional build-in web-server. This web-server provides:
  • A simple web page that show vital performance data
  • A metrics endpoint for the Prometheus (https://prometheus.io) monitoring system
  • A URL endpoint for a restful API
• The web-server can be enabled in the PowerDNS configuration file

# Webserver
webserver=yes
webserver-address=127.0.0.1
webserver-password=powerDNS
webserver-port=8053
webserver-allow-from=127.0.0.1/32

• After a restart, PowerDNS should now listen of the defined TCP port

% lsof -Poni :8053
COMMAND     PID USER   FD   TYPE DEVICE OFFSET NODE NAME
pdns_serv 29892 pdns    9u  IPv4  85197    0t0  TCP 138.68.87.93:8053 (LISTEN)

• The URL endpoint /metrics provides a metrics page that get be used ("scraped") by Prometheus and compatible monitoring systems

• Access to the REST-API must be secured by an API key. At least 16 byte of random key is recommended. Such a key can be generated as a hexadecimal string with openssl:

% openssl rand -hex 16

• The configuration settings below enable the API

api=yes
api-key=<api-key>

• Testing the API from curl. It is required to send the API-Key with the X-API-Key header with every request

# curl -s -H 'X-API-Key: ac90ae012cdaba61101a9061eda492d6' http://ns001a.dnslab.org:8053/api/v1/servers/localhost/zones | jq .
[
  {
    "account": "",
    "dnssec": false,
    "edited_serial": 1007,
    "id": "zone001.dnslab.org.",
    "kind": "Native",
    "last_check": 0,
    "masters": [],
    "name": "zone001.dnslab.org.",
    "notified_serial": 1007,
    "serial": 1007,
    "url": "/api/v1/servers/localhost/zones/zone001.dnslab.org."
  }
]

• The API documentation is available from the PowerDNS server itself (benefit: it is always correct for the running version of PowerDNS). To dump out the API documentation, the API-Key needs to be send as well

$ curl -s -H 'X-API-Key: <api-key>' http://nsNNNa.dnslab.org:8053/api/docs | less

• API call to remove the IPv6 AAAA record from a PowerDNS hosted zone. The name parameter must be fully qualified, ending with the dot .:

% curl -s -X PATCH --data '{"rrsets": [ {"name": "zoneNNN.dnslab.org.", "type": "AAAA", "changetype": "DELETE" } ] }' \
   -H 'X-API-Key: <api-key>' http://pdnsNNNa.dnslab.org:8053/api/v1/servers/localhost/zones/zoneNNN.dnslab.org
% pdnsutil list-zone zoneNNN.dnslab.org

• As the pdnsutil list-zone shows, the SOA serial is not incremented. To increment the SOA record, another API call is required.

• In the hands-on lab sessions for this training, we will be installing the PowerDNS recursor from the Debain 11 repositories

% apt install pdns-recursor

• After installation, the PowerDNS recursor should be up and running with the default configuration. You can use the rec_control tool to check that all PowerDNS recursor threads are alive

% rec_control ping
pong
pong
pong

• The PowerDNS recursor main binary is pdns_recursor. Beside being used to start the PowerDNS recursor process, it can be used to print out the compile time configuration

# pdns_recursor --version
Oct 31 11:48:51 PowerDNS Recursor 4.4.2 (C) 2001-2020 PowerDNS.COM BV
Oct 31 11:48:51 Using 64-bits mode. Built using gcc 10.2.1 20210110.
Oct 31 11:48:51 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Oct 31 11:48:51 Features: fcontext libcrypto-ecdsa libcrypto-ed25519 libcrypto-ed448 libcrypto-eddsa lua protobuf dnstap-framestream snmp sodium
Oct 31 11:48:51 Configured with: " '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include'
                '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
                '--disable-option-checking' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode'
                '--disable-dependency-tracking' '--sysconfdir=/etc/powerdns' '--enable-systemd' '--with-systemd=/lib/systemd/system'
                '--enable-reproducible' '--enable-unit-tests' '--disable-silent-rules' '--with-service-user=pdns' '--with-service-group=pdns'
                '--with-libsodium' '--with-lua' '--with-net-snmp' '--with-protobuf=yes' '--enable-dnstap'
                'build_alias=x86_64-linux-gnu'
                'CFLAGS=-g -O2 -ffile-prefix-map=/build/pdns-recursor-MyWXnC/pdns-recursor-4.4.2=. -fstack-protector-strong -Wformat -Werror=format-security'
                'LDFLAGS=-Wl,-z,relro -Wl,-z,now'
                'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
                'CXXFLAGS=-g -O2 -ffile-prefix-map=/build/pdns-recursor-MyWXnC/pdns-recursor-4.4.2=. -fstack-protector-strong -Wformat -Werror=format-security -DPACKAGEVERSION='\''"4.4.2-3.Debian"'\'''"

• The default security settings of the PowerDNS recursor service is already quite good, as can be seen with systemd-analyze:

% systemd-analyze security pdns-recursor

• However some additional hardening steps are possible (optional):
  • MemoryDenyWriteExecute - enables a function in the memory management unit (and modern CPUs) that makes memory pages either write-able (data) or executable (program code), but never both. This prevents certain attacks, such as buffer overflow attacks
  • umask - Files created by the PowerDNS recursor processes can only be read or written by the PowerDNS recursor user or root, but not by any other user
  • MemoryMax - Restrict the memory consumption of the PowerDNS process to 1 GB. This is usually more than enough for an DNS resolver server and prevents system instability in case of a memory leak. For a DNS resolver with a large user base (millions of user), this setting might need to be adjusted. However values over 2 GB are usually not needed.
  • TasksMax - restricts the number of threads (tasks) of the PowerDNS service to 512. This prevents DoS attacks through a security vulnerability that would allow and external attacker to start new threads. This setting should be monitored and adjusted to the real world scenario where PowerDNS is deloyed (number of CPU cores, network interfaces etc).

MemoryDenyWriteExecute=true
UMask=077
MemoryMax=1G
TasksMax=512

• Slides: DNSSEC Introduction

• PowerDNS comes with the DNSSEC trust anchors for the Internet DNS system
• Using DNSSEC validation is just a matter of switching it on
  • Every DNS resolver for the Internet should have DNSSEC validation enabled, there is no downside of doing DNSSEC validation
• To enable DNSSEC validation, add the dnssec=validate statement to the PowerDNS recursor configuration file

# DNSSEC
dnssec=validate

• The Root Trust Anchor for DNSSEC used in a Debian 13 system can be found in /usr/share/dns/root.key which is loaded from /usr/share/pdns-recursor/lua-config/rootkeys.lua

-- readTrustAnchorsFromFile reads the DNSSEC trust anchors from the provided file
-- and reloads it every 24 hours.
readTrustAnchorsFromFile("/usr/share/dns/root.key")

• Other values for the dnssec setting statement:
  • off - completely disables DNSSEC
  • process-no-validate - (default until version 4.5.0): creates a "security aware, non-validating" nameserver, where PowerDNS will request DNSSEC data with the DO flag and will provide the data to a downstream client when requested, but will not do any DNSSEC validation
  • process - (default since version 4.5.0): the PowerDNS recursor will always request DNSSEC data, but will only validate when requested by the client with either the DO or the AD flag in the query from the client
  • log-fail - same as process, but additionally will log DNSSEC validation issues. This can be used to find DNSSEC validation issues before enabling full validation
  • validate - PowerDNS recursor does full DNSSEC validation on all DNS data that contains DNSSEC information

• allow-trust-anchor-query - Allow trustanchor.server CH TXT and negativetrustanchor.server CH TXT queries to view the configured DNSSEC (negative) trust anchors. Can be used to monitor trust anchors.
• dnssec-log-bogus - Log every DNSSEC validation failure. Note: This is not logged per-query but every time records are validated as Bogus.
• aggressive-nsec-cache-size - The number of records to cache in the aggressive cache. If set to a value greater than 0, the recursor will cache NSEC and NSEC3 records to generate negative answers, as defined in RFC 8198. To use this, DNSSEC processing or validation must be enabled by setting dnssec to process, log-fail or validate.
• max-cache-bogus-ttl - Maximum number of seconds to cache an item in the DNS cache (negative or positive) if its DNSSEC validation failed, no matter what the original TTL specified, to reduce the impact of a broken domain.
• nsec3-max-iterations - Maximum number of iterations allowed for an NSEC3 record. If an answer containing an NSEC3 record with more iterations is received, its DNSSEC validation status is treated as Insecure. New recommendation is to set this to 150 iterations, see Guidance for NSEC3 parameter settings - draft-ietf-dnsop-nsec3-guidance
• signature-inception-skew - Allow the signature inception to be off by this number of seconds. Negative values are not allowed (Default 60 seconds since version 4.2.0).

• DNS Query - what the client sends

• DNS Name Resolution

• DNS Resolver caching

• NXDOMAIN - the domain name does not exist.
• NOERROR/NODATA - the domain name exists, but not the RR type. AKA: NOERROR/NOANSWER.
• For NXDOMAIN and NOERROR/NODATA, the zone's SOA is returned in the authoritative section.
• The negative TTL is the minimum of SOA's TTL and the SOA's RDATA MIN field. (RFC 2308 -Negative Caching of DNS Queries (DNS NCACHE))

• Sometimes wrong DNS data might be stored in the cache (caused by an operational error on the authoritative zone side, or through an cache-poisioning attack
• The command rec_control dump-cache can be used to write the content of the cache into a file in human readable form (RFC 1035 master file format)
  • Such a file can be quite large, up to 400% larger than the (binary) cache content in memory. Check the available space on the file system first before dumping a large cache!

% rec_control dump-cache /some/path/in/the/filesystem/cache.txt
dumped 88 records

• Malicious or wrong content in the cache can be removed with the wipe-cache sub-command of rec_control
  • This command takes one or more domain names and removed all cached entries (all record types) of that name (but only that name).
  • When the domain name is suffixed with a $, the domain name and all sub-domains are removed as well

% rec_control wipe-cache powerdns.com$ sys4.de

• The sub-command wipe-cache-typed can be used to remove a specific record type (A, AAAA, MX, TXT …) of a domain name from the cache

% rec_control wipe-cache-typed MX powerdns.com$ sys4.de

• As a security measure, PowerDNS recursor run with a private /tmp and /var/tmp directory
  • The file is written, but only the same process can see and access it (see systemd hardening above)

% rec_control dump-cache /tmp/cache.txt
dumped 88 records
% less /tmp/cache.txt
/tmp/cache.txt: No such file or directory

• We need to write the file into a non-temp directory

% mkdir /var/cache/pdns
% chown pdns /var/cache/pdns/
% rec_control dump-cache /var/cache/pdns/cache.txt
% less /var/cache/pdns/cache.txt

• The PowerDNS cache function can be tuned in the configuration file
• disable-packetcache - disables the cache. Only recommended in special cases where PowerDNS recursor is used to manipulate DNS data on-the-fly that should not be cached, for example though Lua scripts
• max-cache-entries - Maximum number of DNS record cache entries (default 1 million), shared by all threads. Each entry associates a name and type with a record set. The size of the negative cache is 10% of this number. Having a too large cache can hurt performance, as it fragments DNS cache memory. This value can be changed at runtime with rec_control set-max-cache-entries <num>. This is useful to dynamically re-size the cache to find the best cache size for a workload.
• max-packetcache-entries - While the normal cache holds DNS resource records, the packet cache holds the full DNS response packets. This setting controls the maximum number of Packet Cache entries (default 500.000). Each worker and each distributor thread has a packet cache instance. This number will be divided by the number of worker plus the number of distributor threads to compute the maximum number of entries per cache instance. Can be set at run-time with set-max-packetcache-entries <num>
• max-cache-ttl - Maximum number of seconds to cache an item in the DNS cache (default 86400 or 24 hours), no matter what the original TTL specified. The minimum value of this setting is 15. i.e. setting this to lower than 15 will make this value 15.
• max-negative-ttl - How long are negative responses (NXDOMAIN and NXRRSET/NODATA being cached. Default 3600 (1 hour). Can overwrite the SOA negTTL/minTTL of the authoritative zone
• minimum-ttl-override - This setting artificially raises all TTLs to be at least this long. Setting this to a value greater than 1 technically is an RFC violation (and might break applications and web-pages!), but might improve performance a lot. Using a value of 0 impacts performance of TTL 0 records greatly, since it forces the recursor to contact authoritative servers each time a client requests them. Can be set at runtime using rec_control set-minimum-ttl <num>
• packetcache-ttl - Maximum number of seconds to cache an item in the packet cache, no matter what the original TTL specified. Default 3600 (1 hour)
• packetcache-servfail-ttl Maximum number of seconds to cache an answer indicating a failure to resolve in the packet cache. Before version 4.6.0 only ServFail answers were considered as such. Starting with 4.6.0, all responses with a code other than NoError and NXDomain, or without records in the answer and authority sections, are considered as a failure to resolve.
• record-cache-shards - Sets the number of shards in the record cache (Default 1024). If you have high contention as reported by record-cache-contented / record-cache-acquired (see statistics/metrics), you can try to enlarge this value or run with fewer threads.

• Forwarding in recursive mode is configured with one or more forward-zones-recurse statements in the configuration file
  • This statement takes the domain to be forwarded (including all sub-domains). Use . to generate a global forwarding where all queries to being forwarded to the upstream DNS resolver
  • The second parameter are the IP addresses of the upstream DNS resolver. The addresses are separated by semicolons (not comma or space)!

forward-zones-recurse=example.org=192.0.2.53;2001:db8::53

• Iterative forwarding is configured with the forward-zone statement. The statement format is identical with forward-zone-recurse (IP addresses are separated by semicolons)

forward-zones=internal-zone.loc=10.0.0.1;192.168.10.53

• Sometimes an (internal or external) undelegated DNS zone is secured with DNSSEC and the DNS resolver should validate the data
  • Internal Active Directory domains
  • Private domains
  • Split-DNS domains
• As the zone is not delegated, there cannot be a DNSSEC chain-of-trust from the parent zone (as there is no parent zone)
• The solution is to create a dedicated Trust-Anchor for the undelegated zone(s)
• The PowerDNS recursor uses the Key-Signing-Key (KSK, DNSKEY flag 257) of a zone as the trust anchor
  • All trust anchors should be in a single file and loaded at startup
  • It is possible to inject a trust anchor at runtime, but that trust anchor will be lost after an restart of the service

• It is difficult to monitor the internal operation of a DNS server
  • Classic monitoring has a huge performance impact (on busy DNS servers)
    • Example: PowerDNS query-logging
    • Up to 200% performance loss seen
    • Speed of the disk storage is often the limiting factor
    • Conversation from binary to text format is CPU intensive

• DNSdist slides

1. Installing dnsdist
   
   • We work on the resolver machine pdnsrNNN.dnslab.org
   • We install dnsdist from the Debian 13 repositories

   apt install -y dnsdist
   

2. DNSdist as load balancer for authoritative server
   
   • In this session we build a load balancer for our authoritative PowerDNS servers
   • Save and remove a previous dnsdist configuration /etc/dnsdist/dnsdist.conf and start with a clean file
   • We use Port 65053 on the IPv4 loopback address, as port 53 on all IP addresses is currently occupied by the PowerDNS recursor

   newServer({address="<ip-v6-address-of-server1>",   checkType="SOA", checkType=DNSClass.IN, checkName="zoneNNN.dnslab.org"})
   newServer({address="<ip-v6-address-of-server2>",   checkType="SOA", checkType=DNSClass.IN, checkName="zoneNNN.dnslab.org"})
   newServer({address="<ip-v4-address-of-server1>",   checkType="SOA", checkType=DNSClass.IN, checkName="zoneNNN.dnslab.org"})
   newServer({address="<ip-v4-address-of-server2>",   checkType="SOA", checkType=DNSClass.IN, checkName="zoneNNN.dnslab.org"})
   setServerPolicy(leastOutstanding)
   setLocal("127.0.0.1:65053")
   

   1. Starting dnsdist
      
      • check the configuration of dnsdist for syntax errors

      % /usr/bin/dnsdist -u dnsdist -g dnsdist --check-config
      Configuration '/etc/dnsdist/dnsdist.conf' OK!
      

      • enable and start the dnsdist service (or restart)

      % systemctl restart dnsdist
      

      • check that the service has been started without errors:

      % systemctl status dnsdist
         ● dnsdist.service - DNS Loadbalancer
           Loaded: loaded (/usr/lib/systemd/system/dnsdist.service; enabled; vendor preset: disabled)
           Active: active (running) since Wed 2021-04-21 07:45:47 UTC; 2min 26s ago
             Docs: man:dnsdist(1)
                   https://dnsdist.org
          Process: 27803 ExecStartPre=/usr/bin/dnsdist -u dnsdist -g dnsdist --check-config (code=exited, status=0/SUCCESS)
         Main PID: 27804 (dnsdist)
            Tasks: 25 (limit: 8192)
           Memory: 159.4M
           CGroup: /system.slice/dnsdist.service
                   └─27804 /usr/bin/dnsdist -u dnsdist -g dnsdist --supervised --disable-syslog
      
         Apr 21 07:47:21 dnsdist001 dnsdist[27804]: Marking downstream 149.20.1.73:53 as 'down'
         Apr 21 07:47:24 dnsdist001 dnsdist[27804]: Marking downstream 149.20.1.73:53 as 'up'
         Apr 21 07:47:37 dnsdist001 dnsdist[27804]: Marking downstream [2001:4f8:1:f::73]:53 as 'down'
         Apr 21 07:47:38 dnsdist001 dnsdist[27804]: Marking downstream [2001:4f8:1:f::73]:53 as 'up'
         Apr 21 07:47:50 dnsdist001 dnsdist[27804]: Marking downstream 149.20.1.73:53 as 'down'
         Apr 21 07:47:51 dnsdist001 dnsdist[27804]: Marking downstream 149.20.1.73:53 as 'up'
         Apr 21 07:47:58 dnsdist001 dnsdist[27804]: Marking downstream 149.20.1.73:53 as 'down'
         Apr 21 07:47:59 dnsdist001 dnsdist[27804]: Marking downstream 149.20.1.73:53 as 'up'
         Apr 21 07:48:05 dnsdist001 dnsdist[27804]: Marking downstream 149.20.1.73:53 as 'down'
         Apr 21 07:48:06 dnsdist001 dnsdist[27804]: Marking downstream 149.20.1.73:53 as 'up'
      

   2. Testing the load-balancing setup
      
      • we send the request to the port 65053 where dnsdist is listening

      % dig -p 65053 @localhost zoneNNN.dnslab.org +norec
      ; <<>> DiG 9.16.22-Debian <<>> -p 65053 @localhost zoneNNN.dnslab.org +norec
      ; (1 server found)
      ;; global options: +cmd
      ;; Got answer:
      ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43793
      ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
      
      ;; OPT PSEUDOSECTION:
      ; EDNS: version: 0, flags:; udp: 1232
      ;; QUESTION SECTION:
      ;zoneNNN.dnslab.org.            IN      A
      
      ;; ANSWER SECTION:
      zoneNNN.dnslab.org.     60      IN      A       46.101.218.115
      
      ;; Query time: 3 msec
      ;; SERVER: 127.0.0.1#65053(127.0.0.1)
      ;; WHEN: Tue Nov 02 21:20:54 UTC 2021
      ;; MSG SIZE  rcvd: 63
      

      • The AA Flag tell us that the response comes indeed from an authoritative DNS server
   3. Configuring the dnsdist webserver
      
      • Add the following lines to the dnsdist configuration

      webserver("0.0.0.0:8853")
      setWebserverConfig({acl="0.0.0.0/0",password="dnsdist-is-great"})
      

      • Reload the dnsdist

      % systemctl restart dnsdist
      

      • Access the web-interface from a web browser

      http://pdnsrNNN.dnslab.org:8853/
      

3. DNSdist as load balancer for DNS resolver
   
   • In this session we build a load balancer for multiple DNS resolver. We use the local PowerDNS recursor together with a selection of public DNS resolver
   • Save and remove a previous dnsdist configuration /etc/dnsdist/dnsdist.conf and start with a clean file
   • We use Port 65053 on the IPv4 loopback address, as port 53 on all IP addresses is currently occupied by the PowerDNS recursor

   newServer({address="127.0.0.1", qps=10000, order=1}) -- our PowerDNS recursor
   newServer({address="1.1.1.1",   qps=100, order=3})   -- Cloudflare Public DNS
   newServer({address="8.8.8.8",   qps=100, order=2})   -- Google Public DNS
   -- Cache
   pc = newPacketCache(10000, {maxTTL=86400, minTTL=0, temporaryFailureTTL=60, staleTTL=60, dontAge=false})
   getPool(""):setCache(pc)
   -- Load-balancing Policy
   setServerPolicy(leastOutstanding)
   -- local IP address for dnsdist
   setLocal("127.0.0.1:65053")
   

   1. (Re-)start dnsdist
      
      • check the configuration of dnsdist for syntax errors

      % dnsdist -u dnsdist -g dnsdist --check-config
      Configuration '/etc/dnsdist/dnsdist.conf' OK!
      

      • restart the dnsdist service

      % systemctl restart dnsdist
      

      • check that the service has been started without errors:

      % systemctl status dnsdist
         ● dnsdist.service - DNS Loadbalancer
         Loaded: loaded (/usr/lib/systemd/system/dnsdist.service; enabled; vendor preset: disabled)
         Active: active (running) since Wed 2021-04-21 08:20:01 UTC; 1min 34s ago
           Docs: man:dnsdist(1)
                 https://dnsdist.org
        Process: 29682 ExecStartPre=/usr/bin/dnsdist -u dnsdist -g dnsdist --check-config (code=exited, status=0/SUCCESS)
       Main PID: 29684 (dnsdist)
          Tasks: 23 (limit: 8192)
         Memory: 106.1M
         CGroup: /system.slice/dnsdist.service
                 └─29684 /usr/bin/dnsdist -u dnsdist -g dnsdist --supervised --disable-syslog
      
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: Listening on 127.0.0.1:65053
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: dnsdist 1.6.0-rc1 comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to>
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: ACL allowing queries from: 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.168.0.0/16, ::1/128, fc00>
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: Console ACL allowing connections from: 127.0.0.0/8, ::1/128
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: Webserver launched on 0.0.0.0:8053
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: Marking downstream 8.8.8.8:53 as 'up'
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: Marking downstream 1.1.1.1:53 as 'up'
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: Marking downstream 127.0.0.1:53 as 'up'
      Apr 21 08:20:01 dnsdist001 systemd[1]: Started DNS Loadbalancer.
      Apr 21 08:20:01 dnsdist001 dnsdist[29684]: Polled security status of version 1.6.0-rc1 at startup, no known issues reported: OK
      

   2. Testing the load-balancing setup
      
      • we send the request to the port 65053 where dnsdist is listening

      % dig -p 65053 @localhost powerdns.com
      
      ; <<>> DiG 9.16.22-Debian <<>> -p 65053 @localhost powerdns.com
      ; (1 server found)
      ;; global options: +cmd
      ;; Got answer:
      ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2499
      ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
      
      ;; OPT PSEUDOSECTION:
      ; EDNS: version: 0, flags:; udp: 512
      ;; QUESTION SECTION:
      ;powerdns.com.                  IN      A
      
      ;; ANSWER SECTION:
      powerdns.com.           3600    IN      A       188.166.104.92
      
      ;; Query time: 175 msec
      ;; SERVER: 127.0.0.1#65053(127.0.0.1)
      ;; WHEN: Tue Nov 02 21:53:37 UTC 2021
      ;; MSG SIZE  rcvd: 57
      

      • The RD and AD Flags tell us that the response comes indeed from a DNS resolver
   3. Configuring the dnsdist webserver
      
      • Add the following lines to the dnsdist configuration

      webserver("0.0.0.0:8853")
      setWebserverConfig({acl="0.0.0.0/0",password="dnsdist-is-great"})
      

      • Reload the dnsdist

      % systemctl restart dnsdist
      

      • Access the web-interface from a browser

      http://pdnsrXXX.dnslab.org:8853/
      

   4. More tests
      
      • Performance testing a DNS resolver with dnsblast
      • dnsblast is a simple tool that sends random DNS queries for the com TLD
      • compiling dnsblast

      % apt -y install gcc make
      % git clone https://github.com/jedisct1/dnsblast
      % cd dnsblast
      % make
      

      • We use dnsblast to send 5,000 queries at a rate of 100 qps to our dnsdist load-balancer at port 65053

      % ./dnsblast 127.0.0.1 5000 100 65053
      

      • Watch the web-interface metrics during the test
      • Try different parameters for dnsblast
      • see https://github.com/jedisct1/dnsblast for additional information on the dnsblast tool
4. dnsdist as a DoT/DoH proxy
   
   • In this exercise, we will configure dnsdist as a DoH/DoT proxy for the PowerDNS recursor
   • Save and remove a previous dnsdist configuration /etc/dnsdist/dnsdist.conf and start with a clean file
   • Check that the PowerDNS recursor server is up and running (our pre-installed DNS resolver serving classic DNS over Port 53):

   % dig @localhost powerdns.com
   

   • the response should look like this

   ; <<>> DiG 9.16.22-Debian <<>> @localhost powerdns.com
   ; (1 server found)
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38445
   ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
   
   ;; OPT PSEUDOSECTION:
   ; EDNS: version: 0, flags:; udp: 512
   ;; QUESTION SECTION:
   ;powerdns.com.                  IN      A
   
   ;; ANSWER SECTION:
   powerdns.com.           3278    IN      A       188.166.104.92
   
   ;; Query time: 0 msec
   ;; SERVER: 127.0.0.1#53(127.0.0.1)
   ;; WHEN: Tue Nov 02 21:58:59 UTC 2021
   ;; MSG SIZE  rcvd: 57
   

   1. TLS x509 certificates
      
      • Our server already has x509 certificates for TLS from Let's Encrypt. We need to copy the certificate file and the private key file into the dnsdist configuration directory and adjust the file permissions, so that dnsdist can read the files

      % cp /etc/ssl/private/dnslab.* /etc/dnsdist/
      % chown _dnsdist: /etc/dnsdist/*
      

   2. Configuration for the "upstream" DNS resolver
      
      • first we need to tell dnsdist where to find the upstream (existing) DNS resolver. In our case, it is the PowerDNS recursor instance running on the same machine.
        • in dnsdist, you can specify any number of upstream servers with load-balancing parameters, please see the dnsdist website for documentation.
      • we create the file /etc/dnsdist/dnsdist.conf with one line of configuration that defines one upstream DNS resolver (use your favorite text editor nano, emacs, vim):

      newServer({address="127.0.0.1"})
      

   3. Configuration for DNS-over-TLS (DoT)
      
      • our DNS-over-TLS service will run on the (loop-back) IP-Address 127.0.0.10. In an production environment, this would be one of the external addresses of the proxy machine that is reachable from DNS clients.
        • the new configuration line for DoT defines the listen address, the x509 certificate and the private key matching the certificate
      • our configuration file /etc/dnsdist/dnsdist.conf should now look like this:

      newServer({address="127.0.0.1"})
      addTLSLocal('127.0.0.10',
            '/etc/dnsdist/dnslab.crt',
            '/etc/dnsdist/dnslab.key')
      

   4. Starting dnsdist
      
      • check the configuration of dnsdist for syntax errors

      % dnsdist -u dnsdist -g dnsdist --check-config
      Configuration '/etc/dnsdist/dnsdist.conf' OK!
      

      • restart the dnsdist service

      % systemctl restart dnsdist
      

      • check that the service has been started without errors:

      % systemctl status dnsdist
         ● dnsdist.service - DNS Loadbalancer
            Loaded: loaded (/usr/lib/systemd/system/dnsdist.service; enabled; vendor preset: disabled)
            Active: active (running) since Tue 2021-02-09 09:05:34 UTC; 17s ago
              Docs: man:dnsdist(1)
                    https://dnsdist.org
           Process: 111443 ExecStartPre=/usr/bin/dnsdist -u dnsdist -g dnsdist --check-config (code=exited, status=0/SUCCESS)
          Main PID: 111445 (dnsdist)
             Tasks: 19 (limit: 8192)
            Memory: 28.2M
            CGroup: /system.slice/dnsdist.service
                     └─111445 /usr/bin/dnsdist -u dnsdist -g dnsdist --supervised --disable-syslog
      
         Feb 09 09:05:34 doh01 systemd[1]: Stopped DNS Loadbalancer.
         Feb 09 09:05:34 doh01 systemd[1]: Starting DNS Loadbalancer...
         Feb 09 09:05:34 doh01 dnsdist[111445]: Added downstream server 127.0.0.1:53
         Feb 09 09:05:34 doh01 dnsdist[111445]: Listening on 127.0.0.10:853 for TLS
         Feb 09 09:05:34 doh01 dnsdist[111445]: dnsdist 1.6.0-alpha1 comes with ABSOLUTELY NO WARRANTY. This is free software, and you are we>
         Feb 09 09:05:34 doh01 dnsdist[111445]: ACL allowing queries from: 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0>
         Feb 09 09:05:34 doh01 dnsdist[111445]: Console ACL allowing connections from: 127.0.0.0/8, ::1/128
         Feb 09 09:05:34 doh01 dnsdist[111445]: Marking downstream 127.0.0.1:53 as 'up'
         Feb 09 09:05:34 doh01 systemd[1]: Started DNS Loadbalancer.
         Feb 09 09:05:35 doh01 dnsdist[111445]: Polled security status of version 1.6.0-alpha1 at startup, no known issues reported: OK
      

   5. Testing the DoT setup
      
      • we use kdig, the DNS query tool from the Knot DNS Server, to send DNS over TLS queries to our server. But first we install kdig

      % apt install knot-dnsutils
      

      • and now we test DNS-over-TLS

      % kdig @127.0.0.10 powerdns.com +tls
      ;; TLS session (TLS1.3)-(ECDHE-SECP256R1)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
      ;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 13574
      ;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1
      
      ;; EDNS PSEUDOSECTION:
      ;; Version: 0; flags: ; UDP size: 512 B; ext-rcode: NOERROR
      
      ;; QUESTION SECTION:
      ;; powerdns.com.                IN      A
      
      ;; ANSWER SECTION:
      powerdns.com.           2450    IN      A       188.166.104.92
      
      ;; Received 57 B
      ;; Time 2021-11-02 22:12:47 UTC
      ;; From 127.0.0.10@853(TCP) in 0.0 ms
      

      • kdig will print information on the TLS connection in the first line:

      ;; TLS session (TLS1.3)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
      

   6. Configuration for DNS-over-HTTPS (DoH)
      
      • the configuration for DNS-over-HTTPS is very similar to the DoT configuration. Please add the following line (replace the XXX with your participant number) to the file /etc/dnsdist/dnsdist.conf:

      addDOHLocal('127.0.0.10:8443',
          '/etc/dnsdist/dnslab.crt',
          '/etc/dnsdist/dnslab.key')
      

      • check the configuration file for errors

      % dnsdist -u dnsdist -g dnsdist --check-config
      

      • and if no errors are reported, restart the dnsdist service

      % systemctl restart dnsdist
      

      • check that the service has been (re-)started

      % systemctl status dnsdist
         ● dnsdist.service - DNS Loadbalancer
            Loaded: loaded (/usr/lib/systemd/system/dnsdist.service; enabled; vendor preset: disabled)
            Active: active (running) since Tue 2021-02-09 09:12:03 UTC; 5s ago
              Docs: man:dnsdist(1)
                    https://dnsdist.org
           Process: 112038 ExecStartPre=/usr/bin/dnsdist -u dnsdist -g dnsdist --check-config (code=exited, status=0/SUCCESS)
          Main PID: 112040 (dnsdist)
             Tasks: 21 (limit: 8192)
            Memory: 31.2M
            CGroup: /system.slice/dnsdist.service
                    └─112040 /usr/bin/dnsdist -u dnsdist -g dnsdist --supervised --disable-syslog
      
         Feb 09 09:12:02 doh01 systemd[1]: Starting DNS Loadbalancer...
         Feb 09 09:12:03 doh01 dnsdist[112040]: Added downstream server 127.0.0.1:53
         Feb 09 09:12:03 doh01 dnsdist[112040]: Listening on 127.0.0.10:853 for TLS
         Feb 09 09:12:03 doh01 dnsdist[112040]: Listening on 127.0.0.10:443 for DoH
         Feb 09 09:12:03 doh01 dnsdist[112040]: dnsdist 1.6.0-alpha1 comes with ABSOLUTELY NO WARRANTY. This is free software, and you are we>
         Feb 09 09:12:03 doh01 dnsdist[112040]: ACL allowing queries from: 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0>
         Feb 09 09:12:03 doh01 dnsdist[112040]: Console ACL allowing connections from: 127.0.0.0/8, ::1/128
         Feb 09 09:12:03 doh01 dnsdist[112040]: Marking downstream 127.0.0.1:53 as 'up'
         Feb 09 09:12:03 doh01 systemd[1]: Started DNS Loadbalancer.
         Feb 09 09:12:03 doh01 dnsdist[112040]: Polled security status of version 1.6.0-alpha1 at startup, no known issues reported: OK
      

   7. Testing DNS-over-HTTPS
      
      • Again we use the kdig tool to test our new DNS-over-HTTPS service:

      % kdig -p 8443 @127.0.0.10 powerdns.com +https
      ;; TLS session (TLS1.3)-(ECDHE-SECP256R1)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
      ;; HTTP session (HTTP/2-POST)-(127.0.0.10/dns-query)-(status: 200)
      ;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 0
      ;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1
      
      ;; EDNS PSEUDOSECTION:
      ;; Version: 0; flags: ; UDP size: 512 B; ext-rcode: NOERROR
      
      ;; QUESTION SECTION:
      ;; powerdns.com.                IN      A
      
      ;; ANSWER SECTION:
      powerdns.com.           2159    IN      A       188.166.104.92
      
      ;; Received 57 B
      ;; Time 2021-11-02 22:17:38 UTC
      ;; From 127.0.0.10@8443(TCP) in 0.1 ms
      

      • kdig print information about the TLS and HTTPS connection in the first two lines:

      ;; TLS session (TLS1.3)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
      ;; HTTPS session (HTTP/2-POST)-(127.0.0.10/dns-query)-(status: 200)
      

• Lua Policy Engine can be used to implement RRL https://ripe69.ripe.net/presentations/55-powerdns-ripe69.pdf

• DNSdist can be deployed to implement QPS limits https://dnsdist.org/advanced/qpslimits.html and Packet Actions https://dnsdist.org/rules-actions.html
• DNSdist can even use eBPF inside the Linux-Kernel to block incoming DNS messages deep in the network stack (before it reaches user space): https://dnsdist.org/advanced/ebpf.html

• The PowerDNS recursor implements a query rate limiting when it detects that a upstream authoritative server becomes unresponsive
  • This prevents that the upstream authoritative DNS server get more overloaded by repeated queries from the DNS resolver
  • non-resolving-ns-max-fails - Number of failed address resolves of a nameserver name to start throttling it, 0 is disabled. Default: 5
    • dont-throttle-names exclude name server from throttling, can be set dynamically with rec_control add-dont-throttle-names NAME [NAME…]
    • dont-throttle-netmasks exclude name servers from these networks from throttling, can be set dynamically with rec_control add-dont-throttle-netmasks NETMASK [NETMASK…]
    • non-resolving-ns-max-throttle-time Number of seconds to throttle a nameserver with a name failing to resolve
  • server-down-max-fails If a server has not responded in any way this many times in a row, no longer send it any queries for server-down-throttle-time seconds. Afterwards, the resolver will try a new packet, and if that also gets no response at all, it will again throttle for server-down-throttle-time seconds. Even a single response packet will clear the block.
  • Controlling the throttle lists
    • rec_control get-dont-throttle-names will print out the throttle exceptions for name server
    • rec_control get-dont-throttle-netmasks will print out the throttle exceptions for networks
    • rec_control dump-throttlemap FILENAME will write the throttle list to a file
    • rec_control clear-dont-throttle-names NAME [NAME…] will remove a server from the throttle lust
    • rec_control clear-dont-throttle-netmasks NETMASK [NETMASK…] will remove a network from a throttle list

• Cryptography has four purposes:
  • Confidentiality – keeping data secret
  • Integrity – Is it "as sent"?
  • Authentication – Did it come from the right place?
  • Non-Repudiation – Don’t tell me you didn't say that.
• DNSSEC implements: Authentication & Integrity
• The IETF has added confidentiality since 2016:
  • RFC 7858 "Specification for DNS over Transport Layer Security (TLS)" (DNS-over-TLS) https://tools.ietf.org/html/rfc7858
  • RFC 8484 "DNS Queries over HTTPS (DoH)" (DNS-over-HTTPS) https://tools.ietf.org/html/rfc8484
• DNS uses different authenticity and integrity techniques depending on the application.
• Symmetric Cryptography
  • Message Digests (aka: hashes, hash values, fingerprints)
  • Message Authentication Codes
• Asymmetric Cryptography
  • Digital Signatures

• Symmetric cryptography provides both integrity and authenticity.
• A single key is stored and used by both (all) parties.
  • The key encrypts and decrypts.
  • The key is a shared secret.
  • The system is known as pre-shared key (PSK).
  • Securely getting the key to all parties can be a challenge.
• Symmetric cryptography requires mutual trust.
  • "My security is good, but what about the other person's?"
  • If all sides are administered by one party, trust is a non-issue.
  • For DNS, Pre-Shared-Keys (PSK) work well:
    • between primary and secondary servers (TSIG).
    • for dynamic DNS updates (TSIG).
    • for controlling BIND 9 with rndc (TSIG).
  • For DNS, PSKs do not work well:
    • between DNS Resolver servers and auth servers (DNSSEC).
    • between stub resolvers & DNS Resolver servers (DNSSEC amongst other).

• A Pre-Shared Key (PSK) could be used to encrypt a message.
• If it decrypts, confidentially, integrity, and authenticity are assured.
• However, confidentiality was not a design goal (of DNSSEC).
• Encrypting everything is computationally expensive.
• The alternative is more complicated but less expensive.

• Creating a hash of the message is a light-weight alternative to encrypting everything.
  • A hash is also known as a hash value, a message digest (MD) and a fingerprint.
• The sender runs a cryptographic hashing algorithm on the message to produce a fixed-length hash.
  • The message and hash are sent over an insecure path.
• As the sender did, the receiver hashes the message.
• If the hashes match, the message was not modified. Message integrity is proven.
• However, the receiver does not know if the message and hash were both replaced: Message authenticity is unknown.
• Confidentiality is not provided.

• Hashing, with a symmetric key added to the input, efficiently provides integrity and authenticity.
  • There is no confidentiality.
  • A MAC is a fingerprint (MD, hash) created with the message and a PSK as input.
  • The MAC described here, is a keyed HMAC (Hash-based message authentication code).
    • It is used by DNS.

• Although HMACs efficiently assure both the sender's authenticity, and the message's integrity, not all applications can use (pre)shared keys.
  • "Am I really seeing the website mybank.example?"
  • "Is the email I'm reading really from Edward Snowden?"
  • "Is this RRSet actually for theguardian.com?"

• In DNS, asymmetric cryptography is used for DNSSEC.
• This asymmetric cryptography section is a short overview.
• Asymmetric cryptography uses two keys, a pair.
  • Data encrypted with one key can only be decrypted by the other.
  • A key cannot decrypt what it encrypted!
  • One key of a pair is declared as public, the other as private.
  • The private key is highly sensitive, never shared, and must be well protected.
  • The public key is made widely available without concern for who knows it.
  • The technique is known as public key encryption.

    

• PK encryption is used for privacy, to assure only the intended receiver can read a message.
  • Data integrity is also assured.
  • used in DNS-over-TLS and DNS-over-HTTPS
• PK encryption is used for authenticity, assuring the recipient, that the message came from a specific sender.
  • This is known as signing a message.
  • Data integrity is also assured.
  • used in DNSSEC, as well in DNS-over-TLS and DNS-over-HTTPS

• To assure authenticity, a sender encrypts a message with her private key.
  • Anyone decrypting the message with the public key is assured it came from the holder of the private key.
• The message is encrypted, but there is no privacy.

• For efficiency, the message itself is not encrypted.
  • The message is first hashed to create a fingerprint.
  • The fingerprint is encrypted.
  • The signed fingerprint is a digital signature (aka encrypted fingerprint and encrypted hash).

• TSIGs secure the communication of two endpoints.
  • TSIGs uses HMAC (i.e. symmetric encryption).
  • Trust is required between all systems (all endpoints).
  • Securely installing the key on all systems is external to DNS.
  • The endpoints must have reasonably accurate clocks.
• TSIG is independent of DNSSEC.
• TSIG use cases:
  • DNS dynamic updates (client / dhcp-server <-> server)
  • BIND server control messages (rndc <-> server)
  • Server communication (zone transfers, notifies, queries, …)
  • Queries (client <-> server): Impractical and rare.

    

• A TSIG is a dynamically generated pseudo-RR.
  • TSIG RRs are not found in zone files, but do have the standard format of RRs.
  • TSIGs are never cached (TTL=0).
  • A TSIG RR is sent in the additional section.
  • A TSIG assures the integrity and authenticity of the entire DNS message.

    

• TSIG offers a choice of HMAC algorithms:
  • hmac-md5 (deprecated)
  • hmac-sha1 (deprecated)
  • hmac-sha224
  • hmac-sha256
  • hmac-sha384
  • hmac-sha512
• All the algorithms take a random length input and create a fixed length fingerprint.

  Algorithm	fingerprint length
  MD5	16 byte
  SHA1	20 byte
  SHA256	32 byte
  SHA512	64 byte

• DNSSEC Key parameters
• NSEC or NSEC3

• DNSSEC Keyrollover

• Zones can switch from NSEC to NSEC3 and back at any time
• Use pdnsutil set-nsec3 <zone-name> <nsec3-parameter> to switch to NSEC3
• Use pdnsutil unset-nsec3 <zone-name> to switch back to NSEC

% pdnsutil set-nsec3 zoneNNN.dnslab.org "1 0 10 -"
NSEC3 set, please rectify your zone if your backend needs it
% pdnsutil rectify-zone zoneNNN.dnslab.org
Adding NSEC3 hashed ordering information for 'zoneNNN.dnslab.org'
% pdnsutil increase-serial zoneNNN.dnslab.org
SOA serial for zone zoneNNN.dnslab.org set to 1635973224

• The zone is now secured with NSEC3

# dig @localhost xyz.zoneNNN.dnslab.org +dnssec +multi +norec

; <<>> DiG 9.16.22-Debian <<>> @localhost xyz.zoneNNN.dnslab.org +dnssec +multi +norec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 33387
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;xyz.zoneNNN.dnslab.org.        IN A

;; AUTHORITY SECTION:
zoneNNN.dnslab.org.     60 IN SOA pdns010a.dnslab.org. hostmaster.zoneNNN.dnslab.org. (
                                1636009907 ; serial
                                3600       ; refresh (1 hour)
                                1800       ; retry (30 minutes)
                                3542400    ; expire (5 weeks 6 days)
                                60         ; minimum (1 minute)
                                )
zoneNNN.dnslab.org.     60 IN RRSIG SOA 8 3 60 (
                                20211118000000 20211028000000 14634 zoneNNN.dnslab.org.
                                QOtJT7bfN6OvuVbWBTn326PwCuqJ717jQ+kTo61Nymhe
                                ReOKa6IvIyiUBCKXxyrvzhASxR3CAw0x3CukL3Gwdm4F
                                kS3nILwsyKIv/IeREpXLcU1KrUK4fxAXQ81IJh1R96IH
                                s0xAeurl4hdzGgYeydLqGlRdhAics8vGpuqPtnzBybDp
                                5SImpw7KGI50229jpdbgA67c89E4nAkBszLYI8IfhJKM
                                rN0zj7JXNJPHGmA+oeCQnBy5yh+E/6IeTlA2 )
6egermavujrtij09ssmf3r4kg1bhbihl.zoneNNN.dnslab.org. 60 IN NSEC3 1 0 10 - (
                                6TNSMUD7JF5CKL4QMPHMSJ1VCVNIGP2I
                                A NS SOA RRSIG DNSKEY NSEC3PARAM )
6egermavujrtij09ssmf3r4kg1bhbihl.zoneNNN.dnslab.org. 60 IN RRSIG NSEC3 8 4 60 (
                                20211118000000 20211028000000 14634 zoneNNN.dnslab.org.
                                VXf8bF+/EqVndIzWhbUlsuJJ1lNHaF7bYnOpZ3bm9Ij9
                                XOFtG8Oa+YBg4bAr82B2Ra565jVK4uAZ5tL1/69H7xlj
                                BIkZ74KSRgoeI69yjT1fzN04l7V1RHDEjZtz6/NFy1qG
                                9R+J9/vVZXdhJBPhlia6+A5x5+RDdLlZcB2wrWjdvjpR
                                z4hfoy5T4p3MHx4rXRDp3zNQrRMkcdW3Kg/mK1+a5rYA
                                1/WJl+pCVaWLTspwY6wnybNnmx/eBf6H0bO5 )
6tnsmud7jf5ckl4qmphmsj1vcvnigp2i.zoneNNN.dnslab.org. 60 IN NSEC3 1 0 10 - (
                                34KQL2SE77RDH2S50C8DR19DI2OB75VD
                                TXT RRSIG )
6tnsmud7jf5ckl4qmphmsj1vcvnigp2i.zoneNNN.dnslab.org. 60 IN RRSIG NSEC3 8 4 60 (
                                20211118000000 20211028000000 14634 zoneNNN.dnslab.org.
                                fl0z50DIEOtombxYRST2ANGsLtxUSX2Yo6ceMPYTHnvS
                                5gvw3m7+TDkTRVB39Ezcw/W9y7aPMp6HNCk0+ZCytWHI
                                4E4tV0uzrN/dVYSCv1+ooX54JbF23Zk7C76MYKjYO33V
                                cOarWoA8VYiy3uKaKyE752tUWGhVW6X8iQYQbQ0oZmXM
                                z2gOVlPnxLsrV5IRyhRIaFhKTFfcu4TA54l+Dpgz7rd4
                                x3/eHepeaPPlFsBclKZ3rmZ+gIQ1YqAPzqpz )

;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Nov 04 07:11:47 UTC 2021
;; MSG SIZE  rcvd: 992

• The NSEC3PARAM record in the zone

% dig @localhost nsec3param zoneNNN.dnslab.org +dnssec +multi +norec

; <<>> DiG 9.16.22-Debian <<>> @localhost nsec3param zoneNNN.dnslab.org +dnssec +multi +norec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35689
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;zoneNNN.dnslab.org.    IN NSEC3PARAM

;; ANSWER SECTION:
zoneNNN.dnslab.org.     60 IN NSEC3PARAM 1 0 10 -
zoneNNN.dnslab.org.     60 IN RRSIG NSEC3PARAM 8 3 60 (
                                20211118000000 20211028000000 14634 zoneNNN.dnslab.org.
                                JTPUqdF6YqfhcpUBHMomhqspp5ijzDCOiNMXwRbQm1Xp
                                VGTOo8CNRWNCbCnYgpjILz5itUcLSnJPzz6gdKXGuGVI
                                sxFVR/3OKLweaj+pWrlgSkRG1mskYiFN58Y2PRL42P6j
                                oYCXJOyj/J5VcQBJyKWoJWP801t+581hBNEkZOVaELeQ
                                HjH2shbKX5un/0Bc2f4pKM/8W4a2O/K/0avSeFGKklIK
                                g3C4NZKaYFkAxaGapAMQp5CXrpilt7moAXmp )

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Nov 04 07:12:54 UTC 2021
;; MSG SIZE  rcvd: 306

• DNSSEC Risks

• RPZ is an RDNS security mechanism to protect queriers from Internet dangers.
  • It is like a firewall. RPZ prevents access to domain names identified as disreputable.
• A domain name identified as disreputable triggers a policy with an action that denies normal resolution.
  • E.G. a domain name in a URL known for having malware, gets an NXDOMAIN response.
• RPZs are already being widely implemented, but are still only a draft RFC: https://tools.ietf.org/html/draft-vixie-dnsop-dns-rpz (expired 12/2018)
• other RPZ implementations: Bind 9, Unbound 1.11+, policy.rpz (Knot) - partial support, commercial DNS products, …
• RPZ policies are implemented in zone files.
  • RPZs have some unique values, but conform to Master File Format.
  • The RPZ zones are deployed on DNS resolver, not on auth servers.
  • A DNS resolver is usually the secondary to a central RPZ management server that hosts the primary RPZ zones
  • There is no delegation and the NS RRs are unused.
  • An RPZ typically has policies for unrelated names. E.G.:

    www.verybad.org 30 IN CNAME google.com.
    evilstuff.net   30 IN CNAME google.com.
    

    • Note: the owner names in the example above are not fully qualified (FQDN)
• There are five triggers for an RPZ policy.
  • Two are based on the query.
  • Three are based on the RDATA of the response.
• For some of the triggers, special labels are required in the RR owner name.
• There are six actions that a policy can trigger.
  • NXDOMAIN is one, the others are covered soon.

• Keep your PowerDNS software up-to-date
  • use the package manager of your Linux/Unix distribution
    • configure automatic unattended updates
    • consider using the PowerDNS supplied packages
  • or use a cross-platform package manager
    • pkgsrc — https://www.pkgsrc.org
    • Nix — https://nixos.org/nix/
• subscribe to PowerDNS announce mailing list (low volume, new versions and security announcements only) https://mailman.powerdns.com/mailman/listinfo/pdns-announce