bert hubert finally blogs

Four million pings only - aka 1 dimensional DNS radar

2012-01-15T12:07:00.000-08:00

Quick post as I have no time to work on this for now. Ages ago I read a book, I think by Arthur C. Clarke, where powerful atomic bombs were used to generate radar pulses so powerful, the return signal was used to map the entire solar system in one go. The grandeur of this vision impressed me a lot, and I hope that one day we can do it. (Btw, if anybody knows the name of the book, please share!).

If you send out one powerful 'ping' of radar signal, and only measure the strength of the return over time, you don't get a good picture - you learn how much reflection you get, and from how far away (based on the delay). But you don't get the angle. This is why 'real' radars rotate, so they can sweep the sky. (I know there are other reasons).

One of the 2012 goals for the PowerDNS Recursor is to become the DNS resolver with the best perceived experience for the end-users. This means not so much the highest performance (in terms of hundreds of thousands of queries/second, the usual metric), but in terms of getting the best answer to the user within the shortest amount of time.

In doing the math on this challenge, I needed to know how the response times of authoritative DNS servers are distributed, so I instrumented the PowerDNS Recursor to graph this while I sent it runs of 2,000,000 questions from a list of the most popular domain names. I was naively expecting some sort of Poisson distribution centered around 150ms.

But lo and behold, I got this graph. From this you can see that around 10% of answers came in beteen 1 and 2msec. But this graph isn't any kind of nice smooth distribution at all, and I should have realised that.

(the graph contains two runs, called 'plot' and 'plot.1', plus the combination of both runs in Blue)

The speed of light within a fiberoptic cable is around 200,000km/s, and because the answer needs to come back too, this equals around 100km per msec. So if you multiply the y-axis by a hundred, you get a very rough measure of the distance of all authoritative servers queried. And servers are not distributed smoothly! They tend to cluster around hotspots.

So what are these peaks? Well.. the first one turns out to be mostly ANYcasted servers present very closely to xs.powerdns.com. A secondary peak (24ms) appears to be Milan (actual distance: 1000km, but we lose 20ms somewhere within Level3 for no apparent reason), hosting an instance of a.dns.it, plus an instance of b.gtld-servers.net in Stockholm (actual distance: 1500km).

The big void between 50ms and 75ms might correspond to the Atlantic Ocean.

The peak around 84-87ms matches closely with the East Coast of the US, whereas the somewhat broader peak beyond 158ms might well be California. Or Asia!

250ms is about what you'd expect from Australia, the peak from 350ms might again be Australia, but then 'the wrong way round'.

These analyses are very tentative, but I've now seen the same result on 4 different datasets, one of which was measured a few years ago and based on very different techniques, but gave the same result.

The dataset used to generate the graph above can be found on http://xs.powerdns.com/dns-radar, the format is "microseconds errorcode remote-server:port domain-name".

I might publish more details on how to reproduce later, but for now, I thought this was cool enough to share!

PowerDNS Authoritative Server Security Notification 2012-01

2012-01-10T06:11:00.000-08:00

CVE	CVE-2012-0206
Date	10th of January 2012
Credit	Ray Morris of BetterCGI.com.
Affects	Most PowerDNS Authoritative Server versions < 3.0.1 (with the exception of the just released 2.9.22.5)
Not affected	No versions of the PowerDNS Recursor ('pdns_recursor') are affected.
Severity	High
Impact	Using well crafted UDP packets, one or more PowerDNS servers could be made to enter a tight packet loop, causing temporary denial of service
Exploit	Proof of concept
Risk of system compromise	No
Solution	Upgrade to PowerDNS Authoritative Server 2.9.22.5 or 3.0.1
Workaround	Several, the easiest is setting: cache-ttl=0, which does have a performance impact. Please see below.

Affected versions of the PowerDNS Authoritative Server can be made to respond to DNS responses, thus enabling an attacker to setup a packet loop between two PowerDNS servers, perpetually answering each other's answers. In some scenarios, a server could also be made to talk to itself, achieving the same effect.

If enough bouncing traffic is generated, this will overwhelm the server or network and disrupt service.

As a workaround, if upgrading to a non-affected version is not possible, several options are available. The issue is caused by the packet-cache, which can be disabled by setting 'cache-ttl=0', although this does incur a performance penalty. This can be partially addressed by raising the query-cache-ttl to a (far) higher value.

Alternatively, on Linux systems with a working iptables setup, 'responses' sent to the PowerDNS Authoritative Server 'question' address can be blocked by issuing:

iptables -I INPUT -p udp --dst $AUTHIP --dport 53 \! -f -m u32 --u32 "0>>22&0x3C@8>>15&0x01=1" -j DROP

If this command is used on a router or firewall, substitute FORWARD for INPUT.

To solve this issue, we recommend upgrading to the latest packages available for your system. Tarballs and new static builds (32/64bit, RPM/DEB) of 2.9.22.5 and 3.0.1 have been uploaded to our download site. Kees Monshouwer has provided updated CentOS/RHEL packages in his repository. Debian, Fedora and SuSE should have packages available shortly after this announcement.

For those running custom PowerDNS versions, just applying this patch may be easier:

--- pdns/common_startup.cc   (revision 2326)
+++ pdns/common_startup.cc   (working copy)
@@ -253,7 +253,9 @@
       numreceived4++;
     else
       numreceived6++;
-
+    if(P->d.qr)
+      continue;
+      
     S.ringAccount("queries", P->qdomain+"/"+P->qtype.getName());
     S.ringAccount("remotes",P->getRemote());
     if(logDNSQueries) {

It should apply cleanly to 3.0 and with little trouble to several older releases, including 2.9.22 and 2.9.21.

This bug resurfaced because over time, the check for 'not responding to responses' moved to the wrong place, allowing certain responses to be processed anyhow.

We would like to thank Ray Morris of BetterCGI.com for bringing this issue to our attention and Aki Tuomi for helping us reproduce the problem.

Old blog posts back

2011-11-20T10:49:00.000-08:00

Hi everybody,

More than a year ago my old server died, taking down my blog with it. It was a busy year so I only now got round to reviving the content.

It is back now, and you can find the old content here.

Some of my favorite old posts:

The ultimate SO_LINGER page, or: why is my tcp not reliable
Predictions
Reusing UNIX semantics for fun and profit
Secrets in Public: Diffie-Hellman key exchange
Calculating the chance of spoofing an agile source port randomised resolver
The fourteen stages of any real software project

Neutrinos faster than light?

2011-09-23T02:37:00.000-07:00

I'm really hoping it is true, that neutrinos are actually travelling faster than light! As is usually the case with potential breakthroughs in physics, the evidence is complicated.

It would be wonderful if it had been a simple race between light and neutrinos, and the neutrinos arriving first. Sadly, it is not as simple as this.

In reality, "trains" of millions and millions of neutrinos are generated in Switzerland, from where they travel straight through the earth, to be detected in an experiment in Italy. Over three years of measurements, 16111 neutrinos have been recorded and timed.

Each train of neutrinos is 10500 nanoseconds long, and the average neutrino is measured to appear 60 nanoseconds earlier than expected. The problem is that of the millions of neutrinos sent, only a few get detected.

So when the arrival of a neutrino is measured, it is not certain if it is one from the beginning of the 10500 nanosecond long train, or one at the end.

If the train of neutrinos were exactly uniform, you could just take the average travel time, and be done with it. In the end the length of the train would average out. The problem is that the density of neutrinos being sent is absolutely not uniform.

Through crafty statistics however, and by measuring the precise density shape of the train in Switzerland, it is however possible to still generate an average travel time. And this is the number that is being reported, between 50 and 70 nanoseconds too fast (more or less).

This corresponds to around ~~180~~ 18 meters at the speed of light. The distance to between where the beam starts and where it is measured is known to 20 centimeters precise, so that is not a problem. It also appears that the very best timekeeping people in the whole world have been involved in making sure the clocks are running correctly.

It is an impressive effort. It would have been a lot easier if we could just organize a race between a neutrino and a photon. The result now reported is statistical in nature, but the statistics are impressive. It should however be realized that a lot of calculations are needed to get the 60ns number, and a mistake could hide anywhere.

The result can be compared to trying to measure if a (real) train runs on schedule by timing when people walk out of the train station. If you keep that up long enough, you will be able to get results - but a lot of things could mess with your measurements!

On a final note, it might well turn out that neutrinos do not travel faster than light, but that there is another reason why we are getting these results. That might in itself be almost as interesting!

A prior art post: wifi OR bluetooth for powersaving

2011-07-23T11:44:00.000-07:00

While singing my lovely son to sleep just now, an idea popped up in my head which I'm hoping is not yet patented. If it isn't, this post should kill any chances of that ever happening. Here goes.

Bluetooth and Wifi on smartphones both draw power, even when there is no actual Wifi or Bluetooth connection active - the mere act of listening for a potential pairing costs energy.

However, when I'm on my home Wifi, I am never on Bluetooth. When I'm connected to Bluetooth, I'm never eager for a wireless connection.

Thus, wouldn't it be nice if there was a piece of software that shutdown the Bluetooth listener if connected to Wifi? And if that Wifi goes out of range, turn off the Bluetooth again. If that leads to a connection, shut down the Wifi listener.

I'm pretty sure that for many use cases (people with wifi at home and at the office, and bluetooth handsfree in the car), this would save like.. joules per day! ;-)

This post was of course inspired by my phone shutting down while I was reading the web while singing my son to sleep.

So - feel free to make this appear on the Android market or the appstore. And who knows, it might be there already.

PowerDNS Authoritative Server 3.0 has been released!

2011-07-22T05:29:00.000-07:00

Available from:
 * http://downloads.powerdns.com/releases/pdns-3.0.tar.gz
 * http://downloads.powerdns.com/releases/rpm/pdns-static-3.0-1.x86_64.rpm
 * http://downloads.powerdns.com/releases/deb/pdns-static_3.0-1_amd64.deb
 * http://downloads.powerdns.com/releases/rpm/pdns-static-3.0-1.i386.rpm
 * http://downloads.powerdns.com/releases/deb/pdns-static_3.0-1_i386.deb

These files also come with GPG signatures (append .sig).

RHEL/CentOS "native" RPMs are usually contributed by Kees Monshouwer
(thanks!) pretty quickly after a release on:
http://www.monshouwer.eu/download/3th_party/pdns-server/

The release notes are also available, with clickable links, on 
http://doc.powerdns.com/changelog.html#changelog-auth-3-0

Warning
Version 3.0 of the PowerDNS Authoritative Server is a major upgrade. Please refer to Section 1, “From PowerDNS Authoritative Server 2.9.x to 3.0” for important information on correct and stable operation, as well as notes on performance and memory use. Known issues as of RC3 include: Not all new features are fully documented yet

	Note
	Released on the 22nd of July 2011 RC1 released on the 4th of April 2011 RC2 released on the 19th of April 2011 RC3 released on the 19th of July 2011

AFNIC, the French registry
IPCom's RcodeZero Anycast DNS, a subsidiary of NIC.AT, the Austrian registry
SIDN, the Dutch registry
.. (awaiting details) ..

This release has received exceptional levels of community support, and we'd like to thank the following people in addition to those mentioned explicitly below: Peter Koch (DENIC), Olaf Kolkman (NLNetLabs), Wouter Wijngaards (NLNetLabs), Marco Davids (SIDN), Markus Travaille (SIDN), Leen Besselink, Antoin Verschuren (SIDN), Olafur Gudmundsson (IETF), Dan Kaminsky (Recursion Ventures), Roy Arends (Nominet), Miek Gieben (SIDN), Stephane Bortzmeyer (AFNIC), Michael Braunoeder (nic.at), Peter van Dijk, Maik Zumstrull, Jose Arthur Benetasso Villanova (Locaweb), Stefan Schmidt, Roland van Rijswijk (Surfnet), Paul Bakker (Brainspark/Fox-IT), Mathew Hennessy, Johannes Kuehrer (Austrian World4You GmbH), Marc van de Geijn (bHosted.nl), Stefan Arentz and Martin van Hensbergen (Fox-IT), Christof Meerwald, Detlef Peeters, Jack Lloyd, Frank Altpeter, frederik danerklint, Vasiliy G Tolstov, Brielle Bruns, Evan Hunt, Ralf van der Enden, Marc Laros, Serge Belyshev, Christian Hofstaedtler, Charlie Smurthwaite, Nikolaos Milas, ..
Changes between RC3 and final:

Slight tweak to the pipebackend to ease DNSSEC operations (commit 2239, commit 2247). Also fix pipebackend support in pdnssec tool (commit 2244).
Upgrade the experimental native Lua backend to the latest version from Frederik Danerklint (commit 2240) and include this backend in the .deb packages (commit 2242)
Remove IPv6 dependency, it was only possible to run master/slave operations on a server with at least one IPv6 address. Some very old virtualized setups turned out to have no IPv6 at all. Fix incommit 2246.

Changes between RC2 and RC3:

PowerDNS Authoritative Server could not be configured to use an IPv6 based resolving backend. Solved in commit 2191.
LDAP backend reconfigured the timezone (TZ) setting of the daemon, leading to confusing logfile entries. Fixed by Christian Hofstaedtler in commit 2913, closing ticket 313.
Non-DNSSEC capable backends could crash on DNSSEC queries. Fixed in commit 2194 and commit 2196 (thanks to Charlie Smurthwaite) closing ticket 360.
Errors looking up a UID or GID were reported confusingly ('Success'), fixed in commit 2195, closing ticket 359.
Fix compilation against older MySQL, client libraries (commit 2198, commit 2199, commit 2204), especially for older RHEL/CentOS. Also addresses the failure to look in lib64 directory for PostgreSQL.
Sqlite3 needs write access not just to its database file, but also to the directory it is in. If this wasn't the case, no useful error message was provided. Improvement in commit 2202.
Update of MongoDB backend (commit 2203, commit 2212).
'pdnssec hash-zone-record' emitted an inverted warning about narrow NSEC3 hashes. Spotted by Jan-Piet Mens, fix in commit 2205.
PowerDNS can fill out default fields for SOA records, but neglected to do so if the SOA record was matched by an incoming ANY question. Spotted by Marc Laros & others. Fixes ticket 357, code incommit 2206.
PowerDNS would mistreat binary data in TXT records. Fix in commit 2207. Again spotted by Jan-Piet Mens. Closes ticket 356.
Add experimental Lua backend by our star contributor Fredrik Danerklint. commit 2208.
Christoph Meerwald discovered our RRSIG freshness checking checked more than the intended RRSIG (on the SOA record). Fix in commit 2209.
Christoph Meerwald discovered we got confused by TSIG signed EDNS-adorned queries, since we expected the EDNS OPT pseudorecord to be the very last record. Fix in commit 2214.
Christoph Meerwald discovered that when using SOA outgoing editing we would sign and THEN edit. This was not productive. Fixed in commit 2215.
Add missing-but-documented pdnssec command 'disable-dnssec'. Spotted by Craig Whitmore. Plus fixed misleading --help output. Code in commit 2216.
By popular demand, a tweak which makes an overloaded database no longer restart PowerDNS but to drop queries until the database is available again. Code in commit 2217, lightly tested. Enable by setting 'overload-queue-lengh=100' (for example).
By suggestion of Miek Gieben of SIDN, add SOA-EDIT mode 'EPOCH' which sets the SOA serial number to the 'UNIX time'. Implemented in commit 2218.
Added some US export control & ECCN to documentation, needed because of DNSSEC content. Update in commit 2219.
Fix up various spelling mistakes and badly formatted messages (commit 2220 and commit 2221) by Maik Zumstrull and 'anonymous'.
After a lot of thought, we now handle CNAMEs to names outside our knowledge ('bailiwick') exactly as in BIND 9.8.0, even though our way was standards compliant too. It confused things. Update incommit 2222 and commit 2224.
Tweak sqlite3 library location detection for newer Ubuntu versions. Change in commit 2223.
DNSSEC SQL schema improvements allowing for the use of constraints and foreign keys in commit 2225, by Gerald Gruenberg, closing ticket 371.
Add support for EDNS option 'edns-subnet', based on draft-vandergaast-edns-client-subnet (commit 2226, commit 2228, commit 2229, commit 2230, commit 2231, commit 2233).
Silence SIGCHLD warning from Perl when used to power 'pipe' backends (commit 2232).
Add experimental support, off by default, for draft-edns-subnet. See commit 2233 and commit 2239 for details how to use this feature.
PostgreSQL and LDAP backends can now deal with a restart of their respective servers. Many thanks to Peter van Dijk for debugging and Nikolaos Milas for supplying a reproduction path of the problem (& much nagging). Fixes in commit 2233 and commit 2235.
Jan-Piet Mens discovered that records inserted by Lua on zone retrieval did not get correct 'ordername' and 'auth' fields for DNSSEC. Fixed in commit 2174.
Silenced various relevant and less relevant compilation warnings (commit 2175). Thanks to Serge Belyshev for pointing out the error in our ways.
Steve Bauer discovered we would cache empty recursive answers in some cases. Addressed in commit 2176.
James Cloos reported that 'pdnssec check-zone' tripped over SRV records. Fixed this, and added check-zone to the regression tests. Code in commit 2177.
DNSSEC regression tests were added in commits 2178, 2179, 2182, 2186 We test against the fine tools from NLNetLabs.
Secure DNSSEC delegations to ourselves picked wrong zone to serve the DS record from. Fixed in commit 2180, commit 2181, commit 2183. reported by Niek Willems of InterNLnet.
Stef Van Dessel suggested we made our RPMs state explicitly that they need glibc 2.4 on Linux. Code in commit 2184.
John Leach discovered our MySQL based backends would wait for ages on a failing MySQL server. The patch merged in commit 2189 reduces the timeout significantly, which is especially useful with haproxy and mysqlproxy.
commit 2190 fixes a crash reported by Marc Laros when using a non-DNSSEC capable backend. Should also improve non-DNSSEC performance.

Changes between RC1 and RC2:

Zone2sql sent out the wrong 'COMMIT' statement in sqlite mode. In addition, in this mode, zone2sql would not emit statements to update the domains table unless the 'slave' setting was chosen. Code in commit 2167.
We dropped the Authoritative Answer flag on an out-of-bailiwick CNAME referral, which was unneccessary. Code in commit 2170.
Kees Monshouwer discovered that we failed to detect the location of PostgreSQL on RHEL/CentOS. Fix in commit 2144. In addition, commit 2162 eases detection of MySQL on RHEL/CentOS 64 bits systems.
Marc Laros re-reported an old bug in the internally used 'pdns' backend where details of the SOA record were not filled out correctly. Resolved in commit 2145.
Jan-Piet Mens found that our TSIG signed SOA zone fresheness check was signed incorrectly. Fixed in commit 2147. Improved error messages that helped debug this issue in commit 2148, commit 2149.
Jan-Piet Mens helped debug an issue where some servers were "almost always" unable to transfer a TSIG signed zone correctly. Turns out that the TSIG signing code used an internal timestamp and not the remote timestamp. Because of good NTP synchronization this quite often was not a problem. Fix in commit 2159.
Thor Spruyt of Telenet discovered that the PowerDNS code would try to emit DNS answers over TCP of over 65535 bytes long, which failed. We now truncate such answers properly. Code in commit 2150.
The Slave engine now reuses an existing database connection, removing the need to create a new database connection every minute (and worse, log about it). Code in commit 2153.
Fix a potential Year 2106 bug in the TSIG signing code. Because we care (commit 2156).
Added experimental support for the 'DANE' TLSA record which is used to authenticate SSL certificates via DNSSEC. commit 2161.
Added experimental support for the MongoDB 'NoSQL' backend, contributed by fredrik danerklint in commit 2162.

On to the release notes. Next to DNSSEC, other major new features include:

TSIG for authorizing and authenticating AXFR requests & incoming zone transfers (Code in 2024, 2025, 2033, 2034). This allows for retrieving TSIG protected content, as well as serving it.
Per zone also-notify.
MyDNS compatible backend, allowing for 'instantaneous' migration from this authoritative nameserver. Code in commit 1418, contributed by Jonathan Oddy.
PowerDNS can now slave zones over IPv6 and notify IPv6 remotes of updates. Already. Code in commit 2009 and beyond.
Lua based incoming zone editing, allowing masters or signing slaves to add information to the zone they will (re-)serve. Implemented in commit 2065. To enable, use LUA-AXFR-SCRIPT zone metadata setting.
Native Oracle backend with full DNSSEC support. Contributed by Maik Zumstrull, then at the Steinbuch Centre for Computing at the Karlsruhe Institute of Technology.
"Also-notify" support, implemented by Aki Tuomi in commit 1400. Support for Generic SQL backends and for the BIND backend. Further code in commit 1360.
Support for binding to thousands of IP addresses, code in commit 1443.
Generic MySQL backend now supports stored procedures. Implemented in commit 2084, closing ticket 231.
Generic ODBC backend compiles again, and is reported to work for some users that need it. Code contributed in ticket 309, author unknown.
Massively parallel slaving infrastructure, able to check the freshness of thousands of remote zones per second, plus perform many incoming zone transfers simultaneously. Sponsored by Tyler Hall, code in 1449, 1500, 1859
Core DNS logic replaced completely to deal with the brave new world of DNSSEC.

Bugs fixed:

sqlite2 and sqlite3 backends used MySQL-style escaping, leading to SQL errors in some cases. Discovered by Sten Spans. Fixed in commit 1342.
Internal webserver no longer prints '1e2%'. Bug rediscovered by Jeff Sipek. Fixed in commit 1342.
PowerDNS would refuse to serve domain names with spaces in them, or otherwise non-printable characters. Addressed in commit 2081.
PowerDNS can now serve escaped labels, as described by RFC 4343. Data should be present in backends in that escaped form. Code in commit 2089.
In some cases, we would include duplicate CNAMEs. In addition, we would hand out a full root-referral when not configured to in some cases (ticket ticket 223). Discovered by Andreas Jakum, fixed incommit 1344.
Shane Kerr discovered we would corrupt DNS transaction IDs from the packet cache on big endian systems. Fix in commit 1346, closing ticket 222.
PowerDNS did not use RFC 1982 serial arithmetic, leading to a SOA serial number of 1 to be regarded as older than 4400000000, when in fact it is 'newer'. Issue (re-)discovered by Jan-Piet Mens.
BIND backend got confused of a zone's filename changed after a configuration reload. Fix in commit 1347, closing ticket 228.
When restarted by the Guardian, PowerDNS will perform a full multi-threaded cache cleanup, which took a long time and could crash. Fix in commit 1364.
Under artificial circumstances, PowerDNS would never clean its packet cache. Found by Marcus Goller, fix in commit 1399 and commit 1408. This update also retunes the cleanup frequency.
Packetcache would cache things it should not have been caching. Fixes in commits 1407, 1488, 1869, 1880
When processing incoming notifications, the BIND backend was case-sensitive, and would disregard notifications in the wrong case. Discovered by 'Dolphin', fix in commit 1420.
The init.d script did not mention the 'reload' command. Code in commit 1463, closes ticket 233.
Generic SQL Backends would sometimes emit obscure error messages. Fix in commit 2049.
PowerDNS would be confused by embedded NULs in domain names, and would also mess up the escaping of some characters. Fix in commit 1468, commit 1469, commit 1478, commit 1480,
SOA queries for the name of a delegation point were not referred. Fix in commit 1466, closing ticket 224. In addition, queries for AAAA for a CNAMEd record pointing to a name with no AAAA would deliver a direct SOA, without the CNAME in between. Fix in commit 1542, commit 1607. Also, wildcard CNAMEs pointing to a record without the type requested suffered from the same issue, fix incommit 1543.
On processing an incoming AXFR, once an MX or SRV record had been seen, all future fields got a 'priority' entry as well. This had no operational impact, but looked messy. Fixed in commit 1437.
Aki Tuomi discovered that the BIND zonefile parser would misrepresent 'something IN MX 15 @'. Fix in commit 1621.
Marco Davids discovered the BIND zonefile parser would trip over really long lines. Fix in commit 1624, commit 1625.
Thomas Mieslinger discovered that our webserver would only be started after dropping privileges, which could cause problems. Fix in commit 1629.
Zone2sql did quite often not do exactly what was required, which users fixed by editing the SQL output. Revamped in commit 2032.
An Ubuntu user discovered in Launchpad bug 600479 that restarting database threads cost a lot of memory. Normally this is rare, except in case of problems. Addressed in commit 1676.
BIND backend could crash under (very) high load with very large numbers of zones (hundreds of thousands). Fixed in commit 1690.
Miek Gieben and Marco Davids spotted that PowerDNS would answer the version.bind query in the IN class too. Bug reported via twitter! Fix in commit 1709.
Marcus Lauer and the OpenDNSSEC project discovered that outgoing notifications did not carry the 'aa' flag. Fixed in commit 1746.
Debugging PowerDNS, or backgrounding it, could cause crashes. Fixed by Anders Kaseorg in commit 1747.
Fixed a bug that could cause crashes on launching thousands of backend connections. Never observed to occur, but who knows. Fix in commit 1792.
Under some circumstances, large answers could be truncated in mid-record. While technically legal, this upset a number of resolver implementations (including the PowerDNS Recursor!). Fixed incommit 1830, re-closes ticket 200.
Jan Piet Mens and Florian Weimer discovered we had problems dealing with escaped labels and escaped TXT fields. Fixed in commit 2000.
After 2.2 billion queries, statistics would wrap oddly. Fix in commit 2019, closing ticket 327.

Improvements:

Long TXT records are now split into 255-byte components automatically. Implemented in commit 1340, reported by Darren Gamble in ticket 188.
When receiving large numbers of notifications, PowerDNS would check these synchronously, leading to a slowdown for other services. Fixed in commit 2058, problem diagnosed by Richard Poole of Heart Internet.
Fixed compilation on newer compilers and newer versions of Boost. Changes in 1345 (closes ticket 227), 1391, 1394, 1425, 1427, 1428, 1429, 1440, 1653, thanks to Ruben Kerkhof and others.
Moved Generic PostgreSQL backend over to the newer E'' style escapes. commit 2094.
Compilation fixes for Mac OS X 10.5.7 in commit 1389, thanks to Tobias Markmann.
We can now bind to scoped IPv6 addresses, lack spotted by Darren Gamble. Part of the fix is in commit 2018.
Built-in query cache can now also cache queries which lead to multiple answers. Code in commit 2069.
Prodded on by Jan Piet Mens, we now support 'unknown types' (which look like TYPE65534).
Add 'slave-renotify' to retransmit notifies for slaved zones, which is helpful when acting as a 'signing slave' for a hidden master. Code in commit 1950.
No longer let zone2sql and zone2ldap import BIND 'hint' zones. commit 1998.
Allow for timestamps to explicitly be specified in (s)econds. Code in commit 1398, closing ticket 250.
Zones with URL and MBOXFW records can be transferred over AXFR, code in commit 1464.
Maik Zumstrull cleaned up the BIND Backend makefile, plus taught our init.d script to read /etc/default/pdns. Code in commit 1601, commit 1602.
Generic SQL backends now support multiple masters in the domains table. Code in commit 1857. Additionally, masters can also have :port numbers. Code in commit 1858.

As a community service, the glibc 2.14 'fixed bugs' with descriptions

2011-06-13T12:18:00.000-07:00

Hi everybody,

The venerable GNU C Library just saw the release of 2.14. Within the release notes there is a list of bug numbers that are addressed by this release. Some of these are highlighted in the release notes, but most aren't.

I did some scripting and here is a lightly edited list of things fixed. I added a link to bug 10149, you can guess the URL for the rest. I bolded things that might actually hit my programs (and who knows, yours):

(potentially) security related:

Bug 10149 - stack guard should lead with zero byte to gain protections

Bug 11892 - putenv()/setenv() unbounded alloca()

Bug 12393 - ld.so: insecure handling of privileged programs' RPATHs with

Bug 12671 - multiple vulnerabilities in netdb.h/aliases.h/glob.h

Rest:

Bug 386 - pthread_create returns ENOMEM but should return EAGAIN

Bug 6420 - Mtrace deadlock

Bug 7101 - getopt message for ambiguous options could be more helpful

Bug 10138 - Outdated config.guess/sub

Bug 10157 - Wrong value for sysconf(_SC_CPUTIME) or

Bug 11099 - INT_FIELD_MAYBE_NULL changed behaviour on x86_64

Bug 11257 - need finer control of group unioning in /etc/nsswitch.conf

Bug 11558 - No way to set some options in /etc/resolv.conf

Bug 11634 - tst-audit6.c doesn't compile without AVX support

Bug 11697 - pt_chown doesn't work when the PTY's gid is already correct

Bug 11724 - ld.so - Initialization and Termination Functions incorrectly

Bug 11781 - Interoperability problems between malloc hook and GCC 4.5.0

Bug 11799 - si_code is not SI_USER on raise()

Bug 11820 - sys/user.h requires additional header in x86_64 to define

Bug 11857 - Missing documentation in regex.h

Bug 11895 - pselect incorrecly handles small negative timeouts on old

Bug 11901 - __libc_message(do_abort = 1) will deadlock if called from malloc

Bug 11952 - glibc may use uninitialized DTV slot, return NULL for

Bug 12052 - posix_spawn() nonconformance (POSIX_SPAWN_SETSCHEDPARAM)

Bug 12083 - aio_init() treatment of aio_num argument looks buggy

Bug 12350 - Resolver doesn't save RES_SNGLKUP/RES_SNGLKUPREOP state in

Bug 12420 - On AMD64 linux, getcontext resets FPU exception mask.

Bug 12432 - backtrace() fails with recursive function on 64-bit

Bug 12445 - printf() stack corruption in case of positional parameters +

Bug 12453 - Broken thread local storage (TLS) initialization

Bug 12454 - Inconsistency detected by ld.so: dl-deps.c: 622:

Bug 12460 - AVX audit test failures with gcc 4.6

Bug 12469 - Race condition in configure.in check for necessary ranlib

Bug 12489 - prelinking ldso causes binaries to segfault upon startup

Bug 12509 - dlopen(path_to_lib, RTLD_LOCAL|RTLD_NOLOAD) leaks memory

Bug 12510 - elf/dl-lookup.c: STB_GNU_UNIQUE/ELF_RTYPE_CLASS_COPY lookup

Bug 12511 - elf/dl-lookup.c: STB_GNU_UNIQUE/ELF_RTYPE_CLASS_COPY lookup

Bug 12518 - memcpy acts randomly (and differently) with overlapping areas

Bug 12527 - Off by one bug with ftell() with fmemopen()

Bug 12583 - fnmatch: integer overflow in computation of the required

Bug 12587 - sysconf(_SC_*CACHE) returns 0 for all caches on some CPUs.

Bug 12597 - SSE4 strncmp failure

Bug 12625 - mntent operations provide no indication of failure due to

Bug 12626 - __backtrace_symbols_fd uses of out-of-scope storage in stack

Bug 12631 - wcp[n]cpy are required by POSIX 2008

Bug 12650 - Memory leak with dlopen() and thread-local storage variables

Bug 12653 - undefined references to ssse3 routines when trying to link

Bug 12655 - fix a comment in sysdeps/unix/sysv/linux/sys/syscall.h

Bug 12684 - Multi-request DNS lookups do not properly fall back to

Bug 12685 - fopen doesn't honor last byte of valid modes

Bug 12713 - coreutils-8.12 "make check" thinks glibc-2.13's "getcwd()" is

Bug 12714 - getaddrinfo(AF_INET6) does not return scope_id info provided

Bug 12717 - declaration of getnameinfo() is not POSIX compliant

Bug 12723 - pathconf for a FIFO returns a different value than fpathconf

Bug 12724 - fclose violates POSIX 2008 on seekable input streams

Bug 12734 - resolver failures without even sending a query.

Bug 12766 - SEGV in error_at_line(3)

Bug 12775 - Typo in sysdeps/x86_64/fpu/e_powl.S

Bug 12782 - POSIX strerror_r quality of implementation

Bug 12792 - perror violates POSIX regarding ferror status

Bug 12795 - bits/resource.h is outdated

Bug 12811 - regexec/re_search consumes huge amounts of memory

Bug 12813 - Linux x86_64: glibc should prefer the vDSO over vsyscalls

Locale:

Bug 9730 - sv_FI time format does not match fi_FI

Bug 9732 - dz_BT Dzongkha collation order

Bug 9809 - Please add Kurdish locale for Kurdish Sorani (CKB)

Bug 11258 - es_CR locale update

Bug 11487 - [Patch] to fix yesexpr and noexpr to use Po (Yes) and Jo (No)

Bug 11532 - Support old DOS Lithuanian character sets in iconv

Bug 11578 - sync glibc Latin American paper sizes with CLDR 1.8.1

Bug 11653 - Incorrect LC_MONETARY symbol of es_NI.utf-8

Bug 11668 - Paper Size is wrong for locale es_NI (A4 -> Letter)

Bug 11837 - GB18030-2005 is not supported!

Bug 11869 - LANGUAGE not taken into account unless LC_MESSAGES is set to

Bug 11945 - Month names in Russian Localization should be in lowercase

Bug 11947 - New locale for Meadow Mari language

Bug 11987 - missing info on first day of week in Slovenian (sl_SI) locale

Bug 12158 - Please add the new lij_IT locale

Bug 12178 - New locale wae_CH, request for inclusion

Bug 12200 - Please add the new yue_HK locale file

Bug 12346 - Estonia (et_EE) joins the eurozone on Jan 1 2011

Bug 12449 - Please add the new lb_LU locale

Bug 12541 - update for indian locale for U+20B9 (New Rupee Symbol)

Bug 12545 - [PATCH] localedef: fix error check for size_t < 0

Bug 12551 - New locales for Swahili (Kenya and Tanzania)

Bug 12582 - Incorrect date and time formats in en_SG locale

Bug 12611 - New locale for Fulah (Senegal)

Bug 12601 - iconv(3) doesn't handle invalid sequence properly

Bug 12660 - Recent changes in tk_TM locale

Bug 12681 - New locale for Bemba (Zambia)

Bug 12711 - changes required for adding new currency symbol in indian

Bug 12738 - Please add the new os_RU locale

Bug 12746 - Encoding mismatch in se_NO file

Bug 12777 - iconv mapping of U+0385 in CP1258 is likely incorrect

Bug 12788 - [PATCH] setlocale sets the locale of LC_ALL incorrect to 'C'

Bug 12814 - ISO-2022-JP-2 conversion of U+20AC gives strange result

PowerDNS Authoritative Server 3.0RC1 released! Now with DNSSEC!

2011-04-05T01:24:00.000-07:00

I'm very proud to announce the first Release Candidate for PowerDNS Authoritative Server 3.0, now with full support for DNSSEC, TSIG, IPv6 master/slave, per-zone metadata and Lua zone editing. The DNSSEC support is 'fully automatic' - if everything goes well, all that is required is to set 'pdnssec secure-zone powerdns.com' and your zone is secured.

Read on for more information! To download, head to http://powerdnssec.org/downloads

	Warning
Version 3.0 of the PowerDNS Authoritative Server is a major upgrade. Please refer to Section 1, “From PowerDNS Authoritative Server 2.9.x to 3.0” for important information on correct and stable operation, as well as notes on performance and memory use. Known issues as of RC1 include: Not all new features are documented yet Queries for 'empty non-terminals' may give confusing results We are not 100% convinced all corner cases of NSEC3/NXDOMAIN give correct responses. Common cases function well DNSSEC has only been benchmarked up to 2000 queries/second but not beyond A lot more database connections are made and released

Warning

Version 3.0 of the PowerDNS Authoritative Server is a major upgrade. Please refer to Section 1, “From PowerDNS Authoritative Server 2.9.x to 3.0” for important information on correct and stable operation, as well as notes on performance and memory use.
Known issues as of RC1 include:

Not all new features are documented yet
Queries for 'empty non-terminals' may give confusing results
We are not 100% convinced all corner cases of NSEC3/NXDOMAIN give correct responses. Common cases function well
DNSSEC has only been benchmarked up to 2000 queries/second but not beyond
A lot more database connections are made and released

	Note
	RC1 released on the 4th of April 2011

Version 3.0 of the PowerDNS Authoritative Server brings a number of important features, as well as over two years of accumulated bug fixing.
The largest news in 3.0 is of course the advent of DNSSEC. Not only does PowerDNS now (finally) support DNSSEC, we think that our support of this important protocol is among the easiest to use available. In addition, all important algorithms are supported.
Complete detail can be found in Chapter 12, Serving authoritative DNSSEC data. The goal of 'PowerDNSSEC' is to allow existing PowerDNS installations to start serving DNSSEC with as little hassle as possible, while maintaining performance and achieving high levels of security.
Tutorials and examples of how to use DNSSEC in PowerDNS can be found linked from http://powerdnssec.org.
This release has received exceptional levels of community support, and we'd like to thank the following people in addition to those mentioned explicitly below: Peter Koch (DENIC), Olaf Kolkman (NLNetLabs), Wouter Wijngaards (NLNetLabs), Marco Davids (SIDN), Markus Travaille (SIDN), Leen Besselink, Antoin Verschuren (SIDN), Olafur Gudmundsson (IETF), Dan Kaminsky (Recursion Ventures), Roy Arends (Nominet), Miek Gieben (SIDN), Stephane Bortzmeyer (AFNIC), Michael Braunoeder (nic.at), Peter van Dijk, Maik Zumstrull, Jose Arthur Benetasso Villanova (Locaweb), Stefan Schmidt, Roland van Rijswijk (Surfnet), Paul Bakker (Brainspark/Fox-IT), Mathew Hennessy, Johannes Kuehrer (Austrian World4You GmbH), Marc van de Geijn (bHosted.nl), Stefan Arentz and Martin van Hensbergen (Fox-IT), Christof Meerwald, Detlef Peeters, Jack Lloyd, Frank Altpeter, frederik danerklint, Vasiliy G Tolstov, Brielle Bruns, Evan Hunt, Ralf van der Enden.
On to the release notes. Next to DNSSEC, other major new features include:

TSIG for authorizing and authenticating AXFR requests & incoming zone transfers (Code in 2024, 2025, 2033, 2034). This allows for retrieving TSIG protected content, as well as serving it.
Per zone also-notify.
MyDNS compatible backend, allowing for 'instantaneous' migration from this authoritative nameserver. Code in commit 1418, contributed by Jonathan Oddy.
PowerDNS can now slave zones over IPv6 and notify IPv6 remotes of updates. Already. Code in commit 2009 and beyond.
Lua based incoming zone editing, allowing masters or signing slaves to add information to the zone they will (re-)serve. Implemented in commit 2065. To enable, use LUA-AXFR-SCRIPT zone metadata setting.
Native Oracle backend with full DNSSEC support. Contributed by Maik Zumstrull, then at the Steinbuch Centre for Computing at the Karlsruhe Institute of Technology.
"Also-notify" support, implemented by Aki Tuomi in commit 1400. Support for Generic SQL backends and for the BIND backend. Further code in commit 1360.
Support for binding to thousands of IP addresses, code in commit 1443.
Generic MySQL backend now supports stored procedures. Implemented in commit 2084, closing ticket 231.
Generic ODBC backend compiles again, and is reported to work for some users that need it. Code contributed in ticket 309, author unknown.
Massively parallel slaving infrastructure, able to check the freshness of thousands of remote zones per second, plus perform many incoming zone transfers simultaneously. Sponsored by Tyler Hall, code in 1449, 1500, 1859
Core DNS logic replaced completely to deal with the brave new world of DNSSEC.

Bugs fixed:

sqlite2 and sqlite3 backends used MySQL-style escaping, leading to SQL errors in some cases. Discovered by Sten Spans. Fixed in commit 1342.
Internal webserver no longer prints '1e2%'. Bug rediscovered by Jeff Sipek. Fixed in commit 1342.
PowerDNS would refuse to serve domain names with spaces in them, or otherwise non-printable characters. Addressed in commit 2081.
PowerDNS can now serve escaped labels, as described by RF4343. Data should be present in backends in that escaped form. Code in commit 2089.
In some cases, we would include duplicate CNAMEs. In addition, we would hand out a full root-referral when not configured to in some cases (ticket ticket 223). Discovered by Andreas Jakum, fixed in commit 1344.
Shane Kerr discovered we would corrupt DNS transaction IDs from the packet cache on big endian systems. Fix in commit 1346, closing ticket 222.
PowerDNS did not use RF1982 serial arithmetic, leading to a SOA serial number of 1 to be regarded as older than 4400000000, when in fact it is 'newer'. Issue (re-)discovered by Jan-Piet Mens.
BIND backend got confused of a zone's filename changed after a configuration reload. Fix in commit 1347, closing ticket 228.
When restarted by the Guardian, PowerDNS will perform a full multi-threaded cache cleanup, which took a long time and could crash. Fix in commit 1364.
Under artificial circumstances, PowerDNS would never clean its packet cache. Found by Marcus Goller, fix in commit 1399 and commit 1408. This update also retunes the cleanup frequency.
Packetcache would cache things it should not have been caching. Fixes in commits 1407, 1488, 1869, 1880
When processing incoming notifications, the BIND backend was case-sensitive, and would disregard notifications in the wrong case. Discovered by 'Dolphin', fix in commit 1420.
The init.d script did not mention the 'reload' command. Code in commit 1463, closes ticket 233.
Generic SQL Backends would sometimes emit obscure error messages. Fix in commit 2049.
PowerDNS would be confused by embedded NULs in domain names, and would also mess up the escaping of some characters. Fix in commit 1468, commit 1469, commit 1478, commit 1480,
SOA queries for the name of a delegation point were not referred. Fix in commit 1466, closing ticket 224. In addition, queries for AAAA for a CNAMEd record pointing to a name with no AAAA would deliver a direct SOA, without the CNAME in between. Fix in commit 1542, commit 1607. Also, wildcard CNAMEs pointing to a record without the type requested suffered from the same issue, fix in commit 1543.
On processing an incoming AXFR, once an MX or SRV record had been seen, all future fields got a 'priority' entry as well. This had no operational impact, but looked messy. Fixed in commit 1437.
Aki Tuomi discovered that the BIND zonefile parser would misrepresent 'something IN MX 15 @'. Fix in commit 1621.
Marco Davids discovered the BIND zonefile parser would trip over really long lines. Fix in commit 1624, commit 1625.
Thomas Mieslinger discovered that our webserver would only be started after dropping privileges, which could cause problems. Fix in commit 1629.
Zone2sql did quite often not do exactly what was required, which users fixed by editing the SQL output. Revamped in commit 2032.
An Ubuntu user discovered in Launchpad bug 600479 that restarting database threads cost a lot of memory. Normally this is rare, except in case of problems. Addressed in commit 1676.
BIND backend could crash under (very) high load with very large numbers of zones (hundreds of thousands). Fixed in commit 1690.
Miek Gieben and Marco Davids spotted that PowerDNS would answer the version.bind query in the IN class too. Bug reported via twitter! Fix in commit 1709.
Marcus Lauer and the OpenDNSSEC project discovered that outgoing notifications did not carry the 'aa' flag. Fixed in commit 1746.
Debugging PowerDNS, or backgrounding it, could cause crashes. Fixed by Anders Kaseorg in commit 1747.
Fixed a bug that could cause crashes on launching thousands of backend connections. Never observed to occur, but who knows. Fix in commit 1792.
Under some circumstances, large answers could be truncated in mid-record. While technically legal, this upset a number of resolver implementations (including the PowerDNS Recursor!). Fixed in commit 1830, re-closes ticket 200.
Jan Piet Mens and Florian Weimer discovered we had problems dealing with escaped labels and escaped TXT fields. Fixed in commit 2000.
After 2.2 billion queries, statistics would wrap oddly. Fix in commit 2019, closing ticket 327.

Improvements:

Long TXT records are now split into 255-byte components automatically. Implemented in commit 1340, reported by Darren Gamble in ticket 188.
When receiving large numbers of notifications, PowerDNS would check these synchronously, leading to a slowdown for other services. Fixed in commit 2058, problem diagnosed by Richard Poole of Heart Internet.
Fixed compilation on newer compilers and newer versions of Boost. Changes in 1345 (closes ticket 227), 1391, 1394, 1425, 1427, 1428, 1429, 1440, 1653, thanks to Ruben Kerkhof and others.
Moved Generic PostgreSQL backend over to the newer E'' style escapes. commit 2094.
Compilation fixes for Mac OS X 10.5.7 in commit 1389, thanks to Tobias Markmann.
We can now bind to scoped IPv6 addresses, lack spotted by Darren Gamble. Part of the fix is in commit 2018.
Built-in query cache can now also cache queries which lead to multiple answers. Code in commit 2069.
Prodded on by Jan Piet Mens, we now support 'unknown types' (which look like TYPE65534).
Add 'slave-renotify' to retransmit notifies for slaved zones, which is helpful when acting as a 'signing slave' for a hidden master. Code in commit 1950.
No longer let zone2sql and zone2ldap import BIND 'hint' zones. commit 1998.
Allow for timestamps to explicitly be specified in (s)econds. Code in commit 1398, closing ticket 250.
Zones with URL and MBOXFW records can be transferred over AXFR, code in commit 1464.
Maik Zumstrull cleaned up the BIND Backend makefile, plus taught our init.d script to read /etc/default/pdns. Code in commit 1601, commit 1602.
Generic SQL backends now support multiple masters in the domains table. Code in commit 1857. Additionally, masters can also have :port numbers. Code in commit 1858.

PowerDNSSEC: packages available, ready for light production use

2011-01-11T03:28:00.000-08:00

Dear PowerDNS Community,

With the help of many of you, we've now brought 'PowerDNSSEC' to the point where it is in light production. Several of our important domains have already been migrated to the PowerDNS Authoritative Server 3.0 prereleases. Several PowerDNS users have done the same with their domains (please let us know!).

We are pleased to announce the regular availability of documentation, packages and tarballs for testing. On http://powerdnssec.org/downloads/packages you will find RPM/DEB for 32-bit and 64-bit Linux. On http://powerdnssec.org/downloads you will find tarballs which can be compiled on other systems.

For more information head over to http://www.powerdnssec.org (which of course is powered by PowerDNSSEC).

Documentation is on http://doc.powerdns.com/powerdnssec-auth.html

Even more information is on http://wiki.powerdns.com/trac/wiki/PDNSSEC - including how to get started, and how to get help.

In brief, PowerDNSSEC will allow you to continue operating as normal in many cases, with only slight changes to your installation. There is no need to run signing tools, nor is there a need to rotate keys or run scripts.

Particularly, if you run with Generic MySQL, Generic PostgreSQL or Generic SQLite3, you should have an easy time. A small schema update is required, plus an invocation of 'pdnssec secure-zone domain-name ; pdnssec rectify-zone domain-name' per domain you want to secure. And that should be
it.

Supported are:

NSEC
NSEC3 in ordered mode (pre-hashed records)
NSEC3 in narrow mode (unmodified records)
Zone transfers (for NSEC)
Import of 'standard' private keys from BIND/NSD
Export of 'standard' private keys
RSASHA1
RSASHA256
"Pure" PostgreSQL, SQLite3 & MySQL operations
Hybrid BIND/PostgreSQL/SQLite3/MySQL operation
Front-signing slaved data from legacy installations

See http://doc.powerdns.com/dnssec-supported.html for more specifications.

To join the fun, download the tarball and packages which can be found on the sites above, and let us know how it works for you!

To clarify, we do not recommend taking the current code snapshot into heavy production, but we are getting close.

Kind regards,
Bert

Linux or UNIX programmer? Go get The Linux Programming Interface: A Linux and UNIX System Programming Handbook

2010-12-03T00:33:00.000-08:00

The Linux Programming Interface: A Linux and UNIX System Programming Handbook
Are you a Linux or UNIX programmer? Get this book. Do you know a Linux or UNIX programmer and want to give him (her? ;-)) a gift? Get this book. I thought it was a stack of manpages, which would already have been great, but this book is the true successor to the historic Stevens works on UNIX. If you think you don't need this book since you know everything already, that's what I thought too, and I was wrong. Even if you won't read it, the 1552 pages will look really good on your desk.

So go get the book already.

http://www.amazon.com/Linux-Programming-Interface-System-Handbook/dp/1593272200/ref=sr_1_1?ie=UTF8

PowerDNS Recursor additional Lua hooks for IPv6 DNS64 and Renumbering

2010-11-14T13:47:00.000-08:00

Dear PowerDNS Community,

The PowerDNS Recursor is currently being extended with additional Lua hooks
and extra infrastructure to support flexible DNS64 operations, plus perform
on-the-fly IPv4 or IPv6 renumbering.

DNS64 is described on http://tools.ietf.org/html/draft-ietf-behave-dns64-11
and in brief: 

  "DNS64 is a mechanism for synthesizing AAAA records from A records.  DNS64
   is used with an IPv6/IPv4 translator to enable client-server communication
   between an IPv6-only client and an IPv4-only server, without requiring any
   changes to either the IPv6 or the IPv4 node"

Those of you with an interest in these features are invited to test out the
following *pre-release*, specifically to let us know if the API is sufficient
for your needs:

http://svn.powerdns.com/snapshots/pdns-recursor-3.3-hooks.tar.bz2

It can be compiled like any other PowerDNS Recursor release. 

New in the version are the 'nodata()' and 'postresolve()' Lua hooks. Nodata
functions just like nxdomain(), except that it gets called when a domain
exists, but the requested type doesn't. This is where DNS64 happens.

Postresolve() is different, and very powerful - it gets handed the complete
DNS answer as it would be sent out, ready for modification from Lua. This is
where one might for example perform on the fly IP address renumbering.

In the release you can find powerdns-example-script.lua which contains a
working sample for both of the new hooks. This script can also be viewed on
http://wiki.powerdns.com/trac/browser/branches/pdns-dns64/pdns/powerdns-example-script.lua

Note: DO NOT TAKE THIS SCRIPT INTO PRODUCTION - it blacks out important
sites

To get going without disturbing any existing nameservers on your computer,
compile the PowerDNS Recursor, and start like this:
 $ ./pdns_recursor --local-address=0.0.0.0 --local-port=5300 --daemon=no
   --socket-dir=./ --lua-dns-script=powerdns-example-script.lua 

Known defects are:
 postresolve() can't yet access the original dns rcode
 there is no way for nodata() to set the TTL to the SOA minimum value
  as specified by draft-ietf-behave-dns64

Please let us know your thoughts so we can make sure the API has everything
needed for great DNS64 and renumbering operations!

Kind regards,

Bert Hubert

The "leaky abstraction" of the POSIX file interface

2010-10-02T12:28:00.000-07:00

Hi everybody,

Lately I've been looking into large scale database & key/value storage engine performance, and the results were not very good. Machines that should seemingly be able to load the full .COM zone with no problems turned out to have a very hard time to do so - even though COM zone and indexes all fit comfortably in RAM. Despite this, loads of disk i/o ensued.

This led to some investigations on how Linux and the various storage engines interact. Through tweaking a lot of settings, decent performance was achieved, but this process did drive home the fact that your operating system might have a hard time guessing "what you want".

Any decent operating system is fitted with an in-memory cache to speed up disk access. The difference in speed between a disk read and a memory read is so stunning (many orders of magnitude) that being disk bound or being memory bound can make or break a solution.

Naively, we'd want the operating system to cache exactly the data we want it to cache. However, the operating system can't read our minds, and may decide to not dedicate all of the system memory to do exactly what you want.

This issue is worth a blog post in its own right, because if you want dependable performance, it is not good if your kernel decides on Monday to do the right thing, and on Tuesday to take 2 days to finish a job that previously ran in 15 minutes - simply because something else has decided to use the cache in the meantime!

While investigating how reads and writes actually hit the platter, I wrote the following little "exploit" that tickles most operating systems into a flurry of (at first thought) unexpected disk activity. Try to predict what this does:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

// $ dd if=/dev/zero of=blah bs=1024000 count=1000  # this creates a 1G empty file
// $ sudo sysctl vm.drop_caches=3                   # this empties the caches
// $ vmstat 1 & ./writerreader
// and stand back in awe


int main()
{
  int fd = open("./blah", O_WRONLY);
  if(fd < 0) {
    perror("open");
    exit(0);
  }
  
  char c[2];
  for(int n=0; n < 128000; ++n) {
    if(pwrite(fd, (void*)c, 2, (n+1)*8192 - 1) < 0) {   // write out 2 bytes
      perror("pread");
      exit(0);
    }
  }
  fsync(fd);
  close(fd);
}

Try adding up how much read & write activity was caused by writing out 128KB of data.
Please note that there is very little an OS can do to improve this! It is just a fact of life.

So why did I call this a 'leaky abstraction' in the title of this post? Joel Spolsky (who has a blog, Joel on Software, which you should definitely read) invented this term to describe the situation where any kind of API, which supposedly hides underlying details from you, will from time to time confront you with results of said details.

And in this case, the details mean that writing 128KB of data leads to around 2G of I/O (1G of reads, 1G of writes).

So why does this happen? At the very base, disks operate in terms of (mostly) 512 byte sectors. You can't perform any reads or writes smaller than a sector. In this case, it means that writing 2 bytes requires first reading the sector(s) which straddle the two bytes, adding the two bytes, and then writing them out again.

To make matters worse, internally most operating systems don't think in terms of sectors, but in terms of blocks and pages - which are even larger. The sample program above is tuned to perform 2 byte writes which accurately straddle 2 pages - leading to 2 full pages to be read and written per 2 byte operation. And a page tends to be 4096 bytes!

Getting back to the beginning of this post, the effect described above partially explains the really bad performance observed in some key/value storage engines, engines which perform millions of tiny little pwrites, leading to massive read i/o, where you did not expect it.

In a follow-up post, I will probably delve into how to improve this situation, and perhaps by that time I will have gotten round to a solution that might be generic enough to help more projects than just mine to gain more control over how to optimize loads so that they do not cause more disk i/o than you'd want.

In the meantime, I hope to have entertained you with some arcane knowledge!

You can now own the house from which PowerDNS Recursor 3.3 was released!

2010-09-23T23:39:00.000-07:00

Yes, you too can live in the house from which PowerDNS Recursor 3.3 was released!

Details are here! Special price for you ;-)

Close to the city centers of The Hague and Delft!

PowerDNS Recursor 3.3 released!

2010-09-23T11:59:00.000-07:00

Hi everybody,

We're proud to announce the release of the PowerDNS Recursor 3.3!

It can be downloaded from http://www.powerdns.com/

Version 3.3 fixes a number of small but persistent issues,
rounds off our IPv6 %link-level support and adds an important
feature for many users of the Lua scripts.

In addition, scalability on Solaris 10 is improved.

Bug fixes:

* 'dist-recursor' script was not compatible with pure POSIX
/bin/sh, discovered by Simon Kirby. Fix in commit 1545.
* Simon Bedford, Brad Dameron and Laurient Papier discovered
relatively high TCP/IP loads could cause TCP/IP service to
shut down over time. Addressed in commits 1546, 1640, 1652,
1685, 1698. Additional information provided by Zwane
Mwaikambo, Nicholas Miell and Jeff Roberson. Testing by
Christian Hofstaedtler and Michael Renner.
* The PowerDNS Recursor could not read the 'root zone' (this
is something else than the root hints) because of an
unquoted TXT record. This has now been addressed, allowing
operators to hardcode the root zone. This can improve
security if the root zone used is kept up to date. Change
in commit 1547.
* A return of an old bug, when a domain gets new nameservers,
but the old nameservers continue to contain a copy of the
domain, PowerDNS could get 'stuck' with the old servers.
Fixed in commit 1548.
* Discovered & reported by Alexander Gall of SWITCH, the
Recursor used to try to resolve 'AXFR' records over UDP.
Fix in commit 1619.
* The Recursor embedded authoritative server messed up
parsing a record like '@ IN MX 15 @'. Spotted by Aki Tuomi,
fix in commit 1621.
* The Recursor embedded authoritative server messed up
parsing really really long lines. Spotted by Marco Davids,
fix in commit 1624, commit 1625.
* Packet cache was not DNS class correct. Spotted by "Robin",
fix in commit 1688.
* The packet cache would cache some NXDOMAINS for too long.
Solving this bug exposed an underlying oddity where the
initial NXDOMAIN response had an overly long (untruncated)
TTL, whereas all the next ones would be ok. Solved in
commit 1679, closing ticket 281. Especially important for
RBL operators. Fixed after some nagging by Alex Broens
(thanks).

Improvements:

* The priming of the root now uses more IPv6 addresses.
Change in commit 1550, closes ticket 287. Also, the IPv6
address of I.ROOT-SERVERS.NET was added in commit 1650.
* The rec_control dump-cache command now also dumps the
'negative query' cache. Code in commit 1713.
* PowerDNS Recursor can now bind to fe80 IPv6 space with
'%eth0' link selection. Suggested by Darren Gamble,
implemented with help from Niels Bakker. Change in commit
1620.
* Solaris on x86 has a long standing bug in port_getn(),
which we now work around. Spotted by 'Dirk' and 'AS'.
Solution suggested by the Apache runtime library, update in
commit 1622.
* New runtime statistic: 'tcp-clients' which lists the number
of currently active TCP/IP clients. Code in commit 1623.
* Deal better with UltraDNS style CNAME redirects containing
SOA records. Spotted by Andy Fletcher from UKDedicated in
ticket 303, fix in commit 1628.
* The packet cache, which has 'ready to use' packets
containing answers, now artificially ages the ready to use
packets. Code in commit 1630.
* Lua scripts can now indicate that certain queries will have
'variable' answers, which means that the packet cache will
not touch these answers. This is great for overriding some
domains for some users, but not all of them. Use
setvariable() in Lua to indicate such domains. Code in
commit 1636.
* Add query statistic called 'dont-outqueries', plus add IPv6
address :: and IPv4 address 0.0.0.0 to the default
"dont-query" set, preventing the Recursor from talking to
itself. Code in commit 1637.
* Work around a gcc 4.1 bug, still in wide use on common
platforms. Code in commit 1653.
* Add 'ARCHFLAGS' to PowerDNS Recursor Makefile, easing 64
bit compilation on mainly 32 bit platforms (and vice
versa).
* Under rare circumstances, querying the Recursor for
statistics under very high load could lead to a crash
(although this has never been observed). Bad code removed &
good code unified in commit 1675.
* Spotted by Jeff Sipek, the rec_control manpage did not list
the new get-all command. commit 1677.
* On some platforms, it may be better to have PowerDNS itself
distribute queries over threads (instead of leaving it up
to the kernel). This experimental feature can be enabled
with the 'pdns-distributes-queries' setting. Code in commit
1678 and beyond. Speeds up Solaris measurably.
* Cache cleaning code was cleaned up, unified and expanded to
cover the 'negative cache', which used to be cleaned rather
bluntly. Code in commit 1702, further tweaks in commit
1712, spotted by Darren Gamble, Imre Gergely and Christian
Kovacic.

Changes between RC1, RC2 and RC3.

* RC2: Fixed linking on RHEL5/CENTOS5, which both ship with a
gcc compiler that claims to support atomic operations, but
doesn't. Code in commit 1714. Spotted by 'Bas' and Imre
Gergely.
* RC2: Negative query cache was configured to grow too large,
and was not cleaned efficiently. Code in commit 1712,
spotted by Imre Gergely.
* RC3: Root failed to be renewed automatically, relied on
fallback to make this happen. Code in commit 1716, spotted
by Detlef Peeters.

Some notes on Solaris 10 x86, 64 bit compilation, bugs and memory allocators

2010-08-28T01:20:00.000-07:00

Over the past few months, I've spent a lot of time getting the PowerDNS Recursor to perform well on Solaris 10 on x86. Initially, I thought this could not be a lot of work since there are many happy Recursor users on UltraSPARC. "How hard could it be?"

Turns out that Solaris x86 and Solaris UltraSPARC are different in important respects.

What follows is a rather long winded story of a mostly stranger in a somewhat strange land. I view the world through Linux glasses. Some of the pain described below can indubitably be ascribed to that. However, some of the bits below are plainly caused by Oracle not doing a good job maintaining Solaris on x86. This situation is not bound to improve, it appears.

Before starting the rant in earnest, I'd like to thank one (so far) anonymous Sun/Oracle employee who helped me through the forest of Solaris bugtrackers, 'IDRs' and without whom this problem would definitely not have been solved. I'd also like to thank Ad, Bert, John, Martijn and Robin over at a big PowerDNS deployment for sticking through this whole adventure, and for pressuring Sun to actually fix the issues.

Here goes.

The first thing we noticed was that , the 'Ports' event multiplexer failed to work on x86 applications, as described in long standing Solaris bug 'CR 6268715 "library/libc port_getn(3C) and port_sendn(3C) not working on Solaris x86"'. Apache, libevent and PowerDNS all contain workarounds for this bug, but that workaround does come with performance implications. At the very least it is worrying.

Secondly, it turns out that Solaris 10 on x86 can't link 64 bits binaries as generated by system gcc compiler, at least, not those binaries using Thread Local Storage for objects at global scope. This is Solaris bug 'CR 6354160', aka 'Solaris linker includes more than one copy of code in binary when linking gnu object code', which we worked around by changing PowerDNS so it could be compiled as one big C++ file.

Using the native Sun Studio compiler failed, because it is not compliant enough with the C++ standard to compile PowerDNS, and the changes required were non-trivial.

Although both issues (ports_getn() and 64 bits linking) were known, and fixes were available in OpenSolaris, these had not made it into Solaris 10 production releases.

Eventually, PowerDNS was able to work around both bugs, but in the case of 6268715 at a runtime performance cost (note: Sun has now shipped 'IDR145429-01' which fixes this).

Which brings us to performance. For some reason, even though the PowerDNS Recursor uses 'share nothing' threads, there was no scalability when using multiple threads on Solaris. In fact performance was rather dismal anyhow, even with only one thread.

Firstly, we discovered that having multiple threads try to wait on a single socket does not scale beyond a single thread. This was fixed by having only a single thread wait on the socket, and manually distributing queries over threads in a round-robin fashion.

This turned out to help slightly, but not decisively. We then discovered that the default Solaris x86 memory allocator ('malloc()') is effectively single-threaded (unlike the UltraSPARC variant, which is completely different!). Solaris ships with no less than two alternative mallocs, called -lmtmalloc and -lumem respectively. Using libumem helped for benchmarking.

Finally, for Solaris, we had to bring back an old favorite, the 'fork-trick' which makes the whole PowerDNS Recursor fork itself into multiple processes, which helped bring Solaris performance up to par with our other major platform, Linux. We don't yet know why our 'share nothing' threads end up interfering with each other.

The resulting work was taken into production.. and crashed within 5 minutes of heavy load, indicating an out of memory error. With a 64 bit binary on an 8 gigabyte machine, this seemed doubtful.

After some further investigations, it was found that while libumem certainly was faster for multithreaded code, but that it also wastes memory on a prodigious scale. To be honest, this may be due to the fact that the g++ c++ runtime libraries are not making optimal use of the allocator, or our use of get/set/swap/makecontext(), but the amount of memory used was staggering. Think 450MB for storing 10MB of content.

We studied some of the articles available online, among which was 'A Comparison of Memory Allocators' on the 'Oracle Sun Development Network'. This one indeed showed graphs of libumem using large amounts of memory, and a thing called ptmalloc using very little. Oddly enough, ptmalloc is (more or less) the default allocator for Linux too.

We then built a PowerDNS with all the workarounds, plus ptmalloc linked in, and now finally have something that survives production use!

Rounding this off:

Solaris x86 is remarkably different from Solaris UltraSPARC (different bugs, different allocators)
Do not have n>1 threads wait on a single datagram socket filedescriptor, it does not scale
There now IS an IDR to get ports_getn() working, IDR145429-01, which should also speed up Apache and several other high-performance applications for Solaris
To build 64 bits binaries with thread local storage (__thread) at global scope, concatenate all your C++ into one big file, and compile that one
Be aware that the default allocator on Solaris 10 x86 is single-threaded
Be aware that both mtmalloc and libumem may use prohibitive amounts of memory for some programs
Consider ptmalloc3
We still have to investigate why fork() scales better than pthread_create()
Make sure that you have some friends within Sun engineering ;-)

All in all, we still consider Solaris 10 x86 a 'supported platform' for the PowerDNS Recursor, but along the way we had some doubts.. Solaris 10 on UltraSPARC continues to work very well meanwhile!

Bert

Better statistical regression tests: Release your inner German!

2010-06-13T12:28:00.000-07:00

Hi everybody,

The PowerDNS Recursor 3.2 release is holding up well for almost all users, but still some slight issues have crept up. One of the issues involved, where we needed to work around an artefact/issue/quirk/oddity/bug in the UltraDNS servers (depending on who you talk to), turned out to be.. a faulty workaround.

The workaround looked like it should have caused a lot of problems in production, but apparently did not. The PowerDNS Recursor is very well tested before each release (by replaying billions of anonymized packets donated by large scale Recursor users). Such testing catches large scale problems, but small scale problems can get lost in the noise of the internet - huge amounts of DNS queries produce failures not because of PowerDNS, but because the domains themselves are broken.

To fix this, and also to determine the exact impact of the failed workaround, we now have an automated test tool that tries to resolve all 1 million domains which Alexa regards as the most important. There is a strong WWW bias in their domain names, but we can still be reasonably sure that any regression in PowerDNS that is important is sure to be reflected in the success of resolving these 1 million domains.

The testing tool we wrote, 'dnsbulktest' behaved as expected, and immediately uncovered bugs in our parallel packet sending/receiving infrastructure (part of the PowerDNS Authoritative Server prereleases). In addition, the amounts of traffic generated blew away several firewalls, leading to network downtime. Way to go!

After those issues were addressed, the numbers from the regression tests turned out not to add up. To have any confidence in numbers produced, it helps if the number of timeouts plus the number of received packets eventually equals the number of packets sent. Getting everything to match up took quite some time, but again fixed some bugs here and there unrelated to the testing tool.

And now, my "inner German" is satisfied, and all the numbers match up perfectly:

In this case, all 1 million Alexa domains were queried once with 'www.' prepended, and once without. Quite a number of domains return 'No Data' without the 'www.'. The NXDOMAIN number is truly odd, but when 'dnsbulktest' is run against BIND, a similar number pops up. Apparently, quite a few of the 'one million most popular domain names' are unavailable after 24 hours. Makes you wonder.

Next up is scripting this tool so it will be run frequently and graphing the results, giving us a good indication of the state of the DNS as well as of the state of the PowerDNS Recursor!

Oh, and on a final note, fixing up the workaround mentioned earlier caused a repeatable 1.6% decrease in the number of 'errors'. So that fix has been applied, now with the feeling that it actually fixes more than a single 'broken domain'!

PowerDNSSEC Available For Testing!

2010-04-22T00:17:00.000-07:00

Dear PowerDNS people,

On http://wiki.powerdns.com/trac/wiki/PDNSSEC you will find the newest

version of PowerDNS with DNSSEC support built in. This version is

tentatively called 'PowerDNS Authoritative Server 3.0-pre', to signify its

pre-release status, but also to make it clear that DNSSEC will be part of

the mainline PowerDNS.

The status of PowerDNSSEC is that it is interesting to look at, and

functional enough to experiment with. It is not suitable for production, nor

is PowerDNSSEC guaranteed to remain compatible with its current

configuration form.

However, the good news is that signing a DNSSEC zone is now as simple as

entering 'pdnssec sign-zone powerdnssec.org'. Any changes to your zone are

automatically re-signed, there is no need to do anything by hand.

However, please study http://wiki.powerdns.com/trac/wiki/PDNSSEC for

cautions on what will work and what does not work right now!

Kind regards,

Bert Hubert

A few notes on procurement

2010-04-20T22:00:00.000-07:00

Every once in a while I have to deal with a formal (public) procurement situation. And as a technical guy, this hurts. A lot. It is enough to make you want to pull out your hair and scream in pain.

(dear customers & contacts, if you think this post is about you specifically, it is not - I am venting steam about all procurements I've been involved with. Also, I have come quite well out of several of these procedures. It is just that it hurts!)

Procurement goes something like this. Somewhere in a company is a guy who needs a banana.  But, because of the scale of the company, or simply because they are like that, he can't simply go out and buy a banana.

So, he has to involve the procurement department. This department is filled with legal people, and folks otherwise uninterested in the details of bananas. But they do want to do a good job, so they get down to work.

Questionnaires are drafted. What constitutes a good banana? Is a banana the only choice? Will the supply of bananas be guaranteed? How can we store them? For how long? If the banana fails to please, who is responsible? How will we deal with defective shipments? If the bananas are stolen in transport, but the invoice has already been sent, should it be paid? These are not trivial things.

Eventually, this process ends up with a REQUEST FOR PROPOSAL FOR SUPPLY AND DELIVERY OF SELF-CONTAINED AND PEELABLE NATURAL PRODUCT PROVIDING SUSTENANCE.

In this Request for Proposal is a list of items the delivered product should comply with ('the compliance matrix'). It has such vital requirements as:

Product provides lasting sustenance 


Product must preferrably be yellow 

Product should have limited variability in color

Product can be transported 

Product will be delivered in a suitable vessel/container/ship/boat/car/train 

Product remains edible for 1 hour

Product remains edible for 1 day 

Product remains edible for 1 week 

Product remains edible for 1 month 

Product remains edible for 3 months 

Product remains edible for 1 year 

Product remains edible for 5 years 

Product shall comply with RS232 standard for serial communications 

Product shall not require specific temperature ranges for storage 

Product must comply with ISO-32423-2 humidity requirements 

Product must not cause allergic reactions 

Product must be peelable 

Product must be clearly identified with a sticker 

Product must have a non-edible peel 

Product must optionally be delivered in a bundle of products 

Vendor must describe shape and form of product, including typical curvature   ratios 

Vendor should provide guidance on disposal of product, including, but not   limited to possible slipperiness of peel 

Etc, etc

Update: it happened for real! Thanks to Peter van Dijk for spotting this gem:

Update: And another one!

This compliance matrix will often contain hundreds or even thousands of items. The matrix is affixed with a little note that informs the reader that the procurement process will favour 'lowest cost compliant solution'.

This matrix is then mixed together with no less than 200 pages of general terms and conditions, vendor assessment forms, environmental statements, non-disclosure agreements, ethical statements, delivery and payment conditions.

A variety of fruit vendors receive the Request For Proposals and some shrug their shoulders, but in other places bidding teams will be formed. Such teams often number dozens of people.

These people wade through the hundreds of pages of legalese and requirements, and finally consult an actual farmer, relaying the demands of the procuring party.

This poor guy is then asked if there is a fruit that complies with the requirements, and after a while he might figure out that a banana would suit the bill. Probably.

Then attention is turned to the compliance matrix, and the little note about the importance of full compliance.  Yes, the product remains edible for one hour, and usually 1 week, maybe a month, but definitely not 3 months.

Sad faces all round - so we are not compliant? Well says the farmer, if you take a banana off the tree real early, it might be edible after three months, but not for the first two. No matter says the bidding team, and enters 'COMPLIANT' for 3 months!

Next up, how about a full year? No says the farmer, no way. Ah, but the legal eagle of the bidding team has discovered that the matrix does not provide for who the 'product' should be edible! Would a rat eat a one year old banana? Definitely! COMPLIANT!

But now.. 5 years? Dare we say it? This is where the farmer draws the line, but at a stroke of genius, the legal team okays a statement that says 'PARTIALLY COMPLIANT (*)' and adds wording that after five years of fermenting, bananas can stimulate the growth of nutrient-rich mushrooms!

Next up are the really odd questions. RS232 compliance? Does the customer really want that? Or did he copy paste that in? Much soul searching ensues. The RFP document quite clearly states that the vendor may only contact the procurement department of the procuring party, and that any other contact will lead to disqualification. Clarification requests will delay the process, possibly to such an extent that the response is no longer admissable.

Finally the team cops out with a general statement that RS232 compliant connectors can optionally be supplied.

And thus it continues - the bidding team navigates the ethical boundaries ('no allergic reactions?  put down COMPLIANT!'), and finally delivers an equally astounding 200 page response, including its own (competing & conflicting) general terms and conditions, delivery and payment instructions and whatnot.

Over at the customer, these responses are now marked by the procurement people who disregard all notes and other things, and simply count the number of 'COMPLIANT' requirements.  The most honest responses are immediately disqualified, since they mostly came in as non-compliant ('our banana remains edible for 3 weeks, tops').

Over a thousand pages of responses are now forwarded to the original guy asking for a banana.  The only thing he cared about is getting some really good bananas, and if he would need to pick them up himself.  Oddly enough, the document only asked for pricing per ton, does not specify if the bananas will be delivered, and while it contains a lot of wording on curvature ratios, the actual taste of bananas remains undiscussed.

In the meantime, the farmer would really really just like to ship a crate of bananas as a sample and get down to business.

And the original guy?  He already works somewhere else, and in the end not a single banana was sold..

PowerDNS Recursor 3.2 Release Candidate 1

2010-02-10T12:16:00.001-08:00

Hi everybody,

Please find below the release notes of the PowerDNS Recursor version 3.2,
release candidate 1.

RC1 is already deployed in a number of large places, and it appears to be
holding up well. In addition, a number of future users have performed
stringent testing and performance measurements, and it appears this version
works satisfactorily.

It is also observed that this release candidate provides for vastly improved
performance compared to 3.1.7.*, even bringing us close to the very
impressive numbers measured by users of the Nominum Vantio and Nominum CNS
software.  On modern hardware, the PowerDNS Recursor may in fact be faster,
and certainly better value for money. For more details, please see below.

If you are looking forward to deploying PowerDNS Recursor version 3.2, now
is a good time to testdrive RC1.

We are very interested in hearing your experiences, and look forward to
fixing any issues found before the final release is made. If nothing
important pops up, this is expected to happen next week.

Download from:

* http://svn.powerdns.com/snapshots/rc1/
  (tar.bz2, "universal" i386/x86 .rpm and .deb packages, .md5 and pgp
   signatures)

(Nominum, Nominum CNS & Nominum Vantio are trademarks owned by
Nominum)

Release notes
- -------------
Version with clickable links:
http://doc.powerdns.com/changelog.html#CHANGELOG-RECURSOR-3-2

The 3.2 release is the first major release of the PowerDNS
Recursor in a long time. Partly this is because 3.1.7.*
functioned very well, and delivered satisfying performance,
partly this is because in order to really move forward, some
heavy lifting had to be done.

As always, we are grateful for the large PowerDNS community
that is actively involved in improving the quality of our
software, be it by submitting patches, by testing development
versions of our software or helping debug interesting issues.
We specifically want to thank Stefan Schmidt and Florian
Weimer, who both over the years have helped tremendously in
keeping PowerDNS fast, stable and secure.

This version of the PowerDNS Recursor contains a rather novel
form of lock-free multithreading, a situation that comes close
to the old '--fork' trick, but allows the Recursor to fully
utilize multiple CPUs, while delivering unified statistics and
operational control.

In effect, this delivers the best of both worlds: near linear
scaling, with almost no administrative overhead.

Compared to 'regular multithreading', whereby threads cooperate
more closely, more memory is used, since each thread maintains
its own DNS cache. However, given the economics, and the
relatively limited total amount of memory needed for high
performance, this price is well worth it.

In practical numbers, over 40,000 queries/second sustained
performance has now been measured by a third party, with a
100.0% packet response rate. This means that the needs of
around 400,000 residential connections can now be met by a
single commodity server.

In addition to the above, the PowerDNS Recursor is now
providing resolver service for many more Internet users than
ever before. This has brought with it 24/7 Service Level
Agreements, and 24/7 operational monitoring by networking
personnel at some of the largest telecommunications companies
in the world.

In order to facilitate such operation, more statistics are now
provided that allow the visual verification of proper PowerDNS
Recursor operation. As an example of this there are now graphs
that plot how many queries were dropped by the operating system
because of a CPU overload, plus statistics that can be
monitored to determine if the PowerDNS deployment is under a
spoofing attack.

All in all, this is a large and important PowerDNS Release,
paving the way for further innovation.

Note

    This release removes support for the 'fork' multi-processor
    option. In addition, the default is now to spawn two threads.
    This has been done in such a way that total memory usage will
    remain identical, so each thread will use half of the allocated
    maximum number of cache entries.
Improvements:

 * Multithreading, allowing near linear scaling to multiple
   CPUs or cores. Configured using 'threads=' (many commits).
   This also deprecates the '--fork' option.
 * Added ability to read a configuration item of a running
   PowerDNS Recursor using 'rec_control get-all' (commit
   1243), suggested by Wouter de Jong.
 * Speedups in packet generation (Commits 1258, 1259, 1262)
 * TCP deferred accept() filter is turned on again for slight
   DoS protection. Code in commit 1414.
 * PowerDNS Recursor can now do TCP/IP queries to remote IPv6
   addresses (commit 1412).
 * Solaris 9 '/dev/poll' support added, Solaris 8 now
   deprecated. Changes in commit 1421, commit 1422, commit
   1424, commit 1413.
 * Lua functions can now also see the address _to_ which a
   question was sent, using getlocaladdress(). Implemented in
   commit 1309 and commit 1315.
 * Maximum cache sizes now default to a sensible value.
   Suggested by Roel van der Made, implemented in commit 1354.
 * Domains can now be forwarded to IPv6 addresses too, using
   either ::1 syntax or [::1]:25. Thanks to Wijnand Modderman
   for discovering this issue, fixed in commit 1349.
 * Lua scripts can now load libraries at runtime, for example
   to calculate md5 hashes. Code by Winfried Angele in commit
   1405.
 * Periodic statistics output now includes average queries per
   second, as well as packet cache numbers (commit 1493).
 * New metrics are available for graphing (DOCUMENTATION
   FORTHCOMING), plus added to the default graphs (commit
   1495, commit 1498, commit 1503)
 * Fix errors/crashes on more recent versions of Solaris 10,
   where the ports functions could return ENOENT under some
   circumstances. Reported and debugged by Jan Gyselinck,
   fixed in commit 1372.

New features:

 * Add pdnslog() function for Lua scripts, so errors or other
   messages can be logged properly.
 * rec_control now accepts a --timeout parameter, which can be
   useful when reloading huge Lua scripts. Implemented in
   commit 1366.
 * 'rec_control get-all' now retrieves all statistics in one
   call (commit 1496).
 * Domains can now be forwarded with the 'recursion-desired'
   bit on or off. Feature suggested by Darren Gamble,
   implemented in commit 1451. DOCUMENTATION FORTHCOMING!
 * Access control lists can now be reloaded at runtime
   (implemented in commit 1457).
 * PowerDNS Recursor can now use a pool of
   query-local-addresses to further increase resilience
   against spoofing. Suggested by Ad Spelt, implemented in
   commit 1426. DOCUMENTATION FORTHCOMING!
 * PowerDNS Recursor now also has a packet cache, greatly
   speeding up operations. Implemented in commit 1426, commit
   1433 and further. DOCUMENTATION FORTHCOMING!
 * Cache can be limited in how long it stores records, for
   BIND compatibility. Patch by Winfried Angele in commit
   1438. DOCUMENTATION FORTHCOMING!
 * Cache cleaning turned out to be scanning more of the cache
   than necessary for cache maintenance. In addition, far more
   frequent but smaller cache cleanups improve responsiveness.
   Thanks to Winfried Angele for discovering this issue.
   (commits 1501, 1507)
 * Performance graphs enhanced with separate CPU load and
   cache effectiveness plots, plus display of various overload
   situations (commits 1503)

Compiler/Operating system/Library updates:

 * PowerDNS Recursor can now compile against newer versions of
   Boost. Reported & fixed by Darix in commit 1274. Further
   fixes in commit 1275, commit 1276, commit 1277, commit
   1283.
 * Fix compatibility with newer versions of GCC (closes ticket
   ticket 227, spotted by Ruben Kerkhof, code in commit 1345,
   more fixes in commit 1394, 1416, 1440).
 * Rrdtool update graph is now compatible with FreeBSD out of
   the box. Thanks to Bryan Seitz (commit 1517).
 * Fix up Makefile for older versions of Make (commit 1229).
 * Solaris compilation improvements (out of the box, no
   handwork needed).
 * Solaris 9 MTasker compilation fixes, as suggested by John
   Levon. Changes in commit 1431.

Bug fixes:

 * Under rare circumstances, the recursor could crash on 64
   bit Linux systems running glibc 2.7, as found in Debian
   Lenny. These circumstances became a lot less rare for the
   3.2 release. Discovered by Andreas Jakum and debugged by
   #powerdns, fix in commit 1519.
 * Configuration parser is now resistant against trailing tabs
   and other whitespace (commit 1242)
 * Fix typo in a Lua error message. Close ticket 210, as
   reported by Stefan Schmidt (commit 1319).
 * Profiled-build instructions were broken, discovered & fixes
   suggested by Stefan Schmidt. ticket 239, fix in commit
   1462.
 * Fix up duplicate SOA from a remote authoritative server
   from showing up in our output (commit 1475).
 * All security fixes from 3.1.7.2 are included.
 * Under highly exceptional circumstances on FreeBSD the
   PowerDNS Recursor could crash because of a TCP/IP error.
   Reported and fixed by Andrei Poelov in ticket 192, fixed in
   commit 1280.
 * PowerDNS Recursor can be a root-server again. Error spotted
   by the ever vigilant Darren Gamble (t229), fix in commit
   1458.
 * Rare TCP/IP errors no longer lead to PowerDNS Recursor
   logging errors or becoming confused. Debugged by Josh Berry
   of Plusnet PLC. Code in commit 1457.
 * Do not hammer parent servers in case child zones are
   misconfigured, requery at most once every 10 seconds.
   Reported & investigated by Stefan Schmidt and Andreas
   Jakum, fixed in commit 1265.
 * Properly process answers from remote authoritative servers
   that send error answers without including the original
   question (commit 1329, commit 1327).
 * No longer spontaneously turn on 'export-etc-hosts' after
   reloading zones. Discovered by Paul Cairney, reported in
   ticket 225, addressed in commit 1348.
 * Very abrupt server failure of large numbers of high-volume
   authoritative servers could trigger an out of memory
   situation. Addressed in commit 1505.
 * Make timeouts for queries to remote authoritative servers
   configurable with millisecond granularity. In addition, the
   old code turned out to consider the timeout expired when
   the integral number of seconds since 1970 increased by 1 -
   which *on average* is after 500ms. This might have caused
   spurious timeouts! New default timeout is 1500ms. Code in
   commit 1402. DOCUMENTATION FORTHCOMING!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAktzEy0ACgkQHF7pkNLnFXX6NQCfWLjmCtB17I7/9a278LUvI9Ba
YAoAoMeOq8nVZ+Q2/0NKCkryjV8LxTlk
=v7eH
-----END PGP SIGNATURE-----

Critical PowerDNS Recursor Security Vulnerabilities: please upgrade ASAP to 3.1.7.2

2010-01-06T07:26:00.000-08:00

Dear PowerDNS Users,

Two major vulnerabilities have recently been discovered in the PowerDNS
Recursor (all versions up to and including 3.1.7.1). Over the past two
weeks, these vulnerabilities have been addressed, resulting in PowerDNS
Recursor 3.1.7.2.

Given the nature and magnitude of these vulnerabilities, ALL PowerDNS
RECURSOR USERS ARE URGED TO UPGRADE AT THEIR EARLIEST CONVENIENCE. No
versions of the PowerDNS Authoritative Server are affected.

PowerDNS Recursor 3.1.7.2 as been thoroughly tested, and has in fact been in
production for a week at some major sites already. No problems have been
reported. 3.1.7.2 does not include anything other than security updates.

The two major vulnerabilities can lead to a FULL SYSTEM COMPROMISE, as well
as cache poisoning, connecting your users to possibly malicious IP addresses.

These vulnerabilities were discovered by a third party that for now prefers
not to be named. PowerDNS is however very grateful for their help. More
details are available on:
http://doc.powerdns.com/powerdns-advisory-2010-01.html
http://doc.powerdns.com/powerdns-advisory-2010-02.html

Debian, FreeBSD, Gentoo and SuSE are processing the changed packages, and
will be releasing security updates shortly. Ubuntu does not provide security
updates for PowerDNS, so Ubuntu users must take immediate action and
download our packages.

RHEL4/5, CentOS packages are available (care of Kees Monshouwer) here:
http://www.monshouwer.eu/download/3th_party/pdns-recursor/

Updated packages for .deb based systems are available here:
http://downloads.powerdns.com/releases/deb/pdns-recursor_3.1.7.2-1_i386.deb
http://downloads.powerdns.com/releases/deb/pdns-recursor_3.1.7.2-1_amd64.deb

Updated packages for .rpm based systems are available here:
http://downloads.powerdns.com/releases/rpm/pdns-recursor-3.1.7.2-1.i386.rpm
http://downloads.powerdns.com/releases/rpm/pdns-recursor-3.1.7.2-1.x86_64.rpm

Source code is available here:
http://downloads.powerdns.com/releases/pdns-recursor-3.1.7.2.tar.bz2

Special 'upgrade option of last resort' (old systems)
-----------------------------------------------------
In addition, as a special service, we are also providing two precompiled
fully static Linux binaries as an 'upgrade option of last resort':

http://downloads.powerdns.com/releases/pdns_recursor-3.1.7.2.amd64.static.executable
http://downloads.powerdns.com/releases/pdns_recursor-3.1.7.2.i386.static.executable

These two binaries are suitable of our .deb or .rpm files somehow refuse to
load (which happens on RHEL version 3, for example).

Download the appropriate executable, rename to pdns_recursor, set the
executable bit (chmod a+x pdns_recursor), and 'mv' the executable over
/usr/sbin/pdns_recursor.

If you need any help in upgrading, please do not hesitate to contact us.

Kind regards,

Bert Hubert

Bert

Dit jaar wel studie- en afstudeerbegeleiding

2009-12-25T07:30:00.000-08:00

A rare post in Dutch - if you know anyone in The Netherlands that really needs to graduate or otherwise finish their studies, this message is for them ;-)

Met trots presenteer ik de website van het bedrijf van mijn vrouw Mirjam: ditjaarwel.nl!

Na het nodige proefdraaien met de eerste klanten is het bedrijf nu 'ready for business'. Dus als je iemand kent die dringend af moet studeren, of anderszins over een hobbel in de studie geholpen moet worden, wijs ze dan vooral op Dit Jaar Wel.nl! Ook erg geschikt voor ouders die vinden dat het nu wel mooi geweest is met de studie van zoon of dochterlief.

PS: Leuk om te melden dat ik als 'officiele computernerd in huis' er niet in geslaagd was iets fatsoenlijks en betaalbaars te vinden om een website mee te maken. Mirjam kwam na een rondje googlen met Yola.com, wat precies is wat we zochten. Aanrader.

Addicted to the LHC

2009-12-10T15:24:00.000-08:00

Over the past few weeks, the Large Hadron Collider (LHC), also known as the 'most puissant particle punisher that pounds the protons', has been starting up (again).

The LHC is housed at CERN, and consists of a 27km long tunnel that is so big it needs two countries to contain it, France and Switzerland.

It's goal in life is to smash protons and later lead ions together at stupendous energies.. to see what happens. I bet they have written a loftier description of their aims themselves, but this is what they do.

At a cost that is hard to calculate, but surely more than €10 billion, it is cheaper than propping up a small bank.

The scale of everything they do there is huge. The tunnel is lined with arrays of machines, each of which would be the the proud possession of any physics department in the world. And it all has to work to deliver the big goal: smashing particles together at hitherto unobtainable energies.

Both because of "typos" in newspapers and because of the excitement this machine generates in the nerd crowd, the Large Hadron Collider has also been called the Large Hardon Collider. And it deserves the name.

I'm going to stop gushing now, but it is a seriously impressive setup. I haven't even started about the computing grid that calculates what the collisions resulted in, and if new physics has been discovered.

So, getting back to the title of this post, why am I addicted? In all their 'web 2.0' wisdom (they invented 'web 1.0' over at CERN, btw), the LHC people have decided to be incredibly open. With a little effort, you can find up to the second statistics of all their activities, down to a minute by minute logbook of operations.

And for some reason or other, this makes the LHC like a reality series for me. There is even a form and a chat room to hang out and gossip!

So, to join the fun, head over to the (unofficial) LHC Portal which contains links to all the good stuff! The best page to start is "Page 1" (which can also be found on the LHC Portal).

Have fun!

When DNS is cool and when it is not

2009-11-11T11:28:00.000-08:00

Whenever massive query rates are desired for globally distributed data, with high redundancy and built in positive and negative caching, people think of DNS. Popular examples are of course our day to day use of the Domain Name System (which is a lot more than a protocol) to lookup IP addresses, but also include tremendous amounts of spam lookups (RBLs) to determine if an IP address is likely to be a source of spam.

In addition, "ENUM" has been designed to share reachability information for phone numbers over DNS, telling one for example over which SIP identity the owner of a phone number could be reached using VoIP. This needs many of the aforementioned features of DNS, like high query rates, redundancy and caching.

Periodically, people ponder storing other things in DNS, most often because they are attracted to the huge query rates, built in distribution, redundancy and caching. And indeed, these are things that make the DNS very attractive.

In addition, DNS passes more firewalls by default than almost any other protocol, because the network's resolver acts as a sanctioned proxy.

It turns out however that there are severe limits to what you can do within DNS while retaining the attractive bits.

High query rates
Even with a very limited investment, it is possible to build solutions based on DNS enabling one to ask and answer over a million queries per second. Building such functionality on top of a SQL database would be an order of magnitude more expensive (at least).

Among the reasons why DNS can support such tremendous speeds is its use of the connectionless UDP protocol, which means that a question fits in a single packet, as does the answer. A TCP/IP session goes through at least 6 packets to achieve the same thing.

Passing firewalls
Almost all network environments have DNS connectivity to the outside world, often via the network's resolvers. In addition, these resolvers typically have an undisturbed view of UDP port 53 to the outside world. However, they often do not have such unfettered access to TCP or ICMP.

This is important because UDP packets have severe constraints on their size, with 1500 being the maximum before stuff needs to happen. The stuff that needs to happen either entails sending fragments (which have a hard time passing firewalls), or moving to TCP (which is blocked far more often than UDP for DNS).

DNS has a lot of rules
DNS was originally a replacement for the (then) famous HOSTS.TXT file, which contained IP addresses for host names that people wanted to share with the internet. This file was lovingly maintained by hand, and periodically downloaded by everybody.

When this no longer proved to be sustainable, the DNS was created so everybody could administer their own names, and publish these in an automated fashion.

Look closely however, and the DNS shows its HOSTS.TXT roots. Even though each 'top level domain' can have its own set of servers, in the end fundamentally, the DNS assumes it is actually one uniform list of records ('HOSTS.TXT'). This means that if the root says the nameserver for everything ending on NL are X, Y and Z, that if you ask any of X, Y and Z what the namservers for NL are it *has* to answer X, Y and Z (it may add U, V and W to the answers though).

What it may NOT do is say 'oh, NL, I handed that over to servers A, B and C, ask them'. Because this would violate the 'HOSTS.TXT' view of the DNS, where everything in the root zone has to be identical to the stuff at the lower level.

DNS can only answer simple questions
DNS basically knows only one question 'Do you have information of type X about name Y?'. And as an answer, you'll get all the information about Y of type X that fits in the answer packet. There is no way to say 'give me all names Y that have type X', for example. Nor is there a way to ask for all names that start with 'www'.

You can't mirror the DNS
The DNS is a fully distributed system, and one that can only answer simple questions (see above). There is no reliable way to make a complete copy of the DNS. This means that in order to use it, one has to rely on working network connectivity, and also has to trust other people's systems.

Unlike, say, a SQL database, it is not possible to have a full copy that still works without network connectivity.

So - what do these limitations mean?
Summarising - we like DNS because it is really fast, easily distributed, well cached and passes firewalls easily. However, the above means that if we want to keep all these cool features:

Responses to DNS queries must be small. Large answers mean UDP can't be used, which in turn means a significant slowdown because TCP needs so many more packets. In addition, TCP has a far harder time passing firewalls.
Fundamentally, this means not storing photographs or other large things in DNS
We must only ask simple questions that have direct answers.
Our questions and data distribution must fit the DNS rules.
This means we can't "redelegate". A practical problem that gets hit by this restriction is so called telephony number portability, where a phone number jumps outside of the hierarchy, and is suddenly served by a wholly different company.
We must accept that queries will leave our network, and that we can't have an 'offline copy'

All in all - this means that quite a lot of problems do not fit the constraints that DNS imposes.

But anytime you have simple questions, with small answers and you dare to rely on other people's servers, plus do not desire 'redelegation', DNS may be your best bet.

Some alternatives
Slightly more advanced than DNS is LDAP, which offers the possiblity of asking more complicated questions. Slightly *less* advanced than DNS is memcached, which does however share the very high performance and easy redundancy. It does not offer delegation though.

xs.powerdns.com: PowerDNS Development & Community Server @ xs4all!

2009-11-08T02:32:00.000-08:00

Hi everybody,

Over the past few months, the PowerDNS Wiki and Subversion servers had a hard time and were no longer able to keep up with the growing amounts of traffic. Since these servers also routed my personal email, I had little choice but move the flood of spam to gmail. But no more!

We acquired a lovely Dell PowerEdge R200, and found pioneering PowerDNS user XS4ALL willing to host it! This explains the xs4all logos on wiki.powerdns.com and svn.powerdns.com ;-)

XS4ALL is what I'd like to call an 'old school internet service provider' - which is quite literally true since they were (almost) the first here in The Netherlands. Racking up xs.powerdns.com went without a hitch. For some reason, whenever I have to rack up a server, it turns out the wrong rails have been agreed upon, or the power is wrong or too much, or the IP addresses have not been arranged.

But this time round, we were done in 5 minutes. When I discovered a few days later that no IPv6 was provisioned, this was fixed within an hour. On Sunday.

XS4ALL also was the 'launching customer' of the PowerDNS Recursor, funding the development process and field testing it.

So many thanks to them for hosting xs.powerdns.com!

PowerDNS competitor Nominum lauds its closed source credits!

2009-09-23T03:20:00.000-07:00

This morning, I was unpleasantly surprised by an advertorial on ZDNET, where PowerDNS competitor Nominum stated that since they are closed source, their technology is inherently more secure. They also cleverly compared Open Source to malware. Nice.

In addition, Nominum stated they have not had any security problems, "unlike the freeware legacy DNS", but this simply is not true as can be seen on their own webpage (which will probably be 'cleaned up' shortly).

There are some true gems in the interview, cleverly titled "Why open-source DNS is 'internet's dirty little secret'".

Freeware legacy DNS is the internet's dirty little secret — and it's not even little, it's probably a big secret. Because if you think of all the places outside of where Nominum is today — whether it's the majority of enterprise accounts or some of the smaller ISPs — they all have essentially been running freeware up until now.
Given all the nasty things that have happened this year, freeware is a recipe for problems, and it's just going to get worse.

Followed by:

Correct. So, whether it's Eircom in Ireland or a Brazilian ISP that was attacked earlier this year, all of them were using some variant of freeware. Freeware is not akin to malware, but is opening up those customers to problems. So we've seen the majority of the world's top ISPs migrating away from freeware to a solution that is carrier-grade, commercial-grade and secure.

And the real screamer:

Nominum software was written 100 percent from the ground up, and by having software with source code that is not open for everybody to look at, it is inherently more secure.

Way, way back when, Nominum employees successfully performed a denial of service attack on PowerDNS. I thought they had grown over this kind of behavior, but it appears they didn't.

Nominum used to be a part of the DNS community, interacting with the IETF in the standards setting process. It may be harder for them to credibly contribute anymore if this is their stance on open cooperation..

UPDATE: It is ironic to note that at the time of writing, one of the Nominum.com nameservers was actually running BIND ('freeware, not akin to malware'). In addition, both the webserver and the operating system for the Nominum webpages run on open source software (Apache, Linux).

Guus Hubert has been born!

2009-09-23T02:37:00.000-07:00

Mirjam and I are really happy to report that our lovely son Guus Hubert was born on the 14th of September! Mother and son are doing really well.

Here is the official picture:

You can see that Maurits eyes his new brother with a keen interest!
Guus & family can be congratulated on guus@hubertnet.nl !

PowerDNS & PowerAdmin contributor Jorn Ekkelenkamp has passed away

2009-09-07T03:09:00.000-07:00

I'm almost too hurt to type..

Copy pasted from the PowerDNS mailing lists:

Dear PowerDNS friends,

I'm deeply saddened to have to report that Jorn Ekkelenkamp passed away yesterday, at the tender age of 26. He died battling Leukemia. Jorn was also known as Sjeemz on the #powerdns irc channel.

At ISP Services (later known as Hubris) Jorn was perhaps the very first large scale user of PowerDNS, in addition to authoring the first PowerDNS web management solution, PowerAdmin.

His belief in PowerDNS, and his subsequent deployment, paved the path for much of what we have achieved over the years. PowerAdmin also helped people use and migrate to PowerDNS. In addition, he frequently suggested, tested or even funded new PowerDNS features.

Jorn will be missed very much. He was truly a PowerDNS man from the very first hour.

His girlfriend and family were aware of Jorn's contributions to the open source world and the Internet. If PowerAdmin has made your life or work better, or if you've benefited from his other contributions to the PowerDNS community, please drop me a message, and I'll relay to his family, who will appreciate hearing about what Jorn meant for other people.

In addition, there will be church service Saturday morning - I can relay the details if you want to attend.

I wish his family and everyone who knew Jorn lots of strength in dealing with this tremendous loss.

Bert Hubert
PowerDNS.COM

HAR2009 thoughts, returning back to earth..

2009-08-16T13:44:00.001-07:00

I just got back from HAR2009, and am slowly returning to earth. HAR is the fifth installment (or sixth, depending on who you speak) of the 'HXX' series of hacker events here in The Netherlands. These usually attract in the order of 2000 of the best geeks of Europe (& beyond), and HAR2009 was no exception.

Over 2200 people trekked to Vierhouten, to attend 106 presentations, many many parties, and have lots of fun. Most of them slept on-site in their tents or caravans.

The event lasts four days, but many people arrived early, and some of them are staying late to help with cleanup.

But for me it is over now, and like I said, I'm slowly returning back to earth. It was tremendous. Awesome to the point that I nearly got emotional when it really was time to go home.

I have tried and failed to express in words why events like these are so wonderful (and why HAR2009 was the best yet), but it is not working. It is one of these 'You had to be there' things. Suffice it to say that if you are a geek at heart (and I am definitely one), this was the place to be.

I did a presentation on "DNS Security in the Broadest Sense", here is a photo just before my presentation, while Niels helped me out because my HP #$@#$ 'netbook' failed to switch to the proper resolution, even though I had tested this on the projector a few hours earlier:

(image courtesy of security.nl)

My presentation: pdf

Badly transcoded movie of the presentation, will be replaced at a later point with a better one:

My lovely son Maurits watching the live stream (this photo is NOT shopped!)

(many thanks to my wife Mirjam, who is very pregnant, but insisted that I went to HAR, because she knew it meant so much to me!)

A video Interview in Dutch about my presentation can be found on Security.nl

Update! Interview (in English) on HARFM can be found here (plays in Firefox 3.5, otherwise try VLC)

All in all, it was a TREMENDOUS event, and I am told people liked my presentation. I'm also proud that PowerDNS powered the whole HAR2009 DNS infrastructure, and that it held up and was not compromised. A good thing at a hacker conference.

I'm also proud that both PowerDNS and Fox-IT could play a part in this. Aldert, I don't believe you when you said HAR2009 would not have happened without us, but the thought is much appreciated! It must have been the good food at our BBQ :-)

There are moments from HAR2009 that I will never, ever, forget again. Thanks to everyone that made it happen!

PowerDNS is The Hackers Choice!

2009-08-11T01:51:00.000-07:00

Very brief note to let you know that I'm thrilled that PowerDNS will be serving the HAR 2009 visitors! It has previously also been used by the same people at CCC congresses.

I'm very proud of this, and I hope the Recursor and Authoritative Server will continue to do well for such demanding users. And if there is any issue, the maintainers can rest secure in the knowledge that 24/7 on-site support is available.. from my tent.

PowerDNS Recursor 3.1.7.1 released!

2009-08-02T14:00:00.000-07:00

I'm pretty proud of this release, or to state it better, proud of not having a real reason to do a release for over a year.

Over the past year, the PowerDNS Recursor has gone places I never thought it would go, powering the majority of internet subscribers in some large countries.

It is very rare for a project that services so many people, to exhibit so very little problems. I'm probably just lucky in this respect, but it still feels good.

However, since the world moved on over the past year, version 3.1.7 became somewhat hard to compile on modern Linux and UNIX distributions. In addition, Solaris 10 changed its ABI slightly, causing Recursor to crash quickly under heavy load.

So after a week of testing, 3.1.7.1 has been released today, with no new features, only bug fixes.

For more details, please see the announcement.

Moving closer to pizza perfection!

2009-08-02T12:01:00.000-07:00

I truly love a good pizza, but it is a rare event to find one. So, I've long been baking my own pies, at first in a normal oven and later in a special pizza oven. Although the "G3 Ferrari" looks impressive, mine is bright red, it is a decidedly weird machine.

For example, it has four heat levels, 1, 2, 2.5 and 3. One has to wonder how that happened.

I've read with great gusto the wonderful works by Jeffrey Steingarten which cover with great precision how very good food is made, including pizza. Jeffrey's book emphasized the importance of high heat, and even mentions the very same "G3 Ferrari" oven I have.

So, I've been using it to make pizzas of very varying quality - sometimes tremendously good, sometimes less so, and I never knew why. Like many aspiring pizzaiolos, I blamed my flour, and I assumed the professionals were using special brands.

Then I discovered the page of Jeff Varasano, who is a bigger pizza nut than I would've thought possible. It looks like he spent 10+ years figuring out how to do it, and from him I discovered the stunning secret: all pizza recipes I've ever seen in books, or online, are wrong. So, I set out to follow his instructions to the letter, which indeed led to very good and elastic dough.. and still no good pizzas ensued from my oven!

(in brief, any recipe which starts out by lumping all ingredients of the dough together and instruct to after mixing 'let it rest until it has doubled in size' is pretty far removed from reality. For more information, see Jeff's page)

Several months passed, and this weekend I found myself with some time off (since our son Maurits was spending time with his grandparents), and I decided to try again, this time using science.

Recall the Italian pizza oven with heat level '2.5'? It turns out the thermostat of this oven is a lying through its teeth! With the aid of a high-heat thermometer, I discovered the awful truth that the temperature of the oven has very little to do with the settings of the thermostat.

It turns out that the "G3 Ferrari" only reaches the required temperatures (400+C, around 750F) when the grill is red hot and has been on non-stop for quite some time, no matter what the thermostat may say.

So today, using my trusty thermometer, I timed it such that the oven reached this stunning temperature just when my pizza was ready.

And lo, it was wonderful!

To the non-pizza-enthusiast, this may not sound like a big thing - but this is an important step in my ongoing quest: be able to entertain large amounts of guests with mouth watering pizzas.

To be continued...

Some quick notes on RSA1024 signing performance

2009-07-27T14:17:00.000-07:00

Just so this does not get lost - I've been doing some RSA1024 signing experiments because of my 'DNSSEC on PowerDNS' experiment, and the results were at first confusing.

For starters, friends of mine with Apple OS X reported very low numbers from the version of OpenSSL that ships with OS X (intel). The command to have OpenSSL perform speed tests is: 'openssl speed rsa1024'. Numbers were around half those reported on identical machines running a 32-bit Ubuntu.

Much investigations ensued, and conclusions are:

Apple ships a version of OpenSSL that misses certain optimizations. If you need performance for your applications, investigate which OpenSSL library they link against, and possibly investigate how to recompile or relink.
Go 64-bit, in a hurry. Twice as many bits appear to deliver over twice as much performance.
A modern Core2 based CPU running 64 bits code maxes out at about 1500 RSA1024 signatures/second/core, based on OpenSSL 1.0 beta 3, or Botan linked against GnuMP 4
Non-beta OpenSSLs are quite a bit slower, but not dramatically so
More naive code, that is not as highly optimized (like the otherwise excellent PolarSSL), will deliver around 1200 RSA1024 signatures/second/core (64 bits)
These numbers scale linearly with the number of cores involved - my 600 euro PC delivers 6000 signatures/second ('0.10 euro/signature/second').

It also looks like no worthwhile general purpose RSA hardware accelerators are available for use from Linux - Sun ships one, but its performance is not stellar (a lot more than 0.10 euro/signature/second), but it is not cheap, plus it is only officially supported on Sun hardware. If anyone has better ideas, please let me know!

PS: Why RSA1024? Because this is what DNSSEC is about for the foreseeable future..

So, why did I move my blog?

2009-07-14T11:53:00.000-07:00

Aesthetically, I liked my old blog. The design was clean, it was 100% under my control, but that last part also turned out to be a problem. When spammers discovered blog.netherlabs.nl, they filled it with junk. Junk which was sometimes filtered out, sometimes not, but in any case clogged my poor server. We are talking gigabytes of spam here, literally.

So eventually I caved. If Linus Torvalds can host his blog on blogger.com, it must surely be good enough for me.

The old blog postings are still available here: http://blog.netherlabs.nl/index.html
Be sure to add the '/index.html', because shortly, blog.netherlabs.nl without it will forward you to this site.

So, welcome back dear readers, and I hope to entertain you with things I can't bring myself to shut up about.

This is the new location of Bert Hubert's blog!

2009-07-14T11:45:00.001-07:00

This is the new location of Bert Hubert's blog!

bert hubert finally blogs

Four million pings only - aka 1 dimensional DNS radar

PowerDNS Authoritative Server Security Notification 2012-01

Old blog posts back

The ultimate SO_LINGER page, or: why is my tcp not reliablePredictionsReusing UNIX semantics for fun and profitSecrets in Public: Diffie-Hellman key exchangeCalculating the chance of spoofing an agile source port randomised resolverThe fourteen stages of any real software project

Reusing UNIX semantics for fun and profit

Secrets in Public: Diffie-Hellman key exchange

Calculating the chance of spoofing an agile source port randomised resolver

Neutrinos faster than light?

A prior art post: wifi OR bluetooth for powersaving

PowerDNS Authoritative Server 3.0 has been released!

As a community service, the glibc 2.14 'fixed bugs' with descriptions

PowerDNS Authoritative Server 3.0RC1 released! Now with DNSSEC!

PowerDNSSEC: packages available, ready for light production use

Linux or UNIX programmer? Go get The Linux Programming Interface: A Linux and UNIX System Programming Handbook

PowerDNS Recursor additional Lua hooks for IPv6 DNS64 and Renumbering

The "leaky abstraction" of the POSIX file interface

You can now own the house from which PowerDNS Recursor 3.3 was released!

PowerDNS Recursor 3.3 released!

Some notes on Solaris 10 x86, 64 bit compilation, bugs and memory allocators

Better statistical regression tests: Release your inner German!

PowerDNSSEC Available For Testing!

A few notes on procurement

PowerDNS Recursor 3.2 Release Candidate 1

Critical PowerDNS Recursor Security Vulnerabilities: please upgrade ASAP to 3.1.7.2

Dit jaar wel studie- en afstudeerbegeleiding

Addicted to the LHC

When DNS is cool and when it is not

xs.powerdns.com: PowerDNS Development & Community Server @ xs4all!

PowerDNS competitor Nominum lauds its closed source credits!

Guus Hubert has been born!

PowerDNS & PowerAdmin contributor Jorn Ekkelenkamp has passed away

HAR2009 thoughts, returning back to earth..

PowerDNS is The Hackers Choice!

PowerDNS Recursor 3.1.7.1 released!

Moving closer to pizza perfection!

Some quick notes on RSA1024 signing performance

So, why did I move my blog?

This is the new location of Bert Hubert's blog!

The ultimate SO_LINGER page, or: why is my tcp not reliable
Predictions
Reusing UNIX semantics for fun and profit
Secrets in Public: Diffie-Hellman key exchange
Calculating the chance of spoofing an agile source port randomised resolver
The fourteen stages of any real software project