Thursday, December 10, 2009

Addicted to the LHC

Over the past few weeks, the Large Hadron Collider (LHC), also known as the 'most puissant particle punisher that pounds the protons', has been starting up (again).

The LHC is housed at CERN, and consists of a 27km long tunnel that is so big it needs two countries to contain it, France and Switzerland.

It's goal in life is to smash protons and later lead ions together at stupendous energies.. to see what happens. I bet they have written a loftier description of their aims themselves, but this is what they do.

At a cost that is hard to calculate, but surely more than €10 billion, it is cheaper than propping up a small bank.

The scale of everything they do there is huge. The tunnel is lined with arrays of machines, each of which would be the the proud possession of any physics department in the world. And it all has to work to deliver the big goal: smashing particles together at hitherto unobtainable energies.

Both because of "typos" in newspapers and because of the excitement this machine generates in the nerd crowd, the Large Hadron Collider has also been called the Large Hardon Collider. And it deserves the name.

I'm going to stop gushing now, but it is a seriously impressive setup. I haven't even started about the computing grid that calculates what the collisions resulted in, and if new physics has been discovered.

So, getting back to the title of this post, why am I addicted? In all their 'web 2.0' wisdom (they invented 'web 1.0' over at CERN, btw), the LHC people have decided to be incredibly open. With a little effort, you can find up to the second statistics of all their activities, down to a minute by minute logbook of operations.

And for some reason or other, this makes the LHC like a reality series for me. There is even a form and a chat room to hang out and gossip!
So, to join the fun, head over to the (unofficial) LHC Portal which contains links to all the good stuff! The best page to start is "Page 1" (which can also be found on the LHC Portal).

Have fun!

Wednesday, November 11, 2009

When DNS is cool and when it is not

Whenever massive query rates are desired for globally distributed data, with high redundancy and built in positive and negative caching, people think of DNS. Popular examples are of course our day to day use of the Domain Name System (which is a lot more than a protocol) to lookup IP addresses, but also include tremendous amounts of spam lookups (RBLs) to determine if an IP address is likely to be a source of spam.

In addition, "ENUM" has been designed to share reachability information for phone numbers over DNS, telling one for example over which SIP identity the owner of a phone number could be reached using VoIP. This needs many of the aforementioned features of DNS, like high query rates, redundancy and caching.

Periodically, people ponder storing other things in DNS, most often because they are attracted to the huge query rates, built in distribution, redundancy and caching. And indeed, these are things that make the DNS very attractive.

In addition, DNS passes more firewalls by default than almost any other protocol, because the network's resolver acts as a sanctioned proxy.

It turns out however that there are severe limits to what you can do within DNS while retaining the attractive bits.

High query rates
Even with a very limited investment, it is possible to build solutions based on DNS enabling one to ask and answer over a million queries per second. Building such functionality on top of a SQL database would be an order of magnitude more expensive (at least).

Among the reasons why DNS can support such tremendous speeds is its use of the connectionless UDP protocol, which means that a question fits in a single packet, as does the answer. A TCP/IP session goes through at least 6 packets to achieve the same thing.

Passing firewalls
Almost all network environments have DNS connectivity to the outside world, often via the network's resolvers. In addition, these resolvers typically have an undisturbed view of UDP port 53 to the outside world. However, they often do not have such unfettered access to TCP or ICMP.

This is important because UDP packets have severe constraints on their size, with 1500 being the maximum before stuff needs to happen. The stuff that needs to happen either entails sending fragments (which have a hard time passing firewalls), or moving to TCP (which is blocked far more often than UDP for DNS).

DNS has a lot of rules
DNS was originally a replacement for the (then) famous HOSTS.TXT file, which contained IP addresses for host names that people wanted to share with the internet. This file was lovingly maintained by hand, and periodically downloaded by everybody.

When this no longer proved to be sustainable, the DNS was created so everybody could administer their own names, and publish these in an automated fashion.

Look closely however, and the DNS shows its HOSTS.TXT roots. Even though each 'top level domain' can have its own set of servers, in the end fundamentally, the DNS assumes it is actually one uniform list of records ('HOSTS.TXT'). This means that if the root says the nameserver for everything ending on NL are X, Y and Z, that if you ask any of X, Y and Z what the namservers for NL are it *has* to answer X, Y and Z (it may add U, V and W to the answers though).

What it may NOT do is say 'oh, NL, I handed that over to servers A, B and C, ask them'. Because this would violate the 'HOSTS.TXT' view of the DNS, where everything in the root zone has to be identical to the stuff at the lower level.

DNS can only answer simple questions
DNS basically knows only one question 'Do you have information of type X about name Y?'. And as an answer, you'll get all the information about Y of type X that fits in the answer packet. There is no way to say 'give me all names Y that have type X', for example. Nor is there a way to ask for all names that start with 'www'.

You can't mirror the DNS
The DNS is a fully distributed system, and one that can only answer simple questions (see above). There is no reliable way to make a complete copy of the DNS. This means that in order to use it, one has to rely on working network connectivity, and also has to trust other people's systems.

Unlike, say, a SQL database, it is not possible to have a full copy that still works without network connectivity.

So - what do these limitations mean?
Summarising - we like DNS because it is really fast, easily distributed, well cached and passes firewalls easily. However, the above means that if we want to keep all these cool features:
  • Responses to DNS queries must be small. Large answers mean UDP can't be used, which in turn means a significant slowdown because TCP needs so many more packets. In addition, TCP has a far harder time passing firewalls.
    Fundamentally, this means not storing photographs or other large things in DNS
  • We must only ask simple questions that have direct answers.
  • Our questions and data distribution must fit the DNS rules.
    This means we can't "redelegate". A practical problem that gets hit by this restriction is so called telephony number portability, where a phone number jumps outside of the hierarchy, and is suddenly served by a wholly different company.
  • We must accept that queries will leave our network, and that we can't have an 'offline copy'
All in all - this means that quite a lot of problems do not fit the constraints that DNS imposes.

But anytime you have simple questions, with small answers and you dare to rely on other people's servers, plus do not desire 'redelegation', DNS may be your best bet.

Some alternatives
Slightly more advanced than DNS is LDAP, which offers the possiblity of asking more complicated questions. Slightly *less* advanced than DNS is memcached, which does however share the very high performance and easy redundancy. It does not offer delegation though.

Sunday, November 8, 2009 PowerDNS Development & Community Server @ xs4all!

Hi everybody,

Over the past few months, the PowerDNS Wiki and Subversion servers had a hard time and were no longer able to keep up with the growing amounts of traffic. Since these servers also routed my personal email, I had little choice but move the flood of spam to gmail. But no more!

We acquired a lovely Dell PowerEdge R200, and found pioneering PowerDNS user XS4ALL willing to host it! This explains the xs4all logos on and ;-)

XS4ALL is what I'd like to call an 'old school internet service provider' - which is quite literally true since they were (almost) the first here in The Netherlands. Racking up went without a hitch. For some reason, whenever I have to rack up a server, it turns out the wrong rails have been agreed upon, or the power is wrong or too much, or the IP addresses have not been arranged.

But this time round, we were done in 5 minutes. When I discovered a few days later that no IPv6 was provisioned, this was fixed within an hour. On Sunday.

XS4ALL also was the 'launching customer' of the PowerDNS Recursor, funding the development process and field testing it.

So many thanks to them for hosting!

Wednesday, September 23, 2009

PowerDNS competitor Nominum lauds its closed source credits!

This morning, I was unpleasantly surprised by an advertorial on ZDNET, where PowerDNS competitor Nominum stated that since they are closed source, their technology is inherently more secure. They also cleverly compared Open Source to malware. Nice.

In addition, Nominum stated they have not had any security problems, "unlike the freeware legacy DNS", but this simply is not true as can be seen on their own webpage (which will probably be 'cleaned up' shortly).

There are some true gems in the interview, cleverly titled "Why open-source DNS is 'internet's dirty little secret'".
Freeware legacy DNS is the internet's dirty little secret — and it's not even little, it's probably a big secret. Because if you think of all the places outside of where Nominum is today — whether it's the majority of enterprise accounts or some of the smaller ISPs — they all have essentially been running freeware up until now.

Given all the nasty things that have happened this year, freeware is a recipe for problems, and it's just going to get worse.

Followed by:
Correct. So, whether it's Eircom in Ireland or a Brazilian ISP that was attacked earlier this year, all of them were using some variant of freeware. Freeware is not akin to malware, but is opening up those customers to problems. So we've seen the majority of the world's top ISPs migrating away from freeware to a solution that is carrier-grade, commercial-grade and secure.
And the real screamer:
Nominum software was written 100 percent from the ground up, and by having software with source code that is not open for everybody to look at, it is inherently more secure.
Way, way back when, Nominum employees successfully performed a denial of service attack on PowerDNS. I thought they had grown over this kind of behavior, but it appears they didn't.

Nominum used to be a part of the DNS community, interacting with the IETF in the standards setting process. It may be harder for them to credibly contribute anymore if this is their stance on open cooperation..

UPDATE: It is ironic to note that at the time of writing, one of the nameservers was actually running BIND ('freeware, not akin to malware'). In addition, both the webserver and the operating system for the Nominum webpages run on open source software (Apache, Linux).

Monday, September 7, 2009

PowerDNS & PowerAdmin contributor Jorn Ekkelenkamp has passed away

I'm almost too hurt to type..

Copy pasted from the PowerDNS mailing lists:

Dear PowerDNS friends,

I'm deeply saddened to have to report that Jorn Ekkelenkamp passed away yesterday, at the tender age of 26. He died battling Leukemia. Jorn was also known as Sjeemz on the #powerdns irc channel.

At ISP Services (later known as Hubris) Jorn was perhaps the very first large scale user of PowerDNS, in addition to authoring the first PowerDNS web management solution, PowerAdmin.

His belief in PowerDNS, and his subsequent deployment, paved the path for much of what we have achieved over the years. PowerAdmin also helped people use and migrate to PowerDNS. In addition, he frequently suggested, tested or even funded new PowerDNS features.

Jorn will be missed very much. He was truly a PowerDNS man from the very first hour.

His girlfriend and family were aware of Jorn's contributions to the open source world and the Internet. If PowerAdmin has made your life or work better, or if you've benefited from his other contributions to the PowerDNS community, please drop me a message, and I'll relay to his family, who will appreciate hearing about what Jorn meant for other people.

In addition, there will be church service Saturday morning - I can relay the details if you want to attend.

I wish his family and everyone who knew Jorn lots of strength in dealing with this tremendous loss.

Bert Hubert

Sunday, August 16, 2009

HAR2009 thoughts, returning back to earth..

I just got back from HAR2009, and am slowly returning to earth. HAR is the fifth installment (or sixth, depending on who you speak) of the 'HXX' series of hacker events here in The Netherlands. These usually attract in the order of 2000 of the best geeks of Europe (& beyond), and HAR2009 was no exception.

Over 2200 people trekked to Vierhouten, to attend 106 presentations, many many parties, and have lots of fun. Most of them slept on-site in their tents or caravans.

The event lasts four days, but many people arrived early, and some of them are staying late to help with cleanup.

But for me it is over now, and like I said, I'm slowly returning back to earth. It was tremendous. Awesome to the point that I nearly got emotional when it really was time to go home.

I have tried and failed to express in words why events like these are so wonderful (and why HAR2009 was the best yet), but it is not working. It is one of these 'You had to be there' things. Suffice it to say that if you are a geek at heart (and I am definitely one), this was the place to be.

I did a presentation on "DNS Security in the Broadest Sense", here is a photo just before my presentation, while Niels helped me out because my HP #$@#$ 'netbook' failed to switch to the proper resolution, even though I had tested this on the projector a few hours earlier:

(image courtesy of

My presentation: pdf

Badly transcoded movie of the presentation, will be replaced at a later point with a better one:

My lovely son Maurits watching the live stream (this photo is NOT shopped!)

(many thanks to my wife Mirjam, who is very pregnant, but insisted that I went to HAR, because she knew it meant so much to me!)

A video Interview in Dutch about my presentation can be found on

Update! Interview (in English) on HARFM can be found here (plays in Firefox 3.5, otherwise try VLC)

All in all, it was a TREMENDOUS event, and I am told people liked my presentation. I'm also proud that PowerDNS powered the whole HAR2009 DNS infrastructure, and that it held up and was not compromised. A good thing at a hacker conference.

I'm also proud that both PowerDNS and Fox-IT could play a part in this. Aldert, I don't believe you when you said HAR2009 would not have happened without us, but the thought is much appreciated! It must have been the good food at our BBQ :-)

There are moments from HAR2009 that I will never, ever, forget again. Thanks to everyone that made it happen!

Tuesday, August 11, 2009

PowerDNS is The Hackers Choice!

Very brief note to let you know that I'm thrilled that PowerDNS will be serving the HAR 2009 visitors! It has previously also been used by the same people at CCC congresses.

I'm very proud of this, and I hope the Recursor and Authoritative Server will continue to do well for such demanding users. And if there is any issue, the maintainers can rest secure in the knowledge that 24/7 on-site support is available.. from my tent.

Sunday, August 2, 2009

PowerDNS Recursor released!

I'm pretty proud of this release, or to state it better, proud of not having a real reason to do a release for over a year.

Over the past year, the PowerDNS Recursor has gone places I never thought it would go, powering the majority of internet subscribers in some large countries.

It is very rare for a project that services so many people, to exhibit so very little problems. I'm probably just lucky in this respect, but it still feels good.

However, since the world moved on over the past year, version 3.1.7 became somewhat hard to compile on modern Linux and UNIX distributions. In addition, Solaris 10 changed its ABI slightly, causing Recursor to crash quickly under heavy load.

So after a week of testing, has been released today, with no new features, only bug fixes.

For more details, please see the announcement.

Moving closer to pizza perfection!

I truly love a good pizza, but it is a rare event to find one. So, I've long been baking my own pies, at first in a normal oven and later in a special pizza oven. Although the "G3 Ferrari" looks impressive, mine is bright red, it is a decidedly weird machine.

For example, it has four heat levels, 1, 2, 2.5 and 3. One has to wonder how that happened.

I've read with great gusto the wonderful works by Jeffrey Steingarten which cover with great precision how very good food is made, including pizza. Jeffrey's book emphasized the importance of high heat, and even mentions the very same "G3 Ferrari" oven I have.

So, I've been using it to make pizzas of very varying quality - sometimes tremendously good, sometimes less so, and I never knew why. Like many aspiring pizzaiolos, I blamed my flour, and I assumed the professionals were using special brands.

Then I discovered the page of Jeff Varasano, who is a bigger pizza nut than I would've thought possible. It looks like he spent 10+ years figuring out how to do it, and from him I discovered the stunning secret: all pizza recipes I've ever seen in books, or online, are wrong. So, I set out to follow his instructions to the letter, which indeed led to very good and elastic dough.. and still no good pizzas ensued from my oven!

(in brief, any recipe which starts out by lumping all ingredients of the dough together and instruct to after mixing 'let it rest until it has doubled in size' is pretty far removed from reality. For more information, see Jeff's page)

Several months passed, and this weekend I found myself with some time off (since our son Maurits was spending time with his grandparents), and I decided to try again, this time using science.

Recall the Italian pizza oven with heat level '2.5'? It turns out the thermostat of this oven is a lying through its teeth! With the aid of a high-heat thermometer, I discovered the awful truth that the temperature of the oven has very little to do with the settings of the thermostat.

It turns out that the "G3 Ferrari" only reaches the required temperatures (400+C, around 750F) when the grill is red hot and has been on non-stop for quite some time, no matter what the thermostat may say.

So today, using my trusty thermometer, I timed it such that the oven reached this stunning temperature just when my pizza was ready.

And lo, it was wonderful!

To the non-pizza-enthusiast, this may not sound like a big thing - but this is an important step in my ongoing quest: be able to entertain large amounts of guests with mouth watering pizzas.

To be continued...

Monday, July 27, 2009

Some quick notes on RSA1024 signing performance

Just so this does not get lost - I've been doing some RSA1024 signing experiments because of my 'DNSSEC on PowerDNS' experiment, and the results were at first confusing.

For starters, friends of mine with Apple OS X reported very low numbers from the version of OpenSSL that ships with OS X (intel). The command to have OpenSSL perform speed tests is: 'openssl speed rsa1024'. Numbers were around half those reported on identical machines running a 32-bit Ubuntu.

Much investigations ensued, and conclusions are:
  • Apple ships a version of OpenSSL that misses certain optimizations. If you need performance for your applications, investigate which OpenSSL library they link against, and possibly investigate how to recompile or relink.
  • Go 64-bit, in a hurry. Twice as many bits appear to deliver over twice as much performance.
  • A modern Core2 based CPU running 64 bits code maxes out at about 1500 RSA1024 signatures/second/core, based on OpenSSL 1.0 beta 3, or Botan linked against GnuMP 4
  • Non-beta OpenSSLs are quite a bit slower, but not dramatically so
  • More naive code, that is not as highly optimized (like the otherwise excellent PolarSSL), will deliver around 1200 RSA1024 signatures/second/core (64 bits)
  • These numbers scale linearly with the number of cores involved - my 600 euro PC delivers 6000 signatures/second ('0.10 euro/signature/second').
It also looks like no worthwhile general purpose RSA hardware accelerators are available for use from Linux - Sun ships one, but its performance is not stellar (a lot more than 0.10 euro/signature/second), but it is not cheap, plus it is only officially supported on Sun hardware. If anyone has better ideas, please let me know!

PS: Why RSA1024? Because this is what DNSSEC is about for the foreseeable future..

Tuesday, July 14, 2009

So, why did I move my blog?

Aesthetically, I liked my old blog. The design was clean, it was 100% under my control, but that last part also turned out to be a problem. When spammers discovered, they filled it with junk. Junk which was sometimes filtered out, sometimes not, but in any case clogged my poor server. We are talking gigabytes of spam here, literally.

So eventually I caved. If Linus Torvalds can host his blog on, it must surely be good enough for me.

The old blog postings are still available here:
Be sure to add the '/index.html', because shortly, without it will forward you to this site.

So, welcome back dear readers, and I hope to entertain you with things I can't bring myself to shut up about.

This is the new location of Bert Hubert's blog!

This is the new location of Bert Hubert's blog!