As a community service, the glibc 2.14 'fixed bugs' with descriptions

Hi everybody,

 

The venerable GNU C Library just saw the release of 2.14. Within the release notes there is a list of bug numbers that are addressed by this release. Some of these are highlighted in the release notes, but most aren't.

I did some scripting and here is a lightly edited list of things fixed. I added a link to bug 10149, you can guess the URL for the rest. I bolded things that might actually hit my programs (and who knows, yours):

(potentially) security related:

   Bug 10149 - stack guard should lead with zero byte to gain protections

   Bug 11892 - putenv()/setenv() unbounded alloca()

   Bug 12393 - ld.so: insecure handling of privileged programs' RPATHs with

   Bug 12671 - multiple vulnerabilities in netdb.h/aliases.h/glob.h

Rest:

   Bug 386 - pthread_create returns ENOMEM but should return EAGAIN

   Bug 6420 - Mtrace deadlock

   Bug 7101 - getopt message for ambiguous options could be more helpful

   Bug 10138 - Outdated config.guess/sub

   Bug 10157 - Wrong value for sysconf(_SC_CPUTIME) or

   Bug 11099 - INT_FIELD_MAYBE_NULL changed behaviour on x86_64

   Bug 11257 - need finer control of group unioning in /etc/nsswitch.conf

   Bug 11558 - No way to set some options in /etc/resolv.conf

   Bug 11634 - tst-audit6.c doesn't compile without AVX support

   Bug 11697 - pt_chown doesn't work when the PTY's gid is already correct

   Bug 11724 - ld.so - Initialization and Termination Functions incorrectly

   Bug 11781 - Interoperability problems between malloc hook and GCC 4.5.0

   Bug 11799 - si_code is not SI_USER on raise()

   Bug 11820 - sys/user.h requires additional header in x86_64 to define

   Bug 11857 - Missing documentation in regex.h

   Bug 11895 - pselect incorrecly handles small negative timeouts on old

   Bug 11901 - __libc_message(do_abort = 1) will deadlock if called from malloc

   Bug 11952 - glibc may use uninitialized DTV slot, return NULL for

   Bug 12052 - posix_spawn() nonconformance (POSIX_SPAWN_SETSCHEDPARAM)

   Bug 12083 - aio_init() treatment of aio_num argument looks buggy

   Bug 12350 - Resolver doesn't save RES_SNGLKUP/RES_SNGLKUPREOP state in

   Bug 12420 - On AMD64 linux, getcontext resets FPU exception mask.

   Bug 12432 - backtrace() fails with recursive function on 64-bit

   Bug 12445 - printf() stack corruption in case of positional parameters +

   Bug 12453 - Broken thread local storage (TLS) initialization

   Bug 12454 - Inconsistency detected by ld.so: dl-deps.c: 622:

   Bug 12460 - AVX audit test failures with gcc 4.6

   Bug 12469 - Race condition in configure.in check for necessary ranlib

   Bug 12489 - prelinking ldso causes binaries to segfault upon startup

   Bug 12509 - dlopen(path_to_lib, RTLD_LOCAL|RTLD_NOLOAD) leaks memory

   Bug 12510 - elf/dl-lookup.c: STB_GNU_UNIQUE/ELF_RTYPE_CLASS_COPY lookup

   Bug 12511 - elf/dl-lookup.c: STB_GNU_UNIQUE/ELF_RTYPE_CLASS_COPY lookup

   Bug 12518 - memcpy acts randomly (and differently) with overlapping areas

   Bug 12527 - Off by one bug with ftell() with fmemopen()

   Bug 12583 - fnmatch: integer overflow in computation of the required

   Bug 12587 - sysconf(_SC_*CACHE) returns 0 for all caches on some CPUs.

   Bug 12597 - SSE4 strncmp failure

   Bug 12625 - mntent operations provide no indication of failure due to

   Bug 12626 - __backtrace_symbols_fd uses of out-of-scope storage in stack

   Bug 12631 - wcp[n]cpy are required by POSIX 2008

   Bug 12650 - Memory leak with dlopen() and thread-local storage variables

   Bug 12653 - undefined references to ssse3 routines when trying to link

   Bug 12655 - fix a comment in sysdeps/unix/sysv/linux/sys/syscall.h

   Bug 12684 - Multi-request DNS lookups do not properly fall back to

   Bug 12685 - fopen doesn't honor last byte of valid modes

   Bug 12713 - coreutils-8.12 "make check" thinks glibc-2.13's "getcwd()" is

   Bug 12714 - getaddrinfo(AF_INET6) does not return scope_id info provided

   Bug 12717 - declaration of getnameinfo() is not POSIX compliant

   Bug 12723 - pathconf for a FIFO returns a different value than fpathconf

   Bug 12724 - fclose violates POSIX 2008 on seekable input streams

   Bug 12734 - resolver failures without even sending a query.

   Bug 12766 - SEGV in error_at_line(3)

   Bug 12775 - Typo in sysdeps/x86_64/fpu/e_powl.S

   Bug 12782 - POSIX strerror_r quality of implementation

   Bug 12792 - perror violates POSIX regarding ferror status

   Bug 12795 - bits/resource.h is outdated

   Bug 12811 - regexec/re_search consumes huge amounts of memory

   Bug 12813 - Linux x86_64: glibc should prefer the vDSO over vsyscalls

Locale:

   Bug 9730 - sv_FI time format does not match fi_FI

   Bug 9732 - dz_BT Dzongkha collation order

   Bug 9809 - Please add Kurdish locale for Kurdish Sorani (CKB)

   Bug 11258 - es_CR locale update

   Bug 11487 - [Patch] to fix yesexpr and noexpr to use Po (Yes) and Jo (No)

   Bug 11532 - Support old DOS Lithuanian character sets in iconv

   Bug 11578 - sync glibc Latin American paper sizes with CLDR 1.8.1

   Bug 11653 - Incorrect LC_MONETARY symbol of es_NI.utf-8

   Bug 11668 - Paper Size is wrong for locale es_NI (A4 -> Letter)

   Bug 11837 - GB18030-2005 is not supported!

   Bug 11869 - LANGUAGE not taken into account unless LC_MESSAGES is set to

   Bug 11945 - Month names in Russian Localization should be in lowercase

   Bug 11947 - New locale for Meadow Mari language

   Bug 11987 - missing info on first day of week in Slovenian (sl_SI) locale

   Bug 12158 - Please add the new lij_IT locale

   Bug 12178 - New locale wae_CH, request for inclusion

   Bug 12200 - Please add the new yue_HK locale file

   Bug 12346 - Estonia (et_EE) joins the eurozone on Jan 1 2011

   Bug 12449 - Please add the new lb_LU locale

   Bug 12541 - update for indian locale for U+20B9 (New Rupee Symbol)

   Bug 12545 - [PATCH] localedef: fix error check for size_t < 0

   Bug 12551 - New locales for Swahili (Kenya and Tanzania)

   Bug 12582 - Incorrect date and time formats in en_SG locale

   Bug 12611 - New locale for Fulah (Senegal)

   Bug 12601 - iconv(3) doesn't handle invalid sequence properly

   Bug 12660 - Recent changes in tk_TM locale

   Bug 12681 - New locale for Bemba (Zambia)

   Bug 12711 - changes required for adding new currency symbol in indian

   Bug 12738 - Please add the new os_RU locale

   Bug 12746 - Encoding mismatch in se_NO file

   Bug 12777 - iconv mapping of U+0385 in CP1258 is likely incorrect

   Bug 12788 - [PATCH] setlocale sets the locale of LC_ALL incorrect to 'C'

   Bug 12814 - ISO-2022-JP-2 conversion of U+20AC gives strange result

PowerDNS Authoritative Server 3.0RC1 released! Now with DNSSEC!

I'm very proud to announce the first Release Candidate for PowerDNS Authoritative Server 3.0, now with full support for DNSSEC, TSIG, IPv6 master/slave, per-zone metadata and Lua zone editing. The DNSSEC support is 'fully automatic' - if everything goes well, all that is required is to set 'pdnssec secure-zone powerdns.com' and your zone is secured.

Read on for more information! To download, head to http://powerdnssec.org/downloads 

Warning

Version 3.0 of the PowerDNS Authoritative Server is a major upgrade. Please refer to Section 1, “From PowerDNS Authoritative Server 2.9.x to 3.0” for important information on correct and stable operation, as well as notes on performance and memory use. Known issues as of RC1 include: Not all new features are documented yet Queries for 'empty non-terminals' may give confusing results We are not 100% convinced all corner cases of NSEC3/NXDOMAIN give correct responses. Common cases function well DNSSEC has only been benchmarked up to 2000 queries/second but not beyond A lot more database connections are made and released

	Note
	RC1 released on the 4th of April 2011

Version 3.0 of the PowerDNS Authoritative Server brings a number of important features, as well as over two years of accumulated bug fixing.
The largest news in 3.0 is of course the advent of DNSSEC. Not only does PowerDNS now (finally) support DNSSEC, we think that our support of this important protocol is among the easiest to use available. In addition, all important algorithms are supported.
Complete detail can be found in Chapter 12, Serving authoritative DNSSEC data. The goal of 'PowerDNSSEC' is to allow existing PowerDNS installations to start serving DNSSEC with as little hassle as possible, while maintaining performance and achieving high levels of security.
Tutorials and examples of how to use DNSSEC in PowerDNS can be found linked from http://powerdnssec.org.
This release has received exceptional levels of community support, and we'd like to thank the following people in addition to those mentioned explicitly below: Peter Koch (DENIC), Olaf Kolkman (NLNetLabs), Wouter Wijngaards (NLNetLabs), Marco Davids (SIDN), Markus Travaille (SIDN), Leen Besselink, Antoin Verschuren (SIDN), Olafur Gudmundsson (IETF), Dan Kaminsky (Recursion Ventures), Roy Arends (Nominet), Miek Gieben (SIDN), Stephane Bortzmeyer (AFNIC), Michael Braunoeder (nic.at), Peter van Dijk, Maik Zumstrull, Jose Arthur Benetasso Villanova (Locaweb), Stefan Schmidt, Roland van Rijswijk (Surfnet), Paul Bakker (Brainspark/Fox-IT), Mathew Hennessy, Johannes Kuehrer (Austrian World4You GmbH), Marc van de Geijn (bHosted.nl), Stefan Arentz and Martin van Hensbergen (Fox-IT), Christof Meerwald, Detlef Peeters, Jack Lloyd, Frank Altpeter, frederik danerklint, Vasiliy G Tolstov, Brielle Bruns, Evan Hunt, Ralf van der Enden.
On to the release notes. Next to DNSSEC, other major new features include:

• TSIG for authorizing and authenticating AXFR requests & incoming zone transfers (Code in 2024, 2025, 2033, 2034). This allows for retrieving TSIG protected content, as well as serving it.
• Per zone also-notify.
• MyDNS compatible backend, allowing for 'instantaneous' migration from this authoritative nameserver. Code in commit 1418, contributed by Jonathan Oddy.
• PowerDNS can now slave zones over IPv6 and notify IPv6 remotes of updates. Already. Code in commit 2009 and beyond.
• Lua based incoming zone editing, allowing masters or signing slaves to add information to the zone they will (re-)serve. Implemented in commit 2065. To enable, use LUA-AXFR-SCRIPT zone metadata setting.
• Native Oracle backend with full DNSSEC support. Contributed by Maik Zumstrull, then at the Steinbuch Centre for Computing at the Karlsruhe Institute of Technology.
• "Also-notify" support, implemented by Aki Tuomi in commit 1400. Support for Generic SQL backends and for the BIND backend. Further code in commit 1360.
• Support for binding to thousands of IP addresses, code in commit 1443.
• Generic MySQL backend now supports stored procedures. Implemented in commit 2084, closing ticket 231.
• Generic ODBC backend compiles again, and is reported to work for some users that need it. Code contributed in ticket 309, author unknown.
• Massively parallel slaving infrastructure, able to check the freshness of thousands of remote zones per second, plus perform many incoming zone transfers simultaneously. Sponsored by Tyler Hall, code in 1449, 1500, 1859
• Core DNS logic replaced completely to deal with the brave new world of DNSSEC.

Bugs fixed:

• sqlite2 and sqlite3 backends used MySQL-style escaping, leading to SQL errors in some cases. Discovered by Sten Spans. Fixed in commit 1342.
• Internal webserver no longer prints '1e2%'. Bug rediscovered by Jeff Sipek. Fixed in commit 1342.
• PowerDNS would refuse to serve domain names with spaces in them, or otherwise non-printable characters. Addressed in commit 2081.
• PowerDNS can now serve escaped labels, as described by RF4343. Data should be present in backends in that escaped form. Code in commit 2089.
• In some cases, we would include duplicate CNAMEs. In addition, we would hand out a full root-referral when not configured to in some cases (ticket ticket 223). Discovered by Andreas Jakum, fixed in commit 1344.
• Shane Kerr discovered we would corrupt DNS transaction IDs from the packet cache on big endian systems. Fix in commit 1346, closing ticket 222.
• PowerDNS did not use RF1982 serial arithmetic, leading to a SOA serial number of 1 to be regarded as older than 4400000000, when in fact it is 'newer'. Issue (re-)discovered by Jan-Piet Mens.
• BIND backend got confused of a zone's filename changed after a configuration reload. Fix in commit 1347, closing ticket 228.
• When restarted by the Guardian, PowerDNS will perform a full multi-threaded cache cleanup, which took a long time and could crash. Fix in commit 1364.
• Under artificial circumstances, PowerDNS would never clean its packet cache. Found by Marcus Goller, fix in commit 1399 and commit 1408. This update also retunes the cleanup frequency.
• Packetcache would cache things it should not have been caching. Fixes in commits 1407, 1488, 1869, 1880
• When processing incoming notifications, the BIND backend was case-sensitive, and would disregard notifications in the wrong case. Discovered by 'Dolphin', fix in commit 1420.
• The init.d script did not mention the 'reload' command. Code in commit 1463, closes ticket 233.
• Generic SQL Backends would sometimes emit obscure error messages. Fix in commit 2049.
• PowerDNS would be confused by embedded NULs in domain names, and would also mess up the escaping of some characters. Fix in commit 1468, commit 1469, commit 1478, commit 1480,
• SOA queries for the name of a delegation point were not referred. Fix in commit 1466, closing ticket 224. In addition, queries for AAAA for a CNAMEd record pointing to a name with no AAAA would deliver a direct SOA, without the CNAME in between. Fix in commit 1542, commit 1607. Also, wildcard CNAMEs pointing to a record without the type requested suffered from the same issue, fix in commit 1543.
• On processing an incoming AXFR, once an MX or SRV record had been seen, all future fields got a 'priority' entry as well. This had no operational impact, but looked messy. Fixed in commit 1437.
• Aki Tuomi discovered that the BIND zonefile parser would misrepresent 'something IN MX 15 @'. Fix in commit 1621.
• Marco Davids discovered the BIND zonefile parser would trip over really long lines. Fix in commit 1624, commit 1625.
• Thomas Mieslinger discovered that our webserver would only be started after dropping privileges, which could cause problems. Fix in commit 1629.
• Zone2sql did quite often not do exactly what was required, which users fixed by editing the SQL output. Revamped in commit 2032.
• An Ubuntu user discovered in Launchpad bug 600479 that restarting database threads cost a lot of memory. Normally this is rare, except in case of problems. Addressed in commit 1676.
• BIND backend could crash under (very) high load with very large numbers of zones (hundreds of thousands). Fixed in commit 1690.
• Miek Gieben and Marco Davids spotted that PowerDNS would answer the version.bind query in the IN class too. Bug reported via twitter! Fix in commit 1709.
• Marcus Lauer and the OpenDNSSEC project discovered that outgoing notifications did not carry the 'aa' flag. Fixed in commit 1746.
• Debugging PowerDNS, or backgrounding it, could cause crashes. Fixed by Anders Kaseorg in commit 1747.
• Fixed a bug that could cause crashes on launching thousands of backend connections. Never observed to occur, but who knows. Fix in commit 1792.
• Under some circumstances, large answers could be truncated in mid-record. While technically legal, this upset a number of resolver implementations (including the PowerDNS Recursor!). Fixed in commit 1830, re-closes ticket 200.
• Jan Piet Mens and Florian Weimer discovered we had problems dealing with escaped labels and escaped TXT fields. Fixed in commit 2000.
• After 2.2 billion queries, statistics would wrap oddly. Fix in commit 2019, closing ticket 327.

Improvements:

• Long TXT records are now split into 255-byte components automatically. Implemented in commit 1340, reported by Darren Gamble in ticket 188.
• When receiving large numbers of notifications, PowerDNS would check these synchronously, leading to a slowdown for other services. Fixed in commit 2058, problem diagnosed by Richard Poole of Heart Internet.
• Fixed compilation on newer compilers and newer versions of Boost. Changes in 1345 (closes ticket 227), 1391, 1394, 1425, 1427, 1428, 1429, 1440, 1653, thanks to Ruben Kerkhof and others.
• Moved Generic PostgreSQL backend over to the newer E'' style escapes. commit 2094.
• Compilation fixes for Mac OS X 10.5.7 in commit 1389, thanks to Tobias Markmann.
• We can now bind to scoped IPv6 addresses, lack spotted by Darren Gamble. Part of the fix is in commit 2018.
• Built-in query cache can now also cache queries which lead to multiple answers. Code in commit 2069.
• Prodded on by Jan Piet Mens, we now support 'unknown types' (which look like TYPE65534).
• Add 'slave-renotify' to retransmit notifies for slaved zones, which is helpful when acting as a 'signing slave' for a hidden master. Code in commit 1950.
• No longer let zone2sql and zone2ldap import BIND 'hint' zones. commit 1998.
• Allow for timestamps to explicitly be specified in (s)econds. Code in commit 1398, closing ticket 250.
• Zones with URL and MBOXFW records can be transferred over AXFR, code in commit 1464.
• Maik Zumstrull cleaned up the BIND Backend makefile, plus taught our init.d script to read /etc/default/pdns. Code in commit 1601, commit 1602.
• Generic SQL backends now support multiple masters in the domains table. Code in commit 1857. Additionally, masters can also have :port numbers. Code in commit 1858.
  
  
  

PowerDNSSEC: packages available, ready for light production use

Dear PowerDNS Community,


With the help of many of you, we've now brought 'PowerDNSSEC' to the point where it is in light production. Several of our important domains have already been migrated to the PowerDNS Authoritative Server 3.0 prereleases.  Several PowerDNS users have done the same with their domains (please let us know!).

We are pleased to announce the regular availability of documentation, packages and tarballs for testing. On http://powerdnssec.org/downloads/packages you will find RPM/DEB for 32-bit and 64-bit Linux. On http://powerdnssec.org/downloads you will find tarballs which can be compiled on other systems.

For more information head over to http://www.powerdnssec.org (which of course is powered by PowerDNSSEC).

Documentation is on http://doc.powerdns.com/powerdnssec-auth.html

Even more information is on http://wiki.powerdns.com/trac/wiki/PDNSSEC - including how to get started, and how to get help.

In brief, PowerDNSSEC will allow you to continue operating as normal in many cases, with only slight changes to your installation. There is no need to run signing tools, nor is there a need to rotate keys or run scripts.

Particularly, if you run with Generic MySQL, Generic PostgreSQL or Generic SQLite3, you should have an easy time. A small schema update is required, plus an invocation of 'pdnssec secure-zone domain-name ; pdnssec rectify-zone domain-name' per domain you want to secure. And that should be
it.

Supported are:

• NSEC
• NSEC3 in ordered mode (pre-hashed records)
• NSEC3 in narrow mode (unmodified records)
• Zone transfers (for NSEC)
• Import of 'standard' private keys from BIND/NSD
• Export of 'standard' private keys
• RSASHA1
• RSASHA256
• "Pure" PostgreSQL, SQLite3 & MySQL operations
• Hybrid BIND/PostgreSQL/SQLite3/MySQL operation
• Front-signing slaved data from legacy installations

See http://doc.powerdns.com/dnssec-supported.html for more specifications.

To join the fun, download the tarball and packages which can be found on the sites above, and let us know how it works for you!

To clarify, we do not recommend taking the current code snapshot into heavy production, but we are getting close.

Kind regards,
Bert

Linux or UNIX programmer? Go get The Linux Programming Interface: A Linux and UNIX System Programming Handbook

The Linux Programming Interface: A Linux and UNIX System Programming Handbook
Are you a Linux or UNIX programmer? Get this book. Do you know a Linux or UNIX programmer and want to give him (her? ;-)) a gift? Get this book. I thought it was a stack of manpages, which would already have been great, but this book is the true successor to the historic Stevens works on UNIX. If you think you don't need this book since you know everything already, that's what I thought too, and I was wrong. Even if you won't read it, the 1552 pages will look really good on your desk.

So go get the book already.

http://www.amazon.com/Linux-Programming-Interface-System-Handbook/dp/1593272200/ref=sr_1_1?ie=UTF8

PowerDNS Recursor additional Lua hooks for IPv6 DNS64 and Renumbering

Dear PowerDNS Community,

The PowerDNS Recursor is currently being extended with additional Lua hooks
and extra infrastructure to support flexible DNS64 operations, plus perform
on-the-fly IPv4 or IPv6 renumbering.

DNS64 is described on http://tools.ietf.org/html/draft-ietf-behave-dns64-11
and in brief: 

  "DNS64 is a mechanism for synthesizing AAAA records from A records.  DNS64
   is used with an IPv6/IPv4 translator to enable client-server communication
   between an IPv6-only client and an IPv4-only server, without requiring any
   changes to either the IPv6 or the IPv4 node"

Those of you with an interest in these features are invited to test out the
following *pre-release*, specifically to let us know if the API is sufficient
for your needs:

http://svn.powerdns.com/snapshots/pdns-recursor-3.3-hooks.tar.bz2

It can be compiled like any other PowerDNS Recursor release. 

New in the version are the 'nodata()' and 'postresolve()' Lua hooks. Nodata
functions just like nxdomain(), except that it gets called when a domain
exists, but the requested type doesn't. This is where DNS64 happens.

Postresolve() is different, and very powerful - it gets handed the complete
DNS answer as it would be sent out, ready for modification from Lua. This is
where one might for example perform on the fly IP address renumbering.

In the release you can find powerdns-example-script.lua which contains a
working sample for both of the new hooks. This script can also be viewed on
http://wiki.powerdns.com/trac/browser/branches/pdns-dns64/pdns/powerdns-example-script.lua

Note: DO NOT TAKE THIS SCRIPT INTO PRODUCTION - it blacks out important
sites

To get going without disturbing any existing nameservers on your computer,
compile the PowerDNS Recursor, and start like this:
 $ ./pdns_recursor --local-address=0.0.0.0 --local-port=5300 --daemon=no
   --socket-dir=./ --lua-dns-script=powerdns-example-script.lua 

Known defects are:
 postresolve() can't yet access the original dns rcode
 there is no way for nodata() to set the TTL to the SOA minimum value
  as specified by draft-ietf-behave-dns64

Please let us know your thoughts so we can make sure the API has everything
needed for great DNS64 and renumbering operations!

Kind regards,

Bert Hubert

The "leaky abstraction" of the POSIX file interface

Hi everybody,

Lately I've been looking into large scale database & key/value storage engine performance, and the results were not very good. Machines that should seemingly be able to load the full .COM zone with no problems turned out to have a very hard time to do so - even though COM zone and indexes all fit comfortably in RAM. Despite this, loads of disk i/o ensued.

This led to some investigations on how Linux and the various storage engines interact. Through tweaking a lot of settings, decent performance was achieved, but this process did drive home the fact that your operating system might have a hard time guessing "what you want".

Any decent operating system is fitted with an in-memory cache to speed up disk access. The difference in speed between a disk read and a memory read is so stunning (many orders of magnitude) that being disk bound or being memory bound can make or break a solution.

Naively, we'd want the operating system to cache exactly the data we want it to cache. However, the operating system can't read our minds, and may decide to not dedicate all of the system memory to do exactly what you want.

This issue is worth a blog post in its own right, because if you want dependable performance, it is not good if your kernel decides on Monday to do the right thing, and on Tuesday to take 2 days to finish a job that previously ran in 15 minutes - simply because something else has decided to use the cache in the meantime!

While investigating how reads and writes actually hit the platter, I wrote the following little "exploit" that tickles most operating systems into a flurry of (at first thought) unexpected disk activity. Try to predict what this does:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

// $ dd if=/dev/zero of=blah bs=1024000 count=1000  # this creates a 1G empty file
// $ sudo sysctl vm.drop_caches=3                   # this empties the caches
// $ vmstat 1 & ./writerreader
// and stand back in awe


int main()
{
  int fd = open("./blah", O_WRONLY);
  if(fd < 0) {
    perror("open");
    exit(0);
  }
  
  char c[2];
  for(int n=0; n < 128000; ++n) {
    if(pwrite(fd, (void*)c, 2, (n+1)*8192 - 1) < 0) {   // write out 2 bytes
      perror("pread");
      exit(0);
    }
  }
  fsync(fd);
  close(fd);
}

Try adding up how much read & write activity was caused by writing out 128KB of data.
Please note that there is very little an OS can do to improve this! It is just a fact of life.

So why did I call this a 'leaky abstraction' in the title of this post? Joel Spolsky (who has a blog, Joel on Software, which you should definitely read) invented this term to describe the situation where any kind of API, which supposedly hides underlying details from you, will from time to time confront you with results of said details.

And in this case, the details mean that writing 128KB of data leads to around 2G of I/O (1G of reads, 1G of writes).

So why does this happen? At the very base, disks operate in terms of (mostly) 512 byte sectors. You can't perform any reads or writes smaller than a sector. In this case, it means that writing 2 bytes requires first reading the sector(s) which straddle the two bytes, adding the two bytes, and then writing them out again.

To make matters worse, internally most operating systems don't think in terms of sectors, but in terms of blocks and pages - which are even larger. The sample program above is tuned to perform 2 byte writes which accurately straddle 2 pages - leading to 2 full pages to be read and written per 2 byte operation. And a page tends to be 4096 bytes!

Getting back to the beginning of this post, the effect described above partially explains the really bad performance observed in some key/value storage engines, engines which perform millions of tiny little pwrites, leading to massive read i/o, where you did not expect it.

In a follow-up post, I will probably delve into how to improve this situation, and perhaps by that time I will have gotten round to a solution that might be generic enough to help more projects than just mine to gain more control over how to optimize loads so that they do not cause more disk i/o than you'd want.

In the meantime, I hope to have entertained you with some arcane knowledge!

PowerDNS Recursor 3.3 released!

Hi everybody,

We're proud to announce the release of the PowerDNS Recursor 3.3!

It can be downloaded from http://www.powerdns.com/

Version 3.3 fixes a number of small but persistent issues,
rounds off our IPv6 %link-level support and adds an important
feature for many users of the Lua scripts.

In addition, scalability on Solaris 10 is improved.

Bug fixes:

* 'dist-recursor' script was not compatible with pure POSIX
/bin/sh, discovered by Simon Kirby. Fix in commit 1545.
* Simon Bedford, Brad Dameron and Laurient Papier discovered
relatively high TCP/IP loads could cause TCP/IP service to
shut down over time. Addressed in commits 1546, 1640, 1652,
1685, 1698. Additional information provided by Zwane
Mwaikambo, Nicholas Miell and Jeff Roberson. Testing by
Christian Hofstaedtler and Michael Renner.
* The PowerDNS Recursor could not read the 'root zone' (this
is something else than the root hints) because of an
unquoted TXT record. This has now been addressed, allowing
operators to hardcode the root zone. This can improve
security if the root zone used is kept up to date. Change
in commit 1547.
* A return of an old bug, when a domain gets new nameservers,
but the old nameservers continue to contain a copy of the
domain, PowerDNS could get 'stuck' with the old servers.
Fixed in commit 1548.
* Discovered & reported by Alexander Gall of SWITCH, the
Recursor used to try to resolve 'AXFR' records over UDP.
Fix in commit 1619.
* The Recursor embedded authoritative server messed up
parsing a record like '@ IN MX 15 @'. Spotted by Aki Tuomi,
fix in commit 1621.
* The Recursor embedded authoritative server messed up
parsing really really long lines. Spotted by Marco Davids,
fix in commit 1624, commit 1625.
* Packet cache was not DNS class correct. Spotted by "Robin",
fix in commit 1688.
* The packet cache would cache some NXDOMAINS for too long.
Solving this bug exposed an underlying oddity where the
initial NXDOMAIN response had an overly long (untruncated)
TTL, whereas all the next ones would be ok. Solved in
commit 1679, closing ticket 281. Especially important for
RBL operators. Fixed after some nagging by Alex Broens
(thanks).

Improvements:

* The priming of the root now uses more IPv6 addresses.
Change in commit 1550, closes ticket 287. Also, the IPv6
address of I.ROOT-SERVERS.NET was added in commit 1650.
* The rec_control dump-cache command now also dumps the
'negative query' cache. Code in commit 1713.
* PowerDNS Recursor can now bind to fe80 IPv6 space with
'%eth0' link selection. Suggested by Darren Gamble,
implemented with help from Niels Bakker. Change in commit
1620.
* Solaris on x86 has a long standing bug in port_getn(),
which we now work around. Spotted by 'Dirk' and 'AS'.
Solution suggested by the Apache runtime library, update in
commit 1622.
* New runtime statistic: 'tcp-clients' which lists the number
of currently active TCP/IP clients. Code in commit 1623.
* Deal better with UltraDNS style CNAME redirects containing
SOA records. Spotted by Andy Fletcher from UKDedicated in
ticket 303, fix in commit 1628.
* The packet cache, which has 'ready to use' packets
containing answers, now artificially ages the ready to use
packets. Code in commit 1630.
* Lua scripts can now indicate that certain queries will have
'variable' answers, which means that the packet cache will
not touch these answers. This is great for overriding some
domains for some users, but not all of them. Use
setvariable() in Lua to indicate such domains. Code in
commit 1636.
* Add query statistic called 'dont-outqueries', plus add IPv6
address :: and IPv4 address 0.0.0.0 to the default
"dont-query" set, preventing the Recursor from talking to
itself. Code in commit 1637.
* Work around a gcc 4.1 bug, still in wide use on common
platforms. Code in commit 1653.
* Add 'ARCHFLAGS' to PowerDNS Recursor Makefile, easing 64
bit compilation on mainly 32 bit platforms (and vice
versa).
* Under rare circumstances, querying the Recursor for
statistics under very high load could lead to a crash
(although this has never been observed). Bad code removed &
good code unified in commit 1675.
* Spotted by Jeff Sipek, the rec_control manpage did not list
the new get-all command. commit 1677.
* On some platforms, it may be better to have PowerDNS itself
distribute queries over threads (instead of leaving it up
to the kernel). This experimental feature can be enabled
with the 'pdns-distributes-queries' setting. Code in commit
1678 and beyond. Speeds up Solaris measurably.
* Cache cleaning code was cleaned up, unified and expanded to
cover the 'negative cache', which used to be cleaned rather
bluntly. Code in commit 1702, further tweaks in commit
1712, spotted by Darren Gamble, Imre Gergely and Christian
Kovacic.

Changes between RC1, RC2 and RC3.

* RC2: Fixed linking on RHEL5/CENTOS5, which both ship with a
gcc compiler that claims to support atomic operations, but
doesn't. Code in commit 1714. Spotted by 'Bas' and Imre
Gergely.
* RC2: Negative query cache was configured to grow too large,
and was not cleaned efficiently. Code in commit 1712,
spotted by Imre Gergely.
* RC3: Root failed to be renewed automatically, relied on
fallback to make this happen. Code in commit 1716, spotted
by Detlef Peeters.