Sunday, December 7, 2014

Picking open source technology: pick the community too

A brief post on picking (open source) technology. When we need something for our project, say, a database, we have a large range of choices. There is old and staid software that may not be hip, but comes with a well known track record and lots of features. There are new shiny frameworks that are blazing trails and publishing impressive benchmarks, written in languages still being standardized.

What do we pick? First of course, our dependency has to meet (most of) our needs. Next, we can look at its current reputation - if a project is known to be a bloated mess, or is well known to crash if you wave at it, we can discount the project for now.

But, even then, we are left with a lot of choice. What I care about these days is the community around a project. Because it turns out that the community is the best predictor of the future of a project. We can’t actually predict the future, but we can be sure that we’ll have new needs for our dependencies. We can also be sure we’ll have questions and discover bugs.

And a healthy community almost guarantees that things will end up well. As a recent example, PowerDNS has recently become involved with a customer where we are helping setup a PowerDNS based anycast environment. For this we needed a BGP implementation. A quick consultation with our community reported that ExaBGP would be a very good choice, and indeed, it offered all the features we needed.

On deployment however, we found two small issues that were holding up our deployment. PowerDNS employee of the month Peter van Dijk worked with the ExaBGP people, fixed the two issues, and both fixes have now been merged by the ExaBGP project. They are happy, we are happy. The next release of ExaBGP won’t just meet our needs, it will suit the needs of many more people.

Last week, a developer reported that the most excellent Valgrind tool found some potential errors in the LMDB project, and I was shocked to see LMDB lead developer tell the reported to ‘learn how to use his tools’, and not address his report in any meaningful way.

I asked Howard Chu to reconsider his stance, given my belief that open source can’t be great without a functioning community, something you don’t build this way. Howard told me that how he treated the reporter was entirely intentional. Shortly afterwards, the following was posted on the LMDB list:

“if you post to this list and your message is deemed off-topic or insufficiently researched, you *will* be chided, mocked, and denigrated. There *is* such a thing as a stupid question. If you don't read what's in front of you, if you ignore the list charter, or the text of the welcome message that is sent to every new subscriber, you will be publicly mocked and made unwelcome.”

This is entirely intentional and by design.”

Now of course, everyone is free to run their project in their own way. But I’m also free to pick my dependencies, and I care about the development community being sane and inclusive. “Bitchslapping” reporters of potential valid issues isn’t that. Formalizing this behaviour most definitely isn’t. I won’t be picking LMDB for any future projects unless this changes.

As a parting gift to LMDB, I worked to document the (powerful) semantics of LMDB, and dragged details out of Howard. This document can be found on If you use LMDB, it may be helpful for you.

I spent this evening working with PowerDNS contributor Ruben Kerkhof to merge several of his fixes for PowerDNS issues. Ruben’s company Tilaa uses PowerDNS, and we are Tilaa customers. I’m truly proud of our community and what we achieve together, and I recommend that every open source project works to foster its community as well.

So in short.. before picking a technology to depend on, investigate how they deal with feature requests, bug reports and questions. What you will learn is the best predictor of how the project will serve you (and vice versa!) over the coming years.

Monday, November 3, 2014

Tin cans, can openers and solar power: explaining the snail’s pace of innovation

Innovation is fascinating. It brings us (by definition) all improvements in technology. Simultaneously, it proceeds at a snail’s pace. An example I love to share is the invention of the tin can, which was a breakthrough in food preservation (there were no refrigerators at the time, and tin cans allowed people to safely transport and store previously perishable goods).

Made practicable in the early 19th century, tin cans were sold bearing notices like “Cut round the top near the outer edge with a chisel and hammer”. And then.. it took over 50 years for the first home operable tin opener to arrive in 1870 (in the 1850s, an opener was available for use in stores, who would open your cans for you). The modern easy to use and safe opener was only invented in 1925.

Until the arrival of the opener we know today, opening tin cans was a dangerous exercise. In fact, it was a common cause of injuries and infections, highly dangerous before the advent of antibiotics. Yet it took decades before the problem was solved, even when the need for a solution stared people in the face!

This example is historical, and hard to come to grips with - why did no one invent the current can opener earlier? The mind boggles. Here’s a far more recent example and one that is still developing. It may help shed light on why innovation proceeds so slowly.

(It looks like this post is about how much I like solar, and while this might be the case, the point I’m trying to make is not that solar power is great. The point is to elaborate on why innovation operates so slowly, using a currently happening development. In fact, if you find yourself disagreeing with my description of the bright future of renewable energy, ponder where your disagreement is coming from. That very disagreement likely is what this post is about!)

Renewable energy

Renewable energy sources have long been treated with derision. “What about when there’s no wind and no sunlight?”. Industry professionals and engineers alike shared this disdain, because as seen through their eyes, solar power compares very badly with (for example) a coal powered plant. The coal powered plant delivers energy against low prices (perhaps 3-7 eurocents/kWh), and does so continuously within a short few hours of turning it on, which you can do at any time you desire.

If you hold this next to solar power, photovoltaics do come off very poorly. They roughly operate in three modes - no output (night), 10% output (overcast) or 80-100% output (direct sunlight). One can try to predict when they’ll deliver this output, but you’ll never get it quite right. And, to add insult to injury, the production cost per kWh currently is a lot higher than coal powered plants! So not only don’t you know if and when you’ll get energy, you have to pay more for it too. A mostly similar story applies to wind power.

The upshot of the intermittent nature of availability is that by the time you’ve built enough solar and wind farms to cover your average needs, you’ll have generated vast overcapacity for when the sun shines and it is windy. Your capital costs meanwhile have also been huge compared to boring old power plants.

In short, from the viewpoint of the seasoned energy professional, solar and wind power makes for very unreliable expensive capacity that generates imbalance between power needs and power production. The professional then points out that energy storage (which could average out the imbalance) is prohibitively expensive, and won’t save the day. Repeat this story for decades and decades, and you are in 2008 or so.

Personally, I was fully on board with this line of thinking. I advocated the development of large fusion or even fission plants as the (remote) future.

Then, politics happened

Then, through various plans that weren’t too well thought out (I think), all of a sudden whole countries started building heavily subsidized wind farms and suddenly cheap roof-top solar panels at a massive clip. Germany comes to mind. The Netherlands and other countries are now following behind, even with dwindled subsidies.
How come? Well.. the original energy professionals were thinking entirely on the production side (and who can blame them, that’s where they worked!). The reality is that in many countries, no consumer actually pays anywhere near 3-7 (euro)cents/kWh for electrical power. Transport costs and heavy taxes increase actual ‘grid costs’ to over 20 eurocents/kWh in most parts of Europe.

And since producing energy you use yourself is untaxed, domestic energy users compare the production costs of solar against the delivery and taxed costs of grid power. And guess what, since there is a factor 3 to 6 between these numbers, all of a sudden ‘grid parity’ arrived. Unsubsidised solar energy is now cheaper than grid power in many places.

Google trends for “grid parity

Note how the concept of grid parity, or at least the term, burst on the scene in January 2008 (the Wikipedia page was created in June of that year). Before that date, the old view of solar power as a sort of retarded coal fired plant prevailed. Production parity was what mattered. Grid parity, which is what matters for consumers, was not a factor. (By the way, I checked - the flatline of the graph pre-2008 is real, and does not reflect a lack of data).

Grid parity is when solar is interesting for consumers - production parity is only relevant to power producers.

The new reality

What the staid electricity people thought unacceptable (and perhaps unimaginable) is now the new reality. The centralized generation of power has become an uneconomic and highly ungrateful enterprise to be in. If the sun shines and the wind blows, there is literally no need for the coal fired plant. Effective energy prices now routinely plummet to 0 cents, and have gone negative on occasion.

Note zero pricing around 03:00 and negative prices around 14:20

And while this is clearly unsustainable (nobody can generate power this way and stay in business), this reality isn’t going to go away, although some localities have so far succeeded in outlawing (Cyprus) or heavily taxing (Hawaii) grid connected solar panels.

The practical upshot though is that even today, substantially lower amounts of coal and gas are being shoveled into power plants.

Meanwhile, wily entrepreneurs have cottoned on the new reality of highly variable energy prices, and are scheduling their business needs around them. Processes that lose money at 20 cents/kWh make a lot more sense at 3 cents/kWh! This is the future, and attempting to emulate the flat power prices of the past century is not.

In another shocking development, a decade ago, California suffered from brownouts and even rolling blackouts, mostly on hot days when lots of air conditioners were running. Now it turns out that a lot of this can be blamed on energy fraud by Enron, but the problem of peak energy use was real.

Welcome to the new reality - already in 2012 the peak was mostly gone, and in 2014 it will have become a dip. The new challenge is the ramp-up when the sun goes down. All of this because in 2012 photovoltaics met the concept of ‘grid parity’, a concept very rarely discussed before 2008.

Only now that minds have been liberated, new thinking has arrived. One could for example ponder potential mass adoption of electrical cars, cars that will need to be charged before their owners drive off with them, but conceivably could charge ‘on demand’ during the day. They could even deliver power during unexpected peak net-demand periods (!). Nearly free energy and “the internet of things” could make this a reality.

Lots of other things could happen too, for example electrical heating of houses currently heated by gas, electrical (on demand) heating of water reservoirs for residential use (‘peak shaving’), bulk energy storage centrally, regionally or even in homes, etc. It is a whole new world out there. How things will end up is hard to predict, but we can already be sure that everything will be cheaper and a lot friendlier to the environment. And this leaves undiscussed the geopolitical impact of a world far less reliant on oil and gas!

All this thinking was previously rejected out of hand, “because it did not act like a power plant we know”.

What this means for innovation

In hindsight, the commonly held belief that solar (or renewable energy in general) will never fly until it performs exactly like a traditional power plant was and is ludicrous. But you still hear it today. A lot. And when you hear it, ponder the decades it took to invent a reliable can opener. Innovation happens at a snail’s pace, and only appears obvious in hindsight.

This inability to see the beyond the present concepts appears to be a feature (bug?) of human nature.

If you are personally trying to innovate, read up on the history of previous inventions, and see how they were rejected by experts and professionals. Read on until you get a feel for proper rejection (“the flying car” with present day technology) and failure-of-the-imagination rejection (British Post Office comment on phones “no need, we have lots of messenger boys”).

To innovate, we must free yourselves of dearly held convictions and concepts. Ironically, the people with the most knowledge and thus the best credentials to invent the future tend to cling fastest to previously held concepts. This is the challenge of innovation - using the knowledge out there, but without having it hold us back.

Good luck!
(Thanks to Nicholas Miell, Tsjoi Tsim and Jan van Kranendonk for editing & constructive criticism - all mistakes remain mine!)

Tuesday, August 19, 2014

The absolute minimum difficulty recipe with maximum impact: authentic chicken soup

I like cooking, and I like to share that joy. If  you can’t cook however, your first efforts are likely to be mediocre, which is not very encouraging. “Why bother?!”.

In this post, I’ll share something that can barely be called a recipe, that’s how simple it is. BUT! The results are entirely authentic and spectacular. You can take this with you to a home cooked dinner, and your friends will be very happy with your efforts. They will ask you for the recipe! Little will they know how easy it is. Also, it is almost impossible to mess this up. I dare you to try ;-)

Ingredients: 10-20 raw chicken wings (plain or spiced, NOT breaded!), 2-4 onions, a few large carrots (or more smaller ones), salt, pepper.
Required equipment: oven-proof container, oven, pan, stove, strainer or colander.
Time spent in the kitchen: 15 minutes

Pre-heat oven to 180C (350F). Meanwhile, put the chicken wings in the oven-proof container. If you got plain wings, add some salt and pepper.  If you have it, sprinkle some oil over the chicken wings.

Put container with chicken in oven (if at temperature). Then you can wait 25 minutes, you can wait 35 minutes, you can wait 45 minutes, and in every case you’ll have some pretty good chicken wings. If after 25 minutes they look ready to eat, they ARE. Do not eat unless thoroughly hot inside!

(if you don’t have an oven, you can also fry the chicken wings in a pan on a stove, works just as well, but you need to pay some closer attention to getting them browned all round and well done inside. Will require more oil).

Now for the surprising part. Eat as many of the chicken wings as you feel like. No need to finish them all. This was not your special dinner ;-)

Next, fill a pan with a few liters (quarts) of water, then throw in all your chicken wings, the ones you ate, the ones you didn’t. Peel onions, cut in large chunks, add to water. Chuck the carrots in there too. Bring water to a boil, turn down heat and leave to simmer for 3, 4, 5, hell 6 hours if you feel like it. If during the boiling you note brown stuff floating on the water, remove that with a spoon. Your house will smell lovely meanwhile.

Afterwards, use colander or strainer to filter out all the bones and pieces of meat. (You can add back the pieces of meat if you want.) Now taste the soup and add salt to taste. This part is not optional, it really needs salt.

That’s it! You can take the soup with you to a party and heat it there. In a fridge it will last for days.

Note: to improve on this soup, add more vegetables (doesn’t really matter which ones), or instead of chicken wings, use an entire chicken.

Monday, August 11, 2014

Some tips when travelling through France: toll & internet

Brief post so other people can find this. If you are going on holiday in France by car, I can highly recommend getting a 'telepeage' badge, which allows for contactless payments.  This means you can pay for the highway tolls without the hassle of tickets and credit cards at booths. The telepeage lanes go a lot quicker than the ones for people who need tickets. Many of them you can pass at 30km/hour even! In The Netherlands you can get a badge at the ANWB before you go. I'm also told you can buy them at the largest toll stations.

Secondly, if you want internet, Orange has several interesting options.  There is the "Let's Go" offer which is attractive. It is a WiFi box which can offer internet service to up to 5 devices. It does HSDPA. There's also a 4G offer, which is even cooler.

However, this being France, so life is not made easy for you. When you buy the box, inside you find a form you have to fill out and send in with a copy of a photo ID. Until this has been processed, you can't top up your 500MB internet credit. In my experience, this processing takes more than a week at least.

It so turns out that the store where you buy the Let's Go offer can also verify your ID on the spot, but nobody reminds you of this. So, when you buy it, insist that they register you too. Even then it takes 48 hours to process.

Secondly, even though the "Let's Go" offer is geared for tourists you can only top up your internet credit.. using a France-domiciled credit card!! I'm not making this up. You can however buy credit at "Tabac" shops or Orange boutiques. Not all Tabac shops know their Orange terminal can do this though.

I would recommend that when you buy the Let's Go box, you take care of the identification, and also immediately buy a voucher to top up. For 35 euros, I got 3500MB of internet, which should suffice. (Unless you forget to turn off cloud backups of all photos you take..)

Finally, although many shops are called 'Orange', I found that the ones labeled as 'Partner' on the Orange website do not carry the Let's Go offer. So head one that is actually Orange owned.

With the advice above.. you may save yourself the 6 visits to various Orange stores I needed to get functioning internet access!

Tuesday, July 8, 2014

Are the media right and are we all going to die from antibiotic resistance?

A brief post - I've long been unhappy with the media's reporting on science. It appears that news sites aren't there to disseminate truth and educate the populace, but to sell advertisement and clicks. Took me long enough to find that out.

So we get lots of news on how we are all going to die because all antibiotics have failed. We even see repeated claims that nobody is working on new antibiotics, that the pipeline for new discoveries is empty, and nothing is on the horizon. I saw so much of this that I actually started to believe it too.

Remember MRSA and how it would kill us all? Oddly enough it didn't, and while it is a nasty bug, there are now protocols to get a handle on it. You probably missed the article telling you so because "we're all going to die" gets a lot more clicks than "with careful work, MRSA outbreaks can be controlled".

"We're all going to die" is a potent way to draw in an audience. Simultaneously, scientists are happy to big it up, since it is a real problem, and it deserves more attention than it gets. Don't get me wrong on that! The world of medicine spends way way more on cholesterol lowering (to dubious effect) than on antibiotics (which actually save lives). And a dead patient needs no cholesterol lowering, so they should get on it pronto.

So, where are we? Bacteria can be divided (roughly) into "gram positive" and "gram negative". On the gram positive front, a whole new class of antibiotics has now been trialed for a few years, and they are called lipoglycopeptides. The two most recently tested (dalbavancin and oritavancin) show excellent effectiveness against MRSA. Interestingly, these new antibiotics require a single dose which does its work for the next weeks, so you can't even forget to take the pills. The great Richard Lehmann discusses these over at the BMJ (and here, even more here).

Gram negative bacteria are different, and have an outer membrane that protects them against many antibiotics in the first place, and the situation there is less hopeful, and currently far more worrying than MRSA. The main worry now are carbapenem-resistant Enterobacteriaceae (CRE), equipped with the scarily named New Delhi metallo-beta-lactamase (NDM-1) enzyme. These are the superbugs that hit the news a lot.

What did not hit the news was a result published last week where a whole swath of NDM-1 carrying bacteria was effectively treated (in mice) with regular antibiotics plus a known compound previously considered as a hypertension drug (which sadly failed to lower blood pressure). An infection that with best current treatment killed 100% of test animals now only killed 5%. Of mainstream media, only the Wall Street Journal reported on it. "It it bleeds, it leads" - and modest but important progress does not, it appears.

Now, are we all saved? No, not yet. In the end it is a battle of bacterial genes against our whits, but our arsenal of methods these days is astounding. There is every reason to be sure we'll keep on winning this battle. The next stage will be winning it cheaply and durably.

But my main point is - whenever you see the news reporting "there are no new antibiotics" and that nobody is working on it, think back to this page. Not only are those claims false, a whole new class of antibiotics are now coming out of the lab (53 at last count) and we're also finding ways of revitalizing our existing arsenal.

Meanwhile "we're all going to die" gets a lot more clicks than "we're working on it and making tangible progress"...

So how did my friend lose all that weight/feel all that better etc?

Hello - you probably got referred to this page by a friend that seemingly without effort suddenly dropped a lot of weight and perhaps can’t stop talking about how much better he or she now feels. You might have asked why, and got a lot of enthusiasm, references to giant books and a whole universe of blogs. This page is the short version.

So - the whole world is getting ever fatter, diabetic and demented. These stats are astoundingly scary, especially when we realize we've been bombarded non-stop with advice to eat less, less fat, move more, exercise harder etc. Either we are not listening or the advice is just not working. It turns out that we have been listening, but that the advice was wrong.


Now, I don’t need to convince you that what your friend is doing is working - you've seen it already. But the next step is believing that most current health advice & associated foods are actually harmfully wrong. To convince yourself of that, stare at the graphs a bit more. The two explanations for it are 1) a collective loss of willpower leading to massive overeating 2) the advice and the food are the problem.

Americans with diabetes

So what do your friend and many of us now do? We eat real food. The stuff our bodies were made to eat. Stuff like nuts, meat,fish, vegetables, full fat yogurt, fruit, eggs, butter, beans. Note the stark absence of cereals, bread, pasta, sugary soda, candy, oreos and generally processed food, or as we like to call it “not food”.

We believe that lots of our weight and health issues did not come from a lack of exercise or because we just could not force ourselves to eat less. We believe the issues came from eating the *wrong* things. We believe you can gain health in the gym and lose weight in the kitchen.

The industry narrative is that you need to exercise off any excess food you eat, get vitamins from pills, fortified cereals or giant amounts of vegetables and fruits. Meanwhile, actually eating eggs and meat is then sorta the “bad food”, from which you can recover by eating “heart healthy whole grains” or other food products with logos on them lauding their health effects.

But if you look up the numbers, you find that actually, so called bad foods like butter, eggs, nuts, bacon, liver and fish abound in nutrients we need to thrive. And it has now also been confirmed that the reasons (cholesterol, saturated fat) we were warned not to eat these foods were false. Meanwhile, most grain based or highly processed foods are nutritional wastelands. Margarine is so bad it needs to have vitamins added back to it - and even then it remains a product that comes out of an oil refinery. Eat butter already.

Now, the story above may be hard to swallow. Especially that the hallowed whole grains turn out not to form a healthy diet. So to explain that a bit more - most grain products you are consuming aren't actually anything like whole grains to begin with. Finely milled grains, even if you leave the bran in the final milled product, are virtually identical to sugar. Nobody is denying this anymore. For a long time, it was assumed that the fiber from the bran would sort of magically change the situation, but it doesn't. The fiber sits next to the sugar, it is there in addition to it. Anything finely milled is not a “complex carbohydrate”. It is glucoseSorry. 

We only got in this place because previously both fat and proteins were thought to be bad for people, and simple carbohydrates (‘sugar’) also had a bad rep. So out popped the complex carbohydrates - and they do exist, in vegetables and intact fruit (not fruit juice or concentrate!)

Getting back, your friend lost weight and is feeling a lot better because he or she is getting the nutrients he needs from real food, the kind of food our bodies not only need but know how to deal with. This in turn reliably leads to a spontaneous reduction in hunger and breaking the “three-hour feeding cycle” which requires food at 9AM, noon, snack at 3PM, dinner at 6PM and snacks at 9PM. You think that ever made any kind of biological sense? Actual food powers you throughout the day, without leaving you tired afterward.

So do we have all the answers? Do we know exactly why we got fat and (pre-)diabetic? Was it the (high) fructose (corn syrup)? The sugar in anything? The mutant dwarf grain that is used to make our pasta and bread? The lack of nutrients from highly processed food? The excess of n-6 polyunsaturated fatty acids? Or more out there, lack of sleep, late night overexposure to bright light?

While we all have our theories, your friend and I have found that “just eating the real food” is a plausible solution that works for us, especially if combined with sufficient (quality) sleep.

Great resources to learn more are:

Good luck and most importantly.. enjoy!