tag:blogger.com,1999:blog-62705919205824699052024-03-18T03:12:11.900-07:00bert hubert finally blogsCode, musings and moreUnknownnoreply@blogger.comBlogger72125tag:blogger.com,1999:blog-6270591920582469905.post-83840503514402526702019-05-01T15:06:00.002-07:002019-05-01T15:06:25.770-07:00This blog has moved<div dir="ltr" style="text-align: left;" trbidi="on">
Hi,<br />
<br />
I am no longer publishing new content on platforms-that-might-go-away like Blogspot or Medium. You can find my new stuff on <a href="https://ds9a.nl/">https://ds9a.nl/</a> or more specifically <a href="https://ds9a.nl/articles/">https://ds9a.nl/articles/</a><br />
<br />
Please follow me there!<br />
<br />
Bert</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6270591920582469905.post-73193336613769393242017-03-24T09:38:00.001-07:002017-03-24T09:38:16.097-07:00On Linux vDSO and clock_gettime sometimes being slow<div dir="ltr" style="text-align: left;" trbidi="on">
Like the previous post on this somewhat dormant blog, I want to share an oddity I discovered that no search engine could really find for me - even though once I found what the problem was, it turns out I was by no means the first person to discover this.<br />
<br />
Some system calls that are used extremely frequently in Linux can be speeded up by a mechanism called vDSO: a virtual dynamically linked shared object. In this way, the kernel can publish selected functions that can run straight in userspace. This means a regular program dynamically links in bits of kernel supplied code, which in turn means that there is no overhead to "jump into the kernel" to execute code. All good.<br />
<br />
One way you notice your system call has received the vDSO treatment is that "strace" and friends no longer see it, since there actually is no system call anymore.<br />
<br />
Of specific interest are time related calls, like gettimeofday and clock_gettime. Many programs make a ton of these calls, and little can be done to prevent it. You might want to cache the current time perhaps, but to do so, you'd need to know the time. So quite some code relies on time related system calls being really really fast.<br />
<br />
This explains why the recent discovery that <a href="https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-slower-on-ec2/">the AWS platform does not vDSO gettimeofday</a> was such a big deal.<br />
<br />
Within PowerDNS software (dnsdist), we use clock_gettime() in hopes of getting the kind of timer we want, and also one that is fast and cheap for the kernel to provide. While doing "million QPS" scale benchmarking of dnsdist today, we did a strace to find out what <a href="http://dnsdist.org/">dnsdist</a> was doing, and lo, within there we found millions and millions of system calls to clock_gettime(). Help!<br />
<br />
My first thought was that the platform we were on might perhaps not actually support clock_gettime as vDSO. To figure out what is actually in the kernel supplied vDSO, I used a program called dump-vdso.c that can be found <a href="https://kernel.googlesource.com/pub/scm/linux/kernel/git/luto/misc-tests/+/5655bd41ffedc002af69e3a8d1b0a168c22f2549/dump-vdso.c">strewn across the web</a>. This emits the library on stdout, and we can then run the regular objdump tool on it to get:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ ./dump-vdso > vdso.so</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ objdump -T vdso.so </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">vdso.so: file format elf64-x86-64</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">DYNAMIC SYMBOL TABLE:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000418 l d .rodata<span class="Apple-tab-span" style="white-space: pre;"> </span>0000000000000000 .rodata</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000a20 w DF .text<span class="Apple-tab-span" style="white-space: pre;"> </span>0000000000000305 LINUX_2.6 clock_gettime</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000000 g DO *ABS*<span class="Apple-tab-span" style="white-space: pre;"> </span>0000000000000000 LINUX_2.6 LINUX_2.6</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000d30 g DF .text<span class="Apple-tab-span" style="white-space: pre;"> </span>00000000000001b1 LINUX_2.6 __vdso_gettimeofday</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000f10 g DF .text<span class="Apple-tab-span" style="white-space: pre;"> </span>0000000000000029 LINUX_2.6 __vdso_getcpu</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000d30 w DF .text<span class="Apple-tab-span" style="white-space: pre;"> </span>00000000000001b1 LINUX_2.6 gettimeofday</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000ef0 w DF .text<span class="Apple-tab-span" style="white-space: pre;"> </span>0000000000000015 LINUX_2.6 time</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000f10 w DF .text<span class="Apple-tab-span" style="white-space: pre;"> </span>0000000000000029 LINUX_2.6 getcpu</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000a20 g DF .text<span class="Apple-tab-span" style="white-space: pre;"> </span>0000000000000305 LINUX_2.6 __vdso_clock_gettime</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0000000000000ef0 g DF .text<span class="Apple-tab-span" style="white-space: pre;"> </span>0000000000000015 LINUX_2.6 __vdso_time</span><br />
<br />
From this we see that clock_gettime is in fact in there. So why was it not getting used? I donned the protective gear and the spelunking equipment and entered the caves of glibc, where I found several nested files, each #including a file from a parent directory, in an impressive attempt to abstract out per CPU, per OS and C library logic. I stared at that code for what felt like a long time, but it appeared to check lots of things, to eventually always end up calling __vdso_clock_gettime(). Weird.<br />
<br />
I then headed to __vdso_clock_gettime() in the Linux kernel where things finally became clear. It turns out the vdso code ITSELF will generate an actual system call for many timers you can request. In fact, this happens for all cases except CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_REALTIME_COARSE and CLOCK_MONOTONIC_COARSE (as of Linux 3.13 up to 4.11-rc3).<br />
<br />
So that solved the mystery: the vDSO stuff was working, but it was itself causing an old fashioned system call. Perhaps the other timers are too difficult (or perhaps even impossible) to supply from the userspace context.<br />
<br />
Now that I knew what the problems was, I found lots of other places noting issues with clock_gettime() performance, for example <a href="https://github.com/Netflix/rend/issues/96#issuecomment-245816569">here</a> and <a href="https://bugs.openjdk.java.net/browse/JDK-8006942">there</a>, and other people have written <a href="http://btorpey.github.io/blog/2014/02/18/clock-sources-in-linux/">some harsh words about CLOCK_MONOTONIC_RAW</a> that we attempted to use.<br />
<br />
It is my hope that the next person to run into this will find this blogpost before spending half a day learning about vDSO. Good luck!</div>
Unknownnoreply@blogger.com12tag:blogger.com,1999:blog-6270591920582469905.post-45253384848253180352016-05-19T02:06:00.005-07:002016-05-19T02:10:55.222-07:00Brief note on LuaWrapper and unexpected crashes in destructor & destructor ordering<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
PowerDNS products rely heavily on Lua, and we mostly use the most excellent <a href="https://github.com/ahupowerdns/luawrapper">LuaWrapper</a> to seamlessly share data and code between Lua and C++. LuaWrapper is such a fundamental part of our products that we took over maintenance of LuaWrapper when the genius original author Pierre Krieger moved on to other things.<br />
<br />
Yesterday we traced a weird crash when we fixed a memory leak in the PowerDNS Recursor, and I think I got lucky in finding the cause quite quickly. This could have taken days. My usual technique of searching the web for other people with similar crashes failed.<br />
<br />
So this post is here to create something for your search engine to find. If you destroy the LuaWrapper object and afterwards see crashes in <b>lua_pushlightuserdata()</b>, what is going on is that any objects you copied from Lua are being destroyed *after* the LuaWrapper instance itself got destroyed.<br />
<br />
This means that those objects are trying to deregister themselves with a Lua instance.. that is no longer there.<br />
<br />
If you have a struct like this:<br />
<br />
<blockquote class="tr_bq">
struct Somestuff<br />
{<br />
typedef std::function < bool(std::shared_ptr < DNSQuestion > ) > luacall_t; </blockquote>
</div>
<blockquote class="tr_bq">
lfunc_t d_preresolve;</blockquote>
<blockquote class="tr_bq">
LuaContext d_lw;<br />
};</blockquote>
And within your code you did:<br />
<br />
<blockquote class="tr_bq">
d_preresolve = d_lw.readVariable<lfunc_t>("preresolve");</lfunc_t></blockquote>
<br />
You will get a crash because when your Somestuff instance gets detroyed, destructors run in reverse order from bottom to top. So d_lw is gone, and only THEN does preresolve get destroyed, at which point it tries to deregister with a Lua that is no longer there.<br />
<br />
This is all trivially resolved by putting LuaContext at the very top of your object, and everything that depends on it below.<br />
<br />
The C++ standard is explicit about the order in which destructors get called, so this is safe.<br />
<br />
I hope this was helpful!<br />
<br /></div>
Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-6270591920582469905.post-59885768722313233892015-10-23T11:18:00.001-07:002015-10-23T12:52:39.527-07:00How to do fast canonical ordering of domain names<div dir="ltr" style="text-align: left;" trbidi="on">
A small post to document an arcane subject: how to quickly do a comparison of DNS names in canonical order. First, to recap, what is DNS canonical ordering? It is case insensitive, but 8 bit, based on the labels that make up the DNS name in reverse order.<br />
<br />
So, in human order, xxx.powerdns.com and aaa.powerdns.de, bbb.powerdns.net sort like this:<br />
<br />
<ol style="text-align: left;">
<li>aaa.powerdns.de</li>
<li>bbb.powerdns.net</li>
<li>xxx.powerdns.com</li>
</ol>
<div>
But in DNS canonical order, they sort like this:</div>
<div>
<ol style="text-align: left;">
<li>xxx.powerdns.com</li>
<li>aaa.powerdns.de</li>
<li>bbb.powerdns.net</li>
</ol>
<div>
This is because in the canonical order, we look at the 'com', 'de' and 'net' parts first. And only if those are equal, we look at the second-to-last label in the name. <a href="https://tools.ietf.org/html/rfc4034#section-6">RFC 4034 section 6</a> has all the details. DNS ordering is more than an obscure subject: you need to order your records this way to calculate DNSSEC signatures for example. If you get the ordering wrong, your signatures won't match.</div>
</div>
<div>
<br /></div>
<div>
So how can we do the comparison quickly? The naive way is of course to use one of your language primitives to split up a domain name in labels, reverse the order, and do a case insensitive lexicographical comparison on them. That could look like this in C++ 2011:</div>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
auto ours=getRawLabels(), rhsLabels = rhs.getRawLabels();</blockquote>
<blockquote class="tr_bq">
return std::lexicographical_compare(ours.rbegin(), ours.rend(), rhsLabels.rbegin(), rhsLabels.rend(), CIStringCompare());</blockquote>
</blockquote>
While this is easy enough, it is also astoundingly slow since it splits up your domain name and does loads of allocations. Loading a 1.4 million record long zone into a container with canonical ordering this way took 40 seconds. Ordering based on naive case insensitive human compare loaded in 8 seconds.<br />
<br />
Now, DNS names consist of labels with a length, so www.powerdns.com typically gets stored in a packet as the value 3, then "www," the value 8, then "powerdns", the value 3 and them "com". Note there are no dots in there!<br />
<br />
It is highly recommended to also store DNS names as a series of length/label-content pairs, since otherwise you need to do lots of escaping to deal with embedded nulls, embedded . etc.<br />
<br />
When stored like this however, it is not straightforward to do a canonical compare. So I asked around among our open source friends, and <a href="http://marek.vavrusa.com/">Marek Vavrusa</a> of CZNIC quickly chimed in to explain how the excellent Knot nameserver products do it, and it is quite clever. First, you store the domain in reverse label order, so www.powerdns.com would turn into com.powerdns.www, which would normally look like "3com8powerdns3www" in memory.<br />
<br />
However, if you naively compare 3com8powerdns3www with (say) 2de8powerdns3www, you'd decide that based on the '3' versus the '2', that www.powerdns.de would sort before www.powerdns.com, which is wrong.<br />
<br />
So the clever bit is to zero out the label length fields, so you store the names as '0com0powerdns0www' and '0de0powerdns0www'. And then you can simply do a case-insensitive compare and get the right ordering. And if course there is no need to store the leading 0 in this case.<br />
<br />
Now, there is a downside to this arrangement: you lose the information what the domain actually looked like. If there were embedded 0s in the domain name, and there could be, you can't recover the domain name anymore. However, if you don't care, or if you just use this as a key and have a copy of the original domain name somewhere, this works great. Thanks for the explanation Marek!<br />
<br />
PowerDNS uses the most astoundingly great <a href="http://www.boost.org/doc/libs/1_59_0/libs/multi_index/doc/index.html">Boost Multi Index</a> container. I had the great pleasure of meeting its author <a href="http://bannalia.blogspot.com/">Joaquín Mª López Muñoz</a> recently and I keep learning more about what is possible with this wonderful container. But, Boost Multi Index allows us to index objects based on what is in them, without a key that lives separately. So within PowerDNS we like to just use the DNSName that is embedded in an indexed object to sort, and we don't want to 0 out the label lengths in there.<br />
<br />
After some trial and mostly error, I hit on the following rapid ordering procedure for DNS names stored in DNS native format (so: 3www8powerdns3com).<br />
<br />
<ol style="text-align: left;">
<li>Scan through both labels and note the positions of the label boundaries in a stack-based simple array (so no malloc). Store how many labels each DNS name has.</li>
<li>Starting at the last position in your arrays, which denotes the beginning of the last label, do a lexicographical compare starting at one position beyond the length byte and ending "length byte" bytes after that. This for both DNS names</li>
<li>If this comparison leads to 'smaller', your DNS name is definitely smaller. If it leads to 'larger', your DNS name is definitely not smaller and you are done. </li>
<li>Otherwise, proceed one place back in the array of lengths of both names.</li>
<li>If you ended up at position 0 for one name and not yet for the other, that name is smaller and you are done</li>
<li>If you ended up at position 0 for BOTH DNS names, none is smaller than the other</li>
<li>Go to 2 (except don't look at the last label, but at the 'current' position).</li>
</ol>
<div>
To make this safe, either make two arrays sufficiently large that no legal DNS name could overflow it, or use something plausible as a maximum, and fall back to allocating on the heap if your name is long enough to warrant it.</div>
<div>
<br /></div>
<div>
In code, this looks something like <a href="https://github.com/ahupowerdns/pdns/commit/ddb7e6c67244ae644c2acfcf58da55bff4a1ff13#diff-1b92a4c9465bac4bfa3eb165368a554cR106">this</a>. So, the big question of course, is it fast enough? After a little bit of tuning, the canonical comparison function implemented as above is just as fast as the 'naive' human order comparison. Loading a zone again takes 8 seconds. This is faster than you'd expect, but it turns out our canonical comparison function inlines better since the original version secretly used a C library function for case insensitive comparisons. </div>
<div>
<br /></div>
<div>
I hope this has been helpful - either do what Knot does, if you can get away with it, and then it is super fast, or ponder our suggested stack based array solution.</div>
<div>
<br /></div>
<div>
Good luck!</div>
<div>
<br /></div>
<br />
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<br />
<br /></div>
Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-6270591920582469905.post-68475098397311109402015-08-10T07:41:00.000-07:002015-08-10T07:41:00.205-07:00Startups don’t win RFPs: here’s why you might want to do one anyhow<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(I’d like to thank </span><a href="https://uk.linkedin.com/in/dirkpeeters" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Dirk Peeters</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> who taught me most things RFP and </span><a href="https://nl.linkedin.com/in/remcovm" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Remco van Mook</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for commenting on and improving this post.)</span></div>
<b id="docs-internal-guid-0656845b-180a-f329-f5be-8f6d1fb2716e" style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As I find myself in </span><a href="http://bert-hubert.blogspot.com/2010/04/few-notes-on-procurement.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">RFP-land</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> again, I found myself pondering how my previous startups spent tremendous amounts of time working on these </span><a href="https://en.wikipedia.org/wiki/Request_for_proposal" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Requests For Proposals</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from huge customers. Enough so to warrant a blog post that may be helpful for current startups: how to choose between small customers and large customers, like governments and telcos with procurement departments.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, there are of course rare startups that sell straight to consumers, and their game is different. These typically are the startups everyone knows about, because they deal with customers (you) directly. But chances are your startup either sells to businesses, or will need to reach the consumer through established distributors or vendors that embed your stuff into their product. If you plan to sell to consumers directly, this post is not for you.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For context, most startups eventually have a ‘minimum viable product’ (MVP), or at least something that strives to be that. Product/market fit has not been achieved, let alone a perfect match. In other words, customers may be wanting things you have not gotten round to offering, or did not know they wanted. And meanwhile you added lots of stuff that the perceive as excess baggage. Not only is the product not perfect, neither is your knowledge of the market. There may not BE a market yet!</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At this stage, there should be good contact with potential customers already. There are lots of small ones to talk to and far fewer very large ones. Who should you spend your time on?</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Everyone discovers early that lofty revenue/profit goals will not be achieved with smaller customers in a reasonable timeframe. If you need thousands of business engagements to get to where you want to be, a startup-sized salesforce is not going to get you there in a reasonable timeframe. In fact, the salesforce that could make this happen does not want to work with you. Good salespeople work with companies with established products so </span><a href="http://www.bothsidesofthetable.com/2014/04/03/helping-startups-understand-salespeople-the-sales-culture/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">they know they can make quotum</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (READ THIS LINK, by the way).</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So when a potential </span><a href="http://pmarchive.com/guide_to_startups_part5.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">‘whale’</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of a customer comes along, it is tempting to jump on that and give it all you got. And here is where I want to warn you. Large corporations and governments typically ‘tender’ deals. They don’t just pick a vendor, test their stuff, and make the deal. Instead, they write out a </span><a href="http://bert-hubert.blogspot.com/2010/04/few-notes-on-procurement.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">very confusing and conflicting list of requirements</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and instructions, and send that to any interested parties. Which may include you!</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A typical RFP-process includes a spreadsheet filled with hundreds or thousands of numbered requirements, a set of documents outlining the procedure, and a number of questions like ‘outline security architecture of the product’, ‘provide copy of your sustainable sourcing policy’, and ‘describe in detail how the system deals with errors’. Next up might be a clarification meeting, where you can ask questions about the requirements and procedure. You then send in the huge stack of requested documents, after which you might be invited to present your company in person. This is then followed by interminable rounds of negotiations, references, proof of concept sessions etc.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, if your product is struggling to find a market (and at the beginning, it WILL be), this sure feels like </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">traction</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">! We’re getting somewhere, we have a potential customer, they have requirements, we can try to meet them, we have to show up for presentations etc. It almost feels like the real thing!</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In my startups, I have wasted MONTHS on these processes. </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Turns out however, startups don’t win RFPs</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Not until the word ‘startup’ starts feel wrong for your (by now) serious company.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img alt="rfp.jpg" height="351px;" src="https://lh5.googleusercontent.com/zy67XDrzpgv4nwzHvTcj47npC6yY0_Yh2oFs9uLWQm7lyoagHa6YfUJyKCsNrV02c_upKEOJRee_y1QWR_HiLmdJEGGhQBo45_iJd8tYsbq0AEALAUCRbGGpKeGE7f0SwTwEbigE" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="698px;" /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Then, there is this</span></div>
<b style="font-weight: normal;"><br /></b><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So why don’t you win an RFP as a small startup? For one, there is the kind of company that inflicts the RFP-process on itself. These are not dynamic places. These are not the organisations that </span><a href="http://scott.a16z.com/2011/10/05/looking-bigger/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">want to give a startup a chance</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. That’s why they do an RFP, to make sure nothing is bought where they don’t have it in (credible) writing that the product will do what it promises. You mostly sell to the procurement department, not the actual user. And no matter how fab your product or service, the procurement department sees only risk in your startup. For one they will try to check your financials for the past three years. You have not HAD three years!</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The second reason you don’t win is that an RFP is a compendium of every requirement someone ever voiced in the company. Hundreds of them at least. And this strongly favours incumbent vendors who have had years or decades to add every such feature under the sun, if it makes sense or not. The deck is stacked against you.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This third reason you don’t win an RFP is that is is typically heavily lobbied by existing relationships, making sure that only one vendor qualifies, or that new challengers (you) are immediately disqualified from the process because you don’t have 100 staff, over 10 million in annual revenue or 5 years of profitable business behind you.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In addition, the RFP process is highly depressing:</span></div>
<b style="font-weight: normal;"><br /></b>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Seeing a list of features you don’t have and won’t have anytime soon is painful</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Many requirements are in fact nonsense (‘system MUST be redundant against power failures’ - customer is trying to procure software!) - which makes you wonder about the state of the world</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finding out you didn’t actually have a chance because you are a startup is a blow</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But the siren song of the RFP is still tempting for the business to business startup since it sure feels like progress and traction! </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It may be hard to resist if no other actual sales are going on</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. So, here are some reasons why it might make sense to participate in an RFP anyhow:</span></div>
<b style="font-weight: normal;"><br /></b>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You get a free list of competitor features! Most of them show up as requirements (see the lobbying above)</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In general, the process is great ‘competitive intelligence’ - although this works both ways, your competition often learns about you too!</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The whole process is very educational about how large customers think and operate, something most startup employees have little experience with</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: bold; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Attempting</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to meet the giant list of requirements is a great motivator for your development team, finally something concrete to aim for</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The documents requested in an RFP might come in handy anyhow, like that ‘high-level overview of your architecture’. And with pressure, such documents get written a lot faster</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It gives your salespeople something to do except moan about lack of traction, although the flip side of this is that they waste their time on the RFP and don’t get any actual business done</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Frequently, a whole RFP process fails (very frequently by the way, much more than you’d think), and if you managed to make a great impression, you might get invited to the ‘afterparty’ and do business anyhow</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But always manage the process carefully. Taking part in such a large process can swallow all the time and resources of a small startup, and in the end you might have little to show for it. Be sure to drop out on time when it isn’t working. It’s better to lose quickly than not to win slowly. And in any case don’t neglect the rest of your business as the process goes on! Also, do realize that even if you send in a compliant RFP response, it still only sits in the sales pipeline. It is not a purchase order. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finally, there is the risk that you might actually win! And that is the point where all those ‘FULLY COMPLIANT’s you optimistically put in the spreadsheet come back to haunt you. You don’t get paid until you are actually fully compliant! That and the potential huge size of the deal that could well overwhelm your startup. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So getting back to the beginning of the post, the smaller customers that don’t fit with your lofty long term revenue goals. Well, they are your path to the market. For one, because they themselves are smaller, don’t feel bad about doing business with small companies. In fact, when a small company tries to do business with a huge one, they feel they don’t get the attention they deserve. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Also, because no (formal) procurement department sits in between, if you find a small customer with strange requirements, you can talk to the people with the actual requirement and figure out what they mean, or convince them to drop it. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This does not mean your initial customer should necessarily be tiny. They might even be pretty large, as long as they are still procuring things ‘humanly’, and not by spreadsheet with macros that prevent you from entering explanations (not making this up). Your first goal is to get ANY revenue - it will help you sustain your business or help show (current and future) investors that you really are moving the needle. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once you’ve established yourself through several approachable launching customers, you might start winning RFPs. And it still won’t be fun, but it will get you to your financial targets.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Good luck!</span></div>
<br /><br /><br /><br /><br /></div>
Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-6270591920582469905.post-41663387052272791022015-07-15T12:38:00.000-07:002015-07-15T12:38:53.384-07:00Developing open source: don't listen to the people that want you to live under a bridge<div dir="ltr" style="text-align: left;" trbidi="on">
This post is for open source developers and all other people working on open source, while trying to make a living. You may also get referred to this page if you made an open source developer unhappy with your demands.<br />
<br />
Let's start at the very beginning: authoring, documenting, packaging and supporting any software takes stupendous amounts of time. Quality requires serious, dedicated and sustained effort.<br />
<br />
Secondly, we need to realise that people also need to eat, live somewhere, get health insurance and often eventually raise and support a family. It would also be great if they saved for their retirement.<br />
<br />
Combining these two, doing a non-trivial open source project requires more than 'evening hours and weekend work'. It requires people dedicated to the task. But they also need to make money to live!<br />
<br />
And eventually this collides with some folks' expectations of open source. It turns out you generally can't live on charitable donations, and I'm not even sure if you should - donations come and go, and they may also come with expectations that are contrary to those of your actual users. Most large open source projects will therefore need to make money the traditional way: by actually selling something.<br />
<br />
There are loads of things you can sell. Consulting, support, new features (even open source, people will pay to get the features they need), training, training materials, value added services, perhaps even some non-free software on top of the stuff you give away. <b>But no matter how you do it: if you ask money for things, some people who can't or won't pay get left out in the cold</b>.<br />
<br />
And this frequently leads to anger. People will accuse you of selling out, and this hurts. They may even mention you are stealing from the community. <b>Oddly enough they will also threaten to stop using your software!</b> And all of this because you try to make a living so you can provide this great open source stuff <b>for free</b>.<br />
<br />
So here's my word of encouragement: <b>there is a segment of the open source community that you will never appease. They won't be satisfied until you live under a bridge, sell your body by day so you can code by night. For free.</b><br />
<br />
You won't ever make these people happy. Whatever you do, it will not be free enough, and you should always do more. Their threat to stop using your software should tell you everything you need to know about them. Finally also realise that more often than not, the very people that accuse you of selling out work for horrible companies that would not DREAM of committing anything back to the community!<br />
<br />
So separate out these people that want you to live on the streets from the parts of the community you should be listening to. <b>They will help you guide the complex and challenging landscape of 'making money with open source'</b>. But if you try to make wrong people happy, you'll fail and you and your software will end up badly.<br />
<br />
(On a side note - open source is a community, not just business. There is no need to sell all or even the majority of your work. Some things are just a great idea, and you should add them to your software. Also ponder, users that need other stuff from you might "pay" you in Q&A, documentation work, (performance) testing etc. So don't get me wrong - this post is about making a living, not about asking money for everything!)<br />
<br />
Summarising: making money with open source is ok, because delivering quality for a non-trivial project costs time, and that time can't come after dinner when you are tired from your day job. It should be a real job, and that requires income. Don't feel bad about it and don't try to make the wrong people happy. Do listen to the rest, as there is real tension between open source and making money, and they can guide you.<br />
<br />
Good luck!<br />
<br />
<br />
<br /></div>
Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-6270591920582469905.post-38142573610316546042015-06-06T11:53:00.000-07:002015-06-06T11:53:12.422-07:00Design of DelayPipe, a small class to let you execute millions of things in the near future<div dir="ltr" style="text-align: left;" trbidi="on">
When developing, quite frequently we run into the situation where we want to delay an action a little bit, perhaps for half a second. If your software is not highly concurrent, this is as easy as a call to one of the more granular sleep() calls now available.<br />
<br />
But what if you have a user facing 1000 DNS queries per second, wanting to delay answering all of them by a few hundred milliseconds? We can't spawn 1000 threads per second and just run nanosleep() on them!<br />
<br />
This problem is all the more vexing because many end-users find it very easy to say the words 'just delay the answers a bit!', without realizing that this is in fact not an easy thing to do at high concurrency.<br />
<br />
This week, we saw <a href="https://twitter.com/odintsov_pavel">Pavel Odintsov</a> run into this exact issue, nameservers from a big search engine were flooding him with questions, and Pavel was looking for solutions that would not involve actually blocking the search engine. We suggested adding a delay in answering queries, since this frequently shuts up 'back to back' generators of questions.<br />
<br />
But then we actually had to do it. PowerDNS has a nice office these days, and it takes me 30 minutes of cycling to get there, and I find those 30 minutes are well suited to speccing out interesting solutions to problems.<br />
<br />
What I came up with has now been implemented. It does have downsides, but it is remarkably simple and handles millions of delayed events per second with ease.<br />
<br />
First, we employ a trick I documented back in 2007, namely <a href="http://blog.netherlabs.nl/articles/2007/09/20/reusing-unix-semantics-for-fun-and-profit">using a pipe within the same process to transmit pointers</a>. This gives you 'free' communication between threads, and since you pass pointers over the pipe, there is no need to serialize anything. We use the pipe to send the a tuple containing the event & when it needs to be executed.<br />
<br />
The nice thing too is that pipes guarantee that small writes are atomic. So we can have as many threads as we want put events in the pipe, without any locking on our side, and they will arrive at the worker thread uninterleaved. And no locking (in our code at least) is always a win!<br />
<br />
<i>As an aside, we've been using the pipe trick since 2007 or so, and it has never shown up in any benchmarks (except in a positive fashion). However, people frequently associate pipes with "slow text based command line monstrosities", and assume pipes must be slow. In fact, pipes are core operating system infrastructure, and they are blazing fast.</i><br />
<div>
<br /></div>
Getting back to our 'DelayPipe', the worker thread reads pointers from the pipe, and puts them in a time ordered container. Subsequently, there is the question of how to 'wait until the first event that needs to happen'. And again, we can reuse some POSIX semantics, as follows.<br />
<br />
We don't immediately read from the pipe, but we call poll() on it first with a timeout, and that timeout is equal to the amount of time we must wait until the first event needs to be executed. And if there is no work already queued, we wait infinitely long. For safety, we also check if there is actual work in the queue that is overdue for happening and in that case we don't read from the pipe at all. Once poll() either tells us we have a new event, or a timeout happened, we execute all events that have reached their appointed moment.<br />
<br />
With this simple setup, we get a solution that is thread-safe, since many threads can put things in the pipe simultaneously. We do have only one executor of delayed events, which might be unacceptable if the execution of events is actually slow. However, we could easily spawn more of these threads, and give them their own pipe.<br />
<br />
A potential downside is that this setup malloc()s stuff in a producer thread and free()s them in the worker thread, something that is known to be a heavily locked and potentially slow operation in many mallocs. Benchmarking on recent platforms however has not shown any appreciable overhead because of this.<br />
<br />
Finally, after only a 30 minute trip on my bike, and half a day of coding, we were able to get this up and running:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://pbs.twimg.com/media/CGxEPHhUgAASq_A.png:large" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="150" src="https://pbs.twimg.com/media/CGxEPHhUgAASq_A.png:large" width="320" /></a></div>
<br />
The actual code can be found <a href="https://github.com/ahupowerdns/pdns/blob/delaypipe/pdns/delaypipe.cc">here</a> and <a href="https://github.com/ahupowerdns/pdns/blob/delaypipe/pdns/delaypipe.hh">here</a>. The <a href="https://github.com/ahupowerdns/pdns/blob/delaypipe/pdns/test-delaypipe_hh.cc">testcase</a> shows how it works in practice. I hope this has been useful for you!<br />
<br />
PS: before you cry "you don't know how deep the pipe is!", please do realize we don't actually use the <b>pipe</b> for queueing, we have separate container for that. The pipe is only a nice way to get easy producer/consumer communications between threads with sane semantics.<br />
<br /></div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6270591920582469905.post-17591695853972954332015-03-05T12:47:00.000-08:002015-03-09T15:38:46.461-07:00Some notes on shared_ptr atomicity and sharing configuration state<div dir="ltr" style="text-align: left;" trbidi="on">
At PowerDNS, we've frequently run into this problem: a program has a complicated amount of state and configuration which determines how queries are processed, which happens non-stop. Meanwhile, occasionally we need to change this configuration, while everything is running.<br />
<br />
The naive solution to this problem is to have a state which we access under a read/write lock. The state can in that case only be changed if no thread holds a read lock on it. This has at least two downsides. For one, locks aren't free. Even if they don't involve system calls, atomic operations cause inter-CPU communications and cache evictions. Secondly, if the worker threads hog the read lock (which they may need to do for consistency purposes), we can't guarantee that updates happen in a reasonable timeframe.<br />
<br />
Effectively this means that a change in configuration might take a very long time, while we incur overhead every time we access the configuration, even if it isn't changing.<br />
<br />
A very very tempting solution is to keep the configuration in a <a href="http://en.cppreference.com/w/cpp/memory/shared_ptr">shared_ptr</a>, and that threads access the configuration through this shared_ptr. This would give us unlocked access to a consistent configuration. And, if we read the C++ 2011 standard, it looks like this could work. It talks about how std::shared_ptr is thread safe under various scenarios. Simultaneously, the standard defines <a href="http://en.cppreference.com/w/cpp/memory/shared_ptr/atomic">atomic update functions</a> (<a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3337.pdf">20.7.2.5</a>), which are sadly unimplemented in many modern compilers. This is a hint.<br />
<br />
So here is what one would hope would work:<br />
<blockquote class="tr_bq">
if(!g_config->acl.check(clientIP)) dropPacket();</blockquote>
Where the global g_config would be something like shared_ptr<Config><config>. If the user updates the ACL, we would do this to propagate it:</config><br />
<blockquote class="tr_bq">
auto newConfig = make_shared<Config><config>(*g_config); newConfig->acl=newACL; g_config=newConfig;</config></blockquote>
And we would fervently hope that that the last statement was atomic in nature, so that a user of g_config either gets the old copy, or the new copy, but never anything else. And this would be right at least 999999 out of 1 million cases. And on that other case we crash. I know cause I wrote a testcase for this this afternoon.<br />
<br />
It turns out that internally, a shared_ptr consists of reference counts and the actual object. And sadly, when we assign to a shared_ptr, the reference counts and the object get assigned to separately, sequentially. And a user of g_config above might thus end up with a shared_ptr in an inconsistent state that way.<br />
<br />
By tweaking things a little bit, for example by utilizing swap(), you can increase the success rate of this mode of coding to the point where it fails almost almost never. You could fool yourself you solved the problem. Over at PowerDNS we thought that too, but then suddenly CPUs and compilers change, and it starts breaking again, leading to hard to debug crashes.<br />
<br />
So, to summarise, whatever the C++ 2011 standard may or may not say about shared_ptr, as it stands in 2015, <b>you can't atomically change a shared_ptr instance while someone tries to use it</b>.<br />
<br />
And of course we could add an RW-lock to our every use of g_config, but that would get us back to where we started, with heavy locking on everything we do.<br />
<br />
Now, in general this problem (infrequent updates, non-stop access) is very well known, as is the solution: <a href="http://en.wikipedia.org/wiki/Read-copy-update">Read Copy Update</a>. I'm not a big fan of software patents (to say the least), but I'll lovingly make an exception for RCU. IBM released the patent for use in GPL-licensed software, and unlike most patents, this one doesn't only prohibit other people from doing things, RCU also tells you exactly how to do it well. And RCU is sufficiently non-obvious that you actually need that help to do it well.<br />
<br />
Now, the full glory of RCU may be a bit much, but it turns out we can very easily get most of its benefits:<br />
<br />
<ul style="text-align: left;">
<li>Lock the g_config shared_ptr before changing it (this can be a simple mutex, not even an RW one, although it helps) </li>
<li>Have the threads make a copy of this g_config ten times per second, fully locked. </li>
<li>The threads actually only access this (private) copy</li>
</ul>
<div>
This means that if the configuration is changed, the operational threads will continue with the old configuration for at most 0.1 second. It also means that no matter how staggering the overhead of a lock is, we incur it only ten times per second. Furthermore, since the lock is only held very briefly for a copy, the updates will also happen very quickly.</div>
<div>
<br /></div>
<div>
In this way, we don't rely on unimplemented atomic shared_ptr functions, but we do get all the benefits of almost completely unlocked operations. </div>
<div>
<br />
<span style="color: red;">UPDATE</span>: Many people have pointed out that instead of "10 times per second", do the update if an atomic "generational" global counter no longer matches the local one. But some potential synchronisation issues linger in that case (you might miss a second very rapid change, for example. So while interesting, we do lose simplicity in this case.<br />
<br />
<span style="color: red;">UPDATE</span>: <a href="https://github.com/ahupowerdns/pdns/blob/dnsname/pdns/sholder.hh#L4">https://github.com/ahupowerdns/pdns/blob/dnsname/pdns/sholder.hh#L4</a> has the code for this idea<br />
<br /></div>
<div>
Summarising: don't attempt to rely on potential shared_ptr atomic update behaviour, but infrequently copy it it, but frequently enough that changes in configuration propagate swiftly, but not so frequently that the locking overhead matters.</div>
<div>
<br /></div>
<div>
Enjoy! And if you know about the implementation plans and status of the atomic_load etc family of functions for shared_ptr in the various popular compilers, please let me know!<br />
<br />
<span style="color: red;">UPDATE</span>: Maik Zumstrull found <a href="https://gcc.gnu.org/ml/libstdc++/2013-08/msg00164.html">this thread</a> about the atomic shared_ptr operations in gcc.</div>
<div>
<br /></div>
<br />
<blockquote class="tr_bq">
</blockquote>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6270591920582469905.post-42372386516230781932015-02-13T01:35:00.001-08:002015-02-13T01:36:39.008-08:00Some notes on sendmsg()<div dir="ltr" style="text-align: left;" trbidi="on">
This post is mostly so other people can save themselves <a href="http://blog.powerdns.com/2015/02/12/new-powerdns-employee-the-importance-of-testing-rcs-skipping-3-7-0-world-hosting-days-2015/">the two days of pain</a> PowerDNS just went through.<br />
<br />
When sending or receiving datagrams with metadata, POSIX offers us sendmsg() and recvmsg(). These are complicated calls, but they can do quite magical things. They are the "where does this go" part of the socket API. And a lot went there.<br />
<br />
For example, when you bind to 0.0.0.0 or ::, you receive datagrams sent to any address on the port you bound to. But if you reply, you need to know the right source address because otherwise you might send back a response from another IP address than received the question. recvmsg() and sendmsg() can make this happen for you. We documented how this works <a href="http://blog.powerdns.com/2012/10/08/on-binding-datagram-udp-sockets-to-the-any-addresses/">here</a>.<br />
<br />
So, we learned two important things over the past few days.<br />
<br />
<u>Requesting timestamps</u><br />
To request timestamp information, which is great when you want to plot the *actual* latency of the service you are providing, one uses setsockopt() to set the SO_TIMESTAMP option. This instructs the kernel to deliver packets with a timestamp describing when the packet hit the system. You get this timestamp via recvmsg() by going through the 'control messages' that came with the datagram. <br />
<br />
On Linux, the type of the control message that delivers the timestamp is equal to SO_TIMESTAMP, just like the option we passed to setsockopt(). <b>However, this is a lucky accident.</b> The actual type of the message is <b>SCM</b>_TIMESTAMP. And it only happens to be the case that SO_TIMESTAMP==SCM_TIMESTAMP on Linux. This is not the case on FreeBSD.<br />
<br />
So: to retrieve the timestamp, select the message with type <b>SCM_TIMESTAMP</b>. If you select for SO_TIMESTAMP, you will get no timestamps on FreeBSD.<br />
<br />
<u>Datagrams without control messages</u><br />
Secondly, sendmsg() is not a very well specified system call. Even though <a href="https://tools.ietf.org/html/rfc2292">RFC 2292</a> was written by that master of documentation <a href="https://en.wikipedia.org/wiki/W._Richard_Stevens">Richard Stevens</a>, it does not tell us all the things we need to know. For example, we discovered that if you use sendmsg() to send a packet <b>without control messages</b>, on FreeBSD it is not enough to set the length of the control message buffer to 0 (which suffices on Linux).<br />
<br />
<b>FreeBSD in addition demands that the control message buffer address is 0 too</b>. FreeBSD has a check if the length of the control message buffer is at least 1 control message, unless the address of the control message buffer is 0.<br />
<br />
So: if you use sendmsg() to send a datagram without any control messages, set <b>both</b> msg_control and msg_controllen to 0. This way you are portable.<br />
<br />
We hope the above has been helpful for you.<br />
<br />
<br />
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6270591920582469905.post-36360888683644698802015-01-18T08:19:00.000-08:002015-02-19T11:37:58.887-08:00On C++2011, Quality of Implementation & Continuous Integration<div dir="ltr" style="text-align: left;" trbidi="on">
Over the past years, I've done a <a href="https://github.com/ahuPowerDNS/metronome/">few</a> <a href="https://github.com/beaumontlab/antonie">projects</a> outside the main <a href="https://github.com/PowerDNS/pdns/">PowerDNS</a> tree, and for all of them I've used C++2011. A wonderful mark of how big of an improvement C++2011 is, is how much pain you feel when you return to programming in 'regular' C++.<br />
<br />
Recently, I've started to note that the new powers of C++ can either translate into better productivity (ie 'more functionality added/hour of work') or perhaps more importantly, in higher quality of implementation ('QoI').<br />
<br />
And this drove me to ponder the concept of QoI a bit, as I think it is underrated compared to writing fast (by some measure) and bug free code.<br />
<br />
I recently had to pick a reasonable value as an estimate while writing C++03 code, and I found that my fingers considered it too much work to actually scan three vectors of objects to make a decent estimate. As a result, the code ended up with a hardcoded number which (for now) is reasonable.<br />
<br />
This is not quality of implementation. For example, a low QoI implementation of a generally useful memory allocator functions well for the amount of memory the author used it for - say, 1 gigabyte. Unbeknownst to you, lots of the inner workings are overkill when on an embedded platform, for example an O(N) algorithm that is actually pretty slow for small N. Meanwhile, other parts of the library might scale badly to 2GB of memory arena.<br />
<br />
A high QoI implementation of a generic memory allocator would find ways to scale itself to multiple domains of scale. It would not surprise you over the years as your project evolves. It would adapt.<br />
<br />
We often hear (correctly) 'make it work, make it right, make it fast'. QoI is the part of making it right. It might in turn also make your code fast!<br />
<br />
In my example, C++2011 would've allowed me to scan three different vectors like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">for(const auto& vec : {vecA, vecB, vecC}) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for(const auto& entry : vec)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> totSize += entry.length();</span><br />
<span style="font-family: Courier New, Courier, monospace;">} </span><br />
<br />
Whereas the equivalent in C++03 is something like:<br />
<span style="font-family: Courier New, Courier, monospace;">unsigned int countVec(const vector<dnsresourcerecord>& vec)</dnsresourcerecord></span><br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> int ret=0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for(vector<dnsresourcerecord>::const_iterator iter = vec.begin(); iter!=vec.end(); ++iter)</dnsresourcerecord></span><br />
<span style="font-family: Courier New, Courier, monospace;"> ret += iter->length();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return ret;</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">... lots of code in between ... </span><br />
<span style="font-family: Courier New, Courier, monospace;">totSize = countVec(vecA);</span><br />
<span style="font-family: Courier New, Courier, monospace;">totSize += countVec(vecB);</span><br />
<span style="font-family: Courier New, Courier, monospace;">totSize += countVec(vecC);</span><br />
<br />
You can see how the second variant might not happen ("100kb of entries is a good guess!").<br />
<br />
If for this reason alone, I expect my C++2011 code to not only be more pleasing to read, but also to deliver higher quality of implementation.<br />
<br />
It is therefore that it pains me to report that in 2015, I can't find a Continuous Integration provider taking C++2011 seriously (for free or for money).<br />
<br />
Travis-CI which I otherwise love dearly uses an antiquated version of g++ that doesn't do C++2011 at all. If you modify the platform into installing g++-4.8, you find that the supplied version of Boost predates C++2011 and fails to compile. The deployed version of clang fares better, but can't do threads, and bails out the moment you #include anything thread related.<br />
<br />
Meanwhile, Circle CI does ship with a slightly more recent gcc (but not recent enough), but for some reason uses a version of Ubuntu that can't install 'libboost-all-dev', or even 'libboost-serialization-dev'.<br />
<br />
I spent some hours on it this morning, and I'm sure there are solutions that don't involved compiling Boost for every commit, but I haven't found them yet.<br />
<br />
So pretty please, with sugar on top, could the CI platforms up their game a bit? If the goal of CI is to quickly find bugs and issues, they should surely feel motivated to support a language that offers ways to do this.<br />
<br />
Thanks.<br />
<br /></div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6270591920582469905.post-52107785050439435352014-11-03T07:51:00.000-08:002014-11-03T07:58:21.685-08:00Tin cans, can openers and solar power: explaining the snail’s pace of innovation<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt; text-align: left;">
<span style="font-family: 'Trebuchet MS'; font-size: 13px; line-height: 1.15; white-space: pre-wrap;">Innovation is fascinating. It brings us (by definition) all improvements in technology. Simultaneously, it proceeds at a snail’s pace. An example I love to share is the invention of the tin can, which was a breakthrough in food preservation (there were no refrigerators at the time, and tin cans allowed people to safely transport and store previously perishable goods).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Made practicable in the early 19th century, tin cans were sold bearing notices like “</span><a href="http://en.wikipedia.org/wiki/Can_opener" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Cut round the top near the outer edge with a chisel and hammer</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”. And then.. it took over 50 years for the first home operable tin opener to arrive in 1870 (in the 1850s, </span><a href="http://www.ideafinder.com/history/inventions/canopener.htm" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">an opener was available for use in stores</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, who would open your cans for you). The modern easy to use and safe opener was only </span><a href="http://content.time.com/time/specials/packages/article/0,28804,2023689_2026093_2023780,00.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">invented in 1925</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Until the arrival of the opener we know today, opening tin cans was a dangerous exercise. In fact, it was a common cause of injuries and infections, highly dangerous before the advent of antibiotics. Yet it took decades before the problem was solved, even when the need for a solution stared people in the face!</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This example is historical, and hard to come to grips with - </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">why</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> did no one invent the current can opener earlier? The mind boggles. Here’s a far more recent example and one that is still developing. It may help shed light on why innovation proceeds so slowly. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(It looks like this post is about how much I like solar, and while this might be the case, the point I’m trying to make is not that solar power is great. The point is to elaborate on why innovation operates so slowly, using a currently happening development. In fact, if you find yourself disagreeing with my description of the bright future of renewable energy, ponder where your disagreement is coming from. That very disagreement likely is what this post is about!)</span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Renewable energy</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Renewable energy sources have long been treated with derision. “What about when there’s no wind and no sunlight?”. Industry professionals and engineers alike shared this disdain, because as seen through their eyes, solar power compares very badly with (for example) a coal powered plant. The coal powered plant delivers energy against low prices (perhaps 3-7 eurocents/kWh), and does so continuously within a short few hours of turning it on, which you can do at any time you desire. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you hold this next to solar power, photovoltaics do come off very poorly. They roughly operate in three modes - no output (night), 10% output (overcast) or 80-100% output (direct sunlight). One can try to predict when they’ll deliver this output, but you’ll never get it quite right. And, to add insult to injury, the production cost per kWh currently is a lot higher than coal powered plants! So not only don’t you know if and when you’ll get energy, you have to pay more for it too. A mostly similar story applies to wind power. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The upshot of the intermittent nature of availability is that by the time you’ve built enough solar and wind farms to cover your average needs, you’ll have generated vast overcapacity for when the sun shines and it is windy. Your capital costs meanwhile have also been huge compared to boring old power plants.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In short, from the viewpoint of the seasoned energy professional, solar and wind power makes for very unreliable expensive capacity that generates imbalance between power needs and power production. The professional then points out that energy storage (which could average out the imbalance) is prohibitively expensive, and won’t save the day. Repeat this story for decades and decades, and you are in 2008 or so. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Personally, I was fully on board with this line of thinking. I advocated the development of large fusion or even fission plants as the (remote) future. </span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Then, politics happened</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Then, through various plans that weren’t too well thought out (I think), all of a sudden whole countries started building heavily subsidized wind farms and suddenly cheap roof-top solar panels at a massive clip. Germany comes to mind. The Netherlands and other countries are now following behind, even with dwindled subsidies.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">How come? Well.. the original energy professionals were thinking </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">entirely</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> on the production side (and who can blame them, that’s where they worked!). The reality is that in many countries, no consumer actually pays anywhere </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">near</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 3-7 (euro)cents/kWh for electrical power. Transport costs and heavy taxes increase actual ‘grid costs’ to over 20 eurocents/kWh in most parts of Europe. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And since producing energy you use yourself is untaxed, domestic energy users compare the </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">production costs</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of solar against the </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">delivery and taxed costs</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of grid power. And guess what, since there is a factor 3 to 6 between these numbers, all of a sudden ‘</span><a href="http://en.wikipedia.org/wiki/Grid_parity" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">grid parity</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’ arrived. Unsubsidised solar energy is now cheaper than grid power in many places.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="244px;" src="https://lh4.googleusercontent.com/-g3zQBxP_ampzrFjnf4prJBQLm9EPSPJgHiuNv4rqO4D-8_92hFGRwbdbQGgvhTixI2lEgB1RyMI93dycAtUTtJxRzS6O5y-EeDeiYrUQCf54urYPwceEVQ7M2DcWaBJ0g" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="698px;" /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Google trends for “</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">grid parity</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note how the concept of grid parity, or at least the term, burst on the scene in January 2008 (the Wikipedia page was created in June of that year). Before that date, the old view of solar power as a sort of retarded coal fired plant prevailed. </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Production parity</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> was what mattered. Grid parity, which is what matters for consumers, was not a factor. (By the way, I checked - the flatline of the graph pre-2008 is real, and does not reflect a lack of data).</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Grid parity is when solar is interesting for consumers - production parity is only relevant to power producers.</span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The new reality</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What the staid electricity people thought unacceptable (and perhaps unimaginable) is now the new reality. The centralized generation of power has become an uneconomic and highly ungrateful enterprise to be in. If the sun shines and the wind blows, there is literally no need for the coal fired plant. Effective energy prices now routinely plummet to 0 cents, and have gone negative on occasion. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img alt="aemo-solar-price.jpg" height="427px;" src="https://lh4.googleusercontent.com/Gmoj0LHZDVzRn_iz7P3RJY_1Nx6ufiHYPCvGkiB3KhIe3eK7XAH3UpIIsTXveMqjVu1sa5kPD-Cjzzc91XpNoCeoBTBAdrr1S3YCMvppY0k3dopM8bFsTWsgRx0ndGd5kg" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="679px;" /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note zero pricing around 03:00 and negative prices around 14:20 </span><a href="http://reneweconomy.com.au/wp-content/uploads/2014/07/aemo-solar-price.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://reneweconomy.com.au/wp-content/uploads/2014/07/aemo-solar-price.jpg</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And while this is clearly unsustainable (nobody can generate power this way and stay in business), this reality isn’t going to go away, although some localities have so far succeeded in outlawing (Cyprus) or heavily taxing (Hawaii) grid connected solar panels.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The practical upshot though is that even today, substantially lower amounts of coal and gas are being shoveled into power plants.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Meanwhile, wily entrepreneurs have cottoned on the new reality of highly variable energy prices, and are scheduling their business needs around them. Processes that lose money at 20 cents/kWh make a lot more sense at 3 cents/kWh! This is the future, and attempting to emulate the flat power prices of the past century is not.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In another shocking development, a decade ago, California suffered from brownouts and even rolling blackouts, mostly on hot days when lots of air conditioners were running. Now it turns out that a lot of this can be blamed on </span><a href="http://en.wikipedia.org/wiki/California_electricity_crisis" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">energy fraud by Enron</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, but the problem of peak energy use was real.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="362px;" src="https://lh5.googleusercontent.com/YdyBZOeISbYlrqtCJFT-C4Gt4o4zI-GZNzYt1gaJW7x4kgUXU3-_zDlR9FpOPJns3c1yb9U5Lm8EP9PcdaVChhkT-PqzbJfDaDRBPEbT5C22n-VfF5w5I5AkAes9-QzAEw" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="657px;" /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<a href="http://www.iea.org/publications/freepublications/publication/TechnologyRoadmapSolarPhotovoltaicEnergy_2014edition.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.iea.org/publications/freepublications/publication/TechnologyRoadmapSolarPhotovoltaicEnergy_2014edition.pdf</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Welcome to the new reality - already in 2012 the peak was mostly gone, and in 2014 it will have become a dip. The new challenge is the ramp-up when the sun goes down. All of this because in 2012 photovoltaics met the concept of ‘grid parity’, a concept very rarely discussed before 2008.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Only now that minds have been liberated, new thinking has arrived. One could for example ponder potential mass adoption of electrical cars, cars that will need to be charged before their owners drive off with them, but conceivably could charge ‘on demand’ during the day. They could even </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">deliver</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> power during unexpected peak net-demand periods (!). Nearly free energy and “the internet of things” could make this a reality.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Lots of other things could happen too, for example electrical heating of houses currently heated by gas, electrical (on demand) heating of water reservoirs for residential use (‘peak shaving’), bulk energy storage centrally, regionally or even in homes, etc. It is a whole new world out there. How things will end up is hard to predict, but we can already be sure that everything will be cheaper and a lot friendlier to the environment. And this leaves undiscussed the geopolitical impact of a world far less reliant on oil and gas!</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">All this thinking was previously rejected out of hand, “because it did not act like a power plant we know”.</span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What this means for innovation</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In hindsight, the commonly held belief that solar (or renewable energy in general) will never fly until it performs exactly like a traditional power plant was and is ludicrous. But you still hear it today. A lot. And when you hear it, ponder the decades it took to invent a reliable can opener. Innovation happens at a snail’s pace, and only appears obvious in hindsight. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This inability to see the beyond the present concepts appears to be a feature (bug?) of human nature.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you are personally trying to innovate, read up on the history of previous inventions, and see how they were rejected by experts and professionals. Read on until you get a feel for proper rejection (“the flying car” with present day technology) and </span><a href="http://www.rinkworks.com/said/predictions.shtml" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">failure-of-the-imagination rejection</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (British Post Office comment on phones “no need, we have lots of messenger boys”). </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To innovate, we must free yourselves of dearly held convictions and concepts. Ironically, the people with the most knowledge and thus the best credentials to invent the future tend to cling fastest to previously held concepts. This is the challenge of innovation - using the knowledge out there, but without having it hold us back.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Good luck!</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(Thanks to Nicholas Miell, Tsjoi Tsim and Jan van Kranendonk for editing & constructive criticism - all mistakes remain mine!)</span></div>
<br /></div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6270591920582469905.post-42482201827655371702014-08-19T12:19:00.002-07:002014-08-19T12:36:31.059-07:00The absolute minimum difficulty recipe with maximum impact: authentic chicken soup<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: 'Trebuchet MS'; font-size: 13px; line-height: 1.15; white-space: pre-wrap;">I like cooking, and I like to share that joy. If you can’t cook however, your first efforts are likely to be mediocre, which is not very encouraging. “Why bother?!”.</span></div>
<b id="docs-internal-guid-0ad56077-efb1-3639-3045-26b5daeb0a34" style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In this post, I’ll share something that can barely be called a recipe, that’s how simple it is. BUT! The results are entirely authentic and spectacular. You can take this with you to a home cooked dinner, and your friends will be very happy with your efforts. </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will ask you for the recipe! </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Little will they know how easy it is. Also, it is almost impossible to mess this up. I dare you to try ;-)</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ingredients: 10-20 raw chicken wings (plain or spiced, NOT breaded!), 2-4 onions, a few large carrots (or more smaller ones), salt, pepper. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Required equipment: oven-proof container, oven, pan, stove, strainer or colander. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Time spent in the kitchen: 15 minutes</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Pre-heat oven to 180C (350F). Meanwhile, put the chicken wings in the oven-proof container. If you got plain wings, add some salt and pepper. If you have it, sprinkle some oil over the chicken wings. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Put container with chicken in oven (if at temperature). Then you can wait 25 minutes, you can wait 35 minutes, you can wait 45 minutes, and in every case you’ll have some pretty good chicken wings. If after 25 minutes they look ready to eat, they ARE. Do not eat unless thoroughly hot inside!</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(if you don’t have an oven, you can also fry the chicken wings in a pan on a stove, works just as well, but you need to pay some closer attention to getting them browned all round and well done inside. Will require more oil).</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now for the surprising part. Eat as many of the chicken wings as you feel like.</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> No need to finish them all. This was not your special dinner ;-)</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Next, fill a pan with a few liters (quarts) of water, then throw in all your chicken wings, the ones you ate, the ones you didn’t. Peel onions, cut in large chunks, add to water. Chuck the carrots in there too. Bring water to a boil, turn down heat and leave to simmer for 3, 4, 5, hell 6 hours if you feel like it. If during the boiling you note brown stuff floating on the water, remove that with a spoon. Your house will smell lovely meanwhile. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Afterwards, use colander or strainer to filter out all the bones and pieces of meat. (You can add back the pieces of meat if you want.) Now taste the soup and add salt to taste. This part is not optional, it really needs salt. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">That’s it! You can take the soup with you to a party and heat it there. In a fridge it will last for days.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note: to improve on this soup, add more vegetables (doesn’t really matter which ones), or instead of chicken wings, use an entire chicken. </span></div>
<br /></div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6270591920582469905.post-37800076234271120202014-07-08T05:43:00.000-07:002014-07-08T06:27:31.048-07:00Are the media right and are we all going to die from antibiotic resistance?<div dir="ltr" style="text-align: left;" trbidi="on">
A brief post - I've long been unhappy with the media's reporting on science. It appears that news sites aren't there to disseminate truth and educate the populace, but to sell advertisement and clicks. Took me long enough to find that out.<br />
<br />
So we get lots of news on how we are all going to die because all antibiotics have failed. We even see repeated claims that nobody is working on new antibiotics, that the pipeline for new discoveries is empty, and nothing is on the horizon. I saw so much of this that I actually started to believe it too.<br />
<br />
Remember MRSA and how it would kill us all? Oddly enough it didn't, and while it is a nasty bug, there are now protocols to get a handle on it. You probably missed the article telling you so because "we're all going to die" gets a lot more clicks than "with careful work, MRSA outbreaks can be controlled".<br />
<br />
"We're all going to die" is a potent way to draw in an audience. Simultaneously, scientists are happy to big it up, since it is a real problem, and it deserves more attention than it gets. Don't get me wrong on that! The world of medicine spends way way more on cholesterol lowering (to dubious effect) than on antibiotics (which actually save lives). And a dead patient needs no cholesterol lowering, so they should get on it pronto.<br />
<br />
So, where are we? Bacteria can be divided (roughly) into "gram positive" and "gram negative". On the gram positive front, a whole new class of antibiotics has now been trialed for a few years, and they are called <a href="http://en.wikipedia.org/wiki/Lipoglycopeptide">lipoglycopeptides</a>. The two most recently tested (<a href="http://www.nejm.org/doi/full/10.1056/NEJMoa1310480">dalbavancin</a> and <a href="http://www.nejm.org/doi/full/10.1056/NEJMoa1310422">oritavancin</a>) show excellent effectiveness against MRSA. Interestingly, these new antibiotics require a single dose which does its work for the next weeks, so you can't even forget to take the pills. The great Richard Lehmann discusses these over at the <a href="http://blogs.bmj.com/bmj/2014/06/09/richard-lehmans-journal-review-9-june-2014/">BMJ</a> (and <a href="http://blogs.bmj.com/bmj/2014/05/27/richard-lehmans-journal-review-27-may-2014/">here</a>, even more <a href="http://jama.jamanetwork.com/article.aspx?articleid=1851734">here</a>).<br />
<br />
Gram negative bacteria are different, and have an outer membrane that protects them against many antibiotics in the first place, and the situation there is less hopeful, and currently far more worrying than MRSA. The main worry now are <a href="http://en.wikipedia.org/wiki/Carbapenem_resistant_enterobacteriaceae">carbapenem-resistant Enterobacteriaceae</a> (CRE), equipped with the scarily named <a href="http://en.wikipedia.org/wiki/New_Delhi_metallo-beta-lactamase_1">New Delhi metallo-beta-lactamase</a> (NDM-1) enzyme. These are the <a href="http://en.wikipedia.org/wiki/Super_bug_(bacteria)">superbugs</a> that hit the news a lot.<br />
<br />
What did <b>not </b>hit the news was a <a href="http://www.nature.com/nature/journal/v510/n7506/full/510477a.html">result</a> <a href="http://www.nature.com/nature/journal/v510/n7506/full/nature13445.html">published</a> last week where a whole swath of NDM-1 carrying bacteria was effectively treated (<b>in mice</b>) with regular antibiotics plus <a href="http://en.wikipedia.org/wiki/Aspergillomarasmine_A">a known compound</a> previously considered as a hypertension drug (which sadly failed to lower blood pressure). An infection that with best current treatment killed 100% of test animals now only killed 5%. Of mainstream media, only the Wall Street Journal <a href="http://online.wsj.com/articles/new-weapon-in-fight-against-superbugs-1404175658">reported on it</a>. "It it bleeds, it leads" - and modest but important progress does not, it appears.<br />
<br />
Now, are we all saved? No, not yet. In the end it is a battle of <a href="http://jama.jamanetwork.com/article.aspx?articleid=1851734">bacterial genes against our whits,</a> but our arsenal of methods these days is astounding. There is every reason to be sure we'll keep on winning this battle. The next stage will be winning it cheaply and durably.<br />
<br />
But my main point is - whenever you see the news reporting "<a href="http://blogs.telegraph.co.uk/technology/marthagilltech/100013925/if-david-cameron-really-wants-to-solve-the-antibiotics-crisis-he-needs-to-find-some-cash/">there are no new antibiotics</a>" and that nobody is working on it, think back to this page. Not only are those claims false, a <a href="http://en.wikipedia.org/wiki/Lipoglycopeptide">whole new class</a> of antibiotics are now <a href="http://www.biocentury.com/antibioticsncepipeline.htm">coming out of the lab</a> (53 at last count) and we're also <a href="http://www.ncbi.nlm.nih.gov/pubmed/?term=antibiotic+adjuvant">finding ways</a> of <a href="http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0063158">revitalizing</a> our existing arsenal.<br />
<br />
Meanwhile "we're all going to die" gets a lot more clicks than "we're working on it and making tangible progress"...<br />
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6270591920582469905.post-25437154282313206572014-06-10T03:37:00.001-07:002014-06-10T05:53:37.099-07:00Guns & Bandsaws: how to feel the scale of a technology?<div dir="ltr" style="text-align: left;" trbidi="on">
When we pick technologies (languages, compilers, CPUs, computers, routing platforms, databases etc), we need ones that are able to deal with the size or scale of our problem. Now, what is this scale? And does it matter?<br />
<br />
If you picked it wrong, you will find out. If you pick an operating system, open a million files, and find that the kernel bogs down because it has lots of linear searches over open file descriptors, you know the designers of the operating system weren't thinking of your use case. And perhaps in this specific example, you could work around the issue (you don't actually _need_ a million open files at once), but you'll be guaranteed to run into other linear walks. They just weren't thinking at your scale.<br />
<br />
So what determines the 'natural scale' for which a technology is suited? Mostly, this turns out to be in all the individual parts that break when scaling to larger volumes, larger data rates, larger change rates etc. And to scale up a technology, you need to address all these individual squeaking parts, of which there might be many.<br />
<br />
If you settled on something that scales to 'X', and then your use grows to '10X', you might find yourself being held back by a myriad small things that slow you down. And this will not change quickly - you picked your infrastructure as, well, infrastructure. It is there. It is a given. It will change, but only on multi-year timescales. So it matters, since now you'll be toast for years to come. Pick correctly!<br />
<br />
Now, to drive home this point, and this is the real excuse for this post, here's a fair-use busting quote from the most wonderful <a href="http://en.wikipedia.org/wiki/Cryptonomicon">Cryptonomicon by Neal Stephenson</a>. You should read it. Many times. The first 100 pages are a bit slow, but THEN it delivers. Here is one of the book's heroes, Bobby Shaftoe, brilliantly explaining the concept of a technology's natural scale:<br />
<br />
<blockquote class="tr_bq">
"Now when Bobby Shaftoe had gone through high school, he’d been slotted into a vocational track and ended up taking a lot of shop classes. A certain amount of his time was therefore, naturally, devoted to sawing large pieces of wood or metal into smaller pieces. Numerous saws were available in the shop for that purpose, some better than others. A sawing job that would be just ridiculously hard and lengthy using a hand saw would be accomplished with a power saw. </blockquote>
<blockquote class="tr_bq">
Likewise, certain cuts and materials would cause the smaller power saws to overheat or seize up altogether and therefore called for larger power saws. But even with the biggest power saw in the shop, Bobby Shaftoe always got the sense that he was <b>imposing some kind of stress on the machine</b>. It would slow down when the blade contacted the material, it would vibrate, it would heat up, and if you pushed the material through too fast it would threaten to jam." </blockquote>
<br />
I'd like to focus a bit on this feeling - "imposing stress on the machine". In the computing world, we recognize this. We have an intuitive feeling that our MySQL database will become severely unhappy with a billion rows, but will zoom with 50 million rows, for example. That Bobby Shaftoe talks about this same feeling indicates it might have broader, more universal, technological roots.<br />
<br />
<blockquote class="tr_bq">
"But then one summer he worked in a mill where they had a <a href="http://en.wikipedia.org/wiki/Bandsaw">bandsaw</a>. The bandsaw, its supply of blades, its spare parts, maintenance supplies, special tools and manuals occupied a whole room. It was the only tool he had ever seen with infrastructure. It was the size of a car. The two wheels that drove the blade were giant eight-spoked things that looked to have been salvaged from steam locomotives. Its blades had to be manufactured from long rolls of blade-stuff by unreeling about half a mile of toothed ribbon, cutting it off, and carefully welding the cut ends together into a loop. </blockquote>
<blockquote class="tr_bq">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://upload.wikimedia.org/wikipedia/commons/9/9e/John_Henry_Walker06.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://upload.wikimedia.org/wikipedia/commons/9/9e/John_Henry_Walker06.jpg" height="200" width="147" /></a></div>
When you hit the power switch, nothing would happen for a little while except that a subsonic vibration would slowly rise up out of the earth, as if a freight train were approaching from far away, and finally the blade would begin to move, building speed slowly but inexorably until the teeth disappeared and it became a bolt of pure hellish energy stretched taut between the table and the machinery above it. Anecdotes about accidents involving the bandsaw were told in hushed voices and not usually commingled with other industrial-accident anecdotes. </blockquote>
<blockquote class="tr_bq">
Anyway, the most noteworthy thing about the bandsaw was that you could cut anything with it and not only did it do the job quickly and coolly but <b>it didn’t seem to notice that it was doing anything.</b> It wasn’t even aware that a human being was sliding a great big chunk of stuff through it. It never slowed down. Never heated up."</blockquote>
<br />
And again, we can recognize this feeling. Setup a million routes on your big BGP router? It'll just smile at it. It is doing what it was meant to do. It is not impressed. Reload them all you want. Similarly, do something like that on your Raspberry Pi, and you <b>know</b> you are "imposing on the machine".<br />
<br />
Bobby now expands a bit on the concept:<br />
<br />
<blockquote class="tr_bq">
"In Shaftoe’s post-high-school experience he had found that guns had much in common with saws. Guns could fire bullets all right, but they kicked back and heated up, got dirty, and jammed eventually. They could fire bullets in other words, but<b> it was a big deal for them</b>, it placed a certain amount of stress on them, and they could not take that stress forever."</blockquote>
<br />
The phrase "it was a big deal for them" is one to recognize & compare with your feelings for technology you are considering. Is it a big deal for it to do what you want?<br />
<br />
<blockquote class="tr_bq">
"But the <a href="http://en.wikipedia.org/wiki/Vickers_machine_gun">Vickers</a> in the back of this truck was to other guns as the bandsaw was to other saws. The Vickers was water-cooled. It actually had a f*cking radiator on it. It had infrastructure, just like the bandsaw, and a whole crew of technicians to fuss over it. But once the damn thing was up and running, it could fire continuously for days as long as people kept scurrying up to it with more belts of ammunition. After Private Mikulski opened fire with the Vickers, some of the other Detachment 2702 men, eager to pitch in and do their bit, took potshots at those Germans with their rifles, but doing so made them feel so small and pathetic that they soon gave up and just took cover in the ditch and lit up cigarettes and watched the slow progress of the Vickers’ bullet-stream across the roadblock. </blockquote>
<blockquote class="tr_bq">
Then he ceased firing at last. Shaftoe felt like he should make an entry in a log book"</blockquote>
<br />
Again note the awe the technology inspires, and how it is pitted against the rifles which just operate in a very different class of scale.<br />
<br />
As a case in point, over at <a href="http://www.powerdns.com/">PowerDNS</a> we've been taking <a href="http://symas.com/mdb/">LMDB</a> for a spin. And I can tell you, it is a water cooled Vickers with a radiator. We've thrown everything we've had at it, and it just zooms along.. like it is enjoying the challenge. It chomps on the zones like they aren't even there.<br />
<br />
So, wrapping up this brief post - whenever you select new technology for a project, be it a compiler, a language, a computer, a router, a database - try to feel its scale. And the feelings of 'imposing a strain', 'being a big deal', 'not even noticing the work', described above may well guide you to pick the right technology. Good luck!<br />
<br /></div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6270591920582469905.post-3849583882775880222014-05-12T13:28:00.000-07:002014-05-12T13:33:44.207-07:00Will your startup idea be successful? I don’t know. But here’s a checklist before you approach people who could help you.<div dir="ltr" style="text-align: left;" trbidi="on">
<style type="text/css">P { margin-bottom: 0.08in; }A:link { }</style>
<br />
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">(this post contains
two startup ideas that you might consider doing, if so, please
<a href="http://ds9a.nl/">contact me</a>!) </span>
</div>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Judging startup
ideas. This is notoriously difficult but simultaneously enormously
relevant. It is hard to predict if an idea (or your execution of it)
will be successful. However, it may be possible at an early stage to
determine that it likely won't be. </span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">This post is
intended to be useful to rid your startup idea of easily avoidable
errors <b>before</b> you approach (angel) investors, incubators or
other people that could help you. </span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="font-weight: normal; margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><u>PowerDNS</u></span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">My personal history
in judging startup ideas is shoddy - when PowerDNS was launched late
20th century, I did not realise our idea was doomed by design.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">We wanted to sell
expensive DNS software to a world used to getting it for free with
the operating system. The people we tried to sell it to (system
administrators) understood our sales pitch all too well: Database
driven DNS would save a lot of manpower. Back in 2000 that meant the
people we aimed our marketing at would personally lose their jobs if
they took us up on our offer. And we didn’t know anyone higher in
the food chain. We didn’t get anywhere.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">However, over the
years, my batting average probably improved. PowerDNS is doing very
well, both as open source technology and as a company. My
joint-venture with Fox-IT had sound beginnings (because I learned
from PowerDNS), and achieved the three-year goals we set for it in
only 5.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Recently, I was
asked by the good people of <a href="http://www.yesdelft.nl/">Yes!Delft</a> if I could participate in a
startup event, but I decided I had better things to do, since I
judged the attending startups to be hopeless. <b>And I was rightfully
chided for that - it is incredibly hard to judge a startup idea</b>. </span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><u>New ideas sound
bad</u></span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Almost by
definition, great new ideas sound really bad when you first hear of
them. Because if the idea were so obviously good, it would’ve been
done already! So <i>prima facie</i> plausibility is not a good way to
judge an idea. For example, when launched, Twitter looked like a
truly stupid blogging service (2 sentence blog posts?!). When the
first digital cameras arrived, the only available storage devices
were fragile power hungry hard disks. It made no sense.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">But still we need
ways to distinguish the right kind of bad idea from the truly bad
idea. Are there things that just have to be right, even if we believe
in the concept itself?</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Here’s a
checklist of things to consider:</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ol style="text-align: left;">
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Is the idea
(near-future) implementable?</span></div>
<ul>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">But not so
doable that anyone can do it</span></div>
</li>
<li><span style="font-family: "Trebuchet MS",sans-serif;"><b>If it is
possible, can you explain why it hasn’t been done yet?</b></span></li>
</ul>
</li>
</ol>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">You'd be surprised
how often startups ideas flunk these simple tests. Wishing something
is possible doesn't make it so. The laws of physics (generally) do
not change for you. It is important to separate the two kinds of
“impossibilities”: </span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ul>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">breaking the
speed of sound (hard work,
requires lots of R&D), </span>
</div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">breaking the
speed of light (requires Nobel prizes, whole new laws of physics).</span></div>
</li>
</ul>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Usually you should
prefer 'speed of sound'-hard problems.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ol start="2">
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Do you have a
good mental model of who would buy and/or use your technology</span></div>
<ul>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">How do they
buy things?</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>Would they
buy it from you?</b></span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Is it
conceivable that they’d pay your prices?</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">How much
better do you have to be than what they are currently buying?</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">If they won’t
be paying, do you have a plan to make money anyhow?</span></div>
</li>
</ul>
</li>
</ol>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Note that this item
does not demand a 5-year business plan. No such thing makes sense, least of all for a startup.
The item <b>does</b> ask if you have a <i><span style="font-weight: normal;">working
mental model of your potential customer base</span></i><span style="font-style: normal;"><span style="font-weight: normal;">.
What makes them tick? Does your team have personal experience? Do you
know people who could help you there? Or are you operating on a “of
course they'll buy our stuff, it is cheaper”-grade model? </span></span></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ol start="3" style="text-align: left;">
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Do you have
(people with) special knowledge, capabilities, access that
competitors don’t?</span></div>
<ul>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Technologies</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Skills</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>Execution</b></span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Unique team</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Relations</span></div>
</li>
<li><span style="font-family: "Trebuchet MS",sans-serif;">Intellectual
property rights</span></li>
</ul>
</li>
</ol>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span><span style="font-family: "Trebuchet MS",sans-serif;">
</span><br />
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Companies,
startups, are in perpetual competition. If you don't have some kind
of head start, don't enter the game. Note that you don't have to have
all of these, <b>but never play fair</b>. Make sure you bring
something special to the game. </span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span><span style="font-family: "Trebuchet MS",sans-serif;">
</span><br />
<ol start="4" style="text-align: left;">
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Are you
serving a market that isn’t too crowded already?</span> </div>
<ul>
<li><span style="font-family: Verdana,sans-serif;">Or one that
nobody previously wanted to serve, or thought of serving?</span></li>
</ul>
<span style="font-family: "Trebuchet MS",sans-serif;"></span>
</li>
</ol>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">This one is key
too. Unless you have vast resources, don't try to make a better
iPhone from scratch. It is tempting to target a well known problem,
but the problem with well known problems is that the are, well, well
known. Again, don't play fair in this respect. Tackle something in a
field that is not too crowded. Don't try to out-Google Google! Do
however try to do things other people found <a href="http://en.wikipedia.org/wiki/The_Innovator%27s_Dilemma">beneath their scale</a>, or
too boring. <b>At least don't do what everybody is doing already,
unless you can do it 100 times better. </b></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ol start="5">
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Do you have
access to your market, or can someone easily block you from entering
it?</span></div>
</li>
</ol>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span><span style="font-family: "Trebuchet MS",sans-serif;">
</span><br />
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>When pondering
revolution, don't expect help from the establishment</b>. They've been at
it for decades. Intel kept AMD processors away from everyone by
<a href="http://en.wikipedia.org/wiki/AMD_v._Intel">promising sweet prices only in case of total vendor loyalty.</a>
Microsoft set up its licensing so they got paid per computer shipped,
not per copy of Windows – thus removing any incentive for <a href="http://en.wikipedia.org/wiki/United_States_v._Microsoft_Corp.">shipping anything other than Windows</a>. Don't expect to compete with Amazon if you need AWS to do it.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><u>Does it all have
to be perfect?</u></span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">These things can
compensate each other, and it is a rare startup idea that will score
well on all items of the checklist.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">For example, great
execution can make up for an idea that is not too novel in a
sparingly addressed market. Conversely, a truly great technology can
take off even in the face of adverse markets and lack of relevant
business experience (the launch of Google comes to mind).</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><u>A cautionary
note</u></span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">No battle plan
survives contact with the enemy. And it is a rare startup that
delivers what they originally thought they’d be doing. WhatsApp did
not start as a texting service. American Express was a courier
service which later added credit facilities, and only then dropped
out of the delivery business. Google did not start with advertising.
“The Facebook” launched as a college hangout. </span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>But if you don't
have an initial narrative on how you could succeed, you'll have a
hard time gathering funding, talent and advice! </b></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><u>Some worked
examples.</u></span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">A cell phone
battery that never dies. If we score this idea with the list above: </span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ol>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">yes, it might
be possible (pacemakers used to have <a href="http://en.wikipedia.org/wiki/Radioisotope_thermoelectric_generator">nuclear batteries</a>, and many
decades old spacecraft are still powered by them. Mars Curiosity is
a recent example)</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Everybody who
is sick of short battery life would want to buy it. So everyone.
Possibly cell phone companies too, if they weren’t tied up with
their own technology yet</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>But no, I
don’t actually have the special technology, nor do I know anyone
who does. The nuclear batteries are not compatible with today's
world.</b></span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Everybody is
already trying to serve this market.</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Batteries are
highly patented, so I could easily be blocked even if I had an idea.</span></div>
</li>
</ol>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">So this idea likely
won’t get anywhere. Sorry.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Let’s try another
one. <b>Cheap hearing aids</b>. The most basic hearing aids sell for
500.00 euros a piece at least in large parts of Europe, since that is
what insurance companies reimburse. However, they only reimburse once
every three to five years. If in that time, you lose one, or break
it, you are on your own. Basic hearing aids are very basic, there is
no need for them to cost 500 euros. And the fabled ‘programming’
consists of three knobs. Changing the clock on your microwave is
harder. So, does this idea fly?</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ol>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Yes, the idea
is possible. A quick web search shows that hearing aids, somewhat
different from the ones we know, sell for less than 10 dollars a
piece in other parts of the world. </span>
</div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">I personally
don’t know a lot about retail, but I know enough about hearing &
speech, compliance, the medical world, that I’m sure I can get
these hearing aids acceptable to European markets and regulators. </span>
</div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">I know quite
some (younger) hearing aid users, and they are all unhappy and want
change. </span>
</div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>No one is
doing this since it is easy to make stupendous amounts of money with
regular hearing aids. </b></span>
</div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Siemens,
Phonak and others have lots of patents on advanced hearing aids, but
basic ones have been around for ages and should be public domain.
Traditional hearing aid shops won’t touch it (<b>they are part of
the racket</b>), so we’ll have to work around them. Don't expect
this to be easy.</span></div>
</li>
</ol>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Summing it up, I
guess it is worth it to ponder this idea. Doesn’t mean it would
work, but it would get out of the starting gate. Please contact me if
you want to do it, I have some more ideas on this field. There's a
(little) bit more to it.</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><u>Early stage
startup patterns</u></span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Here are some
common early startup patterns, some of which work:</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span><span style="font-family: "Trebuchet MS",sans-serif;">
</span><br />
<ul>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>Having a
powerful idea, but you need to find people and suppliers for
everything</b>, since your own team (you?) lacks the native ability
to implement the core of the idea. <br /><br />This means you might fail
item 1, because you don’t know. You fail on item 2. And all the
suppliers you need to approach might be beholden to your
competition, so you fail 4 too. This never works. If you need to pay
for all the work (instead of doing it as a company and owning the
result), you are doomed. Gather more talent first, and give them
equity (stock).</span></div>
</li>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>Launching a
‘platform’ for something that would be great</b>
<b>once it was there and everybody was on
it</b>. <br /><br />I recently received a
pitch: “.. will be an online multimedia platform that brings
together four customer groups: biologists, press officers of
research organisations, journalists and the public. We will identify
the supply and demand related to nature information and will
organise and facilitate the efficient, easy and quick search and
exchange of content and contacts via web, apps and social media”.
<br /><br />Once there, it would be great. But the greatness of the idea
does not make it happen - the challenge is how to get there. <b><a href="http://en.wikipedia.org/wiki/Metcalfe%27s_law">Half a platform is no use</a>.</b> The probable
reason it hasn’t been done is that it is very hard to get it off
the ground. So by the metrics above, this fails 4
and 6. It can
be done, it is not too easy, but lots of people would want to do it
already, and everybody who has achieved partial success will try
very hard to block you.</span></div>
</li>
</ul>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ul>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>Great
execution versus sleepy industry</b>. I
keep trying to get people to do this. Traffic lights. They are
stupendously boring, and nobody’s main industry. An afterthought
for an intersection. At least in Europe, most traffic light
installations come with a giant cabinet that is placed somewhere
next to the road. Refrigerator sized. What the f*ck do traffic
lights require that even in 2014 they need controllers that big?
<br /><br />So even without innovating a lot, you could enter this
market simply by being nimble and efficient. You could power 250
intersections from the computing abilities of a cell phone. And once
you have connectivity, add lovely control centers for the city that
manages the intersections. <br /><br />Gather statistics, hand out magic
devices to ambulances that pre-announce their trajectory to the
hospital miles in advance, guaranteeing green lights all round!
<br /><br />There’s no fundamental innovation going on, but it
requires great execution to get this sleepy industry moving! And by
the time the competition wakes up, the only thing they can do is buy
you. Again, if you want to do this, contact me, I have lots more
ideas on traffic lights. I'm like that.</span></div>
</li>
</ul>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ul>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>Relying on
partners as a shortcut </b><b>to</b><b>
scaling</b>. Thing is, actually scaling
an idea is hard work. With notable exceptions, you’ll have to get
people to sell, support and market the stuff around the world. And even if
you have one of the rare ideas that sell themselves, you still need
to build the infrastructure to support your customer base. It is a
common theme for early stage startups to focus on their idea, and
claim to find partners for the rest to speed up their time to
market. I’ve fallen for this one too.<br /><br />So here’s the
problem. Partners that could truly help you, since they are in the
right place and have the capabilities, are likely to be invested in
your competition. Even if not literally, they will ‘think like the
established players’, and not like you. Secondly, to them, you are
really small fry. How much of their attention will they focus on
what to them might end up as 1% of their revenues? Will you get
their best work?<br /><br />So any idea that starts with “we’ll
focus on the core and speed up time to market with partners” needs
more work, like for example, which partners would be interested and
<b>why</b>.
Count on having to do a lot of the heavy lifting yourself. (Note
that it is fine for many places to buy commodity services like
hosting, but this is different from ‘partnering’).</span></div>
</li>
</ul>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<ul>
<li><div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><b>Ideas that
deliver ‘millions’ in savings to ‘billions’ sized companies</b>.
So the thinking goes like this. You have a technology that
objectively saves money, for example by allowing for slightly more
efficient procurement. Say, 5% savings. For large enterprises, that
adds up to millions and millions. If you’d be able to appropriate
20% of those savings, you’d be making good money. Business case
done! I encounter a lot of this thinking among technology students,
and I admit I used to fall for it too. (This
is an example of having a bad mental model of your customer base,
by the way). <br /><br />There
are countless opportunities for increasing efficiency and saving 5%
here and there - if only people would sign up. And there’s the
rub. Your 5% savings are worth their while in absolute terms.
However, unless you make *everything* 5% more efficient over at your
customer, their bottom line (‘profit’) impact will be a lot
smaller. <b>However, the disruption you
cause by getting them to move to your new technology is real</b>.
This means that a company has to decide to move to a startup’s
technology to eventually increase profits by 1% - whereas profits go
up and down by 30% all the time for other reasons. Seen this way, it
is not a compelling case.<br /><br />To overcome this, your startup will
need solid credentials and a great network. As a great example, see
“<a href="http://scott.a16z.com/2011/10/05/looking-bigger/">Looking bigger</a>”.</span></div>
</li>
</ul>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><u>Wrapping up</u></span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;">Even good ideas
tend to look bad initially, but as a founder, you’ll love your idea
anyhow. But the idea is not yet a company, and you should try to work
out the obvious kinks before looking for help. A potential investor
or employee that can shoot instant & serious holes in your idea
will not join you. If you show up with a compelling narrative that at
least in theory could work, you might get the help you need!</span></div>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><br /></span>
</div>
<div style="margin-bottom: 0in;">
<span style="font-family: "Trebuchet MS",sans-serif;"><u>Further reading</u></span></div>
<ul style="text-align: left;">
<li><span style="font-family: "Trebuchet MS",sans-serif;">This presentation was held for physics students and touches on similar themes as this post: <a href="http://xs.powerdns.com/exact-sciences-entrepreneur/vvtp-bert-hubert-company.pdf">http://xs.powerdns.com/exact-sciences-entrepreneur/vvtp-bert-hubert-company.pdf</a></span></li>
<li><span style="font-family: "Trebuchet MS",sans-serif;">My favorite links for startups can be found <a href="http://xs.powerdns.com/tmp/pearls.html">here</a>.</span>
</li>
</ul>
<span style="font-family: "Trebuchet MS",sans-serif;">
</span></div>
Unknownnoreply@blogger.com12tag:blogger.com,1999:blog-6270591920582469905.post-49879291794738629402014-04-01T02:25:00.003-07:002014-04-01T04:00:09.949-07:00Seen that sketch, "The Expert"? Well, I blame the expert too.<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">So this post has been brewing for years now, but this short movie triggered me to finally write it up:</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.youtube.com/embed/BKorP55Aqvg?feature=player_embedded' frameborder='0'></iframe></div>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">(Written & Directed by <a href="http://www.imdb.com/name/nm4567815/">Lauris Beinerts</a>, based on a short story "The Meeting" by Alexey Berezi).</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">It has been widely shared within the computing community, and is often described as too painful to watch and frequently as 'hitting too close to home'. It took me several tries until I was able to view it all the way to the end. It is painfully magnificent. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In the sketch, five people meet, two from the customer, and three from the supplier. The customer voices artsy requirements for the project, involving red lines drawn in blue ink. Such impossible requirements are hauntingly familiar to anyone in the tech field.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">And, viewed on one level, the film is about the plight of 'the expert' surrounded by fools, finally capitulating into spouting the same nonsense: can I inflate the lines as a cat shaped balloon? Why yes I can.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">And seen in this way, the movie has been widely described as being indicative of people's working lives. I can personally share several stories that while not involving inflatable kittens with transparent lines in red, drawn in blue ink, are just as unlikely. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<blockquote class="tr_bq" style="text-align: justify;">
<span style="background-color: #ffd966;"><i><span style="font-family: Trebuchet MS, sans-serif;">My favorite story involves one of the first 'professionally run' Internet Service Providers in The Netherlands that decided to do a promotion around Valentines Day. And in their wisdom, the marketing people had decided not to involve anyone who knew what they were doing, and based it all on a website to be launched on XXX.NL, where XXX is Dutch for three kisses. Of course, XXX.NL was already a hardcore porn site. But everyone involved assumed that the experts could just launch a new site on XXX.NL. In the end, since all the materials had been printed already, XXX.NL had to be rented from the owner at great expense, and marketing still found a way to blame it on the techies. </span></i></span></blockquote>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">As such, the sketch is a great way for us geeks to show the world how we feel about being surrounded by what is often called 'the Damagement', a cheap joke on 'management', which by the way includes everyone not actually touching or creating new technology.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">And in fact, the psyche of the engineer here typically is that they consider themselves the only smart people in the company, and that the rest are bumbling idiots. To be fair, this attitude is typically shared about truly senior management ('Use smaller words and bigger letters, this is for the board').</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In the movie, we see a painfully honest expert (dressed up in short sleeves, with required pen in pocket) being flummoxed by boneheaded stupidity around him. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><b>It is my interpretation however that, we, experts & engineers, are just as much to blame for this happening.</b> It is not just about stupid 'damagement'. But allow me to explain.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In established fields, whereby I mean, things that have been around for centuries, people tend to realize their lack of knowledge. For example, most of us would not attempt to build a house ourselves from parts. In fact, most of us would go one step farther: we wouldn't be qualified to even hire bricklayers etc to do that for us. Instead, we rely of a chain of suppliers to make that happen.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In analogy with the sketch, at the beginning of a building project, customers use artsy terms like 'sense of space and direction', 'challenging the environment'. This stuff we then lay on an actual architect, who is also an artist. They get this artsy stuff and can deal with it.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In turn, the architect draws a very nice vision of the building and goes over it with us until we like it. But then something interesting happens. The architect hands over the nice drawings to a <a href="http://en.wikipedia.org/wiki/Design_engineer">design engineer</a> ('Constructeur' in Dutch, or architectural engineer in other places).</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The design engineer, who does not actually construct the house, checks if the drawing <b>COULD</b> be built. He may come back with 'well, you could do that 12 meter arch, but you'd have to build the house out of solid titanium'. He might instead suggest a nice sturdy pillar to mess with your "sense of space and direction". Thus, design engineer & architect hash it out, and may further negotiate with the customer on what is possible and at what cost. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Once this process has finished, the 'what' of the architect, the 'how' of the design engineer gets turned into an actionable plan by a builder. And thence, actual bricklayers get involved, and get provided charts where to lay their bricks, and what kind etc.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Over the centuries, this chain has developed, and while cost overruns are frequent, as are late deliveries, we generally get buildings right. Short of the <a href="http://www.spiegel.de/international/germany/spiegel-investigation-how-the-new-berlin-airport-project-fell-apart-a-868283.html">Berlin Airport</a>, for example, we rarely hear of building projects that just didn't work. They may be late and expensive, but they get there. IT projects typically go for 3 out of 3: too late, too expensive, don't do what we actually wanted.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Now compare this with the sketch. In theory they got it all right! So we have a customer, two people in between and then an expert. However, both as geeks and management we go about this all wrong.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Programmers will ALWAYS lament that customers never articulate their requirements (or even specifications) properly. In this way, they are operating like bricklayers "tell me where the wall should go, how high it should be, what colour, and we'll get on it". </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><b>Meanwhile, the rest of the world thinks they are talking to all knowing architects. They aren't providing detailed requirements, because they can't. </b></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<blockquote class="tr_bq">
<div style="text-align: justify;">
<span style="background-color: #ffd966; font-family: 'Trebuchet MS', sans-serif;"><i>Ah, the IT architect, we have those! And I've yet to meet one worth his salt. The problem is that the IT architect typically has no recent hands on experience (if he has any at all), but is also held in higher regard than the actual people that have to implement it. In the world of building houses, the design engineer overrules the architect. In the world of IT, the IT architect can just goof off, since he's supposed to know how to do it. If it fails, blame the folks doing the typing or the customer.</i></span></div>
</blockquote>
<br />
<blockquote class="tr_bq">
<span style="font-family: Trebuchet MS, sans-serif;"></span><br />
<div style="background-color: #ffd966; text-align: justify;">
<span style="font-family: Trebuchet MS, sans-serif;"><i>So why don't we have IT design engineers? Well, there are a few, but unlike the building industry, we don't yet have 100 years of experience. Check back in a few decades, and we might have learned how to do it. (Oh, and if you feel I badmouthed you because *you* ARE a good IT architect, I'd venture you are probably one of the few good IT design engineers, which is great!)</i></span></div>
<span style="font-family: Trebuchet MS, sans-serif;">
</span>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><i><br /></i></span></div>
<span style="font-family: Trebuchet MS, sans-serif;">
</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">And verily, we geeks often reinforce the perception that we know everything by frequently reminding everyone of what idiots they are. I personally was made unwelcome in an organization after branding all the non-techies "art school graduates". So is it any wonder that people just lay their bare feelings on us and say 'now you solve it, expert?'</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">But, back to the sketch - in a better world, the expert would still be an expert. But the two people in between would know more than just hand on the question (and even making it worse along the way). </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">They should be bridging the gap 'tell me more about those lines'. Actually draw out the customer, help them articulate their vision, and start the process of figuring out what they really want. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><b>Because the dark secret is, 'the perfect customer' that can tell you exactly what they want doesn't exist. This is for the simple reason that a customer that is that skilled doesn't need any outside help!</b> So in reality, you'll always have to work with them to clarify their needs.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">But, back to the sketch, the two people in the middle should have cushioned all this vagueness, and in private consultations with the expert, have worked on making it 'crisp', something that could be done.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">This would have allowed the expert to be much happier, and not be painted like an idiot. (And, on a side note, even though many claim this sketch describes their working lives, that obviously can't be the case for long. This is not how anyone stays in business!)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">However. The world in which we live is such that the expert is often pitted directly against the 'art school graduates'. Right now, many of us geeks subscribe to the notion that WE are all brilliant, but that management universally sucks (whereby management = anyone that doesn't actually touch or create the technology). And this notion allows us to continue acting like the 'expert', going home frustrated, telling our friends how much everything sucks, and our friends agree.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">But this can't be the truth. This 'management' drives home in fancier cars than ours. And their homes are fancier too. Plus, they are never on call! They must be doing something right! I'm sure their lives must be very empty, knowing nothing but art, but it is not tenable for us geeks to pretend we have it so much better and that everyone else is an idiot!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">So finally getting to the point of my post, as experts and implementors, we should stop complaining about idiotic requirements. Society has for now cast us in the role of architect, design engineer AND bricklayer. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><b>And for better or worse, if we want to get ahead, we should act like it</b>. This may not be easy, but for example, our expert in the sketch started out with stating 'impossible' and informing the customer they are idiots ('your scheme only works if you are colour blind'). This does not aid communication, but it is what honest geeks do: tell you exactly in gory and imposing detail why it can't be done (<b>'.. and even if it COULD be done, you should not be doing this'</b>).</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">At one point, our expert asks the customer to elaborate on what they want, but he's quickly cut off by his colleagues, and this is indeed quite common, as often discussion then continues with FURTHER "impossibles" being raised. However, this is exactly what we as enlightened experts should be doing: talking to the people with the requirements and getting them to elaborate on them.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><b>If people think we are full-blown architects, we should oblige them. </b></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">So, hold off on the 'impossible', or imputing a lack of education. <a href="http://bert-hubert.blogspot.nl/2014/02/how-many-hours-for-multithreading.html">Turn the tables on the customer</a> (or the people with the requirement), and get them to talk. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Don't lament that they are unable to provide decent specifications. And most CERTAINLY hold off on complaining that once you delivered according to bad specifications, it is all their problem for not properly asking what they wanted. That is BRICKLAYER thinking. And guess what, that leads to videos like these.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">(Don't get me wrong on manual laborers - they are important, and I respect them. But don't lay "a sense of space and direction" on them and expect a proper wall that does that).</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Techniques to employ are:</span><br />
<br />
<ul style="text-align: left;">
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Assume they are skilled too at what they do. It may not be true, but it will help you tremendously. If you go into a meeting thinking they'll all be fools, you'll quickly find confirmation, and they'll see that you are not talking them seriously. <b>Just assume they know what they are doing, and they may act like it</b>. Honestly.</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Get people to talk more, draw them out, w</span><span style="font-family: 'Trebuchet MS', sans-serif;">ork on usecases, or 'stories'</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">If they ask for something truly impossible, by all means don't explain them in gory detail why it is not possible. <b>Definitely hold off on the "but you shouldn't be wanting that"</b></span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Non-technical people typically converse one level away from the ground truth. So, "impossible" becomes "exceedingly difficult, possibly a research project", which they will understand as "impossible". In fact, a straight "impossible" is understood as "you are a damn fool". In <a href="http://en.wikipedia.org/wiki/Yes_Minister">some circles this is so extreme</a> that "We're taking your proposal seriously" means exactly not that. So, adjust your own verbiage. Avoid "impossible".</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">If you truly think everyone around you is an idiot, and this may be true, ponder working somewhere else. </span></li>
</ul>
<br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">So, summarizing - the plight of the expert in the sketch is real. But the expert is also making things worse, and with proper technique could have prevented the alienation and having to succumb to making unrealistic promises.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
</div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6270591920582469905.post-47616586425797289552014-02-22T13:42:00.000-08:002014-04-21T08:38:20.814-07:00The C++/Programming books I recommend<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBwgHBgkIBwgKCgkLDRYPDQwMDRsUFRAWIB0iIiAdHx8kKDQsJCYxJx8fLT0tMTU3Ojo6Iys/RD84QzQ5OjcBCgoKDQwNGg8PGjclHyU3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3N//AABEIAKAAeAMBEQACEQEDEQH/xAAbAAACAwEBAQAAAAAAAAAAAAAABAEDBQIGB//EAEgQAAEDAwAEDAMEBQkJAAAAAAECAwQABRESEyExBhQVIjVBUVRzk7LRMmFxgZGUoSNCUrHSByVDVWJywdPhFiSCkqLC4vDx/8QAGgEBAAMBAQEAAAAAAAAAAAAAAAECAwQFBv/EADARAAIBAgQDBgYDAQEAAAAAAAABAgMRBBIhMRNBUVJhcZGh8AUiMoGx4RQjwdFC/9oADAMBAAIRAxEAPwD6/bbfCNuikxI5JZRvaT2D5UAxydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQC9yt8IW6URDjghlf9EnsPyoBi2dGxPBR6RQDVAFAFAI3KaqGEEJBSc6SifgA69m3FUnLLqa0qfEbVzPavri4gkKZQk8XEkoK9uj2DZtP+nbVeI7G8sKlLKnzt9+87dvTrEhTTzKNEBP6VDmUgqxojd17vtT27HEs7MLCqUc0X9ra6b+/HoS3e3FxkOagFaksrASrIw4cY3bx2fSpVTQSwqU8ubtLyIVenUvstFlsqefLKSlwnBG8nZsH+OB10zu6IWGTi3fZX29+9Thm/OOIUTHQFJjiSU63egjOBs2n/AE7aqqra27y08Iov6udtufnsOWu58eUtJbU2tKUqKVb9ucfYQM//ACrxnmMKtHh2d7o0quYhQBQC1z6Nl+Cv0mgItnRsTwUekUBzNcnoWkQo7Dqcc4uvFBB+xJq0VF7sh3Ftfeu4Q/xSv4Ktan2vT9i76Br733CF+LV/BTLT7Xp+yLvoVuLvKyCq2QFYORmUrYf+SmWl2vT9hTktitKbojGLTbUgHI0ZJ2Hy6ZKXX0LurUe7fmIM39CkJDJsBCsaIRc0kEKOzHN25NMlLtehVzk9/wAjImyRAL3ErTxMDS1nHRq9+/OhjfTJT2zehPEqXzc+txY3pMdawWrC2pJ5wNxSkgjt5tOHTXP0DqTe/wCR8puZ0f5ogc3Gj/vG7G7HMqOHS6+g4s+r8y1ty7N/Da4Y2Ac2Sdw6vgqclPtehDlJ7nfGrz/Vsb8Uf4aZYdr0Iu+gcavP9Wx/xX/jTLDtegu+hfBfuDrihNhNMICchSH9PJ7MYFRKMVs7kp3Lbn0bL8FfpNUJItnRsTwUekUA1QBQBQBQHK0hSSDuIxvoDAYisf7VTY2qTqU2yIlKANgAcfxj7hUA8y+bgf5IpAKYYj8kOZ2q0tHQPVjGakHorY3KcuN4bSxGXFXcCHdYTpFJabyAMY/OgPRNKQttK2yFIUMpUDkEUB3QBQBigCgFrn0bL8FfpNARbOjYngo9IoDt9MkqGoebQnrC2ir/ALhQFernd6j/AIc/x0Aaud3mP+HP8dAGrnd5j/hz/HQBq53eo/4dX8dALC2vCY5MS7GEh1sNrcDCslIyQPj6iTj6ntoCrkQclm1/7oYBRqzHMdRSU/s/Hu+VAVq4PhRdJWzl46TmEOjTOMZP6XbsAFAPtsT20JQ3IiJQkAJAjK2DzKA61dx71F/DK/zKANXcu9Rfwyv8ygDV3LvUX8Kr/MoC2OmUknjLzLg6tW0UY+9RoDm59Gy/BX6TQEWzo2J4KPSKAaoAoAoAoDMvd5j2lkKdyt1XwNJ3q9hXRh8NOvK0fMzqVFBanhbjwkukxZxIVHR+wydH899e1SwNGnyu+84p15y5mYZUk5JkP57S4a6eFDovIyzy6j0C/wBzhrymU44n9h4lY/Pd9lYVMFRqbq3gaRrTi9z3Nhv7F2RoY1UhI5zZO8doPXXjYnCToPqup2Uqqn4mzXIbBQBQBQC1z6Nl+Cv0mgItnRsTwUekUA1QBQBQFUl9Edhx504Q2kqUfkKmMXKSiiG7K58ouE96fLckvE6SzkDOxI6gPpX1FGlGlBQieXOWeV2VxGHpkhDEdBW6s4SmrVKkacXKWxEYuTsj1jXAdRb/AE07DnYhvIH57fyry5fFdfljodSwumrPO3i0yLTKDL+FJUMocTuUP8K78PiYV45kYVKbg7MVjSXYkhuQwopcbOUmtakI1IuMtmUUmndH1e3SkToTMlvYl1IVjs+VfL1IOnNxfI9SMs0UxjIqhYrekMsDLzqED+0cVDaW5aMJS+lXCO+3IQVtK0kgkHZjBGwiiaewlFxdmV3Po2X4K/SakqRbOjYngo9IoBqgCgCgMThk6W+D8nBxp6KfvIrrwMb4iJjiHamz5psr6NHmnrP5PmEqmSn1DKkNhKflk7f3CvK+Kz+WMTqwsfmbPdV4x3CFztMW5pbTLSo6BJToqxvrajXnRbcOZSdOM9zIkcHLHGI1xWgk4SC8cq+g3mt38SrLdorDBKf0psZhTGojKIVsYUpKEKWjXr0SUg7SE7VnaeyuGpXlVm5PdnbDCKnH5nptpr67epkyr+Z0RxcCUHgENry2oMICFqIySpQVkY3ZTvFc7q5loz0qeB4VRKrG2rWvzO6V7aK2vgxvg7JW/cVL4qUx34qHW3FNc4LClJWCrJznAIyTs6zU0pXltpYxxtNRpWzXkpNNX5WTTtp4PReBtx06ic+nZovAOj6/Cr9yfvrZK0mefJ5oLu0/1f6WXPo2X4K/3GrGZFs6NieCj0igGqAKAKAwuGjZc4PyMD4ClX/UK68C7YiJjiF/Wz5qN1fRo809b/J6+lMuWwd620qHzwdv768n4pDSMjrwr1aPdV452iVxusO2JQqa6WwskJ5pOcfStaVGdZ2grlJ1Iw+oxk8JLKi5OOiTjWtAaWpXnKT9Owj7q0/gYhS+n1RZ4mDpKN9n38zmRBjTpI4QJuTyGGGtJtTaQCAlSirOkMkbSMbNn2VxVqDhUbno0d+Gxn9HAhBSzPnfnZad/O54q2Sb/fNe7wThxYUSMnUoWoJLikbwgqVnPbjcM1wRdWonwlZI+nr0cDg8scdNzlLVrW19r2VvPdnuuB6uEBgqTwkaZS5sLakY0iOvSA2Z+ldlHi5f7EfOfE1gVVvg27c+n2ua9wIZ1Mg7A2vCz2JVsP2ZwfsrWWmpw01mvHqdXLo2V4K/SasZhbOjYngo9IoBqgCgCgKZcdEqM6w6MocQUn7atGThJSXIhq6sfJp0V2BKdjSBhxs4PYewj5V9RRqxqxUo8zypxcZWZzDmPQpTciOvRcbOQcflU1KcakXGWzIjJxd0ewY4dNaka+E7rMbdBQ0fzryZfCpX+WWh2LFLmjzt7vT14kpccSENtjRQgbcfXtNehhcNGhGy3Zz1arqO5luNrcU3q05VpjA7c7P8a0rSUI5nyIprM8q5+/0fT3bQF8GnLQlejpxlM6fzIxn76+XxLddyb5nsYOr/ABqsKm+Vp+R824K8JV8C3ZlqvEJ45XrMNkZCsAdeMggb68qjW/jtwmj7X4j8OXxZQxOHmtra+P5Nzgu/cZ/CF++3V12LbzpJitSHCAoqxgJSewZ2/OtqLlKo6ktEeX8QhQo4WOEoJSnpmaXTqz2clxU+M4w3GdU26gpK3P0YwRjr535V1t5lax4MI8KalKWq6a/r1LpoULRIDhyvi6tI9p0astjF2vodWzo2J4KPSKkgaoAoAoAoDHv9ij3doFR1chIwh0DOzsPaK6cNip0HpquhlVoqou88NO4OXKGrnRVOI/ba5w9/yr2qWMo1P/VvE4ZUZx5GephadhQsHsKTXUpRezMhuFZbhMUAxEcIP6yhopH2msamJpU/qkaRpzlsj2lk4MsQGlrlYefcQUkjckHeB714mLxjr/KtIndQpcJ5uZqR3pSmUJUwQ6EgLU4QkE9ZGMmuFN2OqUYJ3vp3ErhLfKTIezjaA2kJx9u0/cRRxvuSqqj9K8/dvQtZhMMKK2m0haviXjKj9TvNSopFZVJSVm9PfLYYqSgtc+jZfgr9JoCLZ0bE8FHpFAL3K7sW99Lbz0VtSkFYD0kNkgHaQDvAyNvzoBZ7hFFYVovSLc2dWHMKnJHMOcK3bjg7fkagF0m8JiBBlcVZCxlBXKSNIdoyNoqQWKuK0xuNKbYEfR0tcZKdDR7c7sUAr/tHDKXFGRC0WikOHjiOYVfDnsz1dtAMxLoJqVqhoZfSg6Ki1JQrRPYcUBLNxL6VrZZbcS2ooWUPoISobwewjsoCY89UtlD0aOl5lYylxt9Ckq+hBoCtq8NPKbQyGnFO6WrSmS2SvR+LG3bjroDt648XU2h+OltTmQhK30AqwMnG3bsoDmNd0SjiK2h46AXhqQ2rmncdh3HB20BC7y22046ttKW216C1mQ3hKuwnS2H5UBdbLkzckLcjFKm0nRK0OoWM9Y5pO2gLbn0bL8FfpNARbOjYngo9IoDNuKnUcJ4KmGg6viEnmlejnns9eKA82g3Cw3W4OojR15jNvSG29JaYza3nyShIALmjvI5uedjqFQDW4ER4sNuZBZc1jkVbbaHDjnsatJaUnGzQI0sY2Z0sVIFYqmo3C5UQlCbWuQssJJwgTNBKigdXw6asftFXWNgDHCyS2ZHEwwjjGYrmtKx8HGWwARv35+40AW6WpLl1vU1pLdzbbRFdgtn4SkqKAFH49Mr2KwMggYBBoAsDci03oRZMYtInsBanCpJ1kpHxq2dakkH/AIDQGfYH3LbbG7WyotuXKO1JgkYGgpwAO4H9lR1h/v0BRGgFFvgM2xhPGLXx92G2NmC3ISAgdmkgqR9FGgHnZzV0v9gu0NxLsRcwxo7idoUlUV1xagfmQhP1QaAzeCsN5Vls9yt6NKdDskEpQMAvtlLmm0Se3YR/aSnqzQGtwenRbjwjlLS4hbTjZehZ6znDpAP6w/Rg9YzjrNAb9vksuXW5MNR9W6yW9c4CMOEpyPtAxv8AlvoBm59Gy/BX6TQEWzo2J4KPSKA6XDZXMRLU2C+2koQ51hJ3j7cD7h2UAJhsJlrlpaSJC0BtTn6xSMkD6Akn7T20Biw7fDdusyEYrCGYSGxH1SShSAsEqTpA7sjcMCgKuFDjNutrMViBGehtAvSoymspTHRjTUkD9YEgjt20AveEQrXOjAwIq7WpGtluFJK28LTouaXWkHRJzuA0s82gGJiUS7+7H4tFcl8wtOrQf0aEAK0lYPOwpYwNm1X1oCLyzPaMdyauJLbTJZTHc1BQppS1atX62/RWcHZ1jroC6YuNbrxBYU0w3DiwH30KKMqZCChOEnqGivd8hQCFkeLQDardHYvYkrYITpBCQtIeK9u0gpxnG9aSM9YA6u0GTarap0C3LjQkKfYxGLeoeKtpAC9xC19hHac0Bo2GCIkgiK/bnIyGQyUxWSgt6OChPxqGACrZsxmgHXbJbnmQ0uGzoh4vjA0SlxXxKBG0E5O0dpoBiHBiwgvirCGtPaopG1R7Ses7aALn0bL8FfpNARbOjYngo9IoAk3CHFXoSZTLSyMhLiwk47dtXjTnL6Vcq5JblXLFs/rGL5yfep4FXssZ49TGRcUxrvcZTL9tdakBsN5nJSeak5yMHG0/OnAq9l+QzR6lcUwJr8iRf5VuU6tKGkBqZlOrCdoI2b1lf2YqeBV7L8hmj1KrGtuMIzFwmW5xhiK5F0jMSsrRpDQ0gd50AAr59tOBV7L8hmj1FIkZy2yVyYd3tj64zpREQ9KxrYpQkatatpCgpIwrCshAyMkmo4FXsvyJzLqX3AcozTOMy1sLbbb1LJmBQUtDiVgrOMDITo5AON+3cJ4FXsvyGZdTq5KF1u0V9+RbW4iW1MPN8eBUptSkrVuHa2gYzuKtu4U4FXsvyIzLqVXSI0ze4tysM+3IU2gl1EiZzXFDISDvIBC1jI3HRODjFOBV7L8mM0eoXvTvCi6p+1MgRXGQyZwVpqUUEEnGMABY3H4uqnAq9l+TGaPU24l1jpkgA2yMyrKnFploJUcYGwAff8qcCr2X5DNHqaPK1t/rCJ56fenAq9l+Qzx6l0eXGlaQjSGXtH4tWsKx91UlCUfqViU09ji59Gy/BX6TVSSLZ0bE8FHpFAWuR2XVaTrTaz2qSDUA44lF7sz5YpYBxGJ3ZjyxSwDiMTuzHlilgRxCJ3Vjy0+1LC4cQh91Y8pNLAOIQ+6R/KTSwDk+F3SP5SaWBHJ0Lucfyk+1LAOToXc43lJ9qWBHJsHuUbyU+1LAOTIHco3kp9qkiyLWIsePpahhprS36CAnP3UJOLn0bL8FfpNARbOjYngo9IoBqgCgErtIdjQy6xjT1jaecMjBWAT9xJq9OKlK0ttfwVk2loYsS93BSkKksaKCdqQ0oE807tvaANvaO2t50ILSL9ffIzjUk9xhm6XEFpLzA0gUNv6KSdBZXjO880jb8sjPXijpw1s/Dw/7+SVOXNFrFxlKuS4rreilKiNPQOCBk7/7pb+0q7NkSpxyZl796+hKk72E2L3NcYZW6hDLimC44hSDzVAJIG/bnJOz6dRrV0IJtLXUhTdiHL1c247jvF0q0UJGA2rKVlR3gHdgdR6x27I4NNuyfr796EZ5JXsXcsSyGtBsLcWlSltaBGqwlRwSTvyB1fvqODHW7+/UcR6HfK01SglqOFaYSWs7A5j4xv2HOwZ/MVV0opavrfu6E52alukGVDZfUlSdYnSwpJSR9QdoNZTjlk0aRd1caqpIUAtc+jZfgr9JoCLZ0bE8FHpFANUAUBBGaAw7yi6FyRyeHgdWjVEKTofraWwnOd2PnjqzXRS4Wmf36GU8+tit1u7qCgyp9CFq0PiRpN83405J2ZJGD2DqzROlu7X/AD3ciLTOWxejMy8F6AKRooUAg84gkHOQMbcEHfjfUvg5dPfv7dR89y6Nysh9OuC1MjWAYUnJ0sEZ/unmj5VWXCy/L3e/vuWWcXiR7wdBMlchJGrCiHE4OSdLr7MfmBVpTpK9kuZCUuZakXYoSUpeSvUjWBxSTlzKc6ODuxpUfBv16b7d4+cvsSZ4b/nHXawpG1RGBsHYd+c1Wu6bf9exML8zYAxWBoTQBQC1z6Nl+Cv0mgItnRsTwUekUA1QBQBQHLiw2hS1ZwkEnFAIC9QyFHLvN+IapWR17sUB3ytDwkqcKQrOCUnGw437qAFXaKHA2krWsgEBKCc5Gf3UBwi9QVrCUrcJOQMtKG0fUf8AuKAhF8gOKSlLisqxjmHbk4oDSoAoAoAoBa59Gy/BX6TQC1tnwhbooMuPnUo/pU9g+dAM8oQu+R/NT70AcoQu+R/NT70AcoQu+R/NT70AcoQu+R/NT70BHH4Pe4/mj3oCePwu+R/NT70BHH4Pe4/mj3oCePwu+R/NHvQBx+F3yP5o96ABPgjdLj+an3oA5Qhd8j+an3oA5Qhd8j+an3oA5Qhd8j+an3oBa5T4Zt0oCXHyWV/0qew/OgP/2Q==" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBwgHBgkIBwgKCgkLDRYPDQwMDRsUFRAWIB0iIiAdHx8kKDQsJCYxJx8fLT0tMTU3Ojo6Iys/RD84QzQ5OjcBCgoKDQwNGg8PGjclHyU3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3N//AABEIAKAAeAMBEQACEQEDEQH/xAAbAAACAwEBAQAAAAAAAAAAAAAABAEDBQIGB//EAEgQAAEDAwAEDAMEBQkJAAAAAAECAwQABRESEyExBhQVIjVBUVRzk7LRMmFxgZGUoSNCUrHSByVDVWJywdPhFiSCkqLC4vDx/8QAGgEBAAMBAQEAAAAAAAAAAAAAAAECAwQFBv/EADARAAIBAgQDBgYDAQEAAAAAAAABAgMRBBIhMRNBUVJhcZGh8AUiMoGx4RQjwdFC/9oADAMBAAIRAxEAPwD6/bbfCNuikxI5JZRvaT2D5UAxydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQBydB7nH8pPtQC9yt8IW6URDjghlf9EnsPyoBi2dGxPBR6RQDVAFAFAI3KaqGEEJBSc6SifgA69m3FUnLLqa0qfEbVzPavri4gkKZQk8XEkoK9uj2DZtP+nbVeI7G8sKlLKnzt9+87dvTrEhTTzKNEBP6VDmUgqxojd17vtT27HEs7MLCqUc0X9ra6b+/HoS3e3FxkOagFaksrASrIw4cY3bx2fSpVTQSwqU8ubtLyIVenUvstFlsqefLKSlwnBG8nZsH+OB10zu6IWGTi3fZX29+9Thm/OOIUTHQFJjiSU63egjOBs2n/AE7aqqra27y08Iov6udtufnsOWu58eUtJbU2tKUqKVb9ucfYQM//ACrxnmMKtHh2d7o0quYhQBQC1z6Nl+Cv0mgItnRsTwUekUBzNcnoWkQo7Dqcc4uvFBB+xJq0VF7sh3Ftfeu4Q/xSv4Ktan2vT9i76Br733CF+LV/BTLT7Xp+yLvoVuLvKyCq2QFYORmUrYf+SmWl2vT9hTktitKbojGLTbUgHI0ZJ2Hy6ZKXX0LurUe7fmIM39CkJDJsBCsaIRc0kEKOzHN25NMlLtehVzk9/wAjImyRAL3ErTxMDS1nHRq9+/OhjfTJT2zehPEqXzc+txY3pMdawWrC2pJ5wNxSkgjt5tOHTXP0DqTe/wCR8puZ0f5ogc3Gj/vG7G7HMqOHS6+g4s+r8y1ty7N/Da4Y2Ac2Sdw6vgqclPtehDlJ7nfGrz/Vsb8Uf4aZYdr0Iu+gcavP9Wx/xX/jTLDtegu+hfBfuDrihNhNMICchSH9PJ7MYFRKMVs7kp3Lbn0bL8FfpNUJItnRsTwUekUA1QBQBQBQHK0hSSDuIxvoDAYisf7VTY2qTqU2yIlKANgAcfxj7hUA8y+bgf5IpAKYYj8kOZ2q0tHQPVjGakHorY3KcuN4bSxGXFXcCHdYTpFJabyAMY/OgPRNKQttK2yFIUMpUDkEUB3QBQBigCgFrn0bL8FfpNARbOjYngo9IoDt9MkqGoebQnrC2ir/ALhQFernd6j/AIc/x0Aaud3mP+HP8dAGrnd5j/hz/HQBq53eo/4dX8dALC2vCY5MS7GEh1sNrcDCslIyQPj6iTj6ntoCrkQclm1/7oYBRqzHMdRSU/s/Hu+VAVq4PhRdJWzl46TmEOjTOMZP6XbsAFAPtsT20JQ3IiJQkAJAjK2DzKA61dx71F/DK/zKANXcu9Rfwyv8ygDV3LvUX8Kr/MoC2OmUknjLzLg6tW0UY+9RoDm59Gy/BX6TQEWzo2J4KPSKAaoAoAoAoDMvd5j2lkKdyt1XwNJ3q9hXRh8NOvK0fMzqVFBanhbjwkukxZxIVHR+wydH899e1SwNGnyu+84p15y5mYZUk5JkP57S4a6eFDovIyzy6j0C/wBzhrymU44n9h4lY/Pd9lYVMFRqbq3gaRrTi9z3Nhv7F2RoY1UhI5zZO8doPXXjYnCToPqup2Uqqn4mzXIbBQBQBQC1z6Nl+Cv0mgItnRsTwUekUA1QBQBQFUl9Edhx504Q2kqUfkKmMXKSiiG7K58ouE96fLckvE6SzkDOxI6gPpX1FGlGlBQieXOWeV2VxGHpkhDEdBW6s4SmrVKkacXKWxEYuTsj1jXAdRb/AE07DnYhvIH57fyry5fFdfljodSwumrPO3i0yLTKDL+FJUMocTuUP8K78PiYV45kYVKbg7MVjSXYkhuQwopcbOUmtakI1IuMtmUUmndH1e3SkToTMlvYl1IVjs+VfL1IOnNxfI9SMs0UxjIqhYrekMsDLzqED+0cVDaW5aMJS+lXCO+3IQVtK0kgkHZjBGwiiaewlFxdmV3Po2X4K/SakqRbOjYngo9IoBqgCgCgMThk6W+D8nBxp6KfvIrrwMb4iJjiHamz5psr6NHmnrP5PmEqmSn1DKkNhKflk7f3CvK+Kz+WMTqwsfmbPdV4x3CFztMW5pbTLSo6BJToqxvrajXnRbcOZSdOM9zIkcHLHGI1xWgk4SC8cq+g3mt38SrLdorDBKf0psZhTGojKIVsYUpKEKWjXr0SUg7SE7VnaeyuGpXlVm5PdnbDCKnH5nptpr67epkyr+Z0RxcCUHgENry2oMICFqIySpQVkY3ZTvFc7q5loz0qeB4VRKrG2rWvzO6V7aK2vgxvg7JW/cVL4qUx34qHW3FNc4LClJWCrJznAIyTs6zU0pXltpYxxtNRpWzXkpNNX5WTTtp4PReBtx06ic+nZovAOj6/Cr9yfvrZK0mefJ5oLu0/1f6WXPo2X4K/3GrGZFs6NieCj0igGqAKAKAwuGjZc4PyMD4ClX/UK68C7YiJjiF/Wz5qN1fRo809b/J6+lMuWwd620qHzwdv768n4pDSMjrwr1aPdV452iVxusO2JQqa6WwskJ5pOcfStaVGdZ2grlJ1Iw+oxk8JLKi5OOiTjWtAaWpXnKT9Owj7q0/gYhS+n1RZ4mDpKN9n38zmRBjTpI4QJuTyGGGtJtTaQCAlSirOkMkbSMbNn2VxVqDhUbno0d+Gxn9HAhBSzPnfnZad/O54q2Sb/fNe7wThxYUSMnUoWoJLikbwgqVnPbjcM1wRdWonwlZI+nr0cDg8scdNzlLVrW19r2VvPdnuuB6uEBgqTwkaZS5sLakY0iOvSA2Z+ldlHi5f7EfOfE1gVVvg27c+n2ua9wIZ1Mg7A2vCz2JVsP2ZwfsrWWmpw01mvHqdXLo2V4K/SasZhbOjYngo9IoBqgCgCgKZcdEqM6w6MocQUn7atGThJSXIhq6sfJp0V2BKdjSBhxs4PYewj5V9RRqxqxUo8zypxcZWZzDmPQpTciOvRcbOQcflU1KcakXGWzIjJxd0ewY4dNaka+E7rMbdBQ0fzryZfCpX+WWh2LFLmjzt7vT14kpccSENtjRQgbcfXtNehhcNGhGy3Zz1arqO5luNrcU3q05VpjA7c7P8a0rSUI5nyIprM8q5+/0fT3bQF8GnLQlejpxlM6fzIxn76+XxLddyb5nsYOr/ABqsKm+Vp+R824K8JV8C3ZlqvEJ45XrMNkZCsAdeMggb68qjW/jtwmj7X4j8OXxZQxOHmtra+P5Nzgu/cZ/CF++3V12LbzpJitSHCAoqxgJSewZ2/OtqLlKo6ktEeX8QhQo4WOEoJSnpmaXTqz2clxU+M4w3GdU26gpK3P0YwRjr535V1t5lax4MI8KalKWq6a/r1LpoULRIDhyvi6tI9p0astjF2vodWzo2J4KPSKkgaoAoAoAoDHv9ij3doFR1chIwh0DOzsPaK6cNip0HpquhlVoqou88NO4OXKGrnRVOI/ba5w9/yr2qWMo1P/VvE4ZUZx5GephadhQsHsKTXUpRezMhuFZbhMUAxEcIP6yhopH2msamJpU/qkaRpzlsj2lk4MsQGlrlYefcQUkjckHeB714mLxjr/KtIndQpcJ5uZqR3pSmUJUwQ6EgLU4QkE9ZGMmuFN2OqUYJ3vp3ErhLfKTIezjaA2kJx9u0/cRRxvuSqqj9K8/dvQtZhMMKK2m0haviXjKj9TvNSopFZVJSVm9PfLYYqSgtc+jZfgr9JoCLZ0bE8FHpFAL3K7sW99Lbz0VtSkFYD0kNkgHaQDvAyNvzoBZ7hFFYVovSLc2dWHMKnJHMOcK3bjg7fkagF0m8JiBBlcVZCxlBXKSNIdoyNoqQWKuK0xuNKbYEfR0tcZKdDR7c7sUAr/tHDKXFGRC0WikOHjiOYVfDnsz1dtAMxLoJqVqhoZfSg6Ki1JQrRPYcUBLNxL6VrZZbcS2ooWUPoISobwewjsoCY89UtlD0aOl5lYylxt9Ckq+hBoCtq8NPKbQyGnFO6WrSmS2SvR+LG3bjroDt648XU2h+OltTmQhK30AqwMnG3bsoDmNd0SjiK2h46AXhqQ2rmncdh3HB20BC7y22046ttKW216C1mQ3hKuwnS2H5UBdbLkzckLcjFKm0nRK0OoWM9Y5pO2gLbn0bL8FfpNARbOjYngo9IoDNuKnUcJ4KmGg6viEnmlejnns9eKA82g3Cw3W4OojR15jNvSG29JaYza3nyShIALmjvI5uedjqFQDW4ER4sNuZBZc1jkVbbaHDjnsatJaUnGzQI0sY2Z0sVIFYqmo3C5UQlCbWuQssJJwgTNBKigdXw6asftFXWNgDHCyS2ZHEwwjjGYrmtKx8HGWwARv35+40AW6WpLl1vU1pLdzbbRFdgtn4SkqKAFH49Mr2KwMggYBBoAsDci03oRZMYtInsBanCpJ1kpHxq2dakkH/AIDQGfYH3LbbG7WyotuXKO1JgkYGgpwAO4H9lR1h/v0BRGgFFvgM2xhPGLXx92G2NmC3ISAgdmkgqR9FGgHnZzV0v9gu0NxLsRcwxo7idoUlUV1xagfmQhP1QaAzeCsN5Vls9yt6NKdDskEpQMAvtlLmm0Se3YR/aSnqzQGtwenRbjwjlLS4hbTjZehZ6znDpAP6w/Rg9YzjrNAb9vksuXW5MNR9W6yW9c4CMOEpyPtAxv8AlvoBm59Gy/BX6TQEWzo2J4KPSKA6XDZXMRLU2C+2koQ51hJ3j7cD7h2UAJhsJlrlpaSJC0BtTn6xSMkD6Akn7T20Biw7fDdusyEYrCGYSGxH1SShSAsEqTpA7sjcMCgKuFDjNutrMViBGehtAvSoymspTHRjTUkD9YEgjt20AveEQrXOjAwIq7WpGtluFJK28LTouaXWkHRJzuA0s82gGJiUS7+7H4tFcl8wtOrQf0aEAK0lYPOwpYwNm1X1oCLyzPaMdyauJLbTJZTHc1BQppS1atX62/RWcHZ1jroC6YuNbrxBYU0w3DiwH30KKMqZCChOEnqGivd8hQCFkeLQDardHYvYkrYITpBCQtIeK9u0gpxnG9aSM9YA6u0GTarap0C3LjQkKfYxGLeoeKtpAC9xC19hHac0Bo2GCIkgiK/bnIyGQyUxWSgt6OChPxqGACrZsxmgHXbJbnmQ0uGzoh4vjA0SlxXxKBG0E5O0dpoBiHBiwgvirCGtPaopG1R7Ses7aALn0bL8FfpNARbOjYngo9IoAk3CHFXoSZTLSyMhLiwk47dtXjTnL6Vcq5JblXLFs/rGL5yfep4FXssZ49TGRcUxrvcZTL9tdakBsN5nJSeak5yMHG0/OnAq9l+QzR6lcUwJr8iRf5VuU6tKGkBqZlOrCdoI2b1lf2YqeBV7L8hmj1KrGtuMIzFwmW5xhiK5F0jMSsrRpDQ0gd50AAr59tOBV7L8hmj1FIkZy2yVyYd3tj64zpREQ9KxrYpQkatatpCgpIwrCshAyMkmo4FXsvyJzLqX3AcozTOMy1sLbbb1LJmBQUtDiVgrOMDITo5AON+3cJ4FXsvyGZdTq5KF1u0V9+RbW4iW1MPN8eBUptSkrVuHa2gYzuKtu4U4FXsvyIzLqVXSI0ze4tysM+3IU2gl1EiZzXFDISDvIBC1jI3HRODjFOBV7L8mM0eoXvTvCi6p+1MgRXGQyZwVpqUUEEnGMABY3H4uqnAq9l+TGaPU24l1jpkgA2yMyrKnFploJUcYGwAff8qcCr2X5DNHqaPK1t/rCJ56fenAq9l+Qzx6l0eXGlaQjSGXtH4tWsKx91UlCUfqViU09ji59Gy/BX6TVSSLZ0bE8FHpFAWuR2XVaTrTaz2qSDUA44lF7sz5YpYBxGJ3ZjyxSwDiMTuzHlilgRxCJ3Vjy0+1LC4cQh91Y8pNLAOIQ+6R/KTSwDk+F3SP5SaWBHJ0Lucfyk+1LAOToXc43lJ9qWBHJsHuUbyU+1LAOTIHco3kp9qkiyLWIsePpahhprS36CAnP3UJOLn0bL8FfpNARbOjYngo9IoBqgCgErtIdjQy6xjT1jaecMjBWAT9xJq9OKlK0ttfwVk2loYsS93BSkKksaKCdqQ0oE807tvaANvaO2t50ILSL9ffIzjUk9xhm6XEFpLzA0gUNv6KSdBZXjO880jb8sjPXijpw1s/Dw/7+SVOXNFrFxlKuS4rreilKiNPQOCBk7/7pb+0q7NkSpxyZl796+hKk72E2L3NcYZW6hDLimC44hSDzVAJIG/bnJOz6dRrV0IJtLXUhTdiHL1c247jvF0q0UJGA2rKVlR3gHdgdR6x27I4NNuyfr796EZ5JXsXcsSyGtBsLcWlSltaBGqwlRwSTvyB1fvqODHW7+/UcR6HfK01SglqOFaYSWs7A5j4xv2HOwZ/MVV0opavrfu6E52alukGVDZfUlSdYnSwpJSR9QdoNZTjlk0aRd1caqpIUAtc+jZfgr9JoCLZ0bE8FHpFANUAUBBGaAw7yi6FyRyeHgdWjVEKTofraWwnOd2PnjqzXRS4Wmf36GU8+tit1u7qCgyp9CFq0PiRpN83405J2ZJGD2DqzROlu7X/AD3ciLTOWxejMy8F6AKRooUAg84gkHOQMbcEHfjfUvg5dPfv7dR89y6Nysh9OuC1MjWAYUnJ0sEZ/unmj5VWXCy/L3e/vuWWcXiR7wdBMlchJGrCiHE4OSdLr7MfmBVpTpK9kuZCUuZakXYoSUpeSvUjWBxSTlzKc6ODuxpUfBv16b7d4+cvsSZ4b/nHXawpG1RGBsHYd+c1Wu6bf9exML8zYAxWBoTQBQC1z6Nl+Cv0mgItnRsTwUekUA1QBQBQHLiw2hS1ZwkEnFAIC9QyFHLvN+IapWR17sUB3ytDwkqcKQrOCUnGw437qAFXaKHA2krWsgEBKCc5Gf3UBwi9QVrCUrcJOQMtKG0fUf8AuKAhF8gOKSlLisqxjmHbk4oDSoAoAoAoBa59Gy/BX6TQC1tnwhbooMuPnUo/pU9g+dAM8oQu+R/NT70AcoQu+R/NT70AcoQu+R/NT70AcoQu+R/NT70BHH4Pe4/mj3oCePwu+R/NT70BHH4Pe4/mj3oCePwu+R/NHvQBx+F3yP5o96ABPgjdLj+an3oA5Qhd8j+an3oA5Qhd8j+an3oA5Qhd8j+an3oBa5T4Zt0oCXHyWV/0qew/OgP/2Q==" /></a>People often ask me which C++ and programming books I recommend, perhaps because PowerDNS has a reputation for being "readable C++". This post is intended as my answer to this question, and also contains some broader notes on programming. If you have any additions or favorites I missed, please drop me a note, I intend to keep this page updated as new books come out!<br />
<br />
An initial note: if you want to learn C++, by all means learn C++2011, the newest version. It removes quite a lot of the pain that comes with the power of C++. Also, don't fear books about C++2014, most compilers support it already!<br />
<br />
<h2 style="text-align: left;">
Programming</h2>
<div>
<br /></div>
Important parts of programming are:
<br />
<ul style="text-align: left;">
<li>knowing the syntax of the language, </li>
<li>knowing which features to use and when, </li>
<li>writing readable code</li>
<ul>
<li>the compiler may get it, but will the next human reader?</li>
</ul>
<li>designing programs that make sense on a higher level </li>
</ul>
To learn a language, typically we can head for the book called 'The X Programming Language'. There appears to be a rule that you have to write that book whenever you create a serious language. Learning the language however is like learning how to write cursive. It does not mean you've suddenly become a great author of prose!<br />
<br />
For C++, there are two language books that matter:<br />
<ul style="text-align: left;">
<li><a href="http://en.wikipedia.org/wiki/The_C_Programming_Language">The C Programming Language</a> (Brian W. Kernighan, Denis M. Ritchie, TCPL)<br />This book isn't about C++, but, everything relevant to C is also relevant to C++. For example, all examples in this book compile as valid C++. This is not generally true of C, since C++ is more strict than its predecessor. But most quality C programs compile fine as C++. TCPL is a small book, and is widely hailed as one of the best 'The X Programming Language' books ever written. There is wisdom on almost every page. </li>
<li><a href="http://en.wikipedia.org/wiki/The_C%2B%2B_Programming_Language">The C++ Programming Language</a> (4th edition, Bjarne Stroustrup)<br />An astounding book. It too has wisdom on every page. It also has 1400 pages. A hilarious review by Verity Stob can be found on <a href="http://www.theregister.co.uk/2013/06/17/verity_stob_is_bjarne_again/">The Register</a>. In TCPL we read "C is not a big language, and it is not well served by a big book.". Hence the 1400 pages for C++. Even though the book is far too huge to read from cover to cover (although I think I've now hit almost every page), everyone should have a copy. It does indeed explain all of C++, and it does so pretty well. The book also is strong on C++ idioms that will make your life easier. </li>
</ul>
<div>
I want to reiterate that TC++PL is not a book to learn C++ from. It is a book to keep nearby while you are doing so, however. An extract of TC++PL "<a href="http://www.stroustrup.com/Tour.html">A Tour of C++</a>" is also available separately.</div>
<div>
C++ is not just the language, but also a library. While the entire library is covered in TC++PL, for full details I recommend:</div>
<div>
<ul style="text-align: left;">
<li><a href="http://www.amazon.com/The-Standard-Library-Tutorial-Reference/dp/0321623215/">The Standard C++ Library</a> (Nikolai Josuttis)<br />Like TC++PL, this is a vast book. But it has Everything on the C++ libraries (also known as the Standard Template Library, STL). Get the second edition, which covers C++2011.</li>
</ul>
</div>
If you come from a higher level language like Python, Perl or Java, C and hence C++ can be daunting. Unlike these languages, C/C++ is very close to the actual hardware. This has great benefits in terms of fully exploiting that hardware, but also confronts you with the associated realities. <a href="http://www.chiark.greenend.org.uk/~sgtatham/">Simon Tatham</a> (famous for 1: authoring Putty, 2: <a href="http://www.chiark.greenend.org.uk/~sgtatham/smell.html">lacking the ability to smell</a>. He still uses C, though) has written a wonderful document called <a href="http://www.chiark.greenend.org.uk/~sgtatham/cdescent/">The Descent to C</a>. It may ease your pain and even seasoned C programmers may pick up a thing or two.<br />
<br />
If you've read these three books and links, you'll be able to express yourself well in C++, and make good use of its libraries. However, you've not learned which of the many features to use, nor when. C++ is a powerful language. It has been said that in C, if you make a mistake, <a href="http://www.stroustrup.com/bs_faq.html#really-say-that">you blow off your own foot</a>. C++ is <a href="http://www.eng.uwaterloo.ca/~comp03a/misc/humour/shootfoot.html">then supposed to take your whole leg</a>, and this is true.<br />
<br />
Also, C++ is powerful enough to enable you to continue programming in your old language in C++. "<a href="http://queue.acm.org/detail.cfm?id=1039535">I can write FORTRAN in any language</a>". This will however not help you code better programs!<br />
<br />
<h2 style="text-align: left;">
Making good use of C++</h2>
<div>
<br /></div>
<div>
Here are three books, all by Scott Meyers, that greatly simplify the life of an aspiring C++ programmer:</div>
<div>
<ul style="text-align: left;">
<li><a href="http://www.aristeia.com/books.html">Effective C++</a> - lists common pitfalls, wonderful features and problematic areas to avoid. Most recently updated in 2005.</li>
<li><a href="http://www.aristeia.com/books.html">More Effective C++</a> - more of the same, but getting a bit dated. Still worth your while.</li>
<li><a href="http://www.aristeia.com/books.html">Effective STL</a> - last updated in 2001, and like Effective C++, but then with a stronger focus on the standard library.</li>
</ul>
<div>
If you want to get one of these, get Effective C++. If you want two, get Effective STL too. </div>
</div>
<div>
It is important to note that Scott has mostly completed a new book fully covering C++2014 (which can be seen as a refined version of C++2011. Most current compilers already support C++2014). <a href="http://scottmeyers.blogspot.nl/2014/02/help-me-name-my-book.html">Once this book comes out</a>, get it immediately as the previews already look great. </div>
<div>
<span style="color: red;">UPDATE</span>: Various people have recommended '<a href="http://www.amazon.com/Primer-5th-Edition-Stanley-Lippman/dp/0321714113">C++ Primer</a>' by Stanley Lippman cs. I've since acquired the book, and I recommend it heartily. The most recent edition is updated for C++2011.</div>
<h2 style="text-align: left;">
Writing readable code</h2>
<div>
<br /></div>
<div>
All books mentioned above discuss coding styles, things (not) to do, but it is all rather spread out. Wisdom on how to write readable code be found in:</div>
<div>
<ul style="text-align: left;">
<li><a href="http://www.amazon.com/Practice-Programming-Addison-Wesley-Professional-Computing/dp/020161586X">The Practice of Programming</a> (Brian W. Kernighan, Rob Pike, TPoP) - not specifically a C++ book, but has a a lot to say on how to structure code, when to optimize, when not, how to debug, how to prevent the need for debugging etc. Every office should have a copy.</li>
<li><a href="https://www.kernel.org/doc/Documentation/CodingStyle">Linux Kernel Coding Style</a> (Linus Torvalds) - specifically not about C++, but still has a lot to tell us in chapters 4 ('Naming'), 6 ('Functions'), and 8 ('Commenting').</li>
</ul>
<div>
TPoP should probably be read from cover to cover by every programmer wanting to improve their code.<br />
<br /></div>
<h2 style="text-align: left;">
Designing larger scale code</h2>
</div>
<div>
<br /></div>
<div>
I know of only one book that touches on this, and it has been instrumental in my thinking:</div>
<div>
<ul style="text-align: left;">
<li><a href="http://www.amazon.com/Large-Scale-Software-Design-John-Lakos/dp/0201633620">Large Scale C++ Software Design</a> (John Lakos). Although dated in places, containing advice of use for people building on underpowered machines with very slow storage, this book is where I learned how to <i>decompose</i> complex problems into components that make sense. Of particular note is the material on cyclic dependencies in code. These quickly crop up, and make your components nearly impossible to test, since each component quickly relies on every other component.</li>
</ul>
<div>
Good luck learning C++! </div>
</div>
<div>
<br /></div>
</div>
Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-6270591920582469905.post-75171588024467971052014-02-02T04:58:00.005-08:002014-02-02T12:10:33.965-08:00How many hours for multithreading the server? Or: dealing with overly detailed project planning<div dir="ltr" style="text-align: left;" trbidi="on">
Many developers, me included, dread that moment. Someone sits down with you and wants to know how many "hours" each step of the project will take. And of course the thing is, if you are doing something that has been done many times before, you might be able to provide a detailed estimate. People that build houses work like this, but they still often get it wrong. And typically, what we are doing is new - otherwise people'd be using an off the shelf solution!<br />
<br />
Even very good developers struggle to estimate how long individual steps of a project may take, even if over the years they have developed the ability to give a decent <b>*whole*</b> project estimation. This post is intended for those developers that CAN commit to a final deadline, but balk at the "spreadsheet hours exercise".<br />
<br />
I'm not knocking serious project management - you NEED to know where a project is, you NEED to keep track of dependencies. Good project managers (they know who they are), are your partners in delivering great results. This post is not about them, however. Back to our software developer getting asked for estimates rounded to the closest 15 minutes.<br />
<br />
Everything inside us screams to tell this fool of a project manager "I'll let you know when it is done!". This however runs counter to all their beliefs, so soon your project is split up in a hundred small steps with innocuous names like 'make server multithreaded', 'add coherent content cache', 'implement localization'.<br />
<br />
And now this guy is sitting next to you and asking how much hours the "implementation of ACLs" will take. "Come on, we are both professionals, I don't care what the number is, but you must give me an estimate!"<br />
<br />
Engineers that we are, we take a copy out of Scotty's playbook:<br />
<blockquote class="tr_bq">
<a href="http://imgc.allpostersimages.com/images/P-473-488-90/61/6189/3J41100Z/posters/star-trek-the-original-series-scotty.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://imgc.allpostersimages.com/images/P-473-488-90/61/6189/3J41100Z/posters/star-trek-the-original-series-scotty.jpg" height="150" width="200" /></a>"<b>Geordi La Forge</b>: Yeah, well, I told the Captain I'd have this analysis done in an hour.<br />
<b>Scotty</b>: How long will it really take?<br />
<b>Geordi La Forge</b>: An hour!<br />
<b>Scotty</b>: Oh, you didn't tell him how long it would *really* take, did ya?<br />
<b>Geordi La Forge</b>: Well, of course I did.<br />
<b>Scotty</b>: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker!"</blockquote>
<blockquote class="tr_bq" style="text-align: right;">
<a href="http://www.youtube.com/watch?v=8xRqXYsksFg">http://www.youtube.com/watch?v=8xRqXYsksFg</a></blockquote>
<br />
<br />
So we provide a massively padded estimate. 160 hours for implementing the ACLs. Dilbert also has some engineering overestimation guidance here: <a href="http://dilbert.com/strips/comic/1993-05-02/">http://dilbert.com/strips/comic/1993-05-02/</a><br />
<br />
Soon the project guy adds up all our numbers and comes up with 2 years of work. 'But you told us it could be done in 3 months!', and you probably did. So this doesn't work, and we end up with a compacted and painful schedule that is incredibly detailed and bogus. It rots the soul.<br />
<br />
Sound familiar? While we all know doing project management as outlined above is bunk (even though <a href="http://agilemanifesto.org/">Agile</a> is not all that it is cracked up to be, it is massively more right than "waterfall"), you still get these people asking you for detailed estimates.<br />
<br />
And everything screams in us to tell the guy he's a fool, that things don't work that way, but it still doesn't help. Anger management is a challenge here.<br />
<br />
This stuff has been holding me back for years. A <a href="http://twitter.com/hacktobeer">fellow developer</a> I spoke with last week shares one trick I employ to avoid this mess: finish the project well BEFORE someone with a spreadsheet shows up to ask how long it will take! However, this scales badly, and doesn't allow you to charge serious money for your work ('he finished it before we started!').<br />
<br />
Then, I saw The Light. It might not do me any commercial good to share this trick, but I think it is too good to keep to myself.<br />
<br />
Here goes. WHY is someone asking you for such detailed estimates? Do they really care if it takes you 1 hour to do the ACLs and 120 hours to do the multi-threaded architecture or the other way around? Often the person asking you for estimates doesn't even know what these things are!<br />
<br />
No, a major reason why they are asking this stuff is to give you lots of rope to hang yourself with. For them, it is a big covering of asses exercise. Because if you go faster than you scheduled, it is your issue ('this guy has been overbilling us!'). If you go slower, it is DEFINITELY your issue. If part of the project was easier, but another part was harder, it is still your problem ('he missed the deadline for the layout demo but he mumbled something about the ACLs being ready!').<br />
<br />
<b>Another major reason is that by having YOU be very detailed, it saves THEM from having to make any tough choices</b>. You provide loads of detail, they only have to watch if you are sticking to your prediction. And since you most likely won't, they don't have to live up to THEIR commitments. And for many large organizations, this is key: <b>they have a far harder time committing to things than you! </b><br />
<br />
A final very important reason is that asking for so much detail is a sign of insecurity. They don't know how you do your magic, they hate to rely on it, so they try to quantify it all. This gives folks a semblance of control over your work. And who can blame them for wanting that?<br />
<br />
Summarizing, they have very good reasons to force you to make detailed estimates that you can't possibly deliver on exactly. This works for them.<br />
<br />
Here's what's been working for me lately. If people show up wanting to do project planning, there is common ground: In what order will things happen, and where do we need the other party to deliver something? Work that out together. This builds trust and understanding.<br />
<br />
The customer (or whoever needs your work) must spend time explaining what they want. (In the past, we might've said that they would need to provide detailed specifications and requirements, but the dark secret is, no customer that NEEDS you is in a position to provide specifications that are good enough. If they could do that, they probably could finish the project themselves! So you need serious time from them to together pin down what they want).<br />
<br />
Once you are working, after a while you'll need input, logos and icons from their marketing (for example), and when integrated, an ok on layout, usability etc. They have to commit people to sign off on that. Similarly for integration with their database etc. And eventually, the time comes for testing and rollout. Typically there are many 'wait points' where 'now the other party has to do something'.<br />
<br />
So the common ground first consists of identifying these 'wait points'. And the very first wait point is at the start of the project: provide lots of time with the people that can tell us what they want! (or, if they think they can do it, provide the very detailed specifications and requirements).<br />
<br />
<b><u>And this is where you turn the tables on the people with the spreadsheets.</u></b><br />
<br />
Force THEM to commit to specific times and deliverables! If there are unanswered questions on the project (operating system? virtualized or not? database to use?), block on that. "I can't plan unless I know what the work is!". List ALL the things you need to know.<br />
<br />
If they can't answer that on the spot (and a project manager typically won't be able to), ask them for an exact date when they CAN provide the answers.<br />
<br />
<b>Turn those tables!</b><br />
<br />
Quite quickly you'll find that the customer has as much problems as you do with committing exactly to when they'll have something done! "The children of the cobbler go bare feet".<br />
<br />
Then move on to the other 'hard wait points' where you need them, and see if they can commit to those. Can marketing deliver content 1st of February, can we schedule an ok of the UI on the 25th of February, 9AM? And a go-live meeting on April 2nd? Typically, you'll find that this is hard for them to do.<br />
<br />
If you've done your job right, after a while it will be well established and agreed that committing to specific results at specific times is very hard. Frustration is in the air.<br />
<br />
And now you whip out your gift - under these tired and battered circumstances, allow that it doesn't actually matter that much to you when the UI decision is "shall we put it somewhere in week 12 or 13? Let me know when you pin it down".<br />
<br />
If done well, this "gift" will be accepted with a lot of enthusiasm, and if anyone asks how long the ACLs will take to implement, ask when they'll let you know which database they'll be using. This restores the balance. The upshot is that you now only have a few deadlines left, the moments where THEY have to do something with your project, you have to be ready for them.<br />
<br />
Of course, in reality things will rarely play out exactly as above, but the take home message is: <b>turn the tables on folks asking you for detailed estimates you can't provide. Because in the end, most large organizations have a far harder time committing to things than you do</b>.<br />
<br />
Don't end up supplying padded numbers, just make sure you are asking as much detail from them as they from you. This will level the playing field, allowing you to 'sell' your flexibility in return for them getting off your back in the meantime!<br />
<br />
Finally, I'd like to reiterate that the story above only applies to serious developers that ARE able to commit to actual deadlines like 'UI demo'. If you have a hard time divining the size of a project, turning the tables won't help you! Work on that first - or get your customer to commit to Agile (but not the 'half-assed Agile' as found on <a href="http://www.halfarsedagilemanifesto.org/">http://www.halfarsedagilemanifesto.org/</a> )<br />
<br />
Also, from what I hear of serious project managers, they have little love already for spreadsheets full of micro-deadlines. A good project manager can however be your partner in making sure both you and your customer meet the expectations in delivering a working product!<br />
<br />
Good luck, and let me know if this works for you!<br />
<br /></div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6270591920582469905.post-87577865295481079132013-12-25T04:57:00.000-08:002013-12-25T07:13:16.308-08:00In praise of PPM/PNM: printf-style graphical (debugging) output<div dir="ltr" style="text-align: left;" trbidi="on">
When programming, it is often convenient to emit debugging output, and it is very easy to do so in textual format using printf, syslog, console.log and their friends. And once emitted, there are ample tools to analyse our textual output, or even to graph it (gnuplot comes to mind).<br />
<br />
However, I'd always been stumped by creating debugging.. images. Although most debugging proceeds well via text, some things just require a 2D image. Periodically I'd delve into libpng, or even whole UI libraries, but these always presented me with too many choices to satisfy my 'rapid debugging' urge. I want to quickly get a picture, not choose from many color spaces, calibration curves, compression levels and windowing models.<br />
<br />
Recently, I ran into the awesome story of the <a href="http://fabiensanglard.net/rayTracing_back_of_business_card/">ray tracer demo small enough to fit on a business card</a>, and I tried to understand how he did it. And lo, this most karmic of codes used the venerable PPM format to.. printf an image! Of course I knew of PPM, but for some reason had never considered it as a debugging aid.<br />
<br />
So how does it work? The header (as fully explained <a href="http://netpbm.sourceforge.net/doc/ppm.html">here</a>) goes like this:<br />
<blockquote class="tr_bq">
printf("P6 %d %d %d\n", width, height, maxColor);</blockquote>
maxColor indicates for red, green and blue which numerical value denotes maximum intensity. If this is less than 256 each pixel is stored as 3 bytes. If 256 or higher, each color component requires 2 bytes.<br />
<br />
After this header, simply print pixels one row at a time:<br />
<br />
<blockquote class="tr_bq">
for(int y=0; y < height; ++y)<br />
for(int x=0; x< width; ++x)<br />
printf("%c%c%c", red(x,y), green(x,y), blue(x,y));</blockquote>
And that's all. I've found that most UNIXy image viewers accept PPM files out of the box, and if they don't, pnmtopng or ImageMagicks 'convert' can rapidly make a PNG for you.<br />
<br />
In addition, if you are looking for patterns in your data, Gimp 'adjust levels' or 'adjust curves' is a great way to tease these out of your graph.<br />
<br />
Good luck!<br />
<br />
<br />
<br /><!--256--></div>
Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-6270591920582469905.post-58789164355407911352013-12-09T05:36:00.001-08:002013-12-09T06:07:21.267-08:00Hello, and welcome to the wonderful world of DNA sequencing!<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I tremendously enjoy the television program “</span><a href="http://en.wikipedia.org/wiki/How_It's_Made" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">How it’s made</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”. And even though it is unlikely I’ll ever fabricate a baseball bat, at least I’ve learned how it’s done. This post is intended to similarly transfer that sense of wonder, but not about sports equipment, but about the analysis of the veritable code of life: DNA.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Back in 2002, I became fascinated by DNA, and wrote an article called “DNA as seen through the eyes of a computer programmer”. It can be found at </span><a href="http://ds9a.nl/amazing-dna" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://ds9a.nl/amazing-dna</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Recently, I’ve been able to turn my theoretical enthusiasm into practical work, and as of November, I’ve been working as a guest researcher at the </span><a href="http://bertusbeaumontlab.tudelft.nl/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Beaumontlab</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of the department of </span><a href="http://bn.tudelft.nl/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Bionanoscience at Delft University</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Arriving there, I was overwhelmed by the sheer amount of file formats and </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">stuff</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> involved in doing actual DNA processing. So overwhelmed that I found it easier to </span><a href="http://bertusbeaumontlab.tudelft.nl/software.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">write my own tools</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> than get to grips with the zoo of technologies out there (some excellent, some “research grade”). </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This post recounts my travels, and might perhaps help other folks entering the field afresh. Here goes. It is recommended to read </span><a href="http://ds9a.nl/amazing-dna" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://ds9a.nl/amazing-dna</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> as an introduction first. If you haven’t, please know that DNA is digital, and can be seen as stored in files called chromosomes. Bacteria and many other organisms have only one chromosome, but people for example have 2*23. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A typical bacterial genome is around 5 million DNA characters (or ‘bases’ or ‘nucleotides’), the human genome is approximately 3 billion bases long.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">FASTA Files, Reference Genome</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We store DNA on disk as FASTA files (generally), and we can find reference genomes at </span><a href="http://ftp.ncbi.nlm.nih.gov/genomes/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://ftp.ncbi.nlm.nih.gov/genomes/</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. For reasons I don’t understand, the FASTA file(s) have a .fna file extension. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When we attempt to read the DNA of an organism, we call this ‘sequencing’. It would be grand if we could just read each chromosome linearly, and end up with the organism’s genome, but we can’t.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">While lowly cells are able to perform this stunt with ease (and in minutes or a few hours), we need expensive ($100k+, up to millions) equipment which only delivers small ‘reads’ of DNA, and in no particular order either. Whereas a cell can copy its genome in minutes or hours, as of 2013 a typical sequencing run takes many hours or days.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sequencing DNA can be compared to making lots of copies of a book, and then shredding them all. Next, a sequencer reads all the individual snippets of paper and stores them on disk. Since the shreds are likely to overlap a lot, it is possible to reorder them into the original book. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There are different sequencing technologies, some deliver short reads of high quality, other longer reads of lesser qualities, and older technologies deliver long reads of very high quality, but at tremendous expense. The current (2013) leaders of the field are Illumina, Life Technologies (Ion Torrent) and Pacific Biosciences. Illumina dominates the scene. The Wikipedia offers </span><a href="http://en.wikipedia.org/wiki/DNA_sequencing#Next-generation_methods" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">this useful table</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<img height="273px;" src="https://lh6.googleusercontent.com/0L0xUYxDZt0_5cYnBVS7GBpNKJOts3mu5d3pK9KMRUI4rJ88X3C66rpFiZGXVWPqRyqIaYDyP5JEyc0kp45Th103zuL92BdS1DDxgpMXxmHbqcBAL0f_fsB5hA" width="364px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Illumina MiSeq, 2013</span></div>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></h3>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">FASTQ Files, DNA reads</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There are various proprietary file formats, but the “standard” output is called </span><a href="http://en.wikipedia.org/wiki/FASTQ_format" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">FASTQ</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Each DNA ‘read’ consists of four lines of ASCII text, the first of which has metadata on the read (machine serial number, etc, mostly highly redundant and repetitive). The second line.. is a plus character. Don’t ask me why. The third line are actual DNA nucleotides (‘ACGTGGCAGC..’), and the final line delivers the Q in FASTQ: quality scores. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div style="text-align: center;">
<b style="font-weight: normal;"><img height="273px;" src="https://lh6.googleusercontent.com/qdPpEAlmnFYEfz8vbDq5O_0LuNraiQc6RG6y21B7uTTs2w4o8NMNMCi94cXbgTYp57VDxdxvH0sXVjtgISs5csCsUK12EJmiMI3g6K_OdWzhBsO0iJj0k44VZg" width="364px;" /></b></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is where the FASTQ comes out (there’s also ethernet)</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Error rates vary between reads, locations and sequencing technologies, and are expressed as </span><a href="http://en.wikipedia.org/wiki/Phred_quality_score" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Phred Q scores</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. A Q score of 30 means that the sequencer thinks there is one chance in 1000 that it ‘miss-called’ the base. 30 stands for 10</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 8px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: super; white-space: pre-wrap;">-3</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. A score of 20 means a 1% estimated chance of being wrong, while 40 means 1 in 10,000. In this field we attach magic importance to reads of Q30 or better, but there is no particular reason for this. Statistics on the quality of reads can be calculated using for example </span><a href="http://www.bioinformatics.babraham.ac.uk/projects/fastqc/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">FASTQC</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.broadinstitute.org/gatk/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GATK</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> or </span><a href="http://bertusbeaumontlab.tudelft.nl/software.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Antonie</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (my own tool).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<img height="238px;" src="https://lh6.googleusercontent.com/Wors0gP0j_Yl-2Mmy0jqvVBGJ1sqlVvTO36y9GzP70TwGOLK0QTuuQaBHVTkiffw9kedKjtg2YEy5aYC-s1sJrZuHKZBG6xg8L5WPxO92ru9VMo6VtVJGqH43w" width="459px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Quality scores over the length of a read</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The best reads are typically near Q40, but this still isn’t enough - if we have a 5 million bases long genome and sequence it once, even if all reads were Q40, we’d still end up with 500 errors. If we expect to find maybe one or two actual mutations in our organism, we’d never see them, since they’d be lost in between the 500 “random” errors. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">@M01793:3:000000000-A5GLB:1:1101:17450:2079 1:N:0:1</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ACCTTCCTTGTTATAGTTTGCGGCCAGCGGTGGCAGTGTCGGCGCTTCTACTAAGGATTCAAGCCCCTGATTTGTGGTTGGATCTGTCNNNNNTACACATCTCCGAGCCCACGAGACAGGCAGAAATCTCGTATGCCGTCTTCTGCTTGA</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">+</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CCDDEFFFFFFFGGGGGGGGGGGGGGHGGGGGGHHGHHHHGGGGGGGGHHHHHHHHHHHHHHHHGHGHGHHHHHHHGHGGGHHHHHHH#####??FFGHHHHHHGGGGGGHGGGGGGHHGGHGGHHHHHHHGGHHHHGGGHGHHHHHHH<</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(FASTQ file)</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To solve this, we make sure the DNA sequencer doesn’t just read the whole genome once, but maybe 100 times in total. We call this ‘a depth of 100’. Depending on actual error rate, you may need a higher or lower depth. Also, since the machine performs random reads, an average depth of 100 means that there are many areas you only read 10 times (and about a similar amount of areas you needlessly read 190 times!). </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div style="text-align: center;">
<b style="font-weight: normal;"><img height="209px;" src="https://lh4.googleusercontent.com/-v0OBrFwh1OsjagxmfKmXgy_1VFTonzo6x19WH3H5WP8LLNP4w9117fXNku7QVr5O_I4O_9aMmq0x4pz7eQ_d_bNHAmqYSjn489pTwWsy9Z5cOB7YUN3cY3eDg" width="392px;" /></b></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, we collect a FASTA reference file, prepare our sample, run the DNA sequencer, and get gigabytes of FASTQ file. Now what? Alignment is what. Lots of tools exist for this purpose, famous ones are </span><a href="http://bio-bwa.sourceforge.net/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">BWA</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="http://bowtie-bio.sourceforge.net/index.shtml" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Bowtie</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(2). You can also use my tool </span><a href="http://bertusbeaumontlab.tudelft.nl/software.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Antonie</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. These index the reference FASTA genome, and ‘map’ the FASTQ to it. They record this mapping (or alignment) in a SAM file.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SAM/BAM Files, Sequence/alignment mapping</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This “Sequence Alignment/Map” format records for each ‘read’ to which part of the reference genome it corresponded. Such mapping can be direct, i.e., “it fits here”, but it can also record that a read had extra characters, or conversely, is missing bases with respect to the reference genome. DNA has two strands which are complementary, and are also called the forward and reverse strands. The DNA sequencer doesn’t (can’t) know if it is reading in the backwards or forwards direction, so when mapping, we have to try both directions, and record which one matched.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">M01793:3:000000000-A5GLB:1:1101:14433:2944 147 gi|388476123|ref|NC_007779.1| 3600227 42 150M = 3600116 -261 GTCAATTCATTTGAGTTTTAACCTTGCGGCCGTACTCCCCAGGCGGTCGACTTAACGCGTTAGCTCCGGAAGCCACGCCTCAAGGGCACAACCTCCAAGTCGACATCGTTTACGGCGTGGACTACCAGGGTATCTAATCCTGTTTGCTCC</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">:FHFHFGHHFG0HHHHGHHHFGGCCCFGGFFGFAGGGGGHGGFD?CGGGDHEGGGGGGFHHHEGCGGHHHGEFFG?GHGHHHHHHHGEHG1GHFGFGEFHHHFGHGGHHGFEGGGGHHHHHHHHHGCHGFFGGGGFGGBFFFFF4A@A??</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(SAM file)</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<a href="http://samtools.sourceforge.net/SAMv1.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">SAM files</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> are written in ASCII, but are very hard to read for humans. Computers fare a lot better, and tools like </span><a href="http://bioinf.scri.ac.uk/tablet/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Tablet</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.broadinstitute.org/igv/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">IGV</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="http://seqanswers.com/wiki/Samscope" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">samscope</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can be used to view how the DNA reads map to the reference - where mutations stand out visually. This is important, because some “mutations” may in fact be sequencing artifacts. </span><br />
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<img height="371px;" src="https://lh4.googleusercontent.com/C2lkg-Ee0a-5Zlll1SNbqfOhsp4xphGURME3ltsvt_vejG_X5PxO3X4Iwx-hH82jL4dokFpq8oN1mfqErCboBCMwP8BiZpA7chM5Gu0tYolRIX_iAvIKERuiuA" width="698px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Alignment as viewed in Tablet. White spots are (random) differences between reference & reads</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SAM files tend to be very, very large (gigabytes) since they contain the entire contents of the already large FASTQ files, augmented with mapping instructions.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For this reason, SAM files are often converted to their binary equivalent, the BAM file. In addition to being binary, the BAM format is also blockwise compressed using Gzip. If a BAM file is sorted, it can also be indexed, generating a BAM.BAI file. Some tools, like my own </span><a href="http://bertusbeaumontlab.tudelft.nl/software.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Antonie</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, can emit BAM (and BAI) files directly, but in other scenarios, ‘</span><a href="http://en.wikipedia.org/wiki/SAMtools" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">samtools</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’ is available to convert from SAM to BAM (and vice versa if required). A more advanced successor to the BAM format is called </span><a href="http://www.ebi.ac.uk/ena/about/cram_toolkit" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CRAM</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and it achieves even higher compression. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now that we have the mapping of reads to the reference, we can start looking at the difference between our organism and the reference, and this often leads to the generation of a </span><a href="http://www.1000genomes.org/node/101" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">VCF file</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which notes locations of inserts, deletes (together: indels) or different nucleotides (SNPs, pronounced “snips”).</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To figure out that these mutations might mean, we can consult a feature database. This is available in various formats, like </span><a href="http://www.ncbi.nlm.nih.gov/genbank/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Genbank</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> or </span><a href="http://en.wikipedia.org/wiki/General_feature_format" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GFF3</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. GFF3 is best suited for our purposes here, since it precisely complements the FASTA. This General Feature Format file notes for each ‘locus’ on the reference genome if it is part of a gene, and if so, what protein is associated with it. It also notes a lot of other things about regions.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div style="text-align: center;">
<b style="font-weight: normal;"><img height="164px;" src="https://lh5.googleusercontent.com/pVfyvKiLJfbLfWOh9t12IR_QLs_d4vcD8yCR4gsOhU3vKpg_zGkF1khOL-F9Ev_zU-vbVDbOza1Bxyiz56H7Otm0D_91TvPQnVFTt7PAyf6ez665NRrTTqqa2g" width="698px;" /></b></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Gene annotations</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Summarizing</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, summarizing, a reference genome is stored in a FASTA file, as found on </span><a href="http://ftp.ncbi.nlm.nih.gov/genomes/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://ftp.ncbi.nlm.nih.gov/genomes/</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. A DNA sequencer emits millions of DNA reads in a FASTQ file, which are snippets from random places in the genome. We then ‘map’ these reads to the reference genome, and store this mapping in a large SAM file or a compressed BAM file. Any differences between our organism and the reference can be looked up in Genbank or GFF3 to see what they might mean. Finally, such differences can be stored as a VCF file.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is the relation between the various file formats:</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">FASTA + FASTQ -> SAM -> BAM (+ BAM.BAI)</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BAM + FASTA -> VCF</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But where did the reference come from?</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Above, we discussed how to map reads to a reference. But for the vast majority of organisms, no reference is available yet (although most interesting ones have been done already). So how does that work? If a new organism is sequenced, we only have a FASTQ file. However, with sufficient amount of overlap (‘depth’), the reads can be aligned to themselves in a process known as ‘de novo assembly’. This is hard work, and typically involves multiple sequencing technologies. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As an example, many genomes are rife with repetitive regions. It is very difficult to figure out how reads hang together if they are separated by thousands of identical characters! Such projects are typically major publications, and can take years to finish. Furthermore, over time, reference genomes are typically improved - for example, the human genome currently being used as a reference is ‘revision 19’.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Paired end reads</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Some sequencing technologies read pieces of DNA from two ends simultaneously. This is possible because DNA actually consists of two parallel strands, and each strand can be read individually. When we do this, we get two ‘paired end reads’ from one stretch of DNA, but we don’t know how long this stretch is. In other words, the two individual reads are ‘close by’ on the chromosome, but we don’t exactly know how close. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Paired end reads are still very useful however, especially since they can disambiguate the correct place because their extra length gives us more context to correctly place them.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div style="text-align: center;">
<b style="font-weight: normal;"><img height="213px;" src="https://lh3.googleusercontent.com/zdZlpgLs84kszsn00Gxm22mWjWJUxg-pR_JAhlt6HYYwnVhWyW1yyo4ROaVTQ6zVP4evBtBdiBA8iXPomY-uH5wVB7DkdIK3fuCrze7jYHDk6iWDaXcXNTDoWg" width="411px;" /></b></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Paired end reads are often delivered as two separate FASTQ files, but they can also live together in one file. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Diving in</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Next to the wonderful archive of reference materials at </span><a href="http://ftp.ncbi.nlm.nih.gov/genomes/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://ftp.ncbi.nlm.nih.gov/genomes/</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, there are also repositories of actual DNA sequencer output. The situation is a bit confusing in that the NCBI </span><a href="http://www.ncbi.nlm.nih.gov/Traces/sra/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Sequence Read Archive</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> offers by far the best searching experience, but the </span><a href="http://www.ebi.ac.uk/ena/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">European Nucleotide Archive</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> contains the same data in a more directly useful format. The Sequence Read Archive stores reads in the SRA format, which requires slow tools to convert to FASTQ. The ENA however stores simple Gzip compressed FASTQ, but offers a less powerful web interface. It might be best to search for data in the SRA and then download it from the ENA!</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Introducing Antonie</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So when I entered this field, I had a hard time learning about all the file formats and what tools to use. It also didn’t help that I started out with bad data that was nevertheless accepted without comment by the widely used programs I found - except that meaningless results came out. Any bioinformatician worth her salt would’ve spotted the issue quickly enough, but this had me worried over the process.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If desired answers come out of a sequencing run, everyone will plot their graphs and publish their papers. But if undesired or unexpected answers come out, people will reach for tooling to figure out what went wrong. Possibly they’ll find ways to mend their data, possibly they’ll file their data and not publish. The net effect is a publication bias towards publishing some kind of result - whether the origin is physical/biological, or a preparation error. In effect, this is bad science.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">From this I resolved to write software that would ALWAYS analyse the data in all ways possible, and simply refuse to continue if the input made no sense. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Secondly, I worried about the large number of steps involved in typical DNA analysis. Although tools like </span><a href="http://galaxyproject.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Galaxy</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can create reproducible pipelines, determining the provenance of results with a complicated pipeline is hard. Dedicated bioinformaticians will be able to do it. Biologists and medical researchers under publication pressure may not have the time (or skills).</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">From this I resolved to make Antonie as much of a one-step process as possible. This means that without further conversions, Antonie can go from (compressed) FASTQ straight to an annotated and indexed BAM file with verbose statistics, all in one step. Such analysis would normally require invoking BWA/Bowtie, FASTQC, GATK, samtools view, samtools sort, samtools index & samtools mpileup. I don’t even know what tool would annotate the VCF file with GFF3 data.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div style="text-align: center;">
<b style="font-weight: normal;"><img height="234px;" src="https://lh4.googleusercontent.com/l8-2TEadfvc20V2El9cogwh2Wbduh-My7CO5H1mNMGk91tZAbmuUltXEe5I92HEhrt8hOqxP1yQKU8JvELtwFMLQlMrWX-ubbG6di8zcw5XYRyu-p4CAhfqTXQ" width="399px;" /></b></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“Well-calibrated measurements”</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finally, with decreasing sequencing costs (you can do a very credible whole genome sequencing run of a typical bacterium for 250 euros of consumables these days), the relative costs of analysis are skyrocketing. Put another way, 10,000 euros would a few years ago net you one sequencing run plus analysis. These days (2013), the same amount of money could net you 40 sequencing runs but no analysis, as outlined in “</span><a href="http://www.genomeweb.com/blog/1000-genome-bait-and-switch" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">The $1000 genome, the $100000 analysis?</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Because of this, I think the field should move to being able to operate with (minimal) bioinformatician assistance for common tasks. End-users should be able to confidently run their own analyses, and get robust results. Software should have enough safeguards to spot dodgy or insufficient data, software should be hard to misuse.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Antonie isn’t there yet, but I’m aiming towards making myself redundant at the lab (or conversely, available for more complicated or newer things).</span></div>
<br />
<span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-6270591920582469905.post-17124672598057128352013-11-27T03:02:00.003-08:002013-11-27T03:55:27.678-08:00Mapping DNA nucleotides to numbers... the cool way!<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Trebuchet MS, sans-serif;">So, there are four DNA 'characters', or nucleotides, or bases: A, C, G and T. Says the <a href="http://en.wikipedia.org/wiki/ACGT">Wikipedia</a>: '"A" stands for Adenine and pairs with the "T", which stands for Thymine. The "C" stands for Cytosine and pairs with the "G", Guanine'.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Since there are only four nucleotides, it is wasteful to spend an entire byte (8 bits) on storing these 2 bits of information. And many of the more karmic tools do indeed 'compress' DNA this way, either for storage or for rapid indexing.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">When Antonie implemented this, we used A=0, C=1, G=2, T=3, which makes some kind of lexicographical sense. However, other software specified a different mapping: A=0, C=1, T=2, G=3. So, I wondered if this had some kind of biological background, but it doesn't, it is all computer geekery!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Behold:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">A = ASCII 65, binary 1000<b><u>00</u></b>1 -> & 6 -> 00x -> 0</span> </blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">C = ASCII 67, binary 1000<b><u>01</u></b>1 -> & 6 -> 01x -> 1</span> </blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">G = ASCII 71, binary 1000<b><u>11</u></b>1 -> & 6 -> 11x -> 3</span> </blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">T = ASCII 84, binary 1010<b><u>10</u></b>0 -> & 6 -> 10x -> 2</span></blockquote>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">This is how many tools in fact map: (c&6)>>1, and it has thus become some kind of standard. </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">So now you know. </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6270591920582469905.post-1166397779179546522013-11-13T04:41:00.002-08:002013-11-13T04:44:10.096-08:00Getting the correct 'git hash' in your binaries w/o needless recompiling or linking<div dir="ltr" style="text-align: left;" trbidi="on">
When processing bug reports, or when people doubt the output of your tools, it is tremendously helpful to know the provenance of the binaries. One way of doing this is to embed the git hash in your code.<br />
<br />
A useful hash can be generated using:<br />
<blockquote class="tr_bq">
$ git describe --always --dirty=+</blockquote>
This outputs something like '4120f32+', where the '+' means there have been local changes with respect to the commit that can be identified with '4120f32':<br />
<blockquote class="tr_bq">
$ git show 4120f32</blockquote>
<blockquote class="tr_bq">
commit 4120f32ef7b684eb1ff42d136e37e8733f9811e1</blockquote>
<blockquote class="tr_bq">
Author: bert hubert <bert .hubert="" netherlabs.nl=""></bert></blockquote>
<blockquote class="tr_bq">
Date: Wed Nov 13 12:46:43 2013 +0100</blockquote>
<blockquote class="tr_bq">
rename git-version wording to git-hash, move the hash to a .o file so we only need to relink on a change</blockquote>
<div>
What we want is this:</div>
<blockquote class="tr_bq">
$ antonie --version</blockquote>
<blockquote class="tr_bq">
antonie version: g4120f32+ </blockquote>
To achieve this, we need to convince our Makefile to generate something that includes the hash from git-describe. Secondly, if this output changes, all binaries must be updated to this effect. Finally, if the git hash did not change, we should not get spurious rebuilds.<br />
<br />
Many projects, including PowerDNS, had given up on achieving the last goal. Running 'make' will always relink PowerDNS now, and even recompile a small file, even if nothing changed. And this hurts our sensibilities.<br />
<br />
For <a href="http://bertusbeaumontlab.tudelft.nl/software.html">Antonie</a>, I got stuck on a boring issue today, and decided I needed to solve the git-hash-embedding issue once and for all instead.<br />
<br />
As the first component, enter <a href="https://github.com/beaumontlab/antonie/blob/e92b14543469de7cc81638554d481a4dc7e854da/update-git-hash-if-necessary">update-git-hash-if-necessary</a>, which will update (or create) githash.h to the latest git hash, but only if it is different from what was in there already - effectively preserving the old timestamp if there were no changes.<br />
<br />
Secondly, we need to convince Make to always run update-git-hash-if-necessary before doing anything else. This can be achieved by adding the following to the Makefile:<br />
<blockquote class="tr_bq">
CHEAT_ARG := $(shell ./update-git-hash-if-necessary)</blockquote>
Even though we never use CHEAT_ARG, whenever we now run make to do anything, this will ensure that githash.h is updated if required. This clever trick was found <a href="http://stackoverflow.com/questions/1612278/pre-build-step-in-makefile">here</a>.<br />
<br />
To complete the story, create githash.c which #includes githash.h and make it define a global variable, possibly like this: 'const char* g_gitHash = GIT_HASH;'. Now include githash.o as an object for all programs needing access to the hash.<br />
<br />
Finally, add 'extern const char* g_gitHash;' somewhere so the programs can actually see it. Note that you can't use githash.h for that, since that would trigger recompilation where relinking would suffice.<br />
<br />
If you now run 'make' or even 'make antonie', any change in the git hash will trigger a recompile of githash.o, followed by rapid relinking. If there was no change, nothing will appear to have happened.<br />
<br />
Please let me know if you find ways to improve on the trick above!</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6270591920582469905.post-91789208895651718342013-10-03T06:03:00.001-07:002021-06-19T08:33:04.243-07:00Help, my sin() is slow, and my FPU is inaccurate!<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: red; font-family: 'Trebuchet MS'; font-size: x-large; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">THIS POST HAS BEEN MOVED TO A NEW LOCATION: <a href="https://berthub.eu/articles/posts/help-my-sin-is-slow-and-my-fpu-is-inaccurate/">https://berthub.eu/articles/posts/help-my-sin-is-slow-and-my-fpu-is-inaccurate/</a> WHERE IT HAS ALSO BEEN UPDATED WITH NEW 2021 RESULTS.</span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Recently, I started writing code to do a simulation of biological system, and I got stuck after 20 lines. My test code tried to go through a million time steps doing basically nothing, and it was taking a minute to do it. However, if I reduced this to only 100,000 time steps, it zoomed along.</span></div>
<b id="docs-internal-guid-23da61e8-7e65-d82e-47ae-3e6944b516fe" style="font-weight: normal;"><br /></b>
<br />
<div style="text-align: center;">
<b id="docs-internal-guid-23da61e8-7e65-d82e-47ae-3e6944b516fe" style="font-weight: normal;"><span id="docs-internal-guid-23da61e8-7e65-d82e-47ae-3e6944b516fe"><img height="297px;" src="https://lh5.googleusercontent.com/A1ugvbq7QG1ZlLVfHYvnUE59CZmD2nDkovOG5Rw63BEx-swkc9sgEUS5aA6OELbu--k0Rvm9RjAHRbxxtLHLeYqzXJR1HKNRG3IHh6Yg5EPYrtjJJbGXdqo8ow" width="396px;" /></span></b></div>
<b id="docs-internal-guid-23da61e8-7e65-d82e-47ae-3e6944b516fe" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Number of steps versus runtime for sin() and cos()</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So I checked the usual things - was there a CPU frequency scaling problem, was my CPU overheating, etc, but none of that was the case. Then I changed one call to sin() by a call to cos().. and everything was fast again.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After some further testing, I also discovered that changing M_PI to 3.141 also made things fast again. Color me confused! </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, I delved into the assembly output, where I found that nothing really changed in my code when I changed M_PI or sin() to cos(). What I did note was that an actual call was being made to a sin() function in the C library.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Trawling the web found various bug reports about sin() and cos() being both inaccurate and slow at times. And this blew my mind. I first got access to a computer with a floating point unit in the late 1980s. And over the past thirty years, I’ve been assuming (correctly) that an FPU would come with a dedicated SINCOS instruction. I had also assumed (incorrectly!) that since it came with the CPU, it would be correct & fast, and people would be using it!</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It so turns out that what the FPU offers is neither correct, nor fast. And in fact, it appears nobody is even using the FPU trigonometry instructions. So what gives?</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Range reduction</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Range reduction is what gives. It is entirely possible to very rapidly calculate sin(x) for small x using a simple approximation formula, for example, a </span><a href="http://en.wikipedia.org/wiki/Taylor_series" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Taylor series</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Tables can also be used, and for hardware there is a suitable algorithm known as </span><a href="http://en.wikipedia.org/wiki/CORDIC" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CORDIC</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (for COordinate Rotation DIgital Computer). </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Such approximations only work well for small input though, but since sin() and cos() are periodic, it is entirely possible to map a large input angle to one that is small again.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div style="text-align: center;">
<b style="font-weight: normal;"><img height="278px;" src="https://lh3.googleusercontent.com/oZjkIhbJ8cyd3NllC3H0hlooA_U2L3jo2bIA8GrEojWDZiJNB2fNFiM1qhzWzZ0PxSuRbNWIehNW_F2iPsnINozEYoymscIcVLJFjaNCXoW1wc7loJRcf5hlTA" width="371px;" /></b></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Actual sin(x) versus 11th order Taylor series, tsin(x)</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So let’s say we are trying to calculate sin(1 + 600π), we can map this to trying to calculate sin(1) - the answer will be identical.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To do so, a computer program must take the input value offered, and remove sufficient even multiples of π that the resulting value is small enough that our approximation is accurate.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And here’s the rub. If our input can be all values attainable by a double precision floating point number, to perform this range reduction naively requires knowing π to an astounding precision, </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">a precision unattainable itself in floating point hardware</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">!</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It is easy to see how this works - even a tiny error in the value of π adds up, in our example of sin(1 + 600π), an error of 1/1000 would still lead us to be off by 0.6!</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Of course a CPU can do better than a 1/1000 error, but it will still never be able to reduce (say) 10</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 8px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: super; white-space: pre-wrap;">22</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> back to the correct value. To see this in practice, the proper value of sin(9*10</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 8px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: super; white-space: pre-wrap;">18</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) is -0.48200764357448050434, but if we ask my Intel FPU, it returns -0.47182360331267464426 which is off by more than 2%. To its credit, if we ask the FPU to calculate the sine of larger numbers, it refuses.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So.. this explains why a programming language can’t rely on the FPU to calculate proper values. But it still does not explain how a higher level language might solve this.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Way back to 1982</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The problem of range reduction was known to be hard - so hard that programmers were urged just not to try to calculate the sine of very large angles! “You do the range reduction”. This however is weak sauce, and makes life unnecessarily hard. For example, if modelling the response of a system to a sinusoid input of f Hz, it is typical to calculate F(x) = sin(2π * f * t) for prolonged periods of time. It would truly be inconvenient if we could only run simulations with well chosen time steps or frequencies so that we would be able to trivially adjust our ranges back to single digits.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And then, after telling programmers for years to clean up their act, suddenly two independent groups achieved a breakthrough. Mary H. Payne and Robert N. Hanek of Digital Equipment Corporation published the seminal paper “Radian Reduction for Trigonometric Functions”, </span><a href="http://dl.acm.org/citation.cfm?id=1057602" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">paywalled to this day</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Meanwhile Bob Corbett at UC Berkeley did not write a paper, but did </span><a href="http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/lib/libm/arch/vax/n_sincos.S" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">independently implement</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> the same technique in BSD.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’d love to explain how it works, and it pains me that I can’t. It is very clever though! It is clear that we have to be grateful for Payne, Hanek and Corbett for inventing “infinite precision range reduction”! It appears to be in universal use, and it is quite astounding to realize that prior to 1982, it was effectively very hard to do trigonometry on very large numbers on the hardware that was available!</span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So why was my original code so slow?</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We now know that computers use actual code to calculate sin(x), and don’t leave it to a dedicated FPU instruction. But why was it fast up to 170,000 steps? It turns out that the sin(x) found in my C library contains various strategies for calculating the right answer, and it does indeed use the Payne-Hanek-Corbett technique, as implemented by IBM in their Accurate Mathematical Library.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If it all works out, some very rapid code is used. But if the library detects a case in which it can’t guarantee the required precision, it reverts to a functions actually called ‘slow()’. And they live up to the name. This is supposed to be rare though, so why was it happening to me?</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Viz the test code:</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">int main(int argc, char **argv)<br class="kix-line-break" />{<br class="kix-line-break" /> double t;<br class="kix-line-break" /> double stepsizeMS=0.1;<br class="kix-line-break" /> double Hz=1000;<br class="kix-line-break" /> double val=0;<br class="kix-line-break" /> unsigned int limit = argc > 1 ? atoi(argv[1]) : 200000;<br class="kix-line-break" /><br class="kix-line-break" /> for(unsigned int steps = 0; steps < limit; ++steps) {<br class="kix-line-break" /> t = steps * stepsizeMS;<br class="kix-line-break" /> val = sin(2.0*M_PI*Hz*t);<br class="kix-line-break" /> printf("%f %f\n", t, val);<br class="kix-line-break" /> }<br class="kix-line-break" />}</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">See if you can spot it. I didn’t, but the most excellent Nicolas Miell did. This code is effectively calculating the sine of </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">exact multiples of π</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> since </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Hz*stepsizeMS</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> equals 100.0000, and </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">steps</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is an integer. And apparently, after around 100,000 steps, this triggers the glibc libm implementation to revert to one of the ‘slow()’ functions.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This also explained why the problem went away by changing sin() into cos(), or my removing only a tiny bit of accuracy from M_PI - we no longer hit the slow path all the time.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, I’m not that happy that the system C library exhibits such slowdowns (which have been noted by others, see reference 2), but in this case my simulation would be useless in any case - I’d only be exerting a force of 0 anyhow, and never achieve anything!</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It does turn out there are libraries that offer superior performance, like:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: "Trebuchet MS"; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://software.intel.com/en-us/intel-mkl/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Intel MKL</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (not free, neither as beer, nor as speech)</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: "Trebuchet MS"; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://developer.amd.com/tools-and-sdks/cpu-development/amd-core-math-library-acml/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">AMD ACML</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (free, but no source, limited CPU support)</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: "Trebuchet MS"; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://gruntthepeon.free.fr/ssemath/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Simple SSE and SSE2 (and now NEON) optimized sin, cos, log and exp</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (zlib license, really fast, but does not do range reduction)</span></div>
</li>
</ul>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<a href="http://software.intel.com/en-us/intel-mkl/" style="text-decoration: none;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Summarizing</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://software.intel.com/en-us/intel-mkl/" style="text-decoration: none;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Calculating (co)sines is done “ma</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">nually”, and may proceed at different speeds depending on your input. The output of your FPU might be wrong, and many FPUs refuse to process really large inputs. Better libraries than your standard system library might be needed if you truly need high performance.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">References</span></h2>
<ol style="margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<li style="background-color: transparent; color: black; font-family: "Trebuchet MS"; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Argument Reduction for Huge Arguments: Good to the Last Bit by K. C. Ng and the members of the FP group of SunPro - </span><a href="http://www.validlab.com/arg.txt" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.validlab.com/arg.txt</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
</li>
<li style="background-color: transparent; color: black; font-family: "Trebuchet MS"; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The state of glibc libm (dire) - </span><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="http://gcc.gnu.org/ml/gcc/2012-02/msg00469.html" style="text-decoration: none;">http://gcc.gnu.org/ml/gcc/2012-02/msg00469.html</a></span></div>
</li>
<li style="background-color: transparent; color: black; font-family: "Trebuchet MS"; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Intel overstates FPU accuracy - </span><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="http://notabs.org/fpuaccuracy/" style="text-decoration: none;">http://notabs.org/fpuaccuracy/</a></span></div>
</li>
<li><span style="line-height: 1.15;">A new range reduction algorithm - <a href="http://www.imada.sdu.dk/~kornerup/papers/RR2.pdf">http://www.imada.sdu.dk/~kornerup/papers/RR2.pdf</a></span></li>
</ol>
<br />
<span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6270591920582469905.post-31505472556795899942013-09-16T03:37:00.001-07:002013-09-16T03:37:12.820-07:00The Business Moped Theory of New Product Rejection<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;">So, you or your company invented something radical & new: congratulations! This has happened to me twice, once with </span><a href="http://www.powerdns.com/" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PowerDNS</span></a><span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;"> and once with </span><a href="https://www.fox-it.com/" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Fox-IT</span></a><span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;">. Both times my products had unique features, and both struggled to gain market acceptance initially. Potential customers told us things that were wrong with our product, we’d fix them, and they’d come up with new reasons not to buy our stuff.</span></div>
<b id="docs-internal-guid-5f340e9b-2653-1575-ef31-14caaf76c844" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After a few iterations of this, I formed what I like to call “The Business Moped Theory of New Product Rejection”, and this theory has served me well (a moped is also known as a scooter or “brommer” in Dutch). In this post, I want to share the Business Moped idea, and how it spells bad news, but also what you can do with your new invention to make it sell better.</span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Commuting to the office</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Back when I lived in Delft, parking my car was a big hassle, and more often than not I had to park it (semi-illegally) over a kilometer away from my home. If you live in a city center, this is quite common. Additionally, the office where I worked 10km away had 750 employees and 200 parking places, leading to similar drama there.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, I could go to work on my bicycle, but many of my colleagues lived too far away for that. But we had one guy who’d been commuting on a moped for 20 years now, and he was never late because of traffic and had no trouble parking either at home or at the office. Also, it was very cheap both in maintenance & fuel. He found a great solution, but nobody copied him.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<b style="font-weight: normal; margin-left: 1em; margin-right: 1em;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="258px;" src="https://lh4.googleusercontent.com/HU5y6XYir9y3CgvisABTpQVSWkqQ1FSTuK58vWdFMn-8KaQsHTBTXpf8tNYObGuklsV75hU0N0Xdk2rA6mre_liU32425aOXqN56bxKszcPcWeAvbl1M3gPDCg" width="344px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<a href="http://www.tomosforum.nl/uploads/images/galerij/Brommer.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.tomosforum.nl/uploads/images/galerij/Brommer.jpg</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And why not? Well, the honest reason is of course that it would look silly! You don’t go to your office in business clothes on a moped!</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<br />
<div dir="ltr" style="-webkit-text-stroke-width: 0px; clear: left; color: black; float: left; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 1.15; margin: 0pt 1em 1em 0px; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For evidence:</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://lh6.googleusercontent.com/SFRHXbsl8eUus3DE-oHh6nzN2ZXPV_MrLLW1cKoPzr36Ul-z1SuHWrYmVDOpg0YC8CBdhIDDIhQ4JyHJE2T7FcBKrhEeaVZEW5kPUWeZ0dezuUENJ-6cu1qyDA" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="263px;" src="https://lh6.googleusercontent.com/SFRHXbsl8eUus3DE-oHh6nzN2ZXPV_MrLLW1cKoPzr36Ul-z1SuHWrYmVDOpg0YC8CBdhIDDIhQ4JyHJE2T7FcBKrhEeaVZEW5kPUWeZ0dezuUENJ-6cu1qyDA" width="176px;" /></a></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<a href="http://thumbs.dreamstime.com/x/business-man-scooter-8685824.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://thumbs.dreamstime.com/x/business-man-scooter-8685824.jpg</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“Dork”</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But when you ask people why they continue to spend an hour stuck in traffic and parking everyday, and not just copy the guy with the moped, they don’t come out and say “it would look silly”.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">No, they’d say things like “the car keeps me dry in the rain”, “In winter it is too cold”, “a moped isn’t safe”. So, this set people thinking, as moped vendors had long been trying to crack the business market. So they addressed the concerns, starting with this attempt:</span></div>
<div class="separator" style="clear: both; text-align: center;">
<b style="font-weight: normal; margin-left: 1em; margin-right: 1em;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="344px;" src="https://lh3.googleusercontent.com/pP2f9cQjmcUVDi-ULRQ9HQS3O5ZPyXptVOxXnRSKBNzq5Dxd9v3HWNBdNuvF46Cl4Cg8EpyCPIGtrilp5BgKfKjgcbLzIh8d38ubeo3syQJDezRY6GFu5rCq3A" width="258px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<a href="http://upload.wikimedia.org/wikipedia/commons/5/5f/Benelli_Adiva_01.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://upload.wikimedia.org/wikipedia/commons/5/5f/Benelli_Adiva_01.jpg</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“Benelli Adiva”</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">With a roof, surely people will stop complaining about getting wet in the rain? Well, after 5 years of trying to sell the thing, Benelli gave up. I’ve never seen one in any case. But, the industry kept working on it, perhaps fixing the “rain” issue wasn’t enough? </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Enter the </span><a href="http://en.wikipedia.org/wiki/BMW_C1" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">BMW C1</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which had everything:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BMW: A cool business brand</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Roof</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Safety harness, no need to wear a helmet</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Heated seats!</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Heated steering handles!</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Storage for (rain) clothes</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They even sold it to racing teams, police forces and other promotionally interesting customers:</span></div>
<div class="separator" style="clear: both; text-align: center;">
<b style="font-weight: normal; margin-left: 1em; margin-right: 1em;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="399px;" src="https://lh5.googleusercontent.com/oRDCp9XVJpMHmRs0kG91Uy6i8ZEFS5bIXxOkII0_T__SD4di1c-mpbUYHlY7VERhIIXGMhke7DMcLRsZyHMwRQa3WX7dsdL7vzqDDv82iu4oD7hkfueMDzivNQ" width="532px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<a href="http://upload.wikimedia.org/wikipedia/commons/b/b5/BMW_C1_-_Bayerische_Polizei_1.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://upload.wikimedia.org/wikipedia/commons/b/b5/BMW_C1_-_Bayerische_Polizei_1.jpg</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So now that all customer issues were addressed, it should sell well right? Think again, they sold hundreds!</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But, this is BMW we are talking about, they had the resources to fix whatever complaints were left & up the marketing even further. So, they also offered:</span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div dir="ltr" style="line-height: 1.15; margin-left: 1em; margin-right: 1em; margin-top: 0pt; text-align: left;">
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-left: 1em; margin-right: 1em; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<br />
<ul style="margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<li><span style="font-family: 'Trebuchet MS'; font-size: 13px; line-height: 1.15; white-space: pre-wrap;">Anti-lock brakes</span></li>
<li><span style="font-family: 'Trebuchet MS'; font-size: 13px; line-height: 1.15; white-space: pre-wrap;">“Executive leather seats”</span></li>
<li style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A navigation system</span></div>
</li>
<li style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Powerful audio, with automatic sound volume adjustment to speed</span></div>
</li>
<li style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">An immobilizer system to prevent theft</span></div>
</li>
<li style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A sunroof!</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</li>
<li style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Glove box with power socket</span></div>
<div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://lh6.googleusercontent.com/mLp173-lSaw5gXMTPDgUsUum_4AtofWknOgz1ejExj0XK655JxKhXvvzzRWTiLgaVyXZA-F6LYsLW20gDEQXHgMhQqkdEZcCi1peEIfpcJwl-PO6CEJZBxPoOw" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="576px;" src="https://lh6.googleusercontent.com/mLp173-lSaw5gXMTPDgUsUum_4AtofWknOgz1ejExj0XK655JxKhXvvzzRWTiLgaVyXZA-F6LYsLW20gDEQXHgMhQqkdEZcCi1peEIfpcJwl-PO6CEJZBxPoOw" width="419px;" /></a></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<a href="http://upload.wikimedia.org/wikipedia/commons/7/73/C1RollerBMW.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://upload.wikimedia.org/wikipedia/commons/7/73/C1RollerBMW.jpg</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“Executive edition”</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, now that every *possible* customer complaint was fixed, sales surely took off? Thousands were sold! And after 5 years of trying real hard, even mighty BMW had to give up.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As a last salvo, Italian Piaggio came up with this marvel, for the people who complained that they didn’t want a “bike that could fall over”:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<img height="295px;" src="https://lh5.googleusercontent.com/8crK0NNVoJyHwKw53Raco_0gS4ixAlzJ1yQXH1FoPt0EfrHUOWdlazC-itUyi6svLzVIWinUSQB0JwPOrA4G6J7RhRQCbtUrqn-MorWLt18UteREDKF3ZTVBIw" width="444px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Also, not ever</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<a href="http://www.webwombat.com.au/motoring/images/piaggio-mp3-scooter-3-big.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.webwombat.com.au/motoring/images/piaggio-mp3-scooter-3-big.jpg</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why didn’t it work?</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">People want to buy what they’ve been used to buying. They don’t want to be too different. You can get away with selling something that is vastly more powerful than customers were used to, but still is fundamentally the same thing - as long as buyers trust your company. But even trust in BMW was not enough to convince people that the moped was suitable for taking to the office, </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">no matter how hard they tried to get their scooter to emulate the features of a car. When people secretly want a car, they don’t buy a scooter!</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When as an innovative company you are trying to do something that is too different, that is not what you’ll hear from your customers (they might say it, but </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">hearing</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> it is something else). No, you’ll hear specific complaints that you can address. The pitfall is that even when you solve those complaints, they’ll come up with new ones. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you are trying to sell something that is too different, fixing the superficial issues is not going to save you - even though your potential customers assure you it will. They are just dishing up excuses. They want something “normal”.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 10pt; text-indent: -18pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Not being too different from something they DO know</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I formed this theory a few years ago, and lately something interesting has happened. While it proved to be impossible to convince people to get a weird looking moped for commuting to the office.. it was entirely possible to electrify the venerable bicycle, and keep it looking “normal”:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<b style="font-weight: normal; margin-left: 1em; margin-right: 1em;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="266px;" src="https://lh3.googleusercontent.com/1MsdjdIB-Q7qPvaKIEriD5CRCbT9LvyxQyy1s1CJNhQtQ0739bGNt_tclagQOEBwPbdCfg1gcPWWW5pfHYIL94KrcbaXBFe6k_EocpuQGqVK0Laql-puM2Fe5Q" width="400px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<a href="http://www.post-fietsen.nl/site/webfiles/content/koga-e-bike.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.post-fietsen.nl/site/webfiles/content/koga-e-bike.jpg</span></a><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note the discreet battery pack in the rear</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And this is now taking off in a big way, without any excuses about getting wet etc. The issue was never with getting wet in the rain (they still had their cars as backups, right?). The issue was not wanting something too different from what everybody else had! And an e-bike is sufficiently close to a bike that droves are now using one.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So summarizing - be careful not to follow BMW in blowing a billion Marks on satisfying customers that aren’t telling you the real reason they don’t want your stuff. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Instead, find people that DO have an open mind, or try being close enough to what your customers already know, while still delivering superior value. Another solution is to wait it out until sufficient people start agreeing that what you offer is just better. Might take longer than you have though!</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Good luck!</span></div>
<br /><span style="font-family: 'Trebuchet MS'; font-size: 13px; vertical-align: baseline; white-space: pre-wrap;"></span>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PS: I’ve also been told that the BMW C1 was a pretty crappy bike. That never helps!</span></div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6270591920582469905.post-41332925358682379212013-09-05T01:39:00.001-07:002013-09-05T01:58:17.804-07:00Boekbespreking: Doe eens normaal man (in 7 stappen naar een betere politiek)<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="p1">
<a href="http://www.atlascontact.nl/boek/doe-eens-normaal-man/">"Doe eens normaal man (in 7 stappen naar een betere politiek)"</a> van Kustaw Bessems en Dirk Jacob Nieuwboer</div>
<div class="p1">
<br /></div>
<div class="p1">
Laat ik beginnen met zeggen dat ik vind dat dit boek in 250-voud over het Binnenhof verspreid zou moeten worden, een exemplaar voor ieder kamerlid, minister en staatssecretaris. (<span style="color: red;">Update:</span> Dirk Jacob meldt dat alle 150 Tweede Kamerleden een exemplaar gekregen hebben!) Tevens zou nog een flinke oplage neergelegd kunnen worden in de diverse kamerrestaurants & toiletgelegenheden, voor het geval iemand het gemist had.</div>
<div class="p2">
<br /></div>
<div class="p1">
Van de achterflap "Hoe komt het dat politici zo gek praten? Waarom vinden ze het zo moeilijk sorry te zeggen? Mogen ze trots zijn op een compromis?". En dat raakt me. Ik kom uit een politiek nest, je kon in het verleden meerdere malen stemmen op mijn vader, mijn moeder en zelfs oma, voor provinciale staten, gemeenteraad en we hebben zelfs tweemaal een gedeputeerde gehad. </div>
<div class="p2">
<br /></div>
<div class="p1">
Als kind stonden mijn broer en ik vooraan met prinsjesdag in het gebouw van de tweede kamer, omringd door de kinderen van kamerleden & andere kamermedewerkers. Dolle pret. Mijn moeder zette haar kooktalent (naast haar beleidsmatige werk!) maar al te graag in voor politieke doelen, wat resulteerde in dinertafels vol politici. Van huis uit was de politiek ons dierbaar, en iets van heel dichtbij. </div>
<div class="p2">
<br /></div>
<div class="p1">
En daarom doen de afgelopen decennia mij zo'n pijn. Een politicus breekt een verkiezingsbelofte schofterig, en staat slechts uren later uit te leggen dat hij zo blij is met het resultaat. Er zijn grote fouten gemaakt en er volgt een excuus dat je van een peuter niet zou accepteren ("als ik mensen gekwetst zou hebben betreur ik dat"). Maar ook, een politicus presenteert een visie en krijgt kritiek dat ie geen concreet plan heeft, een andere politicus presenteert een concreet plan en wordt verweten geen visie te hebben. Het is ook nooit goed!</div>
<div class="p2">
<br /></div>
<div class="p1">
In "Doe eens normaal man" leggen Bessems en Nieuwboer (beide journalisten) uit hoe dat nou zo gekomen is, en uniek, ze bieden ook suggesties hoe het op te lossen. </div>
<div class="p2">
<br /></div>
<div class="p1">
Het eerste hoofdstuk begint gelijk heel sterk, en ik denk eigenlijk dat dit als wortel van het boek zelfs nog wat onderbelicht blijft. Een politicus moet weten waarom hij of zij de politiek in ging. Als ie eenmaal aangekomen is in Den Haag moet ie het niet vergeten, en hij moet het ook duidelijk uit blijven spreken. Bij afwijking van de idealen mag je dat best zeggen. Met pakkende voorbeelden over Wouter Bos, Job Cohen, Diederik Samsom en Jeanine Hennis maken de schrijvers duidelijk hoe het (niet) moet.</div>
<div class="p2">
<br /></div>
<div class="p1">
Persoonlijk denk ik dat als je als politicus geen helder antwoord hebt (en durft te geven) op de vraag waarom je nou de politiek in wilde je net zo goed gelijk kunt stoppen. Dit is echt de wortel van het verhaal, wat overigens uitgaat van politici die oprecht hun best willen doen.</div>
<div class="p2">
<br /></div>
<div class="p1">
Het volgende hoofdstuk beschrijft het vermaledijde "draaien". Normale mensen veranderen, gegeven veranderende inzichten of omstandigheden, van mening. Bij politici heet dit gelijk "draaien" en is dat "niet eerlijk". Wederom met voorbeelden leggen de auteurs uit hoe je fatsoenlijk moet en kunt draaien, en dat als je het uitlegt je publiek dat ook zal begrijpen. Zo dom zijn die stemmers niet namelijk.</div>
<div class="p2">
<br /></div>
<div class="p1">
Volgende hoofdstukken geven gelijksoortige inzichten in het sluiten van compromissen (wees eerlijk dat het jouw keuze niet was, leg het niet zo uit alsof je toch gewonnen had), excuses aanbieden (oprecht en zeg wat je geleerd hebt), "geen commentaar geven" (leg uit waarom je niks zegt, ontken niet dat je niks zegt!) en hoe normaal te blijven praten.</div>
<div class="p2">
<br /></div>
<div class="p1">
De rode draad in dit verhaal is dat als je een helder en eerlijk verhaal hebt, en je niet poogt te verstoppen achter Haags gekonkel of slimpraterij, je electoraat dat zal waarderen. En wat hoop ik dat dat waar is! Bessems en Nieuwboer zijn beide goede verstaanders en denken na over wat ze horen en lezen. Hun inzichten (ook buiten dit boek) bewijzen dat. </div>
<div class="p2">
<br /></div>
<div class="p1">
En het doet hun pijn hoe politici vol met wijsheden als "in een vlek moet je niet wrijven" en "stilzitten als je geschoren wordt" steeds vreemder praten en schijnbaar nooit meer met minder dan absoluut succes uit een vergadering komen, en zeker nooit ongelijk hebben gekregen!</div>
<div class="p2">
<br /></div>
<div class="p1">
Ik ben er echter niet zeker van of de door de auteurs aanbevolen duidelijkheid (en ze geven vele goede voorbeelden die op mij erg overtuigend overkomen) ook aansluiting zal vinden bij het publiek en met name de media.</div>
<div class="p2">
<br /></div>
<div class="p1">
Het siert de auteurs dan ook dat ze zelf ook nog een afsluitend hoofdstuk 'media' opgenomen hebben, over hoe journalisten zelf bijgedragen hebben aan de vervreemding van politici. Met prachtige voorbeelden laten ze zien dat "het ook nooit goed is". De media zullen nieuwe duidelijkheid niet moeten gebruiken als excuus gehakt te maken van de helder pratende politicus!</div>
<div class="p2">
<br /></div>
<div class="p1">
Tussen de concrete hoofdstukken bovenstaand genoemd zitten prachtige epistels over het wisselen van zwakke ministers, vijf nieuwe "droompartijen" (ik zou erop stemmen!), de gevaren van tot laat door vergaderen en de onzin van bezuinigen op de politiek zelf.</div>
<div class="p2">
<br /></div>
<div class="p1">
Afsluitend, zoals eerder gezegd zou ik het geweldig vinden als ieder aanstaand kamerlid bij de eerste vergadering een exemplaar van die boek zou aantreffen op zijn of haar zetel. Samen met "<a href="http://www.jorisluyendijk.nl/boeken%20D.html">Je hebt het niet van mij</a>" van Joris Luyendijk (wat een stuk cynischer is) zou dit boek tot de verplichte introductiecursus moeten behoren.</div>
<div class="p2">
<br /></div>
<div class="p1">
En ik hoop van harte dat de media en het publiek de resulterende helderheid en duidelijkheid niet zullen afstraffen, maar waarderen.</div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<br />
<div class="p2">
<br /></div>
</div>
Unknownnoreply@blogger.com0