[Tails-dev] An amusing tale of indeterminism

anonym anonym at lavabit.com
Thu Oct 11 14:09:18 CEST 2012


While testing 0.14~rc1 I discovered that I2P wasn't behaving as it
should, namely that even though it connected to the I2P network just
fine, it never opened its local HTTP proxy. After many confusing tests
that didn't make sense I ended up builing a new image from the 0.14-rc1
tag using the exact same packages (i.e. the *.packages diff was empty),
and in this image *I2P* *worked* *fine*. WTF?

[Below "bad" refers to the 0.14~rc1 image (or its contents) and "good"
refers to this new image.]

This just blew my mind completely. I compared the contents of the good
and bad images' roots, and their filesystem.squashfs:s. A lot of
expectable differences showed up, but one wasn't quite as expectable:


which is part of the i2p-router package. I unpacked the .deb from our
git repo, and it turned out that the i2ptunnel.jar in the good image was
the same as the one in the .deb. So the i2ptunnel.jar in the bad image
has somehow been modified during the build process. I unpacked it and
diffed its contents with the good one, and the only difference was:


That just makes too much sense, since (checking the corresponding .java
file in the I2P sources) this source file is described to:

     * Act as a mini HTTP proxy, handling various different types of
     * requests, forwarding them through I2P appropriately, and
     * displaying the reply. [...]

See attached file for the diff between decompiled versions of the good
and the bad .class file. It shows that something within the readFile()
function changed so that even the decompiler couldn't make sense out of
it (notice the "// ERROR //").

Those who were on #tails-dev yesterday may remember that I joked about
gamma rays flipping bits in my memory, and this starts to look more and
more plausible. Look at what `cmp -l` says when comparing the bad and
the good .jar file byte-by-byte:

    62050  37  33

So, only the 62050:th byte differs; it's (in base 10) 37 in the bad .jar
and 33 in the good .jar. Perhaps this made the CRC fail so the .jar file
wasn't loaded in I2P (I didn't see any warning about this in I2P's logs,
though)? Any way, notice that:

    good:  33 (base 10) = 00100001 (base 2)
    bad:   37 (base 10) = 00100101 (base 2)
                               ^------------- this bit has flipped!

Is this the prank of some mischievous cosmic ray? :) I'm unsure how
worried we should be about this happening again, and if we should
consider exploring preventative measures like only building releases on
machines using ECC memory, or similar. I nevertheless find this whole
story quite intriguing as I've never been able to track down a bit-flip
with this level of confidence before. :)

Any way, I'd rather not consider the I2P issue as a blocker for rc1
since most testing has already been done, and the interesting changes
that need to be tested for 0.14 really are outside of I2P any way. Let's
just make sure no bit-flips breaks it in the final 0.14 release. :)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: I2PTunnelHTTPClient.diff
Type: text/x-diff
Size: 4420 bytes
Desc: not available
URL: </pipermail/tails-dev/attachments/20121011/e3a6a464/attachment.diff>

More information about the tails-dev mailing list