Comments on: Bugs of Doom (aka the Heisenbugs) http://lbrandy.com/blog/2009/02/bugs-of-doom-aka-the-heisenbugs/ { on programming and the internets, every monday } Mon, 23 Aug 2010 09:39:45 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Chris Wash http://lbrandy.com/blog/2009/02/bugs-of-doom-aka-the-heisenbugs/comment-page-1/#comment-2975 Chris Wash Wed, 18 Feb 2009 18:54:09 +0000 http://lbrandy.com/blog/?p=540#comment-2975 This I wouldn't call a bug but an issue that warranted debugging no less. It's a good story. http://www.ibiblio.org/harris/500milemail.html This I wouldn’t call a bug but an issue that warranted debugging no less. It’s a good story.

http://www.ibiblio.org/harris/500milemail.html

]]> By: Steve http://lbrandy.com/blog/2009/02/bugs-of-doom-aka-the-heisenbugs/comment-page-1/#comment-2901 Steve Tue, 17 Feb 2009 12:11:22 +0000 http://lbrandy.com/blog/?p=540#comment-2901 Back in 1998 I was working in a big team on a complex C/C++ settlement system at a clearing bank in Luxembourg. We used Clearcase for version control, which presented version-specific trees of files to our workstations using a modified NFS mechanism. Builds took a long time, so somebody turned on a clever Clearcase feature that wrapped our compiler and magically popped up .o files originally compiled on other developers' workstations with compatible file versions, saving us the need to compile every file ourselves to make our executables. Clever idea in principle, except that after a couple of months, everybody's executables started crashing. It turned out, after a great deal of investigation, that environment variables on one of the developer's machines had influenced a C++ template expansion strategy, but the smart compiler wrapper had happily propagated the resulting incompatible .o files to the central Clearcase server, and therefore to all the other developers' machines. If I recall correctly, the peculiar choice of solution was to periodically clean out the server-cached .o files. Ah, good times, good times. Back in 1998 I was working in a big team on a complex C/C++ settlement system at a clearing bank in Luxembourg. We used Clearcase for version control, which presented version-specific trees of files to our workstations using a modified NFS mechanism. Builds took a long time, so somebody turned on a clever Clearcase feature that wrapped our compiler and magically popped up .o files originally compiled on other developers’ workstations with compatible file versions, saving us the need to compile every file ourselves to make our executables.

Clever idea in principle, except that after a couple of months, everybody’s executables started crashing. It turned out, after a great deal of investigation, that environment variables on one of the developer’s machines had influenced a C++ template expansion strategy, but the smart compiler wrapper had happily propagated the resulting incompatible .o files to the central Clearcase server, and therefore to all the other developers’ machines.

If I recall correctly, the peculiar choice of solution was to periodically clean out the server-cached .o files. Ah, good times, good times.

]]> By: Dexter http://lbrandy.com/blog/2009/02/bugs-of-doom-aka-the-heisenbugs/comment-page-1/#comment-2897 Dexter Tue, 17 Feb 2009 09:28:14 +0000 http://lbrandy.com/blog/?p=540#comment-2897 It took me two weeks to figure out why the pricelist.ini (a very basic INI file) that the web application produced for the barcode reader (a WinCE handheld) was not recognized by the gadget's software. I could make it read a manually made file, but not a genereated one. And they looked identical! Only with the hex editor I realized that the gadget ran a little-endian ARM CPU, so it needed the UTF bytes swapped! A single change in the output encoding did the trick... Dexter It took me two weeks to figure out why the pricelist.ini (a very basic INI file) that the web application produced for the barcode reader (a WinCE handheld) was not recognized by the gadget’s software.
I could make it read a manually made file, but not a genereated one. And they looked identical!
Only with the hex editor I realized that the gadget ran a little-endian ARM CPU, so it needed the UTF bytes swapped!
A single change in the output encoding did the trick…

Dexter

]]> By: Paul Betts http://lbrandy.com/blog/2009/02/bugs-of-doom-aka-the-heisenbugs/comment-page-1/#comment-2874 Paul Betts Tue, 17 Feb 2009 03:35:08 +0000 http://lbrandy.com/blog/?p=540#comment-2874 The way I try to solve these tricky race conditions is through a circular in-memory log - just have it write a 64-bit timestamp, the current thread ID, and a 2-byte tag (like a Pool tag in the Windows kernel). Make sure that the index to the next item is atomic, or else your log will be corrupted; just keep InterlockedIncrement'ing it and mod by LOG_SIZE. This way, when the program dies, you can dump out the log and see the sequence of events that led to this happening. The way I try to solve these tricky race conditions is through a circular in-memory log – just have it write a 64-bit timestamp, the current thread ID, and a 2-byte tag (like a Pool tag in the Windows kernel). Make sure that the index to the next item is atomic, or else your log will be corrupted; just keep InterlockedIncrement’ing it and mod by LOG_SIZE.

This way, when the program dies, you can dump out the log and see the sequence of events that led to this happening.

]]> By: dar7yl http://lbrandy.com/blog/2009/02/bugs-of-doom-aka-the-heisenbugs/comment-page-1/#comment-2873 dar7yl Tue, 17 Feb 2009 03:02:15 +0000 http://lbrandy.com/blog/?p=540#comment-2873 I had an exasperating experience trying to track down a heisenbug in a medical display device I was working on. (l33t project!) The device would randomly lock up, frazzle the display, reboot, or randomly perform uncommanded functions. There was no way to reproduce the problem. When traces were enabled, it would go away. If you watched it for hours, nothing happened, but when you turned your back, BAM! I realized it was a heisenbug, so I bring out my heisentools. First, check for uninitialized variables, the cause of 95% of HB's (IMHO). Well, I found lots of those (oops), but didn't fix the problem. Ok, so next look for race conditions. The process model was fairly simple, and easy to instrument. There were a few weird interactions, but no deadlock potential. Finally, I had a bit of luck, and managed to get a snapshot from just before the crash. There I noticed that some data wasn't in the expected location. That data was supposed to be written from the device driver, under interrupt control. Once spotted, it was easy to track down where I had neglected to set the data direction bit during a block copy, inside that interrupt routine. Turns out that the interrupt inherits the DD from the interrupted process. Most of the time, it is set correctly, but rarely, it is reversed. Then that errant data goes on to scramble control structures and bring the machine down, usually much later. The hunt took 3 days, which really messed up the schedule. On the plus side, the system was vastly improved afterword. There's nothing like a weird bug to make you go through your code with a fine-tooth comb. I had an exasperating experience trying to track down a heisenbug in a medical display device I was working on. (l33t project!)

The device would randomly lock up, frazzle the display, reboot, or randomly perform uncommanded functions.

There was no way to reproduce the problem. When traces were enabled, it would go away. If you watched it for hours, nothing happened, but when you turned your back, BAM!

I realized it was a heisenbug, so I bring out my heisentools.
First, check for uninitialized variables, the cause of 95% of HB’s (IMHO). Well, I found lots of those (oops), but didn’t fix the problem.

Ok, so next look for race conditions. The process model was fairly simple, and easy to instrument. There were a few weird interactions, but no deadlock potential.

Finally, I had a bit of luck, and managed to get a snapshot from just before the crash. There I noticed that some data wasn’t in the expected location. That data was supposed to be written from the device driver, under interrupt control.

Once spotted, it was easy to track down where I had neglected to set the data direction bit during a block copy, inside that interrupt routine. Turns out that the interrupt inherits the DD from the interrupted process. Most of the time, it is set correctly, but rarely, it is reversed. Then that errant data goes on to scramble control structures and bring the machine down, usually much later.

The hunt took 3 days, which really messed up the schedule. On the plus side, the system was vastly improved afterword. There’s nothing like a weird bug to make you go through your code with a fine-tooth comb.

]]> By: Matthew Crumley http://lbrandy.com/blog/2009/02/bugs-of-doom-aka-the-heisenbugs/comment-page-1/#comment-2872 Matthew Crumley Tue, 17 Feb 2009 02:53:42 +0000 http://lbrandy.com/blog/?p=540#comment-2872 When I first started programming on Linux, I was playing around with a program (I don't even remember what it did). I named the executable "test". When I ran it, I was expecting some output, but didn't get any, so I assumed it was crashing. When I stepped though it in the debugger, everything was fine, and I got the expected answer. After cutting out more and more code, until I *knew* it couldn't be crashing, I found out that there's an existing program called test. I didn't know at the time to use "./test", so I was running the existing command. When I ran it with gdb of course, it found the local file and worked. I learned a couple good lessons about Unix/Linux programming from that... When I first started programming on Linux, I was playing around with a program (I don’t even remember what it did). I named the executable “test”. When I ran it, I was expecting some output, but didn’t get any, so I assumed it was crashing. When I stepped though it in the debugger, everything was fine, and I got the expected answer. After cutting out more and more code, until I *knew* it couldn’t be crashing, I found out that there’s an existing program called test. I didn’t know at the time to use “./test”, so I was running the existing command. When I ran it with gdb of course, it found the local file and worked.

I learned a couple good lessons about Unix/Linux programming from that…

]]> By: Bill King http://lbrandy.com/blog/2009/02/bugs-of-doom-aka-the-heisenbugs/comment-page-1/#comment-2871 Bill King Tue, 17 Feb 2009 02:31:29 +0000 http://lbrandy.com/blog/?p=540#comment-2871 I wrote at that time a simple program to listen at the top of a directory tree for change notifications, and scan down/copy over to mirror version. :/ Memory fragmentation fail (after 2 days). Windows (or was it delphi?) doesn't like cleaning up the blocks it allocated, so over time, it'd just grow, and grow, and grow. Plugged in gc, and bug "solved". It was supposed to be a simple one-day to write application *rolls eyes*. I wrote at that time a simple program to listen at the top of a directory tree for change notifications, and scan down/copy over to mirror version. :/ Memory fragmentation fail (after 2 days). Windows (or was it delphi?) doesn’t like cleaning up the blocks it allocated, so over time, it’d just grow, and grow, and grow. Plugged in gc, and bug “solved”. It was supposed to be a simple one-day to write application *rolls eyes*.

]]>