debugging frustration level >9000: tracking down "impossible" bugs in code

[Darren Dirt]

https://threadreaderapp.com/thread/1191802097551003654.html
https://twitter.com/ErrataRob/status/1191802097551003654

As a programmer, you'll eventually encounter a bug that's "impossible", that you can't figure out no matter what you do. It's a wall that you hit, and it's very demotivating. Programming school doesn't warn you that this will happen, or teach you how to cope with it. 

Big engineering organizations are have two groups of programmers, those that get stumped by bugs who must ask others to help them figure out what's wrong, and programmers skilled at figuring out bugs, helping other people out. 

I think the biggest difference is self-confidence. When you believe you can't solve a bug, then you can't. When you've solved many impossible-to-solve bugs, you believe that you can, and will keep working on the problem until it's fixed. 

When you look backwards at an impossible-to-solve, you wonder about the impossibility of it. It's just an obscure thing, one of million possible things, how could anybody have tracked it down? The above example is a great demonstration: small stack size on VDSO. 

The above story explains how to find 1-in-a-million things: bisection. First, reliably and quickly reproduce the bug. Second, divide things in half, and figure out which half has the bug. 

Do this 20 times, and you'll find the 1-in-a-million obscure bug. It doesn't matter that you've entered obscure territory. The above story makes it seem like the programmer already knew all about the VDSO. But maybe he didn't. 

He continually split the problem in half until he figured out the difference was in the module named "VDSO". At this point, he could research all about the VDSO, going from "never having heard of it" to "the expert teaching coworkers about it". 

When you explain your impossible-to-solve bug to your coworkers, they'll be impressed by how much you know about an obscure corner of tech, and how you could possibly have found such an obscure thing. But you started just as ignorant of the problem as them. 

Confident programmers know they can solve impossible-to-solve bugs not because they know all the technical bits, but because they know they can learn the necessary bits as they go along. 

Confident programmers know that they find a needle-in-a-haystack, not by finding the needle, but by getting rid of the hay until only the needle is left. By increasing the area where the bug isn't, they decrease their search until only the bug is left. 

Anyway, the moral of these tweets is that you've got this, that what the guy did to find that bug wasn't as impossible for you as you think. All impossible to find bugs are actually quite possible. 

The only major thing there was reproducibility. He could reproduce the bug that others failed to reproduce. Once you have that, you have the solution, and nothing will stop you find from finding the solution. 

...

^ this cybersecurity hacker guy's thoughts were in response to this:
 https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/

[Mr. Analog]

Generally speaking I can find most bugs given enough time and resources but that's not always what I'm given. Often there is no time or the proper tools are not available heck, even just getting help on a deep rooted problem can be a challenge.

This is the balancing act that many of us play depending on where we work. Finding the root cause of an issue is surprisingly not always what the business is interested in. Many places value the quick fix over the deep analysis some of the more tricky bugs require.

At least that's how its been at a couple places I've worked.

[Darren Dirt]

I read the linked article AFTER reading the Twitter thread.

Wow, that linked article was low-level heavy duty brain work, lots of Star Trek technobabble to me the unitiated of Go and *nix kernel compiling etc...

BUT the Twitter thread is a nice summary of GENERAL guidelines re. smart / efficient debugging approaches that experienced developers eventually figure out our own after many years of pain... I myself do a lot of out loud "rubber ducking" as I take notes pen on paper -- even before I start stepping through a debugger (which is usually just to confirm my suspicions, or to drill down on some state or values that, like, "there's no way that should have that value" but of course it does etc.