The valley of engineering despair

I have delivered a lot of successful engineering projects. When I start on a project, I’m now very (perhaps unreasonably) confident that I will ship it successfully. Even so, in every single one of these projects there is a period - perhaps a day, or even a week - where it feels like everything has gone wrong and the project will be a disaster. I call this the valley of engineering despair. A huge part of becoming good at running projects is anticipating and enduring this period¹.

The start of a project always feels good. I have a clear idea of what needs doing, and there’s plenty of time to do it. The very end of a project usually feels good too - by that point all the important pieces are ready, and it’s just a matter of getting the final tweaks and bugfixes in. The hard part is the middle of the project, when all these things are happening at the same time:

• You’re discovering that some of the things you thought would be easy are actually surprisingly hard
• New requirements have come in (this always happens)
• Now that you’ve generated some tangible screenshots and demos, it’s turning out that you and your product counterpart aren’t as much on the same page as you thought
• The project deadline is starting to feel real

I always feel a little overwhelmed by all this. The trick is to recognize that the feeling is temporary - you can always work through it - and to make sure you don’t do anything stupid in the meantime². For instance, don’t try and solve the remaining problems in an all-nighter: you’ll just panic everyone else and probably cause more bugs than you’d solve. Alternatively, don’t start ranting and raving about how the project is doomed and it’s impossible to work under these conditions. You’ll just force your manager to have to talk you down. (As always, it’s a good idea to avoid becoming a problem your manager has to solve.)

What should you do, then? Slow down and take one problem at a time. Once you’ve solved that - either by directly solving it or by raising it with your product/management/design counterparts and getting agreement to cut it from the scope - move on to the next one.

Start with the scariest problem. If you can solve it, that’ll be a big morale boost. If you can’t solve it, it’s good to know that as early as possible. Putting the scariest problem off is sometimes necessary if you just can’t bring yourself to do it, since it’s always better to do something than nothing, but you should be careful to not put it off for too long.

Overcommunicate about the specific problems you’re running into. If you can, literally write them up as dot points somewhere public. Your product manager and manager may be able to solve them instantly (e.g. by cutting scope). If not, at least they’ll have a sense of what you’re doing.

But really, specific advice about going through the valley is beside the point. The main point is to expect to despair at some point in an engineering project, and to make sure you don’t panic when it happens.