Reflections on software

Best practices are constantly evolving. Sometimes so much so that we're hearing the opposite today of what we heard yesterday.

But there are some fundamentals that tend to stay the same though. One that most of us can agree one is that bugs can be expensive. More specifically, the later we find bugs, the more costly they are to fix. On a whiteboard, the cost of fixing a bug is just about nothing. It can be as simple as replacing one diagram with another. But in production, the costs could be catastrophic, including loss of current or potential customers, direct loss in financial applications, or human life could even be at stake.

On one side of the spectrum, we have the school of thought that tries to solve as many problems initially as possible. After all, if it's cheaper to fix bugs earlier in the process, let's just spend more time earlier in the process. In practice, this doesn't work so well for a number of reasons. The biggest one is probably just the fact that requirements change during the process itself. There's not really anything we can do in this case. The other pitfall here is that most tough problems in the programming space are relatively new. We don't have much experience with these problems, so it's difficult to account for or even predict the tough spots. Usually we have much more insight into problems after we've tried to solve it the wrong way a few times.

I think what typically ends up happening with this approach is that some areas get over designed, and others don't get the attention they deserve. This can lead to getting the abstractions wrong, and leaky abstractions can make bugs difficult to prevent. Not knowing what invariants need to hold for a system can be a big source of errors.

The extreme/agile approach looks at it from a different perspective. Instead of trying to imagine all the scenarios and details up front, let's just do what we can to discover them early in the process. To use a general oversimplification, it boils down to ignoring complexity in some areas in order to create a working prototype faster. The trick then is knowing which parts of the problem to ignore and which to focus on. Trying to tackle the hard parts is usually a solid strategy, except sometimes the hard parts aren't what we think they are.

Agile development also claims to engender quick changes. If the abstractions still fit, and we have a testing suite underneath us, then yes, the changes can be really fast. But if the abstractions fit, then any system is fast to change. When a requirement change breaks the current model though, or forces an interface change, then I would argue that making changes can be even slower, because they propagate out to more code. We not only have to refactor our logic, but the logic in all relevant tests as well. Some would argue that means the tests aren't written well or aren't testing at the right level. But writing tests at the right level is hard. It can take some time before we can learn how to write effective tests, just like it takes time before we can learn how to write effective code.

But the biggest concern I have with agile development is that it can be used as an excuse for poor judgement. Being lazy is not agile, it's being lazy. Always taking the easy way out usually catches up to us, and then it can be painful to dig out of that hole.

Here are my takeaways:

1. If we don't understand the problem, we can't process our way out of it. In other words, if we don't know what the issues really are, no amount of process can help us.
2. We learn best from our own mistakes. This informs our future decisions much more so than any process could.
3. Context matters most.

We try to keep things simple. After all, nobody starts a project thinking, let's make this one complicated for a change. Yet somehow, things like to end up that way.

I like to think of this as a shock, but it's usually not that surprising, all things considered.

Perhaps the most obvious reason is that programmers knowingly introduce this complexity themselves. This complexity is justified by rationalizing that it will anticipate future situations, and account for them elegantly. The goal here is to prevent future maintenance headaches, but ironically it can add complexity if the wrong abstractions are made.

The opposite can also be just as true. As code design debt accumulates, not acting early enough once a problem is recognized can also lead to increased complexity.

Programmers can also have drastically different opinions on subjects. If we take a look around the software world, we'll see that there are many operating systems, langauges, frameworks, and a myriad of all kinds of tools. Each one of these represents a particular area where a programmer identified a gap, and then proceeded to make an attempt at filling it in. This naturally leads to overlap among similar software. This overlap itself isn't bad, in fact it's probably a little bit healthy. Each competitor can focus on what it feels is important, and users can choose accordingly. But merely having more anything can contribute to its own kind of complexity to deal with. After all, it's easier to make a choice given fewer options.

But along with the lack of consensus on what's important, programmers also don't think the same things are unimportant. Some are willing to let documentation drift out of sync. Some aren't interested in repeatable builds. How about automated tests? Commented code? Consistent style? Design patterns? Stable back ends? Intuitive front ends? Practically everyone will agree that it's important to have many of these items, but not everyone will agree on which one is ok to let slip. These differences can help contribute to software complexity as programmers fight against each other.

Programmer 1: I never knew it would be so hard to push a car downhill.
Programmer 2: What do you mean, downhill!?

There's also the idea that things aren't worthwhile unless they are complicated. When things are simple and easily understandable, it's easy to suggest avenues of improvement. But if too many distractions are entertained, the codebase can start developing warts that can be difficult to deal with.

It is easy to make things. It is hard to make things simple.

When faced with a new problem, we start thinking of the different approaches we can take to solve it. We look back on previous experiences, and see if we've solved anything similar in the past. If so, maybe that same type of solution can be applicable to the current problem. If not, maybe it can be adapted. Pattern recognition can be a powerful tool to help us solve problems.

But with our previous experiences come our own biases as well, directing our trains of thought. If my previous problems all had a large focus on security, chances are high that I'm going to be keeping security in the back of my mind. While this is beneficial for preventing security holes, it can also work against us if I try to inject sophisticated security mechanisms when they aren't needed, just in case.

It's important to remember exactly what we're supposed to be building. Who is it for? What are they going to be doing with it? Is this new feature going to help? Is it necessary? The more frills we add on, the greater the chances that the motivation for what we're doing will be lost. Programmers can have a tendency to make things more complicated than they have to be.

At the onset of a project, this is easier to keep in mind. But as time goes on, we keep our heads deeper and deeper in the details, and we can lose the forest for the trees. We add tweaks here and there, spend some time on peripheral tasks, and then get caught surprised wondering why it is we're so far off mark.

It's important to note that what we're building can change over time. In fact, if it's a long project, it probably should change. If not, what we're building may no longer meet its demands.

So when we're lost, we should remember to take a step back, take in a deep breath, and ask ourselves what we're supposed to be building.

Ask a programmer to hang a picture frame, and you'll get a new wall instead.

Always two there are, a master and an apprentice

When learning a trade, the master/apprentice relationship can be tremendously useful. It is a slow path, but an effective one. It's a great way to pass on the lessons from the past.

And not only is the knowledge of skills passed on to future generations, so is the knowledge of common pitfalls. As an apprentice begins to make mistakes, the master can correct them immediately, potentially avoiding much of the headache associated with recovering from the error. And we all know that the earlier an error can be recovered from, the smaller the consequences.

However, you don't see very much of this in the programming world. I think there are a few reasons for this.

1. Being self taught is encouraged. If you can learn yourself just by rtfm, then you haven't wasted anybody else's time.
2. Programmers all think that they can solve the problem better themselves.
3. There tends to be a high turnover in the programming industry.

But then, each generation is doomed to repeating the same mistakes. And with programming, it's difficult to know when you're making a mistake. But, a master can point this out right away, and correct it before it becomes a major problem. This can save an enormous amount of time, and greatly speed up the learning process.

Programmers, and those expecting programming based solutions, are usually impatient. There's enormous pressure to release the next best thing yesterday. The programming field also changes rapidly, and it will probably continue to change at an even faster rate in the future. It should be no surprise then that programmers are in a rush.

But the flip side is that it takes a while to become a great programmer. We really need to experience many failures before we can identify how to architect successes. And contrary to popular belief, it takes years to become proficient.

If there are no silver bullets, then I think that encouraging master/apprentice relationships can help propel us as a whole in the right direction faster.

We've all had our frustrating moments with computers. We bang our heads against the walls for quite some time, and no matter what we try, the computer responds with a clever "I thought you might try that, here's your error." And then, we try talking to someone else about it, and they usually have a brilliant idea that solves everything elegantly. Then we're left scratching our heads, wondering why we didn't think of that before.

Lots of problems get solved like this simply because they are looked at from a fresh perspective. It's easy to get lost in the details of a problem. When we keep our heads down, it's hard to realize that we were just approaching the problem from the wrong angle.

Programming is a game of insight.

I think that sums up the essence of programming. The most significant gains are often those that shed the problem in a new light.

But not all reevaluations of a problem lead to successes. In fact, I would argue that most of them don't. But the ones that do work, usually do so in a big way. Given that programming is this give and take process, progress often isn't linear. There isn't a lot of progress, or it looks like things are getting worse, and then suddenly, there's a big jump.

This makes it especially difficult to measure, assuming it's even possible to measure at all. Any formal attempts at measurements results in programmers optimizing for the local maxima. This ends up detracting from productivity.

But regardless of whether we can keep track of it or not, it's important to foster an environment that encourages creativity. A single idea can change everything.

We all go through programming disasters. It's hard, if not impossible, to always make the right decisions. And after we've steered the ship back on course and we're out of the storm, we try to review if there was any way we could have prevented the storm in the first place. This reflection is crucial, and is one of the best ways we can improve ourselves.

What becomes dangerous however, is when these reviews turn into policies, or mandates. To explain why, I'll summarize an anecdote which I've come across from the Extreme Programming book.

A mother was baking a ham with her daughter, and she noticed that the ends were cut off. She asked her mother why, to which the mom responded: "I don't know. That's the way my mother always did it. I'll ask her." So the mother asked the grandmother why the ends of the ham were cut off, and she said: "I don't know. That's the way my mother always did it. I'll ask my mother". And the great grandmother's response was: "My oven was too small, so I had to cut off the ends to have it fit".

Blindly following policies can lead to extra steps that can work against us. Rather than come up with new policies, it's better to come up with principles that can inform future decisions. It's more important to understand the reasons behind what those policies would have been.

On that note, dogma itself is dangerous. When we start blindly following rules, we start becoming simple machines. It tends to stifle creativity, which is the worst thing that can happen to programmers. Programming itself, after all, is a creative process.

I've seen a trend by programmers to be completely against any form of duplication. After all, there are extremely compelling motivations for this idea. Programmers have all copy/pasted code to get something working, with the reason being that it usually ends up working much faster in the short term. But then when all that new code needs to get updated, we've all forgotten to make the change to all the pieces that required it. And bingo, we have a new bug.

Then we look back on it and what's to blame? Not that we forgot to make the change everywhere, which was just the symptom. The problem was that it was possible to change one of the parts, and forget to update the other. It should have been refactored to avoid the duplication, so that a change in one location would naturally affect all the other paths. This would have prevented the bug from even being possible.

But like most dogmas taken too far, this can get you into trouble. Here's a very contrived example, and granted most of us don't think this way, but I've experienced somewhat similar arguments for avoiding duplication where I thought it was just silly.

a = 7
b = 3 + 12
c = 18 - 2
d = 9 * 6

Try to pretend that these are real calculations, and there are not just hardcoded numbers here. Can you spot the duplication? Normally you'd say there isn't, but I can argue that there is. You have four assignments, and 3 mathematical operations. Isn't that duplication? Here's the "reduced" version:

vars = ['a', 'b', 'c', 'd']
arguments = [(), (3, 12), (18, 2), (9, 6)]
ops = [lambda *x:7, operator.add, operator.sub, operator.mul]
for var, args, op in zip(vars, arguments, ops):
    globals()[var] = op(*args)

Notice how I've eliminated all the duplication? And wasn't it clever of me to fit the simple assignment in the first example to a no-op? The logic is all now in a single line compared to the 4 up above. I can argue that although it's actually more lines of code, I can effectively move all but the loop into configuration. Now people can add more variables to the global namespace, with an arbitrary operation performed on any number of arguments, all through configuration! What a wonderful and extensive system I've created!

The perceptive reader will have discovered that I was being a tad bit sarcastic. (My co-workers will all tell you that I'm subtle). So which is easier to understand? Which would you rather maintain? Readers with no python experience will probably understand the first code snippet. But you probably need to know python to even attempt to understand what's going on in the second.

So if you're going to follow dogmas, policies, or rules, then follow this one:

Always use your brain.

We all have different ideas on when we should clean things up. It could be a dirty room, cluttered desk, messy closet, or just too much stuff lying around. We also prefer to keep things a certain way, which can be very individualistic. From the outside a desk with tons of paper lying around could look like it could use some rearranging, but the desk's owner might be able to find anything at a moment's notice.

Given how different we all are with tolerating visual clutter, it's not too surprising that there's a wide range of opinions on when we should clean up our code. Many books have been written completely focused on this very issue, and in the end, I think it's still much more of an art than a science. In fact, I believe programming itself is much more of an art than a science, but that's the topic for a whole other discussion.

Then there's a time for spring cleaning too. And we're always surprised when we find some really dirty stuff in the nooks and crannys. But in the real world, we usually do end up cleaning it up. (That is, if you're not as lazy as I am). In the virtual world however, some cleanup tasks are truly daunting, and it may be easier to leave the dirty things the way they are. After all, it might not look pretty, but it works.

And similar to how the owner of a cluttered desk can find something for us quickly, old dirty code tends to work predictably as well. But if we change it a bit, we can no longer guarantee that it'll work the same way. Unit tests can help here, but we still can't make any guarantees. And when you move an old couch from one corner of the room to the other, you realize that there was a whole lot more dust under there than you thought. That dust also has to get cleaned up. And after some time cleaning up, we ask ourselves, was it really worth it? That couch really wasn't all that bad where it was.

But, the advantages gained by cleaning up can outweigh its costs and risks. After all, if something becomes simpler and easier to understand, we stand to gain every single time someone works with it. Add up the time for all these future interactions, and the refactoring can more than pay for itself.

However, like weatherman and traders already know, it's hard to predict the future. (They'll never admit it though. What I think these guys really excel at is coming up with excuses. :) ) But, it's very easy to predict the past. And although we may pay a small price each time we end up working around the dirt, we start learning where all the dirty areas are. This puts us in a better position to clean up more effectively in the future.

Refactoring usually alters the abstractions used. A new layer could get added that simplifies complex interactions by handling those details. Or extra layers that get in the way are removed to produce simpler code. But if a new problem comes along that doesn't fit into these new abstractions, then the game is up. Chances are we'll put in a hack to work around that "edge case", and those can start to pile up, especially if other programmers start putting their hands in the mix. And before you know it, we'll be talking about a new refactoring.

I'm not arguing that we should never clean up code though. I'm just pointing out that there are often more factors involved when thinking about cleaning up than just making the code prettier. What cleaning up does seem to do is improve morale. Most programmers would much rather write a 1000 new lines of clean, sparkling code, rather than try to figure out the ten lines in a 10000 line messy codebase that need to be modified. And when bold, noble undertakings like these are launched, I start to feel like Braveheart just gave me a speech before an impossible battle.

And what usually ends up happening is that the small splinter cell team beats the odds. But why? There's too much work and not enough time. Well, I think that the answer is simple: the programmers work harder. And the reason for that is that they are a whole lot more motivated with the prospect of a fresh start instead of trying to keep a big ball of mud together.

In the end, major refactorings are a tough call either way. And the technical issues might not be the whole story. Social issues can also play a role in the decision too. Programmers tend to take attacks on their code personally, and arguments can turn into personal vendettas quickly.

But just like we're always able to find ways around a messy desk, we find ways around our technical problems too. And regardless of the decision, thinking about the problem gives us more insight into it. So ultimately, we're better off anyway after the exploration step. To use a cliche:

The journey is more important than the destination.

When faced with a new programming challenge, it can usually be approached from one of two ways: the front end or the back end. Each has its merits, and completely focusing on one or the other is a recipe for disaster. But, I think that most programmers tend to be a little "backend heavy". I'd almost go so far as to say that it's difficult to call yourself a programmer without being biased this way. After all, programmers enjoy solving complex problems by creating new powerful abstractions. If not, you'd be a little nuts to go through all the frustrations of programming without enjoying watching something work.

What I find surprising is that if a problem is too simple, it's the programmers themselves are the ones that create more complexity. I find myself doing this all the time. "Well what if the user wanted to do foo and not bar. Instead of hard coding this action, I can add a new <insert cool pattern/framework/abstraction here> and the application will then support any action. Then we can make a user preference and new configuration management system that users can tailor to their needs". This cascades of course, and before you know it, you've created a new framework. It's just so easy to get lost in it, that programmers sometimes forget to ask the obvious question: "hold on a sec, why are we doing this again? Let's just cross that bridge when we get to it".

Focusing completely on the front end can be just as problematic though. Security holes, performance bottlenecks, and general lack of flexibility can hold you back too, perhaps even more so. Any one of these issues can be devastating. Then it'll leave you wishing you had spent just a whee bit more on prevention, instead of having to pay so much for the cure.

Right about now it sounds like my point is that everybody should just do things right from the beginning, without wasting time building anything unnecessary. But realistically, I think the most important thing is to "use your brain". The applications that programmers build are supposed to be helping users in some way. Refactoring is supposed to help programmers maintain their code. Writing tests is supposed to help maintain confidence in the code base. User testing is supposed to help inform usability improvements to the application itself. If something isn't working like it should, maybe it needs to change, or be rethought. It's always healthy to take a step back and review how things are going. I think Einstein said it best:

Everything should be made as simple as possible, but not one bit simpler.