Ever since I mis-titled Terry Chay's talk, I've been thinking of putting together a presentation called The Internet is an Orgy. Just like a real orgy, it can be a lot of fun but it just takes one person to screw everyone.
That point has been getting proven today. Amazon's S3 service is experiencing it's second outage of the year. According to Amazon's status site, they're been down for roughly 6 hours right now. Sites like SmugMug are completely down as they rely 100% on S3 to serve photos from. Avatars on Twitter are broken because Twitter uses S3 to offload that content.
And the poor people using EC2. EC2 isn't directly effected, but you have to have access to S3 in order to bring up another server. This is six hours that you've had to make due with just the servers you have running right now. All of that "elasticity" is just that - a marketing word in air quotes if you can't bring more servers online.
Don't think you're immune to this type of problem just because you're a consumer. Case in point, if Ma.gnolia goes down I loose access to all of my bookmarks and the ability to store anything else. I bookmark a lot when I'm wondering around the web.
I've blogged about distributed systems before though. Whenever I see a central point of failure, I cringe, but we move more and more the way of the centralized which I feel is against the very grain of the internet.
There are notable exceptions. Erlang, my current favorite language, is designed to be distributed across machines with little more than a network connection and not fail if that connection disappears. Git enables distributed version control and is being used as a distributed data store for everything from open source projects to distributed ticket tracking systems.
I'm sure if I looked, I could probably find a few more examples. This isn't meant to be a riff on centralized services like Amazon's S3 or EC2. They are great and can be very valuable to scale out quickly, especially when you're just starting out. There is the proverbial "but", however. Don't think they won't go down just like any other service Treat them just like any other cluster. If you can't afford downtime, have a backup plan.
I'm coining a new phrase. Expertise Syndrome.
expertise syndrome (n):
- becoming so skilled in a given topic, methodology, etc., that you can't discuss it with someone who is lesser skilled than yourself as you leave out significant parts by deeming them "common knowledge".
- when knowledge within a community becomes "common knowledge" and is no longer discussed at which point new comers often find it hard to find basic information.
The seeds of this started last summer when Paul M. Jones posted TDD, Test-First, and Ravioli Code. Paul is a really smart guy, but somehow he missed (and he still does) the point of TDD. The only explanation I can come up with for this is that we, the TDD community inside PHP, haven't properly conveyed our message of peace, harmony, and code with fewer bugs and programmers with fewer white hairs since they we're validating our work as we go in the form of executable specifications.
We had apparently left out a key part of our case: namely that unit tests provide us with a way to manage the inevitable change that comes with software by ensuring that the entire system is properly exercised in a repeatable way. Write a test, make it pass, then move on to the next task. Then, once you start to see how things interact, you start refactoring. You already have your expected outcome being verified by the tests. Now all that's left to do is incrementally add new tests to verify the new functionality you add in and refactor the existing code. Your tests serve as the stutter bumps on the side of the highway, giving you an early warning by failing if things aren't going correctly. On the flip side, they can also be used to track down all of the places in your code that are affected by the change you're making providing the true lazy man's methodology: "oh I'll just change my code for the better and my tests will tell me what I need to fix".
But I digress... I ran into another case of my own expertise syndrome few weeks ago with a colleague of mine. We've started a concerted effort at work to get unit tests implemented around our product and since I'm the resident TDD guru, I got tapped to give a presentation on unit testing. I put together about an hours worth of slides on the basic concepts, determined not to do the standard 2 + 2 test case and show some "real-world" use cases mixed in with a few of the test cases that we already have in our test suite along side the basic syntax and theory someone would need to get started writing test cases.
I gave the presentation I would have loved to have been privy to some 4 or 5 years ago when I started down the TDD path. These guys left the presentation having at least heard of "four phase tests" and "intent revealing names". Based on the feedback I've received and the number of people who are now committing tests, I think it went over well enough and that everyone felt like it was a good use of their time.
A few weeks after the presentation, a new test was committed that broke another, unrelated test. An undefined index error or something similar. I started investigating and within just a few minutes found that one of the super-globals had been set explicitly to an array in the new test case's setup() method. The new value didn't contain the value that should always be there, so the next test case that relied on it failed. Classic case of an interacting test and something I just took for granted and a concept I completely ignored in my presentation.
I found myself doing this again a few months ago when I was working on an article for php|architect on PHPT. I was 3/4th of the way through the article before I realized that I had never explained how to actually run the software. Seriously. I had covered installing it, both from its channel and SVN, how to write the tests, and had started writing how to extend the testing framework before it dawned on me that I'd missed the most crucial part: how to run it.
I experienced this syndrome from the opposite side when I made the switch to Git. There's tons of information, but most people inside that community have apparently been using it long enough that they didn't realize that saying "type git init" wasn't enough to convey that doing so created the repository and stored it in the ".git" directory and that you didn't have to checkout a copy of your repository. I spent an afternoon trying to figure it out. I consider myself a pretty intelligent guy, so heaven help the guy who's only exposure to version control thus far has been TortoiseSVN.
Today I had even further confirmation that this is a real syndrome that we, both individually and collectively contribute to it. I received a message from Paul Greenberg in response to notes on his keynote at SugarCon. He specifically addressed my criticism that his slides had too much information on them. He said it I might be right, but that he wanted to include it so he could provide all of the information and context whenever he sent his slides to someone. Fair enough, but I would have never thought to do this since I use S5 and it has the concept of a "handout" section that you can stuff in that narrative or contextual information that's normally in your presentation. My guess—and I might be wrong—is that Paul's never heard of S5 and that the ability to provide contextual information along side the slides that is hidden unless you're "reading" the slides (either by printing or turning the slide show off) is something new to him. And I'll bet this isn't the only tool that I take for granted that others might find useful, if only they knew it existed or how to use it.
Battling this disease
There isn't a cure, but I think its symptoms can be managed allowing its sufferers to effectively interact with the plebeians. ... removing tongue from cheek ... As a programmer, one way to hone your "novice thinking skills" is to constantly add new languages to your repertoire. There's no need to become an expert in them, but learn their methods for approaching problems. Try to think in that language. You'll be amazed at how the shift in paradigms will force you to rethink your assumptions. I recently picked up Erlang and the shift to functional from imperative programming was—no, still is disorienting.
Another suggestion is to explore new tools. See if they fit with how you work by committing to using them for a week or two. It doesn't matter what they are. Could be a new version control system, IDE, or finally putting all of your tasks in some sort of task manager. The point is to try to mix it up and keep from getting into a rut. Note that a rhythm and a rut are two completely different things. The former is great and can make you much more productive, the latter will cause you to stagnate.
Constantly approaching something new with curiosity should help keep you in a novice mindset. The next time I'm writing an article or preparing a presentation I'll make sure to keep "why?" close by. Why is this the best practice? Why should I use this pattern over that one? You've gotta love those three little letters.
