Over the last few days there has been a rather extended outage of this website. It was the unintended side effect of some of the infrastructure work that we wanted to get done, but had not had the time.

As this has always been a side project, time has always been our biggest limitation. So, this time, we (I) got caught out. Started something where I knew there would be no safety net, and then having it not work. Ultimately, we simply ran out of time to resolve the issue when we started the project, and could not get back to it for a day or so. The result? the website was down.

What were we doing? we changed how we deploy updates to the back end of the website, and in the process broke our Continuing Integration processes. The net result? things broke and broke quite badly.

In addition, there was another mistake. We tried to implement multiple changes at once instead of simple step by step. The result, the things we broke in one masked the issue to the point where we spent a couple of hours chasing the wrong problem.

All of these things are things we know better than, but in a moment of limited time, and a desire to GTD, mistakes were made :).

That said, things should now be stable, and we can ( finally ) get back to our regularly scheduled work.