Tough week Mar 3, 2010

We're having a tough week at Jitbit

First, our email server was down for some time.

Then, one of our developers had accidentally removed a file from the web based helpdesk software source codes. Its all right, we have backups. But! An automated backup procedure was initiated shortly after, overwriting the backup copy with the wrong data.

Then, the new version of Helpdesk was automatically published without this critical file, causing a lot of errors from our customers. Our Hosted Helpdesk was partially down for 20 minutes (was not accepting new tickets via web interface).

Somehow I managed to restore the file, even though it was stored on a TrueCrypt-volume (and kudos to TrueCrypt - its volumes even support undelete tools).

Now, how we plan to prevent this in the future.

Our backup plans and deployment procedures are reworked to prevent conflicts and overwrites. But most importantly, we're moving all our source-codes to Subversion. (btw Jeff Atwood has a great article about it). We also decided to host the Subversion database at a third-party server, and picked ProjectLocker as our provider. There's also a free Subversion plug-in for Visual Studio called Ankhasvn which we strongly recommend to all bootstrapping startups.

PS. We're currently working hard to add iPhone-support to our forum software, very excited about it. Please send us any thoughts and suggestions on this.


Blog Archive