Infrastructure update

July 22nd, 2008

After our outage last week. Here’s a comeback on the events.

We found that our servers did not go down; the apache servers did (albeit incompletely without releasing the socket/port) This made the browser to wait forever to connect to the webserver, eventually timing out. This delayed the loading of sites which are part of SezWho community.

To add to this hubbub; the night before, our SMS quota on the monitoring service ran out and as such we did not get an immediate notification (One of our developers out of Bangalore, caught this at around 4am his time). We could have resolved this problem in less than 10 mins if the alerts came as usual. The SMS quota has been set to infinity now :-)

This week, we got our new environment setup with bigger irons and we have started moving a few things there. We’re setting it up in a way to have the browser request time out immediately if the servers are not available. We are also exploring suggestions of externalizing the CSS.

We take the quality of our service very seriously and will be up and will do everything in our power to make sure an outage like this never happens again. We are thankful to our community members for their support, cheers (and jeers!)

Rate this:
4.1 (7 people)
Posted by User ImageIndus Khaitan Filed in infrastructure

4 Comments

Post a comment   |   Trackback URI   |   Comments RSS feed

Filter Comments

  •  
    no image
    Roger (Who am I?)

    4 months ago

    I knew you all what get to the bottom of it!

    Reply | Rate this:
    • Rate this down
    • Rate this up
    		3.5
  •  
    no image
    Greg Larkin (Who am I?)

    4 months ago

    Hi Indus,

    Thanks for posting the information about the outage and the steps you’re taking to improve your infrastructure.

    One thing that jumped out at me is that you use Fedora Core for your server operating system. Can you elaborate on the reasons for that? I don’t use FC, but my impression of it has been that it’s very current, but possibly less stable than its relatives like RHEL and CentOS. This is only my impression, which may be completely wrong.

    I also respectfully submit that FreeBSD works great as a production operating system. Of course, I’m hugely biased, since I use it for SourceHosting.net, and I recently became a project committer. :)

    Thank you, and keep up the good work,
    Greg

    Reply | Rate this:
    • Rate this down
    • Rate this up
    		3.9 (2 people)
  •  
    no image
    Indus Khaitan (Who am I?)

    4 months ago

    Greg,

    FreeBSD was my original choice along with Debian, but I haven’t installed even a single distro of FreeBSD personally; so kept away from that (I should do that sometime on one of my home servers).

    Fedora Core 7 is stable, there have been two more major revisions after that — The problem I see with FC distros is that the open source community has to really catchup in making sure that the existing tools work with active FC versions. We had to compile (not a big deal, though) some tools locally in order to use their recent tars.

    The next time we plan for a rebuild I’m gonna definitely look for something else.

    Indus

    Reply | Rate this:
    • Rate this down
    • Rate this up
    		3.7 (1 person)
  •  
    no image
    Greg Larkin (Who am I?)

    3 months ago

    Hi Indus,

    If you’re ready to try out FreeBSD, I’ve got a VMware image ready to go with version 7.0 pre-installed. There’s a full ports tree (needs updating by now), so you can install any of 18k+ applications with the simple command “make install clean”. Here’s the link:

    http://torrents.sourcehosting.net:10692/torrents/FreeBSD_7.0_VMware.zip.torrent

    Cheers,
    Greg

    Reply | Rate this:
    • Rate this down
    • Rate this up
    		3.1

Trackbacks/Pings

  • No trackbacks or pings yet

Leave a Comment

Comment template by SezWho