Concerning The Latest Service Outage [PSXE Official]

PSX Excessive lately suffered from a relatively extreme server failure, which took our web site offline between December thirtieth, and January sixth. This was the longest unplanned outage in our 24 12 months historical past.

So, what occurred?

To try to simplify a relatively lengthy story, the database server that powers PSX Excessive malfunctioned. Whereas we had tried to restore the database server quite a few occasions, our makes an attempt at repairing have been  unsuccessful. Actually, we really ended up making issues worse. The database that powers our website, turned irreversibly corrupted.

Our solely actual resolution at this level, was to fully wipe our server clear, and reinstall every thing from the bottom up. On paper, this could have been a straightforward factor to do. Reinstall the working system, reconfigure our management panel. Straightforward. Time consuming, little question. However most positively a straightforward job.

Besides, completely nothing had gone accurately.

Downloading The Backups

PSX Excessive has 4 major and completely totally different backup strategies. Every methodology is meant for use for a unique sort of {hardware} or software program failure. As an example, we again our database, posts, pages, and first directories, as much as the cloud as soon as each 24 hours. This methodology of backup is nice for when we have to shortly revert again a day or two. The downsides? It doesn’t again your entire web site and all directories up, however relatively, it’ll solely again up what’s required to maintain the core of our web site operational. In different phrases, absolutely the fundamentals.

We additionally create a full backup of our whole web site, and all directories inside our principal net folder. This methodology of backup is an actual reproduction of our web site because it appeared on the date the backup was created. Sadly, we solely run this clone-based backup methodology as soon as each seven days, which for a high-content web site like PSX Excessive, isn’t essentially the most preferrred of options. Nevertheless, it’s a fallback that’s practically assured to work.

That can be the backup methodology we opted to make use of.

The precise act of downloading the backups from a server, and storing them on our native drive, took roughly 36 hours. PSX Excessive is a big web site, and incorporates over 400GB of whole knowledge.

Straightforward, however time consuming.

Restoring The Backups

Sadly, that is the place issues began to take a flip for the more serious. Whereas the act of downloading the precise backup information wasn’t overly difficult, simply time consuming.  The identical can’t be mentioned for the restoration course of.

We needed to add the compressed backup information to the server, after which run a restore command. Sadly, each single time that we tried to try this, the restore course of failed. We tried to do that a number of occasions, losing roughly three days. Every time, the backup would get to about 95%, after which cling for a number of hours, earlier than finally failing. Since we needed to restore a relatively giant file, having the restore course of cling was regular and anticipated. Having it crash? Not as regular or anticipated.

As soon as we acquired the positioning restored, we tried to revive one among our cloud backups, to get as near our earlier live-site as we may. Sadly, restoring the cloud backup ended up corrupting our database, requiring that we wipe the database and reinstall the unique backup once more. Every time we needed to do a brand new restore, we must sit and babysit the restoration course of for a whopping 4 hours.

So now, an extra eight hours have been wasted on simply attempting to revive a working backup. However lastly, it was completed. Issues have been not crashing. All was good on this planet!

And Now We’re Right here

PSX Excessive is again on-line. Issues should not absolutely secure fairly but, however on the very least, we’re practical. We are able to as soon as once more contribute content material to our website, and all core performance is sweet to go.

And but, issues are nonetheless reasonably unstable. We’re gradual, and have a number of visible bugs and glitches which have but to be mounted, as of this writing. However no less than we’re again on-line, proper?

I wish to thank everybody in your endurance. Restoring PSX Excessive was no simple job, even when it was alleged to be a straightforward job on paper.

Preventative Measures

To try to make sure that this by no means occurs once more, we’ve got applied a brand new caching methodology into our website, which ought to velocity issues up relatively considerably. Past that, we’re additionally going to be creating full cloned copies of your entire public listing each 24 hours, to roughly match our cloud-based backup providers.

We’re additionally going to rely lots much less on distant cloud backups, contemplating as how these haven’t, thus-far, been of any actual worth. This was alleged to be our most safe, and most dependable, methodology of backup and restoration. However as a substitute, it turned the least dependable of the bunch.

We may also be wanting into the opportunity of internet hosting our web site on a unique internet hosting community. Proper now, we run our personal servers, and roughly present and do every thing ourselves. That is positive for when it really works, however as we simply found, is an actual ache within the ass for when issues hit the proverbial fan.

All in all, we’re again on-line. Hopefully for good this time round.

Associated