4x4 Action Group

Folks - the service provider where the forum is hosted apparently experienced drive failure sometime early AM. People who were up bright and early would have noticed that the forum was inaccessible. In the absence of any communication from the service provider at that point in time, I presumed that the issue was with the physical virtual machine, and ultimately restored the VM from backup so that I could get it up and running asap. That worked, but since daily backups are scheduled for once a day at 13h00 (this is provided by the SP - I cannot configure this) it means that any posts from 13h00 yesterday have been lost. Sorry about that.

The joke is that just after restoring the latest backup, I received email from the service provider mentioning the fact that they had a critical hardware failure on the host. Which means that if I had perhaps not restored the backup, things may have been as they were. Maybe. Maybe not. No idea. If the drive failure resulted in my database crashing, I'm pretty sure there would have been other issues as well.

I guess I am going to have to not rely solely on the SP and make more regular backups on the host itself. I've got an idea to implement regarding this, but it is going to take a little scripting time which I don't have time for right now. So I'll look at getting this working over the weekend.

Thanks for all the time and effort you put in to keep this forum up and running smoothly for all of us. Greatly appreciated by the masses.

The issue started last night at about 22:00.

Renier_c wrote: ↑

Thu Nov 17, 2022 9:21 am

The issue started last night at about 22:00.

Thanks - I can actually not say when the issue occurred. I didn't have time to do an in-depth investigation as I was more concerned about getting the system up and running again asap.

I became aware of it early this morning, and presumed that I would have been notified earlier if there were issues.

If I may ask a favour, when things like this happen, please let me know somehow asap.

For reference, my cell number is oh-ate-2-double-three-789-ten. Or ping Manny - he'll get hold of me...

Anyways, it's all up and running again. A few posts lost, which is sad. But I'll implement a better "local" backup system over the weekend.

Well done Ian.

thanks

Mad Manny wrote: ↑

Thu Nov 17, 2022 11:31 am

Chas also made a brilliant, clear, concise and illuminating post that gave us info we didnt have.
Now that the post is gone we have no such posts by Chas on the Forum.

Such a pity....

I have yet to see such a post by Chas. The loss of that post is, indeed, an incredible tragedy....

Ian, you yster!

iandvl wrote: ↑

Thu Nov 17, 2022 9:43 am

Renier_c wrote: ↑

Thu Nov 17, 2022 9:21 am

The issue started last night at about 22:00.

Thanks - I can actually not say when the issue occurred. I didn't have time to do an in-depth investigation as I was more concerned about getting the system up and running again asap.

I became aware of it early this morning, and presumed that I would have been notified earlier if there were issues.

If I may ask a favour, when things like this happen, please let me know somehow asap.

For reference, my cell number is oh-ate-2-double-three-789-ten. Or ping Manny - he'll get hold of me...

Anyways, it's all up and running again. A few posts lost, which is sad. But I'll implement a better "local" backup system over the weekend.

Will so. Sorry.

Ricof4e wrote: ↑

Thu Nov 17, 2022 11:53 am

Ian, you yster!

+1

Just closing the loop here. I have scripted a local backup thing which will sync forum content and dump the databases locally every hour. These local backups I keep for 24 hours.

This will allow me to rollback using the SP's backups, but also to roll forward hourly should the need arise. ie: Just so that I do not miss posts that may have taken place from the last SP backup. There is obviously still a slight window that posts could be lost. For example: should somebody make a post and something crashes before the next hourly backup, the post will be gone. But this provides a little more redundancy than previously.

That is some good RPO, well done.