All systems operational

Rebuilding Servers - Disaster Recovery Validity Check

Completed
Scheduled for Sunday, May 30, 2021 at 6:00:00 AM – 6:46:16 AM

Affects

Email Hosting services
Email servers
Webmail apps
Web Hosting services
Web servers
Web Analytics servers
Updates
  • Completed
    Sunday, May 30, 2021 at 6:46:16 AM
    Completed
    Sunday, May 30, 2021 at 6:46:16 AM

    Restoration was successful. There may be just a few minutes of downtime intermittently over the next 20 minutes approximately, as I continue to make a few small adjustments which require a reboot of the servers. Thank you for patience. Happy to report the backup and restore procedure was a success, and finished a bit earlier than anticipated too. :-) If there are any questions or if you notice any odd behaviours, please let me know right away.

  • Update
    Sunday, May 30, 2021 at 6:24:07 AM
    In progress
    Sunday, May 30, 2021 at 6:24:07 AM

    The restore is nearly complete. There may be a period of time of intermittent unavailability as systems come back up.

  • Update
    Sunday, May 30, 2021 at 6:11:37 AM
    In progress
    Sunday, May 30, 2021 at 6:11:37 AM

    Restore of data is in progress.

  • In progress
    Sunday, May 30, 2021 at 6:01:28 AM
    In progress
    Sunday, May 30, 2021 at 6:01:28 AM

    Maintenance is now in progress.

  • Planned
    Sunday, May 30, 2021 at 6:00:00 AM
    Planned
    Sunday, May 30, 2021 at 6:00:00 AM

    tl;dr

    There will be approximately 60 minutes of downtime tonight at 11 PM Pacific Time while I perform an annual test of the backup & restore process by rebuilding the servers from their backups.

    Why?

    It is best practice to perform a complete restore of the server from backups at least once a year in order to verify that the restore process is still working, so that in the very rare chance of a catastrophic event in the future, we can be confident in the backup and restore process. This is a critical part of any "Disaster Recovery" plan. Companies have been bit by not testing their backups before and unable to restore, so I want to avoid any of those situations and follow the industry-wide best practice.

    A virtual "snapshot" of the server will be taken prior to the maintenance time, so in the incredibly low chance that the regular restore process fails, I will then use the snapshot to restore from instead.

    There will be no loss of data! Everything will be safe.

    Expected Impact

    I anticipate approximately 60 minutes of downtime. I will keep this maintenance event updated as I progress through it tonight.

    Questions? Concerns?

    If there are any concerns with the timing of the maintenance window or if you have any questions at all, please reach out to me via email: dustin@d19.ca