The cloud went down at 03:31.A day later, back to normal.

Waiting for the zone to return wasn’t an option: the provider gave no timeline, and weekend bookings were still on. While the company was buying a new server, we put together a deployment pack from scratch.

The pack listed every service: how it starts, its ports, data and integrations. For the analytics code, the biggest project, it picked the best source for every file among git, working copies and older snapshots, and proved the choice with checksums. One unpleasant finding: about 400 files existed only inside the image on the dead server. The pack also held settings templates (variable names only, no values), install scripts and a four-level launch plan, each level with its own checks and rollback.

Certificates and routes were checked before DNS moved: the new server answered with temporary certificates, and requests went straight to its address. Then the first wave: nine hostnames, including the Art Quest and Morpheus sites, bonuses, the status page and SemCue. They were checked from Moscow, St Petersburg, Germany and the US with real Let’s Encrypt certificates. Every record had a machine-readable rollback. Emergency mode went live with the same wave.

Horror Cinema moved in the second wave, after the company entered the YooKassa shop keys and separately approved the switch. While online booking was down, the sites showed a notice and a form for reaching a manager. The form wrote each enquiry to disk before replying, filtered out bots and duplicates, and let through no more than 5 enquiries a minute from one address. A payment notification was acknowledged only after the payment was checked through the API; otherwise YooKassa was asked to retry later.

Code is the easy part. Data is harder: bookings, guest bonuses, gift cards, the payroll journal. The last full backups were on the same disk. A snapshot of the old disk never happened: 39 attempts ended in cloud errors, and 36 more attempts just to stop the machine got “service is overloaded”.

So the data was rebuilt from what we had, following rules agreed with the company in advance. The main one: an unknown balance can be neither zeroed nor guessed.

Then came a surprise. The old server came back to life and started polling Telegram with the same bot keys. Two servers were pulling the same messages and getting in each other’s way. Only reissuing the tokens fixed it.

Three workstreams ran at once. One brought up the server, bots and apps. The second rebuilt the data. The third switched DNS, republished client reports and kept trying to snapshot the old disk.

To keep them out of each other’s way, every action went into a shared log with UTC, Tashkent and Omsk times. The rules were simple: each stream writes only in its own folder, copies rather than moves, deletes nothing, and keeps an index and checksums in every folder. Payments, messages to customers and DNS changes only after an explicit “yes” from the company.

On the night of 9 October the websites and booking went back to normal, and the final check passed all 45 items. Then we turned to on what had made the outage so painful.

Not everything came back: some data from the last few weeks had nothing left to rebuild it from. Exactly what was lost is recorded in the recovery log, not hidden. The code for the main services now matches the live server, and backups are checked, not just counted: on 14 October the network dropped the download five times in a row, so half an hour later the script started over and accepted the copy only after the checksums matched.