Skip to main content
Background Image
  1. Cloud Exit Mudslide/

Database Deletion Supreme - Google Cloud Nuked a Major Fund's Entire Cloud Account

·536 words·2 mins· ·
Ruohang Feng
Author
Ruohang Feng
Pigsty Founder, @Vonng
Table of Contents

Due to an “unprecedented configuration error”, Google Cloud mistakenly deleted UniSuper’s cloud account.

The Australian pension fund executive and Google Cloud’s global CEO issued a joint statement apologizing for this “extremely frustrating and disappointing” outage.

https://x.com/0xdabbad00/status/1789011008549450025

Due to Google Cloud’s “peerless” configuration mistake, Australian pension fund UniSuper’s entire cloud account was accidentally deleted. Over half a million UniSuper fund members couldn’t access their pension accounts for a week. After the outage, service began recovering last Thursday, with UniSuper stating they would update investment account balances as soon as possible.


UniSuper CEO Peter Chun explained to 620,000 members on Wednesday evening that the disruption was not caused by a cyber attack, and no personal data was leaked during the incident. Chun clearly stated the problem originated from Google’s cloud services.

In a joint statement from Peter Chun and Google Cloud global CEO Thomas Kurian, both apologized to members for the outage, calling the event “extremely frustrating and disappointing.” They pointed out that due to a configuration error, UniSuper’s cloud account was deleted, an unprecedented event on Google Cloud.

Joint Statement from UniSuper CEO and Google Cloud CEO

Google Cloud CEO Thomas Kurian confirmed that the cause of this outage was an oversight during the setup of UniSuper’s private cloud services, ultimately leading to the deletion of UniSuper’s private cloud subscription. Both stated this was an isolated, unprecedented event, and Google Cloud has taken measures to ensure similar incidents won’t happen again.


Although UniSuper typically sets up backups in two different geographical regions for quick recovery during service outages or losses, the deletion of the cloud subscription also simultaneously deleted backups in both locations.

Fortunately, UniSuper had another backup with a different vendor, so they ultimately succeeded in restoring service. These backups greatly mitigated data loss and significantly enhanced UniSuper and Google Cloud’s ability to complete recovery.

“The complete restoration of UniSuper’s private cloud instance was inseparable from the tremendous focused effort of both teams and close cooperation between both parties.” Through joint effort and cooperation between UniSuper and Google Cloud, their private cloud was fully restored, including hundreds of virtual machines, databases, and applications.

UniSuper currently manages approximately $125 billion in funds.


Cloud Exit Feng’s Commentary
#

If Alibaba Cloud’s global service unavailability major outage could be called “epic,” then this Google Cloud outage deserves to be called “peerless.” The former mainly involved service availability, while this outage struck at the core of many enterprises - data integrity.

To my knowledge, this should be a new record in cloud computing history - the first such large-scale database deletion. The last similar data integrity incident was Tencent Cloud and the “Qianyan CNC” case.

But a small startup company and a major fund managing hundreds of billions are completely incomparable; the scope and scale of impact are completely incomparable - everything under the entire cloud account was gone!

This incident once again demonstrates the importance of (off-site, multi-cloud, different vendor) backups - UniSuper was lucky, they had other backups elsewhere.

But if you believe that public cloud providers’ data backups in other regions/availability zones can “cover your back,” then please remember this case - Avoid Vendor Lock-in, and Always have Plan B.

Reference: Guardian UK report on this incident

Related

Cloud Dark Forest: Exploding Cloud Bills with Just S3 Bucket Names
·1275 words·6 mins
The dark forest law has emerged on public cloud: Anyone who knows your S3 object storage bucket name can explode your cloud bill.
Cloudflare Roundtable Interview and Q&A Record
·2774 words·14 mins
As a roundtable guest, I was invited to participate in Cloudflare’s Immerse conference in Shenzhen. During the dinner, I had in-depth discussions with Cloudflare’s APAC CMO, Greater China Technical Director, and front-line engineers about many questions of interest to netizens.
Cloudflare - The Cyber Buddha That Destroys Public Cloud
·2089 words·10 mins
While I’ve always advocated for cloud exit, if it’s about adopting a cyber bodhisattva cloud like Cloudflare, I’m all in with both hands raised.
Can Luo Yonghao Save Toothpaste Cloud?
·811 words·4 mins
Luo Yonghao’s livestream first spent half an hour selling robot vacuums, then Luo himself belatedly appeared to read scripts selling “cloud computing” for forty minutes — before seamlessly transitioning to selling Colgate enzyme-free toothpaste — leaving viewers bewildered between toothpaste and cloud computing.
Analyzing Alibaba Cloud Server Computing Cost
·641 words·4 mins
Alibaba Cloud claimed major price cuts, but a detailed analysis of cloud server costs reveals that cloud computing and storage remain outrageously expensive.
Will DBAs Be Eliminated by Cloud?
·3607 words·17 mins
Two days ago, the ninth episode of Open Source Talks had the theme “Will DBAs Be Eliminated by Cloud?” As the host, I restrained myself from jumping into the debate throughout, so I’m writing this article to discuss this question: Will DBAs be eliminated by cloud?