https://blog.vonng.com/en/tags/incident/Recent content in Incident on VONNGHugoen-USSun, 26 Jul 2026 00:00:00 +0000https://blog.vonng.com/en/cloud/huawei-iam/Sun, 26 Jul 2026 00:00:00 +0000https://blog.vonng.com/en/cloud/huawei-iam/Huawei Cloud confirmed abnormalities affecting some International Site accounts during a scheduled IAM upgrade window. The timing is suggestive, but the causal link remains unconfirmed.https://blog.vonng.com/en/cloud/robo-clog/Wed, 01 Apr 2026 00:00:00 +0000https://blog.vonng.com/en/cloud/robo-clog/On the night of March 31, a large number of Apollo Go robotaxis in Wuhan failed at the same time. The real concern is not merely that autonomous driving failed, but that a centrally controlled, cloud-based architecture may amplify a single-vehicle failure into a city-scale systemic risk.https://blog.vonng.com/en/cloud/claude-outage/Tue, 03 Mar 2026 00:00:00 +0000https://blog.vonng.com/en/cloud/claude-outage/Claude went down globally on March 2, 2026. The cinematic theory blamed drone strikes on AWS in the Middle East, but the failure pattern points much more strongly to a front-end and authentication crunch triggered by explosive user growth.https://blog.vonng.com/en/cloud/aws-me-bomb/Tue, 03 Mar 2026 00:00:00 +0000https://blog.vonng.com/en/cloud/aws-me-bomb/On March 1, 2026, Iranian drones reportedly hit AWS facilities in the UAE and Bahrain. If the reporting is accurate, this may be the first public case of a hyperscale cloud provider suffering direct military damage to data-center infrastructure.https://blog.vonng.com/en/cloud/alipay-crash/Fri, 05 Dec 2025 00:00:00 +0000https://blog.vonng.com/en/cloud/alipay-crash/Dec 4, 2025, Taobao, Alipay, and Xianyu all cratered. Users got charged while orders still showed “unpaid,” a carbon copy of the 2024 Double-11 fiasco.https://blog.vonng.com/en/db/no-docker-pg/Thu, 20 Nov 2025 00:00:00 +0000https://blog.vonng.com/en/db/no-docker-pg/Tons of users running the official docker postgres image got burned during recent minor version upgrades. A friendly reminder: think twice before containerizing production databases.https://blog.vonng.com/en/cloud/cf-ck-down/Wed, 19 Nov 2025 00:00:00 +0000https://blog.vonng.com/en/cloud/cf-ck-down/A ClickHouse permission tweak doubled a feature file, tripped a Rust hard limit, and froze Cloudflare’s core traffic for six hours—their worst outage since 2019. Here’s the full translation plus commentary.https://blog.vonng.com/en/cloud/aws-postmotem/Fri, 24 Oct 2025 00:00:00 +0000https://blog.vonng.com/en/cloud/aws-postmotem/AWS finally published the Oct 20 us-east-1 postmortem. I translated the key parts and added commentary on how one DNS bug toppled half the internet.https://blog.vonng.com/en/cloud/aws-dns-failure/Tue, 21 Oct 2025 00:00:00 +0000https://blog.vonng.com/en/cloud/aws-dns-failure/us-east-1’s DNS control plane faceplanted for 15 hours and dragged 142 AWS services—and a good chunk of the public internet—down with it. Here’s the forensic tour.https://blog.vonng.com/en/db/bad-etcd/Wed, 07 May 2025 00:00:00 +0000https://blog.vonng.com/en/db/bad-etcd/Plenty. If you’re rolling your own Kubernetes, odds are you’ll crash because etcd ships with a 2 GB time bomb.https://blog.vonng.com/en/cloud/openai-failure/Sat, 14 Dec 2024 00:00:00 +0000https://blog.vonng.com/en/cloud/openai-failure/Even trillion-dollar unicorns can be a house of cards when operating outside their core expertise.https://blog.vonng.com/en/cloud/aliyun-ha/Tue, 17 Sep 2024 00:00:00 +0000https://blog.vonng.com/en/cloud/aliyun-ha/Seven days after Singapore Zone C failure, availability not even reaching 8, let alone multiple 9s. But compared to data loss, availability is just a minor issuehttps://blog.vonng.com/en/cloud/netease/Sun, 18 Aug 2024 00:00:00 +0000https://blog.vonng.com/en/cloud/netease/NetEase Cloud Music experienced a two-and-a-half-hour outage this afternoon. Based on circulating online clues, we can deduce that the real cause behind this incident was…https://blog.vonng.com/en/cloud/bsod-friday/Tue, 23 Jul 2024 00:00:00 +0000https://blog.vonng.com/en/cloud/bsod-friday/Both client and vendor failed to control blast radius, leading to this epic global security incident that will greatly benefit local-first software philosophy.https://blog.vonng.com/en/db/cve-2024-6387/Thu, 04 Jul 2024 00:00:00 +0000https://blog.vonng.com/en/db/cve-2024-6387/This vulnerability affects EL9, Ubuntu 22.04, Debian 12. Users should promptly update OpenSSH to fix this vulnerability.https://blog.vonng.com/en/cloud/gcp-unisuper/Sat, 11 May 2024 00:00:00 +0000https://blog.vonng.com/en/cloud/gcp-unisuper/Due to an “unprecedented configuration error,” Google Cloud mistakenly deleted trillion-RMB fund giant UniSuper’s entire cloud account, cloud environment and all off-site backups, setting a new record in cloud computing history!https://blog.vonng.com/en/cloud/qcloud/Sun, 14 Apr 2024 00:00:00 +0000https://blog.vonng.com/en/cloud/qcloud/Tencent Cloud’s epic global outage after Double 11 set industry records. How should we evaluate and view this failure, and what lessons can we learn from it?https://blog.vonng.com/en/cloud/smile/Wed, 29 Nov 2023 00:00:00 +0000https://blog.vonng.com/en/cloud/smile/Alibaba-Cloud and Didi had major outages one after another. This article discusses how to move from cost-reduction jokes to real cost reduction and efficiency — what costs should we really reduce, what efficiency should we improve?https://blog.vonng.com/en/cloud/aliyun/Mon, 13 Nov 2023 00:00:00 +0000https://blog.vonng.com/en/cloud/aliyun/Alibaba-Cloud’s epic global outage after Double 11 set an industry record. How should we evaluate this incident, and what lessons can we learn from it?https://blog.vonng.com/en/pg/pg-filedump/Wed, 27 Sep 2023 00:00:00 +0000https://blog.vonng.com/en/pg/pg-filedump/Backups are a DBA’s lifeline — but what if your PostgreSQL database has already exploded and you have no backups? Maybe pg_filedump can help you!https://blog.vonng.com/en/pg/time-travel/Mon, 22 Feb 2021 00:00:00 +0000https://blog.vonng.com/en/pg/time-travel/Machine restarted due to failure, NTP service corrected PG time after PG startup, causing Patroni to fail to start.https://blog.vonng.com/en/pg/extension/Thu, 13 Jun 2019 00:00:00 +0000https://blog.vonng.com/en/pg/extension/Today encountered an interesting case where a customer reported database connection issues caused by extensions.https://blog.vonng.com/en/pg/pg-dump-failure/Tue, 11 Dec 2018 00:00:00 +0000https://blog.vonng.com/en/pg/pg-dump-failure/Sometimes, interactions between components manifest in subtle ways. For example, using pg_dump to export data from a connection pool can cause connection pool contamination issues.https://blog.vonng.com/en/pg/page-corruption/Thu, 29 Nov 2018 00:00:00 +0000https://blog.vonng.com/en/pg/page-corruption/Using binary editing to repair PostgreSQL data pages, and how to make a primary key query return two records.https://blog.vonng.com/en/pg/sequence-overflow/Fri, 20 Jul 2018 00:00:00 +0000https://blog.vonng.com/en/pg/sequence-overflow/If you use Integer sequences on tables, you should consider potential overflow scenarios.https://blog.vonng.com/en/pg/xid-wrap-around/Fri, 20 Jul 2018 00:00:00 +0000https://blog.vonng.com/en/pg/xid-wrap-around/XID WrapAround is perhaps a unique type of failure specific to PostgreSQLhttps://blog.vonng.com/en/pg/download-failure/Sun, 08 Apr 2018 00:00:00 +0000https://blog.vonng.com/en/pg/download-failure/Recently there was a perplexing incident where a database had half its data volume and load migrated away, but ended up being overwhelmed due to increased load.