Plays with YouTube's privacy-enhanced embed

Dev Deletes Entire Production Database, Chaos Ensues

Name: Dev Deletes Entire Production Database, Chaos Ensues
Description: Kevin Fang comically retells GitLab's 2017 incident in which an engineer deleted the production database, based on the company's postmortems. It walks through the confusion and failed backups.

Kevin Fang Comedy

Open on YouTube First HN thread

1 HN user recommended this comedy video in 1 different month between 2023 and 2024.

2007–2026

1 recommendation 2 citations 1 joke HackerTube score 1

Shared as a funny, accurate retelling of a classic outage, especially on a bad ops day.

Kevin Fang comically retells GitLab's 2017 incident in which an engineer deleted the production database, based on the company's postmortems. It walks through the confusion and failed backups.

Pull quote: “The reenactment by Kevin Fang is always a good source of comedy on a bad day.” (HN, 2024)

Also tagged: database outage postmortem, gitlab, postmortem, production database, devops, incident

What Hacker News said

4 labeled comments from the HN archive; up to 20 are kept per video, favoring different commenters. Each one links back to the original thread.

Recommends xyst August 2024 under Gitlab is reportedly up for sale

Comedic reenactment of the 2017 GitLab incident

Every time GL is mentioned, I am reminded of the 2017 incident where a person 'rm -rf' the primary pg data logs. The reenactment by Kevin Fang is always a good source of comedy on a bad day. [this video] It’s a shame they are selling out, but I suppose it was bound to happen sooner or later. GH is vacuuming up all of the market share. I kind of liked the ability to self host my own GL. But for small/personal use cases. It was a bit overkill.

View on Hacker News

Citations, criticism and jokes

Cites it sovietmudkipz July 2023 under Gitlab.com is down

Retelling of GitLab's production database deletion

Here is a retelling of the time when gitlab went down because someone deleted a production database: "Dev Deletes Entire Production Database, Chaos Ensues" [this video]

View on Hacker News

Cites it laserDinosaur September 2023 under Nginx Unit – Universal web app server

Lack of CLI feedback worsened the GitLab outage

"When a program has nothing surprising to say, it should say nothing." Which sounds good in theory, until you run into issues like this: [this video] TLDW: Most of the Gitlabs outage where they deleted the primary prod and backup prod database could have been solved if they just waited for the command line op to complete - but because of a lack of feedback they thought it had frozen, leading to several other plans which made the situation worse.

View on Hacker News

Makes a joke renegade-otter April 2024 under ElephantSQL Is Shutting Down

If the primary fails, will the replica fail over? How do you know? Are the backups working? Who's checking? Can anyone delete a directory when tired at 6PM on a Friday? [this video]

View on Hacker News