I really wish the default response in dev and operations wasn't "somebody is going to get (yelled at|punished|fired|pile on layers of bureaucracy)". It would be so much better if the response was "how can we change automation in our release process to catch this and the engineer that did this is responsible for fixing it". Blameless post-mortems should be the industry standard.
Indeed, but this is a big "you should have gotten that in QA" kind of mistake. With the aggravating factor the the company markets security as one of iOS tentpoles... Accountability is a bitch, I guess, but in this case this was really in plain sight. I mean, it's not necessarily the dev's ass that's going to be kicked.
There's a nice story about accountability at Apple once recalled in a profile by Adam Lashinski. It's from the Steve Jobs' era, but I guess this kind of policy is still enforced in Cupertino:
One such lesson could be called the “Difference Between the Janitor and the Vice President,” and it’s a sermon Jobs delivers every time an executive reaches the VP level. Jobs imagines his garbage regularly not being emptied in his office, and when he asks the janitor why, he gets an excuse: The locks have been changed, and the janitor doesn’t have a key. This is an acceptable excuse coming from someone who empties trash bins for a living. The janitor gets to explain why something went wrong. Senior people do not. “When you’re the janitor,” Jobs has repeatedly told incoming VPs, “reasons matter.” He continues: “Somewhere between the janitor and the CEO, reasons stop mattering.” That “Rubicon,” he has said, “is crossed when you become a VP.”
Indeed, but this is a big "you should have gotten that in QA" kind of mistake.
It depends what kind of infrastructure you have.
Ideally there should be NO MANUAL STEPS between dev and production. This should not be any need for a person to remember that this data should be inserted for dev and removed before prod. Instead there should be something like a documented way to insert the data or not based on a configuration file that is different for dev and production.
Of course reaching that keeping that ideal requires discipline. And when you slip, it becomes easy to say, "That's QA's job." Over time QA's job will get harder and harder, and that is a guarantee of periodic slip-ups.
Authority for fixing this situation really is the job of someone VP or above. They need to decide whether to fix the process and change what a lot of people do, or accept the inevitable screw-ups as the right choice to maintain the current rate of development.
I want to preface this by saying that if this comes off as condescending that's not my intention.
> Blameless post-mortems should be the industry standard.
Sure, often times there are multiple reasons why something isn't accomplished, something is broken, or something doesn't work as intended. But generally there is someone that's primarily responsible for the piece of code, be it a manager, engineer, etc. It does suck to be on the receiving end of it and if it's major enough you can be out of a job, but blame should be given to the responsible party/parties and the issue should be handled appropriately as it's more direct than what alternatively happens which is to passive-aggesively handle the situation. Plus, how you handle crises like production going down and being blamed for it say just as much about you as an engineer as does your code.
Essentially, I'm against being coddled as an adult. We're adults, someone fucked up and is responsible for it, they should be blamed and handle it, and handle it like an adult. I do think that handling that kind of situation as person on the giving end definitely requires tact, though.
But generally there is someone that's primarily responsible for the piece of code, be it a manager, engineer, etc. It does suck to be on the receiving end of it and if it's major enough you can be out of a job, but blame should be given to the responsible party/parties and the issue should be handled appropriately as it's more direct than what alternatively happens which is to passive-aggesively handle the situation.
The problem is deciding what is the actual goal of a post-mortem. Is it to fix the problem (and prevent it from happening again)? Or is it to figure out who is the person most to blame so that they can be punished. Because, honestly, you can't have it both ways.
Plus, who makes the final judgement as to the party to be blamed and based on what criteria? Is it QA for not catching the issue, is it the engineer for forgetting to remove some debug code, the tech arch for not properly ensuring there were systems in place to prevent it from occurring in production, the program manager for not pushing back on deadlines that were too aggressive, or the CEO for constantly pushing employees to prioritize revenue over product quality?
Plus, how you handle crises like production going down and being blamed for it say just as much about you as an engineer as does your code.
Or your willingness/ability to find a suitable scapegoat.
The problem is that post mortems then become politicized, which means the actual cause may not be exposed, rendering the whole exercise pointless. It's part of the reason the FAA has all kinds of safe guards around reporting aviation hazards, even if the person who reported them is to blame.
Reminds me of the story of a guy who makes a $50,000 mistake. He goes to his boss saying "So I expect you're going to fire me now" and the boss answers "Are you kidding? We just invested $50,000 in your training!"
Should I consider myself lucky that all my bosses have had this mentality? Is ass-kicking actually more prevalent in the industry than rational correction?
