Incident Investigation with Wesley Carter

Blaine Hoffmann: In this episode I am going to share with you an interview I did with the Amplify Your Process Safety podcast folks. Wes Carter and I talked, and what we had to say is extremely applicable to not just PSM but also to general industry, and I wanted to share it with you on my podcast. So consider this a takeover episode, and I want you to go check out the Amplify team and follow the links in the show notes. I hope you enjoy this episode where I talked with Wes on the Amplify Your Process Safety podcast. Let’s get into it.

Welcome to the SafetyPro Podcast, powered by iReportSource, helping you manage safety one episode at a time. With the constant regulatory and workplace culture challenges businesses face, we’ll provide you with all the relevant information necessary to achieve a safer, more productive workplace. No management theories, platitudes or guru speak, just actionable info you can use right away. Now here’s your host, the safety pro himself, Blaine Hoffmann.

Wes Carter: Welcome to the Amplify Your Process Safety podcast. My name is Wesley Carter, and today I have a special guest, Mr. Blaine Hoffmann. Blaine, why don’t you go ahead and introduce yourself for the listening audience.

Blaine Hoffmann: Thanks, Wesley. As Wes mentioned, I’m Blaine Hoffmann. I am the producer and host of the SafetyPro Podcast, and we met through the podcast, so it’s kind of fitting we wanted to do some sort of a joint venture on some topics. I’m looking forward to it. Thank you, sir.

Wes Carter: Yeah, I was really looking forward to this, and unfortunately with travel restrictions, I would prefer to do this in person, but we adjust, right? We make it happen. So today’s topic, like you said, we’re going to do some joint episodes. You know, Blaine, I don’t like to put us in boxes, but for our listening audience, we’ve done an episode together back at the VPPPA conference in New Orleans.

Blaine Hoffmann: Yes, sir.

Wes Carter: Back when we were allowed to travel and do things face to face. But like you said, we adjust, absolutely. And so in that episode we talked about the differences and the overlap, the differences and the similarities between process safety and occupational safety, and that’s the flavor of these episodes we’re doing together. In today’s topic we want to talk about incident investigation. For the listening audience, what you can expect is some key takeaways from this episode from both a process safety and an occupational safety slant: what does the regulation say about incident investigations, what is it specifically in general industry and in PSM and RMP, and why is it important. So, Blaine, I’m going to go ahead and get into what the regulation says for PSM and RMP, and then we’ll talk through that, and then I think we’ll run into some stuff that’s found in general industry, because it’s not as explicit as it’s written in the PSM and RMP regulation.

Blaine Hoffmann: Perfect.

Wes Carter: So what does the regulation say? OSHA 1910.119 paragraph (m) in the OSHA PSM standard says the employer shall investigate each incident which resulted in, and this is important, or could reasonably have resulted in a catastrophic release of highly hazardous chemicals in the workplace. So they’re basically saying you have to do an investigation if you have a release of these highly flammable, highly toxic chemicals, because obviously the hazard potential is pretty high for these. And then they’ve got a few requirements in there that say the investigation has to be initiated as soon as possible, but you can’t go more than 48 hours following the incident. The team has to be pretty diverse, and so at a minimum consists of at least one person knowledgeable in the process, include a contract employee if the incident involved the work of a contractor, and other persons with appropriate knowledge and experience to thoroughly investigate the incident. And then they give some guidance on what the report’s minimum requirements are: the date of the incident, the date the investigation began, a description of the incident, factors that contributed to the incident, and any recommendations resulting from the investigation.

And then there’s another part which is kind of a staple in the regulation, which is the employer shall establish a system to promptly address and resolve the report findings and recommendations. You have to document that, and the report shall be reviewed with all affected personnel whose job tasks are relevant to the incident findings. And then incident investigation reports shall be retained for five years. Now Blaine, RMP just kind of points over to the PSM, similar type stuff. But when I read through that, and I’d like you to transition into some of the stuff that’s in general industry, does any of that sound like a bad idea? What’s your take on what I just read through?

Blaine Hoffmann: Well, it seems pretty straightforward to me, and it’s what safety professionals would expect in the workplace, certainly.

Wes Carter: Yeah. When I read through this part of the regulation, it makes complete sense to me. You had a release, or under slightly different circumstances you could have this catastrophic release, and then they’re saying investigate it, make sure you have a good team. If you involve the contractor, make sure they’re involved in the investigation. If you had any findings, make sure you track them to closure, retain the report, share the findings. It’s pretty good stuff. So what do we find, in your experience, when we get into general industry, general occupational health and safety?

Blaine Hoffmann: Well, we don’t find that level of detail. We have, under the 1904 standard, 29 CFR part 1904, the general recordkeeping and reporting requirements. They basically spell out the forms that you have to fill out. Those forms will direct you, in the context of certain questions, to go seek an understanding as to how something happened. So if you were to look at the OSHA 301 form, you are allowed to use your own accident report as long as that accident report has, at a minimum, the exact same information the 301 requires. But the 301 form is really where you’re going to find this investigation, I’m using air quotes here, where we have some key questions such as: what was the employee doing just prior to the event, what caused the accident. It’s very vague. It actually is not an approach that we would recommend in general industry at all, when we talk about root cause, sort of like what you laid out, where we have defined interested parties that need to be involved, things like that.

So in general industry, it’s really incumbent upon each employer to deploy an approach that is suitable for their work environment. And the aim, ultimately the goal, is to prevent future incidents, to get to these root causes so that you can fix what led up to that event. So it’s kind of vague. It’s not a great approach, but it’s one that we’ve been managing over the years, as you know.

Wes Carter: You know, towards the back end there you started getting into why incident investigation is important, and you were talking about the approach, that filling that out isn’t something that’s recommended. And I’ll get into a fact sheet here a little bit in the podcast about investigating someone who slipped on a puddle of oil all the way to investigating a vapor cloud release of a toxic chemical in the facility. Why is incident investigation, in Blaine Hoffmann’s point of view, important?

Blaine Hoffmann: Well, the main reason is to determine what led up to the event so that you can prevent a similar event in the future. That’s just the very high level, elementary answer that I would give you. There are a lot of things that go into that, though, and this is where people get hung up. Obviously we want to prevent injuries and incidents, even minor ones. And in order to do that, we have to dig deep and really understand all of the potential contributing factors that led up to that event or incident.

Wes Carter: Yeah, you said something there. Even the minor ones. Why is it important to investigate minor incidents as well?

Blaine Hoffmann: So a great example is, and I could get into Heinrich’s triangle and then start that whole debate as to whether that’s even valid, or has or has not been validated.

Wes Carter: That’s the Heinrich’s triangle, that’s the one that says for every so many near misses there’s a first aid, and it kind of works its way up, right? Until ultimately you reach the top of the pyramid, or top of the triangle.

Blaine Hoffmann: Yeah. If we’re trying to stop the serious incident or fatality, to which there are a number of first aids, to which there are hundreds of near misses, then we have to investigate starting at the base. But, not citing that specific study, this theory, this approach: okay, if I have a minor incident, let’s say somebody almost slips on oil on the floor. They don’t slip and fall, though. They lose their footing, catch themselves, and they keep walking. And we’re like, wow, we dodged a bullet there, and we all just shrug and walk on. It’s just a matter of time, right, before we’re going to go back and look at that scenario again, except this time it was a fall. Maybe it was a broken arm or wrist, maybe it was a concussion, or God forbid worse, but you’re going to be looking at that same scenario.

So intervening at the minor level, every time one occurs, you’re going to learn about similar contributing factors. Do we do maintenance in this area that’s not cleaned up? Do we have equipment that’s leaking? Was it a contractor that came through this area and dripped some stuff? Or is it a pipe that’s leaking? Is there a valve that’s faulty? What’s the PM on that? Where did we buy that one? Do we have similar ones? And this is getting into your category, but you see where this goes. Why wait for a catastrophic release or a fatality or a lost time injury to actually start digging into these things. So the approach you take to all of them should be uniform, if anything for a matter of exercise that you go through this. What’s frustrating for us in general industry is that we do take a different approach sometimes when it’s a minor incident versus lost time. Employers get all worked up when we say, oh, we had a lost time injury, we have to do certain things and jump through certain hoops. And we probably just got done talking about two first aids that were minor, dealing with machinery, equipment or guarding that’s loose or something like that, and we didn’t get all worked up about it. So that’s the issue we run into in general industry as a result of not having some clear written guidance like you find in PSM.

Wes Carter: Oh, the guidance is clear, but I can tell you, because the regulation right at the beginning, when I was reading, said you have to investigate an incident that resulted in a catastrophic release of a highly hazardous chemical, flammables, something registered in the appendix of the regulation, one of those chemicals, or, under different circumstances, had the potential to. So that in itself leaves enough gray area that we run into the same thing when you’re talking about these PSM and RMP regulated facilities: what is categorized as a, I’ll use the air quotes again, PSM incident. It’s amazing for me. I do investigations, and I do a lot of what I’ll call PSM incident investigations. So we had a release of highly hazardous chemicals, and maybe we burned down some equipment, or we had a pretty substantial fire, or we had a pretty good vapor cloud release without ignition, something like that. And it’s happened more than once where I’ve sat in the investigation, and when we’re building a timeline and I’m doing some interviews with personnel, I’m asking, what happened, has this happened before? And multiple times.

I’ll go back to my early days as a facilities engineer, upstream. I supported a certain field, and they had a positive displacement pump that caught on fire. And I was talking to the operator out there, because he came up on scene, he shut it in. It was small enough to put out with a fire extinguisher, and then he reported it. And after speaking with him: that was the fourth time that a piece of equipment, a small overpressure protection device, that PSV, had cracked in its connection to the piping on the discharge side and had started spraying hydrocarbons. That was the fourth time. The first three times it was just spraying the hydrocarbons, flammable, without ignition. And it was the fourth time, when it actually caught on fire, that they went and reported it and we did a big investigation.

That was a very minor, small piece of equipment. But you never know, someone could have been in the area at that time. And the damage to that pump, cost wise, maybe we’re talking $50,000 if that. But that happens often when we get into this area where people are talking about, well, under different circumstances it could have been this or that. And then you get into the difficulty. If I’m walking around a facility and I’ve got these overpressure protection devices, sometimes they go to a closed system like a flare. And for the audience: you have this PSV, this pressure safety valve, that’s sitting on a vessel, and if it comes to an overpressure scenario, it’ll relieve. That way the vessel can stay within its mechanical limits, its maximum allowable working pressure. So you’re walking through a facility, and some of these PSVs go to flare, sometimes they go to atmosphere depending on what’s inside. And if an operator walks by and a PSV just kind of chatters and sprays for just a second and then closes, it’s hard to get an operator to report that. You don’t see the same amount of effort go into the major events versus the very minor ones, right? There was some oil on the floor, but we saw it and we reported it, and no one was hurt, there’s no direct consequence. It’s not treated the same as if someone had slipped and broke their leg and it was a day away from work case, right?

Blaine Hoffmann: Right. Hey, misery loves company, that’s what I’m hearing. So the language in the PSM standard at least gets us a little closer, in that they identify interested parties and some things like that. On the general industry side, everywhere else I should say, we’ve got to go to some Q and A or FAQs to get some guidance, and even then it’s not even enforceable. But on the OSHA 301 recordkeeping form, the incident investigation, if you will, I’m using that term loosely, there are four questions. It’s actually five, but the fifth one mostly doesn’t apply. There are four questions: what was the employee doing just before the incident occurred, what happened, what was the injury or illness, what object or substance directly harmed the employee. And then the last one is, if the employee died, when did death occur. So that’s it.

Wes Carter: Oh, that’s all we have. That’s a whole 20% of the questions, if there was a death, right?

Blaine Hoffmann: I know. So we can go to, there is an FAQ, and I’ll shoot you the link to this, it’s the OSHA instruction for the Recordkeeping Policies and Procedures Manual. Way down in the Q and A section, the question is: may employers attach missing information to their accident investigation or workers’ compensation forms to make them an acceptable substitute form for the 301 for recordkeeping purposes. So yeah, you can use a workers’ comp form or other form that does not contain all of the required information the 301 has, provided the form is supplemented to contain the missing information. So they’re saying, yeah, you could use a fishbone, you can use an FMEA, or if you’ve done some sort of bow tie analysis of the event and you’ve identified things prior to the event and different outcomes, you can supply all that as long as you fill in the other things that the 301 form asks for. And there are other things on that form, like date of hire, date of birth, address, things like that. But yeah, it’s very vague. And this is why, as a consultant, and you’ve just illustrated this, we see wide variations in how different organizations approach different types of incidents. There’s no consistent way of approaching this that I’ve found.

Wes Carter: Yeah. And I think also, you’ve probably had similar experience to this: when you get into an incident investigation, unless you work for a large organization that’s established, like, this is our methodology, everyone has the best methodology out there. And what I mean by that: I do the five why, I do fishbone, I only do bow tie, I only do Apollo, I only do, what’s the other one, TapRooT.

Blaine Hoffmann: TapRooT, yeah. That’s the best, the other ones are all crap. That’s what you’ll hear. HOP is the other one, human and organizational performance, which is a big one now, and I like aspects of it. And you’ve heard my past episodes on some of these, the ones that you’ve named. But you’ve got to fit the tool to the circumstance, to the event. Some events, like you mentioned some valves or some mechanical degradation, traditional root cause analysis lends itself very well to things that break. Processes that fail when people are interacting with them, it’s a little different. You may need a combination of traditional root cause analysis, five why or fishbone, and more of a human and organizational performance approach to the human aspect of it. But you’ve got to be well versed in many of these approaches. I don’t think there is a silver bullet. Have you found yet that one thing that works?

Wes Carter: No, no, no. And that’s why, I think you said it well, I mean, you’ve got to have a number of tools in your kit, depending on what you’re trying to investigate, and what’s the right tool. And oftentimes, when someone will ask, what types of incident investigation methodologies have you been trained in, if you leave off their favorite, they’re like, well, you obviously don’t understand what the best methodology is, or something like that. And I think it really is: they all work, you know, just, are you trained in it? Do you have sufficient experience? And is it the right approach for the type of incident you’re investigating? I think that’s kind of my thoughts on it. But with all of them, you’re eventually trying to get to, I’ll use some buzzwords here, the root cause. Blaine, what is a root cause? We have an incident. What is a root cause?

Blaine Hoffmann: Root cause to me is, and I know there’s some word nerd out there that’s going to chime in on this, and that’s okay, but to me, in my mind, a root cause is just one potential contributing factor that set off a chain of events, either directly or indirectly, that led to the outcome we’re looking at. And I always say causes, there’s probably multiple root causes. And again, I’d ask you, in your experience, especially on the PSM side, have you come across this single point of failure? Even single point of failure is sort of a misnomer. How did we get a single point of failure? There were probably multiple things that had to happen to allow that single point of failure to exist. So to me, a root cause is just one of the contributing factors that were at play, that we could touch and we could influence, thus changing the outcome. If you watch Back to the Future, we were watching those over the weekend.

Wes Carter: Oh, I just burned through those this last week.

Blaine Hoffmann: Nice. Yeah, they’re on Netflix. It’s a great example: if you touch one thing, multiple things change. If you touch multiple things, all sorts of stuff changes. So it’s sort of that same theory. To me, a root cause is just one thing we could have influenced to potentially change an outcome.

Wes Carter: I like what you said. In my experience doing these, it’s rare that I can point to a single one. I’m not saying it’s never, but it’s rare. Oftentimes it’s a couple at a minimum, and then the dive in. But regardless of how we define it, there’s an understanding of what a root cause is. And why is it so important when you do an investigation? Because I remember when I was first getting trained, I took a week long course, I worked for a supermajor in oil and gas, and I took a week long course in root cause analysis. And they just beat it into us. They would show us reports where they had this incident, they went through the whole thing, they had all these causes and stuff, but none of them were really a root cause type. And basically they allowed, or they left, the environment or the workplace in such a condition where the event could be repeated, even though they had all these recommendations. So why is it important to get to the root cause of an incident?

Blaine Hoffmann: So I will use an example, and I know we planned on talking about this, but you kind of led into this, which was perfect. Let’s say human error is a root cause, that’s what we determined led to this event. You still have to go deeper. The whole point of the root cause is to almost go to a point where you could not have controlled anything anymore, right? That you’re getting to a point that’s almost ridiculous. It should feel ridiculous. If it doesn’t feel that you’ve gone to the nth degree with this, then you’re probably doing it wrong. So, human error. And I learned this working in aviation, even with the DOD; they were pretty good at going deep. If there was an incident and you deemed it human error, you would have to define what created or caused or led to the error. So let’s say fatigue: a worker wasn’t paying attention, the worker was tired. Okay. What was the last shift worked? How many hours did they work the last shift? When was the last break? Even getting into, is there something going on at home? Do they have a second job? What led to the fatigue? So you have to keep going deeper. You can’t just say human error, a worker needed to be coached.

And you mentioned training, the training you went through. There you go. Let’s say that you went through that training, and you just described it, it still left you wanting, right? It didn’t really go deep enough. And let’s say folks come out of that training, they’re performing to the training standard, and they miss potential contributing factors to incidents, thus leading to recurrences, like a trend. And let’s say higher ups in the organization, or a regulator, is saying, hey, you’ve had multiple incidents of this type, what’s going on? Well, we followed the training, we did the investigation according to our training. Okay, so lack of training, improper training, could have been a contributing factor to the trend that you’re seeing. And so what does that tell you? That doesn’t really say much, right? Well, how was the training designed? When was it last updated? Who oversees training? Do you audit training competency? Do you have a rhythm around retraining? What about when industry standards change, or, I hate to use the word best practices, but best in class approaches or tools are developed, do you have anything to keep up on the training? So you have to go to that ridiculous level, if that makes sense, to really get to a true root cause. I often see folks just scratch the surface and say, oh, I coached the employee because they indicated they didn’t really know they had to do it that way. Really? What’s your onboarding process look like? Did you go and talk to any training personnel? Is there a written procedure for what they were doing? Is it posted? Is it well known? There are so many other questions you need to get to. And we’re just talking about this off the cuff on a podcast, but you can imagine how this should be structured. This should be a managed process in and of itself, and it has to be audited as well.

Wes Carter: Yeah, I think the way you’re going about this, why it’s important: there are systemic things. Organizations have certain things that are systemic. And if you put human error, and like you said, the result is, oh, the employee said they didn’t understand. Well, the next human that they hire, in their onboarding process, if that’s where they should do it, is there ever a box they have to check to inform a new employee of that requirement? And so you really start to get at some systemic things, some management things, some policies that are lacking, when you start pointing to things like this. When I do human error, I often try to put myself in this, I don’t know why I do this, it seems very simple, but I call it operator A, operator B. If I’m ever going to put operator error down somewhere, I think operator A, operator B. That way I can really put myself in their shoes to say, okay, operator A did this in this scenario. If I bring a new operator B into this scenario, is the system set up in such a way that that new operator could have the same failure?

Blaine Hoffmann: Yeah, and that’s a great approach, because one of the other pitfalls, not to go down a rabbit hole of do’s and don’ts of incident investigation, but one of the pitfalls is confirmation bias. And one of the most frequently cited biases that I see is shoot the messenger. So let’s say Blaine reports that he pulled the muscle in his shoulder lifting an object, and everybody rolls their eyes and goes, oh, well, you’ve got to know Blaine. And I would say, no, I don’t actually. I don’t have to know Blaine. I just have to vet out: is what Blaine’s saying valid. That’s it. So objectivity is key. Having a team approach, having multiple people trained to do this. It doesn’t matter if Blaine’s got that personality where he’s always griping about something. And I’ve told my teams in the past, and even now today: we have to be right 100% of the time. It’s that one time we just shrug it off and go, oh, well, you just don’t know Blaine, and something bad is going to happen.

But these tools, these approaches, the various tools in our kit, these are all things that we can pull out to confirm, to validate or invalidate something objectively. And that’s key too. And one of the best things you said, operator A and operator B. One of my favorite things to do is say, okay, well, Blaine says that this is the way he’s lifting or moving these objects or operating this machine. You obviously go through the manuals, you go through the SOPs, you go through the training, but you also go to another operator. Go do another observation of somebody else doing the same job, maybe on a different shift, maybe under a different supervisor, maybe one that’s a little newer, maybe one that’s been around for a while. Try to find some other examples of this task being performed and check against that. And if there is a differential, why? Now you’ve got to chase that. There are so many cool little things you can do to really do this justice. I think the injustice is we just say, oh, well, that’s Blaine, he probably just doesn’t know what he’s doing, so I coached him. Meanwhile, we changed nothing about the environment and the way I have to do what I do. You’ve just made it more difficult for me to do it by putting more restrictions on me, but not changing the physical task itself or the materials that I’m dealing with. You kind of roll the dice, right?

Wes Carter: Yeah, right. You’ve got the same environment, it could happen again, unless you’ve got that operator, that employee, that is better than the rest, the diamond in the rough. No, that’s good. So I wanted to point to some resources for our listening audience that I think will be beneficial for both. One thing that we’ll include: OSHA and the EPA put out a series of informational fact sheets, and there’s one they put out in 2016. This fact sheet is a joint venture between OSHA and EPA, and it’s the importance of root cause analysis during incident investigation. It’s a pretty good read. They go through the PSM and RMP requirements, how to conduct an RCA, the benefits of root cause analysis for employers, root cause tools. And what I really like about this: they have a resources section, and this is only a two or three page document, and one third of it is just resources, and it’s really good. They point to some Center for Chemical Process Safety books, like the Guidelines for Investigating Chemical Process Incidents, Guidelines for Root Cause Analysis by the Department of Energy, a handbook for accident and operational safety analysis. Just lots of good references, if someone is looking for a good one to pick up and read. We’ll include a link to that fact sheet as part of the release. And then, Blaine, this is something you’ve spoken to on your podcast at least more than once, but I think you wanted to point to one specifically, right?

Blaine Hoffmann: Yeah, episode 82. I did a series on safety management systems, and episode 82 was part three: what is root cause analysis. The whole series gets into what is systems thinking, what is PDCA, what is RCA or root cause analysis. So I would point to this one in particular for what we’ve been talking about, because it ties together some of the things that we talked about, but also some things listed in that fact sheet. There are a lot of great resources out there, and training. And getting back to what you mentioned earlier: don’t just latch onto one because it’s comfortable or familiar to you. Diversify a little bit. You never know when you’re going to need a different approach.

Wes Carter: Yeah, absolutely. I really appreciate your time, Blaine, on incident investigation, because I think we’ve both had quite a bit of experience in this kind of area, but from different approaches, me supporting process safety, you being an expert in occupational health and safety. So I really appreciate your take, because just like we have similarities, we also have a little bit of differences, and that’s the flavor of the industry and the disciplines we support. But I think, for our audience, as a closing: organizations benefit from investigating incidents of any type, process safety, a slip, trip and fall. Organizations benefit from this, because without this proper analysis, investigating to a root cause, incidents can reoccur, and that’s what we’re trying to prevent, right?

Blaine Hoffmann: Absolutely. And I read it in a lean safety book years ago, but it talked about: focus on the process, not the person. It is about process improvement. And I will tell you, an assistant secretary of labor years ago, Lord help me, I’m going to quote the assistant secretary of labor, but on this topic the quote was: accurate records are not simply paperwork, but have an important and in fact life-saving purpose. They will enable employers, employees, researchers and the government to identify and eliminate the most serious workplace hazards. So we get hung up on the standards sometimes, right? We all do. It drives at least a minimum, our starting point where we start our work. But what’s missing sometimes is the spirit, the intent of that standard, and how it can transfer to other situations and scenarios when it makes sense. So that quote right there speaks to the spirit and intent, at least in general industry, of the recordkeeping requirements, why we’re required to fill out that 301. And that’s important.

Wes Carter: Absolutely. Well, Blaine, I appreciate your time today on the podcast, and I guess, since this is a joint one, I enjoyed being on your podcast as well.

Blaine Hoffmann: Absolutely, Wes. It’s always a pleasure. I love working together.

Wes Carter: Yes, sir. Until next time, take care of yourself, and thank you, audience, for listening.

Blaine Hoffmann: All right, there you go. Wes, a fine fellow indeed. I enjoy talking with him, and I love all the things they’re doing over there at the Amplify Your Process Safety podcast. Continue to check them out. I encourage you to broaden your horizons as a safety professional. Get to know the PSM standards. You could learn a lot. A lot of these things transfer over to general industry, some best practices and approaches to things. So let me know what you think. What did you get out of this episode? And I look forward to talking to Wes on future episodes as well. And I look forward to talking to you on the next SafetyPro Podcast episode.

Thanks for listening to this episode of the SafetyPro Podcast, powered by iReportSource, the all-in-one digital safety solution. For info on how you can request a free demo of all the capabilities of iReportSource to streamline your business’s safety management system, head over to theSafetyProPodcast.com. We look forward to helping you manage safety on the next episode of the SafetyPro Podcast.