What alarm rationalization uncovers - and why it matters
Alarm rationalization has a reputation for being a big undertaking — and that reputation isn't entirely undeserved. But it is highly recommended, proven in practice and can find and fix minor issues before they become big, expensive and dangerous problems. The right technology makes it more straightforward than most people expect.
If you are not rationalized...
An alarm system that has never been rationalized is also likely to have been created without the benefit of a consistent alarm philosophy. It is often a collection of many different people's ideas about what should be an alarm, implemented over many years and changed on a whim, reflecting personal preferences. There will be major inconsistencies in such a system.
Unrationalized alarm systems will have hundreds of meaningless and distracting alarms that make little sense to the operator. These are often (rightfully) ignored and the operators will have little trust in the overall system. This can result in other truly important alarms being ignored as well, because – how can you tell the difference? Alarm suppression methods may be used in an uncontrolled manner, with some alarms suppressed for days or months or longer. These are upsets and accidents waiting to happen.
The alarm system is supposed to reliably inform operators of process conditions that need their attention. An unrationalized system will not do this.
How to rationalize
Alarm rationalization is a proven methodology for aligning an alarm system with the alarm philosophy and helping ensure it performs as intended. Drawing on experience from thousands of alarm rationalization projects, Octave has identified practices that can help organizations rationalize alarms more efficiently and effectively. In The Alarm Management Handbook – Second Edition, we wrote a highly detailed chapter on how to approach rationalization.
It includes lots of tips and pitfalls to avoid. If you have never done a rationalization, we advise you to seek experienced assistance. Doing so will save you far more time and money than you would spend by going it alone. We know how to minimize the impact on your plant resources – but you must participate. Don't believe anyone who says they can rationalize your alarm system without your involvement.
For this blog we thought it more interesting to write about some of the strange and disturbing things we have found from doing so many successful rationalization projects.
Before and after rationalization
Before beginning rationalization, you will usually find that
Operator frustration is high
Alarms are occurring constantly
The alarm summary display is always many pages long
Most alarms are nuisances, with some even occurring based on timers
Operators are constantly hitting the acknowledge button (which does not mean that the alarm was read or understood)
When you are done with rationalization, what will be the result? Here are some quotes and reactions.
From operators:
"Finally, the alarm system makes sense."
"The alarm system is useful now. It sure wasn't before."
"You can understand the alarms now – they have real meaning."
"I'm not constantly dealing with a bunch of incomprehensible alarms anymore."
"This is the best thing we have ever done."
Quote from an operations supervisor: "… this rationalization project gave us a lot more benefit than just improving the alarm system. In fact, it helped us discover mistakes in our piping and instrument drawings (P&IDs), incorrect procedures, wrong point descriptors and in general, misunderstandings we had about how our plant works. It also found a number of 'gotchas,' which should contribute to improved process safety and operability."
Note that rationalization is not just getting rid of alarms. The process often reveals necessary alarms that have not been configured at all.
What we've found
Here are some of the things we’ve learned from conducting so many rationalization projects.
As noted during a power industry conference, the role alarm handling plays in upsets, emergencies and incidents is not always recognized. Operations and maintenance teams can become so accustomed to alarm floods and nuisance alarms that they begin to accept poor alarm system performance as the status quo.
That acceptance is part of the problem. Excessive alarm rates and persistent nuisance alarms should not be treated as normal operating conditions. They are signs that the alarm system needs attention and can prevent operators from identifying and responding effectively to the alarms that matter most.
Poor MOC and documentation are rampant
Poor management of change (MOC) practices and outdated or inaccurate documentation are surprisingly common. In some cases, alarm rationalization is the first process review conducted since the plant's initial startup. It may also be the first time anyone has systematically compared what is actually configured in the control system with what is documented on P&IDs or in loop diagrams.
The discrepancies uncovered during rationalization can lead to necessary changes in process documentation, operating displays, programming logic, procedures and sometimes the equipment itself.
Unsafe findings include:
Wrong or missing tag descriptors
Wrong alarms configured on a graphic
Important alarms missing from graphics
Instrument loops exist that are not included on the P&ID
Instrument loops are shown on the P&ID that do not exist in the DCS
Demolished equipment still shown on P&IDs
Equipment exists that is not shown on P&IDs
Incorrectly configured instrument ranges (where'd all those bad-PV alarms come from?)
Redundant instruments with different configured ranges and other characteristics
Switches (unreliable) provide critical alarms where a transmitter is available. (This will almost always be found in older plants that have gone through an upgrade)
A rationalization conducted at a major refinery uncovered additional issues, demonstrating that even large, established operations can have significant gaps in alarm configuration and documentation:
A lot of the documentation was missing and needed updating
The management of change procedure and work process were very complex and errors in the documentation showed it was unreliable
P&IDs, equipment manuals and operations training manuals were not updated after project changes
There were a large number of incorrect or contradictory findings in the operating limits and the operating envelopes in use
There were many tags related to safety and interlock systems that did not contain alarms, but should have, which surprised the operators
Maintenance ignores malfunctioning instruments so the diagnostic alarms are just disabled and ignored
Now, a rationalization does not go into these things without reason. When prepping for a rationalization, Octave uses software that imports the settings for every alarm, straight from the control system. Then, in the alarm discussions, sometimes things such as this example occur:
Octave: "OK, for this reactor pre-shutdown alarm, is 350 degrees the correct setpoint?"
Customer: "Yes. The DCS alarm warns the operator that the safety system is about to shutdown the reactor."
Octave, later: "So, the safety system alarms when it shuts down the reactor. Hmmm – this shutdown is set at 350 degrees. Wait – how can the DCS alarm at 350 be a 'pre-alarm' for the operator to react and avoid the shutdown?"
Customer: "Wow, looks like it can't."
Octave: "And look – the other reactor is set at 360 degrees? Shouldn't that be the same?"
Customer: "Hmmm, let's get some more documents. I think there was a project that might have done that…"
A rationalization at a major chemical plant revealed similar issues:
The design basis for the plant itself had undergone some revisions in years past, but the documentation was never fully updated before the project team was reassigned to another project. The resulting P&IDs were a patchwork of long since abandoned processes and improvements.
Operator comments during the rationalization:
"Ok, so I know that's how it's drawn, but that's not how it actually is."
"That hasn't worked in years."
"That system got removed about a decade ago."
"That was for the old process, it needs to be taken out."
"That's been bypassed for ages, we need to get someone to make a call as to whether we're going to keep it."
Projects never altered existing tags in the DCS, they just made new ones and left the old ones idle in the system: "We had to weed through a number of long-obsolete logic blocks and calculations."
Many 40+ year-old power plants either lack P&IDs or have not updated them in years. A surprising but not-uncommon statement: "We know our P&IDs are not reliable at all, we prefer you do not use them."
What rationalization can uncover
On and oil and gas project, there are a huge number of DCS tags mapped through MODBUS from various packaged systems. These tags have absolutely no traceability with the P&ID drawings or DCS graphics for the respective systems. These include inputs, outputs, logic blocks (sequences), diagnostics and more. Almost all of them have alarms configured. The descriptions on many of these tags were identical and sometimes did not even make sense. Most of these alarms turned out to be representing normal events and were removed.
Conducting a rationalization for a refinery that has a Foxboro system. There were about 25,000 points in the system. Alarms were configured as if it were a testing database for the people who controlled logic engineering work during database configuration. You just name it and you will find alarms configured on it. When we were done, not only did they get a rationalized alarm system, but they also got a list of great process and documentation improvements that would help them dig out of constant firefighting mode and focus on long-term improvements.
Alarms and personnel safety
Here are some interesting discussions:
Customer: "That alarm is before the steam safety valve blows. It has to be the highest priority."
Octave: "Why?"
Customer: "Because the safety valve is pointed at the walkway."
(Conversation ensues about how alarm priority does not divert live steam, but a piece of vent piping does.)
There was an alarm on the coal conveyor to indicate a person's hand or body could be caught in it.
Customer: "Set that to low priority."
Octave (aghast): "why?!!"
Customer: "Most of the times it is a false alarm because of equipment malfunction."
(Conversation ensues about alarms that relate to personnel safety, lawyers and the need to maintain important sensors).
Summary
Rationalization isn't just an extremely good idea; it is also a requirement in the ISA 18.2 and IEC 62682 Alarm Management Standards. What you want is a productive and efficient rationalization with the best possible outcome. Alarm rationalization does take some effort, but following proven practices, using the right tools and getting experienced help when needed can make the process much more effective. The Alarm Management Handbook offers additional guidance for those who want to learn more. And the benefits can be significant. Not once have we had a customer say, “That was a big waste of time!” Quite the opposite. We often have operators, a highly skeptical bunch, start lobbying to help other units that have not yet been rationalized. So, don’t keep putting off alarm rationalization.
Ready to tame your alarm system? Let's talk about we can help!
Review other Taming the Wild Alarm System topics in this 7 part blog series:
Part two - The most important alarm improvement technique in existence
Part three - Silence the noise: fixing chattering and fleeting alarms
Part five - What alarm rationalization really uncovers — and why it matters
Part seven - Beyond alarm management – doing more with a powerful tool