Editorial
With the growing complexity of operating models and underlying systems, organizations have a greater risk of debilitating system failures happening on full display to the public.
The potential for catastrophic system failure has always been present across industries. In 2023, we saw large-scale outages hit telecommunications networks and payments systems. The materialization of cybersecurity risks has also left a range of operators unable to function for days or weeks at a time.
In spite of these failures, the world learns from instances where workplace risk materializes in a big way. High-profile incidents tend to drive higher safety standards, stronger regulation, better practice adoption and technological improvements. But this is also a very reactive approach to risk mitigation.
Prevent Accidents: Apply an Emergency Handbrake
Where an incident has triggered government, regulatory or shareholder attention, the victim organization invariably regrets having breached risk thresholds in such a manner in the first place.
Most companies fear an outlier scenario — one they couldn’t test for, or worse, couldn’t even imagine. They’d sooner pull a “handbrake” than find out what happens if a situation is left to run its course.
But to do that, organizations need a handbrake they can reliably pull. They also need best practices to pull that handbrake automatically in circumstances where elevated risk levels are present.
Where Processes Falter
Most organizations keep the lights on through efficient, accountable and responsible process management. But this falters in sectors where process management is manual or spreadsheet-based — and where the onus is on a human operator to correctly identify and act on risk triggers.
Human reactions to elevated risk triggers often lack nuance. It’s almost impossible for humans to monitor for early warning signs in a complex system, and to apply or adjust guardrails or actions delicately or on a sliding scale according to observed risk or potential consequence.
In addition, human-led action may make things worse. There’s always a possibility of errors being made based on various people’s interpretation of rules or limits.
Past experiences may also affect judgement; a person may have previously experienced a similar operating state and taken a known course of action based on that experience. However, this may not be the best guide for future action because these underlying systems may have evolved between the two incidents occurring.
Related Article: How to Identify the Right Workplace Processes to Automate
Keeping the Lights On
Organizations use process automation and management to ensure the correct decision is made, taking into account all available data and intelligence about a risk threshold breach.
Data, service level agreements (SLAs), analytics and reporting function as key inputs for a process management system to determine when it is appropriate to apply a handbrake to mitigate against a risk-based issue.
For data to inform a handbrake decision, it must be standardized and consistent. Standardization of formatting means that an action can be consistently applied across the dataset. Where data quality varies, taking action at scale is much more difficult.
Where the data is numerical, it also means having checks and balances. For example, payment amounts must be accurate and any red flags must be detected and appropriately addressed. There are still frequent stories of bank customers inadvertently receiving huge deposits — and then spending it. Banks have been trying to augment payment processes to prevent these kinds of transactions occurring in the first instance. These not-infrequent examples of transaction errors highlight the importance of due diligence in payment processes and, in cases where recovering the money is difficult, of having the ability to apply a handbrake to mitigate against a potentially large financial loss.
The Importance of Service-Level Agreements
Performance against a service level agreement (SLA) could also be used to inform a handbrake decision. Business SLAs often include specific metrics around acceptable time frames for operations or actions to occur. Exceeding these thresholds may come with some financial liability for the provider, whether that be through service credits or actual financial rebates.
Conference
Oct
20
Gartner IT Symposium/Xpo Orlando 2025
Register
Conference
Oct
27
Gartner HR Symposium/Xpo Orlando 2025
Register
SLAs can be used as triggers for specific actions. If a system capacity issue is detected quickly, the provider then has time to mitigate the issue by increasing capacity, distributing workloads or identifying the highest consumers of compute resources and throttling their use. Applying one or more of these ‘handbrakes’ can help to avoid a bigger, more catastrophic issue from occurring when capacity becomes exhausted and all customers experience an outage.
However, organizations may still not be able to prevent all possible process degradations or failures from occurring, even with all of these handbrake mechanisms. In these instances, analytics and reporting of the incident can lead to additional handbrake mechanisms being implemented. No one is 100% error free; when a mistake does happen, the ability to report on it and incorporate learnings into the process is important to mitigating against any recurrence.
In all instances, the capability should exist for a handbrake to be automatically and autonomously applied based on inputs from data, SLAs, analytics or reporting. When processes are managed and automated like this, we considerably minimize the risk of becoming a case study for catastrophic system failure.
Learn how you can join our contributor community.
