Turning Apples into Bananas: How Big Data undermines safety and what can be done about it?

An able seaman has been ‘reprimanded’ for leaving the gangway hanging out while the vessel was shifting berth. Root cause – ‘lack of awareness’; corrective action – ‘risk assessment’. A third officer has been served a warning letter for missing out on monthly checks on a fire extinguisher resulting in non-conformance during a safety audit. Root cause – ‘complacency’; corrective action – ‘follow the procedures’. A crew member trips over an obstruction on deck and hurts himself. Root cause – ‘lack of awareness’; corrective action – ‘risk assessment’. And a chief officer, who submitted a near miss report stating that he came close to a fishing vessel during the coastal passage has been sent on a refresher training course. Root cause – ‘lack of planning’; corrective action – ‘training and supervisory control’. Go through any reporting system and you will find hundreds of such reports. With intense budget controls and an over-zealous commitment to safety many organisations are turning towards software solutions, colloquially speaking ‘Big Data’, to measure the state of safety.

The deeper human stories

Listening to the other side of these reports is disturbing but insightful. The able seaman points out that moving the vessel alongside the berth with one crew member forward and aft during the night was not unusual. The captain had consciously made a decision not to wake up other crew members in the middle of the night. What was unusual, however, was that a crane swung out that had obscured captain’s sight of gangway from the bridge. When things went wrong, the able seaman was held to blame for not informing the captain that the gangway was hanging out.

The third officer has his own version of the story. As a safety officer he must ensure that each portable fire extinguisher is visually inspected and ticked off every month. With the best will and attention to detail, there are instances when an odd fire extinguisher can be missed out when you have a list of three hundred extinguishers to check. “But nobody is interested that I check other 299. One mistake and I got warning from the captain,” he says. It does not end here. He adds, “now every time we have inspection, I am worried, I cannot sleep for many days. Maybe I forgot something, I am going around at night after my watch to make sure I have not missed anything. It’s not easy to find another job if I lose my job.” This is a watch officer who stays on the bridge for at least eight hours in a day onboard a vessel laden with hydrocarbons.

The chief officer became furious when I probed him about the near miss. “They make me do a three days course, I find it insulting. It is them who should do better planning. I tell them don’t load six high containers on the forward hatches especially on the sides, it obstructs visibility from the bridge. But they don’t care. And when I reported this, they are telling me I was not careful on my watch.”

The question is, can we ever get insights of this kind from data-based reporting systems or are we simply camouflaging them with technology?

Measuring safety

Attempting to measure safety is a key purpose of any reporting tools. Safety is defined as the condition of being protected from harm or injury to an acceptable level. But this definition is not without its problems. To the consumers, stakeholders and society at large (in a legal and media sensitive environment) the notion of an ‘acceptable level’ of safety can easily become arational and unrealistic. Take an example of what is acceptable – ‘if it saves even one life, it is worth the effort’. More eloquent examples include ‘all accidents are preventable’ (AAAP) or an accident free future.

On the face of it, there is nothing wrong with such virtuous statements but when we dig deeper we realise that the thinking behind such statements sit outside the realms of reason. What is acceptable is not so much a calculated decision based on professional judgment. Rather, it is the unfortunate outcome of living in a society where expert decisions and professional judgment can come under attack when something unexpected happens (especially a bad outcome). For anyone tasked with measuring safety a natural response would be that everything is a risk, nothing can be acceptable.

Businesses operating in fierce market competition often struggle with such arational responses. At times the costs can become unsustainable, and on other occasions the responses themselves make a mockery of safety. This is we how end up investing heavily in barriers and protective devices and implementing behavioural safety tools despite no evidence of injury or harm – and sometimes even evidence that the ‘safety’ measures are counterproductive.

A second, and in my view, bigger problem with measuring safety is that it relentlessly aims for perfection. There is a very specific language that underpins this thinking and quite often it has little or nothing to do with safety. For instance, compliance, conformance, all accidents are preventable, zero tolerance, and the zero-accident vision. Perfection is based on a narrow set of goals and a depressingly negative vocabulary – fatalities, lost time injuries, first aid cases and so on. A handful of unrealistic goals are established, and the focus turns towards petty mistakes rather than overall progress. It does not matter if you have checked hundreds of fire extinguishers, what matters is one that is left out. Any improvement that does not fit with those narrow set of goals means nothing. As we build sophisticated reporting tools to measure safety, we need to be aware that the end purpose can become trapped in arational thinking and unrealistic goals.

Big Data or Big Brother?

What about the actual reporting and analysis of data? As copious data is being poured into software models, one is tempted to believe that this would help us predict what lies ahead, alleviate uncertainty and improve business performance. Technology will comb through petabytes of data and provide intriguing insights into individual behaviour. Software systems are already becoming capable of identifying patterns of human behaviour and correlating them with workers, their origin, ethnicities, race, age groups, creed, colour, nationalities and gender. HR software tools extensive data about employees to assist organisations with hiring decisions and predicting employees’ performance at work.

But imagine how spurious correlations can easily go wrong when it comes to measuring safety. Once a few ‘unsafe’ individuals are identified this can lead to detailed monitoring of those individuals, subsequently finding even more problems in their behaviour. The accident-prone workers will remain logged in the system for a long time. In the wake of an accident (or an unsatisfactory inspection outcome) the first thing would be to recall the history of the workers and take them to task. This may exacerbate the problem of accountability and blame.

Artificial Intelligence or Algorithmic Injustice?

At the organisational level, reporting accidents, defects or any shortcomings have long been considered a sign of failure in many safety critical industries (even more in male dominated industries.) One operator states, “our client takes the risks of dropped objects very seriously, so we scan through our incident reports to check for terms such as ‘dropped objects’ and ‘deck’ to ensure we do not have issues there.” What if over a period the computers are trained to carefully ignore those buzzwords and catch-phrases that affect their performance indicators and market situation?

Manipulation has always been a problem with us humans. Why make so much fuss now that we are assigning the analysis to computers? Data scientist Cathy O’Neil, author of the book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, tackles this issue through a series of case studies. O’Neil argues that while we as humans learn to adapt quickly when things don’t seem to work, computers can get fixated onto erroneous correlations that become difficult to break into. When this happens the scale of damage becomes inconceivable for any organisation.

One such example, reported on social media, was recounted by a ship manager, who reported that one of their vessels developed a small oil leak in the main engine while transiting a busy shipping lane. The issue was reported, the vessel anchored, changed the component in question, and was back underway within an hour, problem solved. Later, the company saw that the vessel had been downgraded by a fully automated rating system from five stars to three. It took months to return to five-star status, during which period several chartering opportunities were lost. There was no recourse, and the entry could not be changed, as the system was fully automated. This is a telling example of algorithmic injustice in the digital era.

Turning apples into bananas

Back in 2017 there was a CNN commercial with the photograph of an apple on the screen. The commercial stated: “This is an apple. Some people might try to tell you that it’s a banana. They might scream “Banana, banana, banana” over and over and over again. They might put BANANA in all caps. You might even start to believe that this is a banana. But it’s not. This is an apple.”

It is quite simple to think. Say it a thousand times until accuracy is gone, forgotten and replaced with plausibility. That is the power of scale. Now imagine a database full of pre-determined phrases like ‘complacency’; ‘procedures not followed’; ‘lack of planning’; ‘lack of situational awareness’ etc. Get rid of ‘dropped objects’ from the database when they become a problem. Over a period of time computers will learn to figure out our affinities and aversions. The apples will be left with two choices – disappear or turn into bananas. The same search engines that were once designed to identify problems could become a weapon to conceal them.

The causality credo

There is a further problem with these predefined phrases. Rich and vivid human stories are stripped of their context and simplified into variants of ‘human error’. The thinking behind all this is that for every accident or non-compliance there is a cause, and generally a bad one. Bad causes precede bad consequences and those bad causes can be traced back if we searched far enough (hence the term root cause analysis). This is what Erik Hollnagel refers to as the ‘causality credo’.

But causation is flawed. Causes can be imaginary and fabricated to serve certain purposes, as in the case of the chief officer who was enrolled for a training course. Similarly, the quest for the cause will typically end with the last man or woman in the chain, as in the case of the able seaman who ‘forgot to secure the gangway’.

And then there are instances when causes are mistaken for consequences and vice versa. Consider the crewmember who tripped on deck. Was it because he did not pay enough attention to the obstruction on deck? Or could it be that a bad design caused the crew to stumble? The former puts the blame on the worker, while the latter aims to find ways to design out the obstruction. Could algorithms ever expose the flaws of causation and get to the human stories behind the reports? That depends on the data we chose to collect (and ignore) and the questions that we ask of that data.

Rethinking safety in the digital era

Data analysts would tell us that software tools can work with massive volumes of data and ‘automatically discover trends and patterns’ or find ‘non-obvious causal relationships’ in the data. This should mean that the entire process (from data collection, to reporting and analysis) is exploratory and driven by curiosity and science. This cannot be true at least for safety where the purpose itself is ill-defined, convoluted (to serve multiple interests of which protecting people from harm and injury is just one purpose) and driven by a pre-determined set of narrow goals.

The examples cited through this paper bring to light that there is more to safety data than just observing patterns of individual’s behaviour or actions. As the educational psychologist, Jerome Bruner pointed out, “it is practically impossible to understand a thought, an act, a move of any sort from the situation in which it occurs.” We can think of safety as an individual’s problem or we could think of it as the capacity of our people to get the work done despite some common problems mirrored across organisations and work-sites – poor technological designs, poorly written instructions and procedures, incomplete information, conflicting demands, time and resource constraints.

Technologist ethnographer Tricia Wang, in her instructive TED Talk speaks about the importance of combining Big Data analysis with human stories. Wang’s research shows how Nokia became preoccupied with market research driven by statistical analysis and missed out on rich human stories and why 73% of big data projects turn into a failure. This has nothing to do with safety, nonetheless it exposes the naivety of relying single-handedly on number crunching and statistics in making sense of human behaviour.

If we can combine big data with deeper stories we can rebuild the trust in reporting, demystify the flaws of causation and build a richer understanding and analysis of data. Technology offers us two choices (not necessarily binary but co-existing). We could continue with the same old concepts of safety in the hope that somehow by using software tools we may obtain better results, or we could genuinely transform safety into a business performance tool.

The article was first published in the Seaways International Journal of the Nautical Institute in October 2018.

Disclaimer: The views expressed by the author may not represent the views of the organisation that the author represents.

10 thoughts on “Turning Apples into Bananas: How Big Data undermines safety and what can be done about it?”

  1. Thank-you – Fascinating insight into challenges with Big Data and relationship with Safety Reporting. Reminds me of the comment by a British PM (Benjamin Disraeli) about lies, damn lies and statistics. How often in todays media intensive world are we presented with seemingly plausible data that is portrayed as an immutable truth that has escaped any form of basic evidential scrutiny or peer review.

  2. Nippin, thanks for the post but a ‘virtuous statement’ can only be deemed so if it is based on an ethical anthropology so, ‘all accidents are preventable’ is an unethical statement because it ignores the reality of infallibility and projects perfectionism onto a random world and vulnerable people. The outcome of such language can never be virtuous! Perfectionism is not just a mental health disorder but an unethical construct that can only lead to blame, victimization and delusion.
    As for big data, and other silly language such as ‘predictive analytics’, these are all proposed on the anthropological assumption of behaviourism and zero ideology. Humans are nothing like computers nor comparable to such metaphors or nonsense language as ‘engineering’ or ‘algorithms’.
    Sorry to disappoint Safety, but human ‘being’ is not ordered, is very messy, is fallible, unpredictable, non-mathematical, ecological and inter-relational at all levels. The imposition of a positivist framework on Safety is the problem. The fixation by Safety on measurement and data is an illness. The lack of an holistic anthropology and ‘an ethic of safety’ is the driver of much of the nonsense that exists in the safety industry. Humans are not machines nor the sum of inputs and outputs, despite the attraction to safety that would like it to be so. Safety should be about care, relationships, building trust, conversation, learning, empathy and listening not data, inputs and outputs.

  3. Dear Nippin, as always an insightful and thought provoking article thank you for sharing.. To Rob Long’s point I work in a world that struggles to understand the human and importance of trust when it comes to business KPIs and how to achieve them.

    1. Hi Julie, I think KPIs are the con job of the century and don’t offer little value for either productivity or organisational development. They are at best an over hang from a Deming-like focus on technique (efficiency) but have delivered very little. However, like many business initiatives, once they are in they are impossible to get out. Unfortunately, they lead to the further delusions about measurement and lower-order goals to the detriment of higher-order goals like trust, care, community and helping.

  4. Great paper, Nippin! Your posting aligns well with the emerging practice of Distributed Ethnography.
    We collect safety stories (what ethnographer Tricia Wang calls “Thick Data”) because stories provide context. Stories we hear are what storytellers choose to tell because it’s important to them. We often learn about safety issues that management or expert consultants had no clue about.
    With the means to convert stories into data points, we generate visual graphs to see emerging patterns. Pattern analysis is done by humans, not by algorithms. This method means we avoid the similar problem we have with safety questionnaires where survey designers with known and unknown biases choose the questions.
    I totally agree we are not in a cause & effect space. The purpose of pattern analysis isn’t to measure outcome. It’s to provides clues where we might intervene to shape the safety culture.

  5. I suppose I am a “big data” person. Work with lots of safety, quality, event data at several locations. There are ways to learn information from the data, and from the people. One thing the current “data analytics” folks are forgetting is Dr. Deming’s (and before that, Dr. Shewhart’s) concept of the “stable system”. Keep doing the same thing over and over, keep getting the same results. Put a different worker in the mix, same results. Keep rolling the dice, same results. Unfortunately, the “big data” analytics believe they have “all the data” and don’t have to worry about statistical uncertainty. But there is a big chunk of data they don’t have – future results. Suggestion to folks working with data is to look at Dr. Deming’s Red Bead Experiment (I did a version here ->https://www.youtube.com/playlist?list=PL8E522DD542C4CA69). The Red Beads makes a good parable for the interaction of the worker with management with the system.

    Good article, working with Canadian Nuclear Laboratories to learn these lessons and apply to work there.

    Steve Prevette

Leave a Reply