Safety audits differently

auditSafety audits have become central to safety management and governance. However, many organisations have noted that safety audits may have limited usefulness and some problematic consequences. People I’ve spoken with have raised concerns that:

  • audits may not get a ‘true picture’ of what is going on
  • some projects suffer from audit overload
  • audits may damage cultures of trust
  • audits may drive a commitment to creating acceptable images of work, rather than improving the primary process that the audits are supposed to assess
  • well audited projects still have a seemingly unaudited performance (incidents and injuries occur despite audits showing well compliant systems).

The way I see it, is that some of these problems stem from the belief that productive, efficient and safe work comes from the precise application of standards, best practices, and approved systems of work that have been systematically analysed and tested elsewhere. The assumption seems to be that if we fail to follow a limited set of rules, we will have substandard, inefficient and disorganised performance. Or, what can go wrong when every part has been checked for its compliance with agreed standards?

From this point of view, it makes sense to have regular safety audits and observations – formal, independent, and rational follow-ups of whether a project’s internal workings align with standards and expectations.

Put differently, safety audits focus primarily on the programmatic elements of organisational practice – the plans/the work as imagined/the normative/what should happen. It is this programmatic level which is connected to regulatory demands, international standards, best practices and other requirements. The technological elements (the operational reality/what actually happens/work as done/work as found) tend to be assessed only to the degree that they can support conclusions about the programmatic elements (Power, 1999).

The focus on the programmatic level makes it possible for audits to be disconnected from the very process that gave rise to its need in the first place. This is to say that in its most extreme form, a safety audit does not need to be concerned with safety performance, the meaning of safety to the project, if the processes actually assist work, or otherwise generate information about what actually goes on.

Another aspect driving the focus on the programmatic level is that audits require a certain type of information or proof. To enable comparison of findings across sites, projects, and organisations a standardised scale is needed. However, the messy details of work at the sharp end are local, contextual, and unique. As such, the audit will require abstraction of the answers, or a disregard for the local specifics. Furthermore, while such a scale or measurement should preferably be external and independent to what is being audited, the introduction of an external measure brings about a risk that audits become a dominant reference point for organisations and projects. Audits burden the audited to focus on what is an accepted way of showing and fixing compliance, rather than on improving performance of what actually gets done. This way, audits may impact the contexts in which they are deployed, without creating new knowledge about what happens. Things get confused into clarity (Law, 2004).

Standardisation may be a powerful tool to set limits and govern from afar. But, what if the selective gaze from afar can’t capture and detect what trouble grows locally, outside, and in between standards and procedures? What if many or most of the problems that people face are diffuse, ephemeral, unspecific, fuzzy, emotional, elusive, indistinct, or not yet fully understood? What good is then a blunt view from afar with a yes/no answer? What if workplace risks and capacities to handle these cannot be captured or even adequately mapped out using standardised templates for what should happen?

And even worse, what if enforcing and auditing a ‘one best way’ may prevent us from creating new understanding of what could be going on and simply rehash solutions of what has previously been found acceptable? Audits are based on and wedded with ideas and practices developed for yesterday’s needs, and essentially ask organisations to embrace the future by organising according to the ideas of the past. Can it be that one-sidedly enforcing standardised requirements, simultaneously makes it more difficult to have a culture of innovation?

Furthermore, one may ask what would happen if organisations did not have safety audits. Would systems degenerate? Would performance decrease? Would people lose sight of what should get done? If the people doing the daily work cannot be trusted to achieve reliable, safe and effective performance, but need policing, then what does it say about expectations and respect for the people involved? Audits may further emphasise that trust is supposedly with the experts that perform the audits, and/or with the document standards and evidence used to show compliance. Again, this may undermine, and distrust, the local custodians of a system. As such, audits may work to drive accountability, but not necessarily responsibility.

In summary, safety audits may:

  • fail to pick up what is actually going on,
  • hold back innovation
  • undermine local trust.

In effect, the current audit format may not be helpful in conveying an idea of how well a system actually functions over time, how it supports (or constrains) the performance of people, nor contribute to fostering local ownership.

 

Steps toward a different kind of safety audit

Most people would probably prefer if audits facilitated organisational learning and improvements, rather than being little more than a compliance check. This is not to say that we should do away with safety audits. This may, however, suggest that we need to open up for new ways of knowing about what goes on in organisations.

There is opportunity to design mechanisms that give a richer and more meaningful image and information dissemination of what goes on across projects and sites. To achieve this we may also need to change how we think about accountability. To stand a better chance to have a constructive audit process, we may ultimately need to change how we understand audits, how audits are communicated, and how audits are carried out.

What if:

  • an audit was something that auditors and auditees looked forward to?
  • finding sources for effectiveness and success in an audit was just as likely as finding nonconformances?
  • the audit process contributed to build trust and respect between the auditor and the audited?
  • the audit process itself started a cycle of continuous improvement (not deficit fixing)?

I don’t have the answers for how to accomplish all this. But in relation to the first potential, I’d like to suggest a more experiential approach – an audit through reflection on doing. Instead of asking ‘do you have a procedure in place for X’ ask the audited to talk about one time/event when something worked really well in and around using a particular procedure. Or one time when work was really difficult or challenging. People enjoy talking about what they do, their achievements and success, the daily dramas of how things come together (and not), and about what works and what could work. When sharing experiences of what actually has happened, people involve themselves in the process. It is their experiences that brought about the desired outcomes, and not someone else’s solution or best practice. The audit becomes an opportunity to get an outsiders perspective of what goes on. Sharing the joys and difficulties of managing complex systems are more likely to drive an engaging conversation, and a more insightful one.

Second, in a more experiential audit, the task for the auditor is to listen and seek to understand what helped performance and what made it difficult, ie examine the factors surrounding a procedure, rather than the mere existence of the procedure itself and evidence that it has been used. This is a more inviting way to bring out information that highlights where local system custodians may have the need, not to ‘non-conform’, but to find a way that can better reconcile complex situations. It may of course be categorised as a non-conformance, if that is helpful. However, as the auditor and audited build a richer understanding of the tools, resources, information and strategies that are available to deal with the demands and constraints, calling something a non-conformance is unnecessarily reductionistic.

Three, as organisations tap into the accumulated wisdom, experience and creativity of employees who are closest to the issues, organisations are more likely to get a better understanding of their workplace. People are more likely to feel listened to and respected, resulting in a more engaged and purposeful conversation, cutting through layers of bureaucracy and possibly having an effect that lingers long after the auditors have left. But, appreciating of the experiences of local stewards, are also more likely to have an impact on the auditor. The realisation of how non-conformances can make sense, is more likely to produce a mutual respect of how difficult and messy operational life can be, and produce a compassionate and more holistic response – helping the auditor to learn something about the system as well.

Four, people are more interested to work towards positive outcomes, as opposed to avoiding negative outcomes. Or so it seems anyway. Applying a more appreciative approach in questioning, allows audits to turn into an opportunity to reconnect with the purpose of a project or site, rather than producing yet another distracting deficit focus. So by focusing on what works and what could be done to achieve success, audits have a better chance to leave people with ideas and inspiration for actions that can deliver on the goals, rather than producing fear or threat that only fuel actions for as long there is a problem (van de Wetering, 2010).

What is at stake here is simply not just about safety audits. It is also about what kind of safety governance an organisation would like to exercise, and what kind of workplaces they would like to contribute toward.

Sources of inspiration:
Auret, D., & Barrientos, S. (2004). Participatory social auditing. Institute of development studies: Brighton, UK.
Healy, S. (2003). Epistemological pluralism and the ‘politics of choice’. Futures, 35, 689–701.
Humphrey, C., & Owen, D. (2000). Debating the ‘power’ of audit. International Journal of Auditing, 4, 29–50.
Law, J. (2004). After method: Mess in social science research. Routledge, Oxon, UK.
Power, M. (1999). The audit society: Rituals of verifications. Oxford University Press: Oxford.
van de Wetering, A. (2010). Appreciative Auditing. AI practitioner, 12, 3.
Wynne, B. (1988). Unruly technology: Practical rules, impractical discourses and public understanding. Social Studies of Science, 18, 147–167.

 

16 thoughts on “Safety audits differently”

  1. I’ve read about FRAM being used as an assurance model in health care which I’m keen to try out. Eric H I believe did something with the Danish Health Service which, as you know, identifies and visualises the full range of functions needed in a process and the critical information and resource exchanges and their timing needed to make it happen. If using FRAM as an ‘audit’ tool you could take a team based approach to walk and talk through processes, where all can contribute actual experience of what happens in practice, probing deviations in terms of sequence and that of people interaction which identifies and informs improvements for delivery and resilience through discovery rather than auditing in a linear manner. Engaging and collaborative ‘auditing’ would be more productive than the linear and punitive typical audit approach.

  2. Great post Daniel and I agree with much of your sentiment.
    However, the other significant issue regarding audits is the auditor. Much of what is purporting to be an audit is simply a set of checklist-driven questions and the capability and competence of the auditor could often be challenged.
    To misquote an old “oil” adage “auditors ain’t auditors” always!

    1. Thank you – I agree that the stance of the auditor himself / herself is very important. The auditor may also be in a double bind , in that they must somehow reconcile the (Work as Imagined) criteria to be assessed such as Required Organizational Practice (Healthcare Accreditation in Canada), with the very real, pragmatic (Work as Done) reality of patient care. In my practice in Healthcare Human Factors, I coach our Quality Improvement folks (the auditors) on that very stance…Not easy to do, since Healthcare seems “awash in a sea of linearity” (Robson)

      thanks for the discussion

      Wrae Hill
      Kelowna , BC Canada

  3. Thanks for the nice title Daniel, that’s too much credit for me and you’re putting pressure on me to deliver!

    I think that Shane describs very accurately how FRAM can help. I envision the following steps that can make an audit more productive and aimed at ‘the messy details’ of operational life:
    0 Identify the aim of the audit;
    1 Audit individual operational personnel to begin to understand how the process under investigation really works;
    2 Start developing a first iteration of the FRAM model;
    3 First team session to verify and expand the model;
    4 Follow-up audit-type interviewing;
    5 Follow-up team session: this session is the core of the audit. The model serves as a facilitator of the discussions and then the focus should be on the collective understanding of the process based on the interactions and functions identified in the model. The output is the qualitative account that emerges from the session, if necessary focused on some pre-defined areas of interest (see step 0).

    Does this makes sense? And do we then also adhere to the audit objectives?

    Report user
    1. I can see where this would be very helpful for audit teams to understand processes, and how they’re supposed to work. It looks like you are advocating that the audit personnel are the ones to do the FRAM…would it not be better to have the auditee prepare these ahead of time?

      If you rely on the audit team for this, you may get a better FRAM, but distort the economics of the audit. Unless required by law or an organization has the benefit of superior leadership, I would say a fair number of organizations don’t go past the first recertification audit.

      I think the idea’s great, however the devil (as always) is in the details.

      Thanks David & Daniel!

  4. • Dysfunctional Upstream Reporting

    An inescapable fact is that part of the causation of all unintended harm is that the harmful conditions, behaviors, actions, and inactions that resulted in the harm had not been effectively noticed, reported, and/ or addressed . The downstream manifestation of the harm and/or the harmful conditions, behaviors, actions, and inactions that resulted in the harm is sufficient evidence that all self-assessment and oversight problem identification measures back upstream to the origins were ineffectively deficient.

  5. I have a tool that deploys a ‘good practice’ audit framework in a group setting (what good practice means is a whole different discussion). For each of the identified areas, we discuss how well it is deployed to get a feel for potential improvement opportunities. I score it for the purposes of creating some headline data, but the comments received are the most important part of the process. In an ideal scenario, this discussion takes place at two or three different levels in the business, which also gives a secondary indicator as to how consistent the safety message/understanding is across the business as a quasi-engagement indicator.

    Report user
  6. I’m trying to envisage an audit template that would accommodate a more positive focus. I can’t quite capture it, but I think a simple editing of the language used would be a good start. Maybe my paradigm of an audit template is flawed from the beginning?! ‘Compliant’ is such an underwhelming, non-complimentary description of ‘positive’!
    Your comment “people are more interested to work towards positive outcomes, as opposed to avoiding negative outcomes” reminds me of some research I remember. It showed that law-breaking drivers were content to accumulate demerit points, but less inclined to lose them. A simple arithmetic switch resulted in significant improvements!

    1. When you audit a railroad start with the train wrecks.
      When you do a safety audit start with the fatalities and injuries.
      When you audit Chipotle start with the food poisonings.
      When you audit the Flint, MI water system start with the lead poisonings.
      When you audit Takata start with the shrapnel.

      Start with the enormities
      Then audit to the nonconformities.

      Drop your hook and worm where you already know the fish are biting.

  7. Hi Daniel – very pertinent post and is a “must read” by auditors and organisations requiring audits.

    (1) Audits must be of value to the organisation and recognise and comment on good and poor practices. Tick box audits are very ordinary if they do not contain solid practical comments.
    (2) Daily close out meetings are essential to capture the daily “Good, Bad and the Ugly” for a wider audience and to immediately consider appropriate actions. The auditor comes under close scrutiny and must know what he is talking about . This is good training for the auditor.
    (3) Audits should be a training tool both for the organisation and the auditor – My field verification audits have an appropriate and influential company representative with me at all times and we both learn from the experience – Great “In Field” training
    (4) Auditors must do substantial homework to understand the organisation being audited and the general inputs and outputs of their processes. In many cases the auditor must be technically competent in these processes, other wise they lack credibility and will not be able to develop an trusting relationship with the client

    1. • Third Party Inadequacies

      An inescapable fact is that when harm occurs that one or more third parties were expected/ intended to prevent, that third party performance has been dysfunctionally inadequate . In these cases the third party performance is part of the causation.

      “Who pays the piper calls the tune.”

      “It is safer to give a parking ticket to a bank robber than to engage in a shoot-out.”

Leave a Reply