Metrics associated with 483’s – how some managers just don’t get it!!!
January 8th, 2012 by Peter CalcottEarlier in my career, as head of global quality in a well known company that will remain anonymous, we had experienced a regulatory inspection that had gone quite well. Although it was a full week with two inspectors, we had survived with 2 citations (note I do not say 483’s – so you do not know what agency it was doing the inspection). These citations were relatively minor and very clearly, “one offs”. There were absolutely no repeat violations from previous inspections with the observations being isolated in that no question of systemic issues were involved. Answering the observations were easy even including broad based CAPA’s.
In the debriefing that followed, after the inspectors left, I did my normal congratulations to the team assembled. I indicated that although we had received two citations, these were minor and easily correctable. I continued that we should not rest on our laurels but continue the continuous improvement program taking heed from some of the non-citation comments the inspectors had left us with.
The team was ready to leave when the new head of operations asked to say a few words. He was in his first two weeks on the job and so, I guess wanted to take charge, launched into a monologue that left the whole team cold. He indicated he was embarrassed by the inspection since we had received citations and that the performance was unacceptable. In future, any citation received would bear directly on all employees. Needless to say, that speech went over like a lead balloon. Over the next few months the plant was handed goals that were draconian in nature which brings us to the topic of this blog for this week.
Bad Metrics!!!!
In the world of continuous improvement, I always remember a quote of Demings (or at least I always attribute it to him – but I could be wrong) and it goes something like this.
If you can not measure it, it can’t improve it
So in order to improve performance we set up metrics to measure our operations. Metrics should be measurable, objective (or at least relatively immune to subjective interpretation) and easily collected. So observations or citations from an inspection are a clear item that fits into the category. So how do you formulate the metric?
The naive answer is simply to set a goal of say,
No citations, 483 points or observations
And that is an admirable goal. But does it set the right standard? Compare these two observations and ask whether they are equally serious.
Operator training records are inadequate, in that operator X has not been trained on procedure y
versus
The Quality unit is ineffective in that they fail to follow their procedures. Examples include
a.) Lot disposition procedures were not followed for the following lots……….which were released before QC data were approved.
b.) Annual product reviews were not carried out for years 2001………..2010.
c.) ……………………………….
d.) ……………………………….
e.) ……………………………….
f.) ……………………………….
g.) ………………………………
h.) Complaints were not investigated for the following products……
In the first case, while it may not be fully clear what the importance of procedure y is, there is no indication that training records were inadequate in any other employee. And you know that they would have looked and cited them if they had found any.
The second case really points to not just a systemic issue of problems but also to a system that is failing – the Quality Management System – the cornerstone of GMP operations.
So how should we develop metrics around citations?
Numbers have a place, but the nature, seriousness and depth of the observation are more important.
A good metric might read
No systemic citation pointing to a major system in the plant. No repeat violations from previous inspections. No citation identified and corrected in internal audits. No citation that is likely to have an impact on patient safety.
These are not perfect metrics but they do point to the important elements of compliance and good operation.
The same metrics might also be used for internal audits that are carried out more frequently than inspections.
While shooting for zero – an admirable goal – a more pragmatic goal is more reasonable.
Is an inspection with zero observations possible? Yes it does happen. Sometimes it is because the plant is well managed and it was a fair inspection. In some cases the inspection was so light weight the inspector did not check anything (and it has happened to me at least on one occasion). In other cases, the inspector did a conscientious job but just never spotted the issues areas.
As with audits, inspections are run by humans and, in consequence, variability in inspections and audits is large. I always think of inspections as biological assays with their large standard deviations.
In fact, getting a zero observation inspection is often dreaded by Quality staff because it tends to lull the naive operation person into a false sense of security.
Oh! By the way, the draconian operations guy did not last long at the company before being asked to leave. But until he left, morale at that plant was not the best.