David Amor, MS, CQA, principal at his firm Medgineering, kicked off the 2016 10x Medical Device Conference with a riveting talk about the Theranos 483 warning letters.

It was also a courageous talk – Former Theranos employees refused to give a presentation for fear of litigation and reprisals from the company.

David covered what a warning letter or 483 remediation project is, what a quality deficiency is, and how you can resolve them in a positive collaboration with FDA.

I also invited David back to present (with Jon Speer from greenlight.guru) at the 10x Medical Device Conference for 2017, our fifth annual event. Prices are deeply discounted this far ahead of the event if you’d like to register. (Full refunds until Valentine’s Day.)

Joe Hage: So please put your hands together for David Amor.

(Applause)

David Amor: Thank you, brave, well that’s not something I’ve ever been called but okay we’ll go with that. So thank you everybody for joining today, I am actually, it’s my first time at 10x surprisingly I mean I’ve been doing this for probably about eight or nine years now and I for some reason I’ve never stumbled upon here, which is a shame because it really is a fantastic conference so I’m really happy to here.

I actually want to start a little bit I was actually talking to my grandmother believe it or not, and I’m really from Miami, Miami, Florida so, not to different from San Diego except the people are worse probably, no offence. I’m from Miami, I had to say that. And really I think that what’s really hilarious is when I was a kid growing up Cuban grandmothers they are notorious for one thing.

I’m Cuban by background; my parents are Cuban from Cuba from Havana. Don’t ask me how they feel about the Obama Policy please for the love of God, but anyway. So when I was growing up one of the biggest things that I would, I remember on Saturday and Sunday mornings and throughout the week was my grandmother watching these TV Novellas, they call it, these little soap operas right.

The Americans have their ‘Young and the Restless’ and that type of stuff, well in our family was Guadalupe and all these really unique-sounding Spanish words and Spanish phrases. And I was talking to my grandmother the other night and it was about four or five days ago you know, and I was getting ready to come over to come over to San Diego.

And I said, “Abuela, I think am presenting on a topic actually that’s better than any novella that you’ve ever seen.” This thing is, I mean it’s a roller coaster of a ride and I, it’s not finished by any means and it’s incredible, the story of how we got to where we are with the Theranos company.

Full disclosure I actually wrote an article about a year ago for Theranos defending the company, so I actually was a, still kind of am a fan the technology if it works, but which is something we’ll get into, but I actually wrote an article defending the technology and defending the company a year back.

And then what happened is, I’ll tell you honestly and I hope if there’s people from Theranos in the room, God bless you. I really hope everything goes well for the company, I really do.

But otherwise, you know what happened is over the last year my faith has started to erode a little bit with continuous issues that have emerged, continuous findings, CMS violations, the clear exemptions status, I mean, all these different things have really started to pop up.

And I still hope really for the benefit of their investors, the benefit of the technology and for me personally because when I see a needle I want to pass out. So the first thing we’ll do is, who hasn’t heard about Theranos okay? Who is an expert in the Theranos case so I can stop talking to you? Okay good.

Alright, so we’re going to go through a couple of things here, we’re going to go through a little bit of an overview about the Theranos story for those of you who are not familiar which is good, so there’s not a ton of you who are experts in this case. So we’ll go through how we got to where we are today with Theranos.

And we’ll go through, as a Quality and Regulatory consultant you know I have to bring in the regulations to this, right. So we’ll go through some Quality and regulatory stuff towards the end, but frankly, its stuff that’s applicable to all of you, who’s in a medical device company? Yeah, well most of you, right, so, you’re going to have to learn this stuff anyway so might as well go through it in context of Theranos.

So with that being said, let’s just get started here, so, this is one of my favorite sentences and it is so apropos to this. In God we Trust, all others bring data absolutely. This is, if Theranos would have kind of followed this model I think we would be in less trouble than we are today.

However, if you look at what Theranos has been coming out with in the press and Elizabeth Holmes and her statements they believe that’s the case. And they might have valid data we just haven’t seen the extent of it quite yet.

So who knows who this is? Trivia question, William Deming sounds familiar? Father of Modern Quality … Okay, so he’s a very good friend of mine, not really a good friend of mine, he’s, I’m a little bit young for that, but that still he’s a good mentor good guy.

So today we’ll be going over a couple of things, the good of Theranos so what happened what is Theranos? Some of you are saying like, “You keep saying Theranos who the hell are these people right, or who the heck are these people? So, we’ll go through the good how they got to where they got, the bad and the ugly and that’s kind of all.

Really think about it this way, up until October 2015 everything was going swimmingly for this company. I mean there was, started emerged some questions about the technology and some questions about the founders and some questions about the scientists. But overall they were raising capital left and right, everything was going swell and then October 2015 something happened that really caused a huge downhill. And they’ve really been in defense mode ever since then, we’ll go through that.

And also what can we learn. I think one of the big take-aways is we see other companies struggling, we shouldn’t applaud that we should really say, Hmmh how can we prevent ourselves from going through the same issues?”

And the key theme of that part of the conversation will be, and this is kind of a surprise to a lot of people when I give this talk, but they key surprise here is Theranos is not the first company to go through this stuff. In fact most of the violations that the FDA cited within their quality, with their forty 483 deficiencies were really things that a lot of medical device companies struggle with.

Now there are some potential allegations of fraud and that’s not something that a lot of companies struggle with but overall in terms of just the quality systems deficiencies or the quality deficiencies, it’s not something that’s, “Oh! It’s the first time in the world we’ve seen this,” but because of the press and the way that the press has tackled this story it really has become something that a lot of people are now starting to learn, what a 483 is, what a quality deficiency is, and what we need to do to resolve them.

Who here has been through a warning letter or a 483 remediation project? Not fun, right? So six-month project manager always say, “Argh! Six months and we’re out,” and then six years later is when it happens, so yeah.

So this is Elizabeth Holmes. In 2003, Ms. Holmes dropped out of Stanford University, she was a chemical engineer. I still, regardless am a huge fan of her, I mean I think the things that she’s done regardless of, regardless of the company. Personally she’s a very strong character, she presents very well, she did a TEDMED talk a few years back that I thought was fantastic. So as a person I really actually admire her vision for this company.

In 2003 she drops out of Stanford Bio-med chemical engineering program sorry, she’s 19 years old and starts this company. At that point in time the company was not called Theranos it was called Real-Time Cures. And really the whole point of the company was to try and figure out how to extract a ton of information from a small pin prick of blood.

Everyone has had a blood test, you sit there for quite a long time and you get all these little vials filled up and we’ll see a comparison. Her goal was to see how can we make that process more seamless. So Theranos wasn’t even born yet but by 2006, there was a c
ouple of things that have happened, by 2006 they had raised almost $15-16 million. So did a Series A round and a Series B round.

Now the venture capitalists that raised that money are not what I would say are famous venture capitalists so the Peter Thiels of the world weren’t part of this deal. However, it did have some pretty good backing from some several well-established VC firms as well as private equity. And private equity was the reason that they ended up in this $9 billion evaluation which we’ll get to shortly.

But that’s Theranos, Theranos in 2006. Several patents, a company just on the rise, Elizabeth Holmes started to be featured in some magazines, but really the period between 2006 and 2013 was kind of what they call a stealth mode. There wasn’t a ton of press, there wasn’t a ton of publicity, there were a couple of articles that had come out in 2012 that were all very flattering of Elizabeth and her talents and she was named Top 30 Under 30, Top 40 Under 40 all these different minor things that really ultimately didn’t get picked up by the press, until a little bit later on.

So the Theranos promise, for those of you who are not familiar with the company, the key take-away is, today to take blood samples for different tests, you go to a lab or you go to a hospital or you go to your physician’s office. You might have somebody come to your actual house and take out blood, and that blood draw process is timely, it’s time-consuming, it’s extensive, they have to take out three huge vials depending on the number of tests that you have.

It’s pretty intense and if you have a fear of needles, one of the biggest issues with blood draws that has occurred over the last 20, 30 years the issues is really that a lot of people who should be getting diagnostic test, don’t end up getting diagnostic tests because of their fear of needles.

So Elizabeth Holmes who has a self-proclaimed fear of needles and grew up with an uncle who had several diseases that she wanted to help him get better access to, came up with this technology, or decided that there was this technology that was worth validating. Which was essentially collecting 1.3 cm-high vial called a Nanotainer, a nano-container, really tiny container worth of blood and being able to extrapolate any sort of test of battery of about 50 or 60 tests in the beginning from just that one tiny single drop of blood. So that was the key promise.

Some other key aspects of this specific company, pricing. So one of the things that you get when you have smaller Nanotainers is that you get to do, you basically get a menu. So her theory was instead of going to the doctor getting a prescription and saying, “Hey, go get this blood test,” her theory was, it should be like a grocery store where you have a menu, you say, “I want to know what my Potassium levels are. I want to know what my vitamin D levels are. I want to know what … If I could screen for this cancer, if I could screen for this.

Basically a menu of options that you would choose off the shelf essentially. And some of these options were priced about one-tenth or one-twentieth of what their reimbursed rates are on a normal basis. So obviously this was a huge huge win for the industry. So they got a lot, they started receiving a lot of press starting at about 2013 when that stealth-mode kind of emerged from its cocoon.

So about 140 of these tests starting at 2014 were under $10. So imagine you wake up one day you say I want to check my Creatinine levels and I want to do it for $10. So I go to one of the Walgreens which eventually partnered with Theranos and I take a little bit of blood and I get to check that out for $10. Not a bad deal, right? It’s like going to, I guess here it’s Vons, yes Vons and really buying something off the shelf. So that was the Theranos promise.

Another thing was this whole empowering of patients, so delivering the results right to you, not having to get screened out through a physician. This was really good, the patient empowerment movement was really emerging and 2005, 2006, 2007 and that continued with the Theranos promise. They basically wanted to make sure you had access to everything and you had transparency to everything before you actually underwent this test.

So, very promising, right. I mean if this comes to fruition who would be a huge fan of that option? I think everybody, that’s a really big thing. So this is an example of their blood test catalogue. So you see things like, for example, adenovirus DNA $29. Albumen, so that’s usually reimbursed at $30, that’s something else, $3.37. So they basically had this test menu where you can theoretically go, select your test and call it a day.

And this what the Nanotainer looks like. Their theory or their claim-to-fame was, “one tiny drop changes everything.” And again it’s a 1.29 cm container, how does it compare to other ones? Well this is what it looks like, so if you’re doing a blood draw, and again I just got my blood drawn two weeks ago. I am not a huge fan of needles but because I am big-boned, I often cannot get a needle draw through here, I often have to go through over here and it’s quite painful, it’s not fun.

So for me when I saw this originally a few years ago, I was like, “Man, this is awesome. I hope this pans out,” because sitting there while the lady is trying to stick me 900 times and say, “Uh sir, yes your patient profile is difficult to get this.” And I’m like, “Yeah, okay fine I’m fat, got it I understand.” So anyway, so that’s what this technology ended up looking like.

Now other good things started happening, the company started booming in 2013. Walgreens filed some partnerships with them. At one point they had hundreds of centers that were about to be open, and the goal was really, you’d be at a Walgreens, you go do this quick draw and you get your results.

I mean it was really kind of a, just like there’s like the mini-clinic theory for some of these other healthcare remedies, or healthcare maladies. The Theranos approach was the same thing but for blood containers. So that’s really where they started to emerge as a very very promising company.

At that point in time there was also a private equity firm that committed almost $400 million to this company. At that point in time Elizabeth Holmes became the “richest self-made billionaire in the world.” And basically at that point their valuation was $9 billion almost, $8 billion $9 billion depending on what terms you look at.

But more or less an $8 billion company which, funny enough in an article in Wired Magazine, the first cracks in the Theranos case started emerging. It started saying, “Hmmh, well, this is $9 billion company, we’re trying to look for peer review journals about their technology, we can’t find any. Okay fine, some companies like operating in stealth-mode where they’re very secretive, they want to … They don’t want to release anything, so let’s talk to some experts in the field, they surely must understand how this works.”

No experts in the field understood how it works. “Okay fine, so let’s talk to regulators, regulators definitely understand how this works, so they’re the ones regulating them.” No regulators refused … All regulators refused to speak on public comment regarding Theranos and similar technology.

So, this Wired Magazine article started, there was a couple of questions that started emerging fairly late in 2013 about this technology. Some of this also came from the two big players in the market, so Quest Diagnostics, LabCorp, 90% of you probably use them for your blood test. They represent about 20% of that overall lab market, the other 80% are in hospital, physician visits all that stuff, but 20% of a huge market, so you can imagine that they started raising their own questions as well which is interesting so.

At that point in time again, they were
an $8 billion Unicorn, everybody familiar with the term Unicorn? It’s a company that’s valuated at over a billion dollars. A lot of these unicorns, it’s a derogatory term, because they end up going bust.

At this time and point again, 8 billion, I want to keep emphasizing that, $8 billion dollars, $9 billion valuation. That means that people out there that were investing in this company without having valid data supporting the theories, they were able to put this price tag on this company.

Now one thing I failed to mention that’s interesting to know is that, the only way that this company Theranos claimed they can do their testing on that one drop little blood was through a proprietary mechanism called Edison. It was basically their own certified validated system, that was able to take this small blood drop and it’s the only system in the world that could have extrapolated all these different dozens of tests within that one drop.

That left a lot of scientists scratching their heads saying, “Man, that is fantastic. Your patents don’t really enable this technology so, if that’s the case I’d really love to see how this works, let’s see some peer-reviewed studies.” Radio silence, so keep that in mind.

2013, 2014 Elizabeth Holmes becomes a rock star, she’s a legitimate healthcare rock star. If you have not seen this phase before I’m surprised, you probably are on Facebook too much so please get off and look at your industry.

So, Fortune Inc., Wired, Forbes, funny enough one of the best articles that emerged about Elizabeth Holmes, one of the most flattering was from the Wall Street Journal. And for some of you who are familiar with this story will find out why that’s kind of ironic but Wall Street Journal featured Elizabeth in one of their articles and again, very flattering praise, everything was pitchy-keen perfect.

And I’m trying to find one quote that I love particularly that I’m going to say it for a little bit. But anyway, so 2014, 2013 she’s getting featured on all these articles, Theranos is blowing up, they find additional investors, they sign additional Safeway. There’s another area where 800 clinics are starting to get opened up for them to do their tests, I mean this company of just booming.

And then just when they were starting to get some, what they call haters in the field, right, FDA clears one of their tests. So they’re basically slamming it in everyone’s face saying, “Huh! You see that? FDA cleared one of our tests using the 510(k) mechanism which is a very stringent and complicated and world-class mechanism.” So that’s there. Okay, so they got FDA clearance, everything was going well.

Then this guy shows up. Anybody know who this is? John Carreyrou. Yup, so for months John Carreyrou from the Wall Street Journal had been calling … Again, let me be clear a lot of this is alleged because you cannot fully vet just like they haven’t validated their technology necessarily, I can’t validate some of his stuff, it’s all hearsay.

But he claims that for many months he was requesting an audience with Theranos, he was hearing rumblings in the back about how some of this technology wasn’t valid, he received some whistleblower cases, some whistleblower information from some disgruntled ex-employees. There was a lot of smoke that was starting to emerge about this company that it really couldn’t do what it claimed to do.

So John pursued this consistently over five, six months gave them several opportunities to answer. And he basically indicates that they never ever answered. So basically the bottom line here is, per his side of the story Elizabeth Holmes would take every single opportunity she could, as well as the rest of the Theranos staff to go into flattering pieces, to do featured articles where it was flattering about them.

However, when it came to his article which was questioning their technology a little bit, at that point he indicates he wasn’t planning on doing this destruction piece, he just wanted to get some clarity and actually help them. Because again Wall Street Journal was helping them get out into the public, they were doing a good service for them. So his goal was just to say, “Hey, can we just refute these things real quick and that way you guys have a smooth sailing?”

Some people on the other side say that he had a vendetta. Who knows? But long story short, he ends up publishing an article on October 2015 which really represents, I hope not the death nail, but really the beginning of the end for Theranos, if they end up being an end. I hope not because the technology is so cool and so novel that I do hope it becomes validated. But this was really the cascade of that negative spiral that happened over these last two years, or this last year sorry.

So the article’s entitled “Hot Startup Theranos Has Struggled With Its Blood-Test Technology.” Seems simple enough, nothing too crazy, it’s just saying, “Hey they’ve struggled with some of the technology.” But of the things that emerged from that article were very interesting.

The first thing was that several disgruntled employees said, “Hey, you know that Edison machine we’re supposed to be using to test our blood containers, we use it for less than 10% of our tests. We actually use off-the-shelf stuff from Siemens and from Roche to do our analytical tests in our lab.”

So that started 00:20:01 emerges an interesting comment from these disgruntled employees. Obviously Theranos’s pushback was, “They’re disgruntled employees, they want to just, they want to get back at us.” So that was one of the themes that started emerging.

The other thing that started emerging is that their knowledge of regulatory, compliance and quality practices were fairly deficient. And that will come to light here in a few minutes, but basically at the end of 2014 and by that time the article was published, it was clear that there were some issues with the way that Theranos was conducting their tests.

Now they never disclosed publicly before their fundraising that they were doing those tests with non-Edison machines with off-the-shelf machines. So you can imagine why that would be an issue. If you’re claiming that you can do a specific test or you can do something new that the industry cannot do by extrapolating, by extracting one drop of blood, test using a proprietary Edison tool and basically coming up with results at a lower cost.

Yet what you’re actually doing is extracting that blood, diluting some of the samples and sticking it into, and again allegedly diluting some of the samples and sticking it into off-the-shelf analyzers, then your technology isn’t really validated. So that’s what was emerging.

Now a lot of this was happening in parallel while Elizabeth Holmes was still building her brand. In fact, before the article came out there was a $200 million pledge from an investment group that was supposed to come out and it never emerged because of all these issues. So that was something that I think really did not board well for the company.

Now what happens is that when you have this phrase on your website “one tiny drop changes everything” and now it’s really “maybe” because we’re not sure, a couple of things ended up happening.

A company did a GAP analysis essentially of Theranos’s website pre-October 2015 and post-October 2015 and found seven major changes within the website. And a lot of it was … What do you think the changes were? Give me one example of a change in verbiage or text.

Audience (Female): 00:22:13

David Amor: Sure yeah, making the language more vague. What else?

Audience (Male): 00:22:16

David Amor: Yeah that so language that vagueness. Also started saying basically claiming that they could do things without … They could potentially do with Edison, Edison was one way of many to get the samples. A lot of dif
ferent redactions from their website that really came up during this time frame.

So as you can imagine Theranos fired back a ton of back and forth. Studies were finally released by a lot of these labs that had been saying that Theranos’s test data was not valid.

One of the biggest difficulties is, so remember how I told you that most of their tests were not conducted on their Edison machines. So the ones that were conducted on their Edison machines had significant differences between those tests and tests that were done at typical laboratories or traditional laboratories.

So you can obviously tell that this is a major concern. If you’re getting a lab test for Alkaline phosphatase and your test with Theranos is significantly different than a test you get at a clinic, that can drive a different diagnosis decision. So that’s very much an invalid test. The test is not getting you results that are actionable by a physician.

So this again was the biggest reason that Theranos was again becoming under scrutiny. Now Theranos had an answer for all of this, so in full transparency they do have an answer for absolutely everything that I’ve presented. So if you’re interested in that go to the Theranos website, there’s some information there, do your own due diligence blah blah blah.

But long story short having done this for a bit, there was some stuff there that was a bit questionable. So the full transparency they have been able to answer all these allegations. Not successfully because you still don’t find one big person, a well-known physician, a well-known diagnostic expert defending them. However, this is the case.

So more importantly just this year in January 2016, CMS found significant quality deficiencies with one of their, with several of their labs. And in fact to the point where they’re potentially revoking a lot of their tests so they do not get Medicare coverage which is a, which would be a huge blow for this company.

So CMS got involved, FDA, we haven’t talked about FDA yet but all these different agencies. New York Health Department got involved. A lot of these agencies started basically coming up with independent inspections and reports about how Theranos’s tests were invalid.

And this was the most recent piece of news and this was just this month where Theranos founder, Elizabeth Holmes was under consideration for being banned. Now whether this happens or not, if I had to guess I’d say it probably would not happen unless they don’t resolve their deficiencies in a timely manner.

However, one of their biggest issues has been when they get a deficiency notice, they have not been able to respond in a timely manner so their CSM deficiency had 10 days for response. The FDA when they issued their 483s they had several time frame for response.

Whenever they get asked for peer review journals by an independent publication or an independent organization entity, they have X amount to respond and they usually have failed to respond in a timely manner.

This is what we’re going to focus on the rest of today because this is our world, we live in FDA world. And anybody in an IVD company or a diagnostic company? Okay, so you guys know that medical devices include IVDs 00:25:42 regulations and this is our world; we live under FDA.

We kind of skipped over the FDA aspect of this remember FDA cleared Theranos’s product, you saw that clearance, I didn’t make that up, they cleared it but we’re going to get to why that’s very important.

In October 2015 not only did this Wall Street Journal article came out but boom! Perfect timing, FDA issues two 483s. Everyone familiar with what a 483 is? It’s basically systemic deficiencies within your quality system in your facility.

So this was issued and what I’m going to go through is really some take-aways from these 483s that we can learn about, because a lot of these 483s again are not new. So let’s go through some of them.

There’s fact and fiction in this Theranos case study. The fact is that there are systemic issues at Theranos; it’s clear. Some of the stuff is very basic. How can a company be valued at $9 billion without having a post-market surveillance program?

That’s like a very basic part of medical devices. That’s in special controls. You’re doing a 510(k) you get post-market surveillance. You probably learned that yesterday in the 510(k) workshop. So really that’s a fact, you can’t deny that.

The fiction is that these are unique or first see with Theranos. In fact, if you look at their different types of deficiencies: design control, complaint files, CAPA and regulatory deficiencies.

This is the FDA Trends and warning letters and 483s over the last few years, 2011-2015. So you see kind of some common hitters here: Design Controls, CAPA, Complaint Files. God if companies ever figured out how to do Complain and CAPAs and Design Controls I’d have no job so please please don’t pay attention to this right now. Go back to sleep.

This is the common recurring theme in medical device companies. I’m going to highlight in red which ones Theranos had issues with. At least they received their stuff correctly, that’s good so that’s interesting.

But can you tell that a lot of these issues are not issues quality 00:27:51 deficiency that just Theranos has, a lot of companies do. I mean again I’d say it jokingly but a lot of our work that we do is FDA remediation work so we go to companies that have these issues and fix them. So it’s common, it’s not a matter of if it happens, it’s when it happens.

In fact, if you look at some of these: CAPA, Complaint File, Design Control, Production Process Control isn’t on here but process validation is another recurring issue with companies.

It’s also part of the Quality System Inspection Technique from FDA. So Quality System Inspection Technique for those of you who may not familiar is basically the FDA’s manuals to its inspectors on what are the big focal points of the quality system.

And it lists five different sub-parts. Guess what some of those are: CAPA, Complaints, Production Process Controls, Design Controls, they’re all there so it’s not just companies struggling, FDA acknowledges that these are some of the most critical aspects of being a medical design developer.

So let’s go through some of these observations or deficiencies and see how can we prevent them, how can we avoid them? The first one, design validation did not ensure that the device conforms to defined user needs and intended uses.

So what’s really difficult here is, is the right product being built? Now in context of what you heard for design validation, maybe you don’t know exactly what A2030 says about design validation. Do you think that their design was validated?

Well again we don’t have exposure to their private records, but from what it looks like from the public, again allegedly, the design was not validated properly. It didn’t show that it can actually extract that blood and do those tests in a way that was comparable to a laboratory test.

Remember they were cleared using the 510(k) mechanism meaning thus had to prove substantial equivalence to some other test, lab test. And you saw on that previous comparison when they were put up in independent test versus a typical lab test they failed.

So just by that they were not able to meet their intended use, their indications which in this case was to diagnose certain disease states, certain molecules, certain entities using their technology. So in that point their design was not valid.

One of the things to do when you have this type of issue, one of the things that I find always when a company gets cited under A2030 for design validation is that they fail to separate what their intended use, what their indications say versus what their technical requirements are. They seem to bucket that all together and put it into one document, into one area.

And one of the things that we try to do as companies is make sure we identify who those stakeholders are, who those users are. The users can be the patients, the physicians, it all depends. And those are really what the FDA considers concept documentation per the FDA guidance.

That’s kind of the first step, understand how your intended use it translated to these user needs because ultimately that design validation is going to be performed against these needs.

So that’s one of the first things that you can take a look at within your company is do we do a good job of distinguishing between technical requirements and some of these higher level conceptual requirements that the FDA calls concept documents. So that’s one of the things to consider.

And the easiest thing to ask yourself is are you going to go to a doctor and the doctor is going to say, “Hey, I want a catheter that has a tensile strength of 10 neutons.” No, they’re probably going to say, “Hey, this thing probably shouldn’t break, because we don’t want to kill our patients.” That’s something that’s a user need and that’s something that will be validated through design validation.

Translating those into technical requirements and to design input requirements is that second step. So for example if Theranos had an IVD a Nanotainer and that Nanotainer was supposed to keep a certain volume, that might be a technical requirement. The user need that corresponds to that might be, “Hey, we need sufficient volume to do these tests.” So you go high level, lower level more in detail.

And then design validation itself: Benchtop testing, Clinical Trials, Usability studies, Simulated use testing. It didn’t seem that Theranos really was able to demonstrate to FDA that they did a lot of those things that’s typical of design validation. So really that’s why they were cited under that observation.

One of the key things for design validation is to perform these tests under simulated use conditions. So if Theranos was cited for not performing their design validation testing under simulated use conditions. What that means is you can imagine if you’re doing an assay, this actually happened to me once. We were performing, we were doing an assay for combat zones and within the test plan the combat zones were in Afghanistan and Iraq. So what do you think is something that might be different about those environments versus Minnesota where I’m currently from?

Temperature, right? So if I test this, if I use this assay outside which this assay in particular was used outside often within -20 degrees, which unfortunately is my reality for several week, do you think it might give the same results as if you tested it in 120 degree Fahrenheit? Maybe but we don’t know so we had to validate that.

So that’s simulated use conditions. Ensure that when you’re doing your medical device development you do that. And you obviously heard of some of that yesterday, but just kind of driving the home point … the point home.

Now the other thing that companies fail to do and Theranos failed to do this as well is to perform testing on what’s called Production Equivalent Units.

Production Equivalence is essentially you ask yourself this question, “Hey, the stuff that I got on the market, that’s going to go to market is it the same stuff that I use to test for design validation and to submit to FDA those results with?”

And if your answer is no or you can’t really justify why they’re equivalent, that’s why you’re going to get cited for that particular observation. So how do you do that?

You can do a bomb gap analysis, you look at your building materials. Are they the same product? No. Then why is it, why can’t I say that this material that was used for design verification and validation is “production equivalent.”

Well it has the same tensile strength, it has the same material make-up, it should perform the same as the one that we’re going to go to commercial with. Fine cool.

Use environment. Again there’s a new standard that came out ICE 62366-1, it’s a usability guidance. It’s something that everyone should be aware of. When you’re doing your testing make sure that you do human factors and that really shows that you’re considering the use environment. So all this is part of what Theranos could have done to avoid that issue.

And design input requirements were not adequately documented. We talked about design input requirements a little bit, but basically they didn’t define the product fully. So what that could mean is that in this case they didn’t translate those user needs into those more technical requirements as this observation claim.

This is another big aspect of the observations. Who knows what 14971 is? You guys probably all know it, you guys are all experts in it. Okay, using a consensus standard or a consensus-recognized standard by FDA, or in EU just the harmonized standard is really a good way of showing compliance to risk analysis.

So how do you assess and determine which risks are acceptable? Have you done Risk-Benefit analysis? One aspect of Risk-Benefit analysis which is part of risk analysis is hey what’s out there that does these same tests? It’s a comparison. Alternate therapies. In this case Theranos’s alternate therapies would probably be the diagnostics being run at different clinics.

So they would be, they would want to say, “Why is ours better or as good essentially as those?” So that’s part of the risk analysis, it’s really understanding what are the potential design issues within your product and what are the risks associated with them? And that’s part of the things that they didn’t do.

Now another observation that they had here was documents were not reviewed and approved. So this is one of the biggest things with startups it’s document control, getting things signed off, changes, getting issues, getting design changes, process changes, who approves them? How do you demonstrate that you’ve signed off on them?

Some companies use manual process, some companies use enterprise quality management systems so all these different types of Grand Avenue, Greenlight.guru. All these different platforms that you can use to do document control. Theranos failed to adequately document that their documents were reviewed and approved by somebody.

So essentially if Elizabeth Holmes were having a bad day and she said, “You know what, I want to change the way this process works,” she could have walked in there, changed the process and it never would have been approved by somebody.

So that’s what document control does within your quality system and that’s one of the things that they were cited for.

With document control having a dedicated document management system is the key to prevent those type of observations. It can be manual but a lot times you have an EQMS solution that can help you figure out electronic signatures, electronic records, maintain them, get them approved by the right people. There’s always help in that type of solution.

So this is one solution that I like to use called Greenlight.guru but basically you can kind of put all your stuff into one platform and say, “Hey, I’ve got a couple of documents routing for approval right now. We can’t make this design change until we get the VP of quality, the engineering manager, and the VP of Clinical Labs to approve this.”

So that’s one of the ways that document control can be effective within your company so you don’t get those types of observations that Theranos did.

Another observation here. Devices for which listing is required have not been listed. So this was their Nanotainer. Remember how I said FDA cleared their products? Remember? Okay, so here’s the thing, if you look at the actual clearance letter, I’m going to read it here, “Theranos today announced that it has received the US Food & Drug Administration’s clearance of its test system and test for herpes simplex 1 virus. Okay, why is this important?

If I’m a company and I say that I have 150 types of tests that I conduct using this one drop and I get FDA clearance for herpes simplex, am I allowed to now use this product for the rest of the 149 tests?

Who says true? Who says false? Oh good, you guys are smart. You guys all attended that workshop yesterday so I’m sure you guys are experts in this now. But basically yeah, this is what happened.

So regulators were claiming, “Hey, hold off for a second. We cleared you for herpes, we’re not …” Sorry that sounds really bad, we cleared you for herpes. “We cleared your product for use with herpes simplex, not all these other things that you’re saying.”

If that’s the case you either have to do two things, you have to validate each one of those tests individually or provide some sort of rationale or strategy that, “Look, 30 of these tests we’re doing one test and it’s comparable to these rest so all are validated,” but they didn’t even do that. And obviously there’s been several FOIA freedom of Information Act request for the response.

But long story short, intended use within your 510(k) is critical, so if your indication or your intended use in this case is for this specific disease state for diagnosis, you’re not going to be able to leverage that for the rest of your tests. So that was one of the keys that Theranos got in trouble for.

Clinical rationale as well. So this is a really interesting case that came up in the Scientific American where a huge panel of different physicians and laboratory technicians as well as PhDs in the matter, in the subject matter said, “Who told you,” or basically saying to Theranos theoretically said, “Who told you that blood taken from your finger is different than blood, is the same as blood taken from veins, from other veins.”

Their argument was there are a lot of molecules that you can draw analogous conclusions from if you pin-prick versus systemic blood. However, there’s a lot of proteins, lipids and other molecules that you cannot. In other words this panel came out and they said, “We have been asking for information for years, we’ve never received information. You guys must have this,” I’m paraphrasing, “You guys must have this phenomenal new-age technology that allows you to basically extract 150 type of tests from your finger, and that’s really not the case.” So that was one of the big things that came out.

Long story short, when you’re developing a test or you’re developing your indications in your intended use, you should have clinical rationale for that. I mean I know it sounds like a duh, but you’ll be surprised at how many times a company will emerge with a technology and they won’t defend their clinical rationale for selecting that technology. So that’s something to keep in mind.

Okay, so Complaints. Long story short this observation was, “Hey, you guys have no way of looking at complaints. You guys don’t assess complaints. You don’t have a formal post-market program.”

Now post-market program is part of being a Class II device. So whenever you kind of upgrade in classification for FDA so you start with Class I you end up in Class III, the types of controls you have to follow also increase. In Class I you followed General Controls. That’s things like listing your product, make sure that you’re not misbranding or adultering any of your product. Kind of some very basic, “You have to do this as a medical device manufacturer.”

As soon as you hop into Class II and you designate yourself as a Class II device that requires a 510(k), you’re going to have to file something called Special Controls. And that includes a little bit more detail but not something that a $9 billion valued company should not have invested in. So we’ll talk a little bit about what that is.

For General Controls here’s a couple of things that you need to follow as a medical device manufacturer. The first one is adulteration, misbranding, device registration and listing. What was that observation? What did it say? You failed to register and list one of your medical devices that Nanotainer because that Nanotainer you said works for this specific disease state but not for all these others. So frankly that’s what’s considered a misbranded or adulterated product. A product that is found to be deficient for its quality system and not really meeting its intended use that it was filed on.

So those are three things that Theranos even under General Controls did not comply with, so that’s a huge issue.

Under Special Controls that observation that I told you, the No Complaints Program, they did not implement a postmarket surveillance program. So what that means is once you get your product to market, you have to have a unit or at least designated individuals who are responsible for collecting complaints, customer feedback, and assessing whether or not they’re critical or they’re high-risk enough. Which in this case they didn’t have proper risk analysis so it would have been tough anyway.

But in this case you really have to do that. That’s part of being a 510(k) cleared product; having that postmarket surveillance program. Patient registries, performance standards, special labeling requirements, all this was stuff that they got clearance on for one product, but when it came to their quality system inspection, they failed some of the quality system regulation requirements. And one of those was postmarket surveillance.

So another thing is that CAPA, Corrective And Preventive Actions were not documented. So who knows what CAPA is? Who deals with CAPA on a daily basis? So long story short, when you get bad info or you get info about a potential quality system deficiency or a product deficiency or other type of non-conformance, you have to evaluate it to determine whether or not you need to take any sort of corrective actions to address it, and then preventive actions to make sure it doesn’t happen again, it doesn’t recur.

However, preventive action also includes, or actually specifically is looking at other areas of your quality system of your products to determine to make sure that this doesn’t happen, this type of issue doesn’t happen again.

So that being said CAPA is a lot of the work that we do in remediation. And there’s a couple of things that I usually recommend for CAPA which is the easy way, the easiest way to defend yourself within an FDA inspection or notified body audit. The first thing is an escalation process so a lot of people say, “Hey you know what, everything is a CAPA. Everything requires corrective action. Every non-conformance needs to be escalated.

That’s not true. In fact if you look at the two types of guidance that I’m going to look, I’m going to show you, long story short with corrective actions and preventive actions, the actions you take should be commensurate with the risk of the product, the rest of the issue, and we’ll go into that a little bit here.

But here are some things that might trigger you to have, to implement a corrective action or preventive action. Is it a high risk? Has it happened again? So if something happens over and over and over and you don’t fix it, what does the FDA think? Well you really don’t, you don’t have good management controls over your system. Your quality system is deficient because you don’t address this thing that keeps happening over and over again.

So doing trend analysis saying, “Hey, this specific issue has happened multiple times,” that’s something that might trigger a CAPA.

Now if you look at this CAPA guidance, some of you are not familiar GHTF, Global Harmonization Task Force, they’re now called something else, IMDRF, sorry. They change like every 10 years.

But basically their guidance suggests that the manufacturer should process corrective actions or preventive actions commensurate with the significance of the non-co
nformity. So one of the key things at your company you take away from this from all this hubbub basically is not every single type of nonconformance needs to have a corrective action issued. You can trend it, you can track it, you can determine by risk whether it’s critical. This is something that’s in that guidance as well as the AAMI Quality System Compendium, which is a manual that I love to use.

Says the same thing, “The potential nonconformance escalation should be appropriate for the magnitude of the problem for the risks encountered. Now a lot of these problems that Theranos is encountering, a lot of those are probably internal CAPAs. If they keep getting postmarket complaints, this Wall Street journal article, these other inputs, those are probably CAPA-worthy. Things that are high risk to the company, high risk to the product, they should probably escalate those.

And I’m going to skip over a couple of things in the interest of time. This is just a kind of escalation criteria. You take all these different inputs from audits, complaints, reviews, service and you basically determine if you need to escalate those nonconformances into your CAPA system for improvement.

So I have to skip a lot because I have some time left and obviously I don’t know how to control my time. So we’re going to skip through a couple of other things.

Purchasing controls and quality audits are things that they got in trouble with. I’ll send you these slides so you can take a look at it.

The last thing I’ll talk about though is this which is Management Responsibility. So one of the things with your quality system is your quality system grows. When you’re a small company you might have a tailored quality system that’s lean, that focuses on the four major subparts: Production, Process Controls, CAPA, Complaint Files, Design Controls. As you grow, as you become a $9 billion valuated company, you might want to implement a more robust quality management system.

And that’s part of what this aspect of the quality system regulation or subpart G under subpart B really talks about. How does management determine whether your quality system is suitable and effective? That’s done through management reviews, audits, and other internal and external influences.

So when you’re looking at suitability, is the quality system appropriate for our company? You know one thing that Theranos could have done is if they’re doing a management review they could determine, “Hey, we just got a bunch of 483s from the FDA, CMS is on us. We probably have to determine whether we have some quality system deficiencies that we need to address. And in fact one of their responses did indicate that they’re going to address them.”

Effectiveness. “Are we meeting our quality policy and objectives?” Interval and frequency. ” How often do we need to look at our quality system?” Let me ask you a question, if I’m a Class I dental floss manufacturer, am I going to take a look at my quality system more or less than Theranos who just got two 483s? What do you guys think?

Yeah so my management review might be once a year or once every two years if you will. Once a year is usually the minimum, is the minimum, but Theranos might be doing quarterly management reviews right now because they’re getting such negative publicity and negative issues with their products. So that’s one of the things you want to consider when you’re doing management reviews.

Last but not least, one of the key take-aways, is anybody an investor in this room? Okay, so one of the things that I do that I really enjoy in my job is doing due diligence for companies for investors who are looking to put money into a company. If any of these investors who would have looked at Theranos prior to their investment in 2014 and 2013 would have done a little bit of digging, they would have realized that doing quality and regulatory due diligence is a critical aspect of this process.

It’s understanding what’s their regulatory path, how their quality system looks, so they don’t get to this October 2015 timeline and basically everything hits the fan.

One of the things I very much encourage you, if you’re an investor or even if you’re a company looking to get investment, have an independent due diligence strategy with somebody. Hire a consultant, hire somebody internally, hire one of your board members to do this assessment so that you prevent these type of issues that Theranos went under.

And last but not least can Theranos come back? 23andMe is personal genomics personal gene health company. They got cited by the FDA a couple of years back now 2013 as well. The key difference is they’re now selling their director consumer kits after resolving their issues, resolving their deficiencies, validating their product. It’s not analogous, quite analogous but it’s very similar because they basically were questioned on, “Hey, do we want everybody to have access to this data, and is the data you’re providing valid?”

They underwent a two or three-year remediation effort, and I just actually ordered my 23andMe kit a couple of months ago, I’m still waiting for the results.

So it’s something that companies can come back from. They didn’t get hit as hard from the press.

But the key takeaway here just to end is basically if you’re going to get promoted in the press, especially by the tech press, you’ve got to be able to learn to live and die by them. You’re get brought up by them, you’re going to get crushed down by them as well, so you’ve got to take the good and the bad.

Yep, and that’s kind of the takeaway here. So when you raise $9 billion make sure you have a quality system that’s robust. So that’s it for me.

Joe Hage: Dave, you are a very gifted speaker.

(Applause)

You’re a gifted speaker, a brave speaker, thank you.

David Amor: If I end up dead tonight you know where to go.

Joe Hage: No but don’t leave the podium just yet. While we’re out of time I do want to ask one question. You mentioned one of the things that you do is you help companies do due diligence before investing. Quick plug for yourself, what else might we hire you as a consultant to do for us?

David Amor: So right now the focus of my specific practice is I do a lot of work in combination of a products so 505B2s are kind of our preferred mechanism. It’s basically a combination product, a drug and a device which who regulates it from FDA. Depends on that primary motive action, is it primarily a drug or primarily a device. When you stick them both together you basically submit to the FDA branch that is that motive action.

So we do a lot of that work, the regulatory strategy, quality system strategy as well as other regulatory and quality stuff which I won’t bore you with because you’re probably going to fall asleep so.

Joe Hage: Okay. Thank you. David Amor.

David Amor: Thanks everybody.

(Applause)

Joe Hage: Nice job.

David Amor: Thank you.

:::The End:::