What Happens when Digital Health Fails to Deliver Evidence? A Story of Accountability with Dr. Chelsea Polis

What Happens when Digital Health Fails to Deliver Evidence? A Story of Accountability with Dr. Chelsea Polis

Clinical outcomes and evidence has been the topic du jour in digital health as evidenced by thinkpieces, surveys, and even venture capitalists (VCs) becoming more interested in the space. Overlooking clinical efficacy can have major financial and legal consequences, or at the very least, the “Fear of Looking Foolish” (FOLF) which has its own effects.

More importantly, we need robust research to determine how digital health implementations affect patient care and outcomes. As we are finding in the development of MDisrupt’s Digital Health Intelligence Database, designed to provide a deeper understanding of what the ecosystem’s research outcomes actually are, there are a number of noted gaps. There are companies that do not have research confirming their product’s effectiveness or impact, others do, but the research does not actually support their marketing claims, while others  have research that is not made publicly available for consumption and scrutiny. As a result, there is confusion as to what works and what doesn’t. Clinicians can’t easily figure out which products, if any, to recommend, patients don’t know what to believe, and payers have no basis on which to make any coverage decisions.

This call for proof in digital health with clinical evidence is no longer optional, or a ‘nice to have’. However, this is not new. In this month’s blog, we are joined by Dr. Chelsea Polis, a senior reproductive health epidemiologist with the Center for Biomedical Research at the Population Council. She has been an advocate for clinical evidence in digital applications for many years, well before most of us even started having this conversation. 

Valley Electronics, a company that manufactures a fertility thermometer called Daysy, made claims that Daysy could be used to prevent pregnancy with >99% effectiveness. Valley Electronics claimed that the thermometer was essentially as effective at preventing pregnancy as an intrauterine device (IUD) – which is one of the most highly effective contraceptive methods available with FDA-reviewed outcomes studies showing >99% effectiveness. In 2017, Dr. Polis reached out privately and directly to Valley Electronics to raise her concerns, but those efforts did not result in meaningful changes to the company’s practices. When the company then published a study of the device in 2018, Dr. Polis examined that study and found the methodology to be extremely flawed. She was ‘horrified’ by the study’s methodology and conclusions, and by the fact that flawed data were being used to make misleading claims to people seeking to prevent unintended pregnancy. So, she decided to do something about it. Dr. Polis was kind enough to answer questions about what happened, what is crucial for companies to know, and what she wants to see the industry deliver. 

Thanks for joining us, Dr. Polis! This story is especially important for all of us asking for clinical relevance to support effective clinical decisions and outcomes for their claims. After all, it is relevant and necessary to do so for patients. As a quick recap, I understand that you reached out to Valley Electronics, the makers of Daysy, to discuss your concerns about their evidence base prior to making any public statements or publications. When that failed to result in meaningful action, you also wrote a peer-reviewed commentary debunking their flawed study and calling for their study to be retracted, and published it in the journal their study had been originally published in. The journal considered your concerns, and after independent review, agreed that their methods were flawed and agreed to retract the paper from the scientific literature. What made you decide to comment and sound the alarm in the first place?

This is an important topic – I am so glad to be able to shed some light on it and grateful that you wanted to discuss it with me! Yes, I was incredibly concerned about the flawed publication and the misleading claims that the company had made to current and potential consumers about the contraceptive effectiveness of this device, which was then being sold for about $330. Fertility and contraception are a huge area of growth in digital health and women’s health. The market has been flooded with fertility-related apps and devices over the last few years, however, many do not appear to be based on robust scientific evidence. Such practices negatively affect patients – people could use these technologies, thinking they are using a highly effective form of contraception, but if the reality doesn’t match the marketing claims, they could experience an unintended pregnancy, which can have a wide variety of health, economic, social, and other consequences. 

What was the issue with the publications that Valley Electronics was using to support Daysy’s fertility tracking?

In 2017, after identifying all available studies on Daysy, I determined that (1) the existing evidence base for claims on Daysy’s contraceptive effectiveness was flawed, and (2) the Daysy device was not cleared or approved by the Food and Drug Administration (FDA) to be marketed for contraceptive purposes. So, in September 2017, I directly and privately emailed Valley Electronics to raise concern that the language in their marketing materials was not sufficiently evidence-based, and could place customers at risk of unintended pregnancy. I described several specific methodological concerns with the main study that they were citing at that time – for example, that it was based not on people using the actual Daysy device (but instead, on predecessor devices), that it lacked robust measurement of how many unintended pregnancies had occurred to people using these devices, and that there were major mathematical errors in the way in which they calculated the Pearl Index (a measure of contraceptive effectiveness). When I asked them if they would be willing to do the kind of study necessary to make robust claims about contraceptive effectiveness, they told me that “the costs and benefits are all out of proportion” to undertake such an effort. I found this to be a deeply offensive and frightening perspective, one that failed to consider the costs of a potential unintended pregnancy to their consumers.

Unfortunately, raising these concerns to the company did not stop them from continuing to claim >99% contraceptive effectiveness nor from publishing subsequent inaccurate analyses. In March 2018, they published a new study purporting to measure the contraceptive effectiveness of Daysy. It had many of the same flaws as the prior study, and several additional flaws – such as inappropriately excluding people who may have been at the greatest risk of unintended pregnancy from the analysis, and various problems with how the data were collected and analyzed. On social media, they inaccurately referred to it as a “clinical trial” (it was only a mailed retrospective survey) and inaccurately claimed that the study was “independent” (it was funded by the company and co-authored by company staff). Their flawed findings spread widely on social media, presenting a danger to informed contraceptive choice and public health – perhaps especially for younger people hoping to avoid unintended pregnancy and who are more avid consumers of social media.

So, I wrote and, in June 2018, published a peer-reviewed commentary outlining my concerns. I called for their flawed study to be retracted from the scientific literature and for greater regulatory oversight of marketing materials on contraceptive effectiveness. The journal retracted the company’s study in May 2019 – and the retraction and company’s response to it was covered a bit in the media.  

When I published my commentary, I also submitted an allegation of regulatory misconduct about the company to the FDA. Interestingly, Valley Electronics had received warning letters from FDA in years past for marketing their earlier devices (LadyComp, BabyComp, Pearly) as contraception without FDA approval to do so. Also, in 2016, the company Tweeted: “Did you know the FDA blocks us from calling Daysy birth control or contraception? But she’s 99.3% effective. #ThxBirthControl.” They deleted that Tweet after I pointed it out on Twitter, but I think these examples give a sense of how the company has conducted itself with respect to regulatory requirements. Anyway, I was gratified to eventually learn that my complaint prompted the FDA to make the company change their marketing language.

I looked at the website today. There are more warnings now, although it took some time to get there. What did you hope to achieve?

There has to be more oversight on these kinds of products – they need to be studied for effectiveness using robust scientific methodology, and there has to be regulatory oversight regarding how these products are marketed to people to avoid these situations. I don’t think that most consumers are aware that these devices are sometimes inappropriately marketed as contraceptive methods, and they may assume that these products have the same rigor of evidence as other contraceptives have. The marketing for any contraceptive product needs to be clear and in line with what solid research actually demonstrates. 

I agree completely, patients can be caught in the middle. Going back to Daysy, I understand the company actually sued you for defamation, albeit unsuccessfully. I cannot imagine that was easy even if you had the research to back you. Do you think this dismissal will help others who might want to call out companies without accurate information?

The company claimed that I had defamed them – for example, they felt it was defamatory for me to refer to their retracted study as “junk science”. So, rather than take my concerns seriously – as both a scientific journal and the US FDA did – they sued me. I believe that this was a classic “SLAPP” suit (“Strategic Lawsuit Against Public Participation”) – filed in hopes of silencing my voice as a scientist and contraceptive expert. 

The lawsuit was a long, stressful process – even though I was incredibly fortunate to have an incredible team of lawyers at Arnold & Porter, led by Ms. Dori Hanswirth, who took on my case pro bono. The case was filed against me in May 2020 and dismissed in August 2021. But then, in October 2021, Valley Electronics filed an appeal to the Second Circuit Court of Appeals, and in March 2022, the Second Circuit affirmed the District Court’s decision to dismiss the case. It was of course a huge relief when I found out about each dismissal. I learned that many SLAPP lawsuits are filed without the hope of actually winning, but instead, are filed with a goal of intimidating, silencing, distracting, and potentially bankrupting the defendant. It often works. I believe that they wanted to silence a scientist with expertise on fertility awareness based methods and expertise on calculation of contraceptive effectiveness for speaking out truthfully and in evidence based ways about their scientific, marketing, and regulatory behaviors. Sadly, I am now well aware that I am not the only scientist to whom this kind of thing has happened – and I have learned that we may not be aware of the full magnitude of the issue, since many people who are sued are not able to speak out about it, as I decided to do.

I certainly hope this case helps to put companies on the alert that scientists who speak out about whether scientific evidence supports a given product’s claims have a right to raise well-justified concerns in the public domain. Attempts to silence people who stick up for scientific data and regulatory oversight, on behalf of protecting consumers and public health, are absolutely chilling. The case against me was ultimately dismissed, as it certainly should have been, and let’s hope that precedent has now been strengthened and that future scientists will be less likely to have to face this kind of traumatic experience – which also detracts from the time and energy we have available to focus on pushing forward scientific knowledge. 

I thank you for doing that and standing up for this – patients will not know these kinds of details, they don’t read research publications and unknowingly end up using these products. MDisrupt is committed to getting researcher and clinician voices into digital health. From your perspective, what is the most likely reason companies do not undertake  the right research? Does adding a clinician or researcher earlier to their process help?

As somebody familiar with the process of contraceptive research and development, I know how challenging, time-consuming, and expensive it can be to take a contraceptive method from idea to market. Numerous studies must be conducted to fully demonstrate a product’s safety and effectiveness before it can be offered to the public. Organizations like the Center for Biomedical Research at the Population Council, a non-profit organization that I joined in September 2022, have worked in contraceptive development for decades, and have brought multiple safe and effective contraceptive options to the global public, including IUDs, implants, contraceptive vaginal rings, and more. We take quite seriously the need to abide by appropriate scientific and regulatory procedures – because we care deeply about people’s right to have clear and accurate information about their contraceptive options. Our scientific experts know how to design appropriate scientific studies, and our regulatory experts work with bodies like the FDA to collaboratively ensure appropriate oversight. 

Valley Electronics stated to me that they found the “costs and benefits all out of proportion” of doing the necessary kinds of study to appropriately examine contraceptive effectiveness. We need companies that are committed to protecting public health and human rights, not preoccupied with their bottom line. There are many scientific and regulatory experts who would be thrilled to help companies ensure that they appropriately protect public health. I’m a huge fan and supporter of MDisrupt because they fill an essential gap by ensuring that such experts can be more easily linked with companies to help guide them towards ethically and scientifically robust studies and practices. However, for that formula to work, companies need to be committed to listening to and using the advice that relevant, trusted experts offer to them. When companies are not willing to do the right thing, they need to be called out – and for that, we need to be brave and make use of our right to free speech in the public domain and in evidence-based, well-justified ways.

What is your advice to companies that want to get the right data and research outcomes but are unsure how to do so?

As just one example, MDisrupt is an excellent resource for finding people who want to help you! You can also reach out to other established experts in the relevant field, or to organizations who have successfully put similar products on the market in evidence-based, ethical ways. We all need to work together to ensure that innovation is supported by rigorous science and robust regulatory oversight – in order to make people’s lives easier, better, and healthier. 

Thank you for joining us Chelsea and thank you for all your research into reproductive health and care. 

It should be emphasized that when it comes to health product marketing and claims, it’s crucial to establish a strong foundation of clinical evidence right from the beginning. The goal is not to simply react to concerns as they arise, but to proactively ensure that claims made about a product are backed up by robust clinical data. By doing so, companies can build trust with consumers, health professionals, and regulators, and avoid potential legal and reputational issues down the line. Additionally, investing in rigorous clinical studies can provide valuable insights into the safety and efficacy of a product, and ultimately improve patient outcomes. In short, establishing a strong foundation of clinical evidence is not only necessary for responsible marketing practices, but is also essential for the long-term success of a health product.

Aditi U Joshi

Aditi U Joshi MD, MSc, FACEP

Dr. Aditi U. Joshi is an experienced Emergency Medicine Physician, Telehealth expert, and consultant. She is working on behalf of health systems, companies, and individual practices to determine which solutions best meet health systems buyers’ needs. Nationally, she serves as the Chair for the Telehealth committee of the American College of Emergency Physicians, chairs the Society of Academic Emergency Medicine’s Telehealth Fellowship Committee, part of the American Telemedicine Association’s research review committee and an Advisor for the AMA’s Digital Payment Advisory Group. Her book on Telehealth Success – how to build and understand the ecosystem will be published by Forbes in Spring of 2023.

At MDisrupt we believe that the most impactful health products should make it market quickly. We do this by uniting digital health companies with experts from the healthcare industry to help them accelerate their time to market responsibly.

Our expert consultants span the healthcare continuum and can assist with all stages of health product development: This includes regulatory, clinical studies and evidence generation, payor strategies, commercialization, and channel strategies. If you are building a health product, talk to us.

Raising the Bar on Digital Health: Clinical Outcomes are the New Standard

Raising the Bar on Digital Health: Clinical Outcomes are the New Standard

Another year beginning is a great time to take stock. For those of us who work in digital health, there has been an increasing trend to talk about evidence and outcomes. Despite recent contraction, the digital health market remains a huge ecosystem. For those wanting to know what product to use, it seems impossible to understand, categorize and differentiate between them, especially in their clinical relevance. This disconnect is leading to perhaps ineffective products being used and some good products ignored. Without clear evidence of what something will accomplish, and whether it will improve or maintain quality of care, the engagement into the healthcare system will remain siloed and separated, physicians won’t want to prescribe them, and payors can’t justify paying for them. 

Over the last year there have been more accounts and calls for improving clinical outcomes. The Rock Health/JMIR paper outlined the clinical robustness vs funding of companies finding that there was little correlation – suggesting asymmetry in what works clinically and what is financially valued. While they found that clinical robustness was low in general, companies with higher scores were more likely to be bought by employers. Which means that this robustness and testing is possible and has a real effect on market success. A Finn Partners global analysis of 12,000+ digital health ventures reveals that just 17% have significant clinical strength and predict that investors are going to go from FOMO to FOLF (Fear of Looking Foolish) by increasing due diligence on clinical outcomes. The WHO has guidelines recognizing that clinicians want to see these outcomes to better adopt and use these products. Sachin Jain calls it the epidemic of magical thinking and that we need ‘less virtue signaling and more virtue’. 

2023 gives us a fresh start and opportunity to be better at building, valuing and funding companies and products. Digital Health will not survive or be taken seriously without it. 

The other challenge is actually finding a centralized, organized way to search and compare clinical outcomes. To this end, we at MDisrupt built the MDisrupt Digital Health Intelligence database (MDHI) that categorizes companies by components such as product type, funding round, clinical staffing, and clinical research. The clinical research is the crux of this project. We looked at the type of research design, clinical claims, and the backing of those claims within the research. 

Doing that for 650 companies has been no joke as evaluating research requires actually reading the papers to understand the methods, design and whether the outcomes are ‘real’. Comparing the research in peer reviewed journals vs company’s websites and claims, there were broad categories of companies noted:

    1. a company understands their product and their research aligns.
    2. a company has a potentially good product but:
      • their research studies are not designed to test what their website claims
      • the study does not align with their vision
      • their product can likely have broader use cases than their research suggests
      • products and research studies do not seem to align at all and will lead to confusion in their marketing and claims.
    3. companies and products that only measure patient/consumer engagement and not clinical outcomes
    4. companies and products that have only a sales pitch and no research.

It is frustrating to see the last 3 scenarios, especially as much of that can be avoided with careful planning and bringing on the right expertise early in the company and product’s inception.

I’ve seen interesting findings and companies doing good research. Koa Health, a Series A funded digital CBT app, for example, has some well thought out research papers, good early clinical findings, and also mentions when things are not exactly working. Compare that to Spring Health, a huge unicorn company, with one listed cohort study. Both had good studies although a difference in number and design. But the difference in funding between them is stark. What this means seems to be in line with others’ findings.

Improving Clinical Research & Sharing Outcomes

Over the next few months we will be sharing a number of our findings from the database, what the market looks like, what gaps exist and what is the alignment with clinical outcomes and funding. Until then, we want to share some ways digital health companies can improve their clinical research and share their outcomes. Here’s how:

Adopt Early Clinician Input

This seems obvious but the number of companies that do not have any staff or advisors that are physicians/clinicians is staggering.

  • It is impossible for those who have never practiced, been in a clinical environment or done patient care to truly understand what it is like to do so. Getting the right physician input for this is crucial.
  • Clinicians can help plan out your research strategy – this can include PhDs, statisticians, and researchers as well. Not all research design is the same. For example, randomized controlled trials are the gold standard. However, it is not always feasible to do these, especially early on. If looking for research planning, make sure it is someone who understands that and is up to date on current literature and practice.
  • Ensure you have the right physician voice. It should align with your goals, the community you are trying to affect and the specialty. For example, if your goal is to use a device to decrease readmissions for Congestive Heart Failure (CHF) – ensure you have a doctor who treats that population in the community you are targeting. It is more likely they will be up to date on the research.

I recognize that it might be expensive to hire physicians, especially for early-stage startups. However, having a contractual, advisory, or fractional relationship will allow for it to be flexible and less expensive. Having a couple of hours and having targeted questions with an expert can save lots of time and headache later.

Understand the clinical environment/the market

What has already been accomplished in this space? I do not mean simply in the digital health space. I mean for the disease process that you are targeting – what is the current practice, what has been the research, what are the current interventions etc? Some diseases have been much more studied in the digital health space, such as diabetes. The process some of those successful products have taken can be replicated.

This research is necessary to create clear goals. Not all products need to stand alone – some can be used for multiple purposes or diseases. A pulse oximeter, for example, is not only used for monitoring COVID or COPD. Having knowledge of that can clear the targets or allow for multiple simultaneous trials.

Ensure your strategy and product align

Once you have an idea of what is currently done from a clinician, then it makes sense to try to see it from an outside perspective – as someone who won’t be as committed to doing things as ‘they were’.

What I appreciate about the digital health founders I’ve met is their commitment to changing some of the roadblocks of medicine. Physicians are very good at working around inefficient workflows just to get through the day – having outside perspectives are what can dramatically change our processes for the better.

Create clear research goals

Do you have PICO formated questions?
The first step is to clearly define the question you are looking to answer. The PICO format is a good starting place. Who is the patient or population specifically? The population is never everyone. What is the intervention specifically? Most products have several use cases, but which of those are you testing specifically? What are you comparing it to? What are the other solutions (or lack of solutions) the product will transform? And finally, what is the outcome you will demonstrate, clinical, financial, engagement, cost?

Do you know what your product is truly capable of?
It is okay not to know all the answers and why it is important to get other input. That will help figure out what you can study.

What do you need to know about how it works?
What clinical outcomes and quality measures do you need to learn to prove the solution works (or doesn’t). Much of this does not need to be redone. There are clinical quality measures that already exist that can be applied. A recent study looking at the HEDIS measures and how well telehealth visits adhered to them is a good example.

What will you do with that information?
Is this to ensure it is safe, gain a pilot, improve marketing, expand the use cases, sell to payors, further the research for a disease process etc. Likely it will be multiple reasons.

Which journals and publications will you publish in?
Not all are the same, however, it should be peer reviewed regardless of the impact factor. You can create a list of journals to submit to as rejections are common (and not to be taken personally!).

Are you ready to pivot depending on the outcomes?
Negative studies exist or your product may not work for what you had originally thought. It doesn’t mean it is useless – are you ready to change its use case or tweak it to try again? By the way, the answer can be No. Not everything will work but knowing that is valuable information.

Some of these might be hard to fully understand as we can all be siloed within our own worlds. For example, if you have a solution for treating diabetes, you have to understand how diabetes is currently treated, what the patient and clinician process is, what current digital solutions exist, what their research shows and then plan out your strategy and clinical outcomes. Designing a research study seems easy but requires a thoughtful process and an understanding of the different types, statistics and what can be technically proven.

Engage other stakeholders

This is easier said than done. You can have a well thought out plan and nowhere to run a research study as it requires engaging a place for a pilot with someone who can be a PI and get IRB approval. This is why it is worthwhile to have clinicians and researchers on your staff or advisory board that can help set that up with health systems, clinics, and patients.

There are always innovation hubs, pitch contests and incubators that also can aid with getting in front of the right places.

Share your findings

Bizarrely, the Rock Health group found that not all those with clinical outcomes shared what they found, and claims made by digital startups are low. In an increasingly competitive environment, selling what the clinical outcomes are is going to only help you. This is mainly because

  • Clinicians want to know the product is safe and effective
  • Payors want to be able to invest in a solution that works
  • Patients are increasingly the consumers of their health, and they want to know this is going to help them.

In the example of diabetes, there are many solutions so having evidence-demonstrated ideas of what works is going to help in all 3 categories. Many Type 1 diabetics are especially well educated about their solutions, and so they can give valuable feedback if the clinical outcomes are there and sold to them.

Also, don’t be afraid of negative studies or those that don’t prove that the product worked in the way it was used. That can be used to improve, iterate, or pivot your direction. It will also aid others who want to improve care for a targeted population. It is not often that we get this data as it is far more common to see studies that are positive. A company that published a negative study decided to continue on, pivoting their goals and then demonstrating where their product could work. This instills far more trust in their vision and improves their chance of longstanding success.

Don’t embellish

On the flip side, do not market something you cannot backup. It will come out and when selling to employers or payors, it will become clear. I will give the benefit of the doubt that it is not nefarious – not all research is equal and there are badly designed studies or the results are not actually statistically or practically significant. Make sure you do your due diligence and ask if you do not know.

Share your data

Companies selling to employers had higher clinical evidence and demonstrable claims compared to other customer groups according to Rock Health. So it’s clear that employers and payors want this information. It also has to be targeted to your vision and goals. If you know you want to target patients with CHF who end up in the hospital once a month, that is what you sell to payors. Having a dietary monitoring program may be broader but can be applied to a whole host of different prevention and treatment programs. Just be clear what you are fixing whether narrow or large.

In conclusion, this is healthcare. We are aiming to improve care, quality of life, and do no harm. To aid in the collection of understanding the clinical utility of the digital health products out there, we all have to be part of this change.

  • Good research design is an extension of understanding your product and strategy. Make sure they are in alignment.
  • Product clinical outcomes are necessary for a forward-thinking strategy. No end user will want to use something that is not demonstrated to improve outcomes.
  • A database of companies and products with relevant clinical outcomes is needed. It is impossible to understand what works and what doesn’t without it. This can inform the future of digital health.

Here’s to more and better clinical evidence in 2023!

Aditi U Joshi

Aditi U Joshi MD, MSc, FACEP

Dr. Aditi U. Joshi is an experienced Emergency Medicine Physician, Telehealth expert, and consultant. She is working on behalf of health systems, companies, and individual practices to determine which solutions best meet health systems buyers’ needs. Nationally, she serves as the Chair for the Telehealth committee of the American College of Emergency Physicians, chairs the Society of Academic Emergency Medicine’s Telehealth Fellowship Committee, part of the American Telemedicine Association’s research review committee and an Advisor for the AMA’s Digital Payment Advisory Group. Her book on Telehealth Success – how to build and understand the ecosystem will be published by Forbes in Spring of 2023.

At MDisrupt we believe that the most impactful health products should make it market quickly. We do this by uniting digital health companies with experts from the healthcare industry to help them accelerate their time to market responsibly.

Our expert consultants span the healthcare continuum and can assist with all stages of health product development: This includes regulatory, clinical studies and evidence generation, payor strategies, commercialization, and channel strategies. If you are building a health product, talk to us.