What Happens when Digital Health Fails to Deliver Evidence? A Story of Accountability with Dr. Chelsea Polis

Clinical outcomes and evidence has been the topic du jour in digital health as evidenced by thinkpieces, surveys, and even venture capitalists (VCs) becoming more interested in the space. Overlooking clinical efficacy can have major financial and legal consequences, or at the very least, the “Fear of Looking Foolish” (FOLF) which has its own effects.

More importantly, we need robust research to determine how digital health implementations affect patient care and outcomes. As we are finding in the development of MDisrupt’s Digital Health Intelligence Database, designed to provide a deeper understanding of what the ecosystem’s research outcomes actually are, there are a number of noted gaps. There are companies that do not have research confirming their product’s effectiveness or impact, others do, but the research does not actually support their marketing claims, while others  have research that is not made publicly available for consumption and scrutiny. As a result, there is confusion as to what works and what doesn’t. Clinicians can’t easily figure out which products, if any, to recommend, patients don’t know what to believe, and payers have no basis on which to make any coverage decisions.

This call for proof in digital health with clinical evidence is no longer optional, or a ‘nice to have’. However, this is not new. In this month’s blog, we are joined by Dr. Chelsea Polis, a senior reproductive health epidemiologist with the Center for Biomedical Research at the Population Council. She has been an advocate for clinical evidence in digital applications for many years, well before most of us even started having this conversation. 

Valley Electronics, a company that manufactures a fertility thermometer called Daysy, made claims that Daysy could be used to prevent pregnancy with >99% effectiveness. Valley Electronics claimed that the thermometer was essentially as effective at preventing pregnancy as an intrauterine device (IUD) – which is one of the most highly effective contraceptive methods available with FDA-reviewed outcomes studies showing >99% effectiveness. In 2017, Dr. Polis reached out privately and directly to Valley Electronics to raise her concerns, but those efforts did not result in meaningful changes to the company’s practices. When the company then published a study of the device in 2018, Dr. Polis examined that study and found the methodology to be extremely flawed. She was ‘horrified’ by the study’s methodology and conclusions, and by the fact that flawed data were being used to make misleading claims to people seeking to prevent unintended pregnancy. So, she decided to do something about it. Dr. Polis was kind enough to answer questions about what happened, what is crucial for companies to know, and what she wants to see the industry deliver. 

Thanks for joining us, Dr. Polis! This story is especially important for all of us asking for clinical relevance to support effective clinical decisions and outcomes for their claims. After all, it is relevant and necessary to do so for patients. As a quick recap, I understand that you reached out to Valley Electronics, the makers of Daysy, to discuss your concerns about their evidence base prior to making any public statements or publications. When that failed to result in meaningful action, you also wrote a peer-reviewed commentary debunking their flawed study and calling for their study to be retracted, and published it in the journal their study had been originally published in. The journal considered your concerns, and after independent review, agreed that their methods were flawed and agreed to retract the paper from the scientific literature. What made you decide to comment and sound the alarm in the first place?

This is an important topic – I am so glad to be able to shed some light on it and grateful that you wanted to discuss it with me! Yes, I was incredibly concerned about the flawed publication and the misleading claims that the company had made to current and potential consumers about the contraceptive effectiveness of this device, which was then being sold for about $330. Fertility and contraception are a huge area of growth in digital health and women’s health. The market has been flooded with fertility-related apps and devices over the last few years, however, many do not appear to be based on robust scientific evidence. Such practices negatively affect patients – people could use these technologies, thinking they are using a highly effective form of contraception, but if the reality doesn’t match the marketing claims, they could experience an unintended pregnancy, which can have a wide variety of health, economic, social, and other consequences. 

What was the issue with the publications that Valley Electronics was using to support Daysy’s fertility tracking?

In 2017, after identifying all available studies on Daysy, I determined that (1) the existing evidence base for claims on Daysy’s contraceptive effectiveness was flawed, and (2) the Daysy device was not cleared or approved by the Food and Drug Administration (FDA) to be marketed for contraceptive purposes. So, in September 2017, I directly and privately emailed Valley Electronics to raise concern that the language in their marketing materials was not sufficiently evidence-based, and could place customers at risk of unintended pregnancy. I described several specific methodological concerns with the main study that they were citing at that time – for example, that it was based not on people using the actual Daysy device (but instead, on predecessor devices), that it lacked robust measurement of how many unintended pregnancies had occurred to people using these devices, and that there were major mathematical errors in the way in which they calculated the Pearl Index (a measure of contraceptive effectiveness). When I asked them if they would be willing to do the kind of study necessary to make robust claims about contraceptive effectiveness, they told me that “the costs and benefits are all out of proportion” to undertake such an effort. I found this to be a deeply offensive and frightening perspective, one that failed to consider the costs of a potential unintended pregnancy to their consumers.

Unfortunately, raising these concerns to the company did not stop them from continuing to claim >99% contraceptive effectiveness nor from publishing subsequent inaccurate analyses. In March 2018, they published a new study purporting to measure the contraceptive effectiveness of Daysy. It had many of the same flaws as the prior study, and several additional flaws – such as inappropriately excluding people who may have been at the greatest risk of unintended pregnancy from the analysis, and various problems with how the data were collected and analyzed. On social media, they inaccurately referred to it as a “clinical trial” (it was only a mailed retrospective survey) and inaccurately claimed that the study was “independent” (it was funded by the company and co-authored by company staff). Their flawed findings spread widely on social media, presenting a danger to informed contraceptive choice and public health – perhaps especially for younger people hoping to avoid unintended pregnancy and who are more avid consumers of social media.

So, I wrote and, in June 2018, published a peer-reviewed commentary outlining my concerns. I called for their flawed study to be retracted from the scientific literature and for greater regulatory oversight of marketing materials on contraceptive effectiveness. The journal retracted the company’s study in May 2019 – and the retraction and company’s response to it was covered a bit in the media.  

When I published my commentary, I also submitted an allegation of regulatory misconduct about the company to the FDA. Interestingly, Valley Electronics had received warning letters from FDA in years past for marketing their earlier devices (LadyComp, BabyComp, Pearly) as contraception without FDA approval to do so. Also, in 2016, the company Tweeted: “Did you know the FDA blocks us from calling Daysy birth control or contraception? But she’s 99.3% effective. #ThxBirthControl.” They deleted that Tweet after I pointed it out on Twitter, but I think these examples give a sense of how the company has conducted itself with respect to regulatory requirements. Anyway, I was gratified to eventually learn that my complaint prompted the FDA to make the company change their marketing language.

I looked at the website today. There are more warnings now, although it took some time to get there. What did you hope to achieve?

There has to be more oversight on these kinds of products – they need to be studied for effectiveness using robust scientific methodology, and there has to be regulatory oversight regarding how these products are marketed to people to avoid these situations. I don’t think that most consumers are aware that these devices are sometimes inappropriately marketed as contraceptive methods, and they may assume that these products have the same rigor of evidence as other contraceptives have. The marketing for any contraceptive product needs to be clear and in line with what solid research actually demonstrates. 

I agree completely, patients can be caught in the middle. Going back to Daysy, I understand the company actually sued you for defamation, albeit unsuccessfully. I cannot imagine that was easy even if you had the research to back you. Do you think this dismissal will help others who might want to call out companies without accurate information?

The company claimed that I had defamed them – for example, they felt it was defamatory for me to refer to their retracted study as “junk science”. So, rather than take my concerns seriously – as both a scientific journal and the US FDA did – they sued me. I believe that this was a classic “SLAPP” suit (“Strategic Lawsuit Against Public Participation”) – filed in hopes of silencing my voice as a scientist and contraceptive expert. 

The lawsuit was a long, stressful process – even though I was incredibly fortunate to have an incredible team of lawyers at Arnold & Porter, led by Ms. Dori Hanswirth, who took on my case pro bono. The case was filed against me in May 2020 and dismissed in August 2021. But then, in October 2021, Valley Electronics filed an appeal to the Second Circuit Court of Appeals, and in March 2022, the Second Circuit affirmed the District Court’s decision to dismiss the case. It was of course a huge relief when I found out about each dismissal. I learned that many SLAPP lawsuits are filed without the hope of actually winning, but instead, are filed with a goal of intimidating, silencing, distracting, and potentially bankrupting the defendant. It often works. I believe that they wanted to silence a scientist with expertise on fertility awareness based methods and expertise on calculation of contraceptive effectiveness for speaking out truthfully and in evidence based ways about their scientific, marketing, and regulatory behaviors. Sadly, I am now well aware that I am not the only scientist to whom this kind of thing has happened – and I have learned that we may not be aware of the full magnitude of the issue, since many people who are sued are not able to speak out about it, as I decided to do.

I certainly hope this case helps to put companies on the alert that scientists who speak out about whether scientific evidence supports a given product’s claims have a right to raise well-justified concerns in the public domain. Attempts to silence people who stick up for scientific data and regulatory oversight, on behalf of protecting consumers and public health, are absolutely chilling. The case against me was ultimately dismissed, as it certainly should have been, and let’s hope that precedent has now been strengthened and that future scientists will be less likely to have to face this kind of traumatic experience – which also detracts from the time and energy we have available to focus on pushing forward scientific knowledge. 

I thank you for doing that and standing up for this – patients will not know these kinds of details, they don’t read research publications and unknowingly end up using these products. MDisrupt is committed to getting researcher and clinician voices into digital health. From your perspective, what is the most likely reason companies do not undertake  the right research? Does adding a clinician or researcher earlier to their process help?

As somebody familiar with the process of contraceptive research and development, I know how challenging, time-consuming, and expensive it can be to take a contraceptive method from idea to market. Numerous studies must be conducted to fully demonstrate a product’s safety and effectiveness before it can be offered to the public. Organizations like the Center for Biomedical Research at the Population Council, a non-profit organization that I joined in September 2022, have worked in contraceptive development for decades, and have brought multiple safe and effective contraceptive options to the global public, including IUDs, implants, contraceptive vaginal rings, and more. We take quite seriously the need to abide by appropriate scientific and regulatory procedures – because we care deeply about people’s right to have clear and accurate information about their contraceptive options. Our scientific experts know how to design appropriate scientific studies, and our regulatory experts work with bodies like the FDA to collaboratively ensure appropriate oversight. 

Valley Electronics stated to me that they found the “costs and benefits all out of proportion” of doing the necessary kinds of study to appropriately examine contraceptive effectiveness. We need companies that are committed to protecting public health and human rights, not preoccupied with their bottom line. There are many scientific and regulatory experts who would be thrilled to help companies ensure that they appropriately protect public health. I’m a huge fan and supporter of MDisrupt because they fill an essential gap by ensuring that such experts can be more easily linked with companies to help guide them towards ethically and scientifically robust studies and practices. However, for that formula to work, companies need to be committed to listening to and using the advice that relevant, trusted experts offer to them. When companies are not willing to do the right thing, they need to be called out – and for that, we need to be brave and make use of our right to free speech in the public domain and in evidence-based, well-justified ways.

What is your advice to companies that want to get the right data and research outcomes but are unsure how to do so?

As just one example, MDisrupt is an excellent resource for finding people who want to help you! You can also reach out to other established experts in the relevant field, or to organizations who have successfully put similar products on the market in evidence-based, ethical ways. We all need to work together to ensure that innovation is supported by rigorous science and robust regulatory oversight – in order to make people’s lives easier, better, and healthier. 

Thank you for joining us Chelsea and thank you for all your research into reproductive health and care. 

It should be emphasized that when it comes to health product marketing and claims, it’s crucial to establish a strong foundation of clinical evidence right from the beginning. The goal is not to simply react to concerns as they arise, but to proactively ensure that claims made about a product are backed up by robust clinical data. By doing so, companies can build trust with consumers, health professionals, and regulators, and avoid potential legal and reputational issues down the line. Additionally, investing in rigorous clinical studies can provide valuable insights into the safety and efficacy of a product, and ultimately improve patient outcomes. In short, establishing a strong foundation of clinical evidence is not only necessary for responsible marketing practices, but is also essential for the long-term success of a health product.