What Happens when Digital Health Fails to Deliver Evidence? A Story of Accountability with Dr. Chelsea Polis

What Happens when Digital Health Fails to Deliver Evidence? A Story of Accountability with Dr. Chelsea Polis

Clinical outcomes and evidence has been the topic du jour in digital health as evidenced by thinkpieces, surveys, and even venture capitalists (VCs) becoming more interested in the space. Overlooking clinical efficacy can have major financial and legal consequences, or at the very least, the “Fear of Looking Foolish” (FOLF) which has its own effects.

More importantly, we need robust research to determine how digital health implementations affect patient care and outcomes. As we are finding in the development of MDisrupt’s Digital Health Intelligence Database, designed to provide a deeper understanding of what the ecosystem’s research outcomes actually are, there are a number of noted gaps. There are companies that do not have research confirming their product’s effectiveness or impact, others do, but the research does not actually support their marketing claims, while others  have research that is not made publicly available for consumption and scrutiny. As a result, there is confusion as to what works and what doesn’t. Clinicians can’t easily figure out which products, if any, to recommend, patients don’t know what to believe, and payers have no basis on which to make any coverage decisions.

This call for proof in digital health with clinical evidence is no longer optional, or a ‘nice to have’. However, this is not new. In this month’s blog, we are joined by Dr. Chelsea Polis, a senior reproductive health epidemiologist with the Center for Biomedical Research at the Population Council. She has been an advocate for clinical evidence in digital applications for many years, well before most of us even started having this conversation. 

Valley Electronics, a company that manufactures a fertility thermometer called Daysy, made claims that Daysy could be used to prevent pregnancy with >99% effectiveness. Valley Electronics claimed that the thermometer was essentially as effective at preventing pregnancy as an intrauterine device (IUD) – which is one of the most highly effective contraceptive methods available with FDA-reviewed outcomes studies showing >99% effectiveness. In 2017, Dr. Polis reached out privately and directly to Valley Electronics to raise her concerns, but those efforts did not result in meaningful changes to the company’s practices. When the company then published a study of the device in 2018, Dr. Polis examined that study and found the methodology to be extremely flawed. She was ‘horrified’ by the study’s methodology and conclusions, and by the fact that flawed data were being used to make misleading claims to people seeking to prevent unintended pregnancy. So, she decided to do something about it. Dr. Polis was kind enough to answer questions about what happened, what is crucial for companies to know, and what she wants to see the industry deliver. 

Thanks for joining us, Dr. Polis! This story is especially important for all of us asking for clinical relevance to support effective clinical decisions and outcomes for their claims. After all, it is relevant and necessary to do so for patients. As a quick recap, I understand that you reached out to Valley Electronics, the makers of Daysy, to discuss your concerns about their evidence base prior to making any public statements or publications. When that failed to result in meaningful action, you also wrote a peer-reviewed commentary debunking their flawed study and calling for their study to be retracted, and published it in the journal their study had been originally published in. The journal considered your concerns, and after independent review, agreed that their methods were flawed and agreed to retract the paper from the scientific literature. What made you decide to comment and sound the alarm in the first place?

This is an important topic – I am so glad to be able to shed some light on it and grateful that you wanted to discuss it with me! Yes, I was incredibly concerned about the flawed publication and the misleading claims that the company had made to current and potential consumers about the contraceptive effectiveness of this device, which was then being sold for about $330. Fertility and contraception are a huge area of growth in digital health and women’s health. The market has been flooded with fertility-related apps and devices over the last few years, however, many do not appear to be based on robust scientific evidence. Such practices negatively affect patients – people could use these technologies, thinking they are using a highly effective form of contraception, but if the reality doesn’t match the marketing claims, they could experience an unintended pregnancy, which can have a wide variety of health, economic, social, and other consequences. 

What was the issue with the publications that Valley Electronics was using to support Daysy’s fertility tracking?

In 2017, after identifying all available studies on Daysy, I determined that (1) the existing evidence base for claims on Daysy’s contraceptive effectiveness was flawed, and (2) the Daysy device was not cleared or approved by the Food and Drug Administration (FDA) to be marketed for contraceptive purposes. So, in September 2017, I directly and privately emailed Valley Electronics to raise concern that the language in their marketing materials was not sufficiently evidence-based, and could place customers at risk of unintended pregnancy. I described several specific methodological concerns with the main study that they were citing at that time – for example, that it was based not on people using the actual Daysy device (but instead, on predecessor devices), that it lacked robust measurement of how many unintended pregnancies had occurred to people using these devices, and that there were major mathematical errors in the way in which they calculated the Pearl Index (a measure of contraceptive effectiveness). When I asked them if they would be willing to do the kind of study necessary to make robust claims about contraceptive effectiveness, they told me that “the costs and benefits are all out of proportion” to undertake such an effort. I found this to be a deeply offensive and frightening perspective, one that failed to consider the costs of a potential unintended pregnancy to their consumers.

Unfortunately, raising these concerns to the company did not stop them from continuing to claim >99% contraceptive effectiveness nor from publishing subsequent inaccurate analyses. In March 2018, they published a new study purporting to measure the contraceptive effectiveness of Daysy. It had many of the same flaws as the prior study, and several additional flaws – such as inappropriately excluding people who may have been at the greatest risk of unintended pregnancy from the analysis, and various problems with how the data were collected and analyzed. On social media, they inaccurately referred to it as a “clinical trial” (it was only a mailed retrospective survey) and inaccurately claimed that the study was “independent” (it was funded by the company and co-authored by company staff). Their flawed findings spread widely on social media, presenting a danger to informed contraceptive choice and public health – perhaps especially for younger people hoping to avoid unintended pregnancy and who are more avid consumers of social media.

So, I wrote and, in June 2018, published a peer-reviewed commentary outlining my concerns. I called for their flawed study to be retracted from the scientific literature and for greater regulatory oversight of marketing materials on contraceptive effectiveness. The journal retracted the company’s study in May 2019 – and the retraction and company’s response to it was covered a bit in the media.  

When I published my commentary, I also submitted an allegation of regulatory misconduct about the company to the FDA. Interestingly, Valley Electronics had received warning letters from FDA in years past for marketing their earlier devices (LadyComp, BabyComp, Pearly) as contraception without FDA approval to do so. Also, in 2016, the company Tweeted: “Did you know the FDA blocks us from calling Daysy birth control or contraception? But she’s 99.3% effective. #ThxBirthControl.” They deleted that Tweet after I pointed it out on Twitter, but I think these examples give a sense of how the company has conducted itself with respect to regulatory requirements. Anyway, I was gratified to eventually learn that my complaint prompted the FDA to make the company change their marketing language.

I looked at the website today. There are more warnings now, although it took some time to get there. What did you hope to achieve?

There has to be more oversight on these kinds of products – they need to be studied for effectiveness using robust scientific methodology, and there has to be regulatory oversight regarding how these products are marketed to people to avoid these situations. I don’t think that most consumers are aware that these devices are sometimes inappropriately marketed as contraceptive methods, and they may assume that these products have the same rigor of evidence as other contraceptives have. The marketing for any contraceptive product needs to be clear and in line with what solid research actually demonstrates. 

I agree completely, patients can be caught in the middle. Going back to Daysy, I understand the company actually sued you for defamation, albeit unsuccessfully. I cannot imagine that was easy even if you had the research to back you. Do you think this dismissal will help others who might want to call out companies without accurate information?

The company claimed that I had defamed them – for example, they felt it was defamatory for me to refer to their retracted study as “junk science”. So, rather than take my concerns seriously – as both a scientific journal and the US FDA did – they sued me. I believe that this was a classic “SLAPP” suit (“Strategic Lawsuit Against Public Participation”) – filed in hopes of silencing my voice as a scientist and contraceptive expert. 

The lawsuit was a long, stressful process – even though I was incredibly fortunate to have an incredible team of lawyers at Arnold & Porter, led by Ms. Dori Hanswirth, who took on my case pro bono. The case was filed against me in May 2020 and dismissed in August 2021. But then, in October 2021, Valley Electronics filed an appeal to the Second Circuit Court of Appeals, and in March 2022, the Second Circuit affirmed the District Court’s decision to dismiss the case. It was of course a huge relief when I found out about each dismissal. I learned that many SLAPP lawsuits are filed without the hope of actually winning, but instead, are filed with a goal of intimidating, silencing, distracting, and potentially bankrupting the defendant. It often works. I believe that they wanted to silence a scientist with expertise on fertility awareness based methods and expertise on calculation of contraceptive effectiveness for speaking out truthfully and in evidence based ways about their scientific, marketing, and regulatory behaviors. Sadly, I am now well aware that I am not the only scientist to whom this kind of thing has happened – and I have learned that we may not be aware of the full magnitude of the issue, since many people who are sued are not able to speak out about it, as I decided to do.

I certainly hope this case helps to put companies on the alert that scientists who speak out about whether scientific evidence supports a given product’s claims have a right to raise well-justified concerns in the public domain. Attempts to silence people who stick up for scientific data and regulatory oversight, on behalf of protecting consumers and public health, are absolutely chilling. The case against me was ultimately dismissed, as it certainly should have been, and let’s hope that precedent has now been strengthened and that future scientists will be less likely to have to face this kind of traumatic experience – which also detracts from the time and energy we have available to focus on pushing forward scientific knowledge. 

I thank you for doing that and standing up for this – patients will not know these kinds of details, they don’t read research publications and unknowingly end up using these products. MDisrupt is committed to getting researcher and clinician voices into digital health. From your perspective, what is the most likely reason companies do not undertake  the right research? Does adding a clinician or researcher earlier to their process help?

As somebody familiar with the process of contraceptive research and development, I know how challenging, time-consuming, and expensive it can be to take a contraceptive method from idea to market. Numerous studies must be conducted to fully demonstrate a product’s safety and effectiveness before it can be offered to the public. Organizations like the Center for Biomedical Research at the Population Council, a non-profit organization that I joined in September 2022, have worked in contraceptive development for decades, and have brought multiple safe and effective contraceptive options to the global public, including IUDs, implants, contraceptive vaginal rings, and more. We take quite seriously the need to abide by appropriate scientific and regulatory procedures – because we care deeply about people’s right to have clear and accurate information about their contraceptive options. Our scientific experts know how to design appropriate scientific studies, and our regulatory experts work with bodies like the FDA to collaboratively ensure appropriate oversight. 

Valley Electronics stated to me that they found the “costs and benefits all out of proportion” of doing the necessary kinds of study to appropriately examine contraceptive effectiveness. We need companies that are committed to protecting public health and human rights, not preoccupied with their bottom line. There are many scientific and regulatory experts who would be thrilled to help companies ensure that they appropriately protect public health. I’m a huge fan and supporter of MDisrupt because they fill an essential gap by ensuring that such experts can be more easily linked with companies to help guide them towards ethically and scientifically robust studies and practices. However, for that formula to work, companies need to be committed to listening to and using the advice that relevant, trusted experts offer to them. When companies are not willing to do the right thing, they need to be called out – and for that, we need to be brave and make use of our right to free speech in the public domain and in evidence-based, well-justified ways.

What is your advice to companies that want to get the right data and research outcomes but are unsure how to do so?

As just one example, MDisrupt is an excellent resource for finding people who want to help you! You can also reach out to other established experts in the relevant field, or to organizations who have successfully put similar products on the market in evidence-based, ethical ways. We all need to work together to ensure that innovation is supported by rigorous science and robust regulatory oversight – in order to make people’s lives easier, better, and healthier. 

Thank you for joining us Chelsea and thank you for all your research into reproductive health and care. 

It should be emphasized that when it comes to health product marketing and claims, it’s crucial to establish a strong foundation of clinical evidence right from the beginning. The goal is not to simply react to concerns as they arise, but to proactively ensure that claims made about a product are backed up by robust clinical data. By doing so, companies can build trust with consumers, health professionals, and regulators, and avoid potential legal and reputational issues down the line. Additionally, investing in rigorous clinical studies can provide valuable insights into the safety and efficacy of a product, and ultimately improve patient outcomes. In short, establishing a strong foundation of clinical evidence is not only necessary for responsible marketing practices, but is also essential for the long-term success of a health product.

Aditi U Joshi

Aditi U Joshi MD, MSc, FACEP

Dr. Aditi U. Joshi is an experienced Emergency Medicine Physician, Telehealth expert, and consultant. She is working on behalf of health systems, companies, and individual practices to determine which solutions best meet health systems buyers’ needs. Nationally, she serves as the Chair for the Telehealth committee of the American College of Emergency Physicians, chairs the Society of Academic Emergency Medicine’s Telehealth Fellowship Committee, part of the American Telemedicine Association’s research review committee and an Advisor for the AMA’s Digital Payment Advisory Group. Her book on Telehealth Success – how to build and understand the ecosystem will be published by Forbes in Spring of 2023.

At MDisrupt we believe that the most impactful health products should make it market quickly. We do this by uniting digital health companies with experts from the healthcare industry to help them accelerate their time to market responsibly.

Our expert consultants span the healthcare continuum and can assist with all stages of health product development: This includes regulatory, clinical studies and evidence generation, payor strategies, commercialization, and channel strategies. If you are building a health product, talk to us.

Raising the Bar on Digital Health: Clinical Outcomes are the New Standard

Raising the Bar on Digital Health: Clinical Outcomes are the New Standard

Another year beginning is a great time to take stock. For those of us who work in digital health, there has been an increasing trend to talk about evidence and outcomes. Despite recent contraction, the digital health market remains a huge ecosystem. For those wanting to know what product to use, it seems impossible to understand, categorize and differentiate between them, especially in their clinical relevance. This disconnect is leading to perhaps ineffective products being used and some good products ignored. Without clear evidence of what something will accomplish, and whether it will improve or maintain quality of care, the engagement into the healthcare system will remain siloed and separated, physicians won’t want to prescribe them, and payors can’t justify paying for them. 

Over the last year there have been more accounts and calls for improving clinical outcomes. The Rock Health/JMIR paper outlined the clinical robustness vs funding of companies finding that there was little correlation – suggesting asymmetry in what works clinically and what is financially valued. While they found that clinical robustness was low in general, companies with higher scores were more likely to be bought by employers. Which means that this robustness and testing is possible and has a real effect on market success. A Finn Partners global analysis of 12,000+ digital health ventures reveals that just 17% have significant clinical strength and predict that investors are going to go from FOMO to FOLF (Fear of Looking Foolish) by increasing due diligence on clinical outcomes. The WHO has guidelines recognizing that clinicians want to see these outcomes to better adopt and use these products. Sachin Jain calls it the epidemic of magical thinking and that we need ‘less virtue signaling and more virtue’. 

2023 gives us a fresh start and opportunity to be better at building, valuing and funding companies and products. Digital Health will not survive or be taken seriously without it. 

The other challenge is actually finding a centralized, organized way to search and compare clinical outcomes. To this end, we at MDisrupt built the MDisrupt Digital Health Intelligence database (MDHI) that categorizes companies by components such as product type, funding round, clinical staffing, and clinical research. The clinical research is the crux of this project. We looked at the type of research design, clinical claims, and the backing of those claims within the research. 

Doing that for 650 companies has been no joke as evaluating research requires actually reading the papers to understand the methods, design and whether the outcomes are ‘real’. Comparing the research in peer reviewed journals vs company’s websites and claims, there were broad categories of companies noted:

    1. a company understands their product and their research aligns.
    2. a company has a potentially good product but:
      • their research studies are not designed to test what their website claims
      • the study does not align with their vision
      • their product can likely have broader use cases than their research suggests
      • products and research studies do not seem to align at all and will lead to confusion in their marketing and claims.
    3. companies and products that only measure patient/consumer engagement and not clinical outcomes
    4. companies and products that have only a sales pitch and no research.

It is frustrating to see the last 3 scenarios, especially as much of that can be avoided with careful planning and bringing on the right expertise early in the company and product’s inception.

I’ve seen interesting findings and companies doing good research. Koa Health, a Series A funded digital CBT app, for example, has some well thought out research papers, good early clinical findings, and also mentions when things are not exactly working. Compare that to Spring Health, a huge unicorn company, with one listed cohort study. Both had good studies although a difference in number and design. But the difference in funding between them is stark. What this means seems to be in line with others’ findings.

Improving Clinical Research & Sharing Outcomes

Over the next few months we will be sharing a number of our findings from the database, what the market looks like, what gaps exist and what is the alignment with clinical outcomes and funding. Until then, we want to share some ways digital health companies can improve their clinical research and share their outcomes. Here’s how:

Adopt Early Clinician Input

This seems obvious but the number of companies that do not have any staff or advisors that are physicians/clinicians is staggering.

  • It is impossible for those who have never practiced, been in a clinical environment or done patient care to truly understand what it is like to do so. Getting the right physician input for this is crucial.
  • Clinicians can help plan out your research strategy – this can include PhDs, statisticians, and researchers as well. Not all research design is the same. For example, randomized controlled trials are the gold standard. However, it is not always feasible to do these, especially early on. If looking for research planning, make sure it is someone who understands that and is up to date on current literature and practice.
  • Ensure you have the right physician voice. It should align with your goals, the community you are trying to affect and the specialty. For example, if your goal is to use a device to decrease readmissions for Congestive Heart Failure (CHF) – ensure you have a doctor who treats that population in the community you are targeting. It is more likely they will be up to date on the research.

I recognize that it might be expensive to hire physicians, especially for early-stage startups. However, having a contractual, advisory, or fractional relationship will allow for it to be flexible and less expensive. Having a couple of hours and having targeted questions with an expert can save lots of time and headache later.

Understand the clinical environment/the market

What has already been accomplished in this space? I do not mean simply in the digital health space. I mean for the disease process that you are targeting – what is the current practice, what has been the research, what are the current interventions etc? Some diseases have been much more studied in the digital health space, such as diabetes. The process some of those successful products have taken can be replicated.

This research is necessary to create clear goals. Not all products need to stand alone – some can be used for multiple purposes or diseases. A pulse oximeter, for example, is not only used for monitoring COVID or COPD. Having knowledge of that can clear the targets or allow for multiple simultaneous trials.

Ensure your strategy and product align

Once you have an idea of what is currently done from a clinician, then it makes sense to try to see it from an outside perspective – as someone who won’t be as committed to doing things as ‘they were’.

What I appreciate about the digital health founders I’ve met is their commitment to changing some of the roadblocks of medicine. Physicians are very good at working around inefficient workflows just to get through the day – having outside perspectives are what can dramatically change our processes for the better.

Create clear research goals

Do you have PICO formated questions?
The first step is to clearly define the question you are looking to answer. The PICO format is a good starting place. Who is the patient or population specifically? The population is never everyone. What is the intervention specifically? Most products have several use cases, but which of those are you testing specifically? What are you comparing it to? What are the other solutions (or lack of solutions) the product will transform? And finally, what is the outcome you will demonstrate, clinical, financial, engagement, cost?

Do you know what your product is truly capable of?
It is okay not to know all the answers and why it is important to get other input. That will help figure out what you can study.

What do you need to know about how it works?
What clinical outcomes and quality measures do you need to learn to prove the solution works (or doesn’t). Much of this does not need to be redone. There are clinical quality measures that already exist that can be applied. A recent study looking at the HEDIS measures and how well telehealth visits adhered to them is a good example.

What will you do with that information?
Is this to ensure it is safe, gain a pilot, improve marketing, expand the use cases, sell to payors, further the research for a disease process etc. Likely it will be multiple reasons.

Which journals and publications will you publish in?
Not all are the same, however, it should be peer reviewed regardless of the impact factor. You can create a list of journals to submit to as rejections are common (and not to be taken personally!).

Are you ready to pivot depending on the outcomes?
Negative studies exist or your product may not work for what you had originally thought. It doesn’t mean it is useless – are you ready to change its use case or tweak it to try again? By the way, the answer can be No. Not everything will work but knowing that is valuable information.

Some of these might be hard to fully understand as we can all be siloed within our own worlds. For example, if you have a solution for treating diabetes, you have to understand how diabetes is currently treated, what the patient and clinician process is, what current digital solutions exist, what their research shows and then plan out your strategy and clinical outcomes. Designing a research study seems easy but requires a thoughtful process and an understanding of the different types, statistics and what can be technically proven.

Engage other stakeholders

This is easier said than done. You can have a well thought out plan and nowhere to run a research study as it requires engaging a place for a pilot with someone who can be a PI and get IRB approval. This is why it is worthwhile to have clinicians and researchers on your staff or advisory board that can help set that up with health systems, clinics, and patients.

There are always innovation hubs, pitch contests and incubators that also can aid with getting in front of the right places.

Share your findings

Bizarrely, the Rock Health group found that not all those with clinical outcomes shared what they found, and claims made by digital startups are low. In an increasingly competitive environment, selling what the clinical outcomes are is going to only help you. This is mainly because

  • Clinicians want to know the product is safe and effective
  • Payors want to be able to invest in a solution that works
  • Patients are increasingly the consumers of their health, and they want to know this is going to help them.

In the example of diabetes, there are many solutions so having evidence-demonstrated ideas of what works is going to help in all 3 categories. Many Type 1 diabetics are especially well educated about their solutions, and so they can give valuable feedback if the clinical outcomes are there and sold to them.

Also, don’t be afraid of negative studies or those that don’t prove that the product worked in the way it was used. That can be used to improve, iterate, or pivot your direction. It will also aid others who want to improve care for a targeted population. It is not often that we get this data as it is far more common to see studies that are positive. A company that published a negative study decided to continue on, pivoting their goals and then demonstrating where their product could work. This instills far more trust in their vision and improves their chance of longstanding success.

Don’t embellish

On the flip side, do not market something you cannot backup. It will come out and when selling to employers or payors, it will become clear. I will give the benefit of the doubt that it is not nefarious – not all research is equal and there are badly designed studies or the results are not actually statistically or practically significant. Make sure you do your due diligence and ask if you do not know.

Share your data

Companies selling to employers had higher clinical evidence and demonstrable claims compared to other customer groups according to Rock Health. So it’s clear that employers and payors want this information. It also has to be targeted to your vision and goals. If you know you want to target patients with CHF who end up in the hospital once a month, that is what you sell to payors. Having a dietary monitoring program may be broader but can be applied to a whole host of different prevention and treatment programs. Just be clear what you are fixing whether narrow or large.

In conclusion, this is healthcare. We are aiming to improve care, quality of life, and do no harm. To aid in the collection of understanding the clinical utility of the digital health products out there, we all have to be part of this change.

  • Good research design is an extension of understanding your product and strategy. Make sure they are in alignment.
  • Product clinical outcomes are necessary for a forward-thinking strategy. No end user will want to use something that is not demonstrated to improve outcomes.
  • A database of companies and products with relevant clinical outcomes is needed. It is impossible to understand what works and what doesn’t without it. This can inform the future of digital health.

Here’s to more and better clinical evidence in 2023!

Aditi U Joshi

Aditi U Joshi MD, MSc, FACEP

Dr. Aditi U. Joshi is an experienced Emergency Medicine Physician, Telehealth expert, and consultant. She is working on behalf of health systems, companies, and individual practices to determine which solutions best meet health systems buyers’ needs. Nationally, she serves as the Chair for the Telehealth committee of the American College of Emergency Physicians, chairs the Society of Academic Emergency Medicine’s Telehealth Fellowship Committee, part of the American Telemedicine Association’s research review committee and an Advisor for the AMA’s Digital Payment Advisory Group. Her book on Telehealth Success – how to build and understand the ecosystem will be published by Forbes in Spring of 2023.

At MDisrupt we believe that the most impactful health products should make it market quickly. We do this by uniting digital health companies with experts from the healthcare industry to help them accelerate their time to market responsibly.

Our expert consultants span the healthcare continuum and can assist with all stages of health product development: This includes regulatory, clinical studies and evidence generation, payor strategies, commercialization, and channel strategies. If you are building a health product, talk to us.

Optimizing product/market fit through user research

Optimizing product/market fit through user research

Most digital health companies understand the importance of achieving product/market fit (i.e., the degree to which their products meet an actual demand from customers).

At least in theory.

In practice, a lot of digital health companies still struggle to develop visions for products that reflect actual user needs. Similarly, many digital health companies are struggling to translate technological innovations into products that people can and want to use.

One way to resolve these challenges is through user research – an effective, but underutilized practice that helps product developers understand their prospective users’ needs and gather actionable insights that optimize product designs. Using various methods, such as ethnographic observations, individual and group interviews, surveys, and usability tests, user researchers help ensure that products are designed in a user-centered manner that is likely to increase a product’s success.

How is user research different from market research?

At a glance, user research might seem similar or even equivalent to market research. Both utilize similar methods, such as surveys and interviews, to gather insights about prospective customers. But, the two disciplines apply these methods differently and have different goals for the types of insights they seek to collect.

At a high level, market research involves the investigation of consumer behaviors, opinions, and trends with the goal of identifying insights that will help direct business strategy. By contrast, user research involves the study of users’ hands-on interactions with products, as well as their needs and expectations for future products, with the goal of identifying insights that will help direct product design and development activities.

Market and user research and not substitutes for one another. Rather, they are complementary activities that can build on each other’s strengths.

The right method at the right time

User research can reveal valuable insights at all stages of product development, from early discovery and conceptualization phases through final development and validation. In each phase, different research methods might be more or less suitable.

In the early discovery phases, observation-based research can be especially valuable. I’m still surprised to discover companies that are developing digital health or medtech without ever having visited the clinical environment in which they expect their product to be used. Observation-based research in prospective use-environments can reveal the current challenges that your prospective users are facing and can often be a great source of inspiration for thinking of innovative solutions that actually help users.

Interviewing users, either one-on-one or in a focus group, is another activity that can be particularly helpful during early product development. In particular, interviews are an effective means for developing a deep understanding of users’ opinions about their experiences with relevant products and their expectations for future products.

In comparison to observations and interviews, surveys can provide an efficient means to collect answers from a large number of respondents. Although surveys are less effective at providing rich qualitative information, they can provide reliable answers to specific questions that are likely to arise in later stages of product development.

Other user research methods are especially effective at evaluating evolving product designs. Activities such as expert reviews and usability tests, in which representative users perform actual tasks with prototype or final versions of a product, reveal specific opportunities for improving a product’s design so as to optimize its usability, usefulness, and appeal.

Getting a helping hand

It’s always better to conduct some user research than none at all. That said, there can be downsides to conducting user research without assistance or guidance from experienced user research professionals such as human factors engineers, usability specialists, or user experience (UX) researchers. Without support from trained researchers, product developers might utilize research methods that are not well suited to their research goals or utilize flawed research protocols that result in the collection of misleading information.

Getting started

The good news for digital health companies eager to dive into user research is that it’s never been easier. Industry enthusiasm for the potential of digital health means that many clinicians are eager to contribute to user research, while the rise of remote communication and research tools enables efficient and simple execution of many research activities.

With its large pool of clinical, product, and user research experts, as well as with its new research tools, MDisrupt has seen first hand how digital health companies improve their product/market fit through user research.

Jonathan Kendler

Jonathan Kendler

For over 20 years, Jonathan has been helping medical device and healthcare technology companies make their products safe and easy to use by applying human factors engineering and user-centered design to their product development efforts. He has consulted for a wide range of companies, including both startups and large companies, helping to ensure that their products meet users’ needs, comply with best design practices, and conform to international regulations and standards for usability engineering.

At MDisrupt we believe that the most impactful health products should make it market quickly. We do this by uniting digital health companies with experts from the healthcare industry to help them accelerate their time to market responsibly.

Our expert consultants span the healthcare continuum and can assist with all stages of health product development: This includes regulatory, clinical studies and evidence generation, payor strategies, commercialization, and channel strategies. If you are building a health product, talk to us.

Women’s Health Investment: Top Trends, Issues, and Opportunities (2022)

Women’s Health Investment: Top Trends, Issues, and Opportunities (2022)

Women’s health is finally starting to get more attention from innovators and investors—but there’s still a long way to go. As a female-founded company, we’ve long been focused on the urgent needs in this area. After speaking with numerous women’s health investors, innovators, and advocates at the Current State of Women’s Health Roundtable (sponsored by Goodwin’s Women’s Health+Wellness practice) and within our own network, we’re encouraged by the small wins, but know there are large opportunities and challenges that still need to be addressed.  Here are some key takeaways. 

Women’s health is for life—not just for reproduction.

An article by our webinar moderator Chrissy Farr, principal at OMERS Ventures, points out that a woman’s reproductive years make up only about 30 of a typical 80-year lifespan—yet that’s where most investment tends to focus. 

Farr and other webinar panelists emphasized that there are big unmet health needs all through a woman’s life, and that innovators and investors who take a broader, more holistic view of women’s health will find many more opportunities to make a difference. Yet in the first nine months of 2021, only about 7 percent of venture funding in digital health was directed to women’s health.

Investors and innovators should look for the “white space.”

This phrase came up often during the Goodwin Roundtable; it refers to areas that have traditionally been underfunded but which offer major possibilities for improving health and meeting market needs. 

Menopause care is one “white space” that is ripe for disruption. There are 50-some different symptoms that women may experience, which means that there are many entry points for improving care. Other major areas of unmet need include behavioral health, osteoporosis, endometriosis, and cardiac health. 

Early detection of ovarian and uterine cancers is another white space; many of the companies focused on early cancer detection (e.g. Grail, Exact Sciences, Guardant) focus primarily on colon cancer. Women’s health needs its own early detection solutions. 

To pitch women’s health to investors, focus on normalization and storytelling. 

On normalization: It’s natural for investors to relate most easily to innovations that address issues that they’ve experienced personally—which can be problematic when most investors are men. (And unfortunately, that remains the case: Last year, only 2 percent of venture funding went to female-founded companies, the lowest amount since 2016.)

Successful founders find ways to make women’s health issues relevant to men. “Even if men can’t relate to a particular health issue personally, surely they know that it affects them by way of the women in their lives,” said Kaitlin Christine, founder and CEO of Gabbi. 

On storytelling: Showing how a solution can affect individual lives is important, but so are numbers:  Emphasizing the market potential in areas that an investor may not be familiar with can break through a sense of indifference or irrelevance, said Marina Pavlovic Rivas, founder and CEO of Eli Health. 

Involve clinicians early and often 

As in other areas of digital health, it can be challenging to introduce a new women’s health technology into a clinical environment. “It’s step by step,” said Pavlovic Rivas, of Eli Health. “If it’s not on the market, you need to validate that there is an unmet need. Then involve clinicians in evidence generation for the technology, and for application of the technology into the environment where it will be used. And you need to do that work pre-market.”

Remember that women’s health affects everyone.

“I think we are on the cusp of a transition to a broader recognition that women’s health issues are human health issues and affect all of us.” – Paxton Maeder-York, founder and CEO of Alife Health. 

Mona Schreiber

Mona Schreiber, VP of Marketing, MDisrupt

Mona is a marketing leader with over 15 years b2b experience developing new healthcare markets. Prior to joining MDisrupt, she was an early marketing lead at Invitae Inc, where she helped scale the company from small startup to market leader in the nascent genetic testing industry. She brings with her a deep understanding of global markets from her international marketing work at Invitae and Affymetrix.

At MDisrupt we believe that the most impactful health products should make it market quickly. We do this by uniting digital health companies with experts from the healthcare industry to help them accelerate their time to market responsibly.

Our expert consultants span the healthcare continuum and can assist with all stages of health product development: This includes regulatory, clinical studies and evidence generation, payor strategies, commercialization, and channel strategies. If you are building a health product, talk to us.

What makes a great CMO? Emergency medicine physicians are the answer

What makes a great CMO? Emergency medicine physicians are the answer

Douglas Bernstein, MD

Dr. Bernstein is the Director of Clinical Operations at the Bon Secours Memorial Regional Medical Center Emergency Department, in Richmond, VA. He attended Johns Hopkins University School of Medicine, followed by internship in General Surgery at Duke University Medical Center, and residency in Emergency Medicine at Yale New-Haven Hospital. His predominant areas of administrative interest include process optimization, clinical operations, billing and documentation, and regulatory compliance. He enjoys recreation with his family on the James River, and is making a quixotic attempt to learn guitar.

 

Chief Medical Officers (CMOs), whether in nimble health startups or mature companies and healthcare systems, are most effective when they command a broad set of organizational and communication skills. Of course, a qualified CMO must possess clinical experience and direct knowledge of the practice of medicine. What distinguishes an exceptional CMO is her array of executive and soft skills.

Does she communicate clearly and efficiently with different stakeholders? Does she inspire confidence, trust, and loyalty from her team members? Can she be relied upon to take responsibility for critical decisions, even with imperfect information? Will she demonstrate initiative, take ownership of multiple simultaneous projects, and meet deadlines?

Companies and organizations seeking a CMO should strongly consider physicians with specialized training in Emergency Medicine (EM). The clinical expertise, team leadership, and organizational proficiency necessary for both roles map well onto one another, making experienced EM physicians prime candidates for executive leadership positions. Here’s why.

Decisiveness with Imperfect Information

EM physicians operate in a unique environment within medicine. Critical information is often unreliable or scarce—patients are often unconscious or delirious and their medical records may be unavailable. Idiosyncratic social, legal, and ethical concerns frequently complicate the clinical scenario and require individualized problem solving. Specialist physicians, hospital beds, imaging, and various resources may be difficult to come by. Significant time pressure and constant interruptions typify the Emergency Department; extreme multitasking is the norm, not the exception. With patient stretchers lining packed hallways and an unrelenting stream of ambulance arrivals, EM physicians in a busy department don’t do two things at once, they do twenty. (Sound like a startup?) Within this dynamic and complex environment, EM physicians rely on heuristic reasoning, clinical expertise, and creative problem-solving to make diagnoses and treat every conceivable medical situation, with zero tolerance for failure.

Soft Skills Matter Most

Beyond being a well-trained and smart doctor, an effective EM physician is an expert in communication and leadership. Patients and family members are often experiencing one of the worst times of their lives and may be skeptical of a strange doctor, or have limited medical literacy. Yet, to forge a therapeutic relationship, the physician needs to be able to immediately assess their emotional and cognitive needs to put them at ease and earn their trust. Most of the time, this simply starts with open-ended listening (studies show that most doctors interrupt the patient’s history within about 20 seconds; with that low a bar, any physician can be a stand out listener).

Meanwhile, to lead a diagnostic and treatment process efficiently and safely, the EM physician must gain the loyalty of diverse team members: nurses, technicians, and other physicians, who may need to be cajoled or convinced into getting out of bed in the middle of the night to attend to a specific issue. And never forget the janitor (who may be the linchpin to getting a soiled room turned over), the secretary (without whom timely calls to consultants won’t be made, delaying patient care) and the late-night cafeteria worker (mine is named Theresa; I call her “Mother Theresa”).

An EM physician who is a courteous role model and instructor for the nurses will find himself valorized and exceptionally supported. When he needs the nurses to prioritize a task, they will be there for him (“Sure, doc, I’ve got her next in line for the CT scan.”) A physician who is haughty, aloof, denigrating, or unfriendly will find himself stymied (“I’m doing five different things right now—I’ll draw those labs whenever I can get to it.”)

Emergency Physicians Understand the Healthcare Landscape

Imagine a blind man in a room who walks endlessly, probing and exploring with his fingertips. Eventually, he will know every nook and cranny like the back of his hand, allowing him to “see” the room.

An EM physician starts as the blind man. As she treats tens of thousands of patients of every age, for every medical and traumatic condition, she will be forced out of necessity to explore the boundaries and contours of the medical system. She will figure out how insurance status affects outpatient follow-up and admissions. She will experience how regulatory metrics and billing practices drive documentation and medical decision-making. She will see how reimbursement rates and private practice contracts lead to access—or lack of it—from specialist consultants. In Emergency Medicine, figuring out who is going to manage a 10-year old boy’s testicular torsion, or the large abscess on an intoxicated man’s jawbone, or a wound dehiscence from a spine surgery that was performed at a different hospital, is part of the normal routine. No other medical specialty is like this. A cardiologist or gastroenterologist, for example, while possessing a depth of knowledge in a particular corner of medicine, does not have to contend with the broader landscape in this way.

The Emergency Department is a type of crucible. The infinite social and medical complexities, combined with the unyielding need to figure out solutions to complex problems, as more patients stream through the door 24/7, forces EM physicians to probe the nooks and crannies of our healthcare system. Out of that crucible are forged physicians with a unique set of problem-solving skills and a knowledge of how the American medical system actually functions (and sometimes, how it doesn’t).

The Right Tool for the Job

The Swiss Army Knife is famous the world over for its remarkable array of useful tools. There is something addictively charming about these tools, which stems from the aspiration built into them—to be prepared for every situation—which is probably why they have become a classic coming-of-age gift. I still have the one my father gave me on my Bar Mitzvah, which we clumsily engraved with “Doug” on the small blade, and “Dad” on the large blade.

It is also why EM physicians have come to be known, among all medical specialties, as the “Swiss Army knife of doctors.” EM specialists possess an exceptionally broad range of clinical, communication, and management skills, developed by necessity in a work environment where they must rapidly confront and overcome the infinite complexities of patients who may be 3 minutes to 103 years old.

Similarly, chief medical officers need command of a variety of skills to be effective and strategic executives. The CMO will be communicating with internal team members and external partners. He may be navigating complex regulatory and investment environments. He will review data and assign out tasks, and follow up on them. He will juggle multiple—and sometimes competing—stakeholders, while delivering on short term projects and long term strategic goals.

Given the need for parallel skill sets that combine hard knowledge of clinical practice, the landscape of the American medical system, and the soft skills of persuasion, multi-tasking, and team leadership, the Swiss Army knife of physicians is often the right tool for the role of CMO.

At MDisrupt, we believe that the most impactful health products should make it to market quickly. We connect digital health innovators to the healthcare industry experts and scientists they need to responsibly accelerate product development, commercialization, adoption, and scale.

Our experts span the healthcare continuum and can assist with all stages of health product development: This includes regulatory, clinical studies and evidence generation, payor strategies, commercialization, and channel strategies. If you are building a health product, talk to us.