Another year beginning is a great time to take stock. For those of us who work in digital health, there has been an increasing trend to talk about evidence and outcomes. Despite recent contraction, the digital health market remains a huge ecosystem. For those wanting to know what product to use, it seems impossible to understand, categorize and differentiate between them, especially in their clinical relevance. This disconnect is leading to perhaps ineffective products being used and some good products ignored. Without clear evidence of what something will accomplish, and whether it will improve or maintain quality of care, the engagement into the healthcare system will remain siloed and separated, physicians won’t want to prescribe them, and payors can’t justify paying for them. 

Over the last year there have been more accounts and calls for improving clinical outcomes. The Rock Health/JMIR paper outlined the clinical robustness vs funding of companies finding that there was little correlation – suggesting asymmetry in what works clinically and what is financially valued. While they found that clinical robustness was low in general, companies with higher scores were more likely to be bought by employers. Which means that this robustness and testing is possible and has a real effect on market success. A Finn Partners global analysis of 12,000+ digital health ventures reveals that just 17% have significant clinical strength and predict that investors are going to go from FOMO to FOLF (Fear of Looking Foolish) by increasing due diligence on clinical outcomes. The WHO has guidelines recognizing that clinicians want to see these outcomes to better adopt and use these products. Sachin Jain calls it the epidemic of magical thinking and that we need ‘less virtue signaling and more virtue’. 

2023 gives us a fresh start and opportunity to be better at building, valuing and funding companies and products. Digital Health will not survive or be taken seriously without it. 

The other challenge is actually finding a centralized, organized way to search and compare clinical outcomes. To this end, we at MDisrupt built the MDisrupt Digital Health Intelligence database (MDHI) that categorizes companies by components such as product type, funding round, clinical staffing, and clinical research. The clinical research is the crux of this project. We looked at the type of research design, clinical claims, and the backing of those claims within the research. 

Doing that for 650 companies has been no joke as evaluating research requires actually reading the papers to understand the methods, design and whether the outcomes are ‘real’. Comparing the research in peer reviewed journals vs company’s websites and claims, there were broad categories of companies noted:

1. 1. a company understands their product and their research aligns.
   2. a company has a potentially good product but:
      • their research studies are not designed to test what their website claims
      • the study does not align with their vision
      • their product can likely have broader use cases than their research suggests
      • products and research studies do not seem to align at all and will lead to confusion in their marketing and claims.
   3. companies and products that only measure patient/consumer engagement and not clinical outcomes
   4. companies and products that have only a sales pitch and no research.

It is frustrating to see the last 3 scenarios, especially as much of that can be avoided with careful planning and bringing on the right expertise early in the company and product’s inception.

I’ve seen interesting findings and companies doing good research. Koa Health, a Series A funded digital CBT app, for example, has some well thought out research papers, good early clinical findings, and also mentions when things are not exactly working. Compare that to Spring Health, a huge unicorn company, with one listed cohort study. Both had good studies although a difference in number and design. But the difference in funding between them is stark. What this means seems to be in line with others’ findings.

Improving Clinical Research & Sharing Outcomes

Over the next few months we will be sharing a number of our findings from the database, what the market looks like, what gaps exist and what is the alignment with clinical outcomes and funding. Until then, we want to share some ways digital health companies can improve their clinical research and share their outcomes. Here’s how:

Adopt Early Clinician Input

This seems obvious but the number of companies that do not have any staff or advisors that are physicians/clinicians is staggering.

• It is impossible for those who have never practiced, been in a clinical environment or done patient care to truly understand what it is like to do so. Getting the right physician input for this is crucial.
• Clinicians can help plan out your research strategy – this can include PhDs, statisticians, and researchers as well. Not all research design is the same. For example, randomized controlled trials are the gold standard. However, it is not always feasible to do these, especially early on. If looking for research planning, make sure it is someone who understands that and is up to date on current literature and practice.
• Ensure you have the right physician voice. It should align with your goals, the community you are trying to affect and the specialty. For example, if your goal is to use a device to decrease readmissions for Congestive Heart Failure (CHF) – ensure you have a doctor who treats that population in the community you are targeting. It is more likely they will be up to date on the research.

I recognize that it might be expensive to hire physicians, especially for early-stage startups. However, having a contractual, advisory, or fractional relationship will allow for it to be flexible and less expensive. Having a couple of hours and having targeted questions with an expert can save lots of time and headache later.

Understand the clinical environment/the market

What has already been accomplished in this space? I do not mean simply in the digital health space. I mean for the disease process that you are targeting – what is the current practice, what has been the research, what are the current interventions etc? Some diseases have been much more studied in the digital health space, such as diabetes. The process some of those successful products have taken can be replicated.

This research is necessary to create clear goals. Not all products need to stand alone – some can be used for multiple purposes or diseases. A pulse oximeter, for example, is not only used for monitoring COVID or COPD. Having knowledge of that can clear the targets or allow for multiple simultaneous trials.

Ensure your strategy and product align

Once you have an idea of what is currently done from a clinician, then it makes sense to try to see it from an outside perspective – as someone who won’t be as committed to doing things as ‘they were’.

What I appreciate about the digital health founders I’ve met is their commitment to changing some of the roadblocks of medicine. Physicians are very good at working around inefficient workflows just to get through the day – having outside perspectives are what can dramatically change our processes for the better.

Create clear research goals

Do you have PICO formated questions?
The first step is to clearly define the question you are looking to answer. The PICO format is a good starting place. Who is the patient or population specifically? The population is never everyone. What is the intervention specifically? Most products have several use cases, but which of those are you testing specifically? What are you comparing it to? What are the other solutions (or lack of solutions) the product will transform? And finally, what is the outcome you will demonstrate, clinical, financial, engagement, cost?

Do you know what your product is truly capable of?
It is okay not to know all the answers and why it is important to get other input. That will help figure out what you can study.

What do you need to know about how it works?
What clinical outcomes and quality measures do you need to learn to prove the solution works (or doesn’t). Much of this does not need to be redone. There are clinical quality measures that already exist that can be applied. A recent study looking at the HEDIS measures and how well telehealth visits adhered to them is a good example.

What will you do with that information?
Is this to ensure it is safe, gain a pilot, improve marketing, expand the use cases, sell to payors, further the research for a disease process etc. Likely it will be multiple reasons.

Which journals and publications will you publish in?
Not all are the same, however, it should be peer reviewed regardless of the impact factor. You can create a list of journals to submit to as rejections are common (and not to be taken personally!).

Are you ready to pivot depending on the outcomes?
Negative studies exist or your product may not work for what you had originally thought. It doesn’t mean it is useless – are you ready to change its use case or tweak it to try again? By the way, the answer can be No. Not everything will work but knowing that is valuable information.

Some of these might be hard to fully understand as we can all be siloed within our own worlds. For example, if you have a solution for treating diabetes, you have to understand how diabetes is currently treated, what the patient and clinician process is, what current digital solutions exist, what their research shows and then plan out your strategy and clinical outcomes. Designing a research study seems easy but requires a thoughtful process and an understanding of the different types, statistics and what can be technically proven.

Engage other stakeholders

This is easier said than done. You can have a well thought out plan and nowhere to run a research study as it requires engaging a place for a pilot with someone who can be a PI and get IRB approval. This is why it is worthwhile to have clinicians and researchers on your staff or advisory board that can help set that up with health systems, clinics, and patients.

There are always innovation hubs, pitch contests and incubators that also can aid with getting in front of the right places.

Share your findings

Bizarrely, the Rock Health group found that not all those with clinical outcomes shared what they found, and claims made by digital startups are low. In an increasingly competitive environment, selling what the clinical outcomes are is going to only help you. This is mainly because

• Clinicians want to know the product is safe and effective
• Payors want to be able to invest in a solution that works
• Patients are increasingly the consumers of their health, and they want to know this is going to help them.

In the example of diabetes, there are many solutions so having evidence-demonstrated ideas of what works is going to help in all 3 categories. Many Type 1 diabetics are especially well educated about their solutions, and so they can give valuable feedback if the clinical outcomes are there and sold to them.

Also, don’t be afraid of negative studies or those that don’t prove that the product worked in the way it was used. That can be used to improve, iterate, or pivot your direction. It will also aid others who want to improve care for a targeted population. It is not often that we get this data as it is far more common to see studies that are positive. A company that published a negative study decided to continue on, pivoting their goals and then demonstrating where their product could work. This instills far more trust in their vision and improves their chance of longstanding success.

Don’t embellish

On the flip side, do not market something you cannot backup. It will come out and when selling to employers or payors, it will become clear. I will give the benefit of the doubt that it is not nefarious – not all research is equal and there are badly designed studies or the results are not actually statistically or practically significant. Make sure you do your due diligence and ask if you do not know.

Share your data

Companies selling to employers had higher clinical evidence and demonstrable claims compared to other customer groups according to Rock Health. So it’s clear that employers and payors want this information. It also has to be targeted to your vision and goals. If you know you want to target patients with CHF who end up in the hospital once a month, that is what you sell to payors. Having a dietary monitoring program may be broader but can be applied to a whole host of different prevention and treatment programs. Just be clear what you are fixing whether narrow or large.

In conclusion, this is healthcare. We are aiming to improve care, quality of life, and do no harm. To aid in the collection of understanding the clinical utility of the digital health products out there, we all have to be part of this change.

• Good research design is an extension of understanding your product and strategy. Make sure they are in alignment.
• Product clinical outcomes are necessary for a forward-thinking strategy. No end user will want to use something that is not demonstrated to improve outcomes.
• A database of companies and products with relevant clinical outcomes is needed. It is impossible to understand what works and what doesn’t without it. This can inform the future of digital health.

Here’s to more and better clinical evidence in 2023!