There is an independent body called NODE.Health (Network of Digital Evidence in Health) that assists digital health companies in partnering with healthcare systems to obtain evidence. Additionally, certain ambulatory care centers and large healthcare institutions have innovation platforms which accept submissions to review digital health products and conduct additional studies.

Evidation Health is organization that also operates in the realm of helping digital health companies generate evidence packages for their solutions by acting as a 3rd party virtual site to streamline all study processes from study design, recruitment, execution, to data analysis and manuscript development.

Ideally, evaluations are done by a dispassionate 3d party. This can be users, academic centers who specialize in validation research, companies like Evidation Health or even Contract Research Organizations. Never the company itself due to conflict of interest management concerns.

There is a great movement around citizen science that is important to leverage. Using participatory methods for patient facing technologies is especially critical.

It would be valuable to have a variety of stakeholders participating in the evaluation process. From a venture perspective, it has been really helpful to receive feedback from end users. In addition, having evidence evaluated from both an academic and commercial perspective is critical to getting a complete picture of the technology's potential.

I think the one consistent component in evidence generation is objectivity. These digital health companies should be interested in running studies to generate evidence, but the more hands off they remain in the design, execution, and analysis of the study, the better.

Ultimately, evidence only matters to the extent that it persuades clinicians to change clinical practice towards the use of digital health / digital therapeutics.

The cop out answer would be to say that “individual clinicians and their patients are the ultimate arbiter of what good evidence persuades them.” But the useful answer would be to define clinical guideline writers such as American Diabetes Association, American Psychiatric Association, etc. as ultimately the key arbiters as these organizations literally define standard of care based on explicit evidence requirements stated in their guidelines.

Some of the previous comments referred to independent 3rd party research groups which are critical for generating good, credible evidence that stands up to independent review (as recently defined in FDA SaMD guidance docs), however these groups don’t necessarily have an evidence “scoring” role. There are app curation platforms such as AppScript by IQVIA that do score apps in large part on their clinical validation and use this data to inform app selection by health plans/providers, but ultimately such platforms merely have the power to raise relevant objective evidence to the attention of key clinical makers rather than make the decisions themselves.

First a disclosure. I am on the board of Xcertia, so in that way have a conflict. The FDA and Xcertia are quite different in their goals and execution. To bring certain products into the marketplace, FDA clearance is required. It is not an option. In those cases, companies need to generate evidence that the product complies with FDA requirements. This involves a very targeted kind of evidence gathering focused on patient safety, less often on efficacy. Xcertia on the other hand is a guidelines-producing non-profit. Once the guidelines are formed and published, manufacturers will have a choice regarding whether to build software to these specifications. It is our intent that the guidelines themselves be evidence-based, so in that way evidence generation may be influenced.

Agree with Joseph, I see both organizations having very different roles. I think Xcertia has the potential to have great impact on evidence generation. Since it is a bit of the wild west in terms of digital apps, having guidelines as a starting place will be helpful. Knowing that a company has followed basic guidelines that cover security, privacy and data protection may be helpful for organizations interested in partnering to co-generate evidence. Hopefully such guidelines can shorted the generation cycle.

How does Xcertia compare with NODE Health?

While the dissimilarity is clear (XCERTIA can’t clear a product for marketing), XCERTIA and FDA have some similarities in the potential impact of their guidance/guidelines.

FDA’s recent guidance on SaMD clinical evaluation - for example - has influenced evidence development by explicitly stating that software as a medical device can apply for broader definitions (indications) over time based on real world data. I think that this has influenced orgs like the Digital Therapeutics Alliance in how they define their goals (eg generating RWE).

I personally think XCERTIA has a unique opportunity to fill some gaps in industry best practice. For example, AHRQ has recently called certain diabetes apps out for not conducting extensive drop out analysis. In this regard, I am a believer in a standardized reporting of the full “engagement funnel” of any given study so that the process of delivering digital therapeutics to patients and how they engage is standardized somewhat. Very importantly, I think XCERTIA may be well positioned to make recommendations to FDA on what data should be accepted for regulatory clearance, particularly indication and efficacy claims. For example, some real world data may not be collected by systems that have certain quality systems in place eg 21 CFR compliance - there are questions around whether such data should be accepted by FDA. Finally and most broadly, digital health / digital therapeutics is an emerging industry: XCERTIA may have a role in defining a “Goldilocks principle” for digital health evidence such that evidence requirements are neither too formal nor too informal.

This is a good question. The simple answer is not enough! Most products are designed either by engineers for engineers or by physicians for providers. Rarely are patients involved. One caveat though is that best practices in design involve ways to watch an individual interact with technology as oppose to asking an individual what they want in a solution. The iPad was brought to market with no consumer research because Steve Jobs famously said, "we don't expect consumers to know what they need"

I think these end users are increasingly being involved in the development of digital health tools - but as Joseph said, not nearly enough. As Design Thinking makes its way into the digital health space, we are seeing interdisciplinary teams (physicians, engineers, scientists, business professionals) observe what end-users are actually doing as opposed to what these end users say they are doing. This puts the actual experience and practices of end users at the center of digital health solutions -- allowing them to be more impactful than those that don’t do the same.

As a public health(er) turned entrepreneur, one of the most shocking observations I've had over the last few years working in the digital health industry are how many products are being created by more tech people than healthcare people. We've turned that on its head and created a team of healthcare/public health professionals and the tech guy has the task of deploying what we ask for. Design thinking is a crucial process to make all of this come together. I agree with Dr. Kvedar. Design thinking encourages observation and exploration. Asking patients what they want can be a part of the process, but should not be the only driver for features.

Completely agreed that more clinical and real-world patient input is necessary. However, a caveat: clinical-user feedback can sometimes skew toward the ideal rather than the pragmatic. For instance, while a physician might reasonably want to have hourly data on her patient's symptomatology between visits, that level of granularity might not be tenable, and could even lead to patient fatigue. A talented healthcare communication/product designer will discover the "why" behind patients' and clinicians' requests, and will craft a product that addresses the problems at hand without compromising on usability.

"Perfection is attained, not when there is nothing more to add, but when there is nothing left to take away." -Antoine de Saint-Exupery

Collaboration is so important and right from the start. This question and discussion immediately reminds me of Steve Jobs and his way of re-engineering the process by focusing on the end user experience and working back from it. After all, if your end user does not have a positive experience and find your solution valuable, then you will be forced to change or you will wither.

Hi Laura! At PreventScripts, we started with patients designing the mobile portion of our behavior change platform. If given the choice, patients would prefer to be healthy, self manage, and remain at home. Patients really do want to minimize their interactions with providers if given the choice. They just really need provider help in the blind spot while they are at home. Physician population health visualization and analytics then come second. Of third importance, are the C-Suite goals, which vary widely. In a system who is taking on downside risk, it's purely the behavior change which is of interest, A1c reduction, biometric improvements in weight and blood pressure. In a system who is still partially billing CPT codes and participating in MACRA, there are Prevention Focused CPT codes combined with the QI Objectives of the organization which are top of mind. We are interested to collaborate with organizations and work towards RCT to gather evidence. Please contact us if your organization has interest in RCT or short term demonstration projects!

Traditionally, there has been limited involvement of end-users. However, the opinions and thoughts of end-users have been collected during trials, and during various focus groups. We need more involvement early on in order to create compelling digital health products.

I agree with all the comments that more patients need to be involved in the development of digital health / digital therapeutics.

However I recently had the great honor to attend a DiabetesMine D-Data ExChange event and was blown away by the number of companies not only referencing patients but actually being founded and staffed primarily by people living with diabetes.

If I was a venture firm, I’d take careful note of patient entrepreneurs in other areas as I think diabetes is showing the power of patient knowledge that can be replicated in other areas.

some sort of ROI is imperative. These days with the movement to value-based reimbursements, outcomes are often tied to payment, but not always. It also depends on who is doing the purchasing. A solution that results in reliable lowering of blood pressure in a cohort of individuals may not be of interest to a health plan because the downstream value of lower blood pressure occurs years later and most insured move around from plan to plan. Thus the consumer may not be insured long enough for the investment to yield results. A separate conversation around cost/ROI is most helpful.

Beyond that it depends on what is of value to the purchaser. A purchaser in a FQHC may be interested in how a solution impacts social determinants of health, for instance.

To extend Joseph's comments, if a product's purpose is to improve clinical outcomes related to a specific condition, then some actual indication that is does is imperative. The evidence, however, can come in many forms. For example, the evidence can be economic (reduction in costs); changes in patients' symptoms and signs; the clinical decision support algorithms used and feedback provided; the data sets used (i.e., are they extensive enough to address all the relevant factors); the ability to adapt to workflows and built-in flexibility that enables it to evolve over time; etc. Also, its usability and usefulness (user experience) is important, so a test drive would be helpful.

Great question! This builds off something I touched upon earlier. They type of evidence digital health companies should be looking to generate will largely depend on the types of business partners they are looking to work with. Of course, if the end goal of the digital health solution is to provide a clinically significant change in a patient's health, then that should be a high priority. However, there are other metrics that carry just as much weight amongst different audiences: decreased costs, hospital utilization, improved patient experiences, user engagement, and workplace productivity.

Cant agree enough. I have tremendous concern that the “clinical ROI function” (ie how clinical outcomes improvements map to a specific quantitative ROI) is getting far too complex making it difficult for digital therapeutics to communicate their value - particularly to providers - succinctly.

The most ideal answer is yes as we all inherently believe that evidence based products and services are best for our patients, but in business it is about the sale and in this case the real answer is: It depends on who the buyer is.

For some potential customers, common sense is enough. For instance, some purchasers, particularly in direct to consumer settings, would find a statement like "X improved Y by 50%." compelling. This is a bit like saying "dentists recommend this toothpaste". all that is required is to find at least two dentists who recommend the toothpaste to make this a true statement.

the next level of sophistication is a sale where the buyer is not educated in public health or scientific method (more common in a b to b context). Here the level of evidence is a hair more sophisticated than "X improved Y by 50%" but real-world-based pilots and the like can generate a story that is supportive of the product's value. The level of evidence here is often uncontrolled before and after studies, focus groups and the like.

If you plan to sell to anyone in the provider space or to a buyer that brings public health or a science background to the table, be prepared for your evidence to be picked apart with a fine tooth comb. Bring someone along who understand power calculations, p values and the like and assure your evidence is as high caliber as possible. In this case as close as you can get to randomized controlled trials with robust sample size and statistical methodology is important.

I like Joseph's comment. The claims a vendor makes should never be purposely fabricated nor deceptively presented to make a buck. Sadly, not everyone is so ethical, and truth is often not what it used to be. Hopefully, however, such falsehoods will be uncovered before anyone is seriously harmed, the vendor's lack of credibility is exposed, and it is put out of business.

Another possibility is testing all new products via some set of standards. This, however, often stifles disruptive innovation, disadvantages the "little guy," and favors the status quo. So, what we need are honest disrupters who, in a free marketplace, are able to transform healthcare in positive ways.

Short answer is yes. I advise vendors with new products in the digital health space to connect with accelerator programs (or anything of the like). It's a great way to get exposure and to work with organizations that would like to try your product and will partner in building the evidence needed.

I'd love a reference for the accelerator program you mentioned here.

To be honest, I’ve seen a lot of “1st generation” products sell pretty well, particularly in the employer space, without a lot of evidence. I think that it’s a combination of lack of an evidence based culture and a desire to make traditional boring wellness programs more interesting.

That being said, I think that new entrants will have a harder time selling without evidence because the bar is getting relatively high on both the clinical and economic fronts.

From my experience, there are two main issues - 1) Access to healthcare systems, and 2) cost of acquiring this evidence. We have encouraged digital health companies to build "acquiring evidence" into their financial costs, and be up front with these costs as they fund raise for their specific solution.

A number of digital health vendors try very hard to be accommodating and flexible with their product in attempts to win over customers. However, this leads to big problems on the validation front in that the product is validated in one specialized scenario (and hence not true validity). My suggestion would be to validate their base product and then be specific on what 'flavors' are subsequently validated. Allowing the customer to 'over-customize' can be a big risk.

The biggest challenges I have seen are that companies struggle with 1) timelines and 2) costs associated with gathering evidence for validation, which may be attributable to unfamiliarity with the environment.

We are seeing more and more digital health companies emerge from founders with backgrounds outside of healthcare - so the intricacies of actually running a study - such as protocol development, IRB submission, participant recruiting and ensuring compliance - are likely new to them. As Nitin said, these companies need to be up front about the costs of these studies when fundraising - and to do so, they need to be aware of the intricacies.

This requires time and effort as there are no standards. Some useful tips -

1. beware of marketing lingo and 'naked' statistics. If a company makes claims based on statistics ask how the data backing those claims was acquired (e.g., pilot study, randomized trial, open study, etc.)

2. There are many biases that immediately creep into the evaluation of technology.

a. You really can't do double blind studies. Thus people know they are being studied. The cohort of people who have the motivation to sign up for a clinical trial is different than the cohort of patients in a given population. They are self-selected and more motivated than most.

b. Before and after designs are very common and fall victim to regression to the mean. This is one reason control groups are very important.

3. sample size matters. I recall a start up that touted a 50% improvement of their intervention, but was only tested on 12 people.

Of course all of this calls into question whether you need a public health degree to evaluate a company's claims. You do not. But unlike some other areas of life, common sense can fool you. Ask if they've done peer reviewed publications on their work. if not ask for details on how they validated their technology and get an expert to help you analyze their claims.

Dr. Kvedar's points are excellent. An additional, albeit slightly more difficult, request that a purchaser should make of potential vendors is that of corroboration: clinical validation in the form of randomized controlled trials is necessary but not entirely sufficient, and ideally, the vendor can show that it have reconciled the digital version of the efficacy/effectiveness gap by replicating results in commercial implementations of its solution.

This is a great topic! One of my biggest suggestions is to dig deeper into the claims you see companies making in their marketing efforts to better understand their implications.

Here is an example process:

1: Check to see if the company has featured any publications on their website.
2: Understand how the study was executed. How rigorous was the study design? What were participant demographics? Over what time period were the participants followed?
3: As an additional level of filtering, check whether these studies were executed by an independent organization free from conflict of interest.

A point to note is that sometimes evidence is generated by customers of these companies, meaning the customer owns the evidence (possibly preventing publications). So if a digital health company has a lot of customers, they may have strong evidence!

Lastly, we can seek to understand a company’s plan and roadmap for generating evidence. Those who have an outlined roadmap are likely aware of how evidence generation is tied to success, and could differentiate from those “following the trend.”

the biggest challenge with all of this is that doing first-rate clinical validation is expensive and time-consuming. many investors want to see traction and not wait for clinical trial results. One tool that can be helpful is case-control studies where a patient database, typically from an EMR, serves as as a tool to generate a matched control group of patients who have never seen the intervention. recruiting is focused on the intervention and typically goes faster. While not as pristine as a randomized trial, this method is more efficient both in terms of time and money.

one challenge for all of us is to find ways to validate technology that are time efficient and cost less than randomized trials. right now the market moves faster than the science.

This is an excellent question, and having reviewed opportunities from both an commercial and clinical perspective, there are a few guidelines I usually review: 1) Market - We review if the market has tested the product and how consumers reacted to it. If consumers reacted positively, we then review if the change in clinical outcomes can be translated to a gain in market share. 2) Technology data - We collect data from multiple healthcare sites to verify if the proposed change in outcomes has been witnessed. 3) Projections - We match the projected change in outcomes to the actual outcomes in the field.

In addition to what is noted in the other comments, I find the biggest help is to pick up the phone and contact other customers to get the 'real' story. Information gained has been invaluable....

Excellent discussion here and great insights. We can only hope that the test data & evidence is done with honest & true rigor void of bias, but the reality is that the percentage of this is small. Buyer beware as they say and always be sure to conduct due diligence. It's simply not easy.

After all, in the past decade we have all seen the headlines in most major headlines that have identified serious conflicts of interest and/or dishonest data (e.g. auto co's emissions tests, FDA, etc.) Great discussion!

My rule of thumb is to see if a company either has an RCT that specifically tested their product (beware: a lot of companies reference similar products). If you filter that way, you’ll get down to a very solid list of core products quickly.

To get started, some basic evaluation studies would be helpful and where possible, pre/post metric measurement. I find when you can tie the project to ROI related activities, the organization may be able to obtain research support.

Great question! Although pilots, retrospective analyses, and smaller quality improvement studies are better than nothing, prospective randomized controlled trials are the best way to determine whether a new process, intervention, or innovation is working.

I recommend working with experienced researchers to help determine the minimum trial size required (power analysis). Clinical researchers and biostatisticians will boost quality and credibility to the final case study or journal article. Additionally, for digital tools, consider setting up a "placebo" arm to control for Hawthorne Effect, whereby the simple introduction of a "touch" can confound results.

Now, the question is how can companies actually implement this type of research, since IRB-covered work can be expensive and time-consuming. In short, it's an investment worth making: the risk of digital health "snake oil" is real, but high-quality products will speak for themselves. For startups, make sure to model research in financial projections and budgets. Additionally, many innovative health systems are open to collaborating on high-impact research projects as they choose to adopt new solutions; consider offering discounts for partners who can offer such value.

Above all, make sure you're answering the right question. As Victoria Tiase mentioned, studies in the digital world are generally intended to show ROI (as opposed to equivalence or safety); make sure to align clinical outcomes with financial interests.

When it comes to evidence generation, the gold standard is always going to be a prospective RCT. That being said, Lisa makes a good point in that there are simpler ways for digital health companies to start the evidence generation process. Here are a few that are effective depending on the audience and purpose:

1) Economic Modeling
2) Retrospective (claims) analysis
3) Smaller pilot / feasibility studies

It is important to note that when evaluating evidence, some audiences will look for clinical significance while others may look for economic impact, user engagement, or other agreed upon outcomes.

One consistent factor between the types of evidence these partners are interested in is objectivity. While we are all familiar with the phrase “if you want something done right, do it yourself,” digital health companies will greatly benefit from having an independent third party organization evaluate their solution. Why? Because business partners will want to see data that is free from any biases or conflicts of interest.

The examples of studies and metrics posted above are valid, but first is deciding what it is you're going to evaluate, which should be tied to the app developers' claims. A claim may be that the app:
• Collects and transmits data accurately, reliably, securely and inexpensively
• Provides useful information
• Supports clinical decision
• Simplifies clinical workflows
• Facilitates quality reporting
• Saves money/time
• Enhances care team collaboration and coordination
• Improves patient self-management
• Delivers a delightful user experience
etc.

Each claim has its own set of metrics, and some apps may have claims of providing multiple benefits to persons of different roles.

So, I recommend identifying classes/types of apps based on their claimed deliverables. Then determine what data is needed to evaluate those claims. Then design a method for collecting, analyzing and reporting those data. And if you want to compare apps, there must be a standardized way of doing all this for each type of app so everyone "plays on a level field."

There are so many types of studies that can be pursued. Traditionally, we look at whether RCTs have been conducted. However, more recently, especially in the digital health startup space, feasibility studies have been used due to their lower cost and ability to navigate the complex landscape. It provides the type of information required, especially in the early stages to determine if the digital health solution is a viable one.

I agree that RCT may very well be the best type of study when feasible.

I'm all for RCTs but in digital health startups, that is an aberration...

Victoria is right IMO. RCT may be feasible, however, if the startup was grown and funded, e.g., in a university setting that had the necessary resources to do the study and if the claims did not stray from what could be proven empirically.

Other respondents have done an excellent job enumerating study types.

I wanted to discuss differentiation in an increasingly crowded space.

The fact of the matter is that many industry leaders in the large chronic conditions (eg diabetes, asthma, copd, 27 use cases in total based on recent IQVIA Institute report) now have RCTs. The strategic implication is that to be a leader in the big markets, an observational / retrospective study may no longer cut it.

Our evidence experts at IQVIA increasingly believe that the industry will now set its sights on two very specific types of studies: pragmatic RCTs and real world cost effectiveness studies (not models).

Pragmatic RCTs combine the key component of an RCT (randomization) with the real world settings, patients, and data that makes RWE great. Such studies have the potential to answer many different types of questions about not only narrowly defined “efficacy” but real world effectiveness and ideal practice patterns.

Real World Cost Effectivess Studies measure true cost of care in the context of real world populations as opposed to creating a model to extrapolate toa cost savings number.

It's been noted that close to 200+ healthcare apps are added to the app stores on a weekly basis, making it difficult for both patients and providers to know what is safe and what's not. And safety can be thought of from a not just a medical error perspective, but regarding data security and privacy as well.

In recent years, digital health companies have begun to flood the market - each touting a unique approach to engaging patients and managing disease. However, only a subset actually have the data to back up claims of improved outcomes and reduced costs. With so many players in the field and so few solutions with evidence to support them, it is no surprise that the health ecosystem is hesitant to adopt digital health solutions at scale.

Evidence generation is important in digital health for the same reasons it is so critical in the rest of medicine -- we need proof. Doctors need to know that these solutions work, how well they work, and for whom they work best before they can be expected to prescribe digital approaches and asked to monitor patients through them.

At the end of the day, the evidence generated will speak for itself. Some digital solutions will rise to the top with impressive improvements in health outcomes or lower costs (and some will have both!). Others will prove not to work -- and that’s okay. As digital health solutions are gaining traction and differentiation through data that backs up their claims, the path forward has to be evidence. This allows the broader ecosystem to scale the most impactful digital approaches and unlocks the potential of these approaches to significantly move the needle.

I'd include EMR/EHRs and PHRs in the list of digital apps as having very similar issues as mentioned above. For all of them, issues of security, privacy, data reliability and validity, interoperability, the effect they have on outcomes, the user experience, workflow disruption, influence on provider-patient relationships and care team referrals and interactions, the impact on the discovery and use of new knowledge, the ability to enable the presentation of a patient from a whole person (biopsychosocial) perspective (including SDH), and the impact on shared decision making, personalized medicine, information overload, alert fatigue, clinician burnout, etc.

Digitizing health data has, IMO, tremendous positive potential to improve clinical outcomes and the well-being of populations by enabling us to build, evolve and use the knowledge we need to (a) deliver high quality care with the least economic burden on the patient and (b) economically reward clinicians and health systems that demonstrate it. At the same time, serious downside risks are very real. Reliable objective evidence digital apps’ value must be separated from spurious unsubstantiated claims, and this evidence should be compiled and evaluated continually, presented transparently, and easily accessible.

Varun, great points on the evidence generation piece. I find that many times, digital health falls under 'innovation' or 'disruption' departments so they move quick and fast. Little time is given to evaluate to write up successes/failures... I wonder how we can do a better job in sharing what works and what doesn't in a quick and nimble fashion.

Evidence in digital health guides both the commercialization and clinical arms of healthcare systems. Generating evidence will guide where limited resources should be placed. On the commercialization front, healthcare systems and sophisticated investors can decide which types of companies to fund in order to create desired improvements in clinical outcomes. On the clinical front, healthcare systems need to decide which applications deserve to be implemented across their networks in order to improve patient experience.