The first open enrollment period for the Affordable Care Act (ACA) ended with larger than anticipated enrollment in qualified private health plans and Medicaid. The strong showing occurred despite early and, in many cases, persistent problems with the stability, load capacity, and performance of online state-based Marketplaces and, the federal website used by states without a state-based Marketplace. These technical problems have been widely reported and dissected.

Less attention has been paid to the enrollment sites’ user interfaces. State and federal agencies have the opportunity to learn from 2014 consumer enrollment experiences as they work to make system changes and improvements before the next enrollment period.

To understand how well online interfaces worked for consumers, the California HealthCare Foundation (CHCF) funded real-time user testing of and (California’s state Marketplace). The research yielded fascinating actionable findings that can help improve consumers’ experiences; increase the likelihood that they will provide correct information in the application; and support timely and accurate eligibility determinations.

The consumer usability assessments were conducted by gotomedia, a consulting firm specializing in user experience research and design and based in San Francisco. Unlike typical website performance studies that might track system downtimes, analyze where in the process people exit the system or seek help, or conduct post-enrollment interviews, these usability assessments involved watching consumers as they applied for coverage. By doing so, researchers were able to uncover reasons why consumers resorted to guessing, why they would quit at certain points, and why and where they encountered the most difficulty in applying online.

The unit of analysis in this study was the website: the study focused on characteristics and features of the two sites that elicited particular consumer reactions and emotions. For that reason, conducting an in-depth study of a relatively small number of participants yielded rich and reliable insights.


Researchers undertook the following activities as part of the study, which was conducted near the close of the first open enrollment period (during February 2014 for the California website and March 14–27, 2014, for the federal site):

* Recruited a diverse group of individuals (as to age, gender, household composition, place of residence) who wanted to sign up for health insurance coverage.

* Obtained consent from participants to allow researchers to view their screens remotely and to watch and listen via webcams as participants applied for coverage.

* Visited several participants’ homes to observe enrollment sessions.

* Recorded sessions and created video clips (with personal data removed) to share as part of the findings.

* Analyzed observations and synthesized results in detailed slide decks for and


The assessments generated a variety of findings that, if resolved before the next open enrollment, could improve the consumer user experience. Some would require only simple wording changes to clarify certain questions and minimize guessing. Others call for more complicated fixes and underscore the complexity that consumers face when called upon to evaluate and purchase health plans.

The gotomedia team studied fifteen people using and twenty-eight using Among these forty-three people, three enrolled in coverage during the initial ninety-minute testing sessions, and three more were enrolled when researchers followed up two weeks after the sessions. Those who enrolled were glad to have done so, and even some who did not enroll commented that the process was less onerous than expected. In addition to technical errors with the websites, reasons for not enrolling included:

–          Challenges associated with sorting plan options, choosing a preferred plan, and deciding whether to purchase it.

–          A “dead-end” experience for those eligible for Medicaid: consumers who were found likely eligible for Medicaid found their online options terminated abruptly. Most were left with unanswered questions about next steps.

–          Spending up to an hour in a “plan preview” portion of the site without realizing that the information viewed there neither aligned with actual plan offerings nor fed into the application process. That was because the preview content was displayed prior to the final determination of plans and prices, so the comparisons and trade-offs mentioned were illustrative only.

–          Cumulative irritation with minor issues, such as formatting and data entry problems.

Researchers also observed application sections where consumers guessed at answers to questions, raising concern about the accuracy of the program eligibility assessments. For example, many consumers guessed at the meaning of “household” and “income” in ways that could affect final eligibility results. A video clip of a consumer trying to use the website shows this problem and illuminates how direct observation can be a revealing way to more fully understand the consumer experience.

The findings from these studies have already been put to use by both Covered California and the federal Centers for Medicare and Medicaid Services (CMS) as they work to improve their enrollment portals before the next open enrollment period. In a CHCF public webinar presentation of its findings about Covered California, representatives from Covered California described how the CHCF-funded research broadened their perspectives on the online experience. State leaders and consumer advocates have identified several report recommendations as high priorities when planned enhancements are made to The full reports on and are available on CHCF’s website.

While the research findings are particular to the two websites, other states managing state-based Marketplaces may find both the specific and the more general results to be useful. They may also want to investigate this kind of direct qualitative assessment. Similarly, states transitioning from the Federally Facilitated Marketplace to their own systems may benefit from real-time consumer research assessments prior to launching new sites.


Private-sector businesses often conduct user testing to understand how customers interact with their websites and how that experience can be improved. In the public sector, however, conducting such reviews—particularly when timing is tight, resources are limited, and program administrators face many competing priorities—is not the norm.

In Fall 2013 it was clear to CHCF, as to many others, that the online enrollment experience would be a defining element in the success of the ACA rollout. We were able to use our flexibility and independence to quickly bring new perspectives and potential solutions to the table.

Over the years, CHCF has frequently sought ways to bring to public programs and safety-net care settings the management approaches and design thinking used in other industries. Often we have worked with others in philanthropy to extend such investments. Some examples include using market research techniques to understand why people eligible for public coverage programs did not enroll; creating a design standard for online applications in advance of the rollout of the ACA (UX 2014); and implementing Lean management practices in California public hospitals.

The observations from this research are powerful. Equally powerful is the role that philanthropy can play in identifying new ways to understand the impact of ambitious policies such as the ACA.

Editor’s note—Related resource:

“Open Enrollment, Take Two,” Tricia A. Brooks, Health Affairs Entry Point, June 2014.