As health care costs swell, the private insurance system that covers most working Americans is in crisis. Americans are paying higher and higher premiums for increasingly threadbare coverage, and employers are getting out of the business of providing health care altogether. Rising costs cannot be attributed purely to improving technology or increasing operating costs for providers, because Medicare has controlled per capita spending more effectively than commercial insurers that provide employer-sponsored coverage.

Rather, commercial insurers cannot contain costs because the pricing mechanism for medical services is broken. When it comes to health care, competition simply isn’t working.

Prices in the private sector are out of control. On average, private insurers pay 25 percent more than Medicare for physician services and 30 percent more for hospital care. What’s more, both public and private sector payment rates for doctors in America are far and away the highest in the world, and research suggests that these high rates are among the principal reasons health care is so much more expensive in this country than elsewhere.

These international gaps are much wider in the private sector. For instance, private payments for an office visit in the United States cost 70 percent more than those abroad, while public payments are 27 percent higher.

And prices aren’t just high; they are rising. Data on private insurers is notoriously hard to get, but the industry has chosen to release, on a very limited basis, information about rising health spending to the Health Care Cost Institute. The Institute’s report revealed that the main reason spending is rising from year to year is increasing unit prices, or fees paid to hospitals, doctors and other providers.

Looking at local markets illustrates how prices have become excessive. In Oregon, the cost of many medical services is rising at roughly 10 percent a year, more than three times faster than overall inflation. In California over the last decade, hospital revenues from a day of inpatient care have risen 78 percent for Medicare and 150 percent for private insurers.

Prices are rising on both a totally unsustainable trajectory and a seemingly irrational one. Prices vary enormously from place to place (the cost of a colonoscopy can fluctuate by 400 percent within the state of New Jersey) and within and outside of provider networks (out-of-network charges can rise to 5,000 percent of the Medicare rate for a given procedure). These are not the hallmarks of a system that features efficient pricing.

Average Annual Rate of Price Inflation, Oregon Hospitals, 2005-2009

Archer-Exhibit

Rather than being influenced by competition, health care prices are largely set by insurers and providers with monopoly power to maximize profits. Big hospital chains and provider groups dominate most local markets and extract extremely high rates from dominant insurers, which are motivated by fear of losing market share if they fail to attract these providers to their networks. Research indicates that hospitals can change their business practices and control their costs effectively when faced with competitive pressure, but health care markets have concentrated in the last few decades. Providers simply haven’t had to compete to offer high-value care.

Commercial health plans have little bargaining power when they negotiate prices with monopolistic providers. In fact, even insurance industry lobbyists admit that private health plans cannot hold down the cost of health care. Insurers choose instead to adapt to this non-competitive environment. Sometimes they resort to collusion, as was the case with Partners Health, a massive provider group that entered into an ethically dubious arrangement with Massachusetts Blue Cross, as the Boston Globe reported in 2009. In that deal, Partners agreed to ensure that no other health insurer would pay lower rates than Blue Cross.

While that scandal was an extreme example, the fact is that big insurers regularly negotiate most-favored nation clauses with providers, thereby agreeing to raise rates significantly while guaranteeing that providers will charge other insurers higher rates and passing the additional costs on to consumers and businesses. And why would they seek out innovative ways to absorb risk and deliver high-value care when the current system delivers big, reliable profits? In any case, most insurers face little competition, so there is little appetite for pushing rates downward.

On both the left and the right, serious scholars realize that the system is failing to yield positive-sum competition. Several years ago Michael Porter and Elizabeth Teisberg, conservative business professors who are fiercely opposed to public health insurance, wrote in the Harvard Business Review that every kind of competition now prevalent in the health care system drives up costs and the kinds of competition that might push costs down are absent. Meanwhile liberal scholars like Ted Marmor, Johnathan Oberlander and Joseph White have illustrated that Medicare controls costs much more effectively than private insurers, and these academics advocate the broad adoption of road-tested Medicare mechanisms for influencing prices.

So what is to be done? The Patient Protection and Affordable Care Act (“ACA”) will for the first time guarantee the overwhelming majority of Americans access to good coverage, a huge step forward.  But it expands coverage mostly by relying on the dysfunctional private insurance market. There is no evidence that the newly created exchanges will exert any downward pressure on prices, given the experience of private plans to date.

Congress needs to recognize that players in the private health care marketplace will continue to set excessive rates until they are stopped. These exorbitant rates are not only hurting working people, they are also driving up Medicare costs and imposing a massive burden on taxpayers and the federal government. Doctors and hospitals are conditioned to expect higher and higher rates and demand higher payments from public programs.

Congress has three options to rein in runaway prices: It can use Medicare-style techniques to set rates or rate ceilings in the commercial marketplace, including in the new health insurance exchanges, just as every other developed nation does. It can give people under 65 the choice of a public health insurance plan that works like Medicare, competes against the private health plans, and brings down costs. Or it can do both.

Over the long-term, the federal government might be able to address the broken pricing mechanism by enforcing antitrust laws more aggressively than before to break up monopolies in health care markets. But it is worth remembering that even heavily regulated insurance markets – such as Medicare Advantage or the exchange system in Massachusetts – have not been particularly successful at controlling costs. The simple fact is that the array of existing incentives for commercial insurers does not tend to drive down prices. Given the direction of commercial insurance, relying on the current version of “competition” is destined to jeopardize access to health care for millions of working Americans while driving public spending upward.