The Veterans’ Administration has been promoting its effectiveness for some time, based on both administrative and clinical measures. As a result, some health reformers have even taken to touting the VA as a model for national health reform. Now their findings and the VA’s veracity are being challenged by a new report. The VA still has its defenders, but I have questions about other favorable studies of the agency (see below.)
They’ve done some things very well, but there’s no reason to believe the VA is a model for the nation. The issue should be resolved through a comprehensive analysis of their medical outcomes conducted by unbiased analysts.
An in-depth journalistic investigation now suggests that the VA has been misleading policymakers and the public about its accomplishments for some time. The gamesmanship reported by McClatchy ranged from the relatively trivial (e.g. changing request dates in the appointment system to make wait times appear shorter) to the more substantive.
Deceptions about appointment wait times and patient satisfaction are wrong, and overstating the availability of PTSD treatment is reprehensible. (Why not ask for the money to give vets what they need?) Still, in my opinion, skewing clinical data to overstate results is the most serious charge of all. A RAND study cited by the VA didn’t say what VA officials claimed it said: namely, that the VA performed better than 12 other systems. Officials went on to say that RAND had concluded that veterans “receive better health care than any other patients in America.” The RAND study was not designed for that purpose, and found no such thing.
McClatchy raises questions about the RAND study’s objectivity, given the fact that four of its nine authors were affiliated with the VA and the agency helped fund the study. RAND’s objectivity is well-regarded by me and others, so I’m less concerned about this issue.
The newspaper’s other question about the RAND study is far more telling: It was published in 2004, yet only used data from 1997 to 1999. In my consulting experience, unexpected date ranges raise a red flag – could the data have been cherry-picked to get the best results?
Something like that could well happen without RAND’s complicity – if they were engaged to study a certain data set, that’s what they would do. The selective use of data could occur when the client engaged the consultant/think tank to do the work. I’ve seen it happen a number of times in my data analysis work, although its more typical of the private sector.
Some people like Ezra Klein are still arguing that the VA’s quality is unsurpassed, based on pieces like the ones published in the New England Journal of Medicine. Yet I have similar concerns about the New England Journal pieces, as well. One such article (“Effect of the Transformation of the Veterans Affairs Health Care System On the Quality of Care”) was published in 2003 yet used essentially the same date range as the RAND study – and focused on a fairly limited number of conditions.
McClatchy notes rightly that the VA has performed very well in providing necessary tests and services. And Ezra and other are right to note its excellent performance in a number of areas. But analyzing health data is a tricky affair.
The VA may have more to say in its defense, but I found the explanations they offered to McClatchy unconvincing. Here’s the way to resolve the issue once and for all: fund a comprehensive review of the VA’s outcomes, conducted by independent analysts with full access to all of the VA’s data.
Until that happens, I’m not comfortable with any of the rosy reports on the VA’s effectiveness. They may well be doing a terrific job in outcomes, but there’s no independent verification of that fact. Until there is, I would argue that using the VA as a model for national reform should be put on hold.
