[this post features quoted material from Malcolm Gladwell’s podcast series Revisionist History episode 2, titled Saigon, 1965]
LBJ used to walk around with a summary of Goure’s findings in his back pocket.
We all carry something special in our own figurative pocket. Each of us has a favorite text or journal article that provides evidence for what we do in the clinic. Of course, there is probably an equally compelling text or article that arrives at a different, opposing conclusion, but we elect to ignore that one. The reasons vary, of course: the sample size wasn’t large enough, the effects were too small, the findings didn’t reach statistical significance, the population isn’t representative of who we see in the clinic. Let’s be honest, the reasons [excuses?] are seem limitless.
. . . .
More than fifty years ago, when the United States was trying to figure out how to effectively intervene in Vietnam, they needed data. The wanted to understand their enemy. They hired the RAND Corporation to study the Viet Cong (VC) and the effect of Vietnamese/U.S. military intervention.
RAND is a place that prides itself on objectivity and rigor. Everything is checked, and double-checked, and fact-checked, and reviewed in-house before it’s released.
RAND hired Leon Goure to spearhead their project. He worked together with Mai Elliott to interview over 2,400 VC detainees. Goure’s report, over 60,000 pages in length and published in 1966, concluded:
“The survey of interviews completed in the spring of 1965 noted that the intensification of Vietnamese government and U.S. military activities had had a significant adverse effect on Viet Cong morale, operations, and expectations of victory. The interviews also indicated that the Viet Cong were losing the sympathy of the rural population … These trends have continued, and some have intensified … the Viet Cong are facing greater military, political, and economic problems, that in the experience of those questioned VC morale has become brittle, and that the combat effectiveness of some units seems to have declined. The tendency of the rural population to try to disassociate themselves from the Viet Cong has become more pronounced.”
. . . .
Qualitative data is important, but we need to be very careful about how we collect and interpret the information. We need to keep in mind that information can deceive us, especially when we mistake it for data.
When a patient seeks your care or guidance for a complaint, how do you measure their progress, or lack thereof? Do you assess the patient’s complaints? Findings of strength or ROM? Validated outcome tools? How do you determine what approach should be most effective? Have you thoroughly combed through the relevant research and literature to answer these questions?
If you are looking at the research – and I hope you are – how many studies make great efforts to take mere information and selectively massage data from it? How often are we taking subjective information, and erroneously mining objective data from it? How often do the interpretations of the authors match your own?
. . . .
Looking back, there was one particular interview that Ms. Elliott conducted that has stayed with her all these years – you know, like that one patient who (for one reason or another) you will never forget:
“I walked into this cell and I didn’t know what to expect. And then walks in this man, middle-aged, very briskly, and he looked like a man of authority [he was a general]…I was afraid; I didn’t know what was going to happen, because I had grown up believing the communists were blood-thirsty … I had never met a communist before, face-to-face, so my curiosity just took over and I just asked him a lot of questions about him, and his background, and his beliefs. And he had devoted his whole life fighting the French and now he was fighting the Americans. And he seemed to have a lot of integrity. It really confused me, because I had believed that communists were thugs…someone who is not quite human.”
Malcolm Gladwell explains: “She saw the evidence with her own eyes, she did the interview with the general, but it wasn’t enough… She comes from a family of privilege and the rise of the Viet Cong in the north takes that all away. They end up living in a little hut in Saigon. The Viet Cong is not some abstract force, they were a personal threat to her family. “
The paper in LBJ’s pocket – the information that would guide the U.S. involvement in Vietnam – was informed (in part) by the interview that Ms. Elliott had conducted and that Mr. Goure would interpret.
. . . .
It’s easier to be objective when you don’t have a personal stake in a situation and you can see the evidence … But when you have a deep, strong, personal stake it is a lot harder…
What is your preferred method? Has your method been proposed by one sole thinker or visionary? If so, how likely is it that this single visionary happened upon an idea that no one else had come upon after centuries of anatomic and physiologic study? How did they arrive to their conclusions? How invested are you in their ideas? How many of their courses have you taken? How much money have you spent to learn the method? How much time have you spent marketing these methods? How much of your successful practice is founded on these methods? Can you provide robust evidence, beyond personal and second-hand anecdote to support your method of care?
. . . .
Approximately one year into the project, RAND brought in another individual to work on the project, Konrad Kellen. Kellen’s backstory alone is fascinating, but not relevant to this posting. What is relevant is that Mr. Kellen was to review nearly 1,000 of the interviews earlier conducted and interpreted by Goure and his team. Kellen’s findings were in stark contrast to Goure’s:
One could also make a short statement regarding the enemy’s morale and motivation, such as: morale is high, and the enemy’s main motivation is his belief in the “Revolution.” But such summaries would not contain the essence of the interview materials, nor would summary statements actually made at the conclusion of each section reflect fully the content of the study. Only the complete text can give the reader the full flavor of the responses…
…It is well-known that totalitarian propaganda is often not ‘bought” by the lower levels, at least not in its entirety. There is usually considerable slippage. On the other hand, to the extent that it is accepted, often it is repeated in a parrot-like fashion, which may indicate brittleness…
…Only by immersing himself in these responses can the reader obtain a genuine feeling of how high morale or how strong motivation is on the other side, and under what circumstances these two related forces are likely or unlikely to disintegrate. The enemy’s picture of the world, his country, his mission, and our role in his country is remarkable by its simplicity, clarity, and internal consistency. And the tenor of his response is remarkable by the control of his passion, and by his matter-of-factness and clarity.
. . . .
Again, Mr. Gladwell: “One interview with a Viet Cong officer. One fantastic bit of intelligence. An insight into the enemy’s mind, and yet everyone was in disagreement on what it meant, because everyone was looking at it through a different set of eyes. That is why intelligence failures happen. It’s not because someone screws up or is stupid or lazy. It’s because the people who make sense of intelligence are human beings with their own histories and biases.”
As I listened to this wonderful podcast, I couldn’t help but think of parallels between what I was hearing and my profession’s challenges when trying to tackle, grasp, or gain hold of evidence for therapy care. Sometimes we take subjective information and try to make sense of it. Other times, we analyze objective data for interpretation. We take outcome measures (based on subjective reports) and try to establish cause and effect after intervention.
Our intentions are good – we aim to provide quality care and we do our best to remain objective, but even when we exercise measured intellectual restraint and control, our biases remain ever present. We choose what to read, critique, and assimilate from the evidence. We all do so differently.
. . . .
Yes, Kellen had been right about the morale of the Viet Cong, but he hadn’t been without his biases. His biases had only aligned with the reality on the ground in Vietnam. He understood this, but what he didn’t understand was how others could read reports of the conflict and remain, somehow, disaffected:
“There were a lot of civilians around … who talked about casualties, for instance. They didn’t give a damn about anything. If somebody came back and said [there were] 50-60 casualties … a casualty is not a dead person. A casualty is something theoretical for these people.”
We read studies with subjects and participants; each is used as only a number to the reader. If the study is done properly to remove bias, the evaluator is blinded; they don’t know the individual’s personal history. It is their job not to care, but to act on or measure a person, they render a subject as object; the patient becomes an abstraction. Such data is used in qualitative studies, case studies, case series, randomized control trials, systematic reviews, and meta-analyses. Each is widely-considered to be distributed along a hierarchy of evidence with information culled from RCTs and SR valued more than data from single subject or small group studies. We necessarily view evidence in a hierarchy in an effort to avoid the same pitfalls Goure succumbed to, but if we look at the evidence with a closed mind, without seeing the person as a subject, we fail to live up to Sackett’s justifiably lofty expectations.
What we fail to recognize is that we carry the same thing in our pockets that LBJ carried in his: bias. But the more we lean on our biases, the more we fail to be good clinicians. The more we fail to be good clinicians, the more superficial the impact of science is on our care.
Decades after his research on the morale and motivation of the Viet Cong had concluded, Mr. Kellen looked back on his time with RAND with some disdain:
“The people that I knew who talked a lot about scientific this-and-that were the most unscientific people you can imagine. They just picked somebody and then if they agreed with him (or he agreed with them) then he was an expert, and if he didn’t agree with them then was not an expert and then they wrote it down. It was almost like a comedy, you know. So stupid.”
Have you made every effort to reduce your own bias, or have you become apathetic to your intelligent failures? Are you as smart as you think you are?
Don’t forget: the folks at RAND thought they were pretty smart too.
Keith's Korner 