Monday, April 12, 2010

You Don't Know What You Don't Know

Robin Hanson on a report about the science of solving crime:
The primary social pressure on law court practices is for courts to give the appearance of punishing guilty folks.  Observers have much less info on who is actually guilty.  So the main pressure on legal standards is that officially-accepted evidence seem to the public, juries, and judges to indicate guilt, not that it actually indicate guilt.  We expect the law to be overconfident about its evidence.
The ideal in society is that criminals are investigated, charged and punished based on actual guilt by completely objective observers. While this is the ideal, I think that there is a secondary, less upstanding motive that acts in the heat of the chase that is rarely ever acknowledged.

That motive is what Robin describes: that the investigator makes sure that the public sees that someone who appears guilty has been punished. The way that sentence reads might sound like I am implying that investigators choose not to do their job properly and callously throw people in jail. That's not what I am suggesting.  I have no experience with any kind of investigation and I have a lot of respect for the people that do that kind of work. I'm positing that there is another motive that has a lot of influence but is rarely acknowledged or that the investigators aren't even aware that they are being influenced by it.

The First 48 is a show purporting to accurately showcase real homicide investigations. It goes like this: there's a homicide and the investigators get called out. They usually know nothing about anyone involved. They get to the scene and have to go from knowing nothing to knowing as much as they can about the situation. Sometimes it's easy- the guilty party has been caught on surveillance or multiple witnesses identified the shooter- usually it's not though. Often times there's information that appears to add up and point to a specific person at the start of the investigation but as the layers are peeled back the new information tells a different story. Early on the investigators suspect someone who, based on all the information that they have, appears to be guilty. The investigators are often relying on gut feelings and intuitive judgments of the evidence that they've gathered so far. The suspect's guilt seems probable or even obvious to the investigators (and the audience).

It's at this point that the motive changes- they'll say that they need to bring this guy in and hope to get him to confess because they need more evidence to charge him- the objective isn't to get more evidence to objectively prove their hypothesis that he's guilty but rather to get the evidence to be able to charge him because they are convinced of his guilt based on what they know at that point. It isn't all that rare that the person that they bring in has an alibi or is otherwise proved not to be guilty (for example, non-matching fingerprints). Sometimes a certain person will look like the suspect but as the investigators gather more information it becomes clear that that person wasn't involved at all. It's revealed that the evidence that they had previously wasn't complete. The problem was that they didn't know it was incomplete.

This lesson applies much more broadly than simply solving crime. In everything involving everyone, no one knows what they don't know. Humans see the information in front of them as complete. It isn't in our nature to acknowledge or operate with the assumption that there is information that is relevant but that we simply don't know about at that time. Nassim Nicholas Taleb makes this point in Fooled By Randomness with regard to financial risk calculations which are all based on known risk factors. There's no consideration of any risk factors that we might not know about but the calculation is presumed to be a comprehensive picture of risk. Humans arrogantly presume that in a certain situation they know all possible information - the problem is they have no concept of what they don't know. That has far-reaching implications.

No comments:

Post a Comment