And you don't think law enforcement agencies will quickly make the leap from "closely observe" to "harass and arrest for any tiny transgression in an effort to coerce a confession"?
Exactly. Instead of doing their job manually, or double- checking system output, they begin to take what the software package says as truth. Whether it’s just human nature, or overwork, or whatever, it’ll happen.
Because Law Enforcement generally behaves like people, and (like people), often take cognitive short-cuts when making certain assessments. In this case, the short-cut would be to assume that the individual has to be a criminal, and act as such.
Now, there are step that you can take to reduce these assumptions, but those are steps outside of the program, and would have to be introduced in tandem with the introduction of the program. Unfortunately, policy-makers often assume that any single idea that is funded (i.e. buying a predictive model from Palantir) is a comprehensive solution to the problem (often encouraged by the sales reps peddling whatever the 'solution' is) and fail to recognize that other programs will have to be funded alongside in order to deploy the program effectively.
If the software says Bob probably did it, Johnny Law will assume Bob did it and stop looking elsewhere, precisely because Johnny Law wants to close the case. Throw an overzealous DA in the mix, and it sucks to be Bob.
That's just not how it works. Eventually you have to present actual evidence in an actual court and it won't be "this software you never heard of said so".
