The Problem With Algorithms

Facebook's algorithms are in the news a lot these days, and with good reason. But this isn't about Facebook. I've been thinking about algorithms in general lately. This has been brought on by my wife's insulin pump, the Medtronic 670G. It's the first closed loop pump approved by the FDA. That means it can monitor your blood sugar levels (actually an approximation as it's not testing your blood) and turn your insulin supply on or off as needed to avoid dangerous lows or highs. If you run it on automatic mode it will micro bolus insulin to try to keep you glucose level between 120-180.

That sounds great in theory. However, the "decision" to shut down or reestablish insulin delivery is driven by an algorithm that we have no legal right to know the details on. This thing could literally kill my wife, and we have no real idea how it actually works. We have only a general idea based on observation of the conditions that cause it to do something with insulin delivery. Compounding the concern is that we have documented that the blood glucose monitor accuracy varies with the rate of change of blood glucose readings. If your glucose levels are rising or falling rapidly the CGM readings can lag an actual blood sugar test by 30% or more. So we have a black box system making decisions with less than accurate data in some cases.

Or take another, more speculative example. Imagine an autonomous vehicle doing 55 mph in heavy traffic. A person jumps into the path of the car, braking won't stop the vehicle in time, and swerving right or left will cause an accident and potentially endanger additional innocent people. What does the vehicle do? Now imagine that the developer of the algorithm had certain beliefs about the value of lives based on skin color, and programmed skin color recognition into the danger avoidance algorithm, so that the vehicle might react differently based on the skin color of the potential victim.

If you think this is too far-fetched to be worthy of conversation you haven't been paying attention.

Algorithms are going to continue to take on more and more of the decisions traditionally made by people. What rights do we have to know the underlying logic of these algorithms? It's already in play with things like credit scores, which can pretty much ruin your life. It's only a matter of time before an otherwise innocuous seeming algorithm kills somebody, if it hasn't already happened.

Today we have no rights. I think we should have a lot of rights to understand the logic powering these algorithms.