In all the punditry about chronic underemployment and robotic automation, it is the specific stories of job loss that truly break my heart. Compared to this, the mediocrity we accept by replacing human judgment with automation is but a minor irritation. But sometimes mediocre automation graduates from irritant to cancer. I have just such a story, and it serves as a cautionary tale for how automation can disempower entire communities of practice, in this case innovative new entrepreneurs. So here is a very real story of mediocrity in automation, brought to you in five short acts:
Act 1: The Entrepreneurial Garden of Paradise
This story begins much like many outstanding Kickstarter concepts, with an idea, an announcement and the starting capital to create a new device. In our particular case, we announce a new air quality sensor at South by Southwest and, with fanfare, begin accepting customer pre-orders, using PayPal as our credit card processor and bank. We watch as orders ballooned, and we set to work manufacturing the very first month's air quality sensor shipments.
Act 2: Deliverance Shackled: Robocop enters
When all the sensors are calibrated, we ship orders out in two concentrated days, using PayPal funds to pay the postal service: a hundred payments out, from our account, in a single day. PayPal 's automated fraud system notices that we are much more active than before, and automatically conducts a rule-based audit.
Act 3: Welcome to the Machine
PayPal decides that air quality sensors are high-risk, and freezes $20,000 in the bank account--nearly 90 percent of our money -- stating by email that, henceforth, they will freeze 100% of all incoming payments on sales, until we have accumulated $1,000,000 of frozen reserve. Then they will generously share a fraction of revenues beyond the first million. Yes, you read that right. Now, this initially smelled like either machine classification gone awry, or spam; the entire decision notification was email from an account with no reply option. After all, the numbers are absurd. One hundred percent of all revenue, frozen? One million dollars minimum balance? This for a company eons away from ever seeing such amounts of money. And of course, for a small company, having all your capital frozen precisely when you want to spend money to build and ship is much more than an inconvenience; it's a death sentence. Repeated calls to PayPal did result in eventual human contact. When we asked, why a million bucks, the human said that this was PayPal shorthand for infinity. Now, 99 percent of the readers have never imagined that a company such as PayPal could (a) choose to lock infinite funds, or (b) have the right to legally do this. But take a look- do a few Internet searches, and you will be stunned with the number of such stories, all starting with protection from chargebacks, then metastasizing to levels of indemnity that only computer programs can dream up. Automated rules governing electronic funds are breathtakingly easy to invent and implement, without every considering the innocents that will be caught in the resulting Big Data dragnet. In fact, many successful Kickstarters have been destroyed just like this, because PayPal locked up their money and they could never build the intended artifacts. The stories are compelling, because human beings are being disempowered by automated rule-making machines that are, in turn, entirely heartless.
Act 4: The New Turing Test: is PayPal's Brain Human or Non-Human?
We decided to follow the only rational path available: we immediately switched all payment processing to a PayPal competitor, as any rational business would. Next, we worked on other avenues of funding and continued manufacturing, as if the $20,000 at PayPal had been lost to Internet thievery. Last week we shipped nearly all the locked PayPal orders, then emailed PayPal to unlock our funds. Their first response was telling: an email parsing system decided that our email request meant we wanted to change our business address, and helpfully supplied weblinks for changing administrative options. After all, our email said we had "concerns we wanted to address." Shame on us for using the word address in our mail text. Mediocrity in first response, through automation, no longer costs a company anything when the corporation, like PayPal, is sticky. Customers cannot afford to lose loyalty; they simply become frustrated, and carry on. Calling PayPal repeatedly finally routed us to a human, and when asked explicitly, twice, if any human beings read the emails we send, Sam at PayPal told us, no, the system is entirely automated. No human reads our business queries, ever. When asked how we unlock our frozen assets, she had two answers. First: you don't. The system decides when to unlock it, it's not up to anyone. Second: you won't. When you close your account, then once you have no outstanding charges, you get the money back after a lockout period.
Act 5: Verdict
PayPal clearly includes living, breathing humans. We spoke with one. But the role of automated classification and rule-based decision systems has disempowered the role of human thought. Employees have become automatons, able only to explain what decision-making power they no longer retain. Will PayPal suffer as a result? Not likely, because volume blunts the impact of poor automation when customers have little choice. You do not choose a supermarket because its humans are more pleasant than slow, automated checkout lines. You do not avoid PayPal because of apparently fringe concerns about computers destroying business prospects through the automatic application of rules. But as humans in organizations are disempowered by Big Data, precisely because computers make sense of big data far more readily than we can, then we fall prey to the fundamental inhumanity depicted in the movie Brazil, when the intelligence of machines outclasses the intelligence of humans, no matter the underlying truth. Is this really a future we wish for our children?