Cosma Shalizi: Machine Learning: Data, Models, Intelligence: Weekend Reading
"The 'big data' point... huge opportunity... to really expand the data.... The 'machine learning' point... a tremendous opportunity to use more flexible models, which do a better job of capturing... reality. The 'AI' point is that artificial intelligence is the technology of the future, and always will be...": Cosma Shalizi: The Rise of Intelligent Economies and the Work of the IMF: "We've been asked to talk about AI and machine learning.... I do understand a bit about how you economists work, and it seems to me that there are three important points to make: a point about data, a point about models, and a point about intelligence. The... an opportunity, the second... an opportunity and a clarification, and the third... a clarification and a criticism—so you can tell I'm an academic by taking the privilege of ending on a note of skepticism and critique, rather than being inspirational...
Official statistics... are the highest-quality statistics, but they're also slow, noisy, and imperfectly aligned with your models. There hasn't been much to be done about that for most of the life of the Fund, though, because what was your alternative? What "big data" can offer is the possibility of a huge number of noisy, imperfect measures... lots of noise, but having a great many noisy measurements could give you a lot more information. It's true that basically none of them would be well-aligned with the theoretical variables in macro models, but there are well-established statistical techniques for using lots of imperfect proxies to track a latent, theoretical variable.... Official statistics, slow and imperfect as they are, will always be more reliable and better aligned to your models. But, wearing my statistician hat, my advice to economists here is to get more information, and this is one of the biggest ways you can expand your information set...
The second point... machine learning.... 90% of machine learning is a rebranding of nonparametric regression.... The data will never completely speak for itself, you will always need to bring some assumptions to draw inferences. But it's now possible to make those assumptions vastly weaker, and to let the data say a lot more. Maybe everything will turn out to be nice and linear, but even if that's so, wouldn't it be nice to know that, rather than to just hope?... More flexible models... make... it easier to "over-fit" the data, to create a really complicated model which basically memorizes every little accident and even error in what it was trained on. It may not, when you examine it, look like it's just memorizing, it may seem to give an "explanation" for every little wiggle.... The way to guard against this, and to make sure your model, or the person selling you their model, isn't just BS-ing is to check that it can actually predict out-of-sample, on data it didn't get to see during fitting. This sort of cross-validation has become second nature for (honest and competent) machine learning practitioners. This is also where lots of ML projects die.... Cross-validation should become second nature for economists, and you should be very suspicious of anyone offering you models who can't tell you about their out-of-sample performance....
The third point.... Where we have seen breakthroughs is in the results of applying huge quantities of data to flexible models to do very particular tasks in very particular environments. The systems we get from this are really good at that, but really fragile, in ways that don't mesh well with our intuition about human beings or even other animals. One of the great illustrations of this are what are called "adversarial examples", where you can take an image that a state-of-the-art classifier thinks is, say, a dog, and by tweaking it in tiny ways which are imperceptible to humans, you can make the classifier convinced it's, say, a car....
If we have to talk about our learning machines psychologically, try not to describe them as automating thought or (conscious) intelligence, but rather as automating unconscious perception or reflex action. What's now called "deep learning" used to be called "perceptrons", and it was very much about trying to do the same sort of thing that low-level perception in animals does, extracting features from the environment which work in that environment to make a behaviorally-relevant classification or prediction or immediate action. This is... what a huge amount of our brains are doing... basically inaccessible to consciousness... fast, automatic, and tuned to very, very particular features of the environment. Our current systems are like this, but even more finely tuned to narrow goals and contexts. This is why the have such alien failure-modes....
One reason these genuinely-impressive and often-useful performances don't indicate human competences is that these systems work in very alien ways. So far as we can tell4, there's little or nothing in them that corresponds to the kind of explicit, articulate understanding human intelligence achieves through language and conscious thought. There's even very little in them of the un-conscious, in-articulate but abstract, compositional, combinatorial understanding we (and other animals) show in manipulating our environment, in planning, in social interaction, and in the structure of language.... Lots of people will sell their learning machines as though they were real AI, with human-style competences, and this will lead to a lot of mischief and (perhaps unintentional) fraud, as the machines get deployed in circumstances where their performance just won't be anything like what's intended...
#shouldread
#riseoftherobots
#weekendreading
