Breaking through the ML/AI computational barrier

Abstract

It is a surprise to many, but the frenetic activity spawned in ML and AI over the past decade has not been driven by theoretical advances. The fundamental underpinnings of current ML/AI approaches were published by Arthur Samuelson in 1952, and while there have certainly been improvements, the approach is basically the same. What has instead driven the activity has been available computational power. In 1952, computers would require months to evaluate a simple network. Over the last couple of decades, computational power has reached the point where interesting networks can be evaluated in useful time. “Useful” time is an interesting concept, but it is clear from the number of startups (and failed startups) striving to develop higher-performance lower-power AI accelerators that VCs and entrepreneurs believe that real future for AI involves achieving another 10- 100x acceleration. Whether that speedup is possible is an open question, but there are some definitive statements that that can be made on the question: - The problems that must be solved to achieve this speed up are not new problems. “Many have tried; none have succeeded.” - The solution, if there is one, will involve parallelism. - The solution, if there is one, will not be the development of a new, “magical” parallel hardware architecture. - Following the lessons of Linpack versus LAPACK, data reuse will be a key part of the solution. - Following lessons garnered from the vector world, compilermanaged memory (i.e. registers) will almost certainly be a key part of the solution. This talk will discuss these points and what they suggest about a possible solution to this problem. Not surprisingly, compilers will be a critical tenet of the solution.