Economic

First and most commonly pointed out, agent AIs are more economically competitive as they can replace tool AIs (as in the case of YouTube upgrading from next-video prediction to REINFORCE⁴) or ‘humans in the loop’.⁵ In any sort of process, Amdahl’s law notes that as steps get optimized, the optimization does less and less as the output becomes dominated by the slowest step—if a step only takes 10% of the time or resources, then even infinite optimization of that step down to zero time/resources means that the output will increase by no more than 10%. So if a human overseeing a, say, high-frequency trading (HFT) algorithm, accounts for 50% of the latency in decisions, then the HFT algorithm will never run more than twice as fast as it does now, which is a crippling disadvantage. (Hence, the Knight Capital debacle is not too surprising—no profitable HFT firm could afford to put too many humans into its loops, so when something does go wrong, it can be difficult for humans to figure out the problem & intervene before the losses mount.) As the AI gets better, the gain from replacing the human increases greatly, and may well justify replacing them with an AI inferior in many other respects but superior in some key aspect like cost or speed. This could also apply to error rates—in airline accidents, human error now causes the overwhelming majority of accidents due to their presence as overseers of the autopilots and it’s unclear that a human pilot represents a net safety gain; and in ‘advanced chess’, grandmasters initially chose most moves and used the chess AI for checking for tactical errors and blunders, which transitioned through the late ‘90s and early ’00s to human players (not even grandmasters) turning over most playing to the chess AI but contributing a great deal of win performance by picking & choosing which of several AI-suggested moves to use, but as the chess AIs improved, at some point around 2007_18ya victories increasingly came from the humans making mistakes which the opposing chess AI could exploit, even mistakes as trivial as ’misclicks’ (on the computer screen), and now in advanced chess, human contribution has decreased to largely preparing the chess AIs’ opening books & looking for novel opening moves which their chess AI can be better prepared for.

At some point, there is not much point to keeping the human in the loop at all since they have little ability to check the AI choices and become ‘deskilled’ (think drivers following GPS directions), correcting less than they screw up and demonstrating that toolness is no guarantee of safety nor responsible use. (Hence the old joke: “the factory of the future will be run by a man and a dog; the dog will be there to keep the man away from the factory controls.”) For a successful autonomous program, just keeping up with growth alone makes it difficult to keep humans in the loop; the US drone warfare program has become such a central tool of US warfare that the US Air Force finds it extremely difficult to hire & retain enough human pilots overseeing its drones, and there are indications that operational pressures are slowly eroding the human control & turning them into rubberstamps, and for all its protestations that it would always keep a human in the decision-making loop, the Pentagon is, unsurprisingly, inevitably, sliding towards fully autonomous drone warfare as the next technological step to maintain military superiority over Russia & China. (See “Meet The New Mavericks: An Inside Look At America’s Drone Training Program”; “Future is assured for death-dealing, life-saving drones”; “Sam Altman’s Manifest Destiny”; “The Pentagon’s ‘Terminator Conundrum’: Robots That Could Kill on Their Own”; “Attack of the Killer Robots”. Despite fervent asseveration that the US military would never use fully autonomous drones, within a few years, by 2019, Pentagon whitepapers had begun to walk that back and talk about autonomous weapons that were merely auditable post hoc and laying out AI ethics principles like being “equitable”.)

Fundamentally, autonomous agent AIs are what we and the free market want; everything else is a surrogate or irrelevant loss function. We don’t want low log-loss error on ImageNet, we want to refind a particular personal photo; we don’t want excellent advice on which stock to buy for a few microseconds, we want a money pump spitting cash at us; we don’t want a drone to tell us where Osama bin Laden was an hour ago (but not now), we want to have killed him on sight; we don’t want good advice from Google Maps about what route to drive to our destination, we want to be at our destination without doing any driving etc. Idiosyncratic situations, legal regulation, fears of tail risks from very bad situations, worries about correlated or systematic failures (like hacking a drone fleet), and so on may slow or stop the adoption of Agent AIs—but the pressure will always be there.

So for this reason alone, we expect to see Agent AIs to systematically be preferred over Tool AIs unless they’re considerably worse.

Intelligence

Why will people choose agents? Agent AIs will be chosen over Tool AIs because agents are what users want, lack of agency is something that will be penalized in competitive scenarios such as free markets or military uses, and because people will differ on preferences and some will inevitably choose to use agents.

More importantly, in addition to those reasons, it is probable that, because everything is a decision problem where agency is useful, the best Tool AI’s performance/intelligence will be equal to or worse than the best Agent AI, probably worse, and possibly much worse. Bostrom notes that “Such ‘creative’ [dangerous] plans come into view when the [Tool AI] software’s cognitive abilities reach a sufficiently high level.” We might reverse this to say that to reach a Tool AI of sufficiently high level, we must put such creativity in view. (A linear model may be extremely safe & predictable, but it would be hopeless to expect everyone to use them instead of neural networks.)

An Agent AI clearly benefits from being a better Tool AI, so it can better understand its environment & inputs; but less intuitively, any Tool AI benefits from agentiness. An Agent AI has the potential, often realized in practice, to outperform any Tool AI: it can get better results with less computation, less data, less manual design, less post-processing of its outputs, on harder domains.

(Trivial proof: Agent AIs are supersets of Tool AIs—an Agent AI, by not taking any actions besides communication or random choice, can reduce itself to a Tool AI; so in cases where actions are unhelpful, it performs the same as the Tool AI, and when actions can help, it can perform better; hence, an Agent AI can always match or exceed a Tool AI. At least, assuming sufficient data that in the environments where actions are not helpful, it can learn to stop acting, and in the ones where they are, it has a distant enough horizon to pay for the exploration. Of course, you might agree with this but simply believe that intelligence-wise, Agent AIs == Tool AIs.)

Because reinforcement learning can solve all your problems, it is rarely the best solution—but every sufficiently hard problem becomes a reinforcement learning problem.

For example, not all data is created equal. Not all data points are equally valuable to learn from, require equal amounts of computation, should be treated identically, should inspire identical followup data sampling, or actions. Inference and learning can be much more efficient if the algorithm can choose how to compute on what data with which actions.

There is no hard Cartesian boundary between an algorithm & its environment such that control of the environment is irrelevant to the algorithm and vice-versa and its computation can be carried out without regard to the environment—there are simply many layers between the core of the algorithm and the furthest part of the environment, and the more layers that the algorithm can model & control, the more it can do. Consider Google Maps/Waze⁷. On the surface they are ‘merely’ Tool AIs which produce lists of possible routes which would optimize certain requirements; but the entire point of such Tool AIs—and all large-scale Tool AIs and research in general—is that countless drivers will act on them (what’s the point of getting driving directions if you don’t then drive?), and this will greatly change traffic patterns as drivers become appendages of the ‘Tool’ AI, potentially making driving in an area much worse by their errors or myopic per-driver optimization causing Braess’s paradox (and far from being a theoretical curiosity, GPS, Google Maps, and Waze are regularly accused of that in many places, especially Los Angeles).

This is a highly general point which can be applied on many levels. This point often arises in classical statistics/experimental design/decision theory where adaptive techniques can greatly outperform fixed-sample techniques for both inference and actions/losses: numerical integration can be improved, a sequential analysis trial testing a hypothesis can often terminate after a fraction of the equivalent fixed-sample trial’s sample size (and/or loss) while exploring multiple questions; an adaptive multi-armed bandit will have much lower regret than any non-adaptive solution, but it will also be inferentially better at estimating which arm is best and what the performance of that arm is (see the ‘best-arm problem’: Bubeck et al 2009, Audibert et al 2010, Gabillon et al 2011, Mellor2014, Jamieson & Nowak2014, Kaufmann et al 2014), and an adaptive optimal design can constant-factor (gains of 50% or more are possible compared to naive designs like even allocation; McClelland1997) minimize total variance by focusing on unexpectedly difficult-to-estimate arms (while a fixed-sample trial can be seen as ideal for when one values precise estimates of all arms equally and they have equal variance, which is usually not the case); even a Latin square or blocking or rerandomization design rather than simple randomization can be seen as reflecting this benefit (avoiding the potential for imbalance in allocation across arms by deciding in advance the sequence of ‘actions’ taken in collecting samples). Another example comes from queueing theory’s “the power of two choices”, where selecting the best of 2 possible queues to wait in rather than selecting 1 queue at random improves the expected maximum delay from ?(log n)/(log log n) to instead ?(log log n)/(log d) (and interestingly, almost all the gain comes from being able to make any choice at all, going 1 → 2—choosing from 3 or more queues adds only some constant-factor gains).

The wide variety of uses of action is a major theme in recent work in AI (specifically, deep learning/neural networks) research and increasingly key to achieving the best performance on inferential tasks as well as reinforcement learning/optimization/agent-y tasks. Although these advantages apply to most AI paradigms, because of the power and wide variety of tasks NNs get applied to, and sophisticated architectures, we can see the pervasive advantage of agentiness much more clearly than in narrower contexts like biostatistics.