Most of the current discussion around running AI at the edge seems to focus on a simple question: how much of a large reasoning model can we squeeze onto a smaller computer? I’m beginning to think this may be the wrong question.

Two Kinds of Decisions

For autonomous systems — robots, drones, vehicles, distributed sensors — there are really two very different kinds of decisions to make.

At the higher level, a system needs to reason about a mission:

• Where should we go?
• What are we trying to accomplish?
• How should we react to a changing situation?
• How should multiple assets cooperate?

This looks a lot like what we now call System 2 reasoning: relatively slow, deliberate, expensive, and capable of considering a large amount of context. There is no obvious reason every edge device needs to do this independently.

A swarm or fleet could have a higher-level controller running a capable reasoning model. It could maintain the larger picture, plan missions, coordinate assets, and periodically communicate intentions and constraints to individual nodes.

Fast Judgment at the Edge

But an autonomous device also has another problem. Every fraction of a second it encounters small situations requiring judgment:

• Something appeared in front of me. Is it relevant?
• The terrain changed. Does this matter?
• The perception system produced several detections. Which one deserves attention?
• The current situation differs slightly from what was expected. Should I continue, slow down, investigate, or ask for help?

These aren’t necessarily planning problems. They are closer to System 1 cognition: fast judgment based on a relatively small amount of current context. That distinction could be important for edge AI.

Instead of trying to put a miniature reasoning model on every device, perhaps we should be designing models specifically to act as fast, inexpensive judgment engines.

Such a model doesn’t need to know everything. It doesn’t need a huge context window or the ability to spend seconds constructing a chain of reasoning. It receives a compact representation of the current situation and evaluates it against the current intent.

The Timescale Hierarchy

And importantly, it doesn’t necessarily run at the speed of the control loop. A flight controller may operate hundreds or thousands of times per second. Perception might operate tens of times per second. A judgment model might only need to reconsider the situation once or a few times per second.

Below that layer, conventional deterministic software remains responsible for real-time control. Above it, a larger System 2 model thinks about the mission.

This creates an interesting hierarchy:

Control → Perception → Judgment → Reasoning

Each layer operates at a different timescale and does a different kind of work:

• The control system asks: How do I execute this action?
• Perception asks: What am I seeing?
• The edge System 1 model asks: What does this situation mean right now?
• The System 2 model asks: What should we be trying to accomplish next?

I find this architecture more interesting than the idea of simply shrinking large language models until they fit onto edge hardware.

It also suggests that some models which look unimpressive by today’s chatbot benchmarks might be extremely useful. An edge judgment model doesn’t need to write an essay, solve a mathematics problem, or remember a long conversation. It needs to reliably map a relatively small state description and an intention into a useful judgment — quickly, repeatedly, and with very little power.

Resilience in Disconnected Operations

Communication failures make the distinction even more interesting.

If the connection to the higher-level intelligence disappears, the edge doesn’t suddenly become unintelligent. It still has perception, its current mission intent, deterministic control, and a local judgment capability. It may not be able to rethink the entire mission, but it can continue making reasonable local decisions until higher-level reasoning becomes available again.

Biological systems seem to have discovered something similar. We don’t consciously reason through every sensory observation or movement. A great deal of behavior happens through fast learned judgment, while expensive deliberate reasoning is invoked much less frequently.

Perhaps distributed autonomous systems should work the same way.

The Inverse Question

There is a lot of work right now on making powerful models small enough for edge devices. I wonder whether an equally interesting research direction is almost the inverse:

What is the smallest model that can provide useful System 1 judgment, if we deliberately don’t ask it to reason?

That might turn out to be a much more natural role for language and multimodal models at the edge.