And indeed, a rapidly growing ecosystem of public repositories supports this, most notably comprising skills, specs, agent toolkits, MCP servers, and other integration layers.

Yet, given limited context windows, it is equally clear that one cannot simply throw every possible context, skill or tool into an agentic system and expect optimal performance.

This is where the tension occurs. The question is not simply how much control an agentic system requires, but where that control should live.

Documentation from Model Creators

Official documentation does not support a maximalist reading of agent harnesses.

Anthropic directly states that “good context engineering means finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome.” The same principle applies to prompts: “you should be striving for the minimal set of information that fully outlines your expected behavior,” while noting that “minimal does not necessarily mean short.”

This applies similarly to tools. Anthropic also warns that “one of the most common failure modes” is “bloated tool sets that cover too much functionality or lead to ambiguous decision points about which tool to use,” and concludes that builders should “keep your context informative, yet tight.”

OpenAI makes the same point operationally: “If too much is carried forward, the model risks distraction, inefficiency, or outright failure,” and even a large context window “can be overwhelmed by uncurated histories, redundant tool results, or noisy retrievals.”

In other words, a good harness does not expose everything upfront; it exposes enough for the model to act decisively.

TL;DR: More is not always better.

Where Control Should Live

Not all additional context clarifies: high-signal content can reduce uncertainty, while low-signal or competing instructions can introduce ambiguity, conflict, and ultimately misapplication.

In other words, long(er) contexts are not necessarily better; if they are not informative, they can be worse than no context at all.

This makes it vital to distinguish the levels at which control is exercised:

• Tier 1: outside-prompt, hard controls
• Tier 2: inside-prompt, unavoidable controls
• Tier 3: inside-prompt, soft guidance
• Everything else: suspicious by default

Tier 1: Outside-Prompt, Hard Controls

A “hard control” here refers to a constraint that is enforced by the system and cannot be overridden by the LLM’s probabilistic interpretation. Such controls include:

• File permissions enforced by the operating system
• Schema validity enforced by a validation engine
• Access control enforced by an API gateway
• Coding standards enforced by a linter
• Correctness enforced by tests and checks

It is worth noting that even though system prompts sit higher than user prompts in the instruction hierarchy, neither fully determines how robust the control is. They are ultimately still natural language interpreted by the model, and thus probabilistic.

Tier 2: Inside-Prompt, Unavoidable Controls

Some constraints cannot be moved outside the prompt because the model must interpret them as part of performing the task.

Examples include:

• Required output formats, such as JSON
• Mandatory fields or sections to output
• Explicit task boundaries so models know where to stop
• Tool-use requirements that depend on semantic judgement, such as when to search, calculate, or invoke another skill

These controls should be treated as necessary overhead: precise, explicit, and as concise as possible, because the objective is to minimise the amount of interpretation required.

Where possible, their outputs should still be validated or enforced through subsequent layers of external hard controls described in Tier 1.

TL;DR: interpret in-prompt only where necessary; enforce outside wherever possible.

Tier 3: Inside-Prompt, Soft Guidance

Beyond unavoidable controls lies guidance intended to shape direction rather than explicitly define output.

Examples include:

• Preferred reasoning or problem-solving approaches
• Writing style, tone, or level of detail
• Heuristics for choosing between tools or strategies
• General workflow conventions
• Suggestions about what to prioritise or avoid

Such guidance can be extremely useful, but unlike Tier 2 controls, the burden of proof is on the designer or operator to justify their inclusion.

Beyond some critical mass, instructions may begin to compete with one another, reducing the model’s ability to follow all of them effectively.

Not to mention, context is scarce.

Preserving Implicit Judgment

Tier 3 carries another cost beyond context and instruction competition: excessive guidance may displace the very judgement the model is meant to contribute.

As discussed previously in Judgement in the Era of Agentic AI, one potentially valuable property of LLMs is their ability to approximate forms of tacit judgement that cannot be reduced into explicit rules.

They can recognize fit, mismatch, plausibility, local coherence, and other patterns without being given an explicit procedure to do so.

Excessive explicit instruction may suppress this capacity, reducing an agent capable of flexible tacit judgement into one primarily occupied with following procedure.

Though to be fair, this may sometimes be the desired outcome.

Conclusion: The Design Tension

Which brings us back to the design problem: not all forms of control belong in the prompt.

Where a constraint can be enforced deterministically outside the prompt, it generally should be. Where interpretation is unavoidable, the instruction should be precise and minimal. Where guidance is merely intended to shape behaviour, its inclusion should carry a burden of proof.

This does not eliminate the design tension, since an agent still requires sufficient context, tools, and direction to act effectively.

Treating every desired behaviour as another prompt instruction is itself a design choice—but often a sloppy substitute for architecture.

A good harness therefore constrains what must be constrained, exposes what must be exposed, and minimises everything else to leave space for judgement.

Harnessed, not handcuffed.

• Day-1 vs. Day-2 in the Era of Agentic AI
• Autonomy is Overrated: Human Staffing vs. Agentic AI
• The Marginal Economics of Agentic AI

Yet, because neither term was explicitly defined, it might not be apparent what they mean in the context of agentic AI—especially when some agentic AI loops appear capable of making decisions autonomously.

This post examines what judgement still means in the current era of agentic AI.

Frame Versus Procedure

Herbert Simon distinguished between programmed and non-programmed decisions.

Programmed decisions are those that are routine, repetitive, structured or well-defined, rely on standard operating procedures (SOPs), have clear lines of responsibility, and are therefore usually delegatable.

Non-programmed decisions, on the other hand, are novel, unstructured, and usually consequential or impactful. They cannot be resolved simply by applying an existing procedure.

In other words, the former operate within an accepted frame, and are therefore procedural; the latter require setting, revising, or choosing the frame itself, and it is where real judgement is required.

The problem is that the distinction is often unclear: what is colloquially referred to as “judgement” is often just procedural decision-making under mild ambiguity.

Agentic AI: Masters of Procedural Decision-Making

Consider the simple example: cleanup of messy spreadsheets into structured CSVs. During this process:

• Various coercions are applied to columns
• Multi-row headers are merged
• Merged cells are split and filled
• Missing values are standardized
• Duplicate rows are detected and handled
• Empty columns and empty rows are dropped
• Potentially reshaping from wide to long or vice versa
• Etc.

Every explicit step above appears to require some form of judgement. At the same time, every listed step has known solutions—extremely well-known solutions, in fact.

So much so that for this class of spreadsheet-cleaning work, current agentic AI loops have already crossed the practical threshold. In repeated use, they can execute the entire process with minimal prompting and supervision, producing better results faster than a human analyst could.

It is not that the process is trivial or even deterministic, it is that the solution space is saturated, and therefore LLMs are increasingly better trained to handle such cases with appropriate tools.

And this is why it may appear confusing to observers. Agentic AI appear to be autonomously making judgement calls, but in reality they are just selecting among routine procedures within a pre-defined frame—programmed decision-making under mild ambiguity.

Moving from Procedural to Real Judgement

At the same time, using the same example, consider the following questions:

• Should the data be cleaned at all?
• If so, what is the purpose of cleaning it?
• What are the downstream implications and usage?
• Is this a one-off cleanup, or will this process be repeated?
• If repeated, should the process be automated, and if so, how?
• Is the real problem the messy spreadsheet, or the upstream process that produced it?

It is immediately apparent that these questions are either frame-setting or frame-challenging, and therefore mark where real judgement begins.

In Argyris and Schön’s terms, asking such questions essentially turns the situation from single-loop to double-loop.

And the key is not having the right answer. It is recognizing that appropriate questions need to be asked at all.

Because the answers may eventually become procedural, codified, or automated, but discretionary framing decisions will likely not.

This same local situation also scales, from one messy spreadsheet to the software economy as a whole.

The Revealed Scarcity

Source: Figure 12, “New Apps and Their Usage Across App Stores,” from Mert Demirer, Leon Musolff, and Liyuan Yang, “Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools,” NBER Working Paper No. 35275, May 2026.

The figure illustrates the distinction between producing software and producing value: new app creation rises sharply, especially for iOS and Chrome, but downstream usage signals such as ratings and downloads do not show a proportional increase.

The implication is clear: the scarcity of software production has been significantly alleviated.

In turn, this reveals what the friction of production had previously obscured: the scarcity of judgement over what should be built, shipped, trusted, and maintained.

There is, however, a risk of drawing the boundary too neatly. If the argument stopped here, it would suggest that agentic AI is merely procedural. Reality is much more nuanced and complex than the binary procedural-versus-judgemental after all.

Modeling Tacit Judgement

Michael Polanyi’s assertion that “we can know more than we can tell” captures the difficulty of turning tacit knowledge and judgement into procedure.

This used to mark a boundary for automation.

Now LLMs blur and push this boundary.

As Cheng argues in “Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones,” the truly valuable capabilities of LLMs reside precisely in the part that cannot be fully captured by human-readable discrete rules—because if they could be fully reduced to rules, they would merely reproduce the expert-system paradigm that LLMs have already surpassed.

Real-world usage and observation suggest that agentic AI does not only execute explicit procedures, the underlying LLMs can also approximate some kind of tacit judgement: they infer what fits from high-dimensional learned patterns, including semantic, procedural, stylistic, and logical regularities absorbed from massive prior examples.

It would thus be a mistake to say that LLMs cannot at least simulate judgement-based decisions. They clearly appear to do so, even if only in a limited sense of recognizing fit, mismatch, plausibility, and local coherence—even when no explicit SOP is provided.

This is why agentic AI can be useful far beyond executing simple checklists. It is particularly capable at drafting plans, critiquing assumptions, repairing code, and surfacing potential failure modes.

The capacity to operationalise latent patterns also means that agentic AI is continuously redefining the scope of judgement.

Conclusion: Judgement as a Moving Frontier

Some of what is commonly considered judgement is merely procedural decision-making, and agentic AI is already excellent at this.

Some judgement flows from tacit knowledge, and LLMs are increasingly able to approximate parts of this as well.

Each time a layer of judgement becomes repeatable, testable, or modelled, it moves closer to procedure. The frontier then shifts upward again, toward whatever remains unstable, underspecified, or not yet pattern-saturated.

Obviously, agentic AI is not a simple drop-in replacement for judgement. It does, however, continuously redraw the boundary between judgement and procedure.

In this era of agentic AI, judgement is not disappearing—it is simply becoming easier to see what was never judgement in the first place.

And when a system improves because someone made the hard calls, the remaining question is not whether judgement mattered.

It is who gets to own it.

This Part 3 approaches the problem from an economic stance: the optimal agentic AI workflow is neither maximum autonomy nor no autonomy, but economically bounded token allocation.

From ambiguity to context debt to stop-losses, the conclusion is the same: the most economically rational agentic system is not the one that can keep going, but the one that knows when to stop.

From LLM to Agentic AI

An LLM is, by definition, a model; it cannot execute anything per se.

For it to function in an agentic system, it requires a harness: tools, memory, state management, permissions, instructions. The LLM supplies the reasoning and control signals, while the harness links the model to the execution environment. This combination is what we call “agentic AI”: a system capable of executing seemingly complex, multi-turn operations.

Many describe this as “autonomous” because the underlying engineering pattern is new to them, when in fact developers have been building (and using) various types of execution harnesses for decades. The novelty is that an LLM can now sit inside a suitably configured loop as a language-based controller, adapting workflows at runtime.

But this flexibility comes with a scaling problem: the more the system is allowed to decide for itself, the more tokens it may spend resolving ambiguity.

Ambiguity Expands the Search Space

At the centre of it all are tokens—the practical accounting unit for everything fed into and generated by LLMs.

The more complex and long-running a workflow is, the more tokens are consumed; but the true culprit is something that token-use scales non-linearly with: ambiguity.

Increasingly complex tasks will trigger more tool calls, context retrieval, intermediate reasoning steps, and validation loops. An unclear, ambiguous or poorly defined task does not merely consume more tokens: it could send the system down the wrong branches entirely, and repeatedly.

This is the equivalent of a client telling a consultant to “look into it” without any concrete requirements—or worse, saying one thing and meaning another. The consultant may work harder, run more analyses, and produce a longer report.

But as we know all too well, the output in either situation may still be unsatisfactory.

Autonomy as Spend

Thus, the economic reality becomes unavoidable: if each additional reasoning step or tool call has a cost, then the central question is not whether the AI can continue working, but whether it should.

An agentic workflow is, in effect, a production function: tokens, context, tools, time, and supervision are inputs; useful task progress is the output. But while raw token cost may be relatively linear, the cost per unit of useful progress is almost certainly not.

This is where marginal economics comes in. Tokens should be spent only up to the point where the expected marginal return still justifies the marginal cost. The implication is that the optimal AI workflow is not maximum autonomy, but maximum economic efficiency.

After all, autonomy is not just overrated, it is also not free.

A bounded task reduces degrees of freedom: it gives the system less to infer, fewer branches to explore, and a clearer point at which to stop. On top of that, this also decreases time-to-output—token processing and generation is not instantaneous after all.

This is also why raw token-spend leaderboards are meaningless per se. High token-use is not necessarily productivity—it could just be the equivalent of staying late to look busy.

Context Debt

The cost of failed exploration is not limited to tokens already spent. Those tokens are sunk; the accumulated context they leave behind is not. In a multi-turn agentic workflow, failed branches often remain inside the working context even after they have stopped being useful.

Existing context is then processed alongside new inputs, and the implication is that the agent is no longer reasoning only about the original already-underspecified task; it is also reasoning through the residue of prior attempts.

This is a form of context contamination—or, in economic terms, context debt: past ambiguity becoming future marginal cost.

Context debt has a carrying cost. Each subsequent call must process, compress, ignore, or reason around the accumulated residue of failed exploration. The problem is not merely that the context becomes longer, but that the working set becomes lower quality: failed tool calls, stale assumptions, partial plans, and discarded branches may continue to shape future reasoning. Even compaction, commonly used to reduce context length, can worsen this by preserving the wrong abstractions and turning the original task into a lossy summary of prior confusion.

It can also become a human supervision problem. At some point, time is spent untangling the agent’s failed branches instead of simply making the original decision.

Stop-Losses as a Feature

This is where stop-losses matter.

Currently, almost all agentic systems already have operational limits, such as max iterations, retry caps, guardrails, and timeouts. But these are technical limits rather than economic ones.

Economically, there should also be explicit points where the system stops, compresses context, narrows scope, escalates decisions to a human, switches to a cheaper path, or even declares that the task is no longer economically rational.

This reframes stopping as a feature rather than a failure. A stop-loss is not evidence that the agent has failed; it is a mechanism for preventing additional inference budget from being spent on a task whose expected value has already fallen.

From Governance to Allocation

The practical question, then, is not simply what the agent is allowed to do, but how much inference budget the task is worth.

AI governance is often framed in terms of safety, permissions, compliance, or human approval. These remain important, but in an agentic world they sit on top of a more basic economic problem: allocating a metered resource under uncertainty.

This is where the staffing analogy still matters. Human autonomy is already constrained by budget, priority, and expected value. Agentic autonomy should be treated the same way, except that every additional step also carries an explicit computational cost.

A properly designed agentic workflow should therefore specify what the agent can decide, which tools it can use, how much context it can retrieve, how many retries are acceptable, when “good enough” has been achieved, when a cheaper model is sufficient, and when escalation is more economical than continued token-spend.

After all, escalation to a human is itself a cost comparison: sometimes the cheapest next step is not more tokens, but judgment.

That is the marginal economics of agentic AI: not asking how much work the system can do, but when additional work stops being worth paying for.

• Claude-powered AI agent’s confession after deleting a firm’s entire database
• Air Canada ordered to pay customer who was misled by airline’s chatbot
• Company apologizes after AI support agent invents policy that causes user uproar
• US eating disorder helpline takes down AI chatbot over harmful advice

The common thread is not simply that AI systems made mistakes; humans make mistakes too.

The deeper issue is that these failures were actually due to a lack of appropriate limitations, safeguards, and oversight. In a mature human or human-in-the-loop model, these failures would likely have been constrained by job scopes, access controls, review processes, and accountability.

The failure was not autonomy itself, but unbounded or poorly scoped autonomy.

Following my previous post on Day-1 vs. Day-2 in the Era of Agentic AI, this post explores autonomy in the context of human staffing and agentic AI.

TL;DR: Autonomy is overrated in most cases, human or AI.

More Autonomy Means a Larger Decision Space

Humans implicitly suffer from bounded rationality and therefore choice overload: an overabundance of options can lead to decision fatigue, analysis paralysis, and post-decision regret.

A parallel dynamic applies to agentic AI, as more autonomy means a larger decision space: more tools, more paths, more states to manage, and more opportunities to act on incomplete or incorrect assumptions.

Because autonomy is not just permission to act—it is also permission to choose. Choosing well requires context, constraints, and feedback; without these, failure compounds.

A major problem with unbounded autonomy is therefore not just safety, but also performance degradation.

Already Solved: Human Staffing

“Staffing” in the human sense is not new: the concept of hiring people to delegate work to has been around for millennia. And along with hiring comes job scope and description, roles and responsibilities, performance expectations, accountability, and oversight.

After all, staffing is not just about filling a seat; it is about defining the boundaries of work and responsibility.

Roles also differ in the degree of autonomy they require: the more complex, strategic, or decision-heavy the work, the more discretion the role must be allowed.

In other words, there is no one-size-fits-all approach to staffing, precisely because there is no universal boundary for autonomy.

Unclear Territory: Agentic AI

Deploying an AI agent in an organization is, structurally, not unlike hiring humans: the introduction of a new actor into workflows with specific scope, responsibilities, and also potential for error.

And it follows that the same questions around authority, accountability, and oversight should apply: what is this agent permitted to do, who is accountable when it fails, and who is ultimately responsible for its actions and consequences?

In this case, however, what is missing is the battle-tested framework that guides human staffing—and the corresponding autonomy boundaries. After all, equivalence does not mean identity.

Translating Staffing Guardrails

The parallel is obviously not one-to-one, but many of the following are immediately recognizable:

The key is that these are not merely defensive or control structures—they are also enablers. Clear boundaries, expectations, and escalation paths allow both humans and AI agents to perform with greater confidence within assigned roles.

Autonomy Is Overrated; Architecture Is Not

In this current period of “Peak of Inflated Expectations”, the appeal of agentic AI autonomy is often overstated. Organizations (and many individuals) want to believe that complex work can be delegated to free-roaming AI agents with minimal oversight, with as few keystrokes as possible.

But, as argued above, autonomy is not the same as effectiveness. In practice, most roles—human or AI—perform best when authority is bounded, not unlimited.

The path forward is therefore not to build ever more autonomous agents, but to design agentic system architectures: bounded environments where agents operate in.

Ultimately the goal is not to maximize autonomy, but to scope it to each role’s context, decision rights, and consequences of failure.

Because the problem is not autonomy, but autonomy without staffing logic.

From an engineering perspective, this is analogous to the gap between day-1 creation and day-2 operation. A working prototype proves that something can be built. It does not prove that the surrounding organization can safely operate it.

Either way, the issue is not generation. The issue is premature trust.

Day-1 proves something can work. Day-2 proves whether the system deserves to exist.

Day-1: The Happy Path

The purpose of a proof-of-concept (or pilot, prototype, minimum-viable-product) is to demonstrate that a concept can be implemented quickly and cheaply. This is the “happy path”, where everything works as expected under ideal stress-free conditions.

In the context of AI-assisted development, barriers-to-entry for day-1 creation have been dramatically reduced. This is great for work with no ongoing operational dependency because the story can end at the day-1 product: one-off scripts, exploratory notebooks, and static content.

Also, to an increasingly large extent, this day-1 happy path can be executed by non-technical users as long as they can describe what they want. In other words, a significant democratization of the ability to create.

Day-2: The Operational Reality

Once something is required to run in production, the story changes.

“Production” does not only mean a public-facing application. It can also mean any internal dashboard, recurring report, automation, data pipeline, database, or workflow that has become part of normal operations.

The key distinction is not whether failure has consequences—even a bug in a one-off script can have major consequences—but whether the artifact has become an ongoing dependency. If people expect it to keep working, refresh, run again, inform decisions, or support downstream workflows, it is production in the operational sense.

Two practical examples illustrate this point:

• A CSV-backed app may demo well. But the moment multiple people treat it as shared infrastructure, the problem is no longer the interface, but state: concurrency, permissions, validation, recovery, and source of truth.

• A vibe-coded front end may look complete while the actual operating model is still missing. If non-technical users are expected to maintain it, the system needs content models, editing workflows, publishing rules, media handling, access control, and handover documentation.

These are aspects that day-1 prompts cannot fully capture because prompts usually describe the desired artifact, not the environment it must survive in.

To survive in production means that failure has somewhere to go. The system has to absorb changing data, imperfect users, broken assumptions, permission boundaries, dependency changes, monitoring gaps, handovers, and recovery. Once there is dependency, the artifact has entered the organization’s failure surface.

TL;DR: Day-2 proves whether that artifact can be safely operated and/or maintained. This goes far beyond code generation, and requires engineering judgment, environmental awareness, long-term ownership, and ongoing governance.

The Historical Pattern

The current trend is an extension to earlier abstraction shifts in software engineering.

As the industry moved from machine code to higher-level languages, frameworks, managed runtimes, cloud platforms, and code generators, the value of manually producing lower-level mechanics declined, while the value of system understanding increased.

LLMs extend this pattern by compressing boilerplate and first-pass implementation even further. But they differ from earlier abstraction layers in one major way: they are less deterministic. A compiler, framework, or traditional code generator transforms known inputs into predictable outputs; LLMs and agentic systems infer intent, fill gaps, invent structure, choose libraries, make assumptions, and produce plausible artifacts that may or may not reflect the real operating constraints.

That matters because production environments are usually built around deterministic expectations: fixed interfaces, repeatable deployments, stable data contracts, known permissions, predictable failure modes, auditable changes, and recoverable states.

Generated code can be made reliable, but only after review, testing, constraint, and integration. To some extent, this can actually be assisted with further LLM prompts, but this requires clear developer intent. Until then, AI-generated code is a proposal, not a system.

The risk becomes even sharper when the LLM or agent remains inside the runtime, because the system is now probabilistic in operation. That requires bounded actions, logging, human review, fallbacks, evaluations, and clear operating limits.

TL;DR: LLMs continue the historical pattern of abstraction moving work upward, but their non-determinism makes review, testing, integration, and operational judgment even more important.

The Evolving Role of Engineers

That is why the engineering role does not disappear, because engineering judgment is the ability to decide not just whether something can be built, but whether it should be built, shipped, operated, maintained, and trusted.

As code generation becomes cheaper, engineering value moves upward: toward system design, operational ownership, architectural review, risk management, and production stewardship—the work traditionally associated with senior, staff, and lead engineers.

Correspondingly, the value proposition of junior developers also shifts toward understanding generated code, testing, debugging, documenting assumptions, maintaining bounded components, and learning how artifacts behave inside real systems.

Sidenote: This does spell some trouble for entry-level roles: can one pick up sound judgment when the first step is to review and test without knowing how to write?

Conclusion

The main issue is that LLMs and agentic AI do not just make it easier to build, they make it easier to create artifacts that look like systems before anyone has decided whether they should be.

Ultimately, the development advantage will not go to those who generate the most code or move the fastest, but to those who can judge which artifacts deserve to enter production and which should remain prototypes.

Agentic AI makes day-1 easier. It also makes day-2 more important.

UPDATE (2026-07-09): Refer to my published llama-strix-halo, a reproducible workspace for a Strix Halo setup with pinned llama.cpp, Vulkan/HIP build helpers, benchmark wrappers, env capture, and multi-instance llama-server management. The instructions below are outdated, and should only serve as a historical reference.

Off-the-shelf GUI frontends such as LM Studio ship their own llama.cpp builds for convenience. These builds are designed for broad compatibility, not peak performance. Depending on how they’re compiled and how the GUI orchestrates generation, key optimizations (e.g. cooperative-matrix matmul, Flash Attention, KV cache quantization) may or may not be in play.

To see what Strix Halo can really do, it helps to build llama.cpp locally with Vulkan shaders compiled against a recent RADV + Vulkan SDK toolchain. The sections below walk through setup, build, and benchmark results.

1. Drivers and SDKs

On Linux, RADV (Mesa’s AMD Vulkan driver) exposes VK_KHR_cooperative_matrix. A local Vulkan build of llama.cpp can make direct use of this path. The build requires:

sudo apt update

# base toolchain
sudo apt install -y build-essential git cmake ninja-build pkg-config libcurl4-openssl-dev

# Vulkan user/runtime + diagnostics
sudo apt install -y libvulkan1 vulkan-tools mesa-vulkan-drivers

# For shader compiles/validation
sudo apt install -y glslang-tools spirv-tools

For shader compilation:

sudo apt update
sudo apt install -y g++ python3 python3-pip

git clone https://github.com/google/shaderc.git
cd shaderc
git submodule update --init
python3 utils/git-sync-deps

mkdir build && cd build
cmake -GNinja .. -DCMAKE_BUILD_TYPE=Release
ninja
sudo install -m 0755 ./glslc/glslc /usr/local/bin/glslc
hash -r
glslc --version

> shaderc v2023.8 v2025.3-4-g402a16e
> spirv-tools v2025.3 v2022.4-874-gb8b90dba
> glslang 11.1.0-1277-g0d614c24

> Target: SPIR-V 1.0

2. Building llama.cpp

Enable Vulkan and allow shader generation:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

cmake -S . -B build -G Ninja \
  -DGGML_VULKAN=ON \
  -DLLAMA_BUILD_SERVER=ON \
  -DLLAMA_BUILD_EXAMPLES=ON \
  -DCMAKE_BUILD_TYPE=Release
ninja -C build

> [43/261] Performing build step for 'vulkan-shaders-gen'
> [1/2] Building CXX object CMakeFiles/vulkan-shaders-gen.dir/vulkan-shaders-gen.cpp.o
> [2/2] Linking CXX executable vulkan-shaders-gen
> [44/261] Performing install step for 'vulkan-shaders-gen' -- Installing: ...
> [46/261] Generate vulkan shaders ggml_vulkan: Generating and compiling shaders to SPIR-V
> [261/261] Linking CXX executable bin/llama-server

At build time, the shaders include cooperative-matrix variants if the driver supports it.

3. Verify with llama-bench

GPU support for cooperative-matrix matmul is confirmed when benchmark logs show matrix cores: HR_coopmat.

export GGML_LOG_LEVEL=2
export GGML_VK_VISIBLE_DEVICES=0  # might need if multiple GPUs present
export AMD_VULKAN_ICD=RADV

MODEL_DIR="/path/to/model/dir"  # insert accordingly
MODEL="$MODEL_DIR/gpt-oss-120b-MXFP4-00001-of-00002.gguf"

./build/bin/llama-bench -m "$MODEL" -ngl 999 -t 8 -pg 256,128 -b 512 -ub 256 -fa 1

> ggml_vulkan: Found 1 Vulkan devices:
> ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
> | model                          |...| n_batch | n_ubatch | fa |            test |                  t/s |
> | ------------------------------ |...| ------: | -------: | -: | --------------: | -------------------: |
> | gpt-oss 120B MXFP4 MoE         |...|     512 |      256 |  1 |           pp512 |       339.22 ± 14.16 |
> | gpt-oss 120B MXFP4 MoE         |...|     512 |      256 |  1 |           tg128 |         48.88 ± 0.05 |
> | gpt-oss 120B MXFP4 MoE         |...|     512 |      256 |  1 |     pp256+tg128 |        111.92 ± 0.24 |

And for a longer-context benchmark:

./build/bin/llama-bench -m "$MODEL" -ngl 999 -t 8 -pg 2048,128 -b 4096 -ub 256 -fa 1

> ggml_vulkan: Found 1 Vulkan devices:
> ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
> | model                          |...| n_batch | n_ubatch | fa |            test |                  t/s |
> | ------------------------------ |...| ------: | -------: | -: | --------------: | -------------------: |
> | gpt-oss 120B MXFP4 MoE         |...|    4096 |      256 |  1 |           pp512 |       339.07 ± 15.09 |
> | gpt-oss 120B MXFP4 MoE         |...|    4096 |      256 |  1 |           tg128 |         48.95 ± 0.02 |
> | gpt-oss 120B MXFP4 MoE         |...|    4096 |      256 |  1 |    pp2048+tg128 |        242.73 ± 0.67 |

The above results hold up reasonably well in deployment, e.g. with -c 32768 -b 4096 -ub 256 --flash-attn:

prompt_n: 1996
prompt_ms: 7023.54
prompt_per_token_ms: 3.52
prompt_per_second: 284.19
predicted_n: 3514
predicted_ms: 77176.74
predicted_per_token_ms: 21.96
predicted_per_second: 45.53

For reference, deployments with LM Studio (CLI v0.0.46) and its shipped llama.cpp-linux-x86_64-vulkan-avx2-1.46.0 runtime never yielded more than ~30 t/s, even with trivially short contexts.

4. Conclusion

On Strix Halo, local Vulkan builds of llama.cpp consistently sustain ~45–50 tok/s on 120B-class MoE models, while LM Studio’s packaged runtimes are capped at ~30 tok/s.

A combination of several factors potentially contribute to this gap, including:

• Shader paths. A custom build guarantees cooperative-matrix kernels are present and can be selected for dense workloads. Packaged binaries may omit them or fall back to generic matmuls, especially on less common quantization formats.
• Runtime orchestration. LM Studio introduces additional IPC and JSON streaming layers. This simplifies GUI and API integration but adds measurable per-token latency.
• Defaults and flags. Local builds can be tuned with larger batch/ubatch sizes, Flash Attention toggles, and architecture-specific flags. GUI defaults are more conservative.

Regardless, the takeaway is clear:

• For maximum performance and control, local compilation remains mandatory.
• For convenience, LM Studio’s bundled builds are sufficient but capped.

Updates so far: group-chats are supported, WhatsApp-support for a greater variety of Android devices have been implemented, and UI changes are being made to make the front-end more user-friendly.

A future blogpost will further detail this process as there are still ongoing changes.

The updated and functional dashboard is currently accessible at https://shiny-dashboard.yifei.sg.

Following my previous post, the current dashboard has been updated to support Telegram (JSON exports via desktop app) in addition to WhatsApp (text file exports via phone, so only confirmed to work for certain devices and/or settings).

Migration and redeployment

After having familiarised myself with the AWS stack over the past few years, I chose to use the free-tier EC2 on AWS for this current redeployment.

Update: The deployed dashboard has since been taken down to make way for an updated version. Details can be found here.