
Maze Comparison Using Laya & Jev
Background
Jev launched as a System One model, and I wanted to see how it stacks up against another System One model. Laya was the obvious one to pick.
All of my testing happens on Apple Silicon, so Laya-MLX was the natural build to run locally. That set the shape of this comparison: Jev over the official API against Laya running on my own machine, both handed the same mazes under the same rules.
Overview
Maze Decision Benchmark compares two decision models on identical maze-navigation tasks, with one metric: wall-clock time to reach the goal.
| Candidate | Model | Where it runs |
|---|---|---|
laya-mlx |
Laya 421M (ModernBERT-large), Convai Innovations, Apache-2.0 | Locally, MLX on Apple Silicon |
jev |
Jev 1.13, TypeSafe AI, closed and metered | Official TypeSafe System One API |
Both receive the same maze state and the same typed choice question, answer with a direction, and the app executes the move and repeats until the agent reaches the goal. Steps, invalid moves, shield interventions, and trajectory shape are recorded as supporting detail.
Tech Stack:
- Next.js (App Router) - Maze explorer (
/) and live benchmark view (/benchmark) - TypeScript - Engine, validator, BFS pathfinding, prompt/guard builder, benchmark runner
- SSE streaming -
POST /api/benchmark/runwith per-provider boards and real-time side-by-side replay - Python sidecar -
runtime/laya_service.py, stdlib-only service holding the checkpoint resident on loopback - Vitest - 10 test files covering engine, guard, shield, and maze verification

How a Run Works
Load maze → warmup decision (never counted) → loop:
build decision context (eligible = walkable minus where we came from)
→ 1 eligible passage? forced move (no model call)
→ 2+? call the provider → apply the shield → execute the move (1 step)
→ goal reached? stop the clock → result + full trace
Three design decisions shape everything:
- The guard. Only walkable directions are offered, and the cell the agent just came from is excluded. Corridors and dead ends become free forced moves. No request, no latency, no chance to blunder. The model is consulted only at real junctions.
- The shield (on by default). When the agent stops reaching new cells (stall limit 6), the harness substitutes the first step of a shortest path. A shielded run measures model + solver. The honest model signal is the
shieldInterventionscount. Turn it off (--no-shield/ checkbox) to measure the raw models. - Time is the metric. With the shield on, step counts cluster near optimal and stop discriminating. Wall-clock is what a user of either system actually experiences. One thing to keep in mind: Laya runs on this machine while Jev crosses the network.
5 fixed, deterministic mazes (11×11 → 31×31), frozen in data/ and re-verified by BFS on every test run. One shared prompt builder (lib/maze/context.ts) gives both providers byte-identical state, instructions, and option sets.
Results
Median of 5 runs per provider per level (MacBookPro18,1, MLX 0.32.2, checkpoint aac6fef/laya-typed-decisions-mlx).
| Level | Optimal | Laya-MLX time / steps | Jev time / steps |
|---|---|---|---|
| 1 Easy (11×11) | 21 | 0.4 s / 31 | 2.2 s / 33 |
| 2 Normal (15×15) | 40 | 1.2 s / 98 | 3.0 s / 66 |
| 3 Intermediate (21×21) | 53 | 4.7 s / 203 | 5.1 s / 109 |
| 4 Hard (25×25) | 96 | 1.5 s / 120 | 7.0 s / 190 |
| 5 Expert (31×31) | 108 | 1.2 s / 136 | 4.3 s / 172 |
| Total (sum of medians) | 9.0 s / 588 | 21.6 s / 554 |
All 50 runs solved, zero invalid moves. Honest reading: time favours local Laya (~1.3× overall, topology rather than intelligence); steps slightly favour Jev (554 vs 588, driven mostly by Level 3). Neither model is uniformly better. That is the finding.
Running It
Prerequisites: Apple Silicon Mac (MLX), Python 3.11+, Node.js 20+. One credential: a TypeSafe API key for Jev.
npm install
python3 -m venv .venv
npm run laya:install
npm run laya:download # 843 MB checkpoint, once
cp .env.example .env.local # add JEV_API_KEY
npm run laya:serve # terminal 1: leave running
npm run dev # terminal 2
Open http://localhost:3000/benchmark, pick a level, click Run level.
npm run bench -- --levels 1-5 --runs 5 # CLI, mean/median over repeats
npm run bench -- --level 1 --no-shield # raw models, no solver fallback
Key env vars: LAYA_ENDPOINT (default http://127.0.0.1:8765), JEV_API_KEY (required), JEV_MODEL (default jev-latest), BENCHMARK_STALL_LIMIT (default 6). All values stay server-side.
Verification
npm run verify # typecheck + tests + maze verification + context-window check
tsc --noEmitclean, production build clean- Vitest suite passes (10 files)
mazes:verifyre-derives solvability,optimalSteps, and junction countscontext:checkasserts every maze fits the checkpoint’s 1024-token window
Source Code
Source code: maze-hehe on Gitea.