01 / 11
Internal Discussion Prep

Technology department operating principles, in the AI-code-acceleration era.

2026-07-06 · companion to cto-operating-principles.md
01 · THE SHIFT

Closing a task has never been easier.

That's not a minor efficiency gain. It changes what "done" should mean, and what the department should be attempting. Two consequences follow directly.

1. Be more ambitious
Things genuinely impossible three years ago now need to be on the table.
If this year's plan looks like a 2023 plan, it isn't ambitious enough for what the tools now allow.
2. The role expands
"I built it, QA checks it" no longer covers the job. Building now includes building the agentic structure that tests and evaluates the work.
The report that matters: "matching went from 62% to 81%, here's what moved it," not "I shipped three features."
01 · THE SHIFT

Two things we need conviction on.

That we can build a great AI system
We didn't start knowing how. The horizontal and vertical guides exist because building reliably turned out to be harder than "plug in a model."
The conviction being built isn't "AI is valuable," we know that. It's "we know how to build with it reliably." Still being proven.
What we're actually solving
Not feature count. Opening and closing a feature is easy, for anyone. That was never the hard part.
The hard part: eval accuracy moving 10 points, or real confidence that 8 of 10 emails are handled correctly.
01 · THE SHIFT

The premise underneath all of this.

Code is increasingly written by AI, not by hand. If we agree on that, the job changes shape.

The CTO's real job shifts from writing and reviewing code…
…to deciding where the team plans deliberately versus where it moves agentically without a human in the loop, and building (or having a skill/workflow build) the mechanism that makes that boundary work in practice.
02 · EIGHT KEY AREAS

Eight areas, all new since code got this easy.

01
Eval ownership
Mandatory, not optional. A living evaluation system.
02
Cost per unit of value
Inference spend tracked against value delivered, not just in aggregate.
03
Reliability & degradation
Uptime, speed, email-to-first-draft latency.
04
Quality gates on AI-generated code
What runs after a task is marked done, before it's trusted.
05
Velocity on hard problems
The easy stuff is already easy. The gap is speed on what's hard.
06
Learning loops
How one build's lessons propagate into the next.
07
Protecting proprietary intelligence
What stays legible only to us versus handed wholesale to any LLM vendor.
08
Architecture understanding
A current, real picture of the system, not a memory of the design.
02 · AREA 01

Eval ownership is mandatory.

Not a nice-to-have. Technology ownership means being able to say, unprompted: here are the current results, here's my diagnosis of why they are what they are, here's how I'm going to move them, here's what happened when I did.

A living evaluation system, not a one-time report. If it isn't being tracked continuously, it isn't owned.
02 · AREAS 02–04

What we govern about what AI builds and runs.

02
Cost per unit of value.As usage grows, we need to know what we're spending in inference cost per unit of value delivered, not just in aggregate. Likely untracked today.
03
Reliability & degradation.Uptime, speed, and specifically latency from an inbound email to a first draft. Not urgent today, but must never silently degrade.
04
Quality gates on AI-generated code.Skills/workflows after a task is done: agentic unit tests as standard, how often human-in-the-loop testing runs, and when (if ever) external code review gets pulled in.
02 · AREAS 05–08

What's still open.

05
Velocity on hard problems.A deliberate choice to make, not a byproduct of tooling improving on its own.
06
Learning loops.Needs more discussion: how a lesson from one build reaches the next, instead of staying with whoever hit it.
07
Protecting proprietary intelligence.Not urgent, worth naming early: what should stay deterministic or split across providers versus handed wholesale to any single LLM vendor as full context.
08
Architecture understanding.Closer to a baseline expectation than a new initiative, but easy to let go stale exactly because building moves this fast.
03 · HOW WE WORK

Three things to hold onto, day to day.

01
The win condition isn't "bug fixed"
It's eval accuracy moving, or usage growing. A cleanly closed ticket that moves neither hasn't actually accomplished the goal.
02
Always be asking how to move faster
Parallel agents, loops, experiments. Look for what can be handed to someone non-technical, and what genuinely can't, rather than assuming today's split is fixed.
03
Make it safer through better evals
Not through more process. The same instinct as eval ownership, applied as a safety mechanism, not just a progress mechanism.
04 · FOR THE DISCUSSION

Open questions to bring into the room.

01
What does a "living evaluation system" concretely look likefor aronlight, this month, not eventually?
02
Who owns cost-per-unit-of-value trackingand where does the number actually live?
03
What's the real quality gate todayif we wrote down what actually happens after a task is marked done?
04
Is there a first concrete step on protecting proprietary intelligenceor is this genuinely a "revisit in six months" item?
Full detail

The document behind this deck

cto-operating-principles.md: the full write-up, plus the open questions
ai-system-best-practice-guide-reviewed.md: the audited engineering standard
product-status.md: where aronlight stands today