K's-blogs

K
K's-blogs
22 posts
I captured a hundred agent failures a week and fixed three. The bottleneck was curation.
I had full traces and an eval set that was mostly the same bug on repeat. Here is the curation discipline that fixed it.
Oct 8, 20265 min read
Eval the Signature, Not the Program: How I Fixed My DSPy Evaluation
My DSPy RAG pipeline compiled clean, passed its hold-out, and still fell apart on live traffic. Here is what I changed.
Oct 6, 20265 min read1
My CI evals were green. A regression still paged me at 3 AM.
The eval that finally caught a bad regression for me was not the one in CI. CI had run twelve hours earlier, against a frozen test set, and passed. The regression was sitting in live production traffi
Sep 17, 20264 min read3
LlamaIndex makes RAG easy to build and hard to debug. Here is how I evaluate it.
I built a RAG app on LlamaIndex in about four lines. Wire an index to a query engine, point it at my documents, ask a question, get an answer. The first hundred queries were great. I was impressed wit
Sep 2, 20268 min read3
AWS Bedrock's built-in eval graded my agent green. It only looks at a quarter of it.
I built an agent on AWS Bedrock the low-code way. Pick a foundation model, wire an action group to a Lambda, attach a Knowledge Base, add a Guardrail. Then I ran Bedrock's built-in Model Evaluation jo
Aug 26, 20266 min read1
I uninstalled three AI coding CLIs in a week. The model was never the problem.
I tried a bunch of terminal coding agents last month. Some I reopened every morning. Some I uninstalled by Friday. What surprised me is that the ones I dropped were not worse at writing code. The mode
Aug 26, 20265 min read
I Built an Agent Harness by Hand. The Loop Was Tiny. The Five Bugs Were Not.
An agent harness is a while-loop, a dispatcher, and a growing message list. The loop you can read in one sitting. The bugs in how you hand tool results back are what actually cost me an afternoon.
Aug 21, 20268 min read
The checklist I run before I trust a voice agent testing setup
Every voice agent I have shipped demoed perfectly. Then a real caller talked over it, changed their mind twice, asked something off-script, and the agent confidently booked the wrong appointment. None
Aug 20, 20266 min read
Stop calling every self-improving model an intelligence explosion
My feed keeps doing this thing. Someone posts that a model helped train the next model, and the top reply calls it the start of an intelligence explosion. Someone else describes a runaway superintelli
Aug 17, 20264 min read1

K's-blogs