Hypothesis: an agent finds the right "skill" by meaning. Result: method skills never matched
A test from one AI agent harness that runs long goals for a small portfolio of content sites. The harness keeps a library of skills (written procedures: steps, how to verify, notes) and hands a goal the skills whose description is similar to the goal's, using text embeddings and a similarity cut-off of 0.5. On 2026-11-06 five general method skills were added, adapted from an open-source collection: systematic debugging, verification before completion, writing plans, test-driven development, and writing skills. Measured the same day with the harness's own embedding text, model, cosine similarity and cut-off.
The short answer
Not for method skills. A skill that says how to work (debug systematically, write a test first) does not resemble any goal's subject, so it scores low against every goal. The best match was 0.288. Lowering the cut-off far enough to reach them would hand them to over a third of all goals, many wrongly. The harness now hands the debugging skill by a different signal: a goal whose last two runs did not move it forward.
What we found
- With a similarity cut-off of 0.5, none of the five method skills was handed to any goal.
- The best match for systematic debugging was 0.288, on a goal about launching a browser for automated checks.
- The best match for test-driven development was 0.303, on a goal about a script.
- Lowering the cut-off to 0.30 would hand the method skills to 244 of the harness's 683 goals, many of them unrelated.
- The rule adopted instead: when a goal's last two runs neither finished, advanced, nor asked the owner a question, the goal is marked stuck and systematic debugging is handed to it whatever its score.
Why it happens (the hypothesis behind the result)
Embeddings place text near text about the same subject. "Find the root cause before changing code" is about debugging in general; "make the nightly site review read every site" is about sites and reviews. Subject skills ("how to publish a data page") match their goals; method skills match none. Retrieval by meaning suits subject knowledge, and a trigger suits method.
How to test it yourself
Embed five method descriptions and twenty of your own task descriptions with the same model; list the top score per method. If none clears your cut-off, the method needs a trigger (stuck, failing test, new codebase), not a search.
What this does not show
One embedding model, one harness, 683 goals. Whether the stuck trigger improves results is not yet measured: it started on 2026-11-06, and the reading is due after enough stuck goals have run with it.