Method skills never matched a goal by meaning: best score 0.288 of 683
hypothesess.com, AI-assisted writing · · updated
We tested this on an agent harness that runs long goals. It keeps a library of skills: written procedures with steps, how to verify, and notes. A goal is handed the skills whose description is similar to it, measured by text embeddings and cosine similarity with a cut-off of 0.5 [1]. Cosine similarity scores two vectors by the angle between them, so two texts about the same thing score near 1 and unrelated texts near 0 [2].
The hypothesis
On 2026-11-06 we added five general method skills, adapted from an open-source collection: systematic debugging, verification before completion, writing plans, test-driven development, and writing skills. The hypothesis was that the agent would find them by meaning, as it finds subject skills. We measured it the same day with the harness's own embedding text, model, similarity and cut-off [1].
What we found
- With the cut-off at 0.5, none of the five method skills was handed to any goal [1].
- The best match for systematic debugging was 0.288, on a goal about launching a browser for automated checks [1].
- The best match for test-driven development was 0.303, on a goal about a script [1].
- Lowering the cut-off to 0.30 would hand the method skills to 244 of the harness's 683 goals, many of them unrelated [1].
Type in your own counts to see what a lower cut-off costs:
- Similarity cut-off: 0.5
- Best score a method skill reached: 0.288
- Goals in the library: 683 goals
- Goals the method skills would reach at a cut-off of 0.30: 244 goals
Works out: How far the best match fell short of the cut-off; Share of all goals the method skills would reach at 0.30; Goals still without a method skill at 0.30
Why it happens
Embeddings place text near text about the same subject. "Find the root cause before changing code" is about debugging in general; "make the nightly report read every input" is about reports and inputs. A subject skill, such as how to publish a data page, matches the goals about that subject; a method skill matches none of them [1]. Retrieval by meaning suits subject knowledge. Method needs a trigger.
What we do instead
The harness now hands systematic debugging by a different signal: when a goal's last two runs neither finished it, advanced it, nor asked its user a question, the goal is marked stuck and gets the debugging skill whatever its similarity score [1]. The score still decides subject skills; the trigger decides method.
How to test it on your own agent
Embed five method descriptions and twenty of your own task descriptions with the same model, and list the top score per method. If none clears your cut-off, that method needs a trigger (stuck, failing test, new codebase), not a search [1].
What this does not show
One embedding model, one harness, 683 goals. Whether the stuck trigger improves results is not yet measured: it started on 2026-11-06, and the reading is due once enough stuck goals have run with it [1].
A notebook entry for hypothesess.com, written with AI from the harness's own measurements and checked by hand. The scores and counts are measured; the calculator computes the gap and the share from them.
*Corrected 2026-11-07: Figures match the data page (683 goals, 0.288, 0.303, 244, cut-off 0.5). Removed wording that describes the platform behind the sites: 'a small portfolio of content sites', 'asked the owner', 'the nightly site review read every site' (private-material and site-association rules); removed a REMEMBER callout restating the lead and a CHECKLIST repeating the 'how to test it' section. The data page it cites carries the same wording and cannot be edited through the article route.*