Every number on this directory comes out of the harness described here. So does every skill we deleted. This is the whole method, including the round we had to throw away.
There is no honest way to measure 104 skills properly. One skill takes about twenty model sessions to test the way we test: five runs with it, five without, on real material, each output graded alone by a session that is not told the other arm exists. So we did the arithmetic, decided we were not going to pretend, and split the library instead.
Measured. 5 of them, and two of those carry an asterisk of different kinds. 4 were put through the harness described below, against a control. Two of those ran on real open-source repositories and real pages pulled from the live web. The other two ran on constructed fixtures with defects planted by the case author, which is weaker evidence and we would rather say so than let the word measured cover both. The mitigation is real but partial: the author of each fixture was instructed never to open the skill file and did not, so the criteria could not be fitted to the skill's own contents, which is the failure mode that ruined our first round. A constructed fixture still flatters both arms relative to code nobody wrote as an exercise. Every one of those pages carries the worst case, the median, the p value, and the things the skill makes worse as well as better, and whether it passed was decided by a rule written down before a single run executed.
1 more carries a real number from our first harness, which we later found had a contaminated control arm. We have not re-run it, its page says so in a box before it says anything else, and we count it separately from the others rather than rounding it up into them.
Not yet measured. 99 of them. Written, read for accuracy, free to take, and we have run no experiment on them. Their pages say so in a grey box above everything else, and their cards say so in the grid. We make no claim about what they do to an output, because we have not shown one. They are in the queue, and each page names the test it is waiting for.
Cut. 4 of them, and you cannot install those, because they are not published. A skill that went through the harness and failed does not get a quiet demotion to the unmeasured pile. It gets deleted from the library and its numbers stay in the table below, permanently, as a result.
The distinction that matters most is the one between the second and the third. Unmeasured is not the same as failed. Everything in the second group might turn out to work, might turn out to do nothing, and we will tell you which when we have run it.
Our first round was one run with the skill, one run without, against fixtures with defects we had planted ourselves. Five of the six skills tied their control and were cut.
That test was wrong in a way worth explaining, because the same mistake is in almost every prompt comparison published anywhere. One run per arm can only see the ceiling. A skill that makes a good run slightly better and a bad run dramatically better scores identically to a skill that does nothing, because a single sample has no floor to measure. And planted defects flatter the control: a bug placed to be found sits in a small file, in a directory built for the test, described the way the scoring criteria are described. Finding it is a much easier job than noticing something wrong in code nobody wrote as an exercise.
So we rebuilt it. Real open-source repositories at real size, pinned to a commit. No planted defects: where a case needed ground truth we derived it mechanically from the repository itself, the other places a pattern genuinely occurs, the identifiers the schema genuinely does and does not define. More than half the prompts are the vague thing a person actually types, have a look before I ship this, rather than a well-formed question with the answer implied in it. 5 runs per arm, 120 runs in total, on the strongest model available.
Two of the five came back. The table below is every method we have ever put through the harness, unedited and including the failures: the five from that re-test, plus everything measured in the rounds since. A method that failed is not published, so several rows here have no page to click through to, and that is the point of keeping them.
| Skill | Worst without | Median without | Worst with | Median with | p | Verdict |
|---|---|---|---|---|---|---|
| Blast radius review | 9 of 21 | 11 | 13 | 14 | 0.0079 | Revived |
| Contract drift review | 11 of 18 | 14 | 11 | 14 | 0.5635 | Still cut |
| Design system conformance | 15 of 21 | 17 | 17 | 18 | 0.2778 | Still cut |
| SERP-first content brief | 16 of 19 | 17 | 18 | 18 | 0.0317 | Still cut |
| Pre-publish page audit | 7 of 22 | 9 | 10 | 12 | 0.0238 | Revived |
| Email deliverability audit | 13 of 22 | 16 | 18 | 19 | 0.0079 | Passed |
| Form design audit | 18 of 24 | 19 | 21 | 22 | 0.0079 | Passed |
| Cold email audit | 19 of 22 | 20 | 19 | 21 | 0.5714 | Cut |
Worst and median are out of that case's rubric total, across 5 runs per arm. The p value is an exact permutation test, not an approximation: with 5 runs an arm there are only 252 possible splits, so the whole distribution is computable.
Line the cases up by how well the control did, and the pattern is not subtle.
Where the assistant working alone was already scoring near the top of the rubric, no skill moved anything, and one made things slightly worse. Where the control had real room to fail, on a vague prompt against a messy codebase, the two surviving skills moved the floor a long way.
That is the honest shape of this whole category. A skill is not a general upgrade. It is insurance against the bad run, and if your task is well specified and your model is strong, you are buying insurance against something that was not going to happen. The five skills that failed did not fail because they were badly written. They failed because a capable assistant already does that job when the job is clearly stated.
The round after that sharpened it into something more useful than a prior. We measured two of the newly written skills, chosen deliberately because their content is hard specification detail rather than method: an email deliverability audit built on authentication records and published sender requirements, and a form audit built on the autofill token vocabulary in the HTML standard. Both passed, and both passed decisively rather than marginally. Median 16 to 19 out of 22 on the first, worst case 13 to 18. Median 19 to 22 out of 24 on the second, worst case 18 to 21. Both at p=0.0079.
Put the two rounds together and the rule is not really about vagueness at all. A skill works when it carries a fact the model does not have, and does nothing when it carries a discipline the model already has. The design system skill and the contract drift skill asked for rigour, and rigour is something a capable assistant already brings. The two that just passed carry a 1024-bit key length, a testing flag, a token list, and a coercion that eats a leading zero. There is no amount of trying harder that produces those.
That is also the honest prior for the 99 skills with no number yet, and it cuts both ways. The ones built on specifications and thresholds have a real chance of doing something. The ones whose value is an ordering or a habit probably do not, and we have cut four on exactly that suspicion without measuring them, which is a judgement call we would rather state than hide.
One of the two also came with a cost we did not predict, and it is on its page rather than buried here. The form audit finds four whole classes of defect the control never finds, and it is measurably worse at saying which single defect is the release blocker: 3 of five runs without it, 1 of five with it. Breadth bought at the price of priority. Nobody would have guessed the sign of that in advance, which is the entire argument for measuring.
The one result that surprised us most was a skill that stayed cut. Our contract drift skill, which checks that every identifier a change names actually exists, failed its first test with a control that scored suspiciously well. We later found out why, and it invalidated the test. It did not change the answer: on a genuinely clean control it scores 14 of 18 without and 14 with, p=0.5635. The number was untrustworthy. The conclusion was right anyway.
Everything above is about whether a skill works. This section is about a different failure, and it is ours rather than the method's.
Count the first fifty-four skills by shape and the number is uncomfortable. Forty-three of them were audits, reviews, checks or gates. Eighty per cent. Almost the entire library existed to examine something somebody else had already made.
Nobody decided that. There was no meeting where we agreed the directory should be a review shop. What happened is more ordinary and harder to see. An audit emits a findings list. A findings list can be scored against mechanical ground truth: the identifier is in the schema or it is not, the pattern occurs at eleven call sites or it does not. So an audit is cheap to measure, which means an audit can carry a number, which means an audit clears the bar this page exists to enforce. A generative skill emits an artifact whose quality is partly a matter of judgement, so it is expensive to measure and it usually cannot carry a number at all.
We built a harness, we made it the gate, and then the gate started choosing the work. That is the streetlight effect with a directory attached to it: we were not writing what was most useful, we were writing what our own instrument could see. Every incentive in the project pointed that way and none of us noticed for two rounds.
The correction, and its cost. Two rounds of writing since that finding have added fifty skills that produce the artifact rather than critiquing one, which takes the library from eighty per cent audits to forty-one. Work that makes something is now the majority of what is here, and that is the first time it has been true. Almost none of it can be measured the way an audit can, and we have not measured it. That is the trade and we would rather state it than dress it up: we have swapped some measurability for usefulness, on the grounds that a skill nobody wants is not improved by having a p value attached to it.
What did not change. The bar on knowledge did not move. A generative skill still has to carry a procedure, an ordering, a set of thresholds or a catalogue of named failures that a capable model does not already have. "Write a great landing page" is not a skill and was not accepted as one. What earns a place is the kind of thing that turns out to be true and is not guessable: that the clause after "but" in a damaging admission is the one the reader believes, so the order is load-bearing; that a translated page whose locale segment sits after the section name returns a 404 in every language; that charging by a unit which does not move when the customer succeeds caps you at your worst customer's willingness to pay; and that a tool which declares no annotations is treated as destructive and open-world by default, so a harmless search tool that leaves them out earns itself a confirmation prompt on every single call.
And the labelling did not change either, which is the part that matters most. Every one of these is marked not yet measured, on its card and at the top of its page, exactly like every other unmeasured skill here. Several of them have a real objective spine and are in the queue. Several of them do not, and the honest route for those is the preference method in section 7, which is weaker evidence and will say so when we run it. Writing what is useful and then labelling the evidence honestly is a different discipline from only writing what is easy to score, and the second one was quietly winning.
This is the part we would want to read on somebody else's directory, so here it is in full.
Both arms run with every customisation disabled. No user instruction files, no plugins, no installed skills, no hooks, no MCP servers. The only difference between the two arms is the skill body.
We did not do that at first, and it cost us a whole round of results. A run started from a normal working machine silently inherits that machine's standing instruction file. Ours contained a rule about verifying that a database field really exists before writing code against it, which is precisely the thesis of one of the skills we were testing. It was sitting inside the control arm of its own experiment. You can check this on your own setup in about ten seconds: ask a question that only your instruction file can answer, then ask again with customisations disabled and watch the answer disappear.
The second contamination was worse, because it looks like it cannot happen. Install the skill the normal way, in the project folder, and disable auto-loading, and the control arm finds the file by listing the directory and follows it anyway. We only caught this by planting a nonsense constant in a test skill and watching a control run cite it by name. Skills now get injected directly into the arm that is supposed to have them, and the file never exists anywhere the other arm can read.
Grading is blind. Each output is scored on its own, by a fresh session that is not told another arm exists, against a fixed rubric written by an author who was forbidden from reading any skill file. Every criterion is binary and needs a quoted piece of evidence. Roughly a third of every rubric is negative criteria, the form does NOT report a problem on this page that is genuinely fine, because without those a skill wins simply by being noisier. Before grading, the outputs are scrubbed of anything identifying which arm produced them.
The decision rule was written down before any run happened. A skill ships if it moves the median by 2 or more, or the worst case by 3 or more, at p below 0.10. That file was committed before the first case existed. One skill cleared significance and missed the effect size, and it is not here. Moving the line after seeing the number is the one thing that would make every figure on this page worthless.
Where this is still weak, since nothing else on the page will tell you. One published skill rests on a single case rather than two, and its page says so. Rubric criteria are binary, so a half-right answer scores zero. The skill carried over from our first round, interface state audit, has not been re-run under the new method: its number came from a control that inherited too much rather than too little, which inflates the control, so a pass measured against it understates the gap rather than overstating it. Its page carries that warning in a box. And the largest weakness on this page is simply that 99 skills have no evidence at all yet, which is why every one of them says so before it says anything else.
Nothing here needs an account or our app. Copy the file or download it and it works today. Locul just makes it one click and puts the file where each assistant expects it.