Lineage

MOTH-002 was run against its parent on 480 held-out windows across 160 datasets, 7020 parent fills against 6134. It came out ahead on 240 of 437 decisive windows (median difference +0.07 points). 42 windows were untouched by the change and are excluded. Windows from one dataset share an instrument and their indicator history, so the test is taken over datasets: 95 of 158 went the candidate's way. A split that lopsided would happen 1.3% of the time by chance. That is evidence worth having, and still not a decision: it is one family of instruments over one stretch of history, and promotion remains a deliberate act.

Typical difference per dataset +0.17 points, 95% of resamples between 0.03 and 0.31. That range excludes zero, which narrows the size of the effect without settling whether it survives another corpus.

With 158 datasets this corpus would have detected a consistent edge of about 0.00 points per dataset. It found none, so whatever MOTH-002 does is smaller than that — which is a bound on the effect, not proof there is none.

On its own this reads as 0.013, under the 0.05 line. It is not on its own: 12 candidates are tested against the same corpus, and asking that many times makes one low number ordinary. Corrected across the family it is 0.094, and MOTH-002 stays a candidate.

MOTH-003 was run against its parent on 480 held-out windows across 160 datasets, 7020 parent fills against 6175. It came out ahead on 204 of 346 decisive windows (median difference +0.14 points). 132 windows were untouched by the change and are excluded. Windows from one dataset share an instrument and their indicator history, so the test is taken over datasets: 84 of 139 went the candidate's way. A split that lopsided would happen 1.7% of the time by chance. That is evidence worth having, and still not a decision: it is one family of instruments over one stretch of history, and promotion remains a deliberate act.

Typical difference per dataset +0.09 points, 95% of resamples between 0.03 and 0.19. That range excludes zero, which narrows the size of the effect without settling whether it survives another corpus.

With 139 datasets this corpus would have detected a consistent edge of about 0.00 points per dataset. It found none, so whatever MOTH-003 does is smaller than that — which is a bound on the effect, not proof there is none.

On its own this reads as 0.017, under the 0.05 line. It is not on its own: 12 candidates are tested against the same corpus, and asking that many times makes one low number ordinary. Corrected across the family it is 0.103, and MOTH-003 stays a candidate.

MOTH-004 was run against its parent on 480 held-out windows across 160 datasets, 7020 parent fills against 7602. It came out ahead on 196 of 391 decisive windows (median difference +0.00 points). 87 windows were untouched by the change and are excluded. Windows from one dataset share an instrument and their indicator history, so the test is taken over datasets: 81 of 154 went the candidate's way. A split at least that lopsided happens 57% of the time when a change does nothing at all, so this corpus does not separate it from chance. MOTH-004 stays a candidate.

Typical difference per dataset +0.02 points, 95% of resamples between -0.07 and 0.13. That range contains zero, so on this corpus MOTH-004 is consistent with making no difference at all.

With 154 datasets this corpus would have detected a consistent edge of about 0.07 points per dataset. It found none, so whatever MOTH-004 does is smaller than that — which is a bound on the effect, not proof there is none.

MOTH-005 was run against its parent on 480 held-out windows across 160 datasets, 7020 parent fills against 5733. It came out ahead on 242 of 422 decisive windows (median difference +0.32 points). 57 windows were untouched by the change and are excluded. Windows from one dataset share an instrument and their indicator history, so the test is taken over datasets: 85 of 149 went the candidate's way. A split at least that lopsided happens 10% of the time when a change does nothing at all, so this corpus does not separate it from chance. MOTH-005 stays a candidate.

Typical difference per dataset +0.27 points, 95% of resamples between -0.04 and 0.50. That range contains zero, so on this corpus MOTH-005 is consistent with making no difference at all.

With 149 datasets this corpus would have detected a consistent edge of about 0.05 points per dataset. It found none, so whatever MOTH-005 does is smaller than that — which is a bound on the effect, not proof there is none.

MOSS-002 was run against its parent on 480 held-out windows across 160 datasets, 6138 parent fills against 4538. It came out ahead on 83 of 204 decisive windows (median difference -0.57 points). 276 windows were untouched by the change and are excluded. Windows from one dataset share an instrument and their indicator history, so the test is taken over datasets: 42 of 102 went the candidate's way. A split at least that lopsided happens 9.2% of the time when a change does nothing at all, so this corpus does not separate it from chance. MOSS-002 stays a candidate.

Typical difference per dataset -0.60 points, 95% of resamples between -0.71 and 0.26. That range contains zero, so on this corpus MOSS-002 is consistent with making no difference at all.

With 102 datasets this corpus would have detected a consistent edge of about 0.71 points per dataset. It found none, so whatever MOSS-002 does is smaller than that — which is a bound on the effect, not proof there is none.

MOSS-003 was run against its parent on 471 held-out windows across 157 datasets, 6041 parent fills against 5654. It came out ahead on 45 of 115 decisive windows (median difference -0.27 points). 356 windows were untouched by the change and are excluded. Windows from one dataset share an instrument and their indicator history, so the test is taken over datasets: 30 of 80 went the candidate's way. A split that lopsided happens 3.3% of the time by chance, and it runs against the candidate: on this corpus MOSS-003 is worse than the version it was meant to improve. That is a real finding, and the useful kind — the change is rejected on evidence rather than left open.

Typical difference per dataset -0.24 points, 95% of resamples between -0.96 and -0.05. That range excludes zero, which narrows the size of the effect without settling whether it survives another corpus.

With 80 datasets this corpus would have detected a consistent edge of about 1.12 points per dataset. It found none, so whatever MOSS-003 does is smaller than that — which is a bound on the effect, not proof there is none.

On its own this reads as 0.033, under the 0.05 line. It is not on its own: 12 candidates are tested against the same corpus, and asking that many times makes one low number ordinary. Corrected across the family it is 0.165, and MOSS-003 stays a candidate.

ECHO-002 was run against its parent on 324 held-out windows across 108 datasets, 7137 parent fills against 7468. It came out ahead on 137 of 324 decisive windows (median difference -0.49 points). Windows from one dataset share an instrument and their indicator history, so the test is taken over datasets: 40 of 108 went the candidate's way. A split that lopsided happens 0.91% of the time by chance, and it runs against the candidate: on this corpus ECHO-002 is worse than the version it was meant to improve. That is a real finding, and the useful kind — the change is rejected on evidence rather than left open.

Typical difference per dataset -0.67 points, 95% of resamples between -1.43 and -0.12. That range excludes zero, which narrows the size of the effect without settling whether it survives another corpus.

With 108 datasets this corpus would have detected a consistent edge of about 1.49 points per dataset. It found none, so whatever ECHO-002 does is smaller than that — which is a bound on the effect, not proof there is none.

On its own this reads as 0.009, under the 0.05 line. It is not on its own: 12 candidates are tested against the same corpus, and asking that many times makes one low number ordinary. Corrected across the family it is 0.072, and ECHO-002 stays a candidate.

ECHO-003 was run against its parent on 324 held-out windows across 108 datasets, 7137 parent fills against 7055. It came out ahead on 22 of 70 decisive windows (median difference -0.42 points). 254 windows were untouched by the change and are excluded. Windows from one dataset share an instrument and their indicator history, so the test is taken over datasets: 17 of 49 went the candidate's way. A split that lopsided happens 4.4% of the time by chance, and it runs against the candidate: on this corpus ECHO-003 is worse than the version it was meant to improve. That is a real finding, and the useful kind — the change is rejected on evidence rather than left open.

Typical difference per dataset -0.33 points, 95% of resamples between -0.46 and -0.01. That range excludes zero, which narrows the size of the effect without settling whether it survives another corpus.

With 49 datasets this corpus would have detected a consistent edge of about 0.54 points per dataset. It found none, so whatever ECHO-003 does is smaller than that — which is a bound on the effect, not proof there is none.

On its own this reads as 0.044, under the 0.05 line. It is not on its own: 12 candidates are tested against the same corpus, and asking that many times makes one low number ordinary. Corrected across the family it is 0.178, and ECHO-003 stays a candidate.