HarnessDev tests agents that build their own harnesses

HarnessDev evaluated six creator LLMs across four domains and five downstream benchmarks totaling 2,207 unique instances, asking agents to build and then evolve their own execution harnesses. In the new arXiv benchmark, generated harnesses matched or beat selected human references in writing and machine-learning experimentation but stayed behind on code and search/research; evolution gains were unstable and transferred only partly to held-out tasks. Teams extending Octobench’s harness-level accounting should benchmark harness and model together—self-improvement is not yet a portable capability.