Most of the gains came from fixing a bug + hyperparameters with optuna which is supposed to be already quite automatic (you set the list of all the var with values you want to try and voilà). I guess a simple claude code session would fix that in a few minutes instead of a full day.

To me, I guess the main value of Autoresearch would be to test different kind of architectures. It's sometimes hard to know what to choose and it would probably give a nice overview.

Anyone used it for exploratory modeling?

A lot depends on whether it is expensive to you. I use Claude Code for the smallest of whims and rarely run out of tokens on my Max plan.

For all the folks spending a lot of time and energy in setting up MCP servers, AGENTS.md, etc. I think this represents more that the LLM cannot do what it is being sold as by AI boosters and needs extreme amounts of guidance to reach a desired goal, if it even can. This is not an argument that the tech has no value. It clearly can be useful in certain situations, but this is not what OpenAI/Anthropic/Perplexity are selling and I don’t think the actual use cases have a sustainable business model.

People who spend the energy to tailor the LLMs to their specific workflows and get it to be successful, amazing. Does this scale? What’s going to happen if you don’t have massive amounts of money subsidizing the training and infrastructure? What’s the actual value proposition without all this money propping it up?

This is, of course, only applicable if doing a single test is relatively fast. In my work a single test can take half a day, so I'd rather not let an agent spend a whole night doing a bogus test.

I wrote up some more notes on that here: https://simonwillison.net/2026/Mar/13/liquid/

What about more distant software projects? Give it the CPython source code and say you want it to be faster.

I used it on a data science project to find the best rules for achieving a defined outcome. At first, for fun, then I actually used some of its insights (and it caught a sampling issue I overlooked, oops)

i.e. perhaps minimal changes to autoresearch can take control for cost-effective research to occur.

There always are. You need to think about what those would be, though. Autoresearch outsources the thinking to LLMs.

Non-parametric optimization is not a new idea. I guess the hype is partly because people hope it will be less brute force now.

Yes, for example "swarm optimization".

The difference with "autoresearch" (restricting just to the HPO angle) is that the LLM may (at least we hope) beat conventional algorithmic optimization by making better guesses for each trial.

For example, perhaps the problem has an optimization manifold that has been studied in the past and the LLM either has that study in its training set or finds it from a search and learns the relative importance of all the HP axes. Given that, it "knows" not to vary the unimportant axes much and focus on varying the important ones. Someone else did the hard work to understand the problem in the past and the LLM exploits that (again, we may hope).

Years ago there were big hopes about bayesian hyperparameter optimization, predicting performance with Gaussian processes etc, hyperopt library, but it was often starting wasteful experiments because it really didn't have any idea what the parameters did. People mostly just do grid search and random search with a configuration that you set up by intuition and experience. Meanwhile LLMs can see what each hyperparameter does, it can see what techniques and settings have worked in the literature, it can do something approximating common sense regarding what has a big enough effect. It's surprisingly difficult to precisely define when a training curve has really flattened for example.

So in theory there are many non-LLM approaches but they are not great. Maybe this is also not so great yet. But maybe it will be.

Not true at all. The whole point of ML is to find better mappings from X to Y, even for the same X.

Many benchmarks can’t be solved by just throwing more compute at the problem. They need to learn better functions which traditionally requires humans.

And sometimes an algorithm lets you tap into more data. For example transformers had better parallelism than LSTMs -> better compute efficiency.

in both cases you'd limit it so CC can only talk to the required Anthropic APIs.

So not zero access, but as close to it as you can get.

The docker container didn’t have network access. Claude didn’t have permission to execute anything other than the run.sh bash script, which would orchestrate the docker run