Auto Research Explained: Karpathy's Loop That Lets AI Improve Itself

There is a way to let an AI agent optimize something in your business while you sleep, run thousands of small experiments against it, and keep only the versions that measurably win. It is called auto research, it is open source, and the whole thing fits in three files. I am Madhuranjan Kumar, and this is the practical playbook I use to point that loop at a real outcome and walk away while it grinds toward a better number. By the end you will know exactly what to hand it, what to lock down, and what "done" looks like when the loop has earned its keep.

The origin is worth ten seconds because it explains the whole idea. A researcher had been hand tuning a training script for months, tweaking one setting, running it, checking the result, tweaking again. One day the obvious question landed: why is a human sitting in this loop at all? A machine can form a guess, change a file, run it, and read the score far faster than any person. So he handed the loop to an agent, and auto research was born. The lesson for a business owner is the same. Anywhere you are personally A/B nudging something toward a number, an agent can take that seat and never get tired.

Start by naming the one number you want to move

Everything in this playbook hangs on a single decision, and if you get it wrong nothing else matters. Before you touch any tool, write down one outcome you can measure with a single objective number. Not a feeling, not a vibe, a number a machine can read without a human in the room. Page load time in milliseconds. Form submission rate. Reply rate on a follow up message. Cost per lead. The test is simple: if you can score it automatically and consistently, you can auto research it. If scoring it requires taste or an opinion, stop here, because this loop will fail on that goal and I will show you why later.

Pick a number that actually connects to money or time. It is tempting to optimize something easy to measure but pointless to improve. A faster page that nobody visits changes nothing. A higher form conversion rate on a page that already converts well is fine, but a page that is quietly losing half its visitors is a far better target. Spend real thought here. The scarce skill in this whole method is no longer the optimizing, because the agent does that for free. The scarce skill is choosing the metric that matters and refusing to chase a vanity number.

There is a compounding effect hiding in this choice that most people miss. A team that used to run thirty careful experiments a year against a metric can now run thousands, because the cost of each run drops to almost nothing. When execution becomes that cheap, the bottleneck moves entirely to the quality of the question you asked. That is why I spend more time on this first step than on any other part of the setup. Get the metric right and the loop turns compute into steady gains. Get it wrong and the loop will optimize hard toward something that does not help you, and it will look busy and productive the whole time it does it.

Build the three files that make up the whole system

Auto research is deliberately tiny. It is three files, and once you understand them you understand the entire method. The first file is the goal file. You write it, and it states the objective, the rules, and the constraints in plain language. Make this page load faster without changing what it shows. Rewrite this message to get more replies without making false claims. This is where your judgment lives, so be specific about what the agent is and is not allowed to break.

The second file is the single thing the agent is allowed to change. That is the entire scope of its freedom. It might be a block of code, a configuration setting, a prompt, a formula, or a piece of copy. Handing the agent exactly one editable surface is what keeps the experiment clean, because every change traces back to one place and every score maps to one idea.

The third file is the scoring file, and it is the most important safeguard in the system. It defines what better means and it produces the number the agent is chasing. The agent must never be allowed to touch this file. If it could edit the scoring, it would do the laziest possible thing: rewrite the rules so that any output counts as a win. Lock it. This single lock is the difference between an agent that genuinely improves your outcome and one that hands you a fake trophy. Treat the scoring file the way you would treat the referee in a match. The players never get to rewrite the rulebook mid game.

Set a fixed time budget so only the idea wins

There is a second, quieter way an agent can cheat, and a fixed time budget closes it. If one experiment is allowed to run for two minutes and the next for twenty, the longer run will usually look better for reasons that have nothing to do with the idea inside it. It simply had more time to grind. So you give every experiment the same time allocation. Two minutes each, or five, whatever fits your task. Now the only variable that changes the score is the raw quality of the idea, which is exactly what you want to measure. Same clock for every contestant. The best thought wins, not the one that got the most runway.

Let the loop run hypothesis, change, score, keep or reset

With the three files set and the clock fixed, the loop itself is almost boring, which is the point. The agent forms a hypothesis about what might move the number. It changes the one file it is allowed to touch. It runs the experiment inside the fixed time budget. It reads the score from the locked scoring file. If the number improved, it commits that change so the gain is banked. If the number got worse or stayed flat, it resets to the last good version and tries something else. Then it does it again, and again, for as many cycles as you care to fund.

The practical way to run this is to clone the open source project, write your own goal file, and point a capable coding agent at the loop with one instruction: keep running experiments, commit what works, reset what does not, and do not stop to ask me. Use a version control system underneath so every kept change is a real commit you can inspect and every reset is clean. What makes this feel like magic the first time you watch it is that the improvements are real and they stack. The agent is not guessing at random for long, because each kept result narrows where it searches next.

One detail I always set up before letting go is a clean baseline. Keep the original, unchanged version saved as a reference so that at any point you can compare the current best against where you started. This does two things. It proves the loop actually improved something rather than just wandering, and it gives you a safe rollback if a change that scored well turns out to have a side effect the metric did not capture. The loop is fast and cheap, but you are still the one accountable for the result, so build in the ability to look under the hood and back out. Trust the loop to search, and keep the receipts so you can verify what it found.

A worked example: an electrician's booking page

Let me make this concrete with numbers, and I want to be clear these figures are illustrative, chosen to show the shape of a run rather than to report a specific account. Picture an electrician running a service business whose whole pipeline depends on a website booking page. The owner picks one number: page load time, because a slow page quietly kills calls and load time is perfectly measurable. The goal file says make this booking page load faster without removing any content or changing what the visitor sees. The editable file is the page's front end code. The scoring file measures load time in milliseconds and is locked so the agent cannot fake a win by deleting the very content that makes the page work.

The run starts and the page loads in 50 milliseconds. The agent tries an idea, scores it, and lands at 33. It banks that and tries another, reaching 28. A third kept change brings it to 25. Within minutes the page is twice as fast and no human edited a line of code during the run. Now play that forward. Suppose that page gets 4,000 visits a month and the faster load lifts the booking rate by a modest two percentage points. On a job worth a few hundred dollars, even a handful of extra booked calls a month pays for the entire effort many times over, and the loop can keep hunting while the crew is out on jobs.

The same owner can point the loop at other scoreable parts of the business. The wording of the lead form can be optimized against the rate at which visitors actually submit it, since submissions are a clean number. This pairs naturally with the CRM and website stack that captures those submissions and the automated follow up that fires after. The text of that first automated follow up message can be optimized against reply rate. In each case the owner hands the agent one honest number and lets it run. This kind of measurable optimization also compounds with paid traffic, because a page that converts better makes every dollar of Facebook and Instagram ad campaigns and every click from Google Ads worth more without raising the budget a cent.

Know where the loop breaks so you do not waste it

The honest limit of auto research is the flip side of its strength. It only works when better is a number. Point it at a subjective goal and it falls apart. Brand look and feel, the emotional pull of a design, the right price point, the tone that makes people trust you: none of these have a clean, automatic score, so the agent has no compass. Without an objective metric it will happily optimize in a random direction and hand you confident nonsense. So the discipline is to sort your business into two buckets. In the first bucket are the things where better is a measurable number, and those you feed to the loop. In the second bucket are the judgment calls, and those stay with a human. The electrician's page speed and form conversion go in bucket one. The brand identity and the pricing strategy stay in bucket two. Respect that line and the loop pays you back. Ignore it and you will spend compute chasing a ghost.

What "done" looks like

You will know this is working when three things are true. First, you have a single objective number written down and a locked file that measures it the same way every time. Second, an agent is running experiments against it inside a fixed time budget, committing wins and resetting losses without asking you to babysit each cycle. Third, the number is moving in the right direction and each gain is a real, inspectable change you could roll back if you ever needed to. When your booking page has gone from 50 to 25, or your form conversion has climbed a few points, and you did not personally run a single experiment to get there, the loop has done its job. That is the whole promise: define the score, lock the referee, fix the clock, and let the machine sit in the seat you used to occupy.

You can absolutely build your first loop yourself, and I would push any owner to pick one measurable number this week and try it. If you would rather have someone choose the metric that actually pays back, set the loop up so it cannot cheat, and point it at the right part of your business the first time, that is exactly the kind of build I do, and you are welcome to bring me in to run it for you.