autoresearch – Karpathy's Open-Source AI Autonomous Research Experiment Framework

1. What Is autoresearch

autoresearch is an open-source AI autonomous research experiment framework from former OpenAI research scientist and former Tesla AI director Andrej Karpathy. It fully automates the manual loop of "tune hyperparameters → run experiments → read logs → tune again" in deep learning model training. The framework allows AI Agents to automatically run nanochat training experiments on a single GPU—autonomously modifying code, adjusting hyperparameters, executing training, analyzing validation metrics, and deciding next optimization directions—transforming AI from research subject to independent research assistant. autoresearch's philosophy is minimal, efficient, reproducible: only three core files needed to run, compressing traditional hours-or-days manual tuning into nearly 100 automatic iterations overnight.

Technical positioning and domain: Belongs to AI-assisted research and automated machine learning (AutoML), specifically focused on automatic hyperparameter tuning and experiment management for deep learning training. Unique positioning embeds AI Agent directly into the full workflow of code modification and training execution—unlike traditional AutoML tools doing only hyperparameter or architecture search. More like an "AI research intern" directly editing training scripts, running experiments, and autonomously deciding based on metrics.

Development background: Karpathy is a landmark figure in deep learning, formerly leading Tesla's autonomous driving AI team and creating the popular CS231n course and nanoGPT project. autoresearch's motivation stems from long observation of AI research efficiency: much deep learning research time consumed by manual tuning and repetitive experiments—AI itself should handle this work. A pragmatic exploration of the "AI scientist" concept, emphasizing efficient autonomous research loops under limited compute.

Core value: Solves deep learning research's core pain point—low experiment efficiency. Traditional manual tuning requires repeatedly modifying code, running training, reading logs, adjusting parameters—time-consuming and error-prone. autoresearch fully automates this loop: ~12 experiments per hour, nearly 100 iterations overnight, freeing researchers from repetitive labor to focus on higher-level research strategy. Fixed 5-minute time budget ensures comparable results across hardware platforms, addressing long-standing "reproducibility" challenges.

Technical characteristics: Core is "agent-environment closed loop" design: program.md as Agent instruction file defining research goals and constraints; Agent reads instructions, directly modifies train.py, runs training, evaluates val_bpb metric, autonomously decides keep/discard modifications forming complete automatic iteration. Single-file focus keeps each experiment's diff clear and traceable; fixed time budget and vocab-independent metric ensure fair comparison before/after architecture changes.

2. Key Features

• Autonomous code iteration: AI Agent directly edits train.py—modifying model architecture, optimizer type, learning rate, batch size, and full training pipeline configuration. Agent isn't limited to hyperparameters but can change model structure itself—attention mechanism replacement, regularization layers, loss function modifications—for true "code-level" automatic optimization.

• Fixed time budget experiments: Each training strictly limited to 5 minutes wall-clock regardless of model size, batch size, or hardware performance. Eliminates hardware differences affecting experiment comparability—results on different GPUs (RTX 3090 vs A100) directly comparable, providing unified benchmark for distributed collaborative research.

• Automatic improvement filtering: Uses val_bpb (validation bits per byte) as unified evaluation metric—independent of vocabulary size, ensuring fair comparison before/after architecture changes. Agent automatically keeps modifications lowering val_bpb, discards attempts worsening metrics, forming reinforcement-learning-style "keep-discard" iteration loop.

• Human-controllable research organization: program.md defines agent behavior and research strategy—humans iterate "research organization code" rather than directly editing code. Specify in program.md "prioritize different learning rate schedules" or "explore attention head count changes"—Agent behavior fully controllable and explainable.

• Single-file focus design: Agent modifies only train.py, keeping experiment scope controlled and diffs reviewable. Avoids Agent jumping between files causing chaos; each experiment's changes concentrated in one file for understanding improvement paths and easy rollback.

• High-throughput experiment loop: Optimized workflow completes full cycle from code modification to metric evaluation in ~5 minutes—~12 experiments per hour, nearly 100 automatic iterations overnight, far exceeding manual efficiency, enabling broader parameter space exploration in short time.

• Environment-agnostic design: uv project manager unifies dependency management for consistent experiments across systems. uv sync auto-installs dependencies; uv run executes scripts—eliminating "environment inconsistency causes different results."

3. How to Use

1. Install uv project manager: First install uv, a high-performance Python package manager:

   curl -LsSf https://astral.sh/uv/install.sh | sh
   

   Reload shell config or restart terminal after installation.

2. Clone repository and install dependencies:

   git clone https://github.com/karpathy/autoresearch.git
   cd autoresearch
   uv sync
   

   uv sync parses pyproject.toml dependencies and installs all necessary packages including PyTorch, numpy, etc.

3. Data preparation: Run data preparation script downloading training data and training BPE tokenizer:

   uv run prepare.py
   

   One-time process, ~2 minutes. Generates preprocessed training data and tokenizer files in data/.

4. Verify environment: Manually run training once to confirm correct setup:

   uv run train.py
   

   Normal execution ~5 minutes, outputs training logs and validation metrics. If fails, check GPU drivers, CUDA version, or dependency installation.

5. Launch AI agent: Load AI agent (Claude Code, Codex, Cline, etc.) in IDE (VS Code, Cursor, Windsurf) with working directory at repository root. Ensure agent can read program.md defining behavior rules and research strategy.

6. Start automatic research: Input prompt to start automatic iteration, e.g.:

   "Hi, have a look at program.md and let's kick off a new experiment"
   

   Agent reads program.md instructions, begins modify train.py → run training → evaluate metrics → keep/discard loop. Recommend running in background to avoid interruption.

7. Review results: Next day review experiment logs and train.py modification history. git log shows each commit's diff for understanding agent improvement path. Optimized model weights and training config remain in repository.

Notes:

• Ensure NVIDIA GPU drivers and CUDA toolkit installed (CUDA 11.8+ recommended)
• Reserve at least 10GB disk space for data and model weights
• Test in virtual environment before first run to avoid Python environment conflicts
• If using Claude Code, set API quota limits to avoid unexpected token consumption

4. Pros and Cons

5. Comparison with Similar Tools

Selection recommendations:

For AI researchers focused on LLM training and tuning, autoresearch is currently the best fit. Minimal design and high-throughput experiments especially suit rapid training config exploration under limited resources. For individuals or small teams, "single GPU + nearly 100 iterations overnight" significantly improves research efficiency. However, task scope limited to nanochat training—for other architectures or task types, consider other tools.

For researchers needing full-cycle research support (literature review to paper generation), SciClaw offers more comprehensive features but closed-source and higher hardware requirements may not suit individuals. For general ML tasks, AutoGluon provides mature auto model selection and tuning; Optuna is lightweight hyperparameter search for custom search spaces. For quickly validating best config of general ML models, AutoGluon or Optuna may be more appropriate; for deeply exploring LLM training optimization paths, autoresearch's "code-level auto-iteration" is irreplaceable.

6. Editor's Review

autoresearch represents a pragmatic and visionary exploration in AI-assisted research. Technical innovation embeds AI Agent directly into code modification and training execution—not traditional AutoML's hyperparameter or architecture search only. This "code-level" auto-iteration enables exploration beyond hyperparameters—model architecture, optimizer choice, data processing. Fixed 5-minute budget and val_bpb metric cleverly address long-standing "reproducibility" and "unfair comparison" problems, providing unified benchmark for distributed collaborative research.

Practically, "single GPU + nearly 100 iterations overnight" has significant meaning for individual researchers and small teams. Traditional manual tuning: 5-10 experiments per day; autoresearch 10×+ improvement while freeing researchers to think higher-level research questions. Minimal design (three core files) makes onboarding cost extremely low.

Target audience: AI researchers, LLM training engineers, deep learning students. Senior researchers use it as efficient experiment engine; students observe agent improvement paths and diff records to intuitively understand hyperparameter and architecture impact on performance.

Strong extension potential: future long-term memory (recording historical success/failure patterns), multi-task training, extension to other architectures (diffusion, ViT), even distributed multi-GPU experiments. Karpathy's open-source projects typically gain broad community attention—autoresearch ecosystem likely to grow quickly.

Scoring: High technical innovation (code-level auto-iteration), strong practical value (significant efficiency improvement), low onboarding cost (minimal design). Deductions for limited task scope (nanochat only) and lack of long-term memory. For target users (LLM training researchers), nearly perfect tool.

7. Application Scenarios

• LLM training hyperparameter auto-search: Define hyperparameter search space in program.md (learning rate range, batch size options, optimizer types); Agent auto-explores combinations for optimal config. Nearly 100 experiments overnight quickly converges to best parameters, eliminating tedious manual tuning.

• Architecture innovation validation: When proposing new attention mechanisms, positional encodings, or activation functions, use autoresearch for rapid validation. Specify in program.md "try replacing attention with linear attention" or "test different positional encoding schemes"—Agent autonomously modifies code, runs training, evaluates—multiple architecture variants compared within an hour.

• Low-cost model tuning: For resource-limited individuals or small teams, autoresearch auto-optimizes small models (nanochat-level GPT) on single GPU. No expensive multi-GPU cluster needed—hours of work vs days of manual tuning, significantly lowering research barrier.

• AI research methodology exploration: Concept validation platform for autonomous AI research agents. Design different research strategies in program.md (exploration vs exploitation balance, dynamic search space adjustment), observe agent behavior and experiment results, deepening understanding of AI autonomous research methodology.

• Teaching and training: In deep learning courses, autoresearch as teaching tool for students to observe hyperparameter and architecture impact on performance. Review agent improvement paths and diff records for rapid tuning experience accumulation.

8. FAQ

Q: How much GPU VRAM does autoresearch need?
A: Recommend NVIDIA GPU with ≥8GB VRAM (RTX 3070, RTX 4060 Ti or higher). GPUs under 6GB (GTX 1660) may fail—nanochat is lightweight but training requires gradient, optimizer state, and activation storage. If insufficient VRAM, try reducing batch_size or n_embd (embedding dimension) but may affect experiment quality.

Q: Does autoresearch support multi-GPU training?
A: Current version supports single GPU only—intentional design choice. Fixed 5-minute budget becomes complex with multi-GPU since GPU count and model affect training speed, breaking comparability. Karpathy explicitly states single GPU design maintains fairness and reproducibility. For multi-GPU, use DeepSpeed, FSDP, etc. for manual tuning.

Q: How to customize evaluation metrics?
A: Default uses val_bpb (validation bits per byte). To customize: 1) Find val_bpb calculation in train.py; 2) Add custom metric calculation; 3) Modify agent decision logic for keep/discard based on new metric. Update program.md to inform agent of new evaluation criteria.

Q: Difference between autoresearch and Optuna?
A: Optuna is Bayesian optimization hyperparameter search—user defines search space and trial function, Optuna auto-selects hyperparameter combinations. autoresearch Agent directly modifies code (not just hyperparameters)—architecture, optimizer, data processing. autoresearch's fixed time budget ensures experiment comparability; Optuna lacks built-in time normalization. Optuna is "parameter-level" search; autoresearch is "code-level" search.

Q: How to ensure Agent won't break existing code?
A: autoresearch uses Git version control—Agent creates new branch or commit before each experiment, keeps if improvement valid, rolls back if not. Initialize Git repository and create initial commit before launching Agent. Define safety constraints in program.md like "don't delete existing functions" or "keep data loading logic unchanged."

Q: Can autoresearch be used for other models (ViT, diffusion)?
A: Currently only nanochat (GPT-style) training, but framework theoretically extensible. Modify model definition, data loading, training loop in train.py, adjust val_bpb for new tasks. Generic design with single-file focus and fixed time mechanism should adapt to other models. Note different models have different training times—5-minute fixed budget may not suit all.

9. Project Links

• GitHub repository: https://github.com/karpathy/autoresearch