Polar – NVIDIA's Open-Source Agentic Reinforcement Learning Training Framework

1. What Is Polar

Polar is NVIDIA's open-source agentic reinforcement learning (RL) training framework whose core innovation lets existing agent frameworks plug into GRPO and other RL algorithms without modifying internal code. By placing proxies at LLM API boundaries, it captures token-level interaction data and rebuilds training trajectories—turning Codex CLI, Claude Code, Qwen Code, Pi, and other complex code agent harnesses directly into trainable RL environments. Designed as "Rollout-as-a-Service," Polar decouples training from execution for efficient, scalable distributed asynchronous RL.

Technical positioning and domain: RL training infrastructure for agents—not algorithms themselves but infrastructure converting any LLM-API-based agent system into a trainable RL environment, filling the gap for "black box" agent integration.

Research background: From NVIDIA's NeMo team with deep LLM training and deployment experience. Agents (coding assistants, multi-turn tool users) grew complex but traditional RL frameworks require Gymnasium-style rewrites—high integration cost. Polar enables RL without rewriting agent source, including closed-source binaries.

Core value: Solves integration difficulty, poor compatibility, and low efficiency in agent RL training. Zero-intrusion integration for any LLM-API agent; asynchronous architecture and token-faithful trajectory reconstruction improve efficiency and data quality for large-scale agent RL.

Technical characteristics: Black-box proxy paradigm treats LLM API traffic as rollout boundary; token-faithful reconstruction from inference backends (token IDs and log probs, no re-encoding drift); Prefix Merging (~5.39× shorter training time); asynchronous staged execution (CPU runtime prep doesn't block GPU training).

2. Key Features

• API proxy capture: Anthropic/OpenAI/Google-style compatible proxies transparently forward requests and log prompts, sampled tokens, log probabilities, and responses—foundation for trajectory reconstruction.

• Multi-strategy trajectory reconstruction: Per-request (each model call as one trajectory) and prefix merging (detect token-prefix relationships in multi-turn dialogs, merge append-only chains into longer trajectories, fewer trainer updates).

• Asynchronous service architecture: Rollout Server for scheduling/load balancing; Gateway Nodes for runtime warmup, agent execution, trajectory build, and evaluation—CPU prep parallel to GPU training.

• Multi-harness compatibility: Built-in adapters for Claude Code, Codex, Qwen Code, OpenCode, Pi, Gemini CLI—minimal config to attach existing agents.

• Containerized runtime: Docker and rootless Apptainer for isolated, reproducible environments; rootless Apptainer for multi-tenant security.

• Weight synchronization: Trainer and inference server async weight sync—rollouts sample on old policy while trainer updates when enough trajectories arrive—stable efficient training.

3. How to Use

1. Environment: GPU resources (A100+ recommended), Linux Ubuntu 20.04+, Docker or Apptainer, Python 3.8+, CUDA 11.8+, compatible LLM inference server (SGLang, vLLM).

2. Deploy Polar services: Start Rollout Server: python -m polar.rollout_server --port 50051. Start Gateway Nodes: python -m polar.gateway --rollout_server localhost:50051 --inference_server localhost:8000. Ensure inference server is running.

3. Configure harness: Point agent model base URL to Polar Gateway, e.g., Codex CLI --model-base-url http://localhost:8080.

4. Write adapter: YAML harness adapter—env vars, provider settings, launch command from templates (name, provider, command, env_vars).

5. Submit training task: POST TaskRequest with harness, runtime, evaluator, trajectory strategy:
   curl -X POST http://localhost:50051/api/tasks -d '{"harness":"codex","runtime":"docker","trajectory_strategy":"prefix_merging"}'

6. Connect trainer: Frameworks (Slime, Megatron) receive trajectories via callbacks for GRPO updates. Python SDK: from polar import PolarClient; client = PolarClient("localhost:50051"); trajectories = client.get_trajectories(task_id).

Notes: Inference tokenizer must match training model exactly—SGLang recommended for log prob support. Multiple Gateway Nodes for large-scale parallel rollout.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice:

• Quick RL for existing code agents with minimal code change: Polar optimal—zero intrusion and token-faithful reconstruction.
• Full-stack multi-turn tool agent system following framework interfaces: SkyRL-Agent with complete algorithm and eval toolchain.
• Cross-framework agent training data collection: Agent Lightning standardized data interface.

6. Editor's Review

Polar's "black box proxy" paradigm—LLM API as rollout boundary—solves the rewrite-interface problem without altering harness behavior. Token-faithful reconstruction from inference backends avoids re-encoding distortion critical in RL.

Practical infrastructure for researchers and engineers training complex agents at scale. Async architecture and prefix merging materially improve utilization and iteration speed for code agents and multi-turn tool users.

Audience: RL-experienced AI researchers and teams training complex agents—not beginners without architecture familiarity.

Future: growing agent adoption increases demand for efficient RL infra; zero intrusion and elastic scaling fit future complex agents. NVIDIA investment and community may mature ecosystem quickly.

— Strong innovation and utility; docs and community need time.

7. Use Cases

• Code agent RL fine-tuning: Improve SWE-Bench etc. for Codex, Claude Code—point config to Polar Gateway, no source changes.

• Multi-turn tool agent training: Async architecture and trajectory rebuild for browser, DB, API long workflows.

• Offline SFT data generation: Batch high-quality training data on custom harnesses with evaluators and trajectory strategies.

• Multi-agent collaboration optimization: Black-box proxy captures multiple agents' LLM calls for joint end-to-end RL including sub-agent orchestration and context compression.

• Closed-source agent improvement: Black-box RL on API-captured interactions without source access.

8. FAQ

Q: Must I modify agent code?
A: No. Proxy at API boundary records interactions—set model base URL to Polar Gateway endpoint.

Q: Supported inference backends?
A: Anthropic/OpenAI/Google API styles; SGLang recommended for log probs; vLLM/Triton if log prob APIs available.

Q: Trajectory strategy choice?
A: Per-request for simple cases; prefix merging for multi-turn dialogs—~5.39× fewer trainer updates. Long workflows: prefer prefix merging.

Q: Training efficiency?
A: Async prep + prefix merging significantly beat traditional setups; elastic scaling improves throughput at scale.

Q: Closed-source agents?
A: Yes—any LLM-API agent including closed binaries; proxy captures API traffic for RL.

9. Project Links

• GitHub: https://github.com/NVIDIA-NeMo/ProRL-Agent-Server
• Paper: https://arxiv.org/pdf/2605.24220