GenEvolve – Self-Evolving Image Generation Agent by Meituan and Others

1. What is GenEvolve

GenEvolve is a self-evolving image generation agent jointly developed by the Hong Kong University of Science and Technology (Guangzhou), Meituan, and the National University of Singapore. It formalizes open-ended image generation tasks as tool-orchestrated visual trajectories. By searching for textual evidence, retrieving visual references, and invoking generative knowledge, it ultimately synthesizes an executable prompt-reference program, which is then rendered by any reference-conditioned generator. Based on the Qwen3-VL-8B-Instruct model, the agent adopts a two-stage training strategy—SFT cold start combined with GRPO reinforcement learning and visual experience self-distillation—achieving a paradigm shift from "passive response" to "active planning and self-evolution." On knowledge-intensive image generation benchmarks, it surpasses GPT-4o with only 8B parameters.

Image source: Official article
Image source: official article

Technical Positioning and Domain: GenEvolve belongs to the intersection of multimodal AI agents and image generation. Its core innovation lies in upgrading image generation from a single prompt mapping to a multi-tool collaborative decision-making and execution process. Unlike traditional end-to-end diffusion models, it treats the agent as a "generation strategy brain," compensating for the generative model's lack of world knowledge through external search, knowledge queries, and reference retrieval, thereby achieving more controllable, accurate, and real-world-constraint-compliant image outputs. This positioning fills the gap between "general agent capabilities" and "professional image generation," offering a programmable and evolvable solution for complex visual content production.

Research Background: This project is led by an interdisciplinary team from the Hong Kong University of Science and Technology (Guangzhou), in collaboration with Meituan's Visual Intelligence Department and the School of Computing at the National University of Singapore. The research motivation stems from inherent flaws in current image generation models for knowledge-intensive tasks: models cannot know the exact number of floors of the Eiffel Tower, facial features of public figures, or internal structures of specific products, leading to frequent "hallucinations" in generated results. The team aims to empower the agent with search and reasoning capabilities, so the generation process no longer relies on the model's internal memory but instead retrieves evidence from external knowledge sources in real time, fundamentally improving generation accuracy and credibility.

Core Value: GenEvolve addresses three key issues: First, it expands image generation from a "single prompt" to "multi-step tool orchestration," allowing the agent to autonomously decide when to search text, retrieve images, or invoke generation skills, significantly reducing user reliance on prompt engineering. Second, it introduces a self-evolution mechanism through GRPO trajectory optimization and visual experience self-distillation, enabling the agent to continuously improve its strategy through repeated practice without requiring manual annotation. Third, it achieves generator transferability: the same agent strategy can seamlessly adapt to different backend generators such as Qwen-Image-Edit and Nano Banana Pro, breaking the binding between strategy and generator.

Technical Features: GenEvolve's core technology stack includes: tool invocation capabilities based on a vision-language model (Qwen3-VL-8B), two-stage training (SFT + GRPO + SDL), and visual experience self-distillation for fine-grained credit assignment. Its differentiating capability lies in not only outputting the final image but also the complete decision trajectory (records of search, image_search, and query_knowledge invocations), making the generation process fully interpretable, auditable, and reproducible.

2. Key Features

• Text Search (search): The Agent automatically invokes a search engine before generation to collect external factual evidence, supplementing world knowledge such as real building structures, public figure characteristics, and product specifications, thereby avoiding "hallucinations" caused by the generation model's lack of training data. For example, when generating "Hall of Supreme Harmony in the Forbidden City," the Agent first searches for details like its roof style and number of bays, then injects these constraints into the generation process.

• Image Search (image_search): Retrieves visual reference images to provide visual anchors such as landmark appearances, facial features, and material textures for generation. The Agent can obtain multiple reference images from the internet and automatically select the most relevant ones as conditional inputs, significantly improving the consistency between generated results and real scenes.

• Knowledge Query (query_knowledge): Activates eight internally predefined generation skills, including text rendering, precise counting, spatial layout, attribute binding, anatomical structure, material consistency, aesthetic style, and multi-object interaction. The Agent dynamically queries the Markdown documentation of the corresponding skill based on current needs to obtain specific constraint formulations, enabling programmatic expression of complex requirements.

• Prompt-Reference Program Synthesis: Integrates the outputs of the above three tools into a structured executable program, including search evidence summaries, reference image paths, and knowledge constraint rules. This program can be parsed and rendered into the final image by any reference-conditioned generator (e.g., Qwen-Image-Edit, Nano Banana Pro), achieving decoupling between Agent strategy and generation backend.

• Self-Evolution (GRPO + Visual Experience Self-Distillation): During training, multiple trajectories are sampled for the same request, rendered into images, and jointly scored by a visual scorer (based on VLM) and a text scorer (based on rules). GRPO is used to optimize trajectory-level rewards. Furthermore, through visual experience self-distillation (SDL), the differences between the best and worst trajectories are compared and summarized into a structured Decision Guide. Token-level reverse KL distills teacher preferences into the student model, enabling the Agent to learn decision habits such as "how to search, select references, and organize constraints for similar requests."

• Generator Transferability: The same Agent strategy can adapt to different backend generators without any fine-tuning. Experiments show stable performance when using the open-source Qwen-Image-Edit, and further improvement in image quality when switching to the commercial-grade Nano Banana Pro, demonstrating that the tool orchestration strategy learned by the Agent is universal and does not depend on the internal capabilities of a specific generator.

• Cross-Benchmark Generalization: On the WISE knowledge-intensive benchmark, GenEvolve achieves a KScore of 0.3663 without in-domain fine-tuning, surpassing GPT-4o (0.3493) and Gen-Searcher (0.3493). This indicates that strategies based on tool orchestration utilize external knowledge more effectively than pure language models or simple search-augmented methods, and exhibit better generalization than closed-source large models.

3. How to Use

1. Environment Requirements and Prerequisites: Requires a Linux or Windows machine with at least 16GB GPU memory, NVIDIA A100 or RTX 4090 recommended. Python version must be ≥3.9, with PyTorch 2.0+ installed. Also need to prepare a search engine API key (such as Bing Search API or Google Custom Search API) and at least one backend generator API key (Qwen-Image-Edit or Nano Banana Pro).

2. Clone Repository and Install Dependencies: Clone the GenEvolve codebase from GitHub, execute git clone (link to be updated after official release) install -r requirements.txt to install all dependency packages. Note that flash-attn must be manually installed to accelerate vision-language model inference.

3. Download Model Weights: Download the strategy model weights from HuggingFace using the command git lfs clone (link to be updated after official release) download MeiGen-AI/GenEvolve. The model is based on Qwen3-VL-8B-Instruct, with weights approximately 16GB.

4. Configure Tools and Generator: Set the search engine API key (search_api_key and search_engine parameters) in config.yaml, and specify the generator backend type (generator_type: "qwen_image_edit" or "nano_banana") along with its API key. Place the Markdown files for the eight generation skills into the skills/ directory, ensuring the query_knowledge tool can read them.

5. Initialize Agent and Execute Generation: Instantiate the GenEvolveAgent class in a Python script, passing in the model path, generator configuration, and tool list. Call agent.run("Generate a frontal photo of the Hall of Supreme Harmony in the Forbidden City, requiring golden glazed tiles on the roof and a clear sky"). The Agent will automatically perform search, image retrieval, knowledge query, and program synthesis, finally returning the generated image path. The entire process takes approximately 30-60 seconds, depending on search latency and generator speed.

6. Best Practices: For first-time use, it is recommended to run agent.evaluate() to test simple requests and confirm all tools are functioning properly. If generation results are unsatisfactory, adjust the temperature parameter (default 0.7) to control Agent exploration, or manually intervene with search keywords. Note that search engine APIs have call limits; it is recommended to use a paid account to avoid rate limiting.

4. Pros and Cons Analysis

5. Comparison of Similar Tools

Selection Recommendations: For scenarios requiring high-accuracy knowledge-intensive image generation (e.g., historical buildings, public figures, product displays), GenEvolve is currently the optimal choice due to its tool orchestration and self-evolution capabilities, especially suited for content production teams and commercial design departments. If only quick creative image generation is needed with low factual accuracy requirements, the traditional LLM+SD pipeline offers lower cost and faster speed. For scenarios requiring multi-turn interactive editing and local modifications, Visual ChatGPT provides more flexible tool invocation capabilities, but it lacks systematic knowledge retrieval and evolution mechanisms, performing less effectively under complex constraints compared to GenEvolve. As a representative of search-augmented generation, Gen-Searcher performs close to GenEvolve on simple factual tasks, but lacks self-evolution and generator transferability, limiting its long-term potential.

6. Editor's Take

GenEvolve demonstrates a significant breakthrough in technological innovation: it organically combines reinforcement learning (GRPO) with knowledge distillation (visual experience self-distillation), achieving fine-grained credit assignment and closed-loop self-evolution in the field of image generation agents for the first time. This design not only improves the accuracy of single-generation tasks but, more importantly, endows the agent with the ability for continuous improvement, allowing it to automatically optimize tool invocation strategies through repeated practice—offering greater long-term value than traditional static prompt optimization or simple search augmentation. From a practical standpoint, GenEvolve effectively lowers the barrier to professional image generation: users only need to describe their requirements in natural language, and the agent can autonomously handle complex tasks such as information collection, visual reference screening, and constraint integration, outputting high-quality executable generation programs. This "programmatic generation" paradigm is particularly suited for commercial scenarios that demand precise control over details, such as advertising posters, e-commerce main images, and educational illustrations. The target audience includes AI content creators, designers, product managers, and researchers, especially those with strict requirements for factual accuracy and consistency in generated results. The future development potential is immense: as more backend generators and tool libraries are integrated (e.g., adding 3D generation and video generation tools), GenEvolve's architecture can naturally extend to multimodal content production. Additionally, its self-evolution mechanism is expected to produce stronger emergent strategies under larger-scale data and longer training cycles. Recommendation rating: ★★★★☆ (4.5/5 stars). Deductions are due to: incomplete open-sourcing of training components, heavy reliance on external search APIs, and insufficient optimization for Chinese-language scenarios. Overall, however, GenEvolve represents a crucial step in the evolution of image generation agents from "assistive tools" to "autonomous creative partners," and its technical approach merits industry attention and follow-up.

7. Application Scenarios

• Knowledge-Intensive Image Generation: Scenarios requiring external world knowledge, such as generating precise structural diagrams of the Eiffel Tower, frontal photos of the Hall of Supreme Harmony in the Forbidden City, portraits of specific public figures, or historical event reconstructions. The Agent supplements factual details through search tools, preventing the generator from "hallucinating" erroneous details (e.g., an extra layer on the tower, incorrect facial features). Suitable for fields with high demands for authenticity, such as museums, educational publishing, and news reporting.

• Quality-Constrained Commercial Design: Advertising posters, e-commerce main images, and product packaging designs with strict requirements for text rendering (e.g., bilingual slogans on posters), precise counting (e.g., 12 cups), spatial layout (e.g., "sofa on the left, floor lamp on the right"), attribute binding (e.g., "red leather sofa"), and material realism (e.g., silk, metal). The Agent uses the query_knowledge tool to call corresponding skill documents, converting constraints into program parameters to ensure results comply with commercial standards.

• Reference-Consistent Serialized Content Production: Serialized content requiring identity consistency with reference images (e.g., character design, brand IP image), material uniformity (e.g., specific fabric texture), or style consistency (e.g., Van Gogh's painting style). The Agent retrieves multiple reference images via image_search and explicitly binds reference conditions in the prompt-reference program, enabling coherent generation of the same character across different scenes. Applicable to game concept design, animation storyboarding, and brand visual system development.

• Vague Requirement Clarification and Proceduralization: Transforming vague user descriptions (e.g., "an office with a tech feel") into executable generation programs, clarifying which keywords to search, which style images to reference, and which knowledge constraints to invoke. The Agent automatically deconstructs requirements, reducing users' reliance on prompt engineering. Suitable for non-professional users seeking high-quality images quickly, as well as for enterprises needing batch generation of standardized visual assets.

8. Frequently Asked Questions (FAQ)

Q: How much GPU memory does GenEvolve require to run?
A: It is recommended to use a GPU with at least 16GB of memory (e.g., RTX 4090, A100). The model weights are approximately 16GB, and peak memory usage during inference is around 18GB. If using an RTX 3060 (12GB), you can run it with quantization or reduced precision, but the speed will drop significantly, and a single generation may take over 3 minutes.

Q: Does it support local offline operation?
A: Full offline operation is not supported. The Agent's search and image_search tools require calls to external search engine APIs. The query_knowledge tool can read local Markdown files offline, but the backend generator (Qwen-Image-Edit or Nano Banana Pro) typically also requires cloud service APIs. If using a locally deployed generator (e.g., Stable Diffusion), partial offline operation is theoretically possible, but you would need to modify the code yourself.

Q: Is the training data publicly available? How can the training process be reproduced?
A: The model weights are open-sourced, but the Teacher trajectory dataset used for training, the visual scorer weights, and the scoring rules have not been publicly released. Currently, only inference and the use of pre-trained strategies are possible. Full training reproduction requires waiting for the team to release these resources later or building similar data yourself. It is recommended to follow updates on the GitHub repository.

Q: How effective is GenEvolve at generating Chinese content?
A: The base model Qwen3-VL-8B supports both Chinese and English, but the coverage of Chinese content by search engines and image retrieval is not as good as English. This results in lower accuracy for searches related to Chinese landmarks, people, etc. For Chinese scenarios, it is recommended to use the Bing Search API with Chinese search parameters, or manually provide reference image paths to compensate for retrieval shortcomings.

Q: Can I customize tools or add new generation knowledge?
A: Yes. GenEvolve's tool invocation framework supports extension. You just need to add a new tool class in the tools/ directory and implement the corresponding interface. Generation knowledge skills are managed as Markdown files. Users can add new skill documents in the skills/ directory, and the Agent will automatically read them via query_knowledge. Note that you may need to retrain or fine-tune the Agent to learn how to use the new tools; otherwise, it may not invoke them correctly.

Q: Can the Agent's output program be saved and reused?
A: Yes. The prompt-reference program generated by the Agent is output in JSON format, containing complete information such as search evidence, reference image paths, knowledge constraints, and generation parameters. Users can save this program and later pass it back to the generator for rendering, or modify it as a template for batch production, enabling the reuse and fine-tuning of generation logic.

9. Project Address

• Project Website: https://ephemeral182.github.io/GenEvolve/
• GitHub Repository: https://github.com/MeiGen-AI/GenEvolve
• HuggingFace Model Library: https://huggingface.co/MeiGen-AI/GenEvolve