BigSet – TinyFish's Open-Source Multi-Agent Real-Time Web Scraping Tool

1. What Is BigSet

BigSet is an open-source multi-agent real-time web scraping system from US startup TinyFish, under the AGPL-3.0 license. Users describe needed data in natural language; the system automatically infers table structure, dispatches agents to scrape from the live web, deduplicates and validates, and produces exportable CSV/XLSX structured datasets. Built on the Mastra framework's multi-agent orchestration architecture, combined with Claude Sonnet and Qwen for semantic analysis and task scheduling, it supports scheduled refresh from 30 minutes to weekly to keep datasets current. Targeting data collection and analysis, it aims to reduce traditional crawler development to zero—enabling non-technical users to easily build structured datasets.

Image source: Official article

Technical positioning and domain: BigSet sits at the intersection of NLP and multi-agent systems, focused on real-time web data collection and structured dataset generation. Its unique positioning replaces traditional crawler configuration with a natural language interface for a fully automated "describe to export" closed loop—filling the gap in AI-driven zero-code data collection tools.

R&D background: Developed by TinyFish, which has processed 40M+ agent operations in enterprise search and scraping APIs with deep web data collection experience. BigSet's motivation: traditional crawler development is high-barrier—selectors, anti-scraping, pipeline maintenance—while most business users only need structured data. BigSet deeply integrates multi-agent collaboration, LLM semantic understanding, and automated collection to lower technical barriers.

Core value: BigSet solves three data collection pain points: complex crawler writing, difficult multi-source integration, and tedious update maintenance. Its innovation combines LLM table structure inference with multi-agent parallel scraping—users need not specify URLs or fields; the system discovers data sources and generates relational datasets with scheduled refresh and auto-deduplication, greatly improving data acquisition efficiency.

Technical characteristics: Multi-agent orchestration on Mastra framework—orchestrator handles entity discovery and task decomposition; sub-agents parallel-scrape individual records (max 6 tool calls each). JS closure injection of dataset IDs effectively isolates prompt injection attacks. Underlying TinyFish Search/Fetch/Browser API for live web access; Vercel AI SDK and OpenRouter unify Claude Sonnet and Qwen scheduling.

2. Key Features

• Natural language table creation: Describe needed data in one sentence (e.g., "AI startup funding information"); system auto-infers column names, data types, and primary key constraints without manual Schema design. Based on Claude Sonnet semantic analysis from entity relationships and data characteristics; serves as validation baseline for subsequent collection.

• Multi-agent parallel collection: Orchestrator discovers target entities, splits scraping into independent units, assigns to multiple sub-agents in parallel. Each sub-agent calls TinyFish Search/Fetch/Browser API from live web (max 6 tool calls), completing discovery-to-fill automatically—significantly shortening large-scale collection time.

• Auto-deduplication and provenance: Primary-key-based deduplication ensures unique records. Each row includes source URL—all data traceable and verifiable—solving redundancy and credibility in multi-source collection.

• Scheduled refresh: Five frequencies—30 minutes, 6 hours, 12 hours, daily, weekly—for automatic dataset updates. Keeps data synchronized for dynamic monitoring scenarios.

• Format export: CSV and XLSX direct download for Excel, Tableau, Python Pandas, etc. Export includes full headers and source URL column—no secondary cleaning needed.

• Preset datasets: Nine curated public datasets (tech company lists, industry reports, etc.) out of the box. Quick core feature experience; modify preset descriptions for customized versions.

3. How to Use

1. Environment requirements: Docker (20.10+ recommended) and Make; Linux or macOS suggested. Register at TinyFish, OpenRouter, and Clerk for three API key types: TinyFish (web scraping), OpenRouter (LLM calls), Clerk (authentication).

2. Clone and configure: git clone https://github.com/tinyfish-io/bigset.git. Copy .env.example to .env; fill in all three API keys—correct format, no extra spaces or newlines.

3. Start services: Run make dev—auto-installs dependencies, starts Postgres and Convex (dataset and task state storage). First launch may take minutes; "Service running at localhost:3500" indicates ready.

4. Access and use: Open http://localhost:3500 in browser; register and log in. Describe needed data in natural language (e.g., "2024 AI chip company product specs"); system generates structured dataset. View collection progress and preview in real time.

5. Load preset data (optional): Run make seed-public-datasets to load nine built-in public datasets. View examples and modify descriptions for new datasets.

6. Set scheduled refresh: On dataset detail page, choose frequency (30 min/6 hr/12 hr/daily/weekly). Frequent refresh consumes API quota—set appropriately for data change rate.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice: Business analysts or market researchers needing quick structured datasets from zero without code—BigSet's NL table creation and auto-discovery excel. Scheduled refresh and dedup suit competitive monitoring and market trend tracking. For known target URLs needing deep crawl (e.g., e-commerce product pages), Firecrawl /agent or Scrapy custom crawlers are more flexible. Cloud-hosted large-scale concurrent teams—Apify's Actor ecosystem and scheduling are mature but paid. BigSet suits small-to-medium scale emphasizing data privacy and self-control.

6. Editor's Review

BigSet demonstrates deep fusion of multi-agent systems and LLMs in data collection. Biggest highlight: transforming traditional crawler "technical problems" into "business problems"—users describe data needs; system handles entity discovery, field inference, parallel scraping, dedup, and export. This "describe equals data" paradigm fundamentally restructures data collection workflows, lowering barriers for non-technical users. Practically, BigSet addresses frequent enterprise data needs (competitive monitoring, industry research)—compressing hours or days of crawler development to minutes. Scheduled refresh improves data timeliness for dynamic metric tracking.

Limitations exist. Strong LLM dependency means table inference quality varies by model capability—Chinese or vertical domains may be less precise. Sub-agent 6 tool call limit may incompletely scrape complex entities (login, pagination, API calls). Only Claude Sonnet and Qwen supported—no other LLM switching.

Target users: Business analysts, market researchers, product managers (non-technical), and startups validating data hypotheses quickly. Professional crawler engineers can use BigSet for rapid prototyping; large production may still need Scrapy. Future potential: more model choices, higher tool call limits, custom source rules could make BigSet a data collection "low-code platform."

Deductions for limited model choice and shallow crawl depth; zero-config ease and open controllability provide irreplaceable value in specific scenarios.

7. Application Scenarios

• Competitive monitoring: Auto-track competitor pricing, feature updates, hiring. Describe "Competitor X product pricing and specs"; daily refresh produces intelligence tables for strategic decisions.

• Market research: Collect industry company info, funding, market share. E.g., "2024 global AI chip startup funding rounds and valuations"—cross-site aggregation, dedup, industry database export to Excel for trend analysis.

• Investment analysis: Track stocks, crypto, startup funding and valuation changes. Describe "Past week crypto exchange coin listing announcements"; scheduled scrape supports investment decisions.

• Recruiting intelligence: Monitor target company or industry job postings, skill requirements, salary ranges. E.g., "Beijing AI algorithm engineer JDs and salaries"—multi-platform collection for talent market trends and HR strategy.

• E-commerce price comparison: Scrape multi-platform product price, inventory, reviews. Describe "iPhone 16 Pro Max price and stock on JD, Taobao, Pinduoduo"—comparison dataset with scheduled refresh for consumers or merchants.

8. FAQ

Q: Does BigSet support Chinese data collection?
A: Yes. Underlying LLMs (Claude Sonnet and Qwen) understand Chinese; NL descriptions can be Chinese. Chinese content preserved in results; column names may be English—specify Chinese column names in description if needed.

Q: Are there data volume limits?
A: Limited by TinyFish API quota and sub-agent 6 tool call limit. Each entity max 6 scrape attempts per dataset; overall volume depends on API key tier. Free tier usually has daily call limits—choose tier by data volume.

Q: Can I customize scrape scope or specify websites?
A: Not currently. Orchestrator auto-discovers sources; users cannot specify URLs or filter sites. For precise source control, use Firecrawl or Scrapy. URL whitelist may come in future versions.

Q: How to ensure collected data doesn't infringe copyright?
A: BigSet scrapes publicly accessible web data only. Users must comply with target site robots.txt and laws. Tool does not bypass anti-scraping or paywalls—use only for legal data collection.

Q: What if make dev fails during deployment?
A: Common causes: old Docker or port conflict. Ensure Docker ≥20.10 and port 3500 free. Database startup failure—check .env keys complete and Postgres/Convex images pulled successfully.

Q: Can BigSet integrate into my app or workflow?
A: Yes. REST API (see GitHub) supports HTTP dataset creation, scrape triggering, result retrieval—integrate into data pipelines, BI tools, or custom apps.

9. Project Links

• GitHub: https://github.com/tinyfish-io/bigset
• TinyFish official site: https://tinyfish.io (API keys and docs)
• OpenRouter: https://openrouter.ai (LLM API keys)
• Clerk: https://clerk.com (authentication keys)