The Agent Perception Layer

Who’s winning the race to give AI eyes. A technical reference for builders.

Mar 26, 2026

Week of March 26, 2026. All data sourced from PT-Edge‘s live tracking of 166K+ AI repos across GitHub, PyPI, npm, Docker Hub, HuggingFace, and Hacker News.

MCP gave agents hands — tools they can operate. Now a parallel infrastructure race is playing out to give agents eyes — the ability to perceive, read, and navigate the open internet the way a human researcher would.

The Problem: Agents Are Blind

Foundation models are powerful reasoners but terrible perceivers. An LLM can analyze a dataset brilliantly, but it can’t go get the dataset. It can synthesize Reddit threads into insight, but it can’t read Reddit. It can draft a competitive analysis, but it can’t visit a competitor’s website.

The MCP ecosystem solved tool use — how agents call functions, query databases, trigger workflows. But perception — how agents acquire information from the open internet — is a separate, harder problem. It involves authentication, anti-bot defenses, JavaScript rendering, rate limiting, format conversion, and platform-specific quirks.

A new infrastructure layer is emerging to solve this. It’s fragmenting into four competing approaches, each with different tradeoffs.

The Four Approaches

1. Scrape-First: Turn the Web Into Structured Data

Build a purpose-built engine that fetches web pages, renders JavaScript, and converts the result into clean markdown or structured data. The agent never touches the raw web — it gets pre-processed, LLM-ready content.

2. Browser-First: Give the Agent a Real Browser

Hand the agent a headless Chromium instance it can drive programmatically. Click buttons, fill forms, navigate pages, extract content. The agent operates the browser the same way a human would.

3. CLI-First: Compose Existing Free Tools

Don’t build a scraper or a browser engine. Instead, wire together existing CLI tools — yt-dlp for YouTube, gh for GitHub, bird for Twitter — and teach the agent which tool handles which platform. The agent calls bash commands.

4. API-First: Use Official Platform APIs

The “proper” way. Use Twitter’s API, YouTube’s Data API, Reddit’s API. Rate-limited, expensive, but sanctioned. This is mostly what commercial agent platforms do under the hood.

The Contenders

Firecrawl — firecrawl/firecrawl

98,391 stars, +3,419 this week (+3.6%) · TypeScript · AGPL-3.0

The star count leader by a wide margin. “Turn entire websites into LLM-ready markdown or structured data.” Scrape-first approach, with a hosted API and open-source core. Commercial product with open-source distribution — the stars are partly a marketing engine.

The numbers behind the stars:

90,531 monthly PyPI downloads (firecrawl-py)
153 commits in 30 days, from effectively 1 contributor
6,628 forks on 98K stars = 6.7% fork ratio (lowest in the group)
6 downstream dependents: gpt-researcher, camel-ai, PySpur, PPTAgent, symbolicai, TrustRAG
236 open issues

What this tells you: Firecrawl has the most visibility but the weakest builder signal relative to its size. The low fork ratio and low dependency count suggest people are using it as a hosted service, not embedding it in their own projects. The single-contributor commit pattern confirms this is company-driven. If you want a managed scraping API you call from your agent, Firecrawl is the obvious choice. If you want to build on top of something, look elsewhere.

Crawl4AI — unclecode/crawl4ai

62,581 stars, +378 this week (+0.6%) · Python · Apache-2.0

Star growth has plateaued. Usage has not.

The numbers that matter:

1,463,419 monthly PyPI downloads — by far the highest in the category
385,666 weekly downloads, trending up (345K → 386K over the week)
78 contributors
57 commits in 30 days
11 downstream dependents: camel-ai, PraisonAI, ag2, Rankify, and others
6,391 forks on 63K stars = 10.2% fork ratio
37 open issues (remarkably low for a project this size)

What this tells you: Crawl4AI is the volume winner. 1.46M monthly installs means this is load-bearing production infrastructure for a lot of teams. The Apache-2.0 license (vs. Firecrawl’s AGPL-3.0) removes the commercial integration headache. The 78-contributor base and low issue count suggest a mature, stable project. The star growth plateau with accelerating downloads is the classic pattern of a tool that’s crossed from “interesting” to “standard” — people stopped starring it because it’s just what you use.

For builders: If you need a Python library to crawl web pages and get LLM-ready output, Crawl4AI is the default choice. The download numbers make the argument for you.

browser-use — browser-use/browser-use

84,496 stars, +3,246 this week (+4.0%) · Python · MIT

“Make websites accessible for AI agents.” Browser-first — gives the agent a real Playwright-driven browser.

The numbers that matter:

4,099,498 monthly downloads — the highest raw number in the group
37 downstream dependencies — more than 3x the next closest (Crawl4AI at 11)
100 contributors — the largest contributor base
9,788 forks on 84K stars = 11.6% fork ratio (highest in the group)
155 commits in 30 days
7 releases in 30 days
193 open issues
Key dependents: ag2 (Microsoft’s AutoGen successor)

What this tells you: browser-use is what other projects build on. The 37 downstream dependencies is the single most important number in this entire analysis — it means the ecosystem has chosen browser-use as the base layer for browser-driven agent interaction. The 11.6% fork ratio (highest) means developers aren’t just using it, they’re customizing it. 100 contributors means a real open-source community, not a company project with a GitHub repo. The MIT license makes it embeddable anywhere.

For builders: If you’re building an agent that needs to interact with websites — not just read them, but click, fill forms, navigate — browser-use is the community’s default. The dependency graph has already decided this.

Scrapling — D4Vinci/Scrapling

32,876 stars, +1,766 this week (+5.7%) · Python · BSD-3-Clause

“An adaptive web scraping framework that handles everything from a single request to a full-scale crawl.”

The numbers:

361,197 monthly downloads
66 commits, 13 contributors in 30 days
2,639 forks on 33K stars = 8.0% fork ratio
6 downstream dependents
6 open issues (the lowest in the group)
2 releases in 30 days

What this tells you: Scrapling is the sleeper. It has the fastest star growth rate this week among the established players, solid downloads, and an absurdly low issue count that suggests a well-maintained, stable codebase. The BSD-3-Clause license is the most permissive in the group. It’s not trying to be a platform — it’s a scraping library that does its job cleanly. The “adaptive” angle handles anti-bot detection, stealth mode, and dynamic content without the overhead of a full browser.

For builders: Scrapling occupies the middle ground between Crawl4AI (page-level extraction) and browser-use (full browser control). If you need scraping that handles JavaScript and anti-bot measures but don’t need the agent to actually drive a browser, this is the sweet spot. Watch it — the growth rate suggests it hasn’t peaked yet.

Agent-Reach — Panniantong/Agent-Reach

10,787 stars, +864 this week (+8.7%) · Python · MIT

“Give your AI agent eyes to see the entire internet. One CLI, zero API fees.” CLI-first — composes existing free tools instead of building its own engine.

The numbers:

0 PyPI downloads (it’s a CLI scaffold, not a library — installed via git clone)
0 downstream dependencies
74 commits, 14 contributors in 30 days
789 forks on 11K stars = 7.3% fork ratio
15 open issues
3 releases in 30 days

How it actually works:

14+ platforms supported through composable upstream tools
Web pages via Jina Reader (free web-to-markdown)
YouTube via yt-dlp (supports 1800+ video sites)
Twitter/X via bird CLI (cookie auth, bypasses paid API)
GitHub via gh CLI (official, free)
Reddit via Exa search (AI semantic search)
XiaoHongShu, Douyin via MCP-based services
WeChat via camoufox headless browser
RSS via feedparser
Each platform is a pluggable channel file implementing a check() method
Agents learn available tools via a SKILL.md file — no hardcoded API knowledge
The agent calls upstream tools directly: curl https://r.jina.ai/URL, bird search "keyword", yt-dlp --dump-json URL

The architectural bet: Agent-Reach doesn’t own any of its capabilities. It’s a thin coordination layer over other people’s tools. This means it can add a new platform in one file without building a scraper, but it’s also fragile — if bird CLI breaks or Jina Reader changes its API, Agent-Reach breaks. The explicit tradeoff is breadth over depth, speed of integration over reliability.

What this tells you: Agent-Reach is the most architecturally provocative project in the group. It’s making a bet that the web perception problem is a composition problem, not an engineering problem. The 8.7% weekly star growth (fastest in the group) suggests the idea resonates. But zero package downloads and zero dependencies means nobody’s building on it yet — it’s still in the “people are starring it” phase.

For builders: Watch this one. If the CLI-first pattern works at scale, it could become the default way to give agents internet access — because it’s the cheapest to build and the fastest to extend. The risk is reliability and maintenance burden across 14+ upstream dependencies you don’t control.

Skyvern — Skyvern-AI/skyvern

20,963 stars, +108 this week (+0.5%) · Python · AGPL-3.0

“Automate browser-based workflows with AI.” Browser-first, with a focus on visual understanding (computer vision + LLM).

The numbers:

0 PyPI downloads, 0 downstream dependencies
234 commits in 30 days (highest in the group)
1 contributor (effectively a solo project)
8 releases in 30 days (highest in the group)
1,870 forks, 146 open issues

What this tells you: Skyvern is in full sprint mode — 234 commits and 8 releases in 30 days from a single contributor is an intense pace. The AGPL license and single-contributor pattern suggest a company building in the open. Star growth has slowed to 0.5%/week while commit pace is the highest in the group, which means the team is heads-down building, not optimizing for visibility. Zero downloads and zero dependencies means adoption hasn’t kicked in yet.

For builders: Skyvern’s differentiator is visual understanding — it uses computer vision to interpret what’s on screen, not just DOM parsing. This matters for sites with heavy dynamic content, iframes, or canvas-rendered UIs. But the AGPL license and single-maintainer risk are real considerations for production use.

The Ecosystem Around Them

These tools don’t exist in isolation. A supporting cast is emerging:

Composio (27.5K stars) — 1000+ tool integrations for agents with auth management. Powers the authentication layer that tools like Agent-Reach skip.
Jina Reader — Used by Agent-Reach and others as the free web-to-markdown engine. Becoming a de facto standard for “give me this URL as clean text.”
yt-dlp — The universal video/audio extraction tool. Used by Agent-Reach for YouTube, but supports 1800+ sites. Already ubiquitous in the scraping world, now being adopted by the agent ecosystem.
khoj (33.6K stars, +151/week) — Self-hostable AI second brain. “Get answers from the web or your docs.” Combines search, web reading, and knowledge management.
Deer Flow (46.9K stars, +15,166/week) — ByteDance’s “SuperAgent harness” that includes web research as part of a broader agent orchestration framework. The most stars gained this week in the adjacent space.

What the Data Says Is Winning

Five metrics, five different answers:

Downloads (who’s actually being used)

browser-use — 4.1M/month
Crawl4AI — 1.46M/month
Scrapling — 361K/month
Firecrawl — 91K/month
Agent-Reach, Skyvern — 0 (different distribution model)

Downstream dependencies (who’s being built on)

browser-use — 37
Crawl4AI — 11
Firecrawl — 6
Scrapling — 2
Agent-Reach, Skyvern — 0

Fork ratio (builders vs. stargazers)

browser-use — 11.6%
Crawl4AI — 10.2%
Skyvern — 8.9%
Scrapling — 8.0%
Agent-Reach — 7.3%
Firecrawl — 6.7%

Contributor diversity (community health)

browser-use — 100
Crawl4AI — 78
Agent-Reach — 14
Scrapling — 13
Firecrawl — 1
Skyvern — 1

Star growth rate (momentum)

Agent-Reach — +8.7%
Scrapling — +5.7%
browser-use — +4.0%
Firecrawl — +3.6%
Crawl4AI — +0.6%
Skyvern — +0.5%

The Verdict

browser-use is winning today. It leads on the metrics that matter most for long-term dominance: downstream dependencies (37), contributor diversity (100), fork ratio (11.6%), and raw downloads (4.1M/month). The ecosystem has chosen it as the base layer. When other projects need browser automation for agents, they import browser-use. That network effect is hard to reverse.

Crawl4AI is the volume king. 1.46M monthly PyPI installs, accelerating week over week. It’s crossed from “interesting project” to “standard infrastructure.” If you just need to turn web pages into LLM-ready text, this is what people reach for. The Apache-2.0 license makes it the safe corporate choice.

Scrapling is the sleeper bet. Fastest star growth among the established players (+5.7%), lowest issue count, and a clean BSD license. It occupies the pragmatic middle ground — more capable than a simple fetcher, lighter than a full browser. The kind of project that becomes quietly essential.

Agent-Reach is the most interesting architectural bet. The CLI-first, compose-don’t-build approach is elegant and fast to extend. It has the highest momentum (+8.7% star growth) but zero adoption metrics yet. The question isn’t whether the idea is good — it is — but whether a thin scaffolding over 14 upstream tools can be reliable enough for production use.

Firecrawl has the brand but not the builder signal. 98K stars and only 6 downstream dependents tells you it’s a product, not a platform. If you want a managed service, it’s great. If you want to build on something, the data points elsewhere.

Skyvern is building hard but hasn’t found distribution. The highest commit pace in the group with the lowest star growth. The visual understanding angle is differentiated, but AGPL + solo maintainer + zero downloads is a tough starting position.

What This Means for Builders

The perception layer is stratifying. You don’t pick one tool — you pick a layer:

Need to read a web page? → Crawl4AI
Need to interact with a web page? → browser-use
Need to scrape at scale with anti-bot handling? → Scrapling
Need quick access to 14+ platforms with no infra? → Agent-Reach
Need visual browser understanding? → Skyvern
Need a managed API? → Firecrawl

The browser-first approach is winning the platform war. browser-use’s dependency count means it’s becoming the Playwright of agent perception. If you’re building agent infrastructure, supporting browser-use integration is increasingly table stakes.

Downloads beat stars. Crawl4AI has half the stars of Firecrawl and 16x the downloads. When evaluating tools for production use, check PyPI/npm numbers, not GitHub stars. Stars measure awareness. Downloads measure adoption. Dependencies measure trust.

License matters. browser-use (MIT) and Crawl4AI (Apache-2.0) can be embedded in commercial products freely. Firecrawl and Skyvern (both AGPL-3.0) cannot without releasing your source code or buying a commercial license. For an infrastructure layer that gets embedded into other products, this is a deciding factor. The data already reflects this — the permissively licensed tools have more downstream dependents.

The CLI-first pattern deserves serious attention. Agent-Reach’s approach — compose existing tools, teach the agent bash — is the cheapest possible solution to the perception problem. If reliability can be solved (and that’s a real if), this pattern could win on economics alone. For solo builders and small teams who can’t maintain a scraping engine, it’s already the pragmatic choice.

Agents need to be internet citizens, not just tool-callers. The MCP ecosystem gives agents hands. This layer gives them eyes. The next gap is memory — persistent state across sessions that lets an agent build knowledge over time rather than starting from scratch. Watch for the memory layer to be the next infrastructure battleground.

Generated March 26, 2026 from PT-Edge live data. Star counts, downloads, and growth figures cover the 7-day window ending March 26, 2026.

Discussion about this post

Ready for more?