Depth control
Limit crawl depth, page count, URL patterns, query params, and subdomains.
Crawl API
Turn websites intoAI-ready datasets.
Submit a domain or sitemap and crawl pages into clean markdown, HTML, metadata, and structured records. Control depth, include/exclude rules, freshness, and delivery.
1 API
scrape, crawl, search, extract
195+
proxy countries
JSON
markdown, html, screenshot
Webhook
batch delivery
crawl.sh
curl -X POST https://api.ipbiubiu.com/v1/crawl \
-H "Authorization: Bearer $IPBIUBIU_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.example.com",
"max_depth": 3,
"limit": 500,
"formats": [
"markdown",
"metadata"
],
"webhook": "https://your-app.com/webhooks/ipbiubiu"
}'Response formats
pagesmarkdownlinksmetadatadataset
Built for production data pipelines
IPBiuBiu combines proxy routing, browser automation, retries, parsing, and delivery into one API surface.
LLM-ready markdown
Clean navigation, ads, footers, and boilerplate so the result is ready for RAG pipelines.
Sitemap optional
Start from one URL and discover pages automatically, or provide a sitemap for strict coverage.
Batch delivery
Track crawl status, stream partial results, and deliver final datasets by webhook.
Freshness policies
Cache stable pages, refresh changed pages, and schedule recurring crawls.
Proxy-aware crawling
Use geo-targeting and managed proxy routes for sites that vary by country or block crawlers.
One workflow from URL to dataset
Use the API for real-time calls, or run batch jobs that deliver clean results to your application.
Request
Send URLs, keywords, schemas, locations, and rendering options.
Collect
Proxy rotation, browser rendering, retries, and block handling run automatically.
Deliver
Receive JSON, markdown, screenshots, datasets, or webhook events.
Common use cases
Package the API around business outcomes, not crawler internals.
Knowledge base ingestion
Documentation search
Website monitoring
Competitor content tracking
Lead source discovery
RAG dataset refresh
Ship web data into your AI product faster
Start with the hosted API today, then scale into dedicated proxy pools, custom extractors, and enterprise delivery.