Firecrawl integration

Overview

Firecrawl is LLM-ready web scraping – fetches pages, follows internal links, and returns clean markdown + structured data. The Automize connector covers Scrape, Batch Scrape, Crawl, Map, Search, Extract, and async job polling.

• Regions: Hosted API at api.firecrawl.dev (US). Self-hosted is supported – point the connector at your instance via base URL override.
• Plans: Free has 500 scrapes/month and 1 crawl concurrency. Paid plans raise concurrency + monthly credits and unlock async batch + extract.

Setting up the connection

Auth type: Bearer Token.

1. In Firecrawl → Settings → API Keys, create a key (starts with fc-).
2. Paste it as the API token. The connector sends Authorization: Bearer <token>.
3. Optional: self-hosted? Override base URL to your instance – same auth model.

Find your credentials at https://www.firecrawl.dev/app/settings/api-keys.

Rate limits

Credits, not request rate. 1 scrape = 1 credit; structured extract = 5 credits/page; PDF scrape = 4 credits/page. Connector retries on 429 (concurrency cap) with backoff. Crawl jobs run async – no per-call latency budget concern.

Data model

Firecrawl's job graph:

    Sync       Scrape          single URL, returns content immediately
    Async      Crawl           crawl_id; walks links, multi-page
               Batch Scrape    batch_id; many URLs, one job
               Extract         extract_id; LLM structured extraction
    Helper     Map             URL discovery only (no fetch)
               Search          web search + on-the-fly scrape

Each async job has a status flow:

    scraping → completed (with data) | failed | cancelled

Async jobs return job_id immediately; data is paginated and lives
under a 'next' URL pointer. Polling cadence: 5-10s for Crawl, 10-30s
for Extract (LLM-bound, slower).

Output formats (request multiple in one call):

    markdown    LLM-ready (default, recommended for RAG)
    html        rendered HTML
    rawHtml     pre-render source
    screenshot  base64 PNG, full-page if fullPage=true
    links       extracted href[]
    extract     LLM-derived JSON matching your schema

Self-hosted Firecrawl serves the same API at your URL – set the
base URL on the credential, auth remains Bearer.

Available operations

10 operations available. Click any row to jump to its detail.

Operations

Batch Scrape

Scrape a list of URLs asynchronously

Tips

• Async – scrape many URLs in one job. urls[] + formats[] + (optional) scrapeOptions applied to every URL.
• Returns batch_id immediately. Poll Get Batch Scrape until status='completed'; data is paginated under 'next' URL when result set is large.
• Cheaper than looping Scrape – one job, one Firecrawl-side concurrency slot.

Cancel Crawl

Cancel an in-flight crawl job

Tips

• DELETE /v1/crawl/{crawl_id} – stops the crawl. Pages already scraped remain available via Get Crawl; in-flight pages are abandoned (credits already spent are not refunded).
• Use when you spot a crawl misconfigured (wrong includePaths, runaway link discovery) – saves credits on remaining pages.

Start Crawl

Crawl a site asynchronously; returns a job id

Tips

• POST /v1/crawl – async, returns crawl_id. limit caps total pages; includePaths/excludePaths take regex patterns.
• Poll Get Crawl every 5-10s; status flips to completed when all pages are done. data is paginated under next URL.

Structured Extract

Extract structured JSON from one or more URLs against a schema

Tips

• POST /v1/extract – uses an LLM to pull structured JSON matching your schema. urls=[...] + schema={...} (JSON Schema).
• Returns extract_id; poll Get Extract for status. Pricier per page than Scrape – use only when structure matters.

Get Batch Scrape Status

Get status and results of a batch scrape job

Tips

• Poll every 5-10s. status: scraping → completed (with data[]) | failed. completed jobs include success/total counters per URL.
• Partial results are visible while status='scraping' – early-finished URLs surface in data[] as the batch progresses.

Get Crawl Status

Get status and results for a crawl job

Tips

• GET /v1/crawl/{crawl_id} – same async pattern. completed/total counters, plus paginated data[] (use 'next' URL).
• Crawls auto-complete when limit is hit OR every reachable internal link is scraped, whichever comes first.

Get Extract Status

Check the status of an async extract job

Tips

• GET /v1/extract/{extract_id} – async polling for structured-extract jobs. status flow: processing → completed (with data matching your schema) | failed.
• Failures expose error.message – common: schema_too_strict (the model couldn't fit), page_not_found, content_extraction_blocked.

Map Site

Discover URLs reachable from a starting URL

Tips

• POST /v1/map – fast sitemap-style URL discovery without fetching pages. Use this to scope a Crawl before kicking it off.

Scrape URL

Scrape a single URL and return markdown/html/links/screenshot

Tips

• POST /v1/scrape – single URL, sync. formats=['markdown','html','rawHtml','screenshot','links','extract'].
• onlyMainContent=true strips nav/footer/ads – recommended for RAG ingest. Set waitFor in ms for SPAs.

Search Web

Web search with optional scraping of the top results

Tips

• Web search + scrape in one call. Combines results from a web index with on-the-fly page fetches.
• query + limit + (optional) scrapeOptions. Without scrapeOptions you get search results only (URLs + snippets); with it, each result is fetched + cleaned.

FAQ

Crawl vs Map vs Scrape – when do I use which?

Scrape: one known URL → content. Map: discover URLs on a domain (cheap, no content). Crawl: discover + scrape with depth/path limits (expensive).

Why is content missing for JS-heavy pages?

Set waitFor: 2000 (ms) and actions: [{type:'wait', milliseconds:2000}] for SPAs. For infinite scroll, use actions=[{type:'scroll'}]. PDFs need parsePDF=true.

Can I get screenshots?

Yes – formats: ['screenshot'] returns a base64 PNG. fullPage=true captures beyond the viewport.

How do I respect robots.txt?

Firecrawl honors robots.txt by default. Override with skipTlsVerification + a custom proxy only when scraping infrastructure you own.

Related connectors

• Openai
• Anthropic
• Cohere