Huntsmanby TulipFarm.ai

Turn any web page into clean Markdown.

A self-hosted scraper with a Firecrawl-compatible API, in one Go binary and one container. It renders JavaScript with Lightpanda and strips ads, trackers and cookie banners. No Redis, no Playwright, no Chrome.

Try it on any page

Example: tulipfarm.ai6,622 tokens · 84,800 as HTML

# One platform to build and run your business operations with agents.

Open source. Connected records, decisions, and actions across operations. Bring your existing tools and replace them at your own pace.

[Sign up for Cloud](https://app.tulipfarm.ai/register?utm_source=tulipfarm.ai&utm_medium=website&utm_campaign=signup&utm_content=home_hero)

[Try it yourself](https://tulipfarm.ai/try/) [view source](https://github.com/TulipFarm/tulipfarm)

3.0× the pages in flight of Firecrawl4.0× the pages per secondSame 8 vCPU machine; see the benchmarks

Benchmarks

Each engine self-hosted alone on the same ARM machine, scraping 1,000 live URLs at rising load.

Pages scraped at once

2 vCPU · 4 GB memory

  1. Huntsman454.58 pages/s · 90.9% scraped
  2. Firecrawl80.91 pages/s · 92.5% scraped
  3. Crawl4AICrashed or ran out of memory even at 2 pages at once

5.6× the pages in flight of Firecrawl, and 5.0× the pages per second.

4 vCPU · 8 GB memory

  1. Huntsman757.74 pages/s · 91.8% scraped
  2. Firecrawl151.58 pages/s · 92.2% scraped
  3. Crawl4AI151.35 pages/s · 80.6% scraped

5.0× the pages in flight of Firecrawl, and 4.9× the pages per second.

8 vCPU · 16 GB memory

  1. Huntsman12911.54 pages/s · 90.8% scraped
  2. Firecrawl432.90 pages/s · 92.3% scraped
  3. Crawl4AI221.62 pages/s · 81.8% scraped

3.0× the pages in flight of Firecrawl, and 4.0× the pages per second.

How we measured

1,000 random URLs from the test split of huntsman-scrape-dataset-v1, scraped live on ARM cloud machines with each engine self-hosted alone. A binary search found each engine’s exact limit: a level passes if the engine doesn’t crash, success stays within 10 points of its best level, and the median page takes under 3× as long as at 8 pages. Firecrawl ran in scrape-only mode on the small machine. One run per level; methods and raw results.

Why Huntsman

Everything an agent or search pipeline needs from a page, and nothing it doesn’t.

The page

HTML · 7,764 tokens

In the browser

No ads, trackers or cookie banners

150,000+ blocklist rules stop them before they load.

blocked
www.googletagmanager.com/gtm.js
rule
||googletagmanager.com^
list
easyprivacy.txt:3716

You get

Just the content

Navigation, footers and boilerplate are stripped out.

Markdown · 1,043 tokens
# Go 1.22 is released!

Eli Bendersky, on behalf of the Go team
6 February 2024

Today the Go team is thrilled to release Go 1.22, which you can
get by visiting the [download page](https://go.dev/dl/).

Get started

From an empty machine to your first page of Markdown in about a minute.

  1. Run the container

    The image is published for linux/amd64 and linux/arm64. The volume keeps the job queue and result cache across restarts.

    Run Huntsman with Docker
    docker run -d --name huntsman -p 127.0.0.1:8080:8080 \
  -v huntsman-data:/data ghcr.io/tulipfarm/huntsman:latest

  2. Scrape a page

    Send a URL to /v1/scrape and get back the page’s main content as Markdown, plus its links.

    Scrape example.com
    curl -s localhost:8080/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com"}'
    Example response, shortened
    {
  "success": true,
  "data": {
    "markdown": "This domain is for use in documentation examples …",
    "links": ["https://iana.org/help/example-domains"],
    "metadata": {
      "title": "Example Domain",
      "statusCode": 200,
      "engine": "lightpanda"
    }
  }
}

  3. Size it for your machine

    HUNTSMAN_WORKERS sets how many pages are scraped at once. By default Huntsman sizes it to the machine: 16 per CPU core, capped at one per 96 MB of memory, so 2 vCPU and 4 GB gets 32. Set it yourself to go further; these are the highest values that held up in the benchmark above. Memory grows over runs of many hours, so leave headroom for long-running servers.

    MachineTested max
    Small 2 vCPU · 4 GB45
    Medium 4 vCPU · 8 GB75
    Large 8 vCPU · 16 GB129
    Run Huntsman with 64 workers
    docker run -d --name huntsman -p 127.0.0.1:8080:8080 \
  -v huntsman-data:/data \
  -e HUNTSMAN_WORKERS=64 \
  ghcr.io/tulipfarm/huntsman:latest

  4. Lock it down for production

    Pin a version and set HUNTSMAN_TOKEN. Every endpoint except /health then needs the token as a bearer header. Private, loopback and link-local addresses are blocked by default.

    Run a pinned version with a token
    docker run -d --name huntsman -p 8080:8080 \
  -v huntsman-data:/data \
  -e HUNTSMAN_TOKEN="$(openssl rand -hex 32)" \
  ghcr.io/tulipfarm/huntsman:0.1.0
    Scrape with the token
    curl -s https://scraper.example.com/v1/scrape \
  -H "Authorization: Bearer $HUNTSMAN_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "onlyMainContent": true}'

Reference

API

Requests and responses follow Firecrawl’s /v1 shapes, so most clients only need a new base URL. Unknown fields are rejected, so typos fail loudly.

EndpointWhat it does
POST /v1/scrapeScrape one URL and return Markdown and links
POST /v1/batchQueue 1–1,000 URLs with shared scrape options
GET /v1/jobs/{id}Status and result of a queued job
POST /v1/crawlCrawl a site up to a page limit and depth
GET /v1/crawl/{id}Crawl progress and pages, paged with ?offset=
GET /healthLiveness and storage check

Coming from Firecrawl? Keep calling /v1/scrape. Huntsman deliberately does less: it returns Markdown and links only, with no screenshots, HTML, JSON extraction, actions or search.

Crawl a site

Start a crawl and read its pages
curl -s localhost:8080/v1/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "limit": 50, "maxDepth": 2}'
# → {"success": true, "id": "…"}

curl -s 'localhost:8080/v1/crawl/<id>?offset=0'

Configuration

Everything is an environment variable. The most used ones are below; the full list is in the README.

VariableDefaultMeaning
HUNTSMAN_WORKERS16 per CPUPages scraped at once, capped at one per 96 MB of memory. Container limits count (see step 3 above).
HUNTSMAN_PER_HOST2Most pages scraped at once from a single site, so one site isn’t flooded. Max 64.
HUNTSMAN_TOKENemptyBearer token; empty turns auth off
HUNTSMAN_TOKENSemptyNamed keys, name:token,... One per client, so you can revoke one without the others
HUNTSMAN_CACHE_TTL1hDefault result cache age, up to 168h
HUNTSMAN_OBEY_ROBOTStrueRespect robots.txt
HUNTSMAN_QUEUE_CAPACITY10000Queued jobs before 429

Run it on your own machine.

One Go binary and one container. No Redis, no Playwright, no Chrome.

Get startedGitHub
Run Huntsman with Docker
docker run -d --name huntsman -p 127.0.0.1:8080:8080 \
  -v huntsman-data:/data ghcr.io/tulipfarm/huntsman:latest