Tools

Crawlbase

The Crawling API, with the right token injected by route
View as Markdown

Proxies Crawlbase’s Crawling API, injecting the secret token query parameter:

GET|POST /crawlbase      → https://api.crawlbase.com/?token=<normal>&<your query>
GET|POST /crawlbase/js   → https://api.crawlbase.com/?token=<javascript>&<your query>

Crawlbase issues two tokens, and the route picks which one is injected:

RouteTokenBehavior
/crawlbaseNormalFetches static HTML
/crawlbase/jsJavaScriptLoads the page in a real headless browser

All controls travel in the query string — the target url and any Crawlbase parameter (format, country, page_wait, …). Your query string is passed through byte-for-byte.

What the gateway injects and strips

  • token is injected first, ahead of your parameters, so a partially-encoded target URL can’t swallow it. The route (/crawlbase vs /crawlbase/js) selects which token.
  • A client-supplied token query param is dropped (in any encoding), so the injected token can’t be overridden.
  • The Authorization header is stripped before forwarding — Crawlbase has no use for your gateway token.

Examples

$curl "$GATEWAY/crawlbase?url=https://example.com" \
>  -H "Authorization: Bearer $TOKEN"

The response is Crawlbase’s, returned unchanged. For the full parameter set, see Crawlbase’s documentation.