Tools

Scrapfly

Web scraping API with the key injected for you
View as Markdown

Proxies Scrapfly’s Web Scraping API, injecting the secret key query parameter:

GET|POST|PUT|PATCH|HEAD /scrapfly   → https://api.scrapfly.io/scrape?key=<secret>&<your query>

All controls travel in the query string — the target url and any Scrapfly parameter (render_js, asp, country, proxy_pool, format, …). Your query string is passed through byte-for-byte.

Scrapfly uses the request method as the method it runs against the scraped URL, so GET, POST, PUT, PATCH, and HEAD are all forwarded. (OPTIONS, a CORS preflight, is not a scrape and is not accepted.)

What the gateway injects and strips

  • key is injected first, ahead of your parameters, so a partially-encoded target URL can’t swallow it.
  • A client-supplied key query param is dropped (in any encoding), so the injected key can’t be overridden.
  • The Authorization header is stripped before forwarding — Scrapfly has no use for your gateway token.

Examples

$curl "$GATEWAY/scrapfly?url=https://example.com" \
>  -H "Authorization: Bearer $TOKEN"

The response is Scrapfly’s, returned unchanged. For the full parameter set, see Scrapfly’s documentation.