Tools

Spider

The whole Spider REST API behind one wildcard route
View as Markdown

Proxies the entire Spider REST API — scrape, crawl, search, links, screenshot, transform, and the /data/* retrieval and management endpoints. Like ElevenLabs, it’s a whole API surface, so the gateway mounts a wildcard and forwards the path after /spider/ verbatim:

GET|POST|DELETE /spider/<api path>   → https://api.spider.cloud/<api path>

Because the path passes through unchanged, new Spider endpoints work without any gateway change. The methods mirror what Spider documents — POST for the action endpoints, GET for /data/* retrieval, and DELETE for /data/* deletion.

Common paths

PathPurpose
POST /spider/scrapeScrape a URL
POST /spider/crawlCrawl a site
POST /spider/searchSearch the web
POST /spider/linksExtract links
POST /spider/screenshotCapture a screenshot
GET, DELETE /spider/data/...Retrieve or delete stored data

What the gateway injects and strips

Spider authenticates with the Authorization: Bearer header — the same header you use for your gateway token. So the gateway overwrites it with the injected Spider key, rather than stripping it:

  • The Authorization header is overwritten with the secret Spider key. Your gateway token is consumed by the gateway and never reaches Spider.
  • Spider has no query-param spelling of the key, so nothing is dropped from the query string.

Examples

$curl -X POST "$GATEWAY/spider/scrape" \
>  -H "Authorization: Bearer $TOKEN" \
>  -H "Content-Type: application/json" \
>  -d '{"url": "https://example.com"}'

The response is Spider’s JSON, returned unchanged. For request and response fields, follow Spider’s API documentation — the gateway does not alter them.