> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.auxiliar.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.auxiliar.ai/_mcp/server.

# Spider

Proxies the entire [Spider](https://spider.cloud) REST API — scrape, crawl, search, links, screenshot, transform, and the `/data/*` retrieval and management endpoints. Like ElevenLabs, it's a whole API surface, so the gateway mounts a **wildcard** and forwards the path after `/spider/` verbatim:

```
GET|POST|DELETE /spider/<api path>   → https://api.spider.cloud/<api path>
```

Because the path passes through unchanged, **new Spider endpoints work without any gateway change**. The methods mirror what Spider documents — `POST` for the action endpoints, `GET` for `/data/*` retrieval, and `DELETE` for `/data/*` deletion.

## Common paths

| Path                           | Purpose                        |
| ------------------------------ | ------------------------------ |
| `POST /spider/scrape`          | Scrape a URL                   |
| `POST /spider/crawl`           | Crawl a site                   |
| `POST /spider/search`          | Search the web                 |
| `POST /spider/links`           | Extract links                  |
| `POST /spider/screenshot`      | Capture a screenshot           |
| `GET, DELETE /spider/data/...` | Retrieve or delete stored data |

## What the gateway injects and strips

Spider authenticates with the `Authorization: Bearer` header — the **same** header you use for your gateway token. So the gateway **overwrites** it with the injected Spider key, rather than stripping it:

* **The `Authorization` header is overwritten** with the secret Spider key. Your gateway token is consumed by the gateway and never reaches Spider.
* Spider has no query-param spelling of the key, so nothing is dropped from the query string.

## Examples

```bash
curl -X POST "$GATEWAY/spider/scrape" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

```bash
curl -X POST "$GATEWAY/spider/crawl" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "limit": 50}'
```

The response is Spider's JSON, returned unchanged. For request and response fields, follow [Spider's API documentation](https://spider.cloud/docs/api) — the gateway does not alter them.