# Feeds

Synthient publishes every data stream as **Parquet** snapshots (hourly rolls plus a daily rollup) alongside the [real-time NDJSON streams](https://docs.synthient.com/enterprise/firehose). Snapshots are served as 24-hour presigned R2 URLs and are accompanied by full metadata: size, row count, SHA-256 checksum, and parquet schema.

## Streams

The same seven stream identifiers are used across exports and real-time streaming.

| Stream           | Description                                                              |
| ---------------- | ------------------------------------------------------------------------ |
| `proxies`        | Proxy IP observations: residential, datacenter, and mobile.              |
| `anonymizers`    | VPN, Tor, and relay-class anonymizer ranges.                             |
| `torrents`       | DHT and tracker peer sightings with info hash, metadata, and peers.      |
| `honeypot_http`  | HTTP request captures from Helios honeypot sensors.                      |
| `honeypot_https` | TLS ClientHello captures from Helios honeypot sensors.                   |
| `honeypot_dns`   | DNS resolution observations from Helios honeypot tunnels.                |
| `honeypot_adb`   | Android Debug Bridge shell commands captured by Helios honeypot sensors. |

---

## Authentication

All endpoints require your API key in the `x-api-key` header and are served from `https://api.synthient.com` under `/api/v4`. Each stream is gated by its own `*_FEED` scope, such as `PROXY_FEEDS`, `ANONYMIZERS_FEED`, or `HONEYPOT_HTTP_FEED`.

```bash
curl -G https://api.synthient.com/api/v4/feeds/proxies/export \
  -H "x-api-key: $SYNTHIENT_API_KEY"
```

### Hourly snapshots and rollups

Hourly snapshots are addressable at `/api/v4/feeds/{stream}/export/{date}/{hour}` (with a matching `/meta` variant). **Only the current UTC date** is addressable for hourlies. At **00:30 UTC** each day, the previous day's hourlies are rolled up into the daily snapshot at `/export/{date}` and the per-hour artifacts are deleted. Use the snapshot id `latest` to track the most recent hourly without committing to a specific hour.

The Helios sensors live under their `helio/` URL prefix, for example `/api/v4/feeds/helio/http/export/{date}` (and `/feeds/helio/https/...` for TLS captures), but they appear in this list endpoint under the `honeypot_http`, `honeypot_https`, `honeypot_dns`, and `honeypot_adb` stream identifiers.

---

## List snapshots (GET /api/v4/feeds/\{stream}/export)

Returns one page of available daily and hourly Parquet snapshots for the given stream. Pages are ordered newest-first and capped at 500 rows; pass `next_cursor` from the response back as `cursor` to fetch the next page.

### Path parameters

- `stream` `string` — One of `proxies`, `anonymizers`, `torrents`, `honeypot_http`, `honeypot_https`, `honeypot_dns`, or `honeypot_adb`.

### Query parameters

- `limit` `integer` — Page size. Defaults to `100`. Values above `500` are clamped.
- `cursor` `string` — Opaque pagination token returned in `next_cursor` of the previous response. Omit on the first page.

### Response

- `stream` `string` — The stream identifier the snapshots belong to.
- `feeds` `array<object>` — Page of snapshots, newest-first.
- `feeds[].kind` `string` — Either `hourly` or `daily`.
- `feeds[].date` `string` — UTC `YYYY-MM-DD` of the snapshot.
- `feeds[].hour` `integer` — `0`–`23` for hourly snapshots; omitted for daily rollups.
- `feeds[].size_bytes` `integer` — Parquet file size in bytes.
- `feeds[].row_count` `integer` — Number of rows in the parquet file.
- `feeds[].checksum` `string` — Hex-encoded SHA-256 of the parquet file bytes.
- `feeds[].id` `string` — Stable identifier: `YYYY-MM-DD` for daily rollups, `YYYY-MM-DD/HH` for past hourlies, and the literal `latest` for the most recent hourly snapshot.
- `feeds[].created_at` `integer` — Unix timestamp in **seconds** when the snapshot was indexed server-side.
- `feeds[].download_path` `string` — Relative API path to follow for the 307 download redirect.
- `next_cursor` `string` — Pagination token. Absent on the final page.

**Request** `GET /api/v4/feeds/{stream}/export`

```bash title="cURL"
curl -G https://api.synthient.com/api/v4/feeds/proxies/export \
  -H "x-api-key: $SYNTHIENT_API_KEY" \
  --url-query "limit=50"
```

```bash title="CLI"
synthient feeds snapshots proxies --limit 50
```

**Response**

```json title="200"
{
  "stream": "proxies",
  "feeds": [
    {
      "kind": "hourly",
      "date": "2026-05-07",
      "hour": 22,
      "size_bytes": 620273951,
      "row_count": 43141039,
      "checksum": "fd6c002ad6c6ae73344c2fdf1cb535a303d90edf9252358e0d30a44231649d36",
      "id": "latest",
      "created_at": 1778195206,
      "download_path": "/api/v4/feeds/proxies/export/latest"
    },
    {
      "kind": "hourly",
      "date": "2026-05-07",
      "hour": 21,
      "size_bytes": 612481020,
      "row_count": 42811027,
      "checksum": "d3fb2ec3de2bbf6af66b6028afe668e365c4113133757153ad975be93609d1ea",
      "id": "2026-05-07/21",
      "created_at": 1778192818,
      "download_path": "/api/v4/feeds/proxies/export/2026-05-07/21"
    },
    {
      "kind": "daily",
      "date": "2026-05-06",
      "size_bytes": 14721234567,
      "row_count": 1024411203,
      "checksum": "a3e4b804e1112ed2dd11e4b01a25a637ceb3abae23fa1f7c8503d567639a11a2",
      "id": "2026-05-06",
      "created_at": 1778115429,
      "download_path": "/api/v4/feeds/proxies/export/2026-05-06"
    }
  ],
  "next_cursor": "eyJkIjoiMjAyNi0wNS0wNyIsImgiOjIxfQ.GECAibkM_hEUt0ixRMKzzQ"
}
```

```json title="400"
{
  "type": "https://docs.synthient.com/errors/bad-request",
  "title": "Bad request",
  "status": 400,
  "detail": "invalid cursor"
}
```

```json title="403"
{
  "type": "https://docs.synthient.com/errors/forbidden",
  "title": "Forbidden",
  "status": 403,
  "detail": "Insufficient scope for this resource"
}
```

---

## Download a snapshot (GET /api/v4/feeds/\{stream}/export/\{date})

Returns a `307` redirect to a presigned R2 URL that is valid for **24 hours**. The 307 (rather than 301) is intentional: the URL is minted per request and expires, so intermediaries must not cache it. Follow the redirect to download the parquet file. Use the date string `latest` for the most recent hourly snapshot, or append a specific hour at `/{date}/{hour}` for a particular hourly within the current UTC day.

### Path parameters

- `stream` `string` — One of the seven stream identifiers.
- `date` `string` — Either a `YYYY-MM-DD` UTC date for a daily rollup, or the literal string `latest` for the most recent hourly snapshot.
- `hour` `integer` — Optional. `0`–`23` UTC hour, addressed as `/api/v4/feeds/{stream}/export/{date}/{hour}`. Only the **current UTC date** is addressable for hourlies.

### Responses

| Code  | Meaning                                                                                   |
| ----- | ----------------------------------------------------------------------------------------- |
| `307` | Redirect to a 24-hour presigned download URL.                                             |
| `400` | Bad date format, hour out of range (0..23), or hour requested for a non-current UTC date. |
| `401` | Missing or invalid API key.                                                               |
| `403` | API key lacks the per-stream feed scope.                                                  |
| `404` | No snapshot exists for the requested date/stream.                                         |

**Request** `GET /api/v4/feeds/{stream}/export/{date}`

```bash title="Daily snapshot"
# Follow the 307 with -L and write to a file
curl -L -o proxies-2026-05-06.parquet \
  https://api.synthient.com/api/v4/feeds/proxies/export/2026-05-06 \
  -H "x-api-key: $SYNTHIENT_API_KEY"
```

```bash title="CLI"
synthient download proxies proxies-2026-05-06.parquet --date 2026-05-06
```

```bash title="Latest hourly"
# Most recent hourly snapshot
curl -L -o proxies-latest.parquet \
  https://api.synthient.com/api/v4/feeds/proxies/export/latest \
  -H "x-api-key: $SYNTHIENT_API_KEY"
```

```bash title="CLI"
synthient download proxies proxies-latest.parquet
```

```bash title="Specific hour"
# A specific hourly within the current UTC day
curl -L -o proxies-2026-05-07-21.parquet \
  https://api.synthient.com/api/v4/feeds/proxies/export/2026-05-07/21 \
  -H "x-api-key: $SYNTHIENT_API_KEY"
```

```bash title="CLI"
synthient download proxies proxies-2026-05-07-21.parquet --date 2026-05-07 --hour 21
```

---

## Snapshot metadata (GET /api/v4/feeds/\{stream}/export/\{date}/meta)

Returns JSON metadata for a parquet snapshot: SHA-256 checksum, byte size, row count, parquet schema with column names and types, and the canonical date.

### Response

- `stream` `string` — The stream identifier the snapshot belongs to.
- `kind` `string` — Either `"hourly"` or `"daily"`.
- `hour` `integer` — `0`–`23` for hourly snapshots; omitted for daily rollups.
- `id` `string` — Stable identifier: `YYYY-MM-DD` for daily rollups, `YYYY-MM-DD/HH` for past hourlies, `latest` for the most recent hourly snapshot.
- `format` `string` — Always `"parquet"`.
- `date` `integer` — Unix timestamp in **seconds** for the snapshot instant. Daily rollups use the day's midnight UTC; hourly snapshots use the hour mark.
- `created_at` `integer` — Unix timestamp in **seconds** when the snapshot was indexed server-side.
- `size` `integer` — File size in bytes.
- `rows` `integer` — Number of rows in the parquet file.
- `checksum` `string` — Hex-encoded SHA-256 of the parquet file bytes.
- `schema.fields` `array<object>` — One entry per column in the parquet footer.
- `schema.fields[].name` `string` — Column name.
- `schema.fields[].type` `string` — Column type: one of `string`, `int64`, `uint32`, `uint64`, `bool`, `bytes`.

**Request** `GET /api/v4/feeds/{stream}/export/{date}/meta`

```bash title="cURL"
curl -G https://api.synthient.com/api/v4/feeds/proxies/export/2026-05-02/meta \
  -H "x-api-key: $SYNTHIENT_API_KEY"
```

```bash title="CLI"
synthient feeds meta proxies 2026-05-02
```

**Response**

```json title="200"
{
  "stream": "proxies",
  "kind": "hourly",
  "hour": 22,
  "id": "latest",
  "format": "parquet",
  "date": 1778191200,
  "created_at": 1778195206,
  "size": 620273951,
  "rows": 43141039,
  "checksum": "fd6c002ad6c6ae73344c2fdf1cb535a303d90edf9252358e0d30a44231649d36",
  "schema": {
    "fields": [
      { "name": "ip",           "type": "string" },
      { "name": "provider",     "type": "string" },
      { "name": "type",         "type": "string" },
      { "name": "timestamp",    "type": "int64" },
      { "name": "country_code", "type": "string" },
      { "name": "asn",          "type": "uint32" }
    ]
  }
}
```

---

## Paginating snapshot listings

The list endpoint returns up to 500 snapshots per page, newest-first, with a `next_cursor` you pass back as `cursor` on the next call. When the response no longer carries a `next_cursor`, you've reached the end.

**Page through every snapshot**

```python title="Python"
import os, requests

URL = "https://api.synthient.com/api/v4/feeds/proxies/export"
HEADERS = {"x-api-key": os.environ["SYNTHIENT_API_KEY"]}

cursor = None
while True:
    params = {"limit": 500, **({"cursor": cursor} if cursor else {})}
    page = requests.get(URL, headers=HEADERS, params=params, timeout=30).json()
    for snap in page["feeds"]:
        handle(snap)
    cursor = page.get("next_cursor")
    if not cursor:
        break
```

```go title="Go"
base := "https://api.synthient.com/api/v4/feeds/proxies/export"
cursor := ""
for {
    u := base + "?limit=500"
    if cursor != "" {
        u += "&cursor=" + url.QueryEscape(cursor)
    }
    req, _ := http.NewRequest("GET", u, nil)
    req.Header.Set("x-api-key", os.Getenv("SYNTHIENT_API_KEY"))
    resp, err := http.DefaultClient.Do(req)
    if err != nil { log.Fatal(err) }

    var page struct {
        Feeds      []map[string]any `json:"feeds"`
        NextCursor string           `json:"next_cursor"`
    }
    json.NewDecoder(resp.Body).Decode(&page)
    resp.Body.Close()

    for _, snap := range page.Feeds { handle(snap) }
    if page.NextCursor == "" { break }
    cursor = page.NextCursor
}
```

---

## Response codes

Snapshot downloads answer with a `307` redirect: follow the `Location` header to a 24-hour presigned URL. Every other status follows the shared contract, including `404` when no snapshot exists for the requested date and stream. See [Errors](https://docs.synthient.com/errors) for the full reference and retry guidance.