Feeds

Synthient publishes every data stream as Parquet snapshots (hourly rolls plus a daily rollup) alongside the real-time NDJSON streams. Snapshots are served as 24-hour presigned R2 URLs and are accompanied by full metadata: size, row count, SHA-256 checksum, and parquet schema.

Streams

The same eight stream identifiers are used across exports and real-time streaming.

Authentication

All endpoints require your API key in the x-api-key header and are served from https://api.synthient.com under /api/v4. Each stream is gated by its own *_FEED scope, such as PROXY_FEEDS, ANONYMIZERS_FEED, or HONEYPOT_HTTP_FEED.

curl -G https://api.synthient.com/api/v4/feeds/proxies/export \
  -H "x-api-key: $SYNTHIENT_API_KEY"

Hourly snapshots and rollups

Hourly snapshots are addressable at /api/v4/feeds/{stream}/export/{date}/{hour} (with a matching /meta variant). An hour stays addressable until its day's daily rollup completes, shortly after 00:30 UTC the next day. The rollup then merges that day's hourlies into the daily snapshot at /export/{date} and deletes the per-hour artifacts, and requests for those hours return 400. Use the snapshot id latest to track the freshest snapshot without committing to a specific hour: the newest hourly during the day, or the daily rollup once that day's hourlies are rolled up.

The Helios sensors live under their helio/ URL prefix, for example /api/v4/feeds/helio/http/export/{date} (and /feeds/helio/https/... for TLS captures), but the stream field of export responses reports them as honeypot_http, honeypot_https, honeypot_dns, and honeypot_adb stream identifiers.

List snapshots

Returns one page of available daily and hourly Parquet snapshots for the given stream. Pages are ordered newest-first and capped at 500 rows; pass next_cursor from the response back as cursor to fetch the next page.

curl -G https://api.synthient.com/api/v4/feeds/proxies/export \
  -H "x-api-key: $SYNTHIENT_API_KEY" \
  --url-query "limit=50"

{
  "stream": "proxies",
  "feeds": [
    {
      "kind": "hourly",
      "date": "2026-05-07",
      "hour": 22,
      "size_bytes": 620273951,
      "row_count": 43141039,
      "checksum": "fd6c002ad6c6ae73344c2fdf1cb535a303d90edf9252358e0d30a44231649d36",
      "id": "latest",
      "created_at": 1778195206,
      "download_path": "/api/v4/feeds/proxies/export/latest"
    },
    {
      "kind": "hourly",
      "date": "2026-05-07",
      "hour": 21,
      "size_bytes": 612481020,
      "row_count": 42811027,
      "checksum": "d3fb2ec3de2bbf6af66b6028afe668e365c4113133757153ad975be93609d1ea",
      "id": "2026-05-07/21",
      "created_at": 1778192818,
      "download_path": "/api/v4/feeds/proxies/export/2026-05-07/21"
    },
    {
      "kind": "daily",
      "date": "2026-05-06",
      "size_bytes": 14721234567,
      "row_count": 1024411203,
      "checksum": "a3e4b804e1112ed2dd11e4b01a25a637ceb3abae23fa1f7c8503d567639a11a2",
      "id": "2026-05-06",
      "created_at": 1778115429,
      "download_path": "/api/v4/feeds/proxies/export/2026-05-06"
    }
  ],
  "next_cursor": "eyJkIjoiMjAyNi0wNS0wNyIsImgiOjIxfQ.GECAibkM_hEUt0ixRMKzzQ"
}

Download a snapshot

Returns a 307 redirect to a presigned R2 URL that is valid for 24 hours. The 307 (rather than 301) is intentional: the URL is minted per request and expires, so intermediaries must not cache it. Follow the redirect to download the parquet file. Use the date string latest for the freshest snapshot (the newest hourly during the day, or the daily rollup once that day's hourlies are rolled up), or append a specific hour at /{date}/{hour} for a particular hourly that has not been rolled up yet.

# Follow the 307 with -L and write to a file
curl -L -o proxies-2026-05-06.parquet \
  https://api.synthient.com/api/v4/feeds/proxies/export/2026-05-06 \
  -H "x-api-key: $SYNTHIENT_API_KEY"

Snapshot metadata

Returns JSON metadata for a parquet snapshot: SHA-256 checksum, byte size, row count, parquet schema with column names and types, and the canonical date.

curl -G https://api.synthient.com/api/v4/feeds/proxies/export/2026-05-02/meta \
  -H "x-api-key: $SYNTHIENT_API_KEY"

{
  "stream": "proxies",
  "kind": "hourly",
  "hour": 22,
  "id": "latest",
  "format": "parquet",
  "date": 1778191200,
  "created_at": 1778195206,
  "size": 620273951,
  "rows": 43141039,
  "checksum": "fd6c002ad6c6ae73344c2fdf1cb535a303d90edf9252358e0d30a44231649d36",
  "schema": {
    "fields": [
      { "name": "ip",           "type": "string" },
      { "name": "provider",     "type": "string" },
      { "name": "type",         "type": "string" },
      { "name": "timestamp",    "type": "int64" },
      { "name": "country_code", "type": "string" },
      { "name": "asn",          "type": "uint32" }
    ]
  }
}

Paginating snapshot listings

The list endpoint returns up to 500 snapshots per page, newest-first, with a next_cursor you pass back as cursor on the next call. When the response no longer carries a next_cursor, you've reached the end.

import os, requests

URL = "https://api.synthient.com/api/v4/feeds/proxies/export"
HEADERS = {"x-api-key": os.environ["SYNTHIENT_API_KEY"]}

cursor = None
while True:
    params = {"limit": 500, **({"cursor": cursor} if cursor else {})}
    page = requests.get(URL, headers=HEADERS, params=params, timeout=30).json()
    for snap in page["feeds"]:
        handle(snap)
    cursor = page.get("next_cursor")
    if not cursor:
        break

Response codes

Snapshot downloads answer with a 307 redirect: follow the Location header to a 24-hour presigned URL. Every other status follows the shared contract, including 404 when no snapshot exists for the requested date and stream. See Errors for the full reference and retry guidance.