Feeds

Synthient publishes every data stream as Parquet snapshots hourly rolls plus a daily rollup alongside the real-time NDJSON streams. Snapshots are served as 24-hour presigned R2 URLs and are accompanied by full metadata: size, row count, SHA-256 checksum, and parquet schema.

Streams

The same seven stream identifiers are used across exports and real-time streaming.

Authentication

All endpoints require your API key in the x-api-key header and are served from https://api.synthient.com under /api/v4. Each stream is gated by its own *_FEED scope, such as PROXY_FEEDS, ANONYMIZERS_FEED, or HONEYPOT_HTTP_FEED.

curl -G https://api.synthient.com/api/v4/feeds/proxies/export \
  -H "x-api-key: $API_KEY"

Hourly snapshots and rollups

Hourly snapshots are addressable at /api/v4/feeds/{stream}/export/{date}/{hour} (with a matching /meta variant). Only the current UTC date is addressable for hourlies at 00:30 UTC each day, the previous day's hourlies are rolled up into the daily snapshot at /export/{date} and the per-hour artifacts are deleted. Use the snapshot id latest to track the most recent hourly without committing to a specific hour.

The Helios sensors live under their helio/ URL prefix for example /api/v4/feeds/helio/http/export/{date} (and /feeds/helio/https/... for TLS captures) but they appear in this list endpoint under the honeypot_http, honeypot_https, honeypot_dns, and honeypot_adb stream identifiers.

List snapshots

Returns one page of available daily and hourly Parquet snapshots for the given stream. Pages are ordered newest-first and capped at 500 rows; pass next_cursor from the response back as cursor to fetch the next page.

curl -G https://api.synthient.com/api/v4/feeds/proxies/export \
  -H "x-api-key: $API_KEY" \
  --url-query "limit=50"

{
  "stream": "proxies",
  "feeds": [
    {
      "kind": "hourly",
      "date": "2026-05-07",
      "hour": 22,
      "size_bytes": 620273951,
      "row_count": 43141039,
      "checksum": "fd6c002ad6c6ae73344c2fdf1cb535a303d90edf9252358e0d30a44231649d36",
      "id": "latest",
      "created_at": 1778195206,
      "download_path": "/api/v4/feeds/proxies/export/latest"
    },
    {
      "kind": "hourly",
      "date": "2026-05-07",
      "hour": 21,
      "size_bytes": 612481020,
      "row_count": 42811027,
      "checksum": "d3fb2ec3de2bbf6af66b6028afe668e365c4113133757153ad975be93609d1ea",
      "id": "2026-05-07/21",
      "created_at": 1778192818,
      "download_path": "/api/v4/feeds/proxies/export/2026-05-07/21"
    },
    {
      "kind": "daily",
      "date": "2026-05-06",
      "size_bytes": 14721234567,
      "row_count": 1024411203,
      "checksum": "a3e4b804e1112ed2dd11e4b01a25a637ceb3abae23fa1f7c8503d567639a11a2",
      "id": "2026-05-06",
      "created_at": 1778115429,
      "download_path": "/api/v4/feeds/proxies/export/2026-05-06"
    }
  ],
  "next_cursor": "eyJkIjoiMjAyNi0wNS0wNyIsImgiOjIxfQ.GECAibkM_hEUt0ixRMKzzQ"
}

Download a snapshot

Returns a 307 redirect to a presigned R2 URL that is valid for 24 hours. The 307 (rather than 301) is intentional the URL is minted per request and expires, so intermediaries must not cache it. Follow the redirect to download the parquet file. Use the date string latest for the most recent hourly snapshot, or append a specific hour at /{date}/{hour} for a particular hourly within the current UTC day.

# Follow the 307 with -L and write to a file
curl -L -o proxies-2026-05-06.parquet \
  https://api.synthient.com/api/v4/feeds/proxies/export/2026-05-06 \
  -H "x-api-key: $API_KEY"

Snapshot metadata

Returns JSON metadata for a parquet snapshot SHA-256 checksum, byte size, row count, parquet schema with column names and types, and the canonical date.

curl -G https://api.synthient.com/api/v4/feeds/proxies/export/2026-05-02/meta \
  -H "x-api-key: $API_KEY"

{
  "stream": "proxies",
  "kind": "hourly",
  "hour": 22,
  "id": "latest",
  "format": "parquet",
  "date": 1778191200,
  "created_at": 1778195206,
  "size": 620273951,
  "rows": 43141039,
  "checksum": "fd6c002ad6c6ae73344c2fdf1cb535a303d90edf9252358e0d30a44231649d36",
  "schema": {
    "fields": [
      { "name": "ip",           "type": "string" },
      { "name": "provider",     "type": "string" },
      { "name": "type",         "type": "string" },
      { "name": "timestamp",    "type": "int64" },
      { "name": "country_code", "type": "string" },
      { "name": "asn",          "type": "uint32" }
    ]
  }
}

Paginating snapshot listings

The list endpoint returns up to 500 snapshots per page, newest-first, with a next_cursor you pass back as cursor on the next call. When the response no longer carries a next_cursor, you've reached the end.

import os, requests

URL = "https://api.synthient.com/api/v4/feeds/proxies/export"
HEADERS = {"x-api-key": os.environ["API_KEY"]}

cursor = None
while True:
    params = {"limit": 500, **({"cursor": cursor} if cursor else {})}
    page = requests.get(URL, headers=HEADERS, params=params, timeout=30).json()
    for snap in page["feeds"]:
        handle(snap)
    cursor = page.get("next_cursor")
    if not cursor:
        break

Response Codes