Feeds

Stream each feed's real-time events and download its hourly and daily Parquet exports.

Each feed is a read-only property of Synthient with type Feed<T>, where T is the feed's event type. Every feed has a real-time stream and hourly Parquet exports that roll up into daily exports. Streaming and exports need separate scopes on the API key. The event types are listed under Event types.

What each feed carries:

• proxies carries residential, mobile, and datacenter proxy sightings.
• anonymizers carries VPN and anonymizer IP ranges.
• torrents carries torrents seen on the BitTorrent DHT and trackers.
• ja4t carries JA4T TCP fingerprints.
• The four Helios feeds carry honeypot captures: HTTP requests, TLS ClientHellos, DNS tunnels, and Android Debug Bridge commands.

Feed is exported so you can annotate variables and parameters with it, but its constructor is internal: use the feeds the client creates.

See Firehose, Feeds, and Helios for the data behind each feed.

Streams

stream

Returns an async generator over the feed's real-time events.

stream(options?: RequestOptions): AsyncGenerator<T, void, undefined>;

• Connection: The connection opens on the first next(), so connection errors are thrown there, including a 403 APIError for a missing scope or a 429 for too many concurrent streams. Breaking out of the loop closes the connection. The server keeps each stream open for up to 30 minutes and then closes it, which ends the iteration normally.
• Errors: Any other failure throws from the loop, such as an APIConnectionError when the connection drops or a SynthientError for an event that is not a JSON object. A 429 APIError without retryAfter means the key's monthly request budget is spent or it already has too many open streams.
• Timeout: The client's timeout bounds the gap between data, and the server sends a keepalive about every 15 seconds.
• Reconnecting: The API has no resume cursor, so events sent between connections are missed.

Call stream again to reconnect:

for (;;) {
  for await (const event of client.proxies.stream()) {
    console.log(event.ip, event.provider, event.type);
  }
}

The loop reconnects immediately after a clean close. Opening a connection already uses the SDK's bounded retries, including Retry-After handling. Any error that remains after those retries, or occurs while reading the stream, stops this consumer. A 429 without Retry-After can indicate an exhausted monthly request budget or too many open streams; resolve that condition before reconnecting.

Every feed streams the same way. Only the event type changes:

for await (const event of client.proxies.stream()) {
  console.log(event.ip, event.provider, event.type, event.country_code, event.asn);
}

HeliosHTTPSEvent.details is null when the sensor could not parse the handshake, so check it before use.

Exports

Each feed has hourly Parquet exports that roll up into daily exports.

• Hourly exports: An hourly export is available until its day's daily export is rolled up, shortly after 00:30 UTC the next day. After that, requests for the hour reject with a 400 APIError.
• Daily exports: While a daily export is being rolled up, requests for it reject with a 404 APIError with retryAfter set.

Export IDs

An export ID is one of:

• "latest"
• a UTC day as "YYYY-MM-DD"
• a UTC hour as "YYYY-MM-DD/H", where a zero-padded hour is also accepted

"latest" is the feed's freshest export: the newest hourly export during the day, or the daily export once that day's hours are rolled up. There is no constant for it.

Export.id values can be passed as they are. Any other string, including an impossible date such as "2026-02-30" or an hour past 23, rejects with a TypeError before a request is sent.

dailyExport returns the ID of the daily export for the UTC day containing date. hourlyExport returns the ID of the hourly export for the UTC hour containing date, with the hour unpadded as in Export.id.

function dailyExport(date: Date): string;
function hourlyExport(date: Date): string;

import { dailyExport, hourlyExport } from "@synthient/synthient";

const yesterday = dailyExport(new Date(Date.now() - 24 * 60 * 60 * 1000));
const lastHour = hourlyExport(new Date(Date.now() - 60 * 60 * 1000));
console.log(yesterday, lastHour);

• Input: An invalid Date throws a RangeError.

listExports

Returns one page of the feed's exports, newest first.

listExports(options?: ListExportsOptions): Promise<ExportPage>;

const page = await client.proxies.listExports({ limit: 24 });
for (const item of page.feeds) {
  console.log(item.id, item.kind, item.row_count);
}
console.log("next cursor:", page.next_cursor);

• Paging: Use allExports to iterate over every page.

allExports

Returns an async generator over the feed's exports, newest first.

allExports({ limit, cursor, signal }?: ListExportsOptions): AsyncGenerator<Export, void, undefined>;

for await (const item of client.proxies.allExports()) {
  console.log(item.id, item.kind, item.size_bytes);
}

• Paging: It fetches pages lazily as iteration proceeds, and fetches no more pages when the loop breaks.
• Input: The options are copied when allExports is called.
• Errors: Errors, including invalid options, are thrown from the iteration, as is a SynthientError when the API returns a cursor it already returned.

exportMeta

Returns the checksum, size, row count, and Parquet schema of an export.

exportMeta(id: string, options?: RequestOptions): Promise<ExportMeta>;

const meta = await client.proxies.exportMeta("latest");
console.log(meta.id, meta.rows, meta.size, meta.checksum);
for (const column of meta.schema.fields) {
  console.log(column.name, column.type);
}

downloadURL

Returns a presigned URL for an export's Parquet file.

downloadURL(id: string, options?: RequestOptions): Promise<URL>;

const url = await client.proxies.downloadURL("latest");
console.log(url.host);

• Lifetime: The URL is valid for 24 hours.
• Security: The URL grants access without an API key, so treat it as a secret.
• Use: It is useful for tools that read Parquet over HTTPS directly.

download

Returns the contents of an export's Parquet file as a web ReadableStream.

download(id: string, options?: RequestOptions): Promise<ReadableStream<Uint8Array>>;

• Cleanup: Read it to the end or cancel it to release the connection.
• Security: The API key is sent only to the Synthient API, never to the storage host that serves the file, and errors never include the presigned URL.
• Errors: A non-success response from either rejects with an APIError. Downloads from the storage host are not retried.

The SDK does not verify the file. Compare its SHA-256 with Export.checksum or ExportMeta.checksum:

import { createHash } from "node:crypto";
import { createWriteStream } from "node:fs";
import { Readable, Transform } from "node:stream";
import { pipeline } from "node:stream/promises";
import { dailyExport, type Feed } from "@synthient/synthient";

async function saveExport(feed: Feed<unknown>, id: string, path: string): Promise<void> {
  const meta = await feed.exportMeta(id);
  const body = await feed.download(id);
  const hash = createHash("sha256");
  await pipeline(
    Readable.fromWeb(body),
    new Transform({
      transform(chunk: Buffer, _, callback) {
        hash.update(chunk);
        callback(null, chunk);
      },
    }),
    createWriteStream(path),
  );

  if (hash.digest("hex") !== meta.checksum) {
    throw new Error("checksum mismatch");
  }
}

const yesterday = dailyExport(new Date(Date.now() - 24 * 60 * 60 * 1000));
await saveExport(client.proxies, yesterday, "proxies.parquet");