# Feeds

Stream each feed's real-time events and download its hourly and daily Parquet exports.

Each feed is a read-only property of `Synthient` with type `Feed<T>`, where `T` is the feed's event type. Every feed has a real-time [stream](#streams) and hourly Parquet [exports](#exports) that roll up into daily exports. Streaming and exports need separate scopes on the API key. The event types are listed under [Event types](https://docs.synthient.com/sdks/node/types#event-types).

| Property      | Event type         | Stream scope            | Export scope          |
| ------------- | ------------------ | ----------------------- | --------------------- |
| `proxies`     | `ProxyEvent`       | `PROXY_FIREHOSE`        | `PROXY_FEEDS`         |
| `anonymizers` | `AnonymizerEvent`  | `ANONYMIZERS_STREAM`    | `ANONYMIZERS_FEED`    |
| `torrents`    | `TorrentEvent`     | `TORRENTS_STREAM`       | `TORRENTS_FEED`       |
| `ja4t`        | `JA4TEvent`        | `JA4T_STREAM`           | `JA4T_FEED`           |
| `heliosHTTP`  | `HeliosHTTPEvent`  | `HONEYPOT_HTTP_STREAM`  | `HONEYPOT_HTTP_FEED`  |
| `heliosHTTPS` | `HeliosHTTPSEvent` | `HONEYPOT_HTTPS_STREAM` | `HONEYPOT_HTTPS_FEED` |
| `heliosDNS`   | `HeliosDNSEvent`   | `HONEYPOT_DNS_STREAM`   | `HONEYPOT_DNS_FEED`   |
| `heliosADB`   | `HeliosADBEvent`   | `HONEYPOT_ADB_STREAM`   | `HONEYPOT_ADB_FEED`   |

What each feed carries:

- `proxies` carries residential, mobile, and datacenter proxy sightings.
- `anonymizers` carries VPN and anonymizer IP ranges.
- `torrents` carries torrents seen on the BitTorrent DHT and trackers.
- `ja4t` carries JA4T TCP fingerprints.
- The four Helios feeds carry honeypot captures: HTTP requests, TLS ClientHellos, DNS tunnels, and Android Debug Bridge commands.

`Feed` is exported so you can annotate variables and parameters with it, but its constructor is internal: use the feeds the client creates.

See [Firehose](https://docs.synthient.com/enterprise/firehose), [Feeds](https://docs.synthient.com/enterprise/feeds), and [Helios](https://docs.synthient.com/enterprise/helios) for the data behind each feed.

## Streams

### stream

Returns an async generator over the feed's real-time events.

```ts
stream(options?: RequestOptions): AsyncGenerator<T, void, undefined>;
```

- **Connection:** The connection opens on the first `next()`, so connection errors are thrown there, including a 403 `APIError` for a missing scope or a 429 for too many concurrent streams. Breaking out of the loop closes the connection. The server keeps each stream open for up to 30 minutes and then closes it, which ends the iteration normally.
- **Errors:** Any other failure throws from the loop, such as an `APIConnectionError` when the connection drops or a `SynthientError` for an event that is not a JSON object. A 429 `APIError` without `retryAfter` means the key's monthly request budget is spent or it already has too many open streams.
- **Timeout:** The client's `timeout` bounds the gap between data, and the server sends a keepalive about every 15 seconds.
- **Reconnecting:** The API has no resume cursor, so events sent between connections are missed.

Call `stream` again to reconnect:

```ts
for (;;) {
  for await (const event of client.proxies.stream()) {
    console.log(event.ip, event.provider, event.type);
  }
}
```

The loop reconnects immediately after a clean close. Opening a connection already uses the SDK's bounded retries, including `Retry-After` handling. Any error that remains after those retries, or occurs while reading the stream, stops this consumer. A `429` without `Retry-After` can indicate an exhausted monthly request budget or too many open streams; resolve that condition before reconnecting.

Every feed streams the same way. Only the event type changes:

**Streams**

```ts title="Proxies"
for await (const event of client.proxies.stream()) {
  console.log(event.ip, event.provider, event.type, event.country_code, event.asn);
}
```

```ts title="Anonymizers"
for await (const event of client.anonymizers.stream()) {
  console.log(event.range_start, event.range_end, event.provider, event.type);
}
```

```ts title="Torrents"
for await (const event of client.torrents.stream()) {
  console.log(event.info_hash, event.name, event.peers?.length);
}
```

```ts title="JA4T"
for await (const event of client.ja4t.stream()) {
  console.log(event.ip, event.ja4t, event.provider);
}
```

```ts title="Helios HTTP"
for await (const event of client.heliosHTTP.stream()) {
  console.log(event.domain, event.details?.method, event.details?.uri, event.meta.provider);
}
```

```ts title="Helios HTTPS"
for await (const event of client.heliosHTTPS.stream()) {
  if (!event.details) {
    continue;
  }
  console.log(event.domain, event.details.sni, event.details.handshake_version);
}
```

```ts title="Helios DNS"
for await (const event of client.heliosDNS.stream()) {
  console.log(event.domain, event.port, event.meta.provider);
}
```

```ts title="Helios ADB"
for await (const event of client.heliosADB.stream()) {
  console.log(event.session, event.sequential_id, event.command);
}
```

`HeliosHTTPSEvent.details` is `null` when the sensor could not parse the handshake, so check it before use.

## Exports

Each feed has hourly Parquet exports that roll up into daily exports.

- **Hourly exports:** An hourly export is available until its day's daily export is rolled up, shortly after 00:30 UTC the next day. After that, requests for the hour reject with a 400 `APIError`.
- **Daily exports:** While a daily export is being rolled up, requests for it reject with a 404 `APIError` with `retryAfter` set.

### Export IDs

An export ID is one of:

- `"latest"`
- a UTC day as `"YYYY-MM-DD"`
- a UTC hour as `"YYYY-MM-DD/H"`, where a zero-padded hour is also accepted

`"latest"` is the feed's freshest export: the newest hourly export during the day, or the daily export once that day's hours are rolled up. There is no constant for it.

`Export.id` values can be passed as they are. Any other string, including an impossible date such as `"2026-02-30"` or an hour past 23, rejects with a `TypeError` before a request is sent.

`dailyExport` returns the ID of the daily export for the UTC day containing `date`. `hourlyExport` returns the ID of the hourly export for the UTC hour containing `date`, with the hour unpadded as in `Export.id`.

```ts
function dailyExport(date: Date): string;
function hourlyExport(date: Date): string;
```

```ts
import { dailyExport, hourlyExport } from "@synthient/synthient";

const yesterday = dailyExport(new Date(Date.now() - 24 * 60 * 60 * 1000));
const lastHour = hourlyExport(new Date(Date.now() - 60 * 60 * 1000));
console.log(yesterday, lastHour);
```

- **Input:** An invalid `Date` throws a `RangeError`.

### listExports

Returns one page of the feed's exports, newest first.

```ts
listExports(options?: ListExportsOptions): Promise<ExportPage>;
```

- `ListExportsOptions.limit?` `number | undefined` — The page size. The API default is 100, and it clamps values above 500. A value that is not a positive integer rejects with a `RangeError` before a request.
- `ListExportsOptions.cursor?` `string | undefined` — Continues a listing from a previous page's `next_cursor`. An empty string starts from the first page. `ListExportsOptions` also takes `signal` from `RequestOptions`.

```ts
const page = await client.proxies.listExports({ limit: 24 });
for (const item of page.feeds) {
  console.log(item.id, item.kind, item.row_count);
}
console.log("next cursor:", page.next_cursor);
```

- **Paging:** Use `allExports` to iterate over every page.

### allExports

Returns an async generator over the feed's exports, newest first.

```ts
allExports({ limit, cursor, signal }?: ListExportsOptions): AsyncGenerator<Export, void, undefined>;
```

```ts
for await (const item of client.proxies.allExports()) {
  console.log(item.id, item.kind, item.size_bytes);
}
```

- **Paging:** It fetches pages lazily as iteration proceeds, and fetches no more pages when the loop breaks.
- **Input:** The options are copied when `allExports` is called.
- **Errors:** Errors, including invalid options, are thrown from the iteration, as is a `SynthientError` when the API returns a cursor it already returned.

### exportMeta

Returns the checksum, size, row count, and Parquet schema of an export.

```ts
exportMeta(id: string, options?: RequestOptions): Promise<ExportMeta>;
```

```ts
const meta = await client.proxies.exportMeta("latest");
console.log(meta.id, meta.rows, meta.size, meta.checksum);
for (const column of meta.schema.fields) {
  console.log(column.name, column.type);
}
```

### downloadURL

Returns a presigned URL for an export's Parquet file.

```ts
downloadURL(id: string, options?: RequestOptions): Promise<URL>;
```

```ts
const url = await client.proxies.downloadURL("latest");
console.log(url.host);
```

- **Lifetime:** The URL is valid for 24 hours.
- **Security:** The URL grants access without an API key, so treat it as a secret.
- **Use:** It is useful for tools that read Parquet over HTTPS directly.

### download

Returns the contents of an export's Parquet file as a web `ReadableStream`.

```ts
download(id: string, options?: RequestOptions): Promise<ReadableStream<Uint8Array>>;
```

- **Cleanup:** Read it to the end or cancel it to release the connection.
- **Security:** The API key is sent only to the Synthient API, never to the storage host that serves the file, and errors never include the presigned URL.
- **Errors:** A non-success response from either rejects with an `APIError`. Downloads from the storage host are not retried.

The SDK does not verify the file. Compare its SHA-256 with `Export.checksum` or `ExportMeta.checksum`:

```ts
import { createHash } from "node:crypto";
import { createWriteStream } from "node:fs";
import { Readable, Transform } from "node:stream";
import { pipeline } from "node:stream/promises";
import { dailyExport, type Feed } from "@synthient/synthient";

async function saveExport(feed: Feed<unknown>, id: string, path: string): Promise<void> {
  const meta = await feed.exportMeta(id);
  const body = await feed.download(id);
  const hash = createHash("sha256");
  await pipeline(
    Readable.fromWeb(body),
    new Transform({
      transform(chunk: Buffer, _, callback) {
        hash.update(chunk);
        callback(null, chunk);
      },
    }),
    createWriteStream(path),
  );

  if (hash.digest("hex") !== meta.checksum) {
    throw new Error("checksum mismatch");
  }
}

const yesterday = dailyExport(new Date(Date.now() - 24 * 60 * 60 * 1000));
await saveExport(client.proxies, yesterday, "proxies.parquet");
```