Destinations

Destinations (aka sinks) are where events are delivered β€” either to hyperi dfe, another edgestream-hub or to external systems such as S3, ClickHouse, Kafka, files, or HTTP endpoints.

Available Destinations

Destination

Label

Base (required) fields

TLS

Common fields (high-level groups)

aws_s3

AWS S3

bucket, region, encoding.codec

βœ… (client)

description, acknowledgements.enabled, batch.*, buffer.*, compression, encoding.*, framing.*, filename_*, server_side_encryption, ssekms_key_id*, storage_class, endpoint, timezone, proxy.*, acl, grant_*

clickhouse

Clickhouse

endpoint, table

βœ… (client)

description, acknowledgements.enabled, batch.*, buffer.*, compression, encoding.*, framing.*, endpoint, timezone

file

File

path, encoding.codec

❌ (not applicable)

description, acknowledgements.enabled, buffer.*, compression, encoding.*, framing.*, idle_timeout_secs, timezone

http

Http

uri, decoding.codec

βœ… (HTTPS)

description, acknowledgements.enabled, auth.*, batch.*, buffer.*, compression, encoding.*, framing.*, method, payload_prefix/suffix, proxy.*

kafka

Kafka

bootstrap_servers, topic, encoding.codec

βœ… (client)

description, acknowledgements.enabled, batch.*, buffer.*, compression, encoding.*, headers_key, key_field, message_timeout_ms, sasl.*, socket_timeout_ms, librdkafka_options

vector

Vector (edgestream-hub)

address

βœ… (client/server)

description, acknowledgements.enabled, batch.*, buffer.*, compression

* Fields marked with an asterisk are conditionally required (see details below).

TLS & Security (Quick Reference)

All sinks that communicate over the network can enable TLS via the shared tls.* block:

  • tls.enabled (bool, default: false) – Turn TLS on.

  • tls.alpn_protocols (string) – ALPN protocol list (ordered).

  • tls.ca_file (file) – Additional CA to trust (PEM).

  • tls.crt_file (file) – Client/server certificate (PEM).

  • tls.key_file (file) – Private key for certificate (PEM).

  • tls.key_pass (string) – Passphrase for encrypted key.

  • tls.server_name (string) – SNI override for outbound TLS.

  • tls.verify_certificate (bool, default: true) – Validate peer certificates (set false only for testing).

  • tls.verify_hostname (bool, default: true) – Verify hostname against cert (outbound only).

Auth & proxy helpers (where supported):

  • HTTP sink: auth.strategy (none|basic|bearer), auth.user, auth.password, auth.token.

  • Proxying (HTTP/HTTPS): proxy.enabled, proxy.http, proxy.https, proxy.no_proxy.

  • Kafka: sasl.enabled, sasl.mechanism, sasl.username, sasl.password.

Common Fields Reference

These groups repeat across sinks; each sink may support a subset.

  • Description & Acks

    • description – freeform text for the entry.

    • acknowledgements.enabled (bool, default: false) – end-to-end acknowledgements.

  • Batching

    • batch.max_bytes (default: 10_000_000), batch.max_events (default: 500), batch.timeout_secs (default: 300).

  • Buffering

    • buffer.type (memory|disk, default: memory)

    • buffer.max_events (memory), buffer.max_size (disk, default: 268,435,488 bytes)

    • buffer.when_full (block|drop_newest, default: block)

  • Compression

    • Typical values: gzip|zstd|snappy|zlib|none (varies by sink).

  • Encoding (output format)

    • encoding.codec (e.g., json, text, csv, protobuf, avro, gelf, logfmt, raw_message, native, native_json)

    • Per-codec extras, e.g. encoding.csv.*, encoding.avro.schema, encoding.protobuf.*

    • Field filters: encoding.only_fields, encoding.except_fields

    • encoding.json.pretty (bool)

    • encoding.timestamp (rfc3339|unix|unix_ms|unix_us|unix_ns|unix_float)

Destination Settings