Skip to main content
Convoy supports ingesting events from Kafka topics. This is currently only supported in outgoing projects.

​
Connecting to your Kafka topic

To connect your Kafka topic, you need to supply:
  • The kafka cluster hostname and port (e.g., localhost:9092)
  • The topic name (e.g., audit-log-events)
  • The number of Convoy agent replicas consuming the topic — the number of partitions for the topic should match how you scale agents for that source.
  • The consumer group id —it is recommended that you set one— which by default this is set to an empty string.
  • Supply authentication configured on the cluster

​
Authentication

  • Auth types: PLAIN and SCRAM
  • Hash types: SHA512 and SHA256
  • TLS: Enabled and Disabled

​
Ingestion Options

There are two ways to ingest events into Convoy from your Kafka topic:
  • Format the payload using the structure below as a guide and write it to your topic.
    reference payload
     {
    "event_type": "string, required",
    "data": "object, required",
    "custom_headers": { // optional
        "x-convoy-message-type": "single",
        "sample-header": "sample-value"
    },
    "idempotency_key": "string, optional",
    "owner_id": "string, optional", // if included, the event is sent to all endpoints with the owner-id
    "endpoint_id": "string, optional" // if included, the event is sent to a single endpoint
 }
    If x-convoy-message-type set to broadcast, the event will be sent to all endpoints in the project, ignoring both the endpoint_id and owner_id values.
    For a full list of reference payloads, see our guide on ingesting events
  • Send your arbitrarily formatted event payloads to your topic and use Convoy’s transform functions to mutate them at the point of ingestion.
    transform function
    function transform(payload) {
    return {
        "endpoint_id": "",
        "owner_id": "",
        "event_type": "sample",
        "data": payload, // payload contains arbitrary message data read from your topic
        "custom_headers": {
            "sample-header": "sample-value"
        },
        "idempotency_key": ""
    }
}

​
Things to note

  • Messages are read off the topic one-by-one by each agent consumer reading a partition.
  • Acknowledgements are done per message.
  • Ingestion is subject to the instance ingest rate (CONVOY_INSTANCE_INGEST_RATE / instance_ingest_rate, default 1000/s in code). See Rate limits.