Overview

Apache Kafka is the dominant distributed streaming platform. The Automize Kafka connector talks to Kafka via Confluent's REST Proxy – produce, consume, topics, partitions.

Regions: Wherever your Kafka cluster runs (on-prem, Confluent Cloud, AWS MSK, Azure Event Hubs Kafka surface).
Plans: Open-source Kafka free. Confluent Cloud paid.

Setting up the connection

Auth type: HTTP Basic.

Confluent REST Proxy URL + Basic auth credentials.
In Automize → Settings → Connectors → Kafka. Paste credentials + base URL.

Find your credentials at https://docs.confluent.io/platform/current/kafka-rest/.

Rate limits

Bound by Kafka cluster capacity + Confluent quota.

Data model

Streaming primitives:

    Cluster ─── Topics ─── Partitions ─── Records
            └─ Consumer Groups

Available operations

18 operations available. Click any row to jump to its detail.

Operation	Category	What it does
Create ACL	General	Create an access control list entry
Create Topic	General	Create a new topic
Delete Topic	General	Delete a topic
Get Broker	General	Get broker details
Get Cluster	General	Get cluster details
Get Consumer Group	General	Get consumer group details
Get Partition	General	Get partition details
Get Topic	General	Get topic details
List ACLs	General	List access control lists
List Brokers	General	List all brokers in a cluster
List Clusters	General	List all Kafka clusters
List Consumer Group Lag Summary	General	Get lag summary for a consumer group
List Consumer Groups	General	List all consumer groups
List Partitions	General	List all partitions for a topic
List Topic Configs	General	List configuration for a topic
List Topics	General	List all topics in a cluster
Produce Records	General	Produce records to a topic
Update Topic Config	General	Update a topic configuration parameter

Operations

Create ACL

Create an access control list entry

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text

Create Topic

Create a new topic

Parameter	Required	Type
Cluster Id `cluster_id`	Yes	text
Topic Name `topic_name`	No	text
Partitions Count `partitions_count`	No	text
Replication Factor `replication_factor`	No	text
Configs `configs`	No	text

Delete Topic

Delete a topic

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text
Topic Name `topic_name`	Yes	text

Get Broker

Get broker details

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text
Broker Id `broker_id`	Yes	text

Get Cluster

Get cluster details

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text

Get Consumer Group

Get consumer group details

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text
Consumer Group Id `consumer_group_id`	Yes	text

Get Partition

Get partition details

Parameter	Required	Type
Cluster Id `cluster_id`	Yes	text
Topic Name `topic_name`	Yes	text
Partition Id `partition_id`	Yes	text

Get Topic

Get topic details

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text
Topic Name `topic_name`	Yes	text

Tips

Returns partitions + replication factor + config.

List ACLs

List access control lists

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text

List Brokers

List all brokers in a cluster

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text

List Clusters

List all Kafka clusters

No input parameters.

List Consumer Group Lag Summary

Get lag summary for a consumer group

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text
Consumer Group Id `consumer_group_id`	Yes	text

List Consumer Groups

List all consumer groups

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text

List Partitions

List all partitions for a topic

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text
Topic Name `topic_name`	Yes	text

List Topic Configs

List configuration for a topic

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text
Topic Name `topic_name`	Yes	text

List Topics

List all topics in a cluster

Parameter	Required	Type	Description
Cluster Id `cluster_id`	Yes	text