Using Kafka ACLs

This page describes enabling and configuring Bufstream’s support for Kafka ACLs. It covers how to enable and configure ACL-based authorization, which APIs support ACL-based authorization, and a few Bufstream-specific limitations.

This feature is in preview, and its functionality is subject to change in a future release of Bufstream.

Default setting

ACL-based authorization is disabled by default, so all authorization checks return ALLOW. Any attempt to create, delete, or list ACLs fails with a CLUSTER_AUTHORIZATION_FAILED error.

Enabling and configuring authorization

Enabling Kafka ACLs is simple. Add a new authorization: section, with one or more super users, to Bufstream’s configuration:

kafka:
  # The addition of the new "authorization" key in the configuration enables
  # ACL-based authorization for Bufstream. Requires authentication to be configured.
  #
  # Bufstream will fail to start if there are no super user principals and allow_everyone_if_no_acl_found
  # is false because this would lead to every operation that requires authorization to fail.
  authorization:
    # To minimize the chance of inadvertent system lockout, there must be at
    # least 1 principal given "super user" permissions. This is necessary because
    # ACL-based authorization is default DENY, so enabling it will cause all of the
    # operations listed below to fail until appropriate ACLs are added.
    super_users:
      - security_admin@example.com
    # If true, invert the behavior of the authorizer from denying operations where no ACL can be
    # found to allowing them. This setting is useful when first enabling authorization, since
    # there won't be any ACLs created, but should only be used during the migration phase. Enabling
    # this feature in production can lead to security vulnerabilities.
    allow_everyone_if_no_acl_found: false

Be sure to restart your brokers after making this change.

Using ACLs in Bufstream

A Kafka ACL entry represents the following statement:

Principal [P] is [allowed/denied] operation [O] for resource [R] on host [H].

• The principal must be of the form User:<username> .
  • The special user name * can be used to create an entry that applies to all users.
  • The special user name Anonymous can be used to configure permissions for connections using anonymous SASL or mTLS authentication.
• The operation can be any of: Read, Write, Create, Delete, Alter, Describe, ClusterAction, DescribeConfigs, AlterConfigs, IdempotentWrite, CreateTokens, DescribeTokens, or All.
  • Allowing or denying the All operation also allows or denies all other operations for the specified principal, resource, and host.
• The resource can be a specific resource (type and name), a resource prefix (type and name prefix), or the * wildcard to apply to all resources of a given type.
  • Valid resource types are: topic, group, cluster, transactional ID, delegation token, or user.
• The host can be a specific host IP or * to apply to all hosts.

For more details, refer to the Apache Kafka documentation.

The examples below use the kafka-acls.sh script that's provided with Apache Kafka to manipulate ACLs. These examples are only highlighting the basics. For more details about the full list of options, refer to that script's existing documentation.

Creating ACLs

Once you've enabled authorization and restarted Bufstream, create appropriate ACLs for your organization.

Initially, only the principals that you configure as super users are able to create ACLs. It's best practice to not use an account with super user permissions for “normal” actions, so the first ACLs created should grant permissions to manage ACLs to a different user, or users, by granting them ALTER and DESCRIBE permissions on the cluster. This unlocks the CREATE_ACLS, DELETE_ACLS, and DESCRIBE_ACLS APIs.

# Grants user bob@example.com sufficient permissions to manage ACLs.
> kafka-acls.sh --bootstrap-server bufstream:9092 --add --cluster --allow-principal User:bob@example.com --operation Alter --operation Describe

A common scenario is to add a principal as a producer or consumer. For that, use the --consumer or --producer options for kafka-acls.sh.

# Adds ExampleUser1 as a producer to the "test-topic" topic.
> kafka-acls.sh --bootstrap-server bufstream:9092 --add --allow-principal User:ExampleUser1 --producer --topic example-topic

# Adds ExampleUser2 as a consumer of the same topic.
> kafka-acls.sh --bootstrap-server bufstream:9092 --add --allow-principal User:ExampleUser2 --consumer --topic example-topic

Removing ACLs

Remove ACL entries with the --remove option for kafka-acls.sh:

# Remove the ACLs created in the example above.
> kafka-acls.sh --bootstrap-server bufstream:9092 --remove --allow-principal User:ExampleUser1 --producer --topic example-topic

> kafka-acls.sh --bootstrap-server bufstream:9092 --remove --allow-principal User:ExampleUser2 --consumer --topic example-topic

Listing ACLs

List ACLs with the --list option for kafka-acls.sh:

# Lists all ACL entries that apply to the "example-topic" topic.
> kafka-acls.sh --bootstrap-server bufstream:9092 --list --topic example-topic

Kafka API permissions

Determining the principal

The user principal for Bufstream operations is determined from either the mTLS certificate used to connect or from the SASL login information, depending on how you have your Bufstream deployment configured. Refer to the documentation for SASL and mTLS authentication for more details.

Host-based ACLs

The “host name” for Bufstream connections is populated from the remote address on the TCP connection. Importantly, this is an IP address, not a computer name. Also, because Bufstream’s brokers are interchangeable, this IP address can often be the address of a load balancer, not a user’s machine.

For these reasons, utilizing host-based permissions can be problematic and we recommend against using them.

Permissioned APIs

The tables below show the Kafka APIs that are implemented by Bufstream and require authorization, grouped by resource type, along with the required permissions for each.

Topics

API Key	Requires Permissions
CREATE_TOPICS	Requires CREATE on the cluster or CREATE on the individual topic or topics.
DELETE_TOPICS	Requires DELETE on the cluster or DELETE on the individual topic or topics.
DELETE_RECORDS	Requires DELETE on the topic or topics.
CREATE_PARTITIONS	Requires ALTER on the topic or topics.
DESCRIBE_PRODUCERS	Requires READ on the topic or topics.
METADATA	Requires DESCRIBE on the topic or topics.
FIND_COORDINATOR	Requires DESCRIBE on the topic or topics. Automatically creating topics also requires CREATE on the cluster.
PRODUCE	Requires WRITE permissions on the topic or topics being produced to. Transactional produces also require WRITE permissions on the associated transactional ID.
FETCH	Requires READ on the topic or topics.
LIST_OFFSETS	Requires DESCRIBE on the topic or topics.
OFFSET_COMMIT	Requires READ on the group and the topic or topics.

Consumer Groups

API Key	Requires Permissions
JOIN_GROUP	Requires READ on the consumer group.
LEAVE_GROUP	Requires READ on the consumer group.
SYNC_GROUP	Requires READ on the consumer group.
DESCRIBE_GROUP	Requires DESCRIBE on the consumer group or groups.
HEARTBEAT	Requires READ on the consumer group.
LIST_GROUPS	Requires DESCRIBE on the cluster or DESCRIBE on the individual consumer group or groups.
DELETE_GROUPS	Requires DELETE on the consumer group or groups.
OFFSET_FETCH	Requires DESCRIBE on the consumer group and the topic or topics.
OFFSET_DELETE	Requires DELETE on the consumer group and READ on the topic or topics.

Transactions

API Key	Requires Permissions
INIT_PRODUCER_ID	If the transactional ID is set on the request, requires WRITE on that transaction. Otherwise, requires IDEMPOTENT_WRITE on the cluster and WRITE for at least one topic.
ADD_PARTITIONS_TO_TXN	Requires WRITE on the transaction and on the associated topic or topics.
ADD_OFFSETS_TO_TXN	Requires WRITE on the transaction and READ on the associated consumer group.
TXN_OFFSET_COMMIT	Requires WRITE on the transaction and READ on the associated consumer groups.
END_TXN	Requires WRITE on the transaction.
DESCRIBE_TRANSACTIONS	Requires DESCRIBE on the transactional ID.
LIST_TRANSACTIONS	Requires DESCRIBE on the transactional ID.

Cluster

API Key	Requires Permissions
DESCRIBE_CLUSTER	Requires DESCRIBE on the cluster.
DESCRIBE_CLIENT_QUOTAS	Requires DESCRIBE_CONFIGS on the cluster.
DESCRIBE_CONFIGS	Requires DESCRIBE_CONFIGS on the specified resource (cluster or topic).
ALTER_CONFIGS	Requires ALTER_CONFIGS on the specified resource (cluster or topic).
INCREMENTAL_ALTER_CONFIGS	Requires ALTER_CONFIGS on the specified resource (cluster or topic).
DESCRIBE_LOG_DIRS	Requires DESCRIBE on the cluster.
LIST_PARTITION_REASSIGNMENTS	Requires DESCRIBE on the cluster.
DESCRIBE_ACLS	Requires DESCRIBE on the cluster.
CREATE_ACLS	Requires ALTER on the cluster.
DELETE_ACLS	Requires ALTER on the cluster.
DESCRIBE_USER_SCRAM_CREDENTIALS	Requires DESCRIBE on the cluster.
ALTER_USER_SCRAM_CREDENTIALS	Requires ALTER on the cluster.