Buf Docs

Appearance

Confluent Schema Registry integration overview ​

WARNING

This feature is only available on the Enterprise plan.

The Buf Schema Registry (BSR) includes a Protobuf-first implementation of the Confluent Schema Registry (CSR) APIs. It's designed to strengthen your streaming data quality where it helps the most — at build time, and at the source of truth for your schemas. Because the BSR implements the entirety of the CSR API, you can take advantage of the Confluent ecosystem, including stream processing systems like kSQL and Kafka Connect, without the risk of runtime errors.

  • Creating a CSR instance is a 1-step process, and instances can be easily managed by your BSR admins.
  • The BSR maps Confluent subjects to messages in your schema with straightforward annotations — no migration or conversion to other schema formats required. It brings the breadth and power of the Confluent ecosystem into your existing BSR workflow.
  • In addition to other breaking change checks, the BSR checks schemas for backwards compatibility with CSR subjects when publishing to the BSR, ensuring that only safe and approved changes flow downstream.
  • The CSR integration works with any Confluent-compatible Kafka client, downstream systems like kSQL and Kafka Connect, and management tools like AKHQ, making all the capabilities of the Confluent ecosystem available to your organization.

Getting started ​

Taking advantage of Buf's CSR implementation has three overall steps:

  1. Go to the BSR and create your CSR instance.

  2. Associate your schemas with Confluent subjects by integrating the bufbuild/confluent managed module and annotating your Protobuf messages (see Manage schemas for detailed steps):

    protobuf
    syntax = "proto3"

package demo.analytics;

import "buf/confluent/v1/extensions.proto";
import "google/protobuf/timestamp.proto";

message EmailUpdated {
  google.protobuf.Timestamp updated_at = 1;
  fixed64 user_id = 2;
  string previous_email = 3;
  string new_email = 4;
  bool new_email_verified = 5;

  option (buf.confluent.v1.subject) = {
    instance_name: "default",
    name: "email-updated-value",
  };
}

  3. Push the annotated .proto files to the BSR. Buf's breaking change detection validates the annotated schemas and sends them to the review flow if they're not backward compatible.

Once the push that includes the subject mapping is successful, the Confluent integration automatically creates a subject associated with the message on your CSR instance. For example, the example .proto file above creates a subject named email-updated-value associated with the demo.analytics.EmailUpdated message on the default CSR instance.

Because subjects are defined in your schemas, you're able to view, code review, and manage them the way you do any other source code. In the BSR, you could view the subject above at:

https://buf.example.com/integrations/confluent/default/subjects/email-updated-value/versions/latest

After you create the subject, you use the CSR URL and a Buf token to configure Confluent-aware Kafka producers and consumers to serialize and deserialize the topics defined in your Protobuf schemas. See Integrating with Kafka clients for examples.

Backwards compatibility modes ​

A Confluent subject can have one of three compatibility modes, which determine how the BSR handles breaking changes to that subject's schema:

  • BACKWARD_TRANSITIVE: Default. Enforces that updates to the schema are backwards compatible against all past versions of that subject.
  • BACKWARD: Enforces that updates to the schema are backwards compatibility against the latest version of that subject.
  • NONE: No compatibility checks are performed on updates to the schema.

All Confluent subjects start with the compatibility mode of the CSR instance they belong to. The default compatibility mode of instances when created is BACKWARD_TRANSITIVE.

Next steps ​