In the first post of our Kafka Connect on Google Cloud series, we demonstrated how to replicate your Managed Service for Apache Kafka cluster across regions using our Connect service (available in Preview). We followed with a deep dive into using Google-managed Kafka, Kafka Connect, and our new schema registry feature (available in Preview) to safely write Kafka data to BigQuery. In this post, we’ll demonstrate how to replicate data from a self-managed Kafka cluster hosted on Google Compute Engine (GCE) to a Managed Service for Apache Kafka cluster using Google-managed MirrorMaker 2.0 (MM2) connectors.
Kafka architects overseeing their own cloud-based Kafka deployments have to manage many complex components, from networking to authentication to security to access control, to name just a few. Many of these users come to Managed Service for Apache Kafka so they can stop wasting precious energy on management and focus their attention on enabling developers for new streaming use cases. However, architects have to migrate from one event-based system to another with extra caution.
In the following walkthrough, we will deploy a Google-managed MM2 connector to replicate data from a self-managed environment to Google-managed walkthrough. We will conclude with some key considerations for your own migration planning.
End-to-End Workflow
This exercise includes the following steps:
- Deploy self-managed cluster: We will deploy a Kafka cluster on GCE.
- Deploy Managed Service for Apache Kafka resources: We will create a managed cluster for both the destination Kafka cluster and the Connect cluster.
- Modify networking configurations: We will make slight modifications to Kafka-on-GCE configuration files. We will also create a DNS zone that enables the Kafka-on-GCE cluster to connect to the managed Kafka Connect cluster.
- Create resources on self-managed cluster: We’ll create a topic and some sample messages on our self-hosted cluster.
- Initiate replication: We will deploy Google-managed MM2 connectors.
- Validate message delivery: We will publish messages to the self-managed cluster, then consume those messages in our destination GMK cluster.
Step 1: Deploy self-managed cluster
Note: You might already have a self-hosted Kafka cluster on Google Compute Engine. If that is the case, you can move onto step 2. However, there are important configurations to note for your Kafka-on-GCE cluster that are covered here.
We will first launch a GCE instance. There are several methods to create & start an instance covered here. Most of the default configurations will be sufficient. Make sure to note the VPC network that you have selected — we will use this for future networking configuration.
Once the instance has been created, we can access the machine via SSH through Console, your command line, or third-party tools. We will install the Kafka software onto the machine. The quickstart tutorial to the Apache Kafka site provides a quick rundown of the steps (we will only use steps 1–4 in that tutorial). The commands are listed below for your convenience:
# Install the Java development kit
# Create directory for Java files & download filesmkdir java
cd java
wget https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_linux-x64_bin.tar.gz
# Decompress and extract the Java archive
tar -xzvf openjdk-11.0.2_linux-x64_bin.tar.gz
export PATH=$PATH:/java/jdk-11.0.2/bin/
# Install Apache Kafka
# Create directory for Kakfa files & download files
mkdir kafka
cd kafka
wget https://dlcdn.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz
# Decompress and extract the Kafka archive
tar -xzf kafka_2.13–3.8.0.tgz
Step 2: Deploy Managed Service for Apache Kafka clusters
Next, we will create two clusters: our Managed Service for Apache Kafka cluster, which will serve as our destination cluster; and our Connect cluster, which will host the MirrorMaker2 connector that will replicate data.
Our quickstart guide highlights how to create a Kafka cluster using the Console or gcloud. For the sake of simplicity, we can use most of the defaults for our test cluster. Once again, let’s take note of the VPC network that we are using for our Kafka cluster:
We want to ensure that the VPC network for our self-hosted Kafka cluster is accessible to our Google-managed cluster. The easiest way to do that is to use the same VPC network. If they are not, you can use VPC network peering to ensure they are accessible to one another.
The cluster can take up to 30 minutes to initialize. Once finished, navigate to the Configurations tab in the Cluster Details page. Take a look at the bootstrap URL:
One of the benefits of the managed service is it sets up a private Cloud DNS zone with a URL for each IP address for the bootstrap servers & brokers. The automatically generated DNS is in the following format:
bootstrap.{CLUSTER_NAME}-{REGION}.managedkafka.{PROJECT}.cloud.goog:9092
In the example above, the DNS is highlighted below:
bootstrap.mehran-kafka-test-us-central1.managedkafka.mehran-for-dataflow.cloud.goog:9092
We’ll reuse this information later in this quickstart.
Next, we’ll deploy a managed Connect cluster. Our documentation provides instructions for creating a Connect cluster using the Console, gcloud, or Terraform. You will notice that when we select our test cluster as our primary cluster, the network configuration will inherit identical settings from our Kafka cluster. This ensures that the Connect cluster can read or write from the Kafka cluster. You will also notice the primary cluster’s DNS domain is already added as a resolvable DNS domain. This ensures our Kafka cluster can advertise its brokers IP addresses to the Connect workers.
Step 3: Modify networking configurations
In this step, we’ll add a record to the private DNS zone that was automatically created by our Kafka cluster, then use that DNS record to update two configuration files for the self-hosted Kafka cluster. These steps will ensure that our MM2 connector can read from the self-hosted cluster and write to our managed one.
We will first go back to the GCE instance where our self-managed Kafka is running. When we click on the VM details, we will find a section called Network interfaces that has the information we need:
Make note of that Primary internal IP address — this information will be used in our next step.
Navigate to the Cloud DNS > Zones page, where we will find the private DNS zone created by our Kafka cluster:
Click on that zone, where you will see the DNS associated with that zone. Click on the “+ Add Standard” button so that we can add an A record:
We will add an A record with the Primary internal IP address that we retrieved from our GCE instance:
Once that has been created, jot down the DNS that is associated with that A record: self-hosted.mehran-kafka-test.us-central1.managedkafka.mehran-for-dataflow.cloud.goog.
Next, we will make changes to two configuration files for the self-hosted Kafka cluster, and add a record to the private DNS zone that was automatically created by our Kafka cluster. These steps will ensure that our MM2 connector can read from the self-hosted cluster and write to our managed one.
The first file we will edit is our controller configuration. You will find this file in the ~kafka/kafka_2.13–3.8.0/config/kraft folder. Use your text editor of choice to modify this line in the controller.properties file:
############################# Socket Server Settings #############################
# The address the socket server listens on.
# Note that only the controller listeners are allowed here when `process.roles=controller`, and this listener should be consistent with `controller.quorum.voters` value.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092listeners=CONTROLLER://:9093
By changing this, our KRaft cluster is running dedicated controller nodes (i.e. they do not store topic data or serve client requests). This helps us scale controllers & brokers independently based on metadata and traffic needs, and is often chosen for large-scale production environments.
Deep Dive: Broker Properties
In practice, we do not have to modify the broker file located on the Kafka server, but it’s a good idea to explore broker properties to see how the settings interact with one another. In production environments, you should create an A record matching the fully qualified domain of the Kafka broker listener.
In the same directory as the controller.properties file, we will find the broker configuration. We will be examine parameters in our “Socket Server Settings” section, specifically the listeners & advertised.listeners properties:
############################# Socket Server Settings #############################
# The address the socket server listens on. If not configured, the host name will be equal to the value of
# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=BROKER://self-hosted.mehran-kafka-test-us-central1.managedkafka.mehran-for-dataflow.cloud.goog:9092# Name of listener used for communication between brokers.
inter.broker.listener.name=BROKER
# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
# advertised.listeners=PLAINTEXT://localhost:9092
# A comma-separated list of the names of the listeners used by the controller.
# This is required if running in KRaft mode. On a node with `process.roles=broker`, only the first listed listener will be used by the broker.
controller.listener.names=CONTROLLER
Our listeners parameter now includes the DNS A record from our managed Kafka cluster. This property specifies the network interfaces and ports that the Kafka broker will bind to and listen for incoming connections from Kafka clients & other brokers (for data replication). We commented out (i.e. adding an asterisk in front of) the advertised.listeners parameter. By doing that, it will use the value that we specified for listeners.
To test the settings, we must restart our broker & controller.
# Generate a Kafka cluster ID
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"# Format storage
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/controller.properties
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/broker.properties
# Start controller
bin/kafka-server-start.sh config/kraft/controller.properties
# Start broker
bin/kafka-server-start.sh config/kraft/broker.properties
Step 4: Create resources on self-managed cluster
We will need to create a topic that can be replicated to our Managed Service for Apache Kafka cluster. The Kafka installation comes with helpful shell scripts that make these tasks relatively trivial. Run this command on your self-managed cluster:
# Create topic called "migration-test"bin/kafka-topics.sh \
- create \
- topic migration-test \
- bootstrap-server localhost:9092
Step 5: Initiate replication
We now have all of the necessary clusters in place, along with all of the networking configurations that allow for these clusters to talk to one another. The final step is deploy our MM2 connector on the Google-managed Connect cluster.
MM2 is actually three connectors: a Source connector that replicates that raw data, a Checkpoint connector that replicates offsets across clusters, and a Heartbeat connector that is used to monitor the health of the connector. We will focus on the Source connector in this walkthrough, but we have sample configurations found in our Github repository.
Our MM2 connector guide provides a walkthrough to help deploy your connector using Console, gcloud, or Terraform. We’ll focus on the Console workflow below:
There are two key settings to note after we’ve selected the MM2 connector plugin:
- Set your primary cluster as the target cluster
- Add the DNS address that we created in Step 3 as the bootstrap server for the self-managed cluster. As a reminder, we want to append the port at the end of it, so it should look like this:
self-hosted.mehran-kafka-test.us-central1.managedkafka.mehran-for-dataflow.cloud.goog:9092
We don’t have to make any modifications to the configuration file — our managed service does that under the hood for you. You could define authentication properties here if your use case requires it.
Step 6: Validate message delivery
We now have a working MM2 connector replicating messages from your self-managed GCE cluster to your Google-managed cluster. Let’s put it to the test!
First, we’ll publish messages to our self-managed Kafka. The Kafka library comes with a producer script that can produce messages.
# Use the Kafka producer utility to generate messages in the "migration-test" topicbin/kafka-console-producer.sh \
--topic migration-test \
--bootstrap-server localhost:9092
> hello
> world
Next, we can use the same machine that we are using for self-managed Kafka to consume messages from the managed one. If you want to setup a separate client machine, follow this quickstart.
We should have already seen our migration-test topic show up in our Kafka cluster. We’ll consume a message from the same topic, except now we will be using the bootstrap URL for our managed Kafka cluster:
# Set environment variablesexport PROJECT_ID=mehran-for-dataflow
export CLUSTER_ID=mehran-kafka-test
export REGION=us-central1
export BOOTSTRAP=bootstrap.CLUSTER_ID.REGION.managedkafka.PROJECT_ID.cloud.goog:9092
# List the topics
kafka-topics.sh - list \
- bootstrap-server $BOOTSTRAP \
- command-config client.properties
# Consume message from topic
kafka-console-consumer.sh \
- topic migration-test \
- from-beginning \
- bootstrap-server $BOOTSTRAP \
- consumer.config client.properties
Considerations
Migrating Kafka clusters could potentially affect downstream applications and analytical systems if not carefully planned. You have to consider several aspects as part of your migration planning:
- Client cutover: Now that you have two Kafka clusters running, how do you switch traffic from the legacy to the new? You can weigh the cost-benefits of instituting a maintenance window versus the disruption it would entail. You can publish to both clusters at the expense of redundant costs. You could also choose to move clients over in phases.
- Kafka-related services: You might be using Kafka Connect, Schema Registry, Kafka streams, ksqlDB, or other tools available in the Kafka ecosystem. You will need to consider if you need to keep running them as they are, or if your provider offers a fully managed version. (note that Google Cloud supports both a Connect and schema registry service).
- Operational posture: Switching to a managed service means forfeiting low-level access to your Kafka infrastructure. If you are used to SSH access to brokers, you will have to transition to relying on your managed providers APIs and operational tooling. It’s also important to consider how your security, networking, and high availability approach with the new service.
- Monitoring & performance: Monitor the performance of your new cluster until you feel confident that your new system can at least match your legacy one. You will likely have to adjust to your new provider’s tools. Make sure to understand what levers you have at your disposal to tune performance in your new system.
What’s Next
Google Cloud’s Managed Service for Apache Kafka reduces the toil of running Kafka on your own, and is fully integrated with Google Cloud’s security, networking, and operational stack. To learn more, visit our overview page.
If you are thinking about migrating Kafka clusters from your on-prem, self-managed, or cloud provider environments to Managed Service for Apache Kafka, we have a migration guide to help you along the way.
Please drop us a note at kafka-hotline@google.com if you have any questions or comments about the guide, our service, or the future roadmap. We look forward to hearing from you!
Source Credit: https://medium.com/google-cloud/migrating-kafka-clusters-on-google-cloud-26859e6364a9?source=rss—-e52cf94d98af—4
