To check a Kafka topic

To check a Kafka topic, you can use the Kafka command-line tools or programmatically access the topic using a client library. Below are methods for checking Kafka topics.

1. Using Kafka Command-Line Tools

Kafka includes a built-in script kafka-topics.sh to manage and check topics.

Check All Topics:

Run the following command to list all topics:

kafka-topics.sh --bootstrap-server <broker_host>:<broker_port> --list

Example:

kafka-topics.sh --bootstrap-server localhost:9092 --list

Describe a Specific Topic:

To get details about a specific topic, including partition count, replication factor, and partition leader:

kafka-topics.sh --bootstrap-server <broker_host>:<broker_port> --describe --topic <topic_name>

Example:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test-topic

Output example:

Topic: test-topic   PartitionCount: 3   ReplicationFactor: 2    Configs: segment.bytes=1073741824
    Topic: test-topic   Partition: 0    Leader: 1   Replicas: 1,2   Isr: 1,2
    Topic: test-topic   Partition: 1    Leader: 2   Replicas: 2,3   Isr: 2,3
    Topic: test-topic   Partition: 2    Leader: 3   Replicas: 3,1   Isr: 3,1

2. Programmatic Approach

If you prefer to programmatically check a topic, you can use Kafka client libraries. Here’s an example in Node.js using kafkajs.

Install kafkajs:

npm install kafkajs

Example Code:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092'], // Replace with your Kafka broker addresses
});

const checkTopic = async (topicName) => {
  const admin = kafka.admin();
  await admin.connect();

  // List all topics
  const topics = await admin.listTopics();
  console.log('Available topics:', topics);

  // Describe the specific topic
  if (topics.includes(topicName)) {
    const metadata = await admin.fetchTopicMetadata({ topics: [topicName] });
    console.log(`Metadata for topic "${topicName}":`, metadata);
  } else {
    console.log(`Topic "${topicName}" does not exist.`);
  }

  await admin.disconnect();
};

checkTopic('test-topic').catch(console.error);

3. Using Kafka UI Tools

Several tools provide a GUI to check Kafka topics:

  • Conduktor: A powerful Kafka GUI client for managing topics, consumers, and producers.
  • AKHQ: A lightweight, open-source web-based tool for viewing and managing Kafka clusters.
  • Kafka Manager: A web-based tool for administering Kafka topics and clusters.

Key Points

  • Kafka CLI is best for quick checks.
  • Programmatic access is ideal for integration or automation.
  • GUI tools offer a user-friendly way to explore and monitor topics.

댓글

댓글 쓰기

이 블로그의 인기 게시물

Chromium 개발 환경 세팅, 크로미움 개발 준비하기

Chromium 개발 환경을 구축하는 것은 여러 단계로 이루어져 있으며, 이를 위해서는 적절한 도구와 의존성을 설치해야 합니다. 아래는 일반적인 Chromium 개발 환경을 설정하는 방법입니다. 1. 시스템 요구사항 확인 Chromium은 리눅스, 맥OS, 윈도우에서 빌드할 수 있습니다. 이 가이드에서는 Ubuntu를 기준으로 설명하지만, 다른 운영체제에서도 유사한 과정으로 설정할 수 있습니다. 메모리 : 최소 16GB RAM (권장 32GB 이상) 디스크 공간 : 최소 100GB 이상의 여유 공간 운영체제 : Ubuntu 20.04 이상 권장 2. 필수 패키지 설치 Chromium을 빌드하려면 여러 패키지가 필요합니다. 아래 명령어를 사용하여 설치합니다. sudo apt- get update sudo apt- get install build-essential git python3 python3-pip sudo apt- get install libx11- dev libxcomposite- dev libxcursor- dev libxdamage- dev libxrandr- dev libxtst- dev libxss- dev libglib2 .0 - dev sudo apt- get install libnss3- dev libasound2- dev libpulse- dev libjpeg- dev libpng- dev sudo apt- get install curl nodejs npm 3. Chromium 소스 코드 가져오기 Chromium 소스 코드를 가져오려면 depot_tools 라는 도구가 필요합니다. depot_tools 설치 : git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git export PATH=` pwd `/depot_tools: " $PATH " 이 경로를 영구적으로 PATH에 추...
자세한 내용 보기

"Do you happen to"

The phrase "Do you happen to" is a polite and casual way of asking if someone knows or has something, or if they might be able to do something. It softens the tone of the question, making it sound less direct and more friendly or tentative. Examples: "Do you happen to have a pen I could borrow?" This means: "Do you have a pen I could borrow?" but in a more polite and casual way. "Do you happen to know where the nearest coffee shop is?" This means: "Do you know where the nearest coffee shop is?" "Do you happen to be free this weekend?" This means: "Are you free this weekend?" It adds an element of chance or uncertainty, implying that it's okay if the answer is "no."
자세한 내용 보기

Declaration of custom object or type in Node.js

In Node.js, when you want to assign one object to another that is declared by a custom type or class, you typically need to ensure compatibility between the source object's structure and the target class's expectations. Here's a breakdown: Scenario 1: Assigning Plain Objects to Class Instances If you have a custom class and want to assign properties from a plain object to an instance of that class, you can use one of the following methods: Example: class CustomType { constructor (name, age) { this .name = name; this .age = age; } } // Plain object const plainObject = { name : "John" , age : 30 }; // Assign using Object.assign() const customInstance = new CustomType(); Object .assign(customInstance, plainObject); console .log(customInstance instanceof CustomType); // true console .log(customInstance); // CustomType { name: 'John', age: 30 } Scenario 2: Using a Constructor You can modify your class to accept an object in the c...
자세한 내용 보기

How to change java version on gradle of flutter

To change the Java version used by Gradle in a Flutter project, you need to modify the build.gradle files of your Flutter project. Here's how you can do that: Steps: Locate the build.gradle files: You will find two build.gradle files in your Flutter project: One at the root level: <project_root>/android/build.gradle Another inside the app module: <project_root>/android/app/build.gradle Update the Java version in android/build.gradle : In the root build.gradle file, ensure that the sourceCompatibility and targetCompatibility settings are set to the desired Java version. Open <project_root>/android/build.gradle Look for the compileOptions block (or create it if it's not present). Update the Java version like this: allprojects { repositories { google() mavenCentral() } } subprojects { afterEvaluate { project -> if ( project .hasProperty( "android" )) { android { ...
자세한 내용 보기

Create fixed rounded button by Tailwind

To create a fixed, rounded button using Tailwind CSS , you can combine Tailwind's fixed utility (for positioning) with the rounded classes (for rounding the button). This is useful if you want a button that stays in a fixed position on the screen, such as at the bottom-right corner. Here’s how to do it: Example of a Fixed Rounded Button <button class="fixed bottom -4 right -4 bg-blue -500 text-white px -6 py -3 rounded-full shadow-lg"> Fixed Button </button> Explanation: fixed : Positions the button fixed relative to the viewport. It will stay in the same position even when the user scrolls. bottom-4 : Positions the button 1rem (16px) from the bottom of the screen. right-4 : Positions the button 1rem (16px) from the right side of the screen. bg-blue-500 : Sets the background color to blue. text-white : Sets the text color to white. px-6 py-3 : Adds horizontal ( px-6 ) and vertical ( py-3 ) padding for sizing the button. rounded-full : Mak...
자세한 내용 보기

Sorting a list in python

To sort a list in Python , you can use either the sort() method or the sorted() function. Below are examples for both approaches: 1. Using the sort() Method This method sorts the list in-place (i.e., modifies the original list). It does not return a new list. Example: my_list = [3, 1, 4, 1, 5, 9, 2, 6] # Sort the list in -place (ascending order by default) my_list. sort () print (my_list) # Output: [1, 1, 2, 3, 4, 5, 6, 9] Descending Order: Use reverse=True . ```python my_list = [3, 1, 4, 1, 5, 9, 2, 6] Sort in descending order my_list.sort(reverse=True) print(my_list) # Output: [9, 6, 5, 4, 3, 2, 1, 1] --- ### 2. **Using the `sorted()` Function** - This function returns a **new sorted list** and leaves the original list unchanged. **Example:** ```python my_list = [ 3 , 1 , 4 , 1 , 5 , 9 , 2 , 6 ] # Create a new sorted list (ascending order) sorted_list = sorted(my_list) print(sorted_list) # Output: [1, 1, 2, 3, 4, 5, 6, 9] print(my_list) # Ori...
자세한 내용 보기

Develop a Kafka producer to send messages to a Kafka topic

Here’s a guide and code examples to develop a Kafka producer to send messages to a Kafka topic. The examples cover both plain Kafka Producer API and Spring Kafka Producer for convenience. 1. Plain Kafka Producer API Maven Dependencies Add the Kafka client dependency in your pom.xml : < dependency > < groupId > org.apache.kafka </ groupId > < artifactId > kafka-clients </ artifactId > < version > 3.6.0 </ version > <!-- Replace with your Kafka version --> </ dependency > Java Code Example This example demonstrates producing messages to a Kafka topic using the plain Kafka Producer API. import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.clients.producer.RecordMetadata; import java.util.Properties; public class KafkaMessageProducer { public static void main( String [] args) { // Configure Kafka produce...
자세한 내용 보기

Options for auto.offset.reset in Kafka

The default value of the auto.offset.reset configuration in Apache Kafka depends on the context in which it is used. This configuration determines what the consumer should do when there is no initial offset in Kafka or if the current offset does not exist anymore (e.g., due to log cleanup). Default Value latest is the default value for auto.offset.reset . Options for auto.offset.reset earliest : The consumer starts reading from the beginning of the log (the earliest offset). Useful for reprocessing data from the start. latest (default): The consumer starts reading from the end of the log (the latest offset). It reads only new messages produced after the consumer has started. none : Throws an exception if no offset is found for the consumer's group. Typically used when you want strict control over offsets. Example Configuration Java Code Properties props = new Properties(); props . put (ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"...
자세한 내용 보기

Joining an additional control plane node to an existing Kubernetes cluster

Joining an additional control plane node to an existing Kubernetes cluster involves specific steps to ensure proper synchronization between the control plane components. Below is the detailed process: 1. Pre-Requisites Ensure the following on the new control plane node: Same Kubernetes Version : The new node must have the same Kubernetes version installed as the existing control plane. kubeadm version Kubeadm, Kubelet, and Kubectl Installed : Ensure these tools are installed and configured. Network Configuration : The new control plane must have network connectivity to the existing control plane nodes. 2. Retrieve the Join Command On an existing control plane node, generate the kubeadm join command with the --control-plane flag: kubeadm token create -- print -join-command You will get output similar to: kubeadm join <control-plane-endpoint> : 6443 --token <token> \ --discovery-token- ca -cert-hash sha256 : <hash> \ --control-plane 3. ...
자세한 내용 보기

The logs of the kubelet service

The logs of the kubelet service can be found depending on how the system is set up. Here are the most common locations: 1. Systemd Logs If your system uses systemd to manage services (most modern Linux distributions do), you can view the kubelet logs using journalctl : View Real-time Logs: sudo journalctl -u kubelet -f View Historical Logs: sudo journalctl -u kubelet 2. Log File on Disk On some systems, kubelet writes its logs to a file in /var/log . The exact location depends on the configuration: Common Locations : /var/log/kubelet.log /var/lib/kubelet/logs If the log file is not there, check the kubelet service configuration for custom log paths. 3. Kubernetes Configuration Flags The kubelet log location can be customized using the --log-dir or --log-file flags in the kubelet service configuration. To verify: Check the kubelet service file: cat /etc/systemd/system/kubelet .service .d / 10 -kubeadm.conf Look for logging-related flags, such as: Exec...
자세한 내용 보기