Create topic on Kafka with partition count, 카프카 토픽 생성하기

To create a Kafka topic with 100 partitions, you can use the Kafka command-line tool (kafka-topics.sh or kafka-topics.bat for Windows). When creating a topic, you can specify the number of partitions and the replication factor.

Here’s how to create a topic with 100 partitions:

Step-by-step instructions:

  1. Open a terminal on your machine where Kafka is installed.

  2. Run the kafka-topics.sh command with the necessary options to create the topic:

    kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 100 --topic <topic_name>
    • --bootstrap-server localhost:9092: Specifies the Kafka broker (or cluster) address. Replace localhost:9092 with your actual Kafka broker address if it’s different.
    • --replication-factor 1: Specifies the number of replicas for each partition. In this example, it's set to 1, but in a production environment, you’ll want a higher replication factor for redundancy.
    • --partitions 100: Specifies the number of partitions (in this case, 100).
    • --topic <topic_name>: Replace <topic_name> with the name of the topic you want to create.

Example:

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 100 --topic my-large-topic

3. Verify the Topic Creation

After running the command, you can verify that the topic was created successfully and that it has 100 partitions:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-large-topic

This will give you a detailed description of the topic, including the number of partitions and replication factor.

Notes:

  • Replication factor: In a production Kafka setup with multiple brokers, you'll usually want to set a replication factor higher than 1 for fault tolerance.
  • Number of partitions: More partitions allow for greater parallelism in Kafka but require more resources. Make sure to tune the number of partitions based on your expected workload and cluster size.

댓글

댓글 쓰기

이 블로그의 인기 게시물

Using the MinIO API via curl

Using the MinIO API via curl is straightforward, as MinIO is compatible with Amazon S3 API, so most commands follow a similar syntax. Here’s a guide on how to use curl with the MinIO API for some common operations like uploading, downloading, and managing objects. Prerequisites Access Key and Secret Key : Obtain your MinIO Access Key and Secret Key. MinIO Endpoint : Know your MinIO server endpoint, e.g., http://localhost:9000 . Bucket : You may need an existing bucket name, or create a new one using the commands below. Authentication Header For requests to work with MinIO, you need to include authentication in the headers. MinIO uses AWS Signature Version 4 for signing requests. Common Examples 1. List Buckets To list all buckets in your MinIO account, use: curl -X GET \ - -url "http://localhost:9000/" \ - H "Authorization: AWS <AccessKey>:<Signature>" 2. Create a Bucket To create a new bucket, use: curl -X PUT \ - -url "htt...
자세한 내용 보기

vsftpd default directory

The default directory for FTP users in vsftpd depends on how the FTP server is configured and the user’s home directory setup. Below are different configurations and their impact on the default directory that users are placed in when they connect to the server. 1. Default Directory Behavior with vsftpd When a user logs in via FTP, they are typically placed in their home directory . By default, the home directory is located at /home/<username> , where <username> is the name of the FTP user. For example: If the FTP user is ftpuser , the default login directory will be /home/ftpuser/ . 2. Changing the Default Directory for Users A. Using local_root Configuration You can specify a custom directory (other than the user’s home directory) for FTP users by setting the local_root parameter in the vsftpd configuration file. Open the vsftpd configuration file: sudo nano /etc/ vsftpd.conf Add the following line to specify a new directory for all FTP users: lo...
자세한 내용 보기

[Ubuntu] *.deb 파일 설치 방법

확장자가 .deb 인 파일은 원래 Debian 리눅스에서의 설치파일입니다. Ubuntu 는 원래 Debian 리눅스에서 파생되었기 때문에, 많은 부분에서 서로 닮아 있습니다. 따라서 별다른 변환 절차 없이 아래와 같이 바로 설치가 가능합니다. [user@hostname]# sudo dpkg -i *.deb *.rpm 파일을 *.deb 파일로 변환 [user@hostname]$ sudo apt-get install alien [user@hostname]$ sudo alien -c *.rpm
자세한 내용 보기

Offset out of range error in Kafka, 카프카 트러블슈팅

The "Offset out of range" error in Kafka typically occurs when a consumer tries to read an offset that no longer exists on the broker. This can happen for several reasons, such as: The offset is smaller than the earliest offset available for the topic partition (i.e., the messages have been deleted due to log retention policies). The offset is larger than the latest offset available for the topic partition (i.e., the consumer is trying to read beyond the latest message). Steps to Resolve "Offset Out of Range" Error 1. Check Current Offsets Before making changes, it's useful to understand the current offsets and the offset range available for the topic partitions. Use the kafka-consumer-groups.sh script to describe the consumer group: bin/kafka-consumer-groups.sh --bootstrap- server localhost: 9092 --describe -- group <consumer- group -id> This command will show the current offsets, log end offsets, and the lag for each partition. 2. Reset the...
자세한 내용 보기

리눅스 (cron - 주기적 작업실행 데몬)

cron 데몬의 가동은 /etc/rc.d/init.d/crond 에 있다. 보통 시스템의 어떤 런레벨이든지 cron 데몬은 부팅 시 시작하도록 되어 있는데, 굳이 cron 데몬을 죽이거나 다시 시작하기 위해서는 위의 /etc/rc.d/init.d/crond 실행 스크립트를 실행하면 된다. cron 데몬에 작업을 설정하기 위해서는 cron table 을 설정하는 crontab 이라는 실행파일을 사용한다. cron table 을 편집하기 위해서는 crontab -e 명령을 사용한다. 입력 형식은 아래와 같다. #crontab -e 01 4 * * *  /etc/rc.d/init.d/network stop 15 4 * * * ~/scripts/my_script.sh 00 5 * * * /etc/rc.d/init.d/network start 위의 내용이 cron table 에 입력된 내용인데 이러한 입력형식은 아래와 같다. M H D m d cmd-line 필드명 범위 설명 M 0-59,* 분을 의미하며, *는 모든 볌위를 말함 H 0-23,* 시를 의미함 D 1-31,* 날짜를 의미함 m 1-12,* 달을 의미함 d 0-7,* 요일을 의미하며,0과 7은 일요일(Sun)을 의미함 cmd-line 실행할 명령을 입력 각자의 사용자가 등록한 cron table 은 /var/spool/cron 디렉토리에 저장된다. 입력 내용의 다른 예를 보면, 30  13  *  * 1-5  mail -s "It's 2pm" root% Dear root% % Are you sleeping?? 같은 경우는 월요일부터 금요일까지 오후 1시 30분에 해당 mail 명령을 실행하게 된다. 위에서 %문자는 개행문자 역할을 하며(풀어 쓰면 아래와 같은 명령이 된다.), 한줄이 넘어가는 명령을 쓰기 위해서는 백슬래쉬를 써주고 다시 개행해서 쓰면 된다. $mail -s "It's 2pm" ro...
자세한 내용 보기

리눅스 (하드링크&소프트링크)

장치파일 [root@hostname /root]# df -h Filesystem Size Used Avail Use% Mounted on /dev/hda1 39G 2.5G 35G 7% / /dev/hda5 132G 24G 102G 19% /home tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/hda2 49G 309M 46G 1% /backup [root@systech-dev /root]# ls -l /dev/hda1 brw-r----- 1 root disk 3, 1 2011-12-03 10:58 /dev/hda1 3, 1 은 커널이 장치를 식별하는 식별자이며, Major Number 식별자는 3, Minor Number 식별자는 1 임을 의미한다. Block 장치파일 I/O단위(Block) Character 장치파일 I/O단위(Byte) 하드링크 - inode 데이터블럭 낭비가 없다. [root@systech-dev /root/test]# touch a.txt [root@systech-dev /root/test]# ln a.txt b.txt [root@systech-dev /root/test]# cp -p a.txt c.txt [root@systech-dev /root/test]# ls -l total 0 -rw-r--r-- 2 root root 0 2012-11-29 00:40 a.txt -rw-r--r-- 2 root root 0 2012-11-29 00:40 b.txt -rw-r--r-- 1 root root 0 2012-11-29 00:40 c.txt a.txt 는 원본이고 b.txt 는 링크이다. (원본과 링크의 구별이 어려움) -rw-r--r-- 다음으로 오는 숫자는 하드링크가 걸린 갯수를 나타낸다. c.txt 는 하드링크가 자기 자신밖에 없으므로 1이 된다. ...
자세한 내용 보기

CDPEvents in puppeteer

In Puppeteer , CDPEvents refers to events emitted by the Chrome DevTools Protocol (CDP) . Puppeteer leverages CDP to interact with and control a Chromium-based browser. CDP provides detailed, low-level access to browser internals, such as network traffic, console logs, page lifecycle events, and more. You can listen to these CDP events through Puppeteer’s API to monitor or intercept browser activity. How to Listen for CDP Events in Puppeteer Enable the Required CDP Domain: Some events require enabling a particular domain (e.g., 'Network' , 'Page' , 'Runtime' ). Use page._client() to Access the CDP Session: Although it’s a bit lower-level, Puppeteer allows access to DevTools through the page._client() API. Example: Listening for Network Requests This example demonstrates how to intercept and log network requests using CDP. const puppeteer = require( 'puppeteer' ); ( async () => { const browser = await puppeteer.launch({ headless...
자세한 내용 보기

Using venv in Python

Using venv in Python allows you to create a virtual environment, which is an isolated environment to manage dependencies for a specific project. Here's how you can use it step by step: 1. Create a Virtual Environment Open your terminal or command prompt. Navigate to the folder where you want to create the virtual environment. Run the following command: python -m venv myenv Replace myenv with the name you want for your virtual environment. This will create a directory named myenv containing the virtual environment. 2. Activate the Virtual Environment Activation depends on your operating system: On Windows : myenv \S cripts \a ctivate On macOS/Linux : source myenv /bin/ activate After activation, your terminal prompt will change, showing the name of the virtual environment (e.g., (myenv) ). 3. Install Dependencies With the virtual environment activated, you can now install packages using pip : pip install package _n ame For example: pip install req...
자세한 내용 보기