Create topic on Kafka with partition count, 카프카 토픽 생성하기

To create a Kafka topic with 100 partitions, you can use the Kafka command-line tool (kafka-topics.sh or kafka-topics.bat for Windows). When creating a topic, you can specify the number of partitions and the replication factor.

Here’s how to create a topic with 100 partitions:

Step-by-step instructions:

  1. Open a terminal on your machine where Kafka is installed.

  2. Run the kafka-topics.sh command with the necessary options to create the topic:

    kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 100 --topic <topic_name>
    • --bootstrap-server localhost:9092: Specifies the Kafka broker (or cluster) address. Replace localhost:9092 with your actual Kafka broker address if it’s different.
    • --replication-factor 1: Specifies the number of replicas for each partition. In this example, it's set to 1, but in a production environment, you’ll want a higher replication factor for redundancy.
    • --partitions 100: Specifies the number of partitions (in this case, 100).
    • --topic <topic_name>: Replace <topic_name> with the name of the topic you want to create.

Example:

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 100 --topic my-large-topic

3. Verify the Topic Creation

After running the command, you can verify that the topic was created successfully and that it has 100 partitions:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-large-topic

This will give you a detailed description of the topic, including the number of partitions and replication factor.

Notes:

  • Replication factor: In a production Kafka setup with multiple brokers, you'll usually want to set a replication factor higher than 1 for fault tolerance.
  • Number of partitions: More partitions allow for greater parallelism in Kafka but require more resources. Make sure to tune the number of partitions based on your expected workload and cluster size.

댓글

댓글 쓰기

이 블로그의 인기 게시물

ftplib.error_perm: 550 Permission denied

The 550 Permission denied error in ftplib occurs when the FTP server denies access to the specified file or directory. This error can be caused by several issues such as: Incorrect file path or directory path on the server. Lack of write permissions for the specified directory. The FTP user does not have the required permissions to upload to the given directory. The remote directory may not exist. Steps to Troubleshoot and Resolve the Issue Here are some steps you can try to fix the 550 Permission denied issue: 1. Check the File Path and Directory on the FTP Server Make sure the remote path you are trying to upload to exists on the FTP server. If you're uploading to a specific directory, you may need to create it first. You can use the following code to list the files and directories on the server to ensure your paths are correct: from ftplib import FTP def list_files (ftp_host, ftp_user, ftp_password) : try : ftp = FTP(ftp_host) ftp.login(...
자세한 내용 보기

Screenshot of a web page using Puppeteer with a specific browser width, 웹페이지 스크린샷

To get a screenshot of a web page using Puppeteer with a specific browser width, you can follow these steps: Install Puppeteer by running the command npm install puppeteer in your project directory. Require Puppeteer in your Node.js script: const puppeteer = require( 'puppeteer' ); Launch a new browser instance with the desired viewport size: const browser = await puppeteer.launch({ defaultViewport: { width : 1280 , height : 720 , }, }); Create a new page in the browser: const page = await browser. newPage () ; Navigate to the desired URL: await page. goto ( 'https://example.com' ); Take a screenshot of the page: await page .screenshot ({ path : 'screenshot.png' }); Close the browser instance: await browser.close() ; Here's the complete code: const puppeteer = require ( 'puppeteer' ); ( async ( ) => { const browser = await puppeteer.launch({ defaultViewport : { width : 1280 , ...
자세한 내용 보기

how to describe kafka topics, 카프카 토픽 정보 확인하기

To describe Kafka topics, you can use the kafka-topics.sh script, which is included in the Kafka distribution, or Kafka Admin APIs to programmatically fetch topic information. Here are the ways to describe Kafka topics: 1. Using kafka-topics.sh Script The kafka-topics.sh command-line tool allows you to describe topics on the Kafka cluster. You can use the --describe flag to get details like the number of partitions, replicas, leader, and ISR (in-sync replicas). Command Syntax: bin/kafka - topics . sh - - describe - - topic < topic - name > - - bootstrap - server < broker - address > Example: bin/kafka - topics . sh - - describe - - topic my - topic - - bootstrap - server localhost:9092 Output Example: Topic: my-topic PartitionCount: 3 ReplicationFactor: 2 Configs: segment.bytes= 1073741824 Topic: my-topic Partition: 0 Leader: 1 Replicas: 1 , 2 Isr: 1 , 2 Topic: my-topic Partition: 1 Leader: 2 Replicas: 2 , 1 Is...
자세한 내용 보기

JAVA 실행 스크립트

JAVA 실행 스크립트 launch start script #!/bin/bash PRG="$0" INSTANCE="$1" CATALINA_HOME="/usr/local/tomcat" JAVA_HOME="/usr/local/java" LANG="ko_KR.UTF-8" PRGDIR="$CATALINA_HOME/bin" LOGDIR="$CATALINA_HOME/logs" EXECUTABLE=catalina.sh CATALINA_BASE="/www/tomcat/$INSTANCE" MEMSIZE=`cat /proc/meminfo | fgrep MemTotal | awk '{print$2}'` ((GB=$MEMSIZE/ 1000000)) if [ $GB -ge 4 ]; then XMS=2g XMX=4g else XMS=512m XMX=1g fi JAVA_DEBUG_OPTS=" -Xdebug \ -Xrunjdwp:transport=dt_socket,address=4102,server=y,suspend=n \ -Dcom.sun.management.jmxremote \ -Dcom.sun.management.jmxremote.port=8999 \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.authenticate=false" JAVA_OPTS=" -server -Dserver=$INSTANCE \ -Dfile.encoding=UTF-8 \ ${...
자세한 내용 보기

[Ubuntu] eclipse + subclipse + subversion1.8.x with (javaHL 1.7)

1.이클립스 다운받아 압축 풀기 2.HELP > Eclipse MarketPlace 에서 Subclipse or Subversion 설치하기 3. JavaHL1.7.x 다운로드(SVN을 사용하기 위해 설치해야됨)     http://www.collab.net/downloads/subversion#tab-1 에서 다운로드(registration 필요)     "JavaHL 1.7.4 i386 rpm" 버전을 다운받아서 .deb 파일로 변환하여 설치 4. JavaHL .deb 설치파일 생성하기    .rpm 을 .deb 로 변환하는 프로그램은 alien    미설치시 아래와 같이 설치    $ sudo apt-get install alien    $ sudo alien -c *.rpm           CollabNetSubversion-client-1.7.5-1.i386.rpm -> collabnetsubversion-client_1.7.5-2_i386.deb    $./collabnetsubversion-client_1.7.5-2_i386.deb 5.eclipse 디렉토리의 eclipse.ini 파일을 아래와 같이 수정하고 eclipse 를 재시작 # #    -startup    plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar    --launcher.library    plugins/org.eclipse.equinox.launcher.gtk.linux.x86_1.1.100.v20110505    -product    org.eclipse.epp.package.jee.product ...
자세한 내용 보기

Using the MinIO API via curl

Using the MinIO API via curl is straightforward, as MinIO is compatible with Amazon S3 API, so most commands follow a similar syntax. Here’s a guide on how to use curl with the MinIO API for some common operations like uploading, downloading, and managing objects. Prerequisites Access Key and Secret Key : Obtain your MinIO Access Key and Secret Key. MinIO Endpoint : Know your MinIO server endpoint, e.g., http://localhost:9000 . Bucket : You may need an existing bucket name, or create a new one using the commands below. Authentication Header For requests to work with MinIO, you need to include authentication in the headers. MinIO uses AWS Signature Version 4 for signing requests. Common Examples 1. List Buckets To list all buckets in your MinIO account, use: curl -X GET \ - -url "http://localhost:9000/" \ - H "Authorization: AWS <AccessKey>:<Signature>" 2. Create a Bucket To create a new bucket, use: curl -X PUT \ - -url "htt...
자세한 내용 보기

What are 5 creative things I could do with my kids' art, 아이와 함께하는 창의적 놀이

Absolutely, preserving your kids' art while also finding creative ways to utilize and display it can be a wonderful solution. Here are five creative ideas to consider: Art Gallery Wall: Designate a section of a wall in your home as an art gallery. Frame selected pieces of your kids' art and arrange them in an organized or eclectic manner. You can change the display periodically to showcase different artworks, creating a rotating gallery. Customized Photo Books: Compile your children's art into a digital photo book or album. Many online services allow you to create customized photo books. Arrange the artwork, add captions, and create a keepsake that you can flip through whenever you want to reminisce. Transform into Functional Items: Turn your kids' artwork into functional items such as placemats, coasters, tote bags, or even custom-made wrapping paper. There are companies that can print your children's art onto these items, making them practical and sentime...
자세한 내용 보기

Offset out of range error in Kafka, 카프카 트러블슈팅

The "Offset out of range" error in Kafka typically occurs when a consumer tries to read an offset that no longer exists on the broker. This can happen for several reasons, such as: The offset is smaller than the earliest offset available for the topic partition (i.e., the messages have been deleted due to log retention policies). The offset is larger than the latest offset available for the topic partition (i.e., the consumer is trying to read beyond the latest message). Steps to Resolve "Offset Out of Range" Error 1. Check Current Offsets Before making changes, it's useful to understand the current offsets and the offset range available for the topic partitions. Use the kafka-consumer-groups.sh script to describe the consumer group: bin/kafka-consumer-groups.sh --bootstrap- server localhost: 9092 --describe -- group <consumer- group -id> This command will show the current offsets, log end offsets, and the lag for each partition. 2. Reset the...
자세한 내용 보기

Reactjs datetime range picker

To create a datetime range picker in React, you can use a library such as react-datetime-range-picker . This library provides a customizable date and time range picker component that you can easily integrate into your React application. Here's an example of how you can use it: Install the library using npm or yarn: npm install react-datetime- range -picker Import the necessary components into your React component: import React, { useState } from 'react' ; import DateTimeRangePicker from 'react-datetime-range-picker' ; Create a state to hold the selected date and time range: const [selectedRange, setSelectedRange] = useState({ startDate: null , endDate: null }); Render the DateTimeRangePicker component and handle the selection changes: ```jsx const handleRangeChange = (range) => { setSelectedRange(range); }; return ( ); ``` In the example above, the DateTimeRangePicker component is rendered with the onChange event handler and th...
자세한 내용 보기

[OS_Linux_Ubuntu] telnet & ssh 설치와 설정 (installation / configuration)

1. telnet 설치 및 설정 sudo apt-get install xinetd sudo apt-get install telnetd /etc/xinetd.conf 파일의 defualt{} 함수는 주석 처리 service telnet {      disable = no      flags = REUSE      socket_type = stream      wait = no      user = root      server = /usr/sbin/in.telnetd      log_on_failure += USERID } sudo /etc/init.d/xinetd restart 2. ssh 설치 및 설정 설치     $ sudo apt-get install ssh 포트 변경     $ sudo vim /etc/ssh/sshd_config     # port 22 에서 '#' 를 제거하고 원하는 포트로 변경     IF >>  X-window 사용하고 싶을 경우,                   /etc/ssh/ssh_config         위의 파일의 다음내용을 변경한다                 # ForwardX11 no -> ForwardX11 yes 로 설정변경) 데몬 재시작     $ sudo /etc/init.d/ssh restart ...
자세한 내용 보기