Create topic on Kafka with partition count, 카프카 토픽 생성하기

To create a Kafka topic with 100 partitions, you can use the Kafka command-line tool (kafka-topics.sh or kafka-topics.bat for Windows). When creating a topic, you can specify the number of partitions and the replication factor.

Here’s how to create a topic with 100 partitions:

Step-by-step instructions:

  1. Open a terminal on your machine where Kafka is installed.

  2. Run the kafka-topics.sh command with the necessary options to create the topic:

    kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 100 --topic <topic_name>
    • --bootstrap-server localhost:9092: Specifies the Kafka broker (or cluster) address. Replace localhost:9092 with your actual Kafka broker address if it’s different.
    • --replication-factor 1: Specifies the number of replicas for each partition. In this example, it's set to 1, but in a production environment, you’ll want a higher replication factor for redundancy.
    • --partitions 100: Specifies the number of partitions (in this case, 100).
    • --topic <topic_name>: Replace <topic_name> with the name of the topic you want to create.

Example:

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 100 --topic my-large-topic

3. Verify the Topic Creation

After running the command, you can verify that the topic was created successfully and that it has 100 partitions:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-large-topic

This will give you a detailed description of the topic, including the number of partitions and replication factor.

Notes:

  • Replication factor: In a production Kafka setup with multiple brokers, you'll usually want to set a replication factor higher than 1 for fault tolerance.
  • Number of partitions: More partitions allow for greater parallelism in Kafka but require more resources. Make sure to tune the number of partitions based on your expected workload and cluster size.

댓글

댓글 쓰기

이 블로그의 인기 게시물

To switch to a specific tag in a Git repository

To switch to a specific tag in a Git repository, you can use the git checkout or git switch command. Here's how you can do it: 1. List Available Tags First, you may want to see a list of all available tags in the repository: git tag 2. Checkout a Specific Tag Once you know the tag you want to switch to, you can use either git checkout or git switch (if you're using a newer version of Git, which recommends git switch for checking out branches). Using git checkout : git checkout <tag- name > Using git switch (recommended for newer versions of Git): git switch -- detach <tag- name > The --detach flag is necessary because tags are not branches; they are just pointers to specific commits. Using --detach makes your HEAD point to the tag without modifying any branch. 3. Verify the Checkout After switching to the tag, verify that you're on the correct commit: git status This will show the tag you're currently on (you'll see somethi...
자세한 내용 보기

How to checkout branch of remote git, 깃 리모트 브랜치 체크아웃

To check out a branch from a remote Git repository, you can follow these steps: 1. Fetch the remote branches First, ensure that your local repository is aware of the remote branches by using git fetch . This updates your local references to the remote branches. git fetch origin 2. Check out the remote branch Once the remote branches are fetched, you can check out the desired branch using git checkout -b and specifying the remote branch. Command: git checkout -b < local -branch- name > origin/<remote-branch- name > Explanation: <local-branch-name> : The name you want to give to your local branch. origin/<remote-branch-name> : Refers to the branch on the remote called origin (which is the default name for the main remote repository) and the branch you want to check out. Example: If the remote branch is called feature-branch and you want to create a local branch with the same name, you would run: git checkout -b feature -branch origin / feature -...
자세한 내용 보기

Using the MinIO API via curl

Using the MinIO API via curl is straightforward, as MinIO is compatible with Amazon S3 API, so most commands follow a similar syntax. Here’s a guide on how to use curl with the MinIO API for some common operations like uploading, downloading, and managing objects. Prerequisites Access Key and Secret Key : Obtain your MinIO Access Key and Secret Key. MinIO Endpoint : Know your MinIO server endpoint, e.g., http://localhost:9000 . Bucket : You may need an existing bucket name, or create a new one using the commands below. Authentication Header For requests to work with MinIO, you need to include authentication in the headers. MinIO uses AWS Signature Version 4 for signing requests. Common Examples 1. List Buckets To list all buckets in your MinIO account, use: curl -X GET \ - -url "http://localhost:9000/" \ - H "Authorization: AWS <AccessKey>:<Signature>" 2. Create a Bucket To create a new bucket, use: curl -X PUT \ - -url "htt...
자세한 내용 보기

To download a file from MinIO using Spring Boot, 스프링부트 Minio 사용하기

To download a file from MinIO using Spring Boot, you can utilize the MinIO Java SDK. Here's an example of how you can achieve this: Add the MinIO dependency to your pom.xml file: < dependency > < groupId > io.minio </ groupId > < artifactId > minio </ artifactId > < version > 7.1.0 </ version > </ dependency > Create a configuration file (e.g., MinioConfig.java ) to establish a connection with your MinIO server: ```java import io.minio.MinioClient; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class MinioConfig { @Value( " ${minio.endpoint} " ) private String endpoint; @Value( " ${minio.accessKey} " ) private String accessKey; @Value( " ${minio.secretKey} " ) private String secretKey; @Bean public MinioClient minioClient() { re...
자세한 내용 보기

리눅스의 부팅과정 (프로세스, 서비스 관리)

보통 사용되지 않는 서비스나 사용 빈도가 적은 프로세스의 경우 stop 시켜두는 것이 시스템 사용 상의 이점이 있다. 자신에게 필요 없는 프로세스를 정지 한다거나 nice 값을 조정하여 실행 시에 프로세스의 우선순위를 변화시킬 수 있다. 이렇게 시스템 상황에 따라 적절히 서비스들을 관리하려면 현재 자신의 시스템의 여러가지 상황을 알아야 하는데, 이런 경우 프로세스 및 실시간 시스템 상황을 보는 명령어인 top 을 사용할 수 있다.  top [-][d delay][q][c][S][s][i][n][b] -d delay : delay 의 시간이 경과하면 현재 스크린을 갱신한다. -q : 이 옵션을 사용하면 스크린을 계속 갱신한다. -c : command list 전체를 보여준다. 즉 욥선을 사용한 것까지 모두 보여준다. -i : idle 상태와 zombie 프로세스는 무시한다.   * 이 외의 옵션은 man 파일을 참고 top 실행 화면 1. First Line :  2:59am up 6days, 10:36,  4 users,  load average    : 현재 시간 2:59 am , 부팅된지 6days and 10시간 36분, 접속해 있는 사용자수, 평균부하 2. Second Line : 61 processes: 56 sleeping, 3 running, 1 xombie, 1 stopped 3. Third Line : CPU states: 36.4% user, 63.5% system, 0.0% nice, 0.0% idle     유저모드에서의 CPU 시간이 36.4%     시스템모드(프로세스를 위해서 커널이 사용한 CPU의 시간)에서의 CPU시간이 63.5%     0.0% nice는 nice로 nice value를 음수로 주어 우선순위를 높이는 경우에 해당하는 모드     0.0% idle(...
자세한 내용 보기

Chromium 개발 환경 세팅, 크로미움 개발 준비하기

Chromium 개발 환경을 구축하는 것은 여러 단계로 이루어져 있으며, 이를 위해서는 적절한 도구와 의존성을 설치해야 합니다. 아래는 일반적인 Chromium 개발 환경을 설정하는 방법입니다. 1. 시스템 요구사항 확인 Chromium은 리눅스, 맥OS, 윈도우에서 빌드할 수 있습니다. 이 가이드에서는 Ubuntu를 기준으로 설명하지만, 다른 운영체제에서도 유사한 과정으로 설정할 수 있습니다. 메모리 : 최소 16GB RAM (권장 32GB 이상) 디스크 공간 : 최소 100GB 이상의 여유 공간 운영체제 : Ubuntu 20.04 이상 권장 2. 필수 패키지 설치 Chromium을 빌드하려면 여러 패키지가 필요합니다. 아래 명령어를 사용하여 설치합니다. sudo apt- get update sudo apt- get install build-essential git python3 python3-pip sudo apt- get install libx11- dev libxcomposite- dev libxcursor- dev libxdamage- dev libxrandr- dev libxtst- dev libxss- dev libglib2 .0 - dev sudo apt- get install libnss3- dev libasound2- dev libpulse- dev libjpeg- dev libpng- dev sudo apt- get install curl nodejs npm 3. Chromium 소스 코드 가져오기 Chromium 소스 코드를 가져오려면 depot_tools 라는 도구가 필요합니다. depot_tools 설치 : git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git export PATH=` pwd `/depot_tools: " $PATH " 이 경로를 영구적으로 PATH에 추...
자세한 내용 보기

Joining an additional control plane node to an existing Kubernetes cluster

Joining an additional control plane node to an existing Kubernetes cluster involves specific steps to ensure proper synchronization between the control plane components. Below is the detailed process: 1. Pre-Requisites Ensure the following on the new control plane node: Same Kubernetes Version : The new node must have the same Kubernetes version installed as the existing control plane. kubeadm version Kubeadm, Kubelet, and Kubectl Installed : Ensure these tools are installed and configured. Network Configuration : The new control plane must have network connectivity to the existing control plane nodes. 2. Retrieve the Join Command On an existing control plane node, generate the kubeadm join command with the --control-plane flag: kubeadm token create -- print -join-command You will get output similar to: kubeadm join <control-plane-endpoint> : 6443 --token <token> \ --discovery-token- ca -cert-hash sha256 : <hash> \ --control-plane 3. ...
자세한 내용 보기

urllib3 with proxy settings

To set a proxy using urllib3 , you typically configure it through a ProxyManager , which allows you to route requests through a specific proxy server. Here’s a quick guide: Step 1: Install urllib3 Make sure you have urllib3 installed: pip install urllib3 Step 2: Use ProxyManager Use the ProxyManager class to configure your proxy. Here’s an example: import urllib3 # Define the proxy URL proxy_url = "http://your_proxy_server:port" # Create a ProxyManager instance http = urllib3.ProxyManager(proxy_url) # Send a GET request through the proxy response = http.request( "GET" , "http://example.com" ) # Print the response print(response.data.decode( "utf-8" )) Step 3: Add Authentication (Optional) If your proxy server requires authentication, include it in the proxy URL: proxy_url = "http://username:password@your_proxy_server:port" http = urllib3.ProxyManager(proxy_url) Additional Tips For HTTPS URLs, ProxyManager ...
자세한 내용 보기

CDPEvents in puppeteer

In Puppeteer , CDPEvents refers to events emitted by the Chrome DevTools Protocol (CDP) . Puppeteer leverages CDP to interact with and control a Chromium-based browser. CDP provides detailed, low-level access to browser internals, such as network traffic, console logs, page lifecycle events, and more. You can listen to these CDP events through Puppeteer’s API to monitor or intercept browser activity. How to Listen for CDP Events in Puppeteer Enable the Required CDP Domain: Some events require enabling a particular domain (e.g., 'Network' , 'Page' , 'Runtime' ). Use page._client() to Access the CDP Session: Although it’s a bit lower-level, Puppeteer allows access to DevTools through the page._client() API. Example: Listening for Network Requests This example demonstrates how to intercept and log network requests using CDP. const puppeteer = require( 'puppeteer' ); ( async () => { const browser = await puppeteer.launch({ headless...
자세한 내용 보기

Avro + Grpc in python

Developing a gRPC client in Python that leverages Avro serialization involves several steps. gRPC is a high-performance RPC framework, while Apache Avro is a serialization framework used for efficient data exchange. In this example, we will show how to: Define gRPC services using Protocol Buffers ( .proto ). Serialize and deserialize data using Avro. Implement the gRPC server and client in Python. We’ll walk through the Avro-based gRPC client-server application in Python, using gRPC to transmit messages and Avro to serialize the messages. Overview of the Application Avro is used to serialize complex data structures (e.g., dictionaries) into bytes. gRPC transmits the serialized Avro bytes over the network. Python will be used to implement the gRPC client and server. Steps to Develop the gRPC + Avro Application Step 1: Install Required Dependencies Ensure you have the following dependencies installed: pip install grpcio grpcio-tools avro-python3 grpcio and ...
자세한 내용 보기