Offset out of range error in Kafka, 카프카 트러블슈팅

The "Offset out of range" error in Kafka typically occurs when a consumer tries to read an offset that no longer exists on the broker. This can happen for several reasons, such as:

  • The offset is smaller than the earliest offset available for the topic partition (i.e., the messages have been deleted due to log retention policies).
  • The offset is larger than the latest offset available for the topic partition (i.e., the consumer is trying to read beyond the latest message).

Steps to Resolve "Offset Out of Range" Error

1. Check Current Offsets

Before making changes, it's useful to understand the current offsets and the offset range available for the topic partitions. Use the kafka-consumer-groups.sh script to describe the consumer group:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <consumer-group-id>

This command will show the current offsets, log end offsets, and the lag for each partition.

2. Reset the Offsets

You can reset the offsets to a valid range using the kafka-consumer-groups.sh script. You have several options for resetting offsets:

a. Reset to the Earliest Offset

This sets the consumer's offset to the earliest available offset.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <consumer-group-id> --topic <topic-name> --reset-offsets --to-earliest --execute
b. Reset to the Latest Offset

This sets the consumer's offset to the latest offset (i.e., the end of the log).

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <consumer-group-id> --topic <topic-name> --reset-offsets --to-latest --execute
c. Reset to a Specific Offset

This sets the consumer's offset to a specific value.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <consumer-group-id> --topic <topic-name> --reset-offsets --to-offset <offset-value> --execute
d. Reset by Duration

This sets the consumer's offset to a specific duration ago (e.g., 1 hour ago).

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <consumer-group-id> --topic <topic-name> --reset-offsets --by-duration <duration> --execute

For example, to reset to offsets from 1 hour ago:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <consumer-group-id> --topic <topic-name> --reset-offsets --by-duration 1h --execute

3. Restart the Consumer

After resetting the offsets, restart your consumer application. It should now start consuming messages from the new offsets.

Additional Considerations

  • Retention Policies: Ensure your Kafka retention policies are correctly configured to retain messages for a sufficient period. You can configure log retention settings in the Kafka broker configuration (server.properties):

    log.retention.hours=168 # Retain logs for 7 days
log.retention.bytes=1073741824 # Retain logs until they reach 1 GB

  • Consumer Group Management: Be aware of how consumer groups manage offsets. Each consumer group tracks its own offsets for each partition of the topic. Ensure that consumers in the same group are coordinated correctly.

  • Monitoring and Alerts: Implement monitoring and alerting for your Kafka consumers and brokers to detect and respond to issues such as offset out of range errors.

Example Script for Resetting Offsets

Here's a sample script to reset offsets to the earliest available offset:

#!/bin/bash

# Variables
BOOTSTRAP_SERVER="localhost:9092"
CONSUMER_GROUP="<consumer-group-id>"
TOPIC="<topic-name>"

# Reset offsets to earliest
bin/kafka-consumer-groups.sh --bootstrap-server $BOOTSTRAP_SERVER --group $CONSUMER_GROUP --topic $TOPIC --reset-offsets --to-earliest --execute

# Describe consumer group to verify
bin/kafka-consumer-groups.sh --bootstrap-server $BOOTSTRAP_SERVER --describe --group $CONSUMER_GROUP

Replace <consumer-group-id> and <topic-name> with your specific consumer group ID and topic name.

By following these steps, you can resolve the "Offset out of range" error in Kafka and ensure your consumers are reading messages from valid offsets.

댓글

댓글 쓰기

이 블로그의 인기 게시물

PYTHONPATH, Python 모듈 환경설정

In Python, an environment variable for a module or Python environment is a way to configure certain settings or provide data paths before running a Python program. Environment variables can be used to: Control Python's runtime behavior (e.g., specifying paths for module search). Pass configuration or sensitive data (like API keys) to Python applications. Set up virtual environments for Python project isolation. Here are some common environment variables related to Python and its modules: 1. PYTHONPATH : This variable defines the search path for modules. It allows you to specify additional directories for Python to look for modules and packages. If you want Python to find your custom modules, you can set this environment variable. Example: export PYTHONPATH= "/path/to/your/module: $PYTHONPATH " In Windows: set PYTHONPATH=C:\ path \ to \your\ module ;%PYTHONPATH% This tells Python to also search for modules in /path/to/your/module . 2. PYTHONHOME : ...
자세한 내용 보기

You can use Sublime Text from the command line by utilizing the subl command

You can use Sublime Text from the command line by utilizing the subl command. Here’s how you can set it up and use it: 1. Install Sublime Text If you haven't already installed Sublime Text, download and install it from the Sublime Text website . 2. Add subl Command to Path (if not already done) On macOS: The subl command is typically added automatically, but if it isn't, follow these steps: Open Sublime Text . Press Cmd + Shift + P to open the command palette. Type Install 'subl' command in PATH and select the option to install it. If you prefer to do it manually, you can create a symbolic link: ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" /usr/ local /bin/ subl On Linux: If the subl command isn't available, you can create a symbolic link to it: sudo ln -s /opt/ sublime_text /sublime_text /u sr /local/ bin /subl On Windows: You can add Sublime Text to the system PATH manually by: Right-clicking on This...
자세한 내용 보기

git 명령어

$ git remote -v origin git@github.daumkakao.com:dennis-lee/vertx-sample.git (fetch) origin git@github.daumkakao.com:dennis-lee/vertx-sample.git (push) $ git log -2 commit 69b4285dd8d1df3debdc24c24a9ee2c22c797931 Author: dennis.lee Date: Mon May 29 18:29:43 2017 +0900 json prettify commit de8c8a8f8a55a4b9a4a89d13cfa04e5001004c1e Author: dennis.lee Date: Mon May 29 18:17:22 2017 +0900 added BodyHandler $ git status On branch master Your branch is up-to-date with 'origin/master'. Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git checkout -- ..." to discard changes in working directory) modified: pom.xml Untracked files: (use "git add ..." to include in what will be committed) src/main/java/net/sample/ no changes added to commit (use "git add" and/or "git commit -a") $ git branch b1 $ git branch b1 * master $ git checkout -b b2 M pom.xml Switched ...
자세한 내용 보기

[gRPC] server of Java and client of Typescript

Great! To develop two applications – a gRPC client in TypeScript using Avro and a gRPC server in Java using Avro , we’ll follow these steps. I’ll break it down into the server and client components. Plan Overview Server Application : Java-based gRPC server with Avro serialization. Exposes a gRPC service that sends/receives Avro-encoded data . Uses Gradle to manage dependencies. Client Application : TypeScript-based gRPC client. Communicates with the gRPC server using Avro-encoded messages . Uses grpc-web and a gRPC gateway to bridge between gRPC and the web frontend. 1. gRPC Server in Java with Avro Project Setup for Server Create a new Gradle project. Add the necessary dependencies (gRPC, Avro, and protobuf). Server Directory Structure grpc-server-java/ ├── build.gradle.kts ├── settings.gradle.kts ├── src/ │ ├── main/ │ │ ├── java/ # Java source files │ │ │ └── com /example/service/ # Service implementation │ │ └── proto/ ...
자세한 내용 보기

[Ubuntu] Apache2.4.x 설치

Apache2.4.x 버전부터는 APR 과 APR-util을 별로도 설치하여야 한다. 하위버전에서는 설치파일에 포함되어 있었으나 버전 업이 되면서 삭제되어 configure를 실행하면 아래와 같은 에러메시지를 발생하면서 종료된다. Configuring Apache Portable Runtime library ... Checking for APR... no Confgirue: error: APR not found. Please read the documentation http://apr.apache.org/   에서 다운받아 설치함 *APR(Apache Portable Runtime) 설치 $ wget http://mirror.sdunix.com/apache//apr/apr-1.4.6.tar.gz $ tar xvf apr-1.4.6.tar.gz ~/apr-1.4.6$ ./configure ~/apr-1.4.6$ ./make ~/apr-1.4.6$ ./sudo make install *ARP-Util 설치 $ wget http://apache.tt.co.kr//apr/apr-util-1.4.1.tar.gz $ tar xvf apr-util-1.4.1.tar.gz ~/apr-util-1.4.1$ ./configure --with-apr=/usr/local/apr ~/apr-util-1.4.1$ make ~/apr-util-1.4.1$ sudo make install Apache 재설치시 Error 발생. configure: error: pcre-config for libpcre not found.PCRE is required and available from http://pcre.org/ *PCRE(Perl Compatible Regular Expressions) 설치 $ ...
자세한 내용 보기

Create topic on Kafka with partition count, 카프카 토픽 생성하기

To create a Kafka topic with 100 partitions, you can use the Kafka command-line tool ( kafka-topics.sh or kafka-topics.bat for Windows). When creating a topic, you can specify the number of partitions and the replication factor. Here’s how to create a topic with 100 partitions: Step-by-step instructions: Open a terminal on your machine where Kafka is installed. Run the kafka-topics.sh command with the necessary options to create the topic: kafka - topics . sh - - create - - bootstrap - server localhost:9092 - - replication - factor 1 - - partitions 100 - - topic < topic_name > --bootstrap-server localhost:9092 : Specifies the Kafka broker (or cluster) address. Replace localhost:9092 with your actual Kafka broker address if it’s different. --replication-factor 1 : Specifies the number of replicas for each partition. In this example, it's set to 1 , but in a production environment, you’ll want a higher replication factor for redundancy. --partitions 10...
자세한 내용 보기

리눅스의 부팅과정 (프로세스, 서비스 관리)

보통 사용되지 않는 서비스나 사용 빈도가 적은 프로세스의 경우 stop 시켜두는 것이 시스템 사용 상의 이점이 있다. 자신에게 필요 없는 프로세스를 정지 한다거나 nice 값을 조정하여 실행 시에 프로세스의 우선순위를 변화시킬 수 있다. 이렇게 시스템 상황에 따라 적절히 서비스들을 관리하려면 현재 자신의 시스템의 여러가지 상황을 알아야 하는데, 이런 경우 프로세스 및 실시간 시스템 상황을 보는 명령어인 top 을 사용할 수 있다.  top [-][d delay][q][c][S][s][i][n][b] -d delay : delay 의 시간이 경과하면 현재 스크린을 갱신한다. -q : 이 옵션을 사용하면 스크린을 계속 갱신한다. -c : command list 전체를 보여준다. 즉 욥선을 사용한 것까지 모두 보여준다. -i : idle 상태와 zombie 프로세스는 무시한다.   * 이 외의 옵션은 man 파일을 참고 top 실행 화면 1. First Line :  2:59am up 6days, 10:36,  4 users,  load average    : 현재 시간 2:59 am , 부팅된지 6days and 10시간 36분, 접속해 있는 사용자수, 평균부하 2. Second Line : 61 processes: 56 sleeping, 3 running, 1 xombie, 1 stopped 3. Third Line : CPU states: 36.4% user, 63.5% system, 0.0% nice, 0.0% idle     유저모드에서의 CPU 시간이 36.4%     시스템모드(프로세스를 위해서 커널이 사용한 CPU의 시간)에서의 CPU시간이 63.5%     0.0% nice는 nice로 nice value를 음수로 주어 우선순위를 높이는 경우에 해당하는 모드     0.0% idle(...
자세한 내용 보기

Auto-populate a calendar in an MUI (Material-UI) TextField component

To auto-populate a calendar in an MUI (Material-UI) TextField component, you can make use of the MUI DatePicker component. Here's an example of how you can achieve this: First, make sure you have installed the necessary dependencies. You'll need Material-UI and its pickers library. npm install @material-ui/core @material-ui/pickers Import the required components and functions in your code: import React, { useState } from 'react' ; import TextField from '@material-ui/core/TextField' ; import { KeyboardDatePicker, MuiPickersUtilsProvider } from '@material-ui/pickers' ; import DateFnsUtils from '@date-io/date-fns' ; Create a functional component and define the state to hold the selected date value: const MyForm = () => { const [selectedDate, setSelectedDate] = useState( null ); const handleDateChange = (date) => { setSelectedDate(date); }; return ( <MuiPickersUtilsProvider utils={DateFnsUtils}>...
자세한 내용 보기

The pierce selector in Puppeteer

The pierce selector in Puppeteer is an experimental feature that allows you to query shadow DOM elements more easily. Traditional CSS selectors don't pierce through shadow roots because they encapsulate their content to prevent styles and scripts from leaking in or out. The pierce selector solves this by bypassing the shadow DOM boundary and directly selecting elements within shadow trees. How to Use the Pierce Selector in Puppeteer Here’s an example of how to use it effectively: const puppeteer = require ( 'puppeteer' ); ( async ( ) => { const browser = await puppeteer.launch({ headless : false }); const page = await browser.newPage(); // Navigate to a page with shadow DOM await page.goto( 'https://example.com' ); // Select an element inside a shadow DOM using the pierce selector const element = await page.$( 'pierce:#shadow-element-id' ); if (element) { console .log( 'Shadow element found!' ); awai...
자세한 내용 보기