DEFAULT_KEY_SERDE_CLASS_CONFIG in Kafka

The DEFAULT_KEY_SERDE_CLASS_CONFIG is a configuration property in Apache Kafka Streams. It specifies the default serializer/deserializer (Serde) to be used for the keys of messages in your Kafka Streams application.

What Is a Serde?

A Serde is a combination of a serializer and a deserializer:

  • Serializer: Converts objects into a byte array to send over Kafka.
  • Deserializer: Converts the byte array back into an object.

Kafka Streams requires Serdes to know how to handle data when reading from and writing to topics.

Usage of DEFAULT_KEY_SERDE_CLASS_CONFIG

This property allows you to specify the default Serde class for the key of messages processed by Kafka Streams. For example, if your keys are strings, you would set the default key Serde to Serdes.String().

Example Configuration in Java:

Properties props = new Properties();
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

Example Configuration in application.yml:

spring:
  kafka:
    streams:
      properties:
        default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
        default.value.serde: org.apache.kafka.common.serialization.Serdes$StringSerde

Common Serdes in Kafka Streams

Kafka Streams provides built-in Serdes for common data types:

  • StringSerde: For string keys/values.
    org.apache.kafka.common.serialization.Serdes.StringSerde
  • IntegerSerde: For integer keys/values.
    org.apache.kafka.common.serialization.Serdes.IntegerSerde
  • LongSerde: For long keys/values.
    org.apache.kafka.common.serialization.Serdes.LongSerde
  • DoubleSerde: For double keys/values.
    org.apache.kafka.common.serialization.Serdes.DoubleSerde
  • ByteArraySerde: For raw binary data.
    org.apache.kafka.common.serialization.Serdes.ByteArraySerde

Why Use DEFAULT_KEY_SERDE_CLASS_CONFIG?

  1. Convenience: Sets a global default, so you don't need to specify the Serde for every stream/table.
  2. Consistency: Ensures that all keys are handled the same way unless overridden.

Overriding the Default Serde

If you need to process data differently for a specific stream, you can override the default Serde when defining the stream:

KStream<Integer, String> stream = builder.stream(
    "source-topic",
    Consumed.with(Serdes.Integer(), Serdes.String()) // Custom key/value Serdes
);

Let me know if you'd like to explore custom Serdes!

댓글

댓글 쓰기

이 블로그의 인기 게시물

PYTHONPATH, Python 모듈 환경설정

In Python, an environment variable for a module or Python environment is a way to configure certain settings or provide data paths before running a Python program. Environment variables can be used to: Control Python's runtime behavior (e.g., specifying paths for module search). Pass configuration or sensitive data (like API keys) to Python applications. Set up virtual environments for Python project isolation. Here are some common environment variables related to Python and its modules: 1. PYTHONPATH : This variable defines the search path for modules. It allows you to specify additional directories for Python to look for modules and packages. If you want Python to find your custom modules, you can set this environment variable. Example: export PYTHONPATH= "/path/to/your/module: $PYTHONPATH " In Windows: set PYTHONPATH=C:\ path \ to \your\ module ;%PYTHONPATH% This tells Python to also search for modules in /path/to/your/module . 2. PYTHONHOME : ...
자세한 내용 보기

You can use Sublime Text from the command line by utilizing the subl command

You can use Sublime Text from the command line by utilizing the subl command. Here’s how you can set it up and use it: 1. Install Sublime Text If you haven't already installed Sublime Text, download and install it from the Sublime Text website . 2. Add subl Command to Path (if not already done) On macOS: The subl command is typically added automatically, but if it isn't, follow these steps: Open Sublime Text . Press Cmd + Shift + P to open the command palette. Type Install 'subl' command in PATH and select the option to install it. If you prefer to do it manually, you can create a symbolic link: ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" /usr/ local /bin/ subl On Linux: If the subl command isn't available, you can create a symbolic link to it: sudo ln -s /opt/ sublime_text /sublime_text /u sr /local/ bin /subl On Windows: You can add Sublime Text to the system PATH manually by: Right-clicking on This...
자세한 내용 보기

git 명령어

$ git remote -v origin git@github.daumkakao.com:dennis-lee/vertx-sample.git (fetch) origin git@github.daumkakao.com:dennis-lee/vertx-sample.git (push) $ git log -2 commit 69b4285dd8d1df3debdc24c24a9ee2c22c797931 Author: dennis.lee Date: Mon May 29 18:29:43 2017 +0900 json prettify commit de8c8a8f8a55a4b9a4a89d13cfa04e5001004c1e Author: dennis.lee Date: Mon May 29 18:17:22 2017 +0900 added BodyHandler $ git status On branch master Your branch is up-to-date with 'origin/master'. Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git checkout -- ..." to discard changes in working directory) modified: pom.xml Untracked files: (use "git add ..." to include in what will be committed) src/main/java/net/sample/ no changes added to commit (use "git add" and/or "git commit -a") $ git branch b1 $ git branch b1 * master $ git checkout -b b2 M pom.xml Switched ...
자세한 내용 보기

[gRPC] server of Java and client of Typescript

Great! To develop two applications – a gRPC client in TypeScript using Avro and a gRPC server in Java using Avro , we’ll follow these steps. I’ll break it down into the server and client components. Plan Overview Server Application : Java-based gRPC server with Avro serialization. Exposes a gRPC service that sends/receives Avro-encoded data . Uses Gradle to manage dependencies. Client Application : TypeScript-based gRPC client. Communicates with the gRPC server using Avro-encoded messages . Uses grpc-web and a gRPC gateway to bridge between gRPC and the web frontend. 1. gRPC Server in Java with Avro Project Setup for Server Create a new Gradle project. Add the necessary dependencies (gRPC, Avro, and protobuf). Server Directory Structure grpc-server-java/ ├── build.gradle.kts ├── settings.gradle.kts ├── src/ │ ├── main/ │ │ ├── java/ # Java source files │ │ │ └── com /example/service/ # Service implementation │ │ └── proto/ ...
자세한 내용 보기

[Ubuntu] Apache2.4.x 설치

Apache2.4.x 버전부터는 APR 과 APR-util을 별로도 설치하여야 한다. 하위버전에서는 설치파일에 포함되어 있었으나 버전 업이 되면서 삭제되어 configure를 실행하면 아래와 같은 에러메시지를 발생하면서 종료된다. Configuring Apache Portable Runtime library ... Checking for APR... no Confgirue: error: APR not found. Please read the documentation http://apr.apache.org/   에서 다운받아 설치함 *APR(Apache Portable Runtime) 설치 $ wget http://mirror.sdunix.com/apache//apr/apr-1.4.6.tar.gz $ tar xvf apr-1.4.6.tar.gz ~/apr-1.4.6$ ./configure ~/apr-1.4.6$ ./make ~/apr-1.4.6$ ./sudo make install *ARP-Util 설치 $ wget http://apache.tt.co.kr//apr/apr-util-1.4.1.tar.gz $ tar xvf apr-util-1.4.1.tar.gz ~/apr-util-1.4.1$ ./configure --with-apr=/usr/local/apr ~/apr-util-1.4.1$ make ~/apr-util-1.4.1$ sudo make install Apache 재설치시 Error 발생. configure: error: pcre-config for libpcre not found.PCRE is required and available from http://pcre.org/ *PCRE(Perl Compatible Regular Expressions) 설치 $ ...
자세한 내용 보기

Create topic on Kafka with partition count, 카프카 토픽 생성하기

To create a Kafka topic with 100 partitions, you can use the Kafka command-line tool ( kafka-topics.sh or kafka-topics.bat for Windows). When creating a topic, you can specify the number of partitions and the replication factor. Here’s how to create a topic with 100 partitions: Step-by-step instructions: Open a terminal on your machine where Kafka is installed. Run the kafka-topics.sh command with the necessary options to create the topic: kafka - topics . sh - - create - - bootstrap - server localhost:9092 - - replication - factor 1 - - partitions 100 - - topic < topic_name > --bootstrap-server localhost:9092 : Specifies the Kafka broker (or cluster) address. Replace localhost:9092 with your actual Kafka broker address if it’s different. --replication-factor 1 : Specifies the number of replicas for each partition. In this example, it's set to 1 , but in a production environment, you’ll want a higher replication factor for redundancy. --partitions 10...
자세한 내용 보기

리눅스의 부팅과정 (프로세스, 서비스 관리)

보통 사용되지 않는 서비스나 사용 빈도가 적은 프로세스의 경우 stop 시켜두는 것이 시스템 사용 상의 이점이 있다. 자신에게 필요 없는 프로세스를 정지 한다거나 nice 값을 조정하여 실행 시에 프로세스의 우선순위를 변화시킬 수 있다. 이렇게 시스템 상황에 따라 적절히 서비스들을 관리하려면 현재 자신의 시스템의 여러가지 상황을 알아야 하는데, 이런 경우 프로세스 및 실시간 시스템 상황을 보는 명령어인 top 을 사용할 수 있다.  top [-][d delay][q][c][S][s][i][n][b] -d delay : delay 의 시간이 경과하면 현재 스크린을 갱신한다. -q : 이 옵션을 사용하면 스크린을 계속 갱신한다. -c : command list 전체를 보여준다. 즉 욥선을 사용한 것까지 모두 보여준다. -i : idle 상태와 zombie 프로세스는 무시한다.   * 이 외의 옵션은 man 파일을 참고 top 실행 화면 1. First Line :  2:59am up 6days, 10:36,  4 users,  load average    : 현재 시간 2:59 am , 부팅된지 6days and 10시간 36분, 접속해 있는 사용자수, 평균부하 2. Second Line : 61 processes: 56 sleeping, 3 running, 1 xombie, 1 stopped 3. Third Line : CPU states: 36.4% user, 63.5% system, 0.0% nice, 0.0% idle     유저모드에서의 CPU 시간이 36.4%     시스템모드(프로세스를 위해서 커널이 사용한 CPU의 시간)에서의 CPU시간이 63.5%     0.0% nice는 nice로 nice value를 음수로 주어 우선순위를 높이는 경우에 해당하는 모드     0.0% idle(...
자세한 내용 보기

Auto-populate a calendar in an MUI (Material-UI) TextField component

To auto-populate a calendar in an MUI (Material-UI) TextField component, you can make use of the MUI DatePicker component. Here's an example of how you can achieve this: First, make sure you have installed the necessary dependencies. You'll need Material-UI and its pickers library. npm install @material-ui/core @material-ui/pickers Import the required components and functions in your code: import React, { useState } from 'react' ; import TextField from '@material-ui/core/TextField' ; import { KeyboardDatePicker, MuiPickersUtilsProvider } from '@material-ui/pickers' ; import DateFnsUtils from '@date-io/date-fns' ; Create a functional component and define the state to hold the selected date value: const MyForm = () => { const [selectedDate, setSelectedDate] = useState( null ); const handleDateChange = (date) => { setSelectedDate(date); }; return ( <MuiPickersUtilsProvider utils={DateFnsUtils}>...
자세한 내용 보기

The pierce selector in Puppeteer

The pierce selector in Puppeteer is an experimental feature that allows you to query shadow DOM elements more easily. Traditional CSS selectors don't pierce through shadow roots because they encapsulate their content to prevent styles and scripts from leaking in or out. The pierce selector solves this by bypassing the shadow DOM boundary and directly selecting elements within shadow trees. How to Use the Pierce Selector in Puppeteer Here’s an example of how to use it effectively: const puppeteer = require ( 'puppeteer' ); ( async ( ) => { const browser = await puppeteer.launch({ headless : false }); const page = await browser.newPage(); // Navigate to a page with shadow DOM await page.goto( 'https://example.com' ); // Select an element inside a shadow DOM using the pierce selector const element = await page.$( 'pierce:#shadow-element-id' ); if (element) { console .log( 'Shadow element found!' ); awai...
자세한 내용 보기