In HBase, the "memory to disk" flush operation

In HBase, the "memory to disk" flush operation (or "memstore flush") happens when data in the memstore (in-memory storage) is flushed to disk as HFiles in HDFS. This flush can occur based on multiple triggers rather than a strict time interval, including:

  1. Memstore Size Limit: The primary trigger is when the size of the memstore reaches a defined threshold. The default maximum memstore size per RegionServer is set by the configuration parameter hbase.regionserver.global.memstore.size, typically as a fraction of the total heap size. When the memstore fills up, it triggers an automatic flush to disk.

  2. Time-Based Flush (Optional): Although time-based flushes are not the main flush trigger, HBase provides an optional parameter for a maximum delay, which can ensure that data gets written to disk within a certain time, even if the memstore size limit hasn't been reached. This parameter is:

    • hbase.hregion.memstore.flush.period: This sets the maximum time (in milliseconds) that data can stay in memory before it must be flushed to disk. By default, this is set to 0 (disabled), meaning there is no strict time limit, and only the memstore size triggers the flush. If enabled, it will enforce a flush after the specified time, regardless of the memstore size.

    For example:

    hbase.hregion.memstore.flush.period=3600000  # Flush after 1 hour

  3. Manual Flush: You can also manually flush data by calling flush() on a table or RegionServer using the HBase shell or API. This is useful in certain situations for performance tuning or maintenance.

Best Practice

Leaving hbase.hregion.memstore.flush.period disabled (the default) generally provides optimal performance, as HBase will flush based on memory thresholds instead of time, which is often more efficient. However, in scenarios where you want to ensure data persistence on disk more frequently (such as during a heavy load with risk of data loss in case of crashes), you can configure this parameter to enforce a periodic flush.

댓글

댓글 쓰기

이 블로그의 인기 게시물

Using the MinIO API via curl

Using the MinIO API via curl is straightforward, as MinIO is compatible with Amazon S3 API, so most commands follow a similar syntax. Here’s a guide on how to use curl with the MinIO API for some common operations like uploading, downloading, and managing objects. Prerequisites Access Key and Secret Key : Obtain your MinIO Access Key and Secret Key. MinIO Endpoint : Know your MinIO server endpoint, e.g., http://localhost:9000 . Bucket : You may need an existing bucket name, or create a new one using the commands below. Authentication Header For requests to work with MinIO, you need to include authentication in the headers. MinIO uses AWS Signature Version 4 for signing requests. Common Examples 1. List Buckets To list all buckets in your MinIO account, use: curl -X GET \ - -url "http://localhost:9000/" \ - H "Authorization: AWS <AccessKey>:<Signature>" 2. Create a Bucket To create a new bucket, use: curl -X PUT \ - -url "htt...
자세한 내용 보기

vsftpd default directory

The default directory for FTP users in vsftpd depends on how the FTP server is configured and the user’s home directory setup. Below are different configurations and their impact on the default directory that users are placed in when they connect to the server. 1. Default Directory Behavior with vsftpd When a user logs in via FTP, they are typically placed in their home directory . By default, the home directory is located at /home/<username> , where <username> is the name of the FTP user. For example: If the FTP user is ftpuser , the default login directory will be /home/ftpuser/ . 2. Changing the Default Directory for Users A. Using local_root Configuration You can specify a custom directory (other than the user’s home directory) for FTP users by setting the local_root parameter in the vsftpd configuration file. Open the vsftpd configuration file: sudo nano /etc/ vsftpd.conf Add the following line to specify a new directory for all FTP users: lo...
자세한 내용 보기

[Ubuntu] *.deb 파일 설치 방법

확장자가 .deb 인 파일은 원래 Debian 리눅스에서의 설치파일입니다. Ubuntu 는 원래 Debian 리눅스에서 파생되었기 때문에, 많은 부분에서 서로 닮아 있습니다. 따라서 별다른 변환 절차 없이 아래와 같이 바로 설치가 가능합니다. [user@hostname]# sudo dpkg -i *.deb *.rpm 파일을 *.deb 파일로 변환 [user@hostname]$ sudo apt-get install alien [user@hostname]$ sudo alien -c *.rpm
자세한 내용 보기

Offset out of range error in Kafka, 카프카 트러블슈팅

The "Offset out of range" error in Kafka typically occurs when a consumer tries to read an offset that no longer exists on the broker. This can happen for several reasons, such as: The offset is smaller than the earliest offset available for the topic partition (i.e., the messages have been deleted due to log retention policies). The offset is larger than the latest offset available for the topic partition (i.e., the consumer is trying to read beyond the latest message). Steps to Resolve "Offset Out of Range" Error 1. Check Current Offsets Before making changes, it's useful to understand the current offsets and the offset range available for the topic partitions. Use the kafka-consumer-groups.sh script to describe the consumer group: bin/kafka-consumer-groups.sh --bootstrap- server localhost: 9092 --describe -- group <consumer- group -id> This command will show the current offsets, log end offsets, and the lag for each partition. 2. Reset the...
자세한 내용 보기

리눅스 (cron - 주기적 작업실행 데몬)

cron 데몬의 가동은 /etc/rc.d/init.d/crond 에 있다. 보통 시스템의 어떤 런레벨이든지 cron 데몬은 부팅 시 시작하도록 되어 있는데, 굳이 cron 데몬을 죽이거나 다시 시작하기 위해서는 위의 /etc/rc.d/init.d/crond 실행 스크립트를 실행하면 된다. cron 데몬에 작업을 설정하기 위해서는 cron table 을 설정하는 crontab 이라는 실행파일을 사용한다. cron table 을 편집하기 위해서는 crontab -e 명령을 사용한다. 입력 형식은 아래와 같다. #crontab -e 01 4 * * *  /etc/rc.d/init.d/network stop 15 4 * * * ~/scripts/my_script.sh 00 5 * * * /etc/rc.d/init.d/network start 위의 내용이 cron table 에 입력된 내용인데 이러한 입력형식은 아래와 같다. M H D m d cmd-line 필드명 범위 설명 M 0-59,* 분을 의미하며, *는 모든 볌위를 말함 H 0-23,* 시를 의미함 D 1-31,* 날짜를 의미함 m 1-12,* 달을 의미함 d 0-7,* 요일을 의미하며,0과 7은 일요일(Sun)을 의미함 cmd-line 실행할 명령을 입력 각자의 사용자가 등록한 cron table 은 /var/spool/cron 디렉토리에 저장된다. 입력 내용의 다른 예를 보면, 30  13  *  * 1-5  mail -s "It's 2pm" root% Dear root% % Are you sleeping?? 같은 경우는 월요일부터 금요일까지 오후 1시 30분에 해당 mail 명령을 실행하게 된다. 위에서 %문자는 개행문자 역할을 하며(풀어 쓰면 아래와 같은 명령이 된다.), 한줄이 넘어가는 명령을 쓰기 위해서는 백슬래쉬를 써주고 다시 개행해서 쓰면 된다. $mail -s "It's 2pm" ro...
자세한 내용 보기

리눅스 (하드링크&소프트링크)

장치파일 [root@hostname /root]# df -h Filesystem Size Used Avail Use% Mounted on /dev/hda1 39G 2.5G 35G 7% / /dev/hda5 132G 24G 102G 19% /home tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/hda2 49G 309M 46G 1% /backup [root@systech-dev /root]# ls -l /dev/hda1 brw-r----- 1 root disk 3, 1 2011-12-03 10:58 /dev/hda1 3, 1 은 커널이 장치를 식별하는 식별자이며, Major Number 식별자는 3, Minor Number 식별자는 1 임을 의미한다. Block 장치파일 I/O단위(Block) Character 장치파일 I/O단위(Byte) 하드링크 - inode 데이터블럭 낭비가 없다. [root@systech-dev /root/test]# touch a.txt [root@systech-dev /root/test]# ln a.txt b.txt [root@systech-dev /root/test]# cp -p a.txt c.txt [root@systech-dev /root/test]# ls -l total 0 -rw-r--r-- 2 root root 0 2012-11-29 00:40 a.txt -rw-r--r-- 2 root root 0 2012-11-29 00:40 b.txt -rw-r--r-- 1 root root 0 2012-11-29 00:40 c.txt a.txt 는 원본이고 b.txt 는 링크이다. (원본과 링크의 구별이 어려움) -rw-r--r-- 다음으로 오는 숫자는 하드링크가 걸린 갯수를 나타낸다. c.txt 는 하드링크가 자기 자신밖에 없으므로 1이 된다. ...
자세한 내용 보기

CDPEvents in puppeteer

In Puppeteer , CDPEvents refers to events emitted by the Chrome DevTools Protocol (CDP) . Puppeteer leverages CDP to interact with and control a Chromium-based browser. CDP provides detailed, low-level access to browser internals, such as network traffic, console logs, page lifecycle events, and more. You can listen to these CDP events through Puppeteer’s API to monitor or intercept browser activity. How to Listen for CDP Events in Puppeteer Enable the Required CDP Domain: Some events require enabling a particular domain (e.g., 'Network' , 'Page' , 'Runtime' ). Use page._client() to Access the CDP Session: Although it’s a bit lower-level, Puppeteer allows access to DevTools through the page._client() API. Example: Listening for Network Requests This example demonstrates how to intercept and log network requests using CDP. const puppeteer = require( 'puppeteer' ); ( async () => { const browser = await puppeteer.launch({ headless...
자세한 내용 보기

Using venv in Python

Using venv in Python allows you to create a virtual environment, which is an isolated environment to manage dependencies for a specific project. Here's how you can use it step by step: 1. Create a Virtual Environment Open your terminal or command prompt. Navigate to the folder where you want to create the virtual environment. Run the following command: python -m venv myenv Replace myenv with the name you want for your virtual environment. This will create a directory named myenv containing the virtual environment. 2. Activate the Virtual Environment Activation depends on your operating system: On Windows : myenv \S cripts \a ctivate On macOS/Linux : source myenv /bin/ activate After activation, your terminal prompt will change, showing the name of the virtual environment (e.g., (myenv) ). 3. Install Dependencies With the virtual environment activated, you can now install packages using pip : pip install package _n ame For example: pip install req...
자세한 내용 보기