In HBase, the "memory to disk" flush operation

In HBase, the "memory to disk" flush operation (or "memstore flush") happens when data in the memstore (in-memory storage) is flushed to disk as HFiles in HDFS. This flush can occur based on multiple triggers rather than a strict time interval, including:

  1. Memstore Size Limit: The primary trigger is when the size of the memstore reaches a defined threshold. The default maximum memstore size per RegionServer is set by the configuration parameter hbase.regionserver.global.memstore.size, typically as a fraction of the total heap size. When the memstore fills up, it triggers an automatic flush to disk.

  2. Time-Based Flush (Optional): Although time-based flushes are not the main flush trigger, HBase provides an optional parameter for a maximum delay, which can ensure that data gets written to disk within a certain time, even if the memstore size limit hasn't been reached. This parameter is:

    • hbase.hregion.memstore.flush.period: This sets the maximum time (in milliseconds) that data can stay in memory before it must be flushed to disk. By default, this is set to 0 (disabled), meaning there is no strict time limit, and only the memstore size triggers the flush. If enabled, it will enforce a flush after the specified time, regardless of the memstore size.

    For example:

    hbase.hregion.memstore.flush.period=3600000  # Flush after 1 hour

  3. Manual Flush: You can also manually flush data by calling flush() on a table or RegionServer using the HBase shell or API. This is useful in certain situations for performance tuning or maintenance.

Best Practice

Leaving hbase.hregion.memstore.flush.period disabled (the default) generally provides optimal performance, as HBase will flush based on memory thresholds instead of time, which is often more efficient. However, in scenarios where you want to ensure data persistence on disk more frequently (such as during a heavy load with risk of data loss in case of crashes), you can configure this parameter to enforce a periodic flush.

댓글

댓글 쓰기

이 블로그의 인기 게시물

Develop a Kafka producer to send messages to a Kafka topic

Here’s a guide and code examples to develop a Kafka producer to send messages to a Kafka topic. The examples cover both plain Kafka Producer API and Spring Kafka Producer for convenience. 1. Plain Kafka Producer API Maven Dependencies Add the Kafka client dependency in your pom.xml : < dependency > < groupId > org.apache.kafka </ groupId > < artifactId > kafka-clients </ artifactId > < version > 3.6.0 </ version > <!-- Replace with your Kafka version --> </ dependency > Java Code Example This example demonstrates producing messages to a Kafka topic using the plain Kafka Producer API. import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.clients.producer.RecordMetadata; import java.util.Properties; public class KafkaMessageProducer { public static void main( String [] args) { // Configure Kafka produce...
자세한 내용 보기

Using the MinIO API via curl

Using the MinIO API via curl is straightforward, as MinIO is compatible with Amazon S3 API, so most commands follow a similar syntax. Here’s a guide on how to use curl with the MinIO API for some common operations like uploading, downloading, and managing objects. Prerequisites Access Key and Secret Key : Obtain your MinIO Access Key and Secret Key. MinIO Endpoint : Know your MinIO server endpoint, e.g., http://localhost:9000 . Bucket : You may need an existing bucket name, or create a new one using the commands below. Authentication Header For requests to work with MinIO, you need to include authentication in the headers. MinIO uses AWS Signature Version 4 for signing requests. Common Examples 1. List Buckets To list all buckets in your MinIO account, use: curl -X GET \ - -url "http://localhost:9000/" \ - H "Authorization: AWS <AccessKey>:<Signature>" 2. Create a Bucket To create a new bucket, use: curl -X PUT \ - -url "htt...
자세한 내용 보기

max_active_runs of Airflow

In Apache Airflow, if you want to ensure that a new run of a DAG doesn’t start before the previous one has completed, you can use the max_active_runs parameter in the DAG definition. Setting this parameter to 1 ensures that only one instance of the DAG is running at any given time. Setting max_active_runs for a DAG Here’s an example of how to set up a DAG with max_active_runs=1 : from airflow import DAG from airflow.operators.dummy_operator import DummyOperator from datetime import datetime # Define the DAG with DAG( 'my_dag' , description= 'A sample DAG' , schedule_interval= '@daily' , # Set your schedule start_date=datetime( 2023 , 1 , 1 ), catchup= False , max_active_runs= 1 # This prevents new runs before the previous run completes ) as dag: # Define tasks start = DummyOperator(task_id= 'start' ) end = DummyOperator(task_id= 'end' ) # Define task dependencies ...
자세한 내용 보기

Boilerplate for typescript server programing

Here are some recommended boilerplates and frameworks for TypeScript server programming, depending on your requirements: 1. NestJS (Highly Recommended for Enterprise-Grade Applications) NestJS is a progressive Node.js framework built with TypeScript. It provides an opinionated structure and is excellent for building scalable and maintainable server-side applications. Features : Dependency injection Modular architecture Built-in support for WebSockets, GraphQL, REST APIs Easy integration with tools like Swagger, TypeORM, and Mongoose Getting Started : Install Nest CLI: npm i -g @nestjs/cli nest new project-name Boilerplate is automatically generated with ready-to-use configurations for controllers, services, and modules. Website : https://nestjs.com/ 2. Express with TypeScript Express.js is lightweight and minimal, making it ideal for developers who prefer flexibility over opinionated frameworks. Boilerplate : You can start with an official or community-maintai...
자세한 내용 보기

difference between truncate and truncate_preserve in hbase

In HBase, both truncate and truncate_preserve commands are used to delete all the data in a table, but they behave differently in terms of how they handle the table’s schema and regions. 1. truncate Purpose : Deletes all data and regions in a table and then recreates it with the same schema. What It Does : When you run truncate , it: Disables the table. Deletes all data and metadata (including region splits) in the table. Drops all regions and recreates the table as a single region with the original schema. Re-enables the table. Effect on Regions : After truncate , all region splits are lost, so the table will start with only one region. This is useful for development or quick data purging but is inefficient for large tables that rely on pre-split regions for load balancing. Command Syntax : truncate 'table_name' 2. truncate_preserve Purpose : Deletes all data in a table but preserves the table’s regions and schema. What It Does : When you use truncate_...
자세한 내용 보기

What are 5 creative things I could do with my kids' art, 아이와 함께하는 창의적 놀이

Absolutely, preserving your kids' art while also finding creative ways to utilize and display it can be a wonderful solution. Here are five creative ideas to consider: Art Gallery Wall: Designate a section of a wall in your home as an art gallery. Frame selected pieces of your kids' art and arrange them in an organized or eclectic manner. You can change the display periodically to showcase different artworks, creating a rotating gallery. Customized Photo Books: Compile your children's art into a digital photo book or album. Many online services allow you to create customized photo books. Arrange the artwork, add captions, and create a keepsake that you can flip through whenever you want to reminisce. Transform into Functional Items: Turn your kids' artwork into functional items such as placemats, coasters, tote bags, or even custom-made wrapping paper. There are companies that can print your children's art onto these items, making them practical and sentime...
자세한 내용 보기

Parasite of korean movie

"Parasite" directed by Bong Joon-ho. It won four Academy Awards, including Best Picture, Best Director, Best Original Screenplay, and Best International Feature Film. "Parasite" is a dark comedy-thriller that explores the theme of class conflict in modern-day South Korea. The film follows the impoverished Kim family as they infiltrate the wealthy Park family's home and try to improve their own lives by exploiting the situation. But as their plan unfolds, things start to spiral out of control, leading to unexpected and shocking consequences. It's a brilliantly crafted film with outstanding performances, a captivating storyline, and thought-provoking themes. It's definitely worth watching!
자세한 내용 보기

Kafka consumer in a Spring Boot application using a scheduled task

Here’s how to develop a Kafka consumer in a Spring Boot application using a scheduled task for consuming messages. This approach uses @Scheduled to poll Kafka messages at fixed intervals. Step 1: Add Maven Dependencies Add the required dependencies for Spring Kafka and Scheduling in your pom.xml : < dependency > < groupId > org.springframework.kafka </ groupId > < artifactId > spring-kafka </ artifactId > </ dependency > < dependency > < groupId > org.springframework.boot </ groupId > < artifactId > spring-boot-starter </ artifactId > </ dependency > Step 2: Configure Kafka Consumer in application.yml spring: kafka: bootstrap-servers: localhost: 9092 consumer: group-id: schedule-consumer-group key-deserializer: org .apache .kafka .common .serialization .StringDeserializer value-deserializer: org .apache .kafka .common .serialization .StringDeserialize...
자세한 내용 보기

To integrate Spring Boot with your gRPC service using Gradle, Java, and Avro

To integrate Spring Boot with your gRPC service using Gradle , Java , and Avro , we’ll modify the setup to include Spring Boot dependencies. The goal is to run the gRPC service inside a Spring Boot application, making it easier to manage configurations, beans, and dependencies. Below is a complete guide to adding Spring Boot to your existing gRPC + Avro project. Updated Project Structure grpc-avro-user-service/ │ ├── src/ │ ├── main/ │ │ ├── avro/ │ │ │ └── user .avsc │ │ ├── java/ │ │ │ ├── proto/ │ │ │ │ └── UserService .proto │ │ │ └── com/example/userservice/ │ │ │ ├── UserServiceImpl .java │ │ │ ├── GrpcServerConfig .java │ │ │ └── Application .java ├── build .gradle .kts └── settings .gradle .kts Step 1: Update build.gradle.kts to Add Spring Boot Dependencies Here’s the updated build.gradle.kts to include Spring Boot along with gRPC and Avro : plugins { java id( "com.google.protob...
자세한 내용 보기

kafka polling vs listen

In Kafka, polling and listening are two different approaches to consuming messages, and they correspond to different APIs or frameworks. 1. Kafka Polling Polling refers to the use of Kafka's native poll() method in the Kafka Consumer API . It requires the consumer application to actively request messages from Kafka. Key Characteristics: Active Consumption : The application explicitly calls poll() in a loop to fetch messages. Manual Commit : Developers can choose when to commit offsets, giving fine-grained control over processing and checkpointing. Use Case : Recommended for applications where control over message consumption, offset management, or threading is critical. Example (Java): Properties props = new Properties(); props . put (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092" ); props . put (ConsumerConfig.GROUP_ID_CONFIG, "group1" ); props . put (ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest" ); props . ...
자세한 내용 보기