In HBase, the "memory to disk" flush operation

In HBase, the "memory to disk" flush operation (or "memstore flush") happens when data in the memstore (in-memory storage) is flushed to disk as HFiles in HDFS. This flush can occur based on multiple triggers rather than a strict time interval, including:

  1. Memstore Size Limit: The primary trigger is when the size of the memstore reaches a defined threshold. The default maximum memstore size per RegionServer is set by the configuration parameter hbase.regionserver.global.memstore.size, typically as a fraction of the total heap size. When the memstore fills up, it triggers an automatic flush to disk.

  2. Time-Based Flush (Optional): Although time-based flushes are not the main flush trigger, HBase provides an optional parameter for a maximum delay, which can ensure that data gets written to disk within a certain time, even if the memstore size limit hasn't been reached. This parameter is:

    • hbase.hregion.memstore.flush.period: This sets the maximum time (in milliseconds) that data can stay in memory before it must be flushed to disk. By default, this is set to 0 (disabled), meaning there is no strict time limit, and only the memstore size triggers the flush. If enabled, it will enforce a flush after the specified time, regardless of the memstore size.

    For example:

    hbase.hregion.memstore.flush.period=3600000  # Flush after 1 hour

  3. Manual Flush: You can also manually flush data by calling flush() on a table or RegionServer using the HBase shell or API. This is useful in certain situations for performance tuning or maintenance.

Best Practice

Leaving hbase.hregion.memstore.flush.period disabled (the default) generally provides optimal performance, as HBase will flush based on memory thresholds instead of time, which is often more efficient. However, in scenarios where you want to ensure data persistence on disk more frequently (such as during a heavy load with risk of data loss in case of crashes), you can configure this parameter to enforce a periodic flush.

댓글

댓글 쓰기

이 블로그의 인기 게시물

Creating an synchronous HTTP request using Spring RestTemplate

While RestTemplate is now in maintenance mode and WebClient is recommended for new applications, it’s still widely used in existing projects. Here’s how you can configure and use RestTemplate in a Spring Boot application: 1. Add Dependencies Ensure the following dependency is present in your pom.xml : < dependency > < groupId > org.springframework.boot </ groupId > < artifactId > spring-boot-starter-web </ artifactId > </ dependency > 2. Configure a RestTemplate Bean Define a RestTemplate bean in a configuration class to enable dependency injection: import org.springframework.boot.web.client.RestTemplateBuilder; import org.springframework.context. annotation .Bean; import org.springframework.context. annotation .Configuration; import org.springframework.web.client.RestTemplate; @Configuration public class RestTemplateConfig { @Bean public RestTemplate restTemplate(RestTemplateBuilder builder) { ret...
자세한 내용 보기

동남아 국가들이 사용하는 언어

📄 동남아시아 국가별 주요 언어 정리 국가 주요 언어 및 어족 태국 태국어 (Tai-Kadai어족, 타이어파) 베트남 베트남어 (오스트로아시아어족, 몬크메르어파) 미얀마 (버마) 버마어 (티베트-버마어파, 중국티베트어족) 캄보디아 크메르어 (오스트로아시아어족, 몬크메르어파) 라오스 라오어 (Tai-Kadai어족, 타이어파) 말레이시아 말레이어 (오스트로네시아어족, 말레이폴리네시아어파) 인도네시아 인도네시아어 (오스트로네시아어족, 말레이폴리네시아어파) 필리핀 필리핀어(타갈로그어) 및 영어 (오스트로네시아어족) 동티모르 테툼어(오스트로네시아어족), 포르투갈어 (공식어) 브루나이 말레이어 (오스트로네시아어족) 싱가포르 영어(공식어), 말레이어, 중국어, 타밀어 등 다언어 국가 부가 설명 오스트로네시아어족 : 말레이어, 인도네시아어, 타갈로그어 등 주요 섬 지역 언어 대부분 포함 티베트-버마어파 : 미얀마와 인근 산악 지역 언어 타이-카다이어족 : 태국, 라오스 언어 오스트로아시아어족 : 베트남어, 크메르어 등 대륙부 지역 언어
자세한 내용 보기

형태소 분석 태그

때 형태소는 한국어 형태소 분석을 통해 문장에서 각 단어의 품사와 문법적 역할을 표시하는 것을 말합니다. 일반적으로 형태소 태그는 주로 형태소 분석기를 사용하여 추가되며, 다양한 형태소 태그 체계가 있을 수 있습니다. 가장 널리 사용되는 형태소 태그 체계 중 하나는 세종 형태소 품사 태그 체계입니다. 세종 형태소 품사 태그 체계의 일부 중요한 태그는 다음과 같습니다: 명사(NNG): 일반 명사 고유 명사(NNP): 고유한 개별 명사 대명사(NP): 대명사 동사(VV): 동사 형용사(VA): 형용사 부사(MAG): 일반 부사 관형사(MM): 관형사 감탄사(IC): 감탄사 조사(JKS, JKC, JKG 등): 조사 어미(EP, EC, ETN 등): 어미 접두사(XPN): 접두사 접미사(XSN): 접미사 어근(XR): 어근 한글 이외의 문자(SL): 외국어 및 숫자 등 이 외에도 다양한 형태소 태그가 있을 수 있으며, 분석 도구나 분석기에 따라 세부적인 태그 체계가 조금씩 다를 수 있습니다.
자세한 내용 보기

MySQL GRANT

grant all privileges on TABLE_NAME.* to 'username'@'xxx.xxx.xxx.xxx' identified by 'password'; flush privileges; 모든 아이피에서 접근 허용하기 grant all privileges on TABLE_NAME.* to 'username'@'%' identified by 'password'; flush privileges;
자세한 내용 보기

max_active_runs of Airflow

In Apache Airflow, if you want to ensure that a new run of a DAG doesn’t start before the previous one has completed, you can use the max_active_runs parameter in the DAG definition. Setting this parameter to 1 ensures that only one instance of the DAG is running at any given time. Setting max_active_runs for a DAG Here’s an example of how to set up a DAG with max_active_runs=1 : from airflow import DAG from airflow.operators.dummy_operator import DummyOperator from datetime import datetime # Define the DAG with DAG( 'my_dag' , description= 'A sample DAG' , schedule_interval= '@daily' , # Set your schedule start_date=datetime( 2023 , 1 , 1 ), catchup= False , max_active_runs= 1 # This prevents new runs before the previous run completes ) as dag: # Define tasks start = DummyOperator(task_id= 'start' ) end = DummyOperator(task_id= 'end' ) # Define task dependencies ...
자세한 내용 보기

Screenshot of a web page using Puppeteer, 웹페이지 스크린샷

To get a screenshot of a web page using Puppeteer, you can follow these steps: Install Puppeteer by running the command npm install puppeteer in your project directory. Require Puppeteer in your Node.js script: const puppeteer = require( 'puppeteer' ); Launch a new browser instance: const browser = await puppeteer. launch () ; Create a new page in the browser: const page = await browser. newPage () ; Navigate to the desired URL: await page. goto ( 'https://example.com' ); Take a screenshot of the page: await page .screenshot ({ path : 'screenshot.png' }); Close the browser instance: await browser.close() ; Here's the complete code: const puppeteer = require ( 'puppeteer' ); ( async ( ) => { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto( 'https://example.com' ); await page.screenshot({ path : 'screenshot.png' }); await browse...
자세한 내용 보기

how to delete all issues on project in sentry, 센트리 이슈 삭제하기

As of now, Sentry doesn't have a direct feature to delete all issues in bulk through the web interface. However, there are a few methods you can use to archive, resolve, or delete issues programmatically or by adjusting project settings: 1. Bulk Archive Issues from the Web Interface Though there is no mass delete option, you can bulk archive or resolve issues, which essentially hides them from active issue lists. Steps: Go to your Sentry project. In the issue list view, select the issues you want to archive or resolve. Use the "Select All" checkbox to select multiple issues. Choose "Resolve" or "Archive" from the bulk action dropdown. However, this doesn't delete the issues permanently; they will be archived or marked as resolved. 2. Adjust Retention Policy (Data Retention Settings) If you're looking to remove older issues, you can adjust the retention policy at the organization level. Sentry allows you to configure how long issue...
자세한 내용 보기

Using the MinIO API via curl

Using the MinIO API via curl is straightforward, as MinIO is compatible with Amazon S3 API, so most commands follow a similar syntax. Here’s a guide on how to use curl with the MinIO API for some common operations like uploading, downloading, and managing objects. Prerequisites Access Key and Secret Key : Obtain your MinIO Access Key and Secret Key. MinIO Endpoint : Know your MinIO server endpoint, e.g., http://localhost:9000 . Bucket : You may need an existing bucket name, or create a new one using the commands below. Authentication Header For requests to work with MinIO, you need to include authentication in the headers. MinIO uses AWS Signature Version 4 for signing requests. Common Examples 1. List Buckets To list all buckets in your MinIO account, use: curl -X GET \ - -url "http://localhost:9000/" \ - H "Authorization: AWS <AccessKey>:<Signature>" 2. Create a Bucket To create a new bucket, use: curl -X PUT \ - -url "htt...
자세한 내용 보기

Read a file in Java, 자바 파일 읽기

To convert a File object into an InputStream in Java, you can use the FileInputStream class. Here's an example: import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; public class FileToInputStreamExample { public static void main(String[] args) { File file = new File( "path/to/file.txt" ); try { InputStream inputStream = new FileInputStream(file); // Use the inputStream as needed... // Close the inputStream when you're done inputStream.close(); } catch (IOException e) { e.printStackTrace(); } } } In the code above, replace "path/to/file.txt" with the actual path to your file. The FileInputStream class is used to create an input stream from the file. You can then use the inputStream object to read the contents of the file or perform any other operations you need. Finally, rem...
자세한 내용 보기

dagrun_timeout of Airflow

In Apache Airflow, you can set a timeout for an entire DAG run using the dagrun_timeout parameter in the DAG definition. This parameter allows you to specify the maximum amount of time a DAG run is allowed to execute. If the DAG run exceeds this time limit, it will be marked as failed, and all running tasks in that run will be terminated. Setting dagrun_timeout Here's how to set a timeout for a DAG run: from airflow import DAG from airflow.operators.dummy_operator import DummyOperator from datetime import datetime, timedelta # Define the DAG with DAG( 'my_dag_with_timeout', description='A DAG with a run timeout', schedule_interval='@daily', start_date=datetime(2023, 1 , 1 ), catchup=False, dagrun_timeout=timedelta(hours=2) # Set a 2-hour timeout for the entire DAG run ) as dag: # Define tasks start = DummyOperator( task_id='start') end = DummyOperator( task_id='end') # Define ta...
자세한 내용 보기