Elasticsearch tokenizer filters

Elasticsearch provides various tokenizer filters that can be used to modify the tokenization process of text during indexing and searching. These filters are applied to individual tokens generated by tokenizers and can transform or filter the tokens based on specific criteria. Here are some commonly used Elasticsearch tokenizer filters:

  1. Lowercase Filter: Converts tokens to lowercase. Useful for case-insensitive searches.

  2. Uppercase Filter: Converts tokens to uppercase.

  3. ASCII Folding Filter: Replaces non-ASCII characters with their ASCII equivalents. For example, "é" becomes "e".

  4. Stop Filter: Removes common words (stop words) from the token stream. Stop words are typically frequently occurring words that add little meaning to the search, such as "a," "an," "the," etc.

  5. Stemmer Filter: Applies stemming algorithms to reduce words to their root form. For example, "running," "runs," and "ran" would all be stemmed to "run."

  6. Synonym Filter: Replaces tokens with their synonyms based on a configured synonym dictionary. Useful for expanding the search scope to include similar terms.

  7. Word Delimiter Filter: Splits tokens into subwords and applies various rules like splitting on camel case, punctuation, or numeric changes. For example, "HelloWorld" can be split into "Hello" and "World."

  8. Length Filter: Filters out tokens based on their length. Tokens shorter or longer than specified thresholds can be excluded.

  9. Pattern Replace Filter: Replaces tokens that match a specified regular expression pattern with a replacement string.

  10. Phonetic Token Filters: Provide phonetic algorithms like Soundex or Metaphone to generate tokens based on their phonetic representation. This allows for approximate matching based on pronunciation.

These are just a few examples, and Elasticsearch provides many more token filters that can be customized and combined to suit specific requirements. Token filters are typically used in conjunction with tokenizers to control the indexing and search behavior of text in Elasticsearch.

댓글

댓글 쓰기

이 블로그의 인기 게시물

Using the MinIO API via curl

Using the MinIO API via curl is straightforward, as MinIO is compatible with Amazon S3 API, so most commands follow a similar syntax. Here’s a guide on how to use curl with the MinIO API for some common operations like uploading, downloading, and managing objects. Prerequisites Access Key and Secret Key : Obtain your MinIO Access Key and Secret Key. MinIO Endpoint : Know your MinIO server endpoint, e.g., http://localhost:9000 . Bucket : You may need an existing bucket name, or create a new one using the commands below. Authentication Header For requests to work with MinIO, you need to include authentication in the headers. MinIO uses AWS Signature Version 4 for signing requests. Common Examples 1. List Buckets To list all buckets in your MinIO account, use: curl -X GET \ - -url "http://localhost:9000/" \ - H "Authorization: AWS <AccessKey>:<Signature>" 2. Create a Bucket To create a new bucket, use: curl -X PUT \ - -url "htt...
자세한 내용 보기

Fundamentals of English Grammar #1

That's a great textbook table of contents! It covers the fundamental tenses in English. Here is a summary of the highlighted sections, with example sentences and key learning tips. PRESENT TIME Item Topic Example Sentence(s) Learning Summary 1-1 The simple present and the present progressive Simple Present: The sun rises in the east. (Fact/Habit) / Present Progressive: She is studying for her exam right now. (Action in progress) Simple Present is for facts, habits, and routines . Present Progressive is for actions happening at the moment of speaking or temporary situations . 1-3 Frequency adverbs I always drink coffee in the morning. / He is never late for class. These adverbs ( always, usually, often, sometimes, rarely, never ) describe how often an action occurs. Their position is key: before the main verb, but after the verb to be or auxiliary verbs. 1-4, 1-5 Final -s / Spelling of final -s/-es He read s a book. / She watch es TV. The final -s (or -es ) is adde...
자세한 내용 보기

리눅스의 부팅과정 (프로세스, 서비스 관리)

보통 사용되지 않는 서비스나 사용 빈도가 적은 프로세스의 경우 stop 시켜두는 것이 시스템 사용 상의 이점이 있다. 자신에게 필요 없는 프로세스를 정지 한다거나 nice 값을 조정하여 실행 시에 프로세스의 우선순위를 변화시킬 수 있다. 이렇게 시스템 상황에 따라 적절히 서비스들을 관리하려면 현재 자신의 시스템의 여러가지 상황을 알아야 하는데, 이런 경우 프로세스 및 실시간 시스템 상황을 보는 명령어인 top 을 사용할 수 있다.  top [-][d delay][q][c][S][s][i][n][b] -d delay : delay 의 시간이 경과하면 현재 스크린을 갱신한다. -q : 이 옵션을 사용하면 스크린을 계속 갱신한다. -c : command list 전체를 보여준다. 즉 욥선을 사용한 것까지 모두 보여준다. -i : idle 상태와 zombie 프로세스는 무시한다.   * 이 외의 옵션은 man 파일을 참고 top 실행 화면 1. First Line :  2:59am up 6days, 10:36,  4 users,  load average    : 현재 시간 2:59 am , 부팅된지 6days and 10시간 36분, 접속해 있는 사용자수, 평균부하 2. Second Line : 61 processes: 56 sleeping, 3 running, 1 xombie, 1 stopped 3. Third Line : CPU states: 36.4% user, 63.5% system, 0.0% nice, 0.0% idle     유저모드에서의 CPU 시간이 36.4%     시스템모드(프로세스를 위해서 커널이 사용한 CPU의 시간)에서의 CPU시간이 63.5%     0.0% nice는 nice로 nice value를 음수로 주어 우선순위를 높이는 경우에 해당하는 모드     0.0% idle(...
자세한 내용 보기

How to checkout branch of remote git, 깃 리모트 브랜치 체크아웃

To check out a branch from a remote Git repository, you can follow these steps: 1. Fetch the remote branches First, ensure that your local repository is aware of the remote branches by using git fetch . This updates your local references to the remote branches. git fetch origin 2. Check out the remote branch Once the remote branches are fetched, you can check out the desired branch using git checkout -b and specifying the remote branch. Command: git checkout -b < local -branch- name > origin/<remote-branch- name > Explanation: <local-branch-name> : The name you want to give to your local branch. origin/<remote-branch-name> : Refers to the branch on the remote called origin (which is the default name for the main remote repository) and the branch you want to check out. Example: If the remote branch is called feature-branch and you want to create a local branch with the same name, you would run: git checkout -b feature -branch origin / feature -...
자세한 내용 보기

In HBase, the "memory to disk" flush operation

In HBase, the "memory to disk" flush operation (or "memstore flush") happens when data in the memstore (in-memory storage) is flushed to disk as HFiles in HDFS. This flush can occur based on multiple triggers rather than a strict time interval, including: Memstore Size Limit : The primary trigger is when the size of the memstore reaches a defined threshold. The default maximum memstore size per RegionServer is set by the configuration parameter hbase.regionserver.global.memstore.size , typically as a fraction of the total heap size. When the memstore fills up, it triggers an automatic flush to disk. Time-Based Flush (Optional) : Although time-based flushes are not the main flush trigger, HBase provides an optional parameter for a maximum delay, which can ensure that data gets written to disk within a certain time, even if the memstore size limit hasn't been reached. This parameter is: hbase.hregion.memstore.flush.period : This sets the maximum time (in mil...
자세한 내용 보기

Chromium 개발 환경 세팅, 크로미움 개발 준비하기

Chromium 개발 환경을 구축하는 것은 여러 단계로 이루어져 있으며, 이를 위해서는 적절한 도구와 의존성을 설치해야 합니다. 아래는 일반적인 Chromium 개발 환경을 설정하는 방법입니다. 1. 시스템 요구사항 확인 Chromium은 리눅스, 맥OS, 윈도우에서 빌드할 수 있습니다. 이 가이드에서는 Ubuntu를 기준으로 설명하지만, 다른 운영체제에서도 유사한 과정으로 설정할 수 있습니다. 메모리 : 최소 16GB RAM (권장 32GB 이상) 디스크 공간 : 최소 100GB 이상의 여유 공간 운영체제 : Ubuntu 20.04 이상 권장 2. 필수 패키지 설치 Chromium을 빌드하려면 여러 패키지가 필요합니다. 아래 명령어를 사용하여 설치합니다. sudo apt- get update sudo apt- get install build-essential git python3 python3-pip sudo apt- get install libx11- dev libxcomposite- dev libxcursor- dev libxdamage- dev libxrandr- dev libxtst- dev libxss- dev libglib2 .0 - dev sudo apt- get install libnss3- dev libasound2- dev libpulse- dev libjpeg- dev libpng- dev sudo apt- get install curl nodejs npm 3. Chromium 소스 코드 가져오기 Chromium 소스 코드를 가져오려면 depot_tools 라는 도구가 필요합니다. depot_tools 설치 : git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git export PATH=` pwd `/depot_tools: " $PATH " 이 경로를 영구적으로 PATH에 추...
자세한 내용 보기

To switch to a specific tag in a Git repository

To switch to a specific tag in a Git repository, you can use the git checkout or git switch command. Here's how you can do it: 1. List Available Tags First, you may want to see a list of all available tags in the repository: git tag 2. Checkout a Specific Tag Once you know the tag you want to switch to, you can use either git checkout or git switch (if you're using a newer version of Git, which recommends git switch for checking out branches). Using git checkout : git checkout <tag- name > Using git switch (recommended for newer versions of Git): git switch -- detach <tag- name > The --detach flag is necessary because tags are not branches; they are just pointers to specific commits. Using --detach makes your HEAD point to the tag without modifying any branch. 3. Verify the Checkout After switching to the tag, verify that you're on the correct commit: git status This will show the tag you're currently on (you'll see somethi...
자세한 내용 보기

urllib3 with proxy settings

To set a proxy using urllib3 , you typically configure it through a ProxyManager , which allows you to route requests through a specific proxy server. Here’s a quick guide: Step 1: Install urllib3 Make sure you have urllib3 installed: pip install urllib3 Step 2: Use ProxyManager Use the ProxyManager class to configure your proxy. Here’s an example: import urllib3 # Define the proxy URL proxy_url = "http://your_proxy_server:port" # Create a ProxyManager instance http = urllib3.ProxyManager(proxy_url) # Send a GET request through the proxy response = http.request( "GET" , "http://example.com" ) # Print the response print(response.data.decode( "utf-8" )) Step 3: Add Authentication (Optional) If your proxy server requires authentication, include it in the proxy URL: proxy_url = "http://username:password@your_proxy_server:port" http = urllib3.ProxyManager(proxy_url) Additional Tips For HTTPS URLs, ProxyManager ...
자세한 내용 보기

To download a file from MinIO using Spring Boot, 스프링부트 Minio 사용하기

To download a file from MinIO using Spring Boot, you can utilize the MinIO Java SDK. Here's an example of how you can achieve this: Add the MinIO dependency to your pom.xml file: < dependency > < groupId > io.minio </ groupId > < artifactId > minio </ artifactId > < version > 7.1.0 </ version > </ dependency > Create a configuration file (e.g., MinioConfig.java ) to establish a connection with your MinIO server: ```java import io.minio.MinioClient; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class MinioConfig { @Value( " ${minio.endpoint} " ) private String endpoint; @Value( " ${minio.accessKey} " ) private String accessKey; @Value( " ${minio.secretKey} " ) private String secretKey; @Bean public MinioClient minioClient() { re...
자세한 내용 보기

CDPEvents in puppeteer

In Puppeteer , CDPEvents refers to events emitted by the Chrome DevTools Protocol (CDP) . Puppeteer leverages CDP to interact with and control a Chromium-based browser. CDP provides detailed, low-level access to browser internals, such as network traffic, console logs, page lifecycle events, and more. You can listen to these CDP events through Puppeteer’s API to monitor or intercept browser activity. How to Listen for CDP Events in Puppeteer Enable the Required CDP Domain: Some events require enabling a particular domain (e.g., 'Network' , 'Page' , 'Runtime' ). Use page._client() to Access the CDP Session: Although it’s a bit lower-level, Puppeteer allows access to DevTools through the page._client() API. Example: Listening for Network Requests This example demonstrates how to intercept and log network requests using CDP. const puppeteer = require( 'puppeteer' ); ( async () => { const browser = await puppeteer.launch({ headless...
자세한 내용 보기