Elasticsearch tokenizer filters

Elasticsearch provides various tokenizer filters that can be used to modify the tokenization process of text during indexing and searching. These filters are applied to individual tokens generated by tokenizers and can transform or filter the tokens based on specific criteria. Here are some commonly used Elasticsearch tokenizer filters:

  1. Lowercase Filter: Converts tokens to lowercase. Useful for case-insensitive searches.

  2. Uppercase Filter: Converts tokens to uppercase.

  3. ASCII Folding Filter: Replaces non-ASCII characters with their ASCII equivalents. For example, "é" becomes "e".

  4. Stop Filter: Removes common words (stop words) from the token stream. Stop words are typically frequently occurring words that add little meaning to the search, such as "a," "an," "the," etc.

  5. Stemmer Filter: Applies stemming algorithms to reduce words to their root form. For example, "running," "runs," and "ran" would all be stemmed to "run."

  6. Synonym Filter: Replaces tokens with their synonyms based on a configured synonym dictionary. Useful for expanding the search scope to include similar terms.

  7. Word Delimiter Filter: Splits tokens into subwords and applies various rules like splitting on camel case, punctuation, or numeric changes. For example, "HelloWorld" can be split into "Hello" and "World."

  8. Length Filter: Filters out tokens based on their length. Tokens shorter or longer than specified thresholds can be excluded.

  9. Pattern Replace Filter: Replaces tokens that match a specified regular expression pattern with a replacement string.

  10. Phonetic Token Filters: Provide phonetic algorithms like Soundex or Metaphone to generate tokens based on their phonetic representation. This allows for approximate matching based on pronunciation.

These are just a few examples, and Elasticsearch provides many more token filters that can be customized and combined to suit specific requirements. Token filters are typically used in conjunction with tokenizers to control the indexing and search behavior of text in Elasticsearch.

댓글

댓글 쓰기

이 블로그의 인기 게시물

Tomcat(톰캣) SSL 포트 설정

https룰 사용하기 위해 톰캣의 server.xml 에 아래의 설정 추가 <Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol" maxThreads="150" SSLEnabled="true" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" URIEncoding="UTF-8" /> .keystore 생성 [/user@hostname]$ $JAVA_HOME/bin/keytool -genkey -alias tomcat -keypass changeit -storepass changeit -keyalg RSA -keystore ${HOME}/.keystore -dname "CN=localhost, OU=OrgUnit, O=MyCompany, C=KR"
자세한 내용 보기

How to split a list into chunks of 100 items in JavaScript, 자바스크립트 리스트 쪼개기

To split a list into chunks of 100 items in JavaScript, you can create a function that loops through the list and slices it into smaller subarrays of size 100. Here’s how you can do it: Example Code: function splitListBy100( array ) { const chunkSize = 100 ; const result = []; for ( let i = 0 ; i < array .length; i += chunkSize) { result .push( array .slice(i, i + chunkSize)); } return result ; } // Example usage: const myArray = [... Array ( 350 ).keys()]; // Creating an array of numbers from 0 to 349 const chunks = splitListBy100(myArray); console.log(chunks); Explanation: chunkSize : Defines the size of each chunk, which in this case is 100. for loop : Iterates over the array, incrementing the index by 100 each time. array.slice(i, i + chunkSize) : Extracts a chunk from the array starting from index i and ending at i + chunkSize . result.push() : Adds the chunk to the result array. The function returns an array of arrays (...
자세한 내용 보기

How to catch exceptions on Future in Java, 자바 Future 에러 잡기

In Java, catching exceptions in a Future object is slightly different from catching exceptions in regular synchronous code. A Future represents the result of an asynchronous computation. If an exception is thrown during the asynchronous task execution, it will be captured and rethrown when you try to retrieve the result using the get() method. Here's how to catch an exception in a Future : Example using Future and ExecutorService : import java.util.concurrent.Callable; import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; public class FutureExceptionHandling { public static void main(String[] args) { ExecutorService executor = Executors.newSingleThreadExecutor(); // Submit a Callable task that throws an exception Future<Integer> future = executor.submit( new Callable<Integer>() { @Override ...
자세한 내용 보기

Using the MinIO API via curl

Using the MinIO API via curl is straightforward, as MinIO is compatible with Amazon S3 API, so most commands follow a similar syntax. Here’s a guide on how to use curl with the MinIO API for some common operations like uploading, downloading, and managing objects. Prerequisites Access Key and Secret Key : Obtain your MinIO Access Key and Secret Key. MinIO Endpoint : Know your MinIO server endpoint, e.g., http://localhost:9000 . Bucket : You may need an existing bucket name, or create a new one using the commands below. Authentication Header For requests to work with MinIO, you need to include authentication in the headers. MinIO uses AWS Signature Version 4 for signing requests. Common Examples 1. List Buckets To list all buckets in your MinIO account, use: curl -X GET \ - -url "http://localhost:9000/" \ - H "Authorization: AWS <AccessKey>:<Signature>" 2. Create a Bucket To create a new bucket, use: curl -X PUT \ - -url "htt...
자세한 내용 보기

운영체제를 개발 방법

운영체제(OS)를 개발하는 것은 매우 복잡하고 기술적으로 도전적인 작업입니다. 이는 컴퓨터 하드웨어와 소프트웨어 간의 상호작용을 관리하는 핵심 소프트웨어이기 때문입니다. 아래는 새로운 운영체제를 개발하는 데 필요한 주요 단계를 정리한 내용입니다. 1. 목적과 요구사항 정의 운영체제의 목적 : 개인용, 서버용, 임베디드 시스템 등 어떤 용도로 사용할지 결정하세요. 핵심 기능 정의 : 파일 시스템, 프로세스 관리, 메모리 관리, 네트워크 지원 등 필수 기능을 정리합니다. 목표 플랫폼 : x86, ARM, RISC-V 등 어떤 하드웨어 아키텍처를 지원할지 결정합니다. 2. 운영체제 설계 커널 타입 선택 : 모놀리식 커널 : 성능은 좋지만 복잡성이 높음. 마이크로커널 : 모듈화가 잘 되어 유지보수가 쉬움. 하이브리드 커널 : 위 두 가지의 장점을 결합. 사용자 인터페이스(UI) : CLI(Command Line Interface)와 GUI 중 어떤 인터페이스를 제공할지 결정. 멀티태스킹 방식 : 선점형(Premptive) 또는 비선점형(Non-preemptive) 멀티태스킹 구현 여부. 3. 개발 환경 설정 개발 언어 선택 : 운영체제 개발에는 일반적으로 C와 C++가 사용됩니다. 부트로더는 종종 어셈블리어로 작성됩니다. 툴체인 준비 : 컴파일러(GCC, Clang 등) 디버거(QEMU, GDB 등) 에뮬레이터(QEMU) 또는 실제 하드웨어. 버전 관리 시스템 : Git 등으로 소스 코드를 관리. 4. 부트로더 개발 부트로더는 컴퓨터가 켜질 때 하드웨어를 초기화하고 운영체제를 메모리에 로드합니다. 대표적인 부트로더 예: GRUB, LILO. 자신만의 부트로더를 제작하거나 기존 부트로더를 커스터마이즈할 수 있습니다. 5. 커널 개발 프로세스 관리 : 프로세스 생성, 스케줄링, 종료 구현. 메모리 관리 : 물리 메모리와 가상 메모리 관리. 페이징 또는 세그멘테이션 지원. 입출력 관...
자세한 내용 보기

dagrun_timeout of Airflow

In Apache Airflow, you can set a timeout for an entire DAG run using the dagrun_timeout parameter in the DAG definition. This parameter allows you to specify the maximum amount of time a DAG run is allowed to execute. If the DAG run exceeds this time limit, it will be marked as failed, and all running tasks in that run will be terminated. Setting dagrun_timeout Here's how to set a timeout for a DAG run: from airflow import DAG from airflow.operators.dummy_operator import DummyOperator from datetime import datetime, timedelta # Define the DAG with DAG( 'my_dag_with_timeout', description='A DAG with a run timeout', schedule_interval='@daily', start_date=datetime(2023, 1 , 1 ), catchup=False, dagrun_timeout=timedelta(hours=2) # Set a 2-hour timeout for the entire DAG run ) as dag: # Define tasks start = DummyOperator( task_id='start') end = DummyOperator( task_id='end') # Define ta...
자세한 내용 보기

What are 5 creative things I could do with my kids' art, 아이와 함께하는 창의적 놀이

Absolutely, preserving your kids' art while also finding creative ways to utilize and display it can be a wonderful solution. Here are five creative ideas to consider: Art Gallery Wall: Designate a section of a wall in your home as an art gallery. Frame selected pieces of your kids' art and arrange them in an organized or eclectic manner. You can change the display periodically to showcase different artworks, creating a rotating gallery. Customized Photo Books: Compile your children's art into a digital photo book or album. Many online services allow you to create customized photo books. Arrange the artwork, add captions, and create a keepsake that you can flip through whenever you want to reminisce. Transform into Functional Items: Turn your kids' artwork into functional items such as placemats, coasters, tote bags, or even custom-made wrapping paper. There are companies that can print your children's art onto these items, making them practical and sentime...
자세한 내용 보기

In HBase, the "memory to disk" flush operation

In HBase, the "memory to disk" flush operation (or "memstore flush") happens when data in the memstore (in-memory storage) is flushed to disk as HFiles in HDFS. This flush can occur based on multiple triggers rather than a strict time interval, including: Memstore Size Limit : The primary trigger is when the size of the memstore reaches a defined threshold. The default maximum memstore size per RegionServer is set by the configuration parameter hbase.regionserver.global.memstore.size , typically as a fraction of the total heap size. When the memstore fills up, it triggers an automatic flush to disk. Time-Based Flush (Optional) : Although time-based flushes are not the main flush trigger, HBase provides an optional parameter for a maximum delay, which can ensure that data gets written to disk within a certain time, even if the memstore size limit hasn't been reached. This parameter is: hbase.hregion.memstore.flush.period : This sets the maximum time (in mil...
자세한 내용 보기

[OS_Linux_Ubuntu] telnet & ssh 설치와 설정 (installation / configuration)

1. telnet 설치 및 설정 sudo apt-get install xinetd sudo apt-get install telnetd /etc/xinetd.conf 파일의 defualt{} 함수는 주석 처리 service telnet {      disable = no      flags = REUSE      socket_type = stream      wait = no      user = root      server = /usr/sbin/in.telnetd      log_on_failure += USERID } sudo /etc/init.d/xinetd restart 2. ssh 설치 및 설정 설치     $ sudo apt-get install ssh 포트 변경     $ sudo vim /etc/ssh/sshd_config     # port 22 에서 '#' 를 제거하고 원하는 포트로 변경     IF >>  X-window 사용하고 싶을 경우,                   /etc/ssh/ssh_config         위의 파일의 다음내용을 변경한다                 # ForwardX11 no -> ForwardX11 yes 로 설정변경) 데몬 재시작     $ sudo /etc/init.d/ssh restart ...
자세한 내용 보기

[Ubuntu] *.deb 파일 설치 방법

확장자가 .deb 인 파일은 원래 Debian 리눅스에서의 설치파일입니다. Ubuntu 는 원래 Debian 리눅스에서 파생되었기 때문에, 많은 부분에서 서로 닮아 있습니다. 따라서 별다른 변환 절차 없이 아래와 같이 바로 설치가 가능합니다. [user@hostname]# sudo dpkg -i *.deb *.rpm 파일을 *.deb 파일로 변환 [user@hostname]$ sudo apt-get install alien [user@hostname]$ sudo alien -c *.rpm
자세한 내용 보기