dagrun_timeout of Airflow

In Apache Airflow, you can set a timeout for an entire DAG run using the dagrun_timeout parameter in the DAG definition. This parameter allows you to specify the maximum amount of time a DAG run is allowed to execute. If the DAG run exceeds this time limit, it will be marked as failed, and all running tasks in that run will be terminated.

Setting dagrun_timeout

Here's how to set a timeout for a DAG run:

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from datetime import datetime, timedelta

# Define the DAG
with DAG(
    'my_dag_with_timeout',
    description='A DAG with a run timeout',
    schedule_interval='@daily',
    start_date=datetime(2023, 1, 1),
    catchup=False,
    dagrun_timeout=timedelta(hours=2)  # Set a 2-hour timeout for the entire DAG run
) as dag:

    # Define tasks
    start = DummyOperator(task_id='start')
    end = DummyOperator(task_id='end')

    # Define task dependencies
    start >> end

Explanation

  • dagrun_timeout: Specifies the maximum time allowed for a DAG run. In the example, dagrun_timeout=timedelta(hours=2) limits the DAG run to 2 hours.
  • Behavior on Timeout: If the DAG run exceeds the timeout, it is marked as failed, and any running tasks will be stopped.

Additional Timeout Options for Tasks

In addition to dagrun_timeout for the entire DAG, you can set timeouts for individual tasks using the execution_timeout parameter:

from airflow.operators.python_operator import PythonOperator

def my_task():
    # Task logic
    pass

# Define a task with an execution timeout of 30 minutes
task = PythonOperator(
    task_id='my_task',
    python_callable=my_task,
    execution_timeout=timedelta(minutes=30),
    dag=dag
)

Summary

  • Use dagrun_timeout for setting a timeout on the entire DAG run.
  • Use execution_timeout for setting a timeout on individual tasks within the DAG.

댓글

댓글 쓰기

이 블로그의 인기 게시물

Consume a topic from kafka by Python

Sure, I can help you develop Python code to consume topics from Apache Kafka using the confluent-kafka library. Make sure you have Kafka installed and running, and you have the confluent-kafka library installed using pip install confluent-kafka . Here's an example of how you can consume messages from a Kafka topic: from confluent_kafka import Consumer, KafkaError # Kafka configuration config = { 'bootstrap.servers' : 'localhost:9092' , # Replace with your Kafka broker addresses 'group.id' : 'my-group' , # Consumer group ID 'auto.offset.reset' : 'earliest' # Start consuming from the beginning of the topic } # Create Kafka consumer instance consumer = Consumer(config) # Subscribe to a topic topic = 'my-topic' # Replace with the topic you want to consume from consumer.subscribe([topic]) # Consume messages try : while True : msg = consumer.poll( 1.0 ) # Poll for new messages with a...
자세한 내용 보기

What are 5 creative things I could do with my kids' art, 아이와 함께하는 창의적 놀이

Absolutely, preserving your kids' art while also finding creative ways to utilize and display it can be a wonderful solution. Here are five creative ideas to consider: Art Gallery Wall: Designate a section of a wall in your home as an art gallery. Frame selected pieces of your kids' art and arrange them in an organized or eclectic manner. You can change the display periodically to showcase different artworks, creating a rotating gallery. Customized Photo Books: Compile your children's art into a digital photo book or album. Many online services allow you to create customized photo books. Arrange the artwork, add captions, and create a keepsake that you can flip through whenever you want to reminisce. Transform into Functional Items: Turn your kids' artwork into functional items such as placemats, coasters, tote bags, or even custom-made wrapping paper. There are companies that can print your children's art onto these items, making them practical and sentime...
자세한 내용 보기

Chromium development gclient

gclient is a command-line tool used primarily in the context of Chromium development. It is part of the depot_tools suite, which is a collection of tools maintained by the Chromium project team to aid in managing large codebases like Chromium. Here are some key aspects of gclient : Source Code Management : gclient is used to manage the source code repositories that make up Chromium and its dependencies. It handles tasks such as cloning repositories, fetching updates, and managing dependencies. Synchronization : It helps synchronize the source code across different repositories and ensures that all dependencies are up to date. This is crucial in a project like Chromium, which consists of multiple interconnected modules and libraries. Usage with fetch : The most common use of gclient is in conjunction with the fetch command. For example, fetch chromium will clone the Chromium repository and all necessary dependencies into a specified directory ( src by default). Configura...
자세한 내용 보기

ftplib.error_perm: 550 Permission denied

The 550 Permission denied error in ftplib occurs when the FTP server denies access to the specified file or directory. This error can be caused by several issues such as: Incorrect file path or directory path on the server. Lack of write permissions for the specified directory. The FTP user does not have the required permissions to upload to the given directory. The remote directory may not exist. Steps to Troubleshoot and Resolve the Issue Here are some steps you can try to fix the 550 Permission denied issue: 1. Check the File Path and Directory on the FTP Server Make sure the remote path you are trying to upload to exists on the FTP server. If you're uploading to a specific directory, you may need to create it first. You can use the following code to list the files and directories on the server to ensure your paths are correct: from ftplib import FTP def list_files (ftp_host, ftp_user, ftp_password) : try : ftp = FTP(ftp_host) ftp.login(...
자세한 내용 보기

리눅스의 부팅과정 (프로세스, 서비스 관리)

보통 사용되지 않는 서비스나 사용 빈도가 적은 프로세스의 경우 stop 시켜두는 것이 시스템 사용 상의 이점이 있다. 자신에게 필요 없는 프로세스를 정지 한다거나 nice 값을 조정하여 실행 시에 프로세스의 우선순위를 변화시킬 수 있다. 이렇게 시스템 상황에 따라 적절히 서비스들을 관리하려면 현재 자신의 시스템의 여러가지 상황을 알아야 하는데, 이런 경우 프로세스 및 실시간 시스템 상황을 보는 명령어인 top 을 사용할 수 있다.  top [-][d delay][q][c][S][s][i][n][b] -d delay : delay 의 시간이 경과하면 현재 스크린을 갱신한다. -q : 이 옵션을 사용하면 스크린을 계속 갱신한다. -c : command list 전체를 보여준다. 즉 욥선을 사용한 것까지 모두 보여준다. -i : idle 상태와 zombie 프로세스는 무시한다.   * 이 외의 옵션은 man 파일을 참고 top 실행 화면 1. First Line :  2:59am up 6days, 10:36,  4 users,  load average    : 현재 시간 2:59 am , 부팅된지 6days and 10시간 36분, 접속해 있는 사용자수, 평균부하 2. Second Line : 61 processes: 56 sleeping, 3 running, 1 xombie, 1 stopped 3. Third Line : CPU states: 36.4% user, 63.5% system, 0.0% nice, 0.0% idle     유저모드에서의 CPU 시간이 36.4%     시스템모드(프로세스를 위해서 커널이 사용한 CPU의 시간)에서의 CPU시간이 63.5%     0.0% nice는 nice로 nice value를 음수로 주어 우선순위를 높이는 경우에 해당하는 모드     0.0% idle(...
자세한 내용 보기

ssh-copy-id in ansible

ssh-copy-id is a command-line tool used to copy your public SSH key to a remote server's authorized_keys file, allowing you to log in to the server without entering a password. Ansible is an open-source automation tool used to manage and configure multiple remote servers simultaneously. In Ansible, you can use the authorized_key module to manage the authorized keys of remote servers. This module allows you to add, remove, or replace SSH keys in a remote user's authorized_keys file. To add your public SSH key to a remote server using Ansible, you can use the authorized_key module. Here's an example playbook that adds your public SSH key to a remote server's authorized_keys file: - name: Add SSH key to authorized_keys hosts: your_server become: true tasks: - name: Install SSH key for current user authorized_key: user: your_username key: " {{ lookup('file', '/path/to/your/public/key/id_rsa.pub') }} " ...
자세한 내용 보기

how to make inactive kafka topic, 카프카 토픽 비활성화

To make a Kafka topic inactive, meaning it no longer receives new messages and its data is eventually deleted, there are several strategies you can follow. Kafka doesn’t have a specific “inactive” state for topics, but you can achieve this effect by adjusting the configuration settings and permissions. Here are some methods to make a Kafka topic "inactive": 1. Restrict Producer Access (Prevent Writing New Messages) One way to make a Kafka topic inactive is to prevent producers from writing new messages to it. You can do this by modifying access control lists (ACLs) or configurations, depending on your Kafka setup. Steps: Modify ACLs to restrict producer access to the topic. If you have security enabled, you can revoke the producer’s permission to write to the topic. bin/kafka - acls . sh - - bootstrap - server localhost:9092 - - remove - - allow - principal User: < producer - user > - - operation Write - - topic < topic - name > This command will ...
자세한 내용 보기

Screenshot of a web page using Puppeteer with a specific browser width, 웹페이지 스크린샷

To get a screenshot of a web page using Puppeteer with a specific browser width, you can follow these steps: Install Puppeteer by running the command npm install puppeteer in your project directory. Require Puppeteer in your Node.js script: const puppeteer = require( 'puppeteer' ); Launch a new browser instance with the desired viewport size: const browser = await puppeteer.launch({ defaultViewport: { width : 1280 , height : 720 , }, }); Create a new page in the browser: const page = await browser. newPage () ; Navigate to the desired URL: await page. goto ( 'https://example.com' ); Take a screenshot of the page: await page .screenshot ({ path : 'screenshot.png' }); Close the browser instance: await browser.close() ; Here's the complete code: const puppeteer = require ( 'puppeteer' ); ( async ( ) => { const browser = await puppeteer.launch({ defaultViewport : { width : 1280 , ...
자세한 내용 보기

운영체제를 개발 방법

운영체제(OS)를 개발하는 것은 매우 복잡하고 기술적으로 도전적인 작업입니다. 이는 컴퓨터 하드웨어와 소프트웨어 간의 상호작용을 관리하는 핵심 소프트웨어이기 때문입니다. 아래는 새로운 운영체제를 개발하는 데 필요한 주요 단계를 정리한 내용입니다. 1. 목적과 요구사항 정의 운영체제의 목적 : 개인용, 서버용, 임베디드 시스템 등 어떤 용도로 사용할지 결정하세요. 핵심 기능 정의 : 파일 시스템, 프로세스 관리, 메모리 관리, 네트워크 지원 등 필수 기능을 정리합니다. 목표 플랫폼 : x86, ARM, RISC-V 등 어떤 하드웨어 아키텍처를 지원할지 결정합니다. 2. 운영체제 설계 커널 타입 선택 : 모놀리식 커널 : 성능은 좋지만 복잡성이 높음. 마이크로커널 : 모듈화가 잘 되어 유지보수가 쉬움. 하이브리드 커널 : 위 두 가지의 장점을 결합. 사용자 인터페이스(UI) : CLI(Command Line Interface)와 GUI 중 어떤 인터페이스를 제공할지 결정. 멀티태스킹 방식 : 선점형(Premptive) 또는 비선점형(Non-preemptive) 멀티태스킹 구현 여부. 3. 개발 환경 설정 개발 언어 선택 : 운영체제 개발에는 일반적으로 C와 C++가 사용됩니다. 부트로더는 종종 어셈블리어로 작성됩니다. 툴체인 준비 : 컴파일러(GCC, Clang 등) 디버거(QEMU, GDB 등) 에뮬레이터(QEMU) 또는 실제 하드웨어. 버전 관리 시스템 : Git 등으로 소스 코드를 관리. 4. 부트로더 개발 부트로더는 컴퓨터가 켜질 때 하드웨어를 초기화하고 운영체제를 메모리에 로드합니다. 대표적인 부트로더 예: GRUB, LILO. 자신만의 부트로더를 제작하거나 기존 부트로더를 커스터마이즈할 수 있습니다. 5. 커널 개발 프로세스 관리 : 프로세스 생성, 스케줄링, 종료 구현. 메모리 관리 : 물리 메모리와 가상 메모리 관리. 페이징 또는 세그멘테이션 지원. 입출력 관...
자세한 내용 보기