Multi-thread program sample in Python

Here's a template for working with multithreading in Python, designed to be easy to adapt to a variety of use cases. It uses the threading module and includes safe thread synchronization with a queue to manage shared resources.

Multithreading Template

import threading
import queue
import time

# Define the worker function
def worker(task_queue, thread_id):
    """
    Worker function to process tasks from the queue.
    """
    while not task_queue.empty():
        try:
            # Get a task from the queue
            task = task_queue.get_nowait()
            print(f"Thread {thread_id} processing task: {task}")

            # Simulate work
            time.sleep(1)

            print(f"Thread {thread_id} finished task: {task}")
        except queue.Empty:
            # Break if the queue is empty
            break
        finally:
            task_queue.task_done()

# Main function
def main():
    # Number of threads
    num_threads = 4

    # Create a queue to hold tasks
    task_queue = queue.Queue()

    # Add tasks to the queue
    tasks = [f"Task-{i}" for i in range(10)]  # Replace with your tasks
    for task in tasks:
        task_queue.put(task)

    # Create and start threads
    threads = []
    for i in range(num_threads):
        thread = threading.Thread(target=worker, args=(task_queue, i))
        threads.append(thread)
        thread.start()

    # Wait for all threads to finish
    for thread in threads:
        thread.join()

    print("All tasks completed!")

if __name__ == "__main__":
    main()

Explanation

1. Task Queue (queue.Queue):

   • Tasks are stored in a thread-safe queue, which ensures no two threads work on the same task simultaneously.

2. Worker Function:

   • Each thread runs this function. It continuously fetches and processes tasks from the queue until the queue is empty.

3. Thread Creation:

   • threading.Thread is used to create threads. Each thread runs the worker function.

4. Synchronization:

   • task_queue.task_done() ensures proper synchronization between threads and the main thread.

5. Dynamic Task Handling:

   • The queue ensures that tasks can be dynamically added and processed, making it flexible for many applications.

When to Use This

• When tasks are independent and can run concurrently (e.g., processing data, downloading files, etc.).
• When you need to limit the number of threads but want to process many tasks.

Notes

• Global Interpreter Lock (GIL): Python's GIL may limit the performance of CPU-bound tasks. Use the multiprocessing module instead for CPU-intensive work.
• For I/O-bound tasks (e.g., file I/O, network requests), this template works well because threads can run during I/O operations.