Concurrent programming with Python

Python provides a number of ways through which parallelism or concurrency can be achieved. All of these methods have their own pros and cons, and differ fundamentally in terms of how they are implemented, and a choice needs to be made about which method to use when, keeping the use case in mind.

One of the methods provided by Python for implementing concurrency is performed at the thread level by allowing the application to launch multiple threads, each executing a job. These threads provide an easy-to-use concurrency mechanism and execute inside a single Python interpreter process, and hence are lightweight.

Another mechanism for achieving parallelism is through the use of multiple processes in place of multiple threads. With this approach, every process performs a separate task inside its own separate Python interpreter process. This approach provides some workarounds to the problems that a multithreaded Python program may face in the presence of the Global Interpreter Lock (GIL), which we will discuss in later sections of the chapter, but may also add to the additional overhead of managing multiple processes and increased memory usage.

So, first let's take a look at how we can achieve concurrency with the use of threads and discuss the benefits and drawbacks they come packaged with.