Composable infrastructures are becoming increasingly important in high-performance computing. WHY? Because they treat compute, storage, and network devices as pools of resources that can be provisioned as needed, depending on what different workloads require for optimum performance.
As the number of CPU cores continues to grow, numeric libraries such as NumPy, SciPy, Dask, and Numba continue to exploit multi-threading and provide higher application throughput. But when these libraries are used in a single Python application, un-orchestrated use of threads can lead to significant overhead. The result: inefficient use of system cores.
Enter the Intel® Distribution for Python*, with new modules that let developers access easy and safe composable parallelism and efficient thread scheduling, and deliver fast Python applications fully optimized for today’s multi-core systems.
In this video, software engineer Anton Malakhov addresses the limitations of existing approaches, and introduces a new method for composable parallelism that supports both multi-threading and multi-processing types of parallelism in Python.
Be sure to download the tool. It’s free.