Parallel Processing¶
Some of the computational tasks performed in scqubits can benefit significantly from parallelization. The scqubits package leverages parallel-processing capabilities provided by the Python Standard Library multiprocessing
module. For better pickling support, scqubits further supports use of pathos
and dill
.
One important consideration for parallelization of tasks like parameter sweeps is the fact that Numpy and Scipy tend to make use of multi-threading internally. (Details of that will depend on how they were built on the machine in question.) This will generally lead to competition between multi-threading on the Numpy/Scipy level and parallelization of map
methods via multiprocessing
or pathos
.
In many cases, best performance is obtained by limiting the number of threads used by Numpy to “a few”. (Precise numbers will be machine dependent and need to be determined on a case by case basis.) Limiting this thread number can be achieved from within a Python script or Jupyter and is accomplished by setting environment variables.
Note
Limiting the number of threads will only be effective if environment variables are set before the first import of Numpy.
Several environment variables can play a role, and which one is needed may again be machine-dependent. We thus simply set them all:
[ ]:
import os
NUM_THREADS = "1"
os.environ["OMP_NUM_THREADS"] = NUM_THREADS
os.environ["OPENBLAS_NUM_THREADS"] = NUM_THREADS
os.environ["MKL_NUM_THREADS"] = NUM_THREADS
os.environ["VECLIB_MAXIMUM_THREADS"] = NUM_THREADS
os.environ["NUMEXPR_NUM_THREADS"] = NUM_THREADS
At this point, Numpy import and import of scqubits can proceed.
[7]:
import numpy as np
import scqubits
from scqubits import HilbertSpace, InteractionTerm, ParameterSweep
Enabling parallel processing¶
Parallel processing is enabled for appropriate scqubits methods and classes by passing the number of cores to be used through the keyword argument num_cpus
. The following classes and class methods support parallelization:
Classes and class methods supporting parallelization¶
Class or class method |
---|
|
|
|
|
|
|
To use parallelization, the keyword argument num_cpus
must be passed, specifying the number of cores to be used as an integer, e.g.
[ ]:
transmon.get_spectrum_vs_paramvals(..., num_cpus=4)
[ ]:
sweep = ParameterSweep(
param_name=param_name,
...,
...,
num_cpus=4
)
Once num_cpus
exceeds the value 1 when passed, scqubits starts a parallel processing pool of the desired number of processes.
Global num_cpus default¶
The global default for num_cpus
is stored in scqubits.settings.NUM_CPUS
. Upon import of scqubits, that constant has the value 1
(no parallelization). To change this default and use a user-defined core number by default (say 6), set
[ ]:
scqubits.settings.NUM_CPUS = 6
multiprocessing vs. pathos¶
scqubits supports parallelization through multiprocessing
as well as pathos
. The latter is the default option and is more robust thanks to the advanced pickling methods enabled through dill
.
To switch from use of pathos
/dill
to multiprocessing
, simply alter the following setting:
[ ]:
scqubits.settings.MULTIPROC = 'multiprocessing'