The HPDS2 Procedure

The HPDS2 procedure controls the number of parallel threads used for execution of the DS2 language statements. In contrast to the THREADS PACKAGE DS2 (whose syntax provides single-node scalability as part of the DS2 syntax), PROC HPDS2 provides threading that operates outside the syntax of the language.

By default, the number of copies of the DS2 program that are executed in parallel is determined by the HPDS2 procedure based on the number of CPUs (cores) available on the host machine. The default is to execute one instance of the DS2 program in a dedicated thread per CPU. You can change the default by specifying the NTHREADS= option in the PERFORMANCE statement. For example, if you specify NTHREADS=n, then the HPDS2 procedure runs n copies of the DS2 program in parallel.

An important characteristic of multithreaded applications is that they might produce nondeterministic or unpredictable results. The HPDS2 procedure does not examine whether the DS2 code that is submitted produces meaningful and reproducible results. It simply executes the DS2 code that is provided on each thread. Each instance of the DS2 program operates on a subset of the data. The results that are produced by each unit of work are then gathered, without further aggregation, into the output data set.

Because the DS2 code instances are executed in parallel, consideration must be given to the DS2 language elements that are included in the DS2 code block of the HPDS2 procedure. Not all DS2 language elements can be meaningfully used in multithreaded applications. For example, lagging or retaining of variables can imply ordering of observations. A deterministic order of observations does not exist in this context.