Data Science Pilot Action Set
The exploreData Action
One of the defining characteristics of modern data sets is high dimensionality along with low signal-to-noise ratio, because of a potentially large number of variables that are irrelevant to the downstream analytics. As a result, variable transformation, which is aimed at improving model performance, is a significant part of the predictive modeling workflow. However, high dimensionality precludes an interactive, variable-by-variable exploration and transformation. To handle this issue of scale (high dimensionality), practitioners consider data-quality issues iteratively and treat variables in one pass. For example, you can identify and treat variables that have a significant missing rate, then identify and treat those that have significant skewness. However, this approach prevents the effective use of the solutions available for most data-quality problems.
You can use the exploreData action to address these challenges. The main logical output from the action is a hierarchical variable grouping in which the groups are characterized by comprehensive and multidimensional statistical metrics. You can use the output from the action and perform similar variable transformations and feature generation on groups of variables that share the same statistical profile.
The exploreData action creates its hierarchical output by applying the explorationPolicy parameter. There are two levels in the hierarchy. The first level groups variables according to whether they are interval, nominal, date, time, or datetime variables. The second group applies a set of statistical metrics to further divide the interval and nominal groups. The date, time, and datetime variables are not part of the second-level grouping.
The statistics that are used for the second-level grouping are as follows:
missing rate: the missing rate of the variables; applicable to both interval and nominal variables
cardinality: the number of distinct levels of nominal variables
entropy: the Gini and Shannon entropy statistics for nominal variables
index of qualitative variation (IQV): based on the mode index of qualitative variation, the ratio of most frequent and least frequent levels, and the ratio of the two most frequent levels
skewness: moment and average quantile skewness for interval variables
kurtosis: moment and average quantile kurtosis for interval variables
outlier: outlier based on interquartile (IQR) and robust IQR method for interval variables
coefficient of variation (CV): moment and robust CV for interval variables
The exploreData action discretizes the values of each of these statistics into low (1), medium (2), and high (3) values on the basis of the thresholds specified in the explorationPolicy parameter. The action uses low-medium and medium-high thresholds for the discretization. You have to specify valid values for both thresholds. Otherwise, the action uses default values for both. The exploreData action refers to the output from the discretization as rated statistics, in contrast to the raw values that contain the actual values. The action outputs a CAS output table that contains a flattened representation of the logical hierarchical structure. Each row of the CAS output table contains a variable’s attributes: the name, the type, and the rated and raw statistics.