Active Learning Action Set

This section contains PROC CAS code.

Note: Input data must be accessible in your CAS session, either as a CAS table or as a transient-scope table. A CAS table has a two-level name: the first level is your CAS engine libref, and the second level is the table name. You refer to this table in the CAS procedure by specifying only the second level. For more information about two-level names, see Chapter 2, Shared Concepts (SAS Viya: Machine Learning Procedures). A transient-scope table is called directly from the action and exists in memory for the duration of the action. For more information about accessing data, see SAS Viya: System Programming Guide. For more information about PROC CAS and programming in CASL, see SAS Cloud Analytic Services: CASL Programmer’s Guide and SAS Cloud Analytic Services: CASL Reference.

This example analyzes annotated comments in Wikipedia from the Wikipedia Detox project (https://meta.wikimedia.org/wiki/Research:Detox). The data set contains 50 input, one ID variable, and one target that labels whether the comment is a personal attack or not.

Download the wikidetox data set to the Work library from the GitHub repository: https://github.com/sassoftware/sas-viya-machine-learning/tree/master/data/wikidetox. You can then load the Work.wikidetox data set into your CAS session by naming your CAS engine libref in the first statement of the following DATA step. These statements assume that your CAS engine libref is named mycas, but you can substitute any appropriately defined CAS engine libref.

data mycas.wikidetox;
    set work.wikidetox;
run;

This example uses the iterate action from the activeLearn action set to iterate the active learning process. For each iteration, the uncertaintySampling action selects the most informative set of instances from the training data set and obtains labels from the ground truth data set, which contains all true labels. The purpose of executing the iteration is to assess the performance of active learning: whether it can improve the accuracy of predictive modeling in a reasonable number of iterations. You can compare the final results with the results of a passive supervised learning process that uses the same amount of labeled data randomly drawn from the ground truth data set.

The following DATA step creates the ID variable alucIndex, splits the wikidetox data set into four data tables, and loads the data tables into your CAS session. The wikitest data table holds out 10% of the instances from wikidetox for assessing model performance; the wikigroundTruth data table contains all the instances and their labels except the data for testing; the wikilabeled data table contains 0.1% of the instances from wikidetox and their labels; and the wikiunlabeled data table contains the remaining instances from wikilabeled but without labels.

proc partition data=mycas.wikidetox samppct=10 samppct2=0.1 PARTIND seed=12345;
output out=mycas.partitioned;
run;

data mycas.wikitest;
   set mycas.partitioned;
   where _PartInd_ = 1;
   drop _PartInd_ ;
run;

data mycas.wikigroundTruth;
set mycas.partitioned;
   where _PartInd_ ^= 1;
   drop _PartInd_ ;
run;

data mycas.wikilabeled;
set mycas.partitioned;
   where _PartInd_ = 2;
   drop _PartInd_ ;
run;

data mycas.wikiunlabeled;
   set mycas.partitioned;
   where _PartInd_ = 0;
   flagged = '';
   drop _PartInd_ ;
run;

These statements assume that your CAS engine libref is named mycas, but you can substitute any appropriately defined CAS engine libref.

The following code loads the iterate action from the activeLearn action set to perform active learning iteratively based on the output from the Bayesian network model. Active learning starts with 115 labeled samples and the remaining 103,721 unlabeled samples. At each iteration, active learning selects the most important 25 unlabeled samples according to the entropy values and obtains labels from the wikigroundTruth data set. In this case, the active learning process stops after 100 iterations.

proc cas;
   modelProgram = "action bayesianNetClassifier.bnet status=s result=r,
      table={name=trainData.name, caslib=trainData.caslib},
      inputs=${V2-V51},
      target='flagged',
      id={'rev_id'},
      structure = {'NAIVE'},
      numBin=20,
      maxParents=2,
      preScreening='ZERO',
      savestate={name=model.name, caslib=model.caslib, replace=TRUE}; ";

   action activeLearn.iterate /
      table={name='wikiunlabeled', caslib=''}
      annotatedTable={name='wikilabeled', caslib=''}
      groundTruthTable={name='wikigroundTruth', caslib=''}
      testTable={name='wikitest', caslib=''}
      id = 'rev_id'
      target = 'flagged'
      inputs = ${V2-V51}
      nominals = ${flagged}
      event = 'TRUE'
      modelProgram = modelProgram
      selectQuery = {method="uncertainty", probVar='P_flaggedTRUE'}
      topK = 25
      nIterations = 100
      outIterationHistory={name='iter_history', caslib=''}
      outQueryHistory={name='query_history', caslib=''};
   run;
quit;

The table parameter names the input data table to be analyzed. The modelProgram parameter specifies the predictive model to estimate the probability of the event of the target variable. The outIterationHistory parameter specifies the output table to store all the model assessment measures across iterations; and the outQueryHistory specifies the output table to store all the queries across iterations.

The following code trains the Bayesian network model by using the wikigroundTruth data set, which contains 103,836 labeled samples, and then scores by using the wikitest data table. The predictions are then used to assess the performance of the Bayesian network model by using all 103,836 instances and labels. This code produces the graph shown in Figure 1.

proc cas;
   action bayesianNetClassifier.bnet status=s result=r/
      table={name='wikigroundTruth'},
      inputs=${V2-V51},
      target='flagged',
      id={'rev_id'},
      structure = {'NAIVE'},
      numBin=20,
      maxParents=2,
      preScreening='ZERO',
      saveState={name='BNetOutModel', replace=TRUE};
   run;
   aStore.score result=r status=s/
      table = {name='wikitest'}
      copyVars = ${V2-V51 flagged}
      out = {name='predOut', replace=TRUE}
      rstore = {name='BNetOutModel'};
      end;
   run;
   action percentile.assess result=r status=s/
      table = {name='predout'}
      inputs = 'P_flaggedTRUE'
      response = 'flagged'
      event = 'TRUE'
      pEvent = 'FALSE'
      pVar = 'P_flaggedFALSE'
      cutStep = 0.1
      includeLift = false
      fitStatOut = {name='fitStat', replace=TRUE}
      rocOut = {name='rocTest', replace=TRUE};
   end;
quit;

data mycas.roctest;
   set mycas.roctest;
   call symput('refRUC', _C_);
run;

proc sgplot data = mycas.iter_history;
   series x = trainSize y = testAuc;
   refline &refRUC/label='BNet(103,836 labels)';
   title 'AUC over 100 Iterations';
run;

Figure 1: AUC over 100 Iterations

Figure 1 shows the learning curve of active learning. It starts with only a few labels and so has a low receiver operating characteristic (ROC) value. However, the curve rises quickly after the initial several iterations with only a few more labels added: 25 more labels for each iteration. After 50 iterations with 1,365 labels, active learning already exceeds the performance of the Bayesian network model training on all 103,836 labels.