Machine Learning Tools Action Set
Cross Validation of a Forest Model
This section contains PROC CAS code.
Note: Input data must be accessible in your CAS session, either as a CAS table or as a transient-scope table. A CAS table has a two-level name: the first level is your CAS engine libref, and the second level is the table name. You refer to this table in the CAS procedure by specifying only the second level. For more information about two-level names, see Chapter 2, Shared Concepts (SAS Visual Data Mining and Machine Learning: Procedures). A transient-scope table is called directly from the action and exists in memory for the duration of the action. For more information about accessing data, see SAS Viya: System Programming Guide. For more information about PROC CAS and programming in CASL, see SAS Cloud Analytic Services: CASL Programmer’s Guide and SAS Cloud Analytic Services: CASL Reference.
This example shows how to use the crossValidate action to perform k-fold cross validation of a forest model. The following DATA step loads the JunkMail data set from the Sashelp library into a data table named mycas.JunkMail. These statements assume that the CAS engine libref is named mycas, but you can substitute any appropriately defined CAS engine libref.
data mycas.JunkMail;
set sashelp.JunkMail;
run;
The following PROC CAS code uses the crossValidate action to automatically perform k-fold cross validation of a forest model that is trained on the JunkMail data table:
proc cas noqueue;
mltools.crossValidate /
modelType='forest'
kFolds=3
seed=12345
casOut='junkmail_forest_score'
trainOptions={
table={
name='junkmail',
where='Class NE .'
},
inputs={
"Address", "Addresses", "All", "Bracket",
"Business", "CS", "CapAvg", "CapLong",
"CapTotal", "Conference", "Credit", "Data",
"Direct", "Dollar", "Edu", "Email",
"Exclamation", "Font", "Free", "George",
"HP", "HPL", "Internet", "Lab",
"Labs", "Mail", "Make", "Meeting",
"Money", "Order", "Original", "Our",
"Over", "PM", "Paren", "Parts",
"People", "Pound", "Project", "RE",
"Receive", "Remove", "Semicolon", "Table",
"Technology", "Telnet", "Will", "You",
"Your", "_000", "_85", "_415",
"_650", "_857", "_1999", "_3D"
},
target='Class',
nominals={
{name='Class'}
},
casOut={
name='forest_junkmail_model', replace=true
}
}
;
run;
quit;
The "Cross Validation Fit Statistics" table in Output 21.1.1 displays the fit statistics for each fold and the fit statistic values averaged across all folds.
Output 21.1.1: Cross Validation Fit Statistics Output
| Cross-Validation Fit Statistics | ||||||
|---|---|---|---|---|---|---|
| Fold | Number of Observations | Squared Error | Mean Consequential Error | Multiclass Log Loss | ||
| Divisor of Average | Average | Root Average | ||||
| Fold 1 | 1535 | 1535 | 0.079870 | 0.282614 | 0.081433 | 0.299239 |
| Fold 2 | 1533 | 1533 | 0.079762 | 0.282421 | 0.086106 | 0.296797 |
| Fold 3 | 1533 | 1533 | 0.080818 | 0.284285 | 0.092629 | 0.298521 |
| Average | 1533.7 | 1533.667 | 0.080150 | 0.283107 | 0.086723 | 0.298186 |
Cross Validation of a Forest Model
This section contains Lua code for the analysis in the CASL version of this example, which contains details about the results.
Note: In order to run this code, the data that are described in the CASL version need to be accessible to the CAS server. One way to do this is to convert the junkmail data to the comma-separated-value (CSV) file junkmail.csv and then use the following code to load the CSV file into CAS:
s:loadtable{casLib="casuser", path="junkmail.csv"}
For more information about coding in Lua, see Getting Started with SAS Viya for Lua and SAS Viya: System Programming Guide.
The following Lua code uses the crossValidate action to automatically perform k-fold cross validation of a forest model that is trained on the JunkMail data table:
result = s:mltools_crossValidate{
modelType='forest',
kFolds=3,
seed=12345,
casOut='junkmail_forest_score',
trainOptions={
table={
name='junkmail',
where='Class NE .'
},
inputs={
'Address', 'Addresses', 'All', 'Bracket',
'Business', 'CS', 'CapAvg', 'CapLong',
'CapTotal', 'Conference', 'Credit', 'Data',
'Direct', 'Dollar', 'Edu', 'Email',
'Exclamation', 'Font', 'Free', 'George',
'HP', 'HPL', 'Internet', 'Lab',
'Labs', 'Mail', 'Make', 'Meeting',
'Money', 'Order', 'Original', 'Our',
'Over', 'PM', 'Paren', 'Parts',
'People', 'Pound', 'Project', 'RE',
'Receive', 'Remove', 'Semicolon', 'Table',
'Technology', 'Telnet', 'Will', 'You',
'Your', '_000', '_85', '_415',
'_650', '_857', '_1999', '_3D'
},
target='Class',
nominals={
{name='Class'}
},
casOut={
name='forest_junkmail_model', replace=true
}
}
}
The following command displays the results table that is produced by this action call:
For details about the results of this analysis, see the PROC CAS version of this example.
Cross Validation of a Forest Model
This section contains Python code for the analysis in the CASL version of this example, which contains details about the results.
Note: In order to run this code, the data that are described in the CASL version need to be accessible to the CAS server. One way to do this is to convert the junkmail data to the comma-separated-value (CSV) file junkmail.csv and then use the following code to load the CSV file into CAS:
s.upload_file('junkmail.csv')
For more information about coding in Python, see Getting Started with SAS Viya for Python and SAS Viya: System Programming Guide.
The following Python code uses the crossValidate action to automatically perform k-fold cross validation of a forest model that is trained on the JunkMail data table:
result = s.mltools.crossValidate(
modelType='forest',
kFolds=3,
seed=12345,
casOut='junkmail_forest_score',
trainOptions={
"table":{
"name":"junkmail",
"where":"Class NE ."
},
"inputs":{
"Address", "Addresses", "All", "Bracket",
"Business", "CS", "CapAvg", "CapLong",
"CapTotal", "Conference", "Credit", "Data",
"Direct", "Dollar", "Edu", "Email",
"Exclamation", "Font", "Free", "George",
"HP", "HPL", "Internet", "Lab",
"Labs", "Mail", "Make", "Meeting",
"Money", "Order", "Original", "Our",
"Over", "PM", "Paren", "Parts",
"People", "Pound", "Project", "RE",
"Receive", "Remove", "Semicolon", "Table",
"Technology", "Telnet", "Will", "You",
"Your", "_000", "_85", "_415",
"_650", "_857", "_1999", "_3D"
},
"target":"Class",
"nominals":{
"Class"
},
"casOut":{"name":"forest_junkmail_model", "replace":True}
}
)
The following command displays the results table that is produced by this action call:
For details about the results of this analysis, see the PROC CAS version of this example.
Cross Validation of a Forest Model
This example is not available for the R programming language.