Language Model Action Set

This example is not available for the Lua programming language.

This section contains Python code. For more information about coding in Python, see Getting Started with SAS Viya for Python and SAS Viya: System Programming Guide.

The following code shows how you can use the textToNgram action to convert text data into n-grams :

                     casInVarList = ["_textVar_"],
                     nGrams       = 3,
                     casOut       = {"name": "ngram_table"})

You need to modify values for the table, casInVarList, nGrams and casOut parameters to fit the in-memory table of text data, the list of available variables, the maximum n-gram size, and the output table, respectively, that you want to use.

An example of a table that contains text data is illustrated in Figure 10.

Figure 10: Sample Table

The table has three rows, which are three sentences: "I LOVE SAS," "I LIKE SAS," and "I WORK IN SAS." The sentences are listed in the _textVar_ column of the table that is specified in the casInVarList parameter.

An example of the output is shown in Figure 11.

Figure 11: Sample Output Table

The table has four columns: "_NGRAM_1_," "_NGRAM_2_," and "_NGRAM_3_," and "_COUNT_." Each "_NGRAM_X_" column contains a token. The maximum n-gram order is defined in the nGrams parameter; here it is three. The "_COUNT_" column shows how many times the n-gram occurs. In massively parallel processing (MPP) mode, the textToNgram action counts the n-grams only locally; it does not merge the redundant n-grams globally. So identical n-grams might exist in the output table. The output table of the textToNgram action is also called the n-gram table. It is used as the input table of the nGramCount action.

This example is not available for the R programming language.