The HMM Procedure

In natural language processing (NLP), practitioners want to learn the structure of sentences and predict the sequences of words. HMMs have been widely applied for characterizing the statistical features of natural languages; for more information, see Jurafsky and Martin (2019). This example runs the model from Cave and Neuwirth (1980) to illustrate how to use the finite HMM to analyze the English language. The data set used in this example was extracted and converted from the classic novel Vanity Fair. To process the raw text, first remove all punctuation and numbers and convert all letters to lowercase.

The following statements create the table for the first 5,000 symbols. The table has two columns: column t is the position of the symbol, and column character is the value at time t.

data text;
   input t character $8. ;
datalines;
1   c
2   h
3   a
4   p
5   t
6   e
7   r
8

   ... more lines ...   

4992    c
4993    k
4994
4995    o
4996    n
4997
4998    h
4999    e
5000    r
;

When you get data set text, you can translate the column value into integers. The following statements relabel the character variables as the unique integers that the finite HMM procedure uses:

data tbltemp;
   set text;
   _temp_count_ = 1;
run;

proc sql;
   create table f0 as
     SELECT tbl.character, /* any other columns to be grouped */
            sum(tbl._temp_count_) as count
     FROM tbltemp tbl
     group by tbl.character
   ;
quit;

data f0Id;
   set f0;
   y = _n_;
   keep character y;
run;

proc print data=f0Id; run;

proc sort data=text out=textSort;
   by character;
run;

PROC SQL;
   CREATE TABLE textLeft AS
   SELECT A.*, B.y
   FROM textSort A LEFT JOIN f0Id B
   ON A.character = B.character
   ORDER BY character;
QUIT;

proc sort data=textLeft out=textLeft;
   by t;
run;

data text2;
   set textLeft;
run;

data mycas.text;;
   set text2;
run;

This converts the novel into a sequence of 26 distinct letters and the word space, for a total of 27 symbols. Then, translate the characters to numbers, a to 2, b to 3, …, z to 27, and word space to 1.

The conversion matching table is shown in Output 18.4.1.

The following statements print the table after value conversion. The table, which is shown in Output 18.4.2, contains three columns: column t is the position of the symbol, column character is the original symbol, and column y is the translated value of the symbol at position t:

proc print data=text2(obs=10) noobs;
run;

Cave and Neuwirth’s model is assumed to have an underlying Markov chain with two hidden states. For each of these hidden states, assume that the 27 symbols appear according to a fixed probability distribution. The following statements estimate a finite HMM in which NSTATE=2 and NCATEGORY=27. There might be a lot of local maximums for the finite HMM; hence, you specify the multistart mode by specifying MULTISTART=1 in the OPTIMIZE statement.

data mycas.text;
   set text;
run;
proc hmm data=mycas.text ;
   id time=t;
   model y / type=finite nstate=2 nCategory=27;
   optimize algorithm=interiorpoint multistart=1;
run;

The number of observations and model information are shown in Output 18.4.3.

The parameter estimates are shown in Output 18.4.4, and the fit statistics are shown in Output 18.4.5.

The most interesting part of this result is the result of the category probability matrix (CPM). In state 1, the probabilities of containing 3, 4, 5, …, 26, which correspond to the consonants, are significant. In state 2, the probabilities of containing 1, 2, 6, 10, 16, 22, which correspond to the vowels a, e, i, o, and u and the space, are significant. Without your having made any assumptions about the two hidden states, the CPM shows that one hidden state contains the vowels and the other hidden state contains the consonants. Interestingly, the space is treated more like a vowel in the context of English. The PROC HMM result shows that the distinction between vowels and consonants is a statistically significant feature inherent in the language. And thanks to HMMs, this could easily be deduced by someone who has no background knowledge of the language.