DOCUMENTATION
Sign In

Get access to My SAS, trials, communities and more.

 

Vocabulary Editor

The Vocabulary Editor enables you to build a vocabulary. When the parsing system needs to categorize a word, it can then easily search a single Vocabulary rather than multiple text files. It is recommended that you build one Vocabulary per parse definition. Before using the Customize Vocabulary Editor, you should define all of your basic categories and implement them in a Grammar, create your parse definitions, and create a text file for each basic category.

Each word in the Vocabulary is defined as belonging to one or more categories, which are defined in an associated Grammar. Each word is also assigned a likelihood, a score indicating a level of confidence (VERY HIGH, HIGH, MEDIUM, LOW, VERY LOW) that a word belongs to a certain category. For example, the next display shows the likelihood associated with each category for the word "Kim," a word in the EN Gender Analysis Vocabulary.

Vocabulary Editor

In the example above, the word "Kim" has two categories: FGW (Female Given Name Word) and MGW (Male Given Name Word). Both of these categories have a likelihood of MEDIUM. In some contexts, both the category and likelihood values are used to make a determination about a word, such as the gender associated with the word. In other contexts, only one of these values might be used to make a determination.

Within the Vocabulary Editor, you must specify which input text sources you want to combine to develop a Vocabulary, and indicate which category's data each library represents. Note that a Vocabulary is stored in a proprietary format. To help ensure Vocabulary integrity, you should not attempt to create or edit a Vocabulary directly, but rather through the Vocabulary Editor.

Building Vocabularies

Create a New Vocabulary File

You can use the Vocabulary Editor to build Vocabularies. It is recommended that one Vocabulary be created for each parse definition. The following steps assume that you have registered a QKB as described in Registering (Adding) a QKB. .

Perform these steps to add a new vocabulary file:

  1. Select Tools > Other QKB Editors > Vocabulary Editor from the main window. The Vocabulary Editor is displayed, and you are prompted to specify a locale.
  2. Specify a locale in the Select Locale(s) and click OK. Typically, this locale would be one with vocabularies that you want to view or maintain in the Vocabulary Editor.
  3. In the Vocabulary Editor, select File > New. You are prompted to specify a locale where the new vocabulary file should be stored.
  4. Specify a locale in the Select Locale(s) and click OK. A new, empty vocabulary file opens in the Vocabulary Editor.

Import Categories from a Grammar

You can import categories and likelihoods so that you can associate those values with the words that you add to your vocabulary. Each word must be associated with one or more categories from the Grammar.

  1. To import categories into the Vocabulary Editor, choose Options > Categories.
  2. Click Import to display the Select Grammar option.
  3. Select the Grammar that you want to associate with your new Vocabulary, and then click OK.
  4. On the Categories screen, the Grammar's standard category abbreviations and descriptions appear. (Derived categories are not imported.)
  5. Use the Delete button to delete any unwanted categories, and then click Close.

Import Words into the Vocabulary

After you import categories, you can import the words to create a vocabulary that fits those categories.

Planning Your Import

Before you import a vocabulary file, review its contents. Make sure that the file meets the criteria described in this section. You can import vocabularies in the following formats:

  • Vocabulary
  • Scheme
  • Text files
  • Delimited text file

For the delimited text file format, the file to be imported must have the following format:

<word><delimiter><category><delimiter><likelihood>

For example, the following is an example of a delimited text file, which uses commas as the delimiter and contains a header row.

Delimited Text File Example

The delimiter character is specified as part of the import action.

If the chosen delimiter character occurs as part of a word, category, or likelihood value, you must encapsulate the word, category, and likelihood values with a text qualifier in the import action. For example, if the delimiter is a space value and the word, category, and likelihood values contain space values, you might use a double-quotation mark to demarcate the values as follows:

“This is the word” “This is the category” “Very High”

If the text identifier occurs naturally within a word or category value, it must be escaped as well. For example:

"This is an escaped ""quotation"" in a word" "Quoted Words category" "Very Low"

The valid likelihood values are: Very Low, Low, Medium, High, and Very High. All other values are considered an error and causes the import to fail.

All spaces before and after the delimiter are removed during the import.

Performing the Import

  1. In the Vocabulary Editor, select to display Import Words. A window similar to the following displays:
    Vocabulary Editor - Import Words

    The following describes the parameters for Import Words.

    Note: Some parameters are enabled or disabled depending on choices made on other parameters.
    • Type - Specify the input file type.
    • Set QKB to Import From - Specify which Quality Knowledge Base to import the Vocabulary or Scheme from if it is not the current QKB.
    • File name - Specify the name of the input file to import.
      TipSelect the proper file extension filter in the file to find the desired input file (that is, *.csv for comma separated files, *.tab for tab delimited files).
    • Encoding - Specify the encoding of the input file.
    • Delimiter - Specify the delimiter used in the delimited text file.
    • Text qualifier - Specify the text qualifier used in the delimited text file.
    • Rows to skip - Specify the number of rows to skip at the beginning of the delimited text file.
    • Filter Word List - Specify a category filter so that only words that have a category, which matches the filter is imported.
    • Use categories from imported vocabulary words - Specify if the categories for a given imported vocabulary word should be added when the word already exists in the vocabulary being imported into.
    • In case of likelihood conflict during merge - If the imported word exists in the vocabulary where it is being imported into, if both words share categories, and the likelihood values differ, you must decide how to resolve the likelihood conflict. If this situation is encountered during the import, use this option to specify whether the existing likelihood should be preserved, the imported word should be used, or a prompt should be displayed to reconcile the conflict.
    • Categories to add - Specify the categories and likelihoods to assign to every word imported. For each category that you select, you can select Overwrite Likelihood. Choosing this option adds your specified likelihood in place of a different value that can already exist for that word and category in the vocabulary being imported into.
  2. Once all inputs have been provided, click Import to import words from the specified file. If you are importing categories, which do not already exist in the vocabulary being imported to, you are prompted to enter a description of each new category being imported. Enter a description for each new category and click OK to continue.
  3. When the import is complete, a status message about the import displays. Click OK to dismiss the status message.
  4. Click Close to close Import Words. The imported words display as shown below.
    Imported Words

Review Categories and Likelihoods

Now that you are looking at the imported words and categories in your new Vocabulary, you can add or delete individual categories, or change likelihood values.

Here is an example of when you might want to update a likelihood value. If your vocabulary contains the name Scott, then that word might have the categories Family Name Word (FNW) and Given Name Word (GNW). You might determine that Scott is more likely to be a Given Name Word than a Family Name Word. You could then increase the likelihood value of the Given Name category for the word Scott.

To change a likelihood value, select the word and click Edit.

Although there can be some adjustments that you want to make to the likelihoods at this point, later testing with the Parse Test Tool reveals other necessary adjustments to give the desired result.

Save the Vocabulary

Now that your Vocabulary is built, you need to save it. Select File > Save. If this is a newly built Vocabulary, the Vocabulary Editor prompts you for a name.

Modifying Vocabularies

Other than altering the likelihood for specific words in a Vocabulary, it is recommended that you do not make many other modifications. However, certain situations can warrant it, so the Vocabulary Editor allows these operations. This can provide a good way to temporarily make changes for testing purposes.

Add a Word to a Vocabulary

  1. On Vocabulary Editor, select File > Open.
  2. Select the Vocabulary to which you want to add a word, and then click Open. The Vocabulary's details appear in the Vocabulary Editor.
  3. Select Edit > Add Word.
  4. Enter your new word, and then click OK. The word now appears selected under Word in the Vocabulary Editor.
  5. On the right side of the screen, add at least one category with a likelihood value.
Note: The Vocabulary Editor alerts you if you attempt to add a word that already exists in the Vocabulary.

Modify a Word in a Vocabulary

  1. In the Vocabulary Editor, select File > Open.
  2. Select the Vocabulary that contains the word that you want to modify, and then click Open. The Vocabulary's details appear in the Vocabulary Editor.
  3. Under Word, select the word or words that you want to modify. The word's categories appear on the right side.
  4. Click Add to display Add Word Category. If you change the likelihood of a category that already belongs to the selected word or words, the Overwrite Category option appears. The Overwrite Category enables you to accept or refuse one or more changed likelihood values.
  5. Select a category and click Edit to change category settings and likelihood values.
  6. Select a category and click Delete to remove that category from the selected word or words.

Delete a Word from a Vocabulary

  1. In the Vocabulary Editor, select File > Open.
  2. Select the Vocabulary that contains the word that you want to delete, and then click Open. The Vocabulary's details appear on the Vocabulary Editor.
  3. Under Word, select the word that you want to delete.
  4. Select Edit > Delete Word.

Navigating the Vocabulary Editor

A vocabulary is a type of QKB file that can be used to categorize words in text data. After a word is categorized by a vocabulary, it can be used in analysis and pattern-matching in subsequent nodes in a QKB definition. For example, the next display shows some of the words listed in the EN Name vocabulary in the English locale in the SAS Quality Knowledge Base.

Vocabulary Editor

In this example, the word Dave is assigned to two categories: Family Name Word (FNW) and Given Name Word (GNW). The FNW category has a likelihood score of Medium and the GNW category has a likelihood score of High. These scores reflect the fact that the word Dave is likely to be a name, and it is more likely to be a given name than a family name.

This information can be used in subsequent nodes in a QKB definition. The next display shows the EN Name vocabulary in the context of the Name parse definition in the English locale.

Vocabulary in a QKB Definition

Vocabularies Tab

When you open a QKB in DataFlux Data Management Studio, the contents of the QKB are displayed in a set of tabs on the right-hand side of the screen. The Vocabularies tab shows a list of vocabularies that are contained in the QKB.

The list of vocabularies in the Vocabularies tab has several columns, as shown in the next display.

Main Columns for the Vocabularies Tab

The Name column shows the name of a vocabulary. The Description column contains text about the vocabulary. The File column contains the physical path to the vocabulary file.

The Locale column shows the QKB locale with which the vocabulary is associated. Note that a vocabulary might be listed even if it is not associated with the locale that you have selected in the Quality Knowledge Bases tree. This is because locales can inherit vocabularies from ancestors.

If you select a locale and look at the values in the Locale column, you see that some vocabularies are associated with the selected locale. Others are associated with the locale’s language, and still others are associated with the Global set of objects. This information is also presented in the icons that are shown for the vocabularies in the list. As shown in the previous display, if an icon has a flag overlay, the vocabulary is associated directly with a locale. If the icon has a word balloon overlay, it means that the vocabulary is associated with a language. An earth overlay means that the vocabulary belongs to the Global set of vocabularies.

If you want to view only the vocabularies that are associated with a language, select that language in the tree for your QKB in the Administration riser.

If you want to view only the vocabularies in the Global set, click on Global in the tree for your QKB.

Find a Vocabulary by Name

To find a vocabulary with a specific name, click the Find icon (binocular icon) in the lower toolbar. This toggles on the Find pane. To toggle off the Find pane, click the Find icon again.

Rename and Duplicate a Vocabulary

You can rename a vocabulary by right-clicking the name and selecting Rename from the pop-up menu. You can also click the name and select Rename from the menu on the lower toolbar.

You can make a copy of a vocabulary by right-clicking the name and selecting Duplicate from the pop-up menu. You can also click the name and select Duplicate from the menu on the lower toolbar.

Create a New Vocabulary

To create a new vocabulary, click the New Vocabulary button on the lower toolbar, or select New Vocabulary from the menu on the lower toolbar. After you have created the new vocabulary, the vocabulary is shown in the vocabulary list, and the Vocabulary Editor appears.

Open a Vocabulary for Editing

To open a vocabulary for editing, double-click on the vocabulary in the list, or select the vocabulary and click Open on the lower toolbar. You might also select the vocabulary and choose Open in the pop-up menu or in the menu on the lower toolbar. See Vocabulary Editor for information about how to edit a vocabulary.

View Definitions That Use a Given Vocabulary

To view a list of QKB definitions that use a given vocabulary, select the vocabulary and choose Usage in the pop-up menu or in the menu on the lower toolbar. A list of the definitions that use the selected vocabulary is displayed.

Delete a Vocabulary

To delete a vocabulary, select the vocabulary and click the delete icon in the lower toolbar, or choose Delete from the pop-up menu or in the menu in the lower toolbar. The vocabulary is removed from the QKB.

Last updated: April 3, 2025