Reduce words to their root forms
Stemming refers to a text normalization technique in natural language processing that reduces words to their root forms. Stemming is done primarily by removing affixes of the words, which may result in an invalid dictionary word. For example, upon stemming, the words “requiring,” “required,” and “requirement” will reduce to “require.”
Stemming is commonly used for:
- Information retrieval, where stemmed words are used as synonyms to expand search criteria
- Engineering applications to reduce dimensionality, where stemming results in a fewer words to be tracked and used in a model with machine learning algorithms
The Porter stemmer algorithm is one of the most popular stemming approaches for the English language, and is based on simple heuristic rules. This stemming approach is fast but may not always be accurate. In subsequent years, many other algorithms were proposed, but Porter’s stemming algorithm remains popular due to its speed and simplicity.
A related, but more sophisticated approach, to stemming is lemmatization. Compared to stemming,
- Lemmatization uses vocabulary and morphological analysis and stemming uses simple heuristic rules
- Lemmatization returns dictionary forms of the words, whereas stemming may result in invalid words
The differences between lemmatization and stemming are shown below.
To learn more about stemming and building models with text data, see Text Analytics Toolbox™.