removeLongWords

Remove long words from documents or bag-of-words model

Syntax

newDocuments = removeLongWords(documents,len)

newBag = removeLongWords(bag,len)

Description

newDocuments = removeLongWords(documents,len) removes words of length len or greater from documents.

newBag = removeLongWords(bag,len) removes words of length len or greater from the bagOfWords object bag.

Examples

Remove Long Words from Document

Remove the words with seven or greater characters from a document.

document = tokenizedDocument("An example of a short sentence");
newDocument = removeLongWords(document,7)

newDocument = 
  tokenizedDocument:

   4 tokens: An of a short

Remove Long Words from Bag-of-Words Model

Remove the words with seven or greater characters from a bag-of-words model.

documents = tokenizedDocument([ ...
    "an example of a short sentence"
    "a second short sentence"]);
bag = bagOfWords(documents);
newBag = removeLongWords(bag,7)

newBag = 
  bagOfWords with properties:

          Counts: [2x5 double]
      Vocabulary: ["an"    "of"    "a"    "short"    "second"]
        NumWords: 5
    NumDocuments: 2

Input Arguments

documents — Input documents

tokenizedDocument array

Input documents, specified as a tokenizedDocument array.

bag — Input bag-of-words model

bagOfWords object

Input bag-of-words model, specified as a bagOfWords object.

len — Minimum length of words to remove

positive integer

Minimum length of words to remove, specified as a positive integer. The function removes words with len or greater characters.

Output Arguments

newDocuments — Output documents

tokenizedDocument array

Output documents, returned as a tokenizedDocument array.

newBag — Output bag-of-words model

bagOfWords object

Output bag-of-words model, returned as a bagOfWords object.

Version History

Introduced in R2017b