Count the number of times a word begins with "co" in a text using Text Analytics Toolbox

Dear community,

I have a pdf with news headlines, and I need to count the number of words each title has and the number of times the words starting with "co" and the word "price" appear in each title. I have not much experience using the Text Analytics Toolbox in Matlab. As far as I can see, "tokenizedDocument" already gives you the total number of words (or tokens) per headline, and "context" counts a specific word. However, I do not know how to ask Matlab to look for words starting with "co". Also, how do I get this information displayed in a table?

I leave my pdf and my code.

I really appreciate any help you can provide!

filename = "Factiva_sample_headlines_1.pdf";
str = extractFileText(filename);
textData = split(str,[newline newline]); %split the text into separate news using split
textData = textData(cellfun(@(s)isempty(regexp(s,'Page')),textData)); %Erase data related to number of page
cleanedDocuments = tokenizedDocument(textData); %Create an array of tokenized documents.

[Stephen23]

Just like any text analytics, the devil is in the detail.

It is worth noting right at the start that PDF format is not intended for data exchange, it actually a language that arranges graphical objects for consistent visual display. How the that data is displayed can be quite different to how it is stored in the file or when extracted.

In this case, I cannot find any one simple pattern that uniquely identifies all of the headlines (blue text in the PDF):

• most headlines are contained on one line, but some are two lines.
• most headlines have two leading newines, but so does the start of the 2nd page (which is not a headline).
• most headlines have a folliwing line of the publisher and date... except for the last headline.
• Most headlines start with alphabetic characters, but some start with quotation marks.

Using a better file format for data exchange would probably make this task easier.

Also note the question is incomplete/badly formed:

• Do you want to match the case or ignore the case?
• Do you want to match only "price" as the entire word, or also as part of a word? E.g. "prices".
• What is a "word"? Is "time-share" one or two words? Is "(JCPOA)" a word? What about "45%" ?

[Angelavtc]

@Stephen All the points you mention are valid. I am just learning how to use the text analytics toolbox. Unfortunately, the documentation and examples I have found are not detailed enough to understand everything from 0. For example, the question of what is a word? It's something I don't know how the tool interprets it. My idea was to apply all the commands matlab proposes to "clean" the data [tokenizedDocument,removeStopWords,normalizeWords,erasePunctuation] and once done, start counting and searching for words. However, the commands proposed here do not apply to data of the type "TokenizedDocument". The headings and dates problem is something I wonder how to solve, because you are absolutely right. Unfortunately the news site only proposes to download the files in pdf and rtf. As far as I understand, rtf does not work for matlab. So I don't know how to solve the problem with pdf. Finally, if you have any document that you can provide me that will help me to understand the tool from scratch, I would appreciate it very much. The examples that matlab has, are not enough. Thank you!

[Stephen23]

@Angelavtc: I tried using EXTRACTFILETEXT but it returned different outputs depending on the OS and/or MATLAB version.

Perhaps the RTF file would be more reliable: could you please upload a sample .RTF file and I will take a look.

[Angelavtc]

@Stephen Thank you very much for your interest. Unfortunately, Matlab does not allow me to load any rtf format file, but here is a link on googledrive where you could download it. Basically, I would like to distinguish between the title, source/date, and news body.

https://drive.google.com/file/d/10OPy9cg5Uq-YjXD0cHZEwG1bHTUaOIh8/view?usp=sharing

Again, thank you very much!

[Angelavtc]

@Stephen Here I also attach the same format but in pdf. As you can see, now I have the news body. Ideally, this is the format in which I am looking to apply textual analysis. As you can see, I would like to group each news by HD, PD and LP. I am not interested in the rest of the field codes. Maybe the codes allow to group the news line by line? Thank you!

[Stephen23]

@Angelavtc: how many of these files do you need to process: 1, 10, 100, 1000, 10000, ... ?

[Angelavtc]

@Stephen About 1000 :/

[Stephen23]

@Angelavtc: what OS are you using? Do you have MS Excel installed ?

[Angelavtc]

@Stephen macOS 10.13.6 version and yes, I do have MS Excel

[Stephen23]

The .RTF file is easy to open using Excel from which it is easy to save as an XLSX. Such an XLSX would then be trivial to import using READTABLE, making the rest of your task faily easy and more robust.

Converting 1000 such files should be possible via VBA in Office or perhaps via ACTXSERVER in MATLAB, also not very complex.

Although .RTF file format contains only ASCII characters, trying to reverse engineer the "markup" that it uses would probably not be a trival task (judged on my quick look now, with my experience processing unusual textfile formats).

[Angelavtc]

Oh la la, it seems more complex than expected :( perhaps I should move to another software 😭. In any case, thank you very much @Stephen!

[Angelavtc]

@Stephen Sorry for the inconvenience again, but I have managed to transform the file to html format (https://drive.google.com/file/d/1Z5bW98_gWohr2appS8zKgxCC1_mlLpzc/view?usp=sharing) Now the problem is that when I use :

filename = "Factiva_1.html";
str = extractFileText(filename);

I only get one article loaded. Any idea how to make matlab read all of them and classify them by title, date and body?

Thank you very much!

Answer 1

[Jonas]

are your searching for something like in this example, applied to your textData?

a={'cotrol', 'alcotro','conect','trial','co'};
cellfun(@(in) strcmp(in(1:2),'co'),a)
ans =

  1×5 logical array

   1   0   1   0   1

you can sum that array to get the total number of words starting with "co"

[Stephen23]

Simpler and much more efficient with STRNCMP/I, e.g.:

C = {'cotrol', 'alcotro','conect','trial','co'};
X = strncmp(C,'co',2)

X = 1×5 logical array
   1   0   1   0   1

[Angelavtc]

@Jonas and @Stephen Thank you for your suggestions, but I am looking more for somthing like this:

str = [
    "an example of a short sentence" 
    "a second short sentence"]

Lets say I am looking for the number of words starting with "se" for each string line. In this case, for the 1st string: "sentence". For the 2nd: "second" and "sentence". For each string line, I want to know the number of times this happens, the specific words starting with "se" and also the the total number of words composing each the line. Is this possible? Thank you again!

[Jonas]

str = [ "se an example of a short sentence" "a second short sentence"]

str=append(" ",str );

numberOfSePerLine=cellfun(@numel,strfind(str, " se"))

for the number of "se"

[Angelavtc]

Thank you @Jonas, what if, for example, I want to find words starting with "se" and "sh" at the same time? How can I modify your code to count for more than one "restrictions"?

[Jonas]

you could add another complicated arrayfun or cellfun instead of the strfind pattern:

cellfun(@(in) strfind(str,in),{" se" " sh"},'UniformOutput',false)

after that i would switch to a for statement for better readability

str = [ "se an example of a short sentence short shark" "a second short sentence" " another thing witout matches" ];
str=append(" ",str ); % if the first word starts with the searched characters, we need to add a space

startingPattern=[" se" " sh"];

% for eafh search term you get one cell containing a cell
whereIsWhat=arrayfun(@(in) strfind(str,in),startingPattern,'UniformOutput',false);

% if you want to store it you can to this eg as matrix, because all results
% have the same number of entries because they search in eual number of
% texts

howOftenIsThisTermnInText=zeros(numel(str),numel(startingPattern)); % search Terms from left to right, results for specific text from top to bottom
for searchTermNumber=1:numel(startingPattern)

    howOftenIsThisTermnInText(:,searchTermNumber)=cellfun(@numel,whereIsWhat{searchTermNumber});

end

[Angelavtc]

@Jonas Than you very much. This community is the best :)

[Walter Roberson]

See also contains() and patternarray

(but I am old fashioned and find regexp easier to work with... but it can get tricky!)

[Angelavtc]

I will @Walter Roberson, thank you!