Problem 42485. Eliminate Outliers Using Interquartile Range

Created by Monica Roberts

Solve Later

{"relationships":[{"relationshipType":"http://schemas.mathworks.com/matlab/code/2013/relationships/document","targetMode":"","relationshipId":"rId1","target":"/matlab/document.xml"},{"relationshipType":"http://schemas.mathworks.com/matlab/code/2013/relationships/output","targetMode":"","relationshipId":"rId2","target":"/matlab/output.xml"}],"parts":[{"partUri":"/matlab/document.xml","relationship":[],"contentType":"application/vnd.mathworks.matlab.code.document+xml","content":"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<w:document xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\"><w:body><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Given a vector with your \"data\" find the outliers and remove them.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>To determine whether data contains an outlier:</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"ListParagraph\"/><w:numPr><w:numId w:val=\"2\"/></w:numPr></w:pPr><w:r><w:t>Identify the point furthest from the mean of the data.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"ListParagraph\"/><w:numPr><w:numId w:val=\"2\"/></w:numPr></w:pPr><w:r><w:t>Determine whether that point is further than 1.5*IQR away from the mean.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"ListParagraph\"/><w:numPr><w:numId w:val=\"2\"/></w:numPr></w:pPr><w:r><w:t>If so, that point is an outlier and should be eliminated from the data resulting in a new set of data.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"ListParagraph\"/><w:numPr><w:numId w:val=\"2\"/></w:numPr></w:pPr><w:r><w:t>Repeat steps to determine if new data set contains an outlier until dataset no longer contains outlier.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>IQR: Interquartile Range is the range between the median of the upper half and the median of the lower half of data:</w:t></w:r><w:r><w:t> </w:t></w:r><w:hyperlink w:docLocation=\"http://www.wikihow.com/Find-the-IQR\"><w:r><w:t>http://www.wikihow.com/Find-the-IQR</w:t></w:r></w:hyperlink></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>To find an outlier by hand:</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Data: [ 53 55 51 50 60 52 ] we will check for outliers.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Sorted: [ 50</w:t></w:r><w:r><w:t> </w:t></w:r><w:r><w:rPr><w:b/></w:rPr><w:t>51</w:t></w:r><w:r><w:t> 52 53</w:t></w:r><w:r><w:t> </w:t></w:r><w:r><w:rPr><w:b/></w:rPr><w:t>55</w:t></w:r><w:r><w:t> 60 ] where the mean is 53.5 and 60 is the furthest away (60-53.5 &gt; 53.5-50).</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>1.5 * IQR = 1.5 * (55-51) = 6</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Since 60-53.5 = 6.5 &gt; 6, 60 is an outlier.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>New Data: [ 53 55 51 50 52 ] we will check for outliers.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>New Data Sorted: [</w:t></w:r><w:r><w:t> </w:t></w:r><w:r><w:rPr><w:b/></w:rPr><w:t>50 51</w:t></w:r><w:r><w:t> 52</w:t></w:r><w:r><w:t> </w:t></w:r><w:r><w:rPr><w:b/></w:rPr><w:t>53 55</w:t></w:r><w:r><w:t> ] where the mean is 52.2 and 55 is the furthest away.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>1.5* IQR = 1.5 * (54-50.5) = 4.5</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Since 55-52.2 = 2.8 &lt; 4.5, 55 is NOT an outlier.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Our original data had one outlier, which was 60.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Example:</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"code\"/></w:pPr><w:r><w:t><![CDATA[Input data = [53 55 51 50 60 52]\n\nOutput new_data = [53 55 51 50 52]]]></w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>since 60 is an outlier, it is removed</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:rPr><w:b/></w:rPr><w:t>*Note: A number may be repeated within a dataset that is an outlier. You should not remove all instances, but remove only the first instance and check the new dataset to determine whether this number is still an outlier (see 5th test suite).</w:t></w:r><w:r><w:t>*</w:t></w:r></w:p></w:body></w:document>"},{"partUri":"/matlab/output.xml","contentType":"text/xml","content":"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\" ?><embeddedOutputs><metaData><evaluationState>manual</evaluationState><layoutState>code</layoutState><outputStatus>ready</outputStatus></metaData><outputArray type=\"array\"/><regionArray type=\"array\"/></embeddedOutputs>"}]}

Given a vector with your "data" find the outliers and remove them.To determine whether data contains an outlier:<ol><li>Identify the point furthest from the mean of the data.</li><li>Determine whether that point is further than 1.5*IQR away from the mean.</li><li>If so, that point is an outlier and should be eliminated from the data resulting in a new set of data.</li><li>Repeat steps to determine if new data set contains an outlier until dataset no longer contains outlier.</li></ol>IQR: Interquartile Range is the range between the median of the upper half and the median of the lower half of data: <a href = "http://www.wikihow.com/Find-the-IQR">http://www.wikihow.com/Find-the-IQR</a>To find an outlier by hand:Data: [ 53 55 51 50 60 52 ] we will check for outliers.Sorted: [ 50 51 52 53 55 60 ] where the mean is 53.5 and 60 is the furthest away (60-53.5 &gt; 53.5-50).1.5 * IQR = 1.5 * (55-51) = 6Since 60-53.5 = 6.5 &gt; 6, 60 is an outlier.New Data: [ 53 55 51 50 52 ] we will check for outliers.New Data Sorted: [ 50 51 52 53 55 ] where the mean is 52.2 and 55 is the furthest away.1.5* IQR = 1.5 * (54-50.5) = 4.5Since 55-52.2 = 2.8 &lt; 4.5, 55 is NOT an outlier.Our original data had one outlier, which was 60.Example:<pre class="language-matlab">Input data = [53 55 51 50 60 52]
</pre><pre class="language-matlab">Output new_data = [53 55 51 50 52]
</pre>since 60 is an outlier, it is removed*Note: A number may be repeated within a dataset that is an outlier. You should not remove all instances, but remove only the first instance and check the new dataset to determine whether this number is still an outlier (see 5th test suite).*

Given a vector with your "data" find the outliers and remove them.

To determine whether data contains an outlier:

# Identify the point furthest from the mean of the data.
# Determine whether that point is further than 1.5*IQR away from the mean.
# If so, that point is an outlier and should be eliminated from the data resulting in a new set of data.
# Repeat steps to determine if new data set contains an outlier until dataset no longer contains outlier.

IQR: Interquartile Range is the range between the median of the upper half and the median of the lower half of data: http://www.wikihow.com/Find-the-IQR

To find an outlier by hand:

Data:  [ 53 55 51 50 60 52 ] we will check for outliers.

Sorted: [ 50 *51* 52 53 *55* 60 ]   where the mean is 53.5 and 60 is the furthest away (60-53.5 > 53.5-50).

1.5 * IQR = 1.5 * (55-51) = 6

Since 60-53.5 = 6.5 > 6, 60 is an outlier.

New Data: [ 53 55 51 50 52 ] we will check for outliers.

New Data Sorted: [ *50 51* 52 *53 55* ] where the mean is 52.2 and 55 is the furthest away.

1.5* IQR = 1.5 * (54-50.5) = 4.5

Since 55-52.2 = 2.8 < 4.5, 55 is NOT an outlier.

Our original data had one outlier, which was 60.

Example:

Input data = [53 55 51 50 60 52]
  
  Output new_data = [53 55 51 50 52]

since 60 is an outlier, it is removed

**Note: A number may be repeated within a dataset that is an outlier. You should not remove all instances, but remove only the first instance and check the new dataset to determine whether this number is still an outlier (see 5th test suite).**

Solve

Solution Stats

112 Solutions
24 Solvers

Last Solution submitted on Jun 17, 2023

Last 200 Solutions

Problem Comments

7 Comments

7 Comments

Show 4 older comments

Lessmann on 4 Aug 2015

Hi,
first of all, nice problem.
But there might be a problem with the 5th test case. At one point the 61 is further away from the mean than the 65.

Tim on 4 Aug 2015

I agree: it looks like the two 61's in correct_data should be 65's.

Monica Roberts on 4 Aug 2015

Fixed. Thanks for catching.

goc3 on 4 Aug 2015

58.5 should be 53.5 a few places in the description. Also, I'm getting weird behavior with this problem: I can get my function to pass all the test cases on my local machine, but all test cases are failed on the Cody server no matter what I've tried to far.

Monica Roberts on 4 Aug 2015

Fixed the typos, thanks for noticing. I'm not sure why your code may not be working. I'd suggest checking you aren't using something like iqr which is only in the Statistics (I think?) Toolbox.

goc3 on 4 Aug 2015

That was it. I wish that Cody would give a warning that an unsupported function was being used. Better yet, I wish the Cody computer had all the toolboxes activated.

Thabo Komane on 4 May 2019

But why, in your example< do you say that 1.5 * (54-50.5)= 4.5 ? shouldn't this be 5.25?

Solution Comments

Show comments

Problem Recent Solvers24

Problem Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Problem 42485. Eliminate Outliers Using Interquartile Range

Solution Stats

Last 200 Solutions

Problem Comments

Solution Comments

Problem Recent Solvers24

Suggested Problems

More from this Author1

Problem Tags

Community Treasure Hunt

Problem 42485. Eliminate Outliers Using Interquartile Range

Solution Stats

Last 200 Solutions

Problem Comments

Solution Comments

Problem Recent Solvers24

Suggested Problems

More from this Author1

Problem Tags

Community Treasure Hunt

Players