Remove duplicate rows in CSV file

hello dear mathworkers,

I have a dataset consist of approximatlly 4 millions records, and i want to remove the duplicated rows or records, can any one help me with the way, i am using matlab 2018a . thanks in advance

[madhan ravi]

Upload a sample file.

[Alex Mcaulley]

You can use unique with 'rows' option

[mohammad Alsajri]

dear madhan ravi this is a sample of the whol data

[Shameer Parmar]

@Mohammad Alsajri: You mentioned that the rows are duplicated, but in you sample sheet, I tried searching for duplicated data, but I could not find duplicated data..

For example, if I filtered the data for value '215' (at column E), still I dont see the duplicated value at other columns like column 'W' and 'X'..

So let us know.. what is mean by duplicate entries and how to consider it..

[mohammad Alsajri]

@Shameer Parmar duplicate means the entire row match another row for all columns , this is just sample it is 4 million records so of cours there is a duplicated rows

[madhan ravi]

Mohammed: Alex's solution should have solved your problem.

[mohammad Alsajri]

thanks for help guys

Answer 1

[Alex Mcaulley]

Since all is numeric data, you can use:

data = xlsread('kdd.xlsx');
datanew = unique(data,'rows');

[Shameer Parmar]

This is not working, because non of data is similar.. I dont find duplicate entries in this sheet provided by Mohammad Alsajri..

using your command, the 'data' and 'datanew' both are getting exact same..

[Alex Mcaulley]

This code works!

I guess the excel provided by Mohammad is just a small portion of the dataset (4 million of rows).