Want to replace for loop with something faster

Hello Everyone,

I am working on a script where I have to read the information from the mysql database, process it, compare it with another information and print the output.

The events information is stored in mysql database table, which I have to read and process, table contains 19000 entires.

As of now I tried using for loop to read the information from the mysql table but this process take approx 3 to 4 minutes.

Also came across the datastore function in matlab, but unable to store such a huge information there.

Could you please guide and advise me on, how I can reduce the execution time and avoid using for loop?

My table looks like this, this are just 10 entries for example, there are 19000 in total

 id                                                                                          titel                                                                                           titel_en                            url                             type    type_wanted          startdate                 enddate                                               wer                                                              werurl                                                  ort                                                        referent                                                         info                                                                                                      notice                                                                               notice_en    format    comment    uid       name                  mail                                                                                  log                                                                         uniteil    fakultaet    presseurl    jubel    homepage    homepage_en    prio    lang
    __    __________________________________________________________________________________________________________________________________________________________________________________    ________    ____________________________________________________    ____    ___________    _____________________    _____________________    ___________________________________________________________________________    ___________________________________________________    __________________________________________________    _________________________________________________________________    ____________________________________________________    ___________________________________________________________________________________________________________________________________________________________    _________    ______    _______    ___    ___________    ________________________    ____________________________________________________________________________________________________________________________________________    _______    _________    _________    _____    ________    ___________    ____    ____

     1    'Durch Kundenkommunikation Marktführerschaft erreichen'                                                                                                                                  ''       ''                                                       2           0         '1998-06-24 18:30:00'    '1998-06-24 20:00:00'    'MTP'                                                                          ''                                                     'Reichenhainer Straße 70/ B102'                       'Mettler Toledo'                                                     ''                                                      ''                                                                                                                                                                ''          2         ''       ''     'F.Richter'    'fri@hrz'                   ''                                                                                                                                                ''          ''           ''          0         0             0          0      'de'
     2    'DDR-Außenpolitik'                                                                                                                                                                       ''       'http://www.tu-chemnitz.de/phil/politik/veranst.htm'     2           0         '1998-06-24 10:45:00'    '1998-06-24 12:15:00'    'Philosophische Fakultät, Politikwissenschaft, Internationale Politik'         'http://www.tu-chemnitz.de/phil/politik/index.html'    'Uni-Teil 3, Reichenhainer Straße 70, Hörsaal B 3'    'Dr. Benno Siebs, Universität München'                               ''                                                      'Strukturen und Schwerpunkte der DDR-Außenpolitik in der Ära Honecker.'                                                                                           ''          2         ''       ''     'F.R.'         'fri@hrz'                   ''                                                                                                                                                ''          ''           ''          0         0             0          0      'de'
     3    'Vortragsreihe "Medizin im Spannungsfeld zwischen Wissenschaft und Gesellschaft": Persönlichkeit in der Normalität und Abnormalität - Verrücktheit als Spielball des Alltäglichen'       ''       ''                                                       2           0         '1998-06-24 19:30:00'    '1998-06-24 21:00:00'    'Gemeinsames Kolloquium von TU Chemnitz und Klinikum Chemnitz'                 'http://www.tu-chemnitz.de/index.html'                 'Uni-Teil 1, Straße der Nationen 62, Hörsaal 201'     'OA Dr. med. Frank Postrach, Nervenklinik des Klinikums Chemnitz'    'Dipl.-Ing. Mario Steinebach, Tel. [phone]'          ''                                                                                                                                                                ''          2         ''       ''     'F.R.'         'fri@hrz'                   ''                                                                                                                                                ''          ''           ''          0         0             0          0      'de'
     4    'UNIX-Stammtisch in Sachsen: URS - ein erweiterter URL ? '                                                                                                                               ''       'http://www.tu-chemnitz.de/urz/stammtisch/'              5           0         '1998-06-30 17:30:00'    '1998-06-30 21:00:00'    'URZ'                                                                          'http://www.tu-chemnitz.de/urz/'                       'Mensa StraNa'                                        'Bernd-Michael Paschke (bmp.de Informationssysteme Berlin)'          ''                                                      ''                                                                                                                                                                ''          2         ''       ''     'F.R.'         'fri@hrz'                   '26.06.1998 12:07 [e-mail]↵26.06.1998 13:11 [e-mail]↵26.06.1998 13:12 [e-mail]↵'      ''          ''           ''          0         0             0          0      'de'
     5    'Physikalisches Kolloquium: "Symmetrien und Symmetriebruch in den Naturwissenschaften"'                                                                                                  ''       ''                                                       1           0         '1998-07-08 16:30:00'    '1998-07-08 16:30:00'    'Fakultät für Naturwissenschaften, Institut für Physik'                        'http://www.tu-chemnitz.de/physik/'                    'Uni-Teil 3, Reichenhainer Straße 70, Hörsaal 3'      'Prof. Dr. Henning Genz, Universität Karlsruhe'                      'Prof. Dr. Peter Häußler, Tel. [phone]'              ''                                                                                                                                                                ''          2         ''       ''     'F.R.'         'fri@hrz'                   ''                                                                                                                                                ''          ''           ''          0         0             0          0      'de'
     6    'Frauenringvorlesung "Männerkörper - Frauenkörper": "Körperstrategien im Leistungssport" '                                                                                               ''       ''                                                       2           0         '1998-07-01 18:30:00'    '1998-07-01 18:30:00'    ' Philosophische Fakultät, Fachgebiet Germanistik'                             ''                                                     'Uni-Teil 1, Straße der Nationen 62, Raum 375'        'Dr. Gabriele Sobiech, Münster'                                      'Dr. Margarete Hubrath,Tel. [phone]'                 ''                                                                                                                                                                ''          2         ''       ''     'F.R.'         'fri@hrz'                   ''                                                                                                                                                ''          ''           ''          0         0             0          0      'de'
     7    'Vortragsreihe "Rasse und Politik in den Vereinigten Staaten"'                                                                                                                           ''       ''                                                       2           0         '1998-06-30 19:30:00'    '1998-06-30 19:30:00'    'Philosophische Fakultät'                                                      'http://www.tu-chemnitz.de/phil/'                      'Wilhelm-Raabe-Straße 43, Raum 206'                   ''                                                                   ''                                                      'Der US-Historiker Prof. Dr. Clarence Taylor spricht am zum Thema "African-American Religion and Its Political Significance". Der Vortrag ist in Englisch.'       ''          2         ''       ''     'fri'          '[e-mail]'    ''                                                                                                                                                ''          ''           ''          0         0             0          0      'de'
     8    'Collision Processes at ultralow temperatures'                                                                                                                                           ''       'http://www.asu.edu/clas/chemistry/'                     2           0         '1998-06-26 13:00:00'    '1998-06-26 14:30:00'    'Innovationskolleg "Methoden und Materialsysteme für den Nanometerbereich"'    'http://www.tu-chemnitz.de/physik/INK/ink.htm'         'Uni-Teil 3, Reichenhainer Straße 70, Hörsaal 3'      'Prof. M. A. Smith (Dep. of Chemestry, University of Arizona)'       'Prof. Hietschold; [e-mail]'    ''                                                                                                                                                                ''          2         ''       ''     'F.R.'         '[e-mail]'    '26.06.1998 13:41 [e-mail]↵'                                                                                                  ''          ''           ''          0         0             0          0      'de'
     9    'Nanobeben in Halbleitermikrostrukturen '                                                                                                                                                ''       ''                                                       2           0         '1998-07-03 13:00:00'    '1998-07-03 14:30:00'    'Innovationskolleg "Methoden und Materialsysteme für den Nanometerbereich"'    'http://www.tu-chemnitz.de/physik/INK/ink.htm'         'Uni-Teil 3, Reichenhainer Straße 70, Hörsaal 3'      'Prof. A. Wixforth (Universität München, Physik)'                    'Prof. Hietschold; [e-mail]'    ''                                                                                                                                                                ''          2         ''       ''     'fri'          '[e-mail]'    '03.07.1998 08:14 [e-mail]↵'                                                                                                  ''          ''           ''          0         0             0          0      'de'
    10    'Strukturwandel durch Globalisierung'                                                                                                                                                    ''       ''                                                       2           0         '1998-07-07 16:30:00'    '1998-07-07 16:30:00'    ' Elektrotechnisches Institut'                                                 ''                                                     'Uni-Teil 3, Reichenhainer Straße 70, Hörsaal 3'      'Dr. Brian Rampp, SIEMENS / München'                                 ''                                                      ''                                                                                                                                                                ''          2         ''       ''     'fri'          '[e-mail]'    '03.07.1998 08:23 [e-mail]↵'                                                                                                  ''          ''           ''          0         0             0          0      'de'

[per isakson]

"such a huge information" How many GB? How large is the physical memory of your system?

[per isakson]

See Profile to Improve Performance. Find the bottlenecks.

[Raj Tailor]

Hello Per Isakson,

Thank you for your reply.

When I read that databsase in matlab, it shows around 80 Mb.

My system has 256 GB of physical storage.

I will refer the link provided by you.

[per isakson]

I assume the calls to MySQL is the bottle-neck. The solution would be to use SQL to create the entire table and return it to Matlab in one call. Or did I miss how your code works?

[Image Analyst]

80 MB is not that large, even assuming you have only 4 or 8 GB of RAM. I assume your 256 GB is your drive. Is it an SSD or a HD? An SSD would be a lot faster.

The for loop itself is not the problem. I recently did a test for another poster and found out I could do one hundred million iterations in only 0.09 seconds. So I think it's the databse calls or memory swapping or something else that's causing the slowdown, not the for loop itself.