How should I organize my MATLAB files?

I’m not a very sophisticated programmer and I have never really seen a primer on how to organize MATLAB-related files on my personal machine. Obviously this is somewhat a matter of personal preference, but there must be some good practices to follow. Among the questions I’m thinking of are:

- Should I store all my MATLAB code within a single directory and its subdirectories (e.g. to facilitate use of the path browser)?

- Should I avoid putting code in the same folder as non-code files (e.g. figures and movies generated by the code and project-related PDFs)? Should I store the non-code files close to the code or keep them completely separate from all code?

- Should I store MATLAB code close to code from other languages?

- Should I store code high up in the directory structure to make pathnames easier to write?

For context, I'm a Ph.D. student, and I have used MATLAB for research as well as various classes.

[Jan]

@David: +1, a good question! Many scientists do not care about software engineering. But a pile of code and data is not a usable tool and I've seen many "evaluations" performed manually in Excel, wuch that the "results" cannot be reproduced some years later anymore. You ask for a method to keep your tools clean to keep the overview and improve the robustness and reliability of your software. Fine.

[per isakson]

See

• Software Carpentry, Teaching basic lab skills for research computing
• Data Carpentry, [...] fundamental data skills needed to conduct research

[Adam]

We keep all our code in a single parent directory which is our repository root for the version control system, but under that we have a whole tree of folders. Some files can equally well go in multiple places though and I often have a dilemma between project-based folder locations and more specific folder locations. Generally I go for the latter - e.g. I have folders for Signal Processing, Image Processing, Optimisation, Statistics, etc, etc, but some code just sits better in a specific project file as even if it is related to processing an image it is far too specialised to want to go in a generic Image processing folder.

Any mex code goes into the same structure, but in our company we use Matlab for R&D prototyping and C++ or C# or other languages for our commercial code so this sits in a totally different repository.

Data and the like I tend to have strewn around all over the place. Lots of it goes into a subfolder within my Matlab hierarchy that is specified to be ignore by the revision control system and lots of other data sits in a totally different place as I use it outside of Matlab too. I don't generally mix figures and input data into the same folder as .m files except where the data is a .mat file specifically linked to a .m script file, which is usually a temporary thing anyway, or at least one that is not part of our main ship-able code.

Answer 1

[ES]

As you have said, its all a personal preference. For my masters project, I used the following structure. It worked well.

projectFiles - Top level folder
  apiFiles - contains all APIs of the project
  configFiles - contains all configuration files
  logFiles - contains all log files
  utilFiles - contains all utility files (common functions like writing to log, creating directories etc)
  outputFiles - contains all output files
  pathdef_project.m - path definition file for the whole project
  ProjectWrapper.m - main wrapper file - single point use of the full project
  ProjectWrapper.fig - GUI file if needed

Answer 2

[Walter Roberson]

"Should I store all my MATLAB code within a single directory and its subdirectories (e.g. to facilitate use of the path browser)?"

No, you should be developing independent libraries of code that should have their own directories. All of those could go into a common directory, in about the same sense that linux stores all user files somewhere under the user's "home directory".

"Should I avoid putting code in the same folder as non-code files (e.g. figures and movies generated by the code and project-related PDFs)? Should I store the non-code files close to the code or keep them completely separate from all code?"

Imagine that for your code to work, it needed a table of material properties. That table might have a existence logically separate from the program. For example perhaps the purpose of the program is to read in a user specification of a component geometry, and the program had to calculate the thermal expansion of the component.

Now, that table of material properties does not vary with the user geometry specification that is used for any one run. That data table arguably belongs with the code, because it is pretty much just control information that it was easier to save in a different form than to output the specifications into a .m file that did appropriate assignments. Contrawise, the user component specification that is input for a run should probably go in a different place than the code.

Sometimes it make sense to use a different output directory for each program run; sometimes it makes sense put the output for multiple runs in the same location. In particular, a common programming pattern is to have the code look in the directory, find all files of a particular type, and process them one by one, often outputting to a file name related to the original name.

You should be looking ahead to having "users" of your program, who will copy/download the code -- and are only going to want to have one copy of the code (and relevant control files), and the users are going to want to be able to designate a directory to work in that contains files relevant to the run they want to do.

Speaking of having users: you should avoid hard-coding directory names or file names. Either assume the user's current directory is where you need to work, or use uigetdir() or uigetfile() to ask the user which directory or file to work on. Someone who pulls your code out of File Exchange or github should not have to edit more than one configuration file to localize it; better yet is nothing about the code needs to be edited. It is legitimate to use mfilename('fullpath') to figure out where the code has been installed, and to reference files relative to that directory.

"Should I store MATLAB code close to code from other languages?"

Yes. MATLAB is just one of the tools that you will need for any successful project. Except perhaps for academic reasons (to prove it can be done, or to do timing), you should use the most efficient tool for each subsection of your project -- where "most efficient" reflects a mix of time to write and debug the code, and time to execute the code. For example it is a waste of my time to rewrite the Unix "find" facility in MATLAB to look for files with specific properties, and it is especially a waste of my time to rewrite the Unix "find" / "xargs" tool chain. Likewise, writing a language parser in MATLAB is probably not productive compared to using yacc and lex.

Chances are that you will end up with mex source calling out to some C or C++ or fortran code. It would be a waste to artificially separate those out, in my opinion.

"Should I store code high up in the directory structure to make pathnames easier to write?"

No, you can write code that figures out where the directories are and does addpath() as needed. The user should not be writing more than a very small number of paths to code, so their length does not become especially relevant. Longer is generally more error-prone, but the user not having to write any path names to code is even less error prone.

Answer 3

[Rob Purser]

Hi, I wanted to make you aware of a new resource MathWorks has created for MATLAB Toolbox Authors: "Best Practices for MATLAB Toolbox Development."

Check it out now!

This comprehensive guide provides valuable guidelines and recommendations for high-quality and user-friendly MATLAB toolboxes. By following these best practices, you can create toolboxes that enhance productivity and deliver an exceptional user experience. We also have included a complete example (link) that implements all the best practices.

This has been released under a permissive license so that people can redistribute them freely and incorporate them in their own organization’s best practices. If you have comments or suggestions, open an issue or participate in the discussions on GitHub.

Check out our guide today and start building exceptional MATLAB toolboxes!

https://github.com/mathworks/toolboxdesign

Answer 4

[Jan]

• If the project is large (>1000 lines of code), a version control is recommended to support debugging and development.
• Keeping the program and the data in separate folders has the advantage, that the same code can be used to process a set of different data.
• Separating GUI and the code for processing in different files allows to call the program in bacth mode also. Reduce the dependencies between the GUI and the processing to the minimum. This allows to modify the one or the other without interferences.
• I keep the Matlab and C files belonging to a project in the same folder.
• It is useful to separate the functions for a specific project and "tools" in different folders. "Tools" means functions, which can be re-used in other projects also, e.g. statistics, special kind of diagrams or string handling. This allows for writing unit-test functions for all tools, and integration tests for the high-level routines.
• Unit-testing is extremely useful, if you run a project on different computers or Matlab versions. There is no other way to control the quality of teh code. I keep all unit-test function in subfolders of the corresponding routine, such that I can run them automatically by searching in the path for subfolders called "UnitTest_".
• The length of the path names does not matter in my work. The folders of a project code are defined relative to a base folder and included in the path automatically. The base path of the data files is stored in a local text file or persistent variable (see setpref). Then I can move the project to an USB drive or different computer easily.
• For a PhD or published articles the outputs must be reproducible for 10 years. If the tools, high-level project code and data are kept in a clear hierachy in seperate files, you can create a copy on the backup easily. This would also be the case, if you keep the data, GUI and processing code in the same folder, but then the re-using and maintenance of the code is much harder.