Dependency management & package manager like pip in ML

Hi all,

Matlab is officially still lacking a proper package manager like npm or pip. I do not understand how people structure projects with dependencies. I understand, there are

• ML projects
• ML toolboxes
• several mpm versions

Toolboxes have a version number attached, projects have not. Both can be shared as a package. Projects can reference other projects, but that 1. requires the same folder structure on every developer PC and 2. requires that the correct version is checked out from the VCS (which is GIT in our case). There's no way to define "project abc@v1.2.0" as a dependency. Just folders/prj files.

How do you do complex app developments? Like a frontent GUI app that needs several self-created toolboxes and projects? The only way is to use GIT (submodules in worst case) to do dependency managment, and that's definitely a red flag.

Then, there's Matlab's mpm matlab-dockerfile/MPM.md at main · mathworks-ref-arch/matlab-dockerfile (github.com) but that's meant to be used to install different product versions and addons, and there's a third party mpm mobeets/mpm: Simple Matlab package management inspired by pip (github.com) which seems to be interesting, but there's no direct support from Mathworks.

Then, it seems like there's another mpm built-in in ML 2022b that mentions a Matlab -packages flag which is very interesting:

mpm

Error using mpm
This command is not supported when the packages feature is not enabled. Start MATLAB with the -packages command line switch.

There's no information about that. Starting ML with -packages reveals more:

Which seems to support different versions. But there's no documentation anywhere, and it's a built-in function © 2022.

What does that mean? Will there be a Mathworks' supported package manager? What exactly is a package by the way in ML? So far, packages were a synonym to namespaces, using the +package folders. How does that work together with ML projects?

I'm in the process of re-structuring code due to a migration from Bitbucket to Gitlab and I'm at a point where a proper dependency management must take place, and that must work in CI/CD, too.

Would be very happy to hear your thoughts.

Thanks,

Jan

[Jan Kappen]

Any updates from Mathworks?

I'd really like to know if theres something on its way or if I have to create my own dependency manager, probably in Python.

Thanks

Answer 1

[Michelle Hirsch]

We are working on a package management system for MATLAB. The mpm comand you found in MATLAB was from an early beta test we ran in R2022b to flesh out some of the basic principles of the system we are designing.

I think the best bet would be for us to connect you with an appropriate expert or two at MathWorks to talk through ideas of how to best approach meeting your needs now while setting yourself up to adopt our system when it is ready. I'll email you through your author profile to make the connection.

[Ernst van der Pols]

Thanks Jan for raising this question and Michelle for giving an update.

We at KROHNE have a Nexus server for storage and distribution of software artifacts, and are still in the process of introducing this technology in the various software-creating communities within the company. Mainly .NET (nuget), and a little PowerShell (nuget) and Python (pip) are already there.

Leveraging this for our MATLAB software would mean a great boost for improving the quality and ease of sharing the abundance of MATLAB scripts and models.

I looked into some options to distribute MATLAB Toolbox packages via Nexus. Since a MATLAB Toolbox (.mltbx) is an OPC container, I tested the combination of cloaking an mltbx as NuGet package, which uses also OPC technology. With cloaking I mean building the package such that it can be handled by both MATLAB and NuGet, with at most only a manual fix of the package file name extension (from .nupkg to .mltbx vv.). Alas, this had no satisfactory results.

NuGet is already not only used for .NET packaging, but also for PowerShell packaging. Currently using the Semantic Versioning standard and a rather simpel Software-Bill-Of-Material (nuspec), it is easy to use, and to integrate in a build process.

Next best option would be to wrap the mltbx in a NuGet package, but before starting that up we first focus on the creation of a Toolbox in the first place. Hopefully we can expect some managed versioning solution that we can employ in-house soon.

[Michelle Hirsch]

@David Sampson do you have any advice you could offer Ernst? I'm guessing you've supported others using NuGet.

[Jan Kappen]

@Michelle Hirsch I'd be very interested in that, too.

[Michelle Hirsch]

@Ernst van der Pols in cases it's helpful, here's some input I received from one of MathWorks consultants who has helped a lot of users set up package management tooling. He hasn't worked directly with NuGet with MLTBX, but has heard from some users who have. Here are his observations:

• It’s pretty simple to put an mltbx in a NuGet package, I don’t know of anyone making an mltbx be a NuGet package.
• NuGet packages can have dependencies and pull them in automatically.
• There is no requirement in .NET for one version of any given library, so it makes no effort to resolve dependency ranges.
• NuGet will allow you to search packages and pull the mltbx files, plus dependencies, into a folder, but can’t install them.
• You need to write a bunch of MATLAB code to take the folder of mltbxs and install them (and with dependencies, work out which ones to install)
• And more code to avoid downloading everything every time.

Let me know if you'd like to get on a call to discuss how our consultants might be able to help you in the short term, and to learn more about your usage to inform our future development plans. We do expect to support integration between the forthcoming MATLAB package management system and artifact repositories, but don't have a timeline yet.

[Ashley Trowell]

I'm glad to see activity on this.

[Jan Kappen]

Just FYI, I'm very happy that the package manager has arrived (I'm on R2024b, but might be available in R2024a already): Package Management - MATLAB & Simulink

Only missing item is the Git integration. And a self-hostable package/registry server in GitLab (similar to the mentioned NuGet integration). @Ernst van der Pols have you progressed with the NuGet server? Any experience with the package manager in MATLAB?

Thanks :)

[Michelle Hirsch]

Thanks, @Jan Kappen. What do you have in mind with Git integration, beyond a self-hostable package registry server in GitLab or the like? MATLAB already has decent git integration for the development of the package source code. I do think that the biggest missing piece is flexible support for package repositories beyond just a shared file system, e.g. Artifactory, or a GitLab package registry.

[Jan Kappen]

Dear @Michelle Hirsch,

> I do think that the biggest missing piece is flexible support for package repositories beyond just a shared file system, e.g. Artifactory, or a GitLab package registry.

I agree :)

Until this exists, a local package meta data repository could make sense, something like a big json file telling MATLAB "this package is available in this and that version and can be found under this git link + revision". If the package does not exist in a local cache, it gets downloaded.

That way, we would avoid that all packages must be accessible from all MATLAB PCs, and instead just the meta data file can be distributed.

Or even easier: have a dependency list in each project with the link to git in the current version (url + rev), similar to buildinspace/peru: a generic package manager, for including other people's code in your projects

[Jan Kappen]

Dear @Michelle Hirsch are there any shareable news regarding the self-hosted pacakge registry on Artifactory or GitLab? I'd like to avoid developing something on my own and get it from MathWorks in the next release ;)

Thanks!

[Michelle Hirsch]

Sorry, but we don't yet have a firm date for either of these.