There is currently no off-the-shelf functionality in MATLAB to add a licensing procedure to a custom toolbox. However, there are several possibilities to control and restrict the execution of your application.
A very simple way would be to use a shared file which can be accessed by your applications, possibly running on several clients connected by a network. This shared file would contain information about the number of your currently running applications. Whenever a new application is started, it first reads the shared file in order to check how many of other of your applications are already running. If the number of currently running applications is sufficiently low, it leaves a tag in the file. In the simplest case, this tag would be a counter which is increased by one.
This tag could also be some encrypted information containing user name, time, host name, etc. When the application is closed, it removes its tag from the file.
What about if an application has been terminated without having been able to remove its tag, e.g. crash, error in the application, etc.? This would lead to accumulation of incorrect information about the number of currently running applications in the shared file. This can be prevented by continuously updating the entries in the file, say every 15 minutes using the timer function. Therefore, a newly started application should not just count the number of allegedly running applications. It should only count the number of actually running applications by taking into account the current time and the last heartbeat from the registered applications and remove the tags from dead applications.
Of course, you have to make sure that the source code is not visible for the customer. You could either convert your MATLAB files to p-files using the PCODE function provided that the customer owns a MATLAB license and the necessary toolboxes. Or you could use the MATLAB Compiler to generate a standalone application if the customer does not have a MATLAB license.
