Is it possible to disable the debugger?

pcode is a method to obscure an m-file's code such that a user cannot easily read the file. However, when executed within Matlab, the variable names appear to be maintained and viewable through the debugger and the function can be easily stepped through line by line. The line text is still not accessible, however you can monitor how the variables change.

Is there a way to disable the debugger before the code execution so there is no way for the debugger to step into nor through the pcode function?

Answer 1

[Jan]

It is very hard to disable the debugger reliably. I claim it is even not possible.

You can include at least some obfuscation to impede the debugging:

function dbstep(varargin)
function dbcont
function dbup
function dbdown
function [Q,I]=dbstack(varargin)  % Do not care about "Q" or "I": Obfuscation!
function [varargout]=inmem(varargin),varargout=cell(1,nargout);
function S = who(varargin),      S = {};
function S = whos(varargin),     S = builtin('whos', 'fraud_');

And in addition shadow even these functions by local variables:

dbstep = [];
dbcont = [];
etc.

Of course you should clear all breakpoints in the code by "dbclear funcname", but they can be set again, when any of the called toolbox functions can be stopped by a breakpoint.

I've tried a lot of more tricks:

• Putting all code in one line help a lot, because the debugger steps through the code line by line. But still you can set breakpoints in all subfunctions.
• dbclear of all functions of Matlab's toolboxes to avoid that the user can break the process. But an attacker can shadow "dbclear", "builtin", etc.
• AES encrypted strings, which are EVAL'ed - bad idea: the key must be stored in the function and can be obtained.
• One single wrapper function called "Q" and an encrypted message, which determines the action to perform. Finally the M-code was such ugly, that a P-coding was not needed anymore...
• Opening a second Matlab instance and evaluate the function there
• Check the timing using the high accuracy timer of Windows in a Mex to detect, if the code processing is slowed down by a curious user. Of course a detected fraud is should not lead to an immediate action, because this would reveal the location of the test.
• I've learned more dirty tricks from the cheaters at Cody.

There are more techniques like obtaining the memory address of the loaded debugger functions and inject code for an early return. But without doubt this would be a reverse engineering of Matlab and therefore a conflict with the license conditions. It is a bad idea to protect your code by cracking another one.

If you find a way to block Matlab's debugger, an assiduous user can still use a disassembler to debug Matlab itself while it runs your P-code.

My conclusion: P-coded files are not readable and observing, what's going on inside, is possible. The debugger can not be disabled. Obfuscation increases the time is to analyze a function, but in consequence the code looses its maintainability. Therefore the main advantage of P-coded function is, that a user cannot claim to have copied a part of the code unintentionally. Analyzing a P-file is obviously a reverse engineering. Therefore I think, that a solid non-disclosure agreement is much better than any crude obfuscation tricks. Here "solid" means a physical sheet of paper and the requirement of a hand-written sign - the check-box NDAs used by all standard software are ignored by 100% of the users.

[Adam]

could you explain what you mean with the above?

[Jan]

@Adam: This question is too general to be answered efficiently. So let me answer generally, although I thought the detailed answer above would be more helpful: No, you cannot disable a debugging efficiently. There is always a trick to reveal the processes inside a P-coded function. If you invest weeks of programming time, you can increase the time required for debugging from minutes to hours.

@Readers: I hesitated to post details about the methods to obfuscate and reveal the contents of P-coded functions. Finally I do not think that anything I've found by easy and legal investigations has the status of magic secret knowledge. I think Matlab users should know, that P-coding hides the source code efficiently, but it does not conceal the internal processing. The concept of interpreted languages like Matlab does not allow to hide information securely.

Answer 2

[Daniel Shub]

Jan has provided some links to CSSM threads about p code security here.

[Adam]

I am completely aware of these and is the express reason for the original question. P-coding your file alone only gets you half way to "protection" where a large vulnerability is easily available through the debugger.

However, if there is a command(s) that one can add to the beginning of the original function which disables the debugger, then that would mitigate that vulnerability.

Example:

    >> dbstop in test.p
    >> test.p

The debugger immediately starts at the first line of test and you can 'dbstep' through and view each variable as they change in the Workspace.

An idea could be to use 'dbclear all' at the beginning of the file, however that doesn't seem to work as it places the breakpoint at the 'dbclear all' line so when executed, you enter the debugger before the commend is issued so then when you step, it eliminates the breakpoint but you are already in the debugger.

If the debugger was just completely disabled that would mitigate this overall vulnerability regardless of if an attacker overloaded a function or not.

[Sean de Wolski]

One thing you can do is put all of the commands on one line so that only the first line is visible.

a = pi;b=17;c=a*b;

[Adam]

Unfortunately, thats not really feasible on over 1000 lines of code. Additionally, even with your example, unless you clear a and b on that same line, someone would still see those variables while debugging.

This method is not realistically manageable.

[Jan]

@Sean: You can overload subsasgn and insert a breakpoint in the new code. Then you still cannot observe the left hand side of the assignment, but you can a kind of step through the code and let the workspace browser show, what happens in the previous step.

[Daniel Shub]

@Adam I would think that a post processor could transform multiline code into single line code prior to p-coding. A "clearvars -except" could help with clearing the intermediate values.