Handling errors of type application/json when using webread

Having trouble with how MATLAB interpret's the message of a server error when type is of application/json.

Problem

Calling a local API, the following error is returned:

{"errorCode":401,"message":"Access to mylocalapi not authorized"}

Calling this using webread in MATLAB, the following exception is generated. Note the message has been stripped out of the exception.

Error using matlab.internal.webservices.HTTPConnectionAdapter/copyContentToByteArray

The server returned the status 401 with message "" in response to the request to URL http://mylocalapi

Error in webread (line 125)

[varargout{1:nargout}] = readContentFromWebService(connection, options);

Assessment

The error handler in HTTPConnectionAdapter does not seem to find the message part of the error. The response was compliant with the odata and Google API standards, but need to recognise there is no clear standard here. My problem is that the content has been stripped out by the time the response is returned as an MException object.

The only way around this seems to be to define a custom ContentReader in weboptions, however this has a performance impact across the full application, due to local disk i/o and poor ContentReader code on the part of the user.

Otherwise, the HTTPConnector generates the following MException

e = 
  MException with properties:
    identifier: 'MATLAB:webservices:StatusError'
       message: 'The server returned the status 401 with message "" in response to the request to URL http://mylocalapi.'
         cause: {0×1 cell}
         stack: [11×1 struct]
e.stack(1)
ans = 
  struct with fields:
    file: 'D:\MATLAB\R2018b\toolbox\matlab\external\interfaces\webservices\restful\+matlab\+internal\+webservices\HTTPConnector.m'
    name: 'HTTPConnector.copyContentToByteArray'
    line: 361

Might it be possible to extend the MException class to have a HTTPException with additional fields such as the full error response?

For info, the following weboptions are being sent with the request.

options = 

  weboptions with properties:

      CharacterEncoding: 'auto'
              UserAgent: 'MATLAB 9.5.0.1067069 (R2018b) Update 4'
                Timeout: 5
               Username: ''
               Password: ''
                KeyName: ''
               KeyValue: ''
            ContentType: 'json'
          ContentReader: []
              MediaType: 'application/json'
          RequestMethod: 'get'
            ArrayFormat: 'json'
           HeaderFields: {1×2 cell}
    CertificateFilename: 'D:\MATLAB\R2018b\sys\certificates\ca\rootcerts.pem'
    

Apologies if this is non-reproducable, however it is using a local service.

Answer 1

[Sylvain Lacaze]

Hi Ciaran,

I'm not sure I completely understand what your API returns. It's not clear to me that the HTTP Response has a status code 401 and a Body "Access to mylocalapi not authorized", or if it returns some 2XX code with a JSON payload as you describe.

I'm not aware of a way to get "webread" to return the response payload in case of non 2XX or 3XX status code.

Maybe to get a better understanding of this, could you use the HTTP interface to see what response you get, using this starting snippet?

request = matlab.net.http.RequestMessage();
request.Method = 'GET';
request.Header = [ ...
    matlab.net.http.field.ContentTypeField( 'application/json' ), ...
    matlab.net.http.field.AcceptField( 'application/json' ), ...
    ];
response = request.send( 'https://www.google.com/doesntexist' );
disp( response )
disp( response.Body )

For reference, not the outcome of:

webreadResponse = webread( 'https://www.google.com/doesntexist' )

From this, you can see you get an MException, whose message is completely formed from the response.StatusLine. It makes no use of the response payload, that you can see in the reponse.Body.

Can you post what the reponse and response.Body would be in your case?

[Ciaran McAndrew]

Hi Sylvain,

Firstly, your response really helped and gave a good and simple example of how to use the low level HTTP interface. I have been able to solve my issue by writing my own error handler that interprets the response payload which contains the error message in JSON.

To answer your question and to understand the issue better, the API is returning a status code of 401, plus a payload with the JSON package:

{"errorCode":401,"message":"Access to mylocalapi not authorized"}

Showing this using HTTP interface: (different error but same mechanics)

disp(response)
  ResponseMessage with properties:

    StatusLine: 'HTTP/1.1 404 '
    StatusCode: NotFound
        Header: [1×12 matlab.net.http.HeaderField]
          Body: [1×1 matlab.net.http.MessageBody]
     Completed: 0
     
disp(response.Body)
  MessageBody with properties:

             Data: [1×1 struct]
          Payload: []
      ContentType: [1×1 matlab.net.http.MediaType]
    ContentCoding: [0×0 string]
    
disp(response.Body.Data)
    errorCode: 404
      message: 'Data does not exist'      

This approach in generally in-line with the odata specification (http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_ErrorResponseBody) and the Google cloud API (https://cloud.google.com/apis/design/errors)

While the approach of using the low-level HTTP interface does what I need, I feel that it seems a bit overkill when webread provides 99% of the required functionality. I have had to add significant complexity to my code, which has a long-term cost implication.

If I understand what MATLAB is doing, it seems webread will only handle an error message if it is in plain text in the reponse payload. I would recommend MathWorks consider whether webread can be made more flexible, as the above examples of how REST APIs are becoming richer in terms of exception details.

One idea may be to provide a field in weboptions to specify a custom error or response handler, a bit like how a custom error handler works in cellfun.

This could work as follows:

function data = myResponseHandler( response )
% where response is of type matlab.net.http.ResponseMessage
    switch response.StatusCode
        case matlab.net.http.StatusCode.Continue % + other conditions
            data = response.Body.Data;
        otherwise
            error('The server returned the status %d with message "%s" in response to the request to URL %s', response.StatusCode, response.Body.Data.Message, url );
    end
end
      
opts = weboptions( 'ResponseHandler', @myResponseHandler );
webread( url, opts );

[Sylvain Lacaze]

Hi Ciaran,

I'm glad I could help. One clarification on this excerpt of your comment:

it seems webread will only handle an error message if it is in plain text in the reponse payload

Webread uses the text from the StatusLine, and completely ignores the payload. The StatusLine is dictated by the HTTP spec, which universal. The payload on the other hand, could be anything. Also OData, and Google APIs are widely used, every API could return anything in the payload, making it unreliable to parse. Some API even return binary payloads.

In any case, I'll make sure your feedback gets logged in our system. Development will assess the idea of a customer error handler.

Best,

Sylvain