Obtain Kafka Event Stream Log Files

When processing Kafka^® stream events using a KafkaStream object, use log files to help debug event streaming issues. These files contain all warnings, errors, and other information related to reading and writing events to and from Kafka streams. You can generate log files from these sources:

KafkaStream objects connected to the Kafka topic — Each object generates a log file containing information about reads from the Kafka server.
librdkafka Kafka C/C++ client library — This library generates a log file containing information about writes from the Kafka server.

For both sources, you can configure the log level, which controls the amount of output written to the log files. You can set these log levels, listed in order from least to most verbose:

off — Logging not enabled (default for librdkafka library)
fatal — Log only errors that force the Kafka connection to shut down
error — Log all errors (default for kafkaStream objects)
warn — Log all errors and warnings
info — Log errors, warnings, and high-level information about major streaming activities
debug — Log debugging information in addition to information in previously described options
trace — Log stack trace information in addition to information in previously described options

You can also configure the log level in your deployed applications.

Configure `kafkaStream` Object Logging

The log file for each KafkaStream object is generated to this file in your current folder.

log\topic_name.log

topic_name is the name of the Kafka topic that the KafkaStream object is connected to. If the log folder does not exist, the KafkaStream object creates it.

By default, the log level for these objects is set to error. To change the log level, update the log4j.properties.template file contained in your installation of Streaming Data Framework for MATLAB^® Production Server™. Open the file by entering this command at the MATLAB command prompt.

open(fullfile(matlabshared.supportpkg.getSupportPackageRoot,'toolbox','mps','streaming', ...
                 '+matlab','+io','+stream','+event','+kafka','collector','log4j.properties.template'))

To change the log level in the log4j.properties.template file, you must update the log level in all locations where the log level is specified. For example, to change the log level from error to debug, in the log4j.properties.template file, search for the lines containing this code.

.level = error

Within each line, update this code to the following:

.level = debug

Alternatively, to quickly review information only about the last error received from the Kafka topic, use the loggederror function. Pass your KafkaStream object as an input argument to this function.

To generate the logs, the streaming data framework uses version 2 of the Log4j Java^® library. For more details on this library, see https://logging.apache.org/log4j/2.x/.

Configure `librdkafka` Library Logging

When logging for the librdkafka library is enabled, the library generates a single file in the current folder named mw_rdkafka.log. If the log file does not already exist, the KafkaStream object creates it.

By default, the log level for the librdkafka library is set to off. To enabled logging, use this MATLAB command:

matlab.io.stream.event.kafka.internal.admin(log=logLevel);

logLevel is a string or character vector indicating the log level described earlier, such as "error", "warn" or "debug". For example, setting logLevel to "debug" enables logging and sets the log level to debug mode.

Configure Logging in Deployed Applications

When you package a deployed application using a project file created by the package or streamingDataCompiler functions, the packaging process copies configuration files into the output folder. You use these files when you start the Kafka Connector to push streaming data from your Kafka topic to your application hosted by MATLAB Production Server. To enable logging in the Kafka Connector, copy the log4j.properties.template file to a new file named log4j.properties. Then, edit this new file to choose the log level and set the name of the log file.

To set the log level, set the *.level properties to the desired log level, as described in the Configure kafkaStream Object Logging section. To set the name of the log file, change the text !ArchiveName! to the name of the desired log file. The !ArchiveName! text appears on these lines:

appender.rolling.fileName = ${basePath}!ArchiveName!.out
appender.rolling.filePattern = ${basePath}!ArchiveName!%i.out

For example, to send output log messages to the file RecamanSum.out, change these lines to:

appender.rolling.fileName = ${basePath}RecamanSum.out
appender.rolling.filePattern = ${basePath}RecamanSum%i.out

To control librdkafka logging, edit the rdkafkalog.properties file, which is located at this path:

open(fullfile(matlabshared.supportpkg.getSupportPackageRoot,'toolbox','mps','streaming', ...
                 '+matlab','+io','+stream','+event','+kafka','collector','rdkafkalog.properties'))

This file is installed with read-only access, so you must make the file writable before editing it. For your changes to take effect, you must edit this file before using the package or streamingDataCompiler function to create the deployable archive. Packaging incorporates the rdkafka.properties file into your application. Although you cannot change this file after packaging, you can use an environment variable to control the location from which MATLAB Production Server loads the rdkafka.properties file.

You can change the log level and log file name by setting values in rdkafka.properties. For example, to change the log level from fatal (default) to info, change the level=fatal line to level=info.

Similarly, the file property sets the full path to the output log file. The default is mw_rdkafka.log, as in:

file=mw_rdkafka.log

With no folder specification, this file is in the top-level folder of the MATLAB Production Server instance that hosts the deployed streaming application, which is the parent folder of the auto_deploy folder. If you deploy multiple streaming applications to the same MATLAB Production Server instance, make this filename application-specific. For example:

file=recamanSum_rdkafka.log

If you leave the filename unchanged, this file is shared by all streaming applications hosted by that server.

To change the folder where the file appears, specify a full or relative path using platform-appropriate syntax.

Windows^®	Linux^® or Mac
`file=c:\mps_logs\recamanSum_rdkafka.log`	`file=/myusername/mps_logs/recamanSum_rdkafka.log`

To temporarily change the location from which MATLAB Production Server loads the rdkafka.properties file, use the environment variable MW_IOSTREAM_RDKAFKA_LOG_CONFIG. You must set this environment variable before you start the MATLAB Production Server instance hosting your application. You must also set this variable in the environment from which you start that instance. Otherwise, MATLAB Production Server is unable to locate the environment variable. This environment variable setting changes the location of rdkafka.properties for all applications hosted by that instance of MATLAB Production Server. Use this environment variable for temporary troubleshooting only.

To temporarily cause all streaming applications to configure rdkafka logging according to the file and log level set in a nondeployed rdkafka.properties file, set MW_IOSTREAM_RDKAFKA_LOG_CONFIG to the full path of the nondeployed file. For example:

Windows	Linux or Mac
set MW_IOSTREAM_RDKAFKA_LOG_CONFIG=c:\mps_logs\recamanSum_rdkafka.log	setenv MW_IOSTREAM_RDKAFKA_LOG_CONFIG /myusername/mps_logs/recamanSum_rdkafka.log

Provide Log Files to MathWorks Technical Support

For additional help debugging, you can provide your log files to MathWorks^® Technical Support. Before providing the log files, set the log level to trace.

Locate the log files, using the previously provided information.
Attach the log files to a new or existing ticket for MathWorks Technical Support. See Contact Support.