Logging Configuration - Storm Streaming Server

Storm Streaming Server utilizes popular and powerful Apache Log4j2 logging library. A single XML configuration file is provided with each server package.

Sample Configuration

You can easily configure the way Storm logs most important operations and errors. To do so, please edit the config/log4j2.xml file. The default configuration file will look like this:

xml

<Configuration status="INFO" monitorInterval="30">

    <!-- Logging Properties -->
    <Properties>
        <Property name="LOG_PATTERN">%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n</Property>
        <Property name="APP_LOG_ROOT">${sys:storm.defaultDirectory}/logs</Property>
        <Property name="CONSOLE_WARN_COLOR">yellow</Property>
        <Property name="CONSOLE_ERROR_COLOR">red</Property>
        <Property name="CONSOLE_INFO_COLOR">white</Property>
        <Property name="CONSOLE_DEBUG_COLOR">bright_white</Property>
        <Property name="CONSOLE_TRACE_COLOR">blue</Property>
    </Properties>

    <Appenders>
        <Console name="Console" target="SYSTEM_OUT" follow="true">
            <PatternLayout disableAnsi="false"
                           pattern="%highlight{%d{HH:mm:ss} %-5p %c{1} - %m%n}{FATAL=${CONSOLE_ERROR_COLOR} blink, ERROR=${CONSOLE_ERROR_COLOR}, WARN=${CONSOLE_WARN_COLOR} bold, INFO=${CONSOLE_INFO_COLOR}, DEBUG=${CONSOLE_DEBUG_COLOR} bold, TRACE=${CONSOLE_TRACE_COLOR}}"/>
        </Console>

        <RollingFile name="errorLog"
                     fileName="${APP_LOG_ROOT}/log-error.log"
                     filePattern="${APP_LOG_ROOT}/log-error-%d{yyyy-MM-dd}-%i.log">
            <LevelRangeFilter minLevel="ERROR" maxLevel="ERROR" onMatch="ACCEPT" onMismatch="DENY"/>
            <PatternLayout pattern="${LOG_PATTERN}"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="100MB" />
            </Policies>
            <DefaultRolloverStrategy max="50"/>
        </RollingFile>

        <RollingFile name="allLog"
                     fileName="${APP_LOG_ROOT}/log-all.log"
                     filePattern="${APP_LOG_ROOT}/log-all-%d{yyyy-MM-dd}-%i.log">
            <LevelRangeFilter minLevel="INFO" maxLevel="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
            <PatternLayout pattern="${LOG_PATTERN}"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="100MB" />
            </Policies>
            <DefaultRolloverStrategy max="100"/>
        </RollingFile>
    </Appenders>

    <Loggers>
        <Logger name="software.amazon.awssdk" level="WARN" additivity="false">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="allLog"/>
            <AppenderRef ref="errorLog"/>
        </Logger>
        <Logger name="org.apache.http" level="WARN" additivity="false">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="allLog"/>
            <AppenderRef ref="errorLog"/>
        </Logger>
        <Logger name="org.apache.http.wire" level="WARN" additivity="false">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="allLog"/>
            <AppenderRef ref="errorLog"/>
        </Logger>

        <Root level="debug">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="allLog"/>
            <AppenderRef ref="errorLog"/>
        </Root>
    </Loggers>

</Configuration>

Configuration Explanation

For detailed explanation and configuration option please check Apache log4j2 page: https://logging.apache.org/log4j/2.x/. Below you'll find a general overview for basic options.

The line below defines minimal logging level via status property. There are 7 levels in total: ALL, DEBUG, INFO, WARN, ERROR, FATAL, OFF. If you wish to log everything that console outputs you can replace status value with INFO. Parameter "monitorInterval" defines how often changes in the configuration file will be checked for changes (you don't have to restart server to modify logging settings).

xml

<Configuration status="WARN" monitorInterval="30">

Log patterns defines how each entry is formatted.

xml

<Property name="LOG_PATTERN">%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n</Property>

Field explanation:

Pattern	Description
`%d{yyyy-MM-dd HH:mm:ss}`	Element defines how date and time are logged.
`%-5p`	Element represents logging level e.g. INFO, ERROR, WARN.
`%c{1}`	Element represents class name.
`%m%n`	Element represent the message and new line (so the next message is added below).

This part controls what output is visible in the console (terminal / Powershell etc.). It uses modified log pattern to include colors.

xml

<Console name="Console" target="SYSTEM_OUT" follow="true">
    <PatternLayout
        disableAnsi="false"
        pattern="%highlight{%d{HH:mm:ss} %-5p %c{1} - %m%n}{FATAL=${CONSOLE_ERROR_COLOR} blink,
                 ERROR=${CONSOLE_ERROR_COLOR}, WARN=${CONSOLE_WARN_COLOR} bold, INFO=${CONSOLE_INFO_COLOR},
                 DEBUG=${CONSOLE_DEBUG_COLOR} bold, TRACE=${CONSOLE_TRACE_COLOR}}"/>
</Console>

Rolling adapter saves logs to a specified file. Once the number of entries or file size exceeds certain values, file is renamed to include date and a new one is created in its place.

xml

<RollingFile name="errorLog"
             fileName="${APP_LOG_ROOT}/log-error.log"
             filePattern="${APP_LOG_ROOT}/log-error-%d{yyyy-MM-dd}-%i.log">
    <LevelRangeFilter minLevel="ERROR" maxLevel="ERROR" onMatch="ACCEPT" onMismatch="DENY"/>
    <PatternLayout pattern="${LOG_PATTERN}"/>
    <Policies>
        <SizeBasedTriggeringPolicy size="100MB" />
    </Policies>
    <DefaultRolloverStrategy max="50"/>
</RollingFile>

Field explanation:

Parameter	Description
minLevel	Only logs above this level will be saved to a file.
maxLevel	Only logs below this level will be saved to a file.
SizeBasedTriggeringPolicy:size	Maximum file size.
DefaultRolloverStrategy:max	Max entries count.

Support Needed?

If you have any questions or need assistance, please create a support ticket and our team will help you.