Tuesday, March 1, 2016

OWASP Security Logging Project Explored



The OWASP Security Logging Project is a security logging API that extends popular logging technologies providing powerful security logging capabilities to your projects.  If your project already uses a popular logging API like log4j, logj2, logback, etc. then most of your work is done.  Even for legacy projects there are some benefits.  In this post I provide some of the projects more important features and benefits as well as some code samples.  The security logging project is lead by Sytze van Konigsveld, August Detlefsen, and myself.  We are the project leaders to contact if you have a question or request.  More importantly we are the team working on the security logging project code.   First things first, how can you include OWASP Security Logging in your project?

From Maven, add OWASP Security Logging with log4j2 support to your project's pom.xml (check for updates)

<dependency>
<groupId>org.owasp</groupId>
<artifactId>security-logging</artifactId>
<version>1.1.0</version>
</dependency> 

Use and initialize your SLF4J compliant logger in the usual way, add the import, retrieve a logger instance, and start logging.  With the startup out of the way, let's discuss the benefits of the security logging project and why you need security logging in your projects.

SLF4J and JDK Compatibility
SLF4J is a logging interoperability specification facilitating interoperability among heterogeneous logging systems and used by the security logging project.  Photo 1, shows some of the common binders for SLF4J and logging frameworks supported by the security logging project.

Photo 1:  SLF4J bindings from slf4j.org website (click to enlarge)

Java logging compliance is based upon JSR-47 and while SLF4J compliant loggers like log4j2 appear similar there's some history and technical difference you may wish to explore on your own[1].  SLF4J requires JDK 1.5+ therefore security logging supports JDK1.5+.  If you have a need we have not anticipated please send us a note or open a ticket on the projects github site.

Legacy Support
Most development efforts require at least some support for legacy systems.  In the case of logging, this means non-compliant SLF4J applications or perhaps some non-compliant application components.  By non-compliant I mean the application logs messages to system streams like System.out and System.err.  The security logging platform includes utility code to redirect these streams to SLF4J compliant loggers.  Logging to streams is not as desirable as native SLF4J support but the some ancillary benefits exist like messaging filtering and distribution.  Intercepted streams can also be provided to SIEM tools and used for event correlation, diagnostics, and forensics.  As an example, you can capture console data and to a remote centralized logging server, scan for event triggers, and alert IT staff of attacker activity.  The security logging project helps with collecting and organizing the log information.  You will need open source, commercial tools, or develop your own for analysis of the log information.
If you already using SLF4J compliant Java logging like, log4j2, log4j, or logback you are ready to use security logging now.  You can begin receiving benefits immediately without recompiling your code.
Mapped Diagnostic Context Support(Pre-populated log message metadata)
Even if you are using an SLF4J compliant logger it's likely your not using powerful features like Mapped Diagnostic Context(MDC).  MDC's are similar to a Map, simple key value pairs.  You assign key/value pairs in one place and optionally include them with your log messages everywhere you decide to log a message.  Once assigned, this extra data provides more meaning or context to your log messages.  The security logging project provides an easy way to leverage MDC's and pre-populate them with data from common usage scenarios.  With compressed project schedules, we understand logging is not at the forefront of everyone's mind.  It's easy to overlook logging and instead concentrate on project features.   The security logging project is helpful since it will log application metadata by default depending upon the type of application your developing making it available for easy use.
  • Standalone (Console/JavaFX/Swing applications), MDC includes: command line options, system properties
  • Application server(Servlet), includes the standalone metadata plus J2EE metadata like: container startup options, servlet properties, and application runtime properties
A concrete example, a typical web application has many users operating concurrently and it's often beneficial for forensic/diagnostic purposes to correlate the ID of the user currently logged on with the corresponding activity, data base update for example. Consider a log message with included MDC metadata like the ID of the currently logged on user may look like the following,

08:59:51.204 [Thread-0] INFO JediKnight com.sample.servlet.alliance Adding employee DarthVader to database.

The previous log message was issued by the application to add the new employee and it's performed by JediKnight.  If the database activity occurred outside the scope of the user request then the user ID would be dropped from the message or alternatively the application designer can include the applications service account.  From a forensics standpoint it makes it easy to understand which activities were initiated by people using the application or services interacting with each other in the cloud.  Likewise, other application activities occur within the scope of the application servers process or HttpServlet, some activities within the scope of the user's HttpSession, still more granular activities within the HttpServletRequest.   The security logger provides the framework to encourage rich log messages.  You can choose the metadata want to log via configuration settings, it's up to you. 

Security Markers (Supports information classification)
For organizations that classify data assets, usually governments, the security logging platform provides features to support your information classification efforts.  An example, in your application you may wish to log CONFIDENTIAL messages like the following.

08:59:51.204 [Thread-0] CONFIDENTIAL JediKnight com.sample.servlet.alliance Employee record for DarthVader updated, disposition=evil.

You can also handle messages differently depending upon their classification.  For instance, you can have CONFIDENTIAL messages logged to a vaulted server with stringent access control whereas less sensitive messages on a server more accessible to project developers.  Additionally, the security logging framework provides features to flag unclassified messages for follow-up action.  Unclassified messages incur some risk since they may contain sensitive information.  Define your classifications and message handling to suite your organizations needs.

Security Filters/Masks(Handling for sensitive content)
Concerned about developers leaving debug features enabled and logging sensitive credentials?  The security logging framework provides features to optionally place asterisks over sensitive content like passwords or so they don't show up in logs.  Security filtering is the way to get this done[5].  An example from the project Wiki.

In Java source code, add the CONFIDENTIAL marker to log statements that could contain confidential information:

LOGGER.info("userid={}", userid);  

LOGGER.info(SecurityMarkers.CONFIDENTIAL, "password={}", password);

The previous configuration will produce the following output in the log:

2014-12-16 13:54:48,860 [main] INFO - userid=joebob

2014-12-16 13:54:48,860 [main] [CONFIDENTIAL] INFO - password=***********

This feature is helpful since developers can see the information during debug and testing sessions whereas it's blocked in production.  You can also extend this to filter other information like, Social Security Numbers or perhaps credit card numbers.

Interval Status Logging(Record of system health)
Often during security forensics or software debugging understanding the conditions at or prior to a particular event of interest like an attack are helpful to speed resolution.  The Interval Status Logger is a background worker thread that logs various system activities of interest at predetermined user defined intervals.  We developed information we thought most people would like to see printed in their logs at regular intervals.  An example of the default interval status message looks like the following,

20:10:10.204 [Thread-0] INFO Watchdog: MemoryTotal=64.5MB, FreeMemory=58.2MB, MaxMemory=947.7MB, Threads Total=5, Threads New=0, Threads Runnable=3, Threads Blocked=0, Threads Waiting=2, Threads Terminated=0 

The interval logger uses the familiar Model View Controller pattern so developers can customize both the metadata content available and format of interval log messages to meet individual logging needs.  The security interval logger is initialize like the following.

logger.info("interval logging started");
IntervalLoggerController wd = SecurityLoggingFactory.getControllerInstance();
wd.start();

The default interval to log this message is 15 seconds and is adjustable. In our test harness we provide a more rich example of editing the interval logger content.  An example is provided in JavaDocs to update the interval log message with the number of open files in the system[2].  You may wish to include number of users logged into your web application and max number of users logged into your application.  What you log on your system and what's important to you is entirely up to you.  

Forward Looking Features
Following is not an exhaustive list and subject to change but is roughly some areas/features we are considering improving or actively engaged in improving.
  • New MDC metadata content, future planned content/property settings to optionally include in log messages
  • Auditing support, the requirements of auditing are more stringent than security.  We have some ideas to include features that add value for logging.  Off site message vaulting, tamper proof logging(signed messages), secure log transport(HTTPS), authenticated clients, and more
  • Internationalization, today some areas of code support it better than others.  Needs some consistency applied thought the implementation.  We don't have plans to localize in anything except US English at this time
  • Asynchronous logging, log messages locally to cache/spool then forward to destination.  Improve client logging performance since message is sent offline.  Some logging platform provide functionality in this area so it's still under investigation.
  • Guaranteed delivery, some systems have strict requirements to preserve all messages.  For example, storage and handling of log messages considered evidence to support regulatory compliance.  Guaranteed delivery is a sort of transaction control to ensure a logged message exists either on the client, or on the server, but not in both locations at the same time.  Further that log messages are never lost in transit.  This feature requires some investigation.
OWASP security logging is open source project so there's room to participate if your interested.  If your having problems with security logging you can contact us on the OWASP site.  Of course, if you having success we like to know that as well.  Security logging is a labor of love and we do this for free and for the betterment of the application development community.  Our employers also support OWASP, public security improvement efforts, and our project.  A big thanks to Oracle for me!  If you find this project useful or have suggestions we are interested to hear.

For official project news please refer to the OWASP project page [3].

[1] JSR-47 vs. log4j, by Ceki Gülcü
[2] ExtendedIntervalLoggingTest.java, demonstrates adding user defined interval logging content.
[3] OWASP Security Logging Project, main landing page for the security logging project on OWASP.
[4] OWASP Security Logging Project Roadmap, authoritative project roadmap.
[5] OWASP Security Logging Project Wiki(filtering), information on log messaging filtering.

Share It!