How to configure Oracle/SUN JVM to send notification in case of JVM fatal error/crash

Recently we had JVM crash in production server and of course we had notified by some monitoring tools but not as fast as we could expect. So I configured our server with following JVM option:
-XX:OnError="sendjvmcrashsms". 'sendjvmcrashsms' is a Linux shell/bash file which is of course in user PATH, and here is example of the file

#! /bin/bash
echo "Production JVM crashed" | mail -s "+33000000000"

It will send mail to address with subject '+33000000000'. In our server we have configured SMS gateway which will send SMS to number(s) defined in subject, and here is docs:
-XX:OnError="<cmd args>;<cmd args>" - Run user-defined commands on fatal error

Experience with slf4j/logback markers

I want to share my experience with slf4j/logback markers, hope can be useful for others.

Problem: want to have possibility to mail for particular log messages regardless log level.

Solution: Using slf4j/logback markers to mark :) interested messages and logback Filter.

First I've defined simple class to keep like global registry for slf4j markers:

public class LOGMarkers {
      public static final Marker SEND_MAIL_MARKER = MarkerFactory.getMarker("SEND_MAIL");
}

and here is code example which is actually using marker:

LOGGER.info(LOGMarkers.SEND_MAIL_MARKER, "Cannot save document with id: {}", instanceId);

or

try {
  .................
} catch (SomeException e) {
     LOGGER.warn(LOGMarkers.SEND_MAIL_MARKER, "", e);
}

also I have to write logback filters and use it in logback configuration file:

package info.sargis.logging.filter;

import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.filter.AbstractMatcherFilter;
import ch.qos.logback.core.spi.FilterReply;
import org.slf4j.Marker;
import org.slf4j.MarkerFactory;

/**
* Created by IntelliJ IDEA.
* User: Sargis Harutyunyan
* Date: 10 oct. 2010
* Time: 20:12:12
*/
public class MarkerFilter extends AbstractMatcherFilter {

Marker markerToMatch;

public void start() {
     if (this.markerToMatch != null) {
          super.start();
     } else {
          addError(String.format("The marker property must be set for [%s]", getName()));
     }
}

public FilterReply decide(ILoggingEvent event) {
     Marker marker = event.getMarker();
     if (!isStarted()) {
          return FilterReply.NEUTRAL;
     }

     if (marker == null) {
          return onMismatch;
     }

     if (markerToMatch.contains(marker)) {
          return onMatch;
     }
     return onMismatch;
}

public void setMarker(String markerStr) {
     if (markerStr != null) {
          markerToMatch = MarkerFactory.getMarker(markerStr);
     }
}

}

 and finally logback config file example:


    <appender name="MARKER_EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
        <filter class="info.sargis.logging.filter.MarkerFilter">
            <marker>SEND_MAIL</marker>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <evaluator class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
            <marker>SEND_MAIL</marker>
        </evaluator>
        <SMTPHost>localhost</SMTPHost>
        <To>sargis@localhost</To>
        <From>twm@localhost</From>
        <Subject>EPAYMAIL: %date %-5level - %message</Subject>
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>%date [%thread] %-5level U:%X{vaspId} - %message%n</Pattern>
        </layout>
    </appender>

    <logger name="info.sargis" level="INFO">
        <appender-ref ref="EPAYCONSOLE"/>
        <appender-ref ref="MARKER_EMAIL"/>
    </logger>

and voila :)