Receiving Notifications When Enterprise Manager is Down

Enterprise Manager monitors itself for its own availability. It does this by having a Management Agent run an external monitoring script that periodically checks the availability of the Management Repository database and the Management Services. (The Management Repository and Management Services are logically combined and modeled as a single target called the “Management Services and Repository” target). If the Management Agent detects that the Management Repository database is down or if all the Management Services are down, then it runs a script that can be customized to notify administrators of the problem. This script will be run outside of Enterprise Manager's regular notification mechanism because the notification mechanism itself will be unavailable if the Management Services are down.

The Management Agent that is responsible for checking the availability of the “Management Services and Repository” target is called the Monitoring Agent. You can determine the monitoring Agent for your Enterprise Manager installation by viewing it in the Management Services and Repository Overview page in the Enterprise Manager console.

The script that the Monitoring Agent runs to determine availability of the “Management Services and Repository” target is called emrepresp.pl (located in the $Oracle_Home/sysman/admin/scripts directory of the Monitoring Agent). The Monitoring Agents runs this script every 5 minutes. If it determines that the management repository database is down, or if all Oracle Management Services are unavailable, it will run a notification script called emrepdown.sh. This notification script is located in the Oracle_home/bin directory of the Monitoring Agent. By default, the notification script will send e-mail (via mailx) indicating that Enterprise Manager is down, and will include relevant error information. This emrepdown.sh script should be customized for your environments.

If you want the emrepdown.sh script to continue to use the mailx command to send e-mail notifications, make sure you specify the appropriate e-mail addresses. These e-mail addresses are specified via the following properties in the Oracle_home/sysman/config/emd.properties file (where Oracle_home is the oracle home directory of the monitoring agent):

  1. emd_e-mail_address: specify the e-mail address to which the e-mail notification will be sent. You can specify more than one e-mail address, separated by commas.
  2. emd_from_e-mail_address: specify the e-mail address of the sender of the e-mail. Specify any valid e-mail address for this property. Note that since the script uses mailx, it will run only on certain platforms. Specifically, it will run if your Monitoring Agent is on any of the following platforms: Sun SPARC Solaris, HP-UX, HP Tru64, Intel Linux, IBM AIX, and OSF1.

Example of an emrepdown.pl Script

#!/usr/local/bin/perl

#

# insert the commands you wish to execute when eml fails to stay running.

#

#

# Mail to specified user

#

# uncomment out the following lines and specify a valid e-mail address

# in place of nobody@somewhere.nowhere to send the failure e-mail.

# The message text is passed in as the first parameter.

# The subject is passed in as the optional second parameter. Default subject is 'Enterprise Manager is down'

#

 

use strict;

use Oraperl;

use Time::HiRes;

 

require "emd_common.pl";

require "semd_common.pl";

 

sub send_mailx

{

  my $start_time = Time::HiRes::time;

  my $targetname = "$ENV{EM_REPOS_USER}1234";

  my $fn = get_tmp_filename ($targetname, "emrepdown");

  system "date > $fn";

  if(!open(FH, ">>$fn"))

  {

  EMD_PERL_ERROR("Could not open $fn for error messages");

  }

  my $subject="Enterprise Manager is Down";

  if ($#ARGV+1 > 0)

  {

  if ($#ARGV+1 > 1)

  {

  print FH "$ARGV[1]/n";

  print FH "Error message: ";

  print FH "$ARGV[0]/n";

  $subject=$ARGV[1];

  }

  else

  {

  print FH "Enterprise Manager is down./n";

  print FH "Error message: ";

  print FH "$ARGV[0]/n";

  }

  }

  else

  {

  print FH "Enterprise Manager is down./n";

  }

 

  if(!close FH)

  {

  EMD_PERL_ERROR("Could not close $fn");

  }

 

  my $filename="../sysman/config/emd.properties";

  if ( $ENV{EMDROOT} ne "" )

  {

  $filename="$ENV{EMDROOT}/sysman/config/emd.properties";

  }

 

  my $command1="cat $filename | grep -i EMD_EMAIL_ADDRESS= | sed s?EMD_EMAIL_ADDRESS=??i | awk '{print $1}'";

  EMD_PERL_DEBUG ("command1=$command1");

  print STDOUT "command1=$command1";

  my $list=`$command1` or EMD_PERL_ERROR("Could not execute $command1");

  $command1="cat $filename | grep -i EMD_FROM_EMAIL_ADDRESS= | sed s?EMD_FROM_EMAIL_ADDRESS=??i | awk '{print $1}'";

  EMD_PERL_DEBUG ("command1=$command1");

  print STDOUT "command1=$command1";

  my $return=`$command1` or EMD_PERL_ERROR("Could not execute $command1");

  my $out;

  chomp($return);

  chomp($list);

  EMD_PERL_DEBUG ("list=$list");

  EMD_PERL_DEBUG ("return=$return");

 

  if ( "$list" eq "" )

  {

  EMD_PERL_ERROR("e-mail address list is empty");

  }

  else

  {

  if ("$return" eq "" )

  {

  $command1 = "mailx -s /"$subject/" $list /< $fn";

  EMD_PERL_DEBUG("command1=$command1");

  print STDOUT "command1=$command1";

  system $command1;

  }

  else

  {

  $command1 = "mailx -s /"$subject/" -r $return $list < $fn";

  EMD_PERL_DEBUG("command1=$command1");

  print STDOUT "command1=$command1";

  system $command1;

  }

  }

  $out = `rm $fn` or EMD_PERL_ERROR("Could not remove file $fn");

}

 

my $platform = get_osType();

if(($platform eq "SOL") ||

  ($platform eq "HP") ||

  ($platform eq "LNX") ||

  ($platform eq "AIX") ||

  ($platform eq "OSF1"))

{

  send_mailx();

}

else

{

  EMD_PERL_ERROR("Platform $platform does not support mailx.");

}

 

Example of E-mail Notification

Subject: Enterprise Manager is Down

Date: Thu, 18 Jul 2005 07:37:55 -0700 (PDT)

From: John Smith <jsmith@jsmith-sun.us.oracle.com>

To: john.doe@oracle.com

Thu Jul 18 07:37:55 PDT 2005

Enterprise Manager with repository jdoe-sun:1522:emdw1 is down

Error message:

ORA-01034: ORACLE not available ORA-27101: shared memory realm does not exist SVR4 Error: 2: No such file or directory (DBD ERROR: OCISessionBegin)

Related Topics

About Notifications

Enterprise Manager Information Roadmap