Search

Script Monitoring

Script Monitoring

In Offsite Disaster Backups, I outlined three methods I use to move backup data to a backup machine with removable hard drives.  Essentially, this backup system replaced our tapes almost entirely.  The one thing that I didn’t like about it was the way that each of the systems to be backed up were being logged.  I used three methods: NTBackup, Robocopy, and Veeam.

  • The native NTBackup utility copied all of the log files to a directory and rotated the log file names.  This was difficult to manage in its native form so I compiled a workaround script to copy the log files over to the backup media.  The trouble with this workaround is that I had to manually go and clear the logs off the backup drive.  This is not a significant problem but it is a hassle.
  • The Robocopy scripts with the /MIR switch are pretty straightforward.  The log file path can be determined in the script which I usually put alongside the mirrored file structure on the backup disk.
  • Veeam e-mails me when it is done copying to the media.  The ghettovcb.sh script also logs to a file and sends e-mails to me saying that a backup is commencing and when it’s finished.

The real concern I have is coordinating all of these events and reading entire separate and individual log files scattered around the disk.  Should something go wrong, how would I know without looking at the logs?  Research HO!

Deja Vu

I had a similar issue with browsing event logs of the numerous Windows systems and servers that I support.  Every day, we’d have to go to the Event Viewer of every server and look at the system and security event log to find out whether a service failed to start, the system has encountered some sort of problem, or other thing that the Dude wasn’t configured to monitor because we didn’t know about a potential for problem until it already happened.  These Windows logs are very “noisy” and don’t always apply.  The Microsoft interface makes it really hard to sort the wheat from the chaff.

In Windows Events to Syslog I described the way I configured Syslogagent (freeware) to forward Windows event logs to the Dude via Syslog.  The post also describes how to separate, filter and otherwise notify certain syslog messages using regexp from the windows events.  I also described how to monitor a Windows file on the file system by sending changes to that file to syslog.  I used the DHCP log file was configured to send all updates to DHCP so that I know in real-time when a computer requests a DHCP operation.

Then, Ben went on to describe how to setup CDR for our phone system which was being logged to a text file on our Shoretel phone system in the Shoretel Events to Syslog post.  Pretty much any text log file could be monitored via our NMS, The Dude.

The Problem

We needed to find a way to alert us when our backup scripts have completed or have failed.  Because of my backup method uses task schedulers, it’s difficult to find out why, when, and how something failed without looking at the individual logs or looking at the Task Scheduler for an error code.

The Solution

I’ve done this before.  Recalling all the building blocks we used in the past as described above, I thought that it would be worthwhile to modify our scheduled windows scripts to notify us of their progress via syslog.  This way, all alerts will be sent to our NMS the Dude and we can monitor almost everything from the Dude’s event log instead of hopping around all the systems looking for problems.

The Steps

Basically, I setup these items to accomplish the alerting task:

  • SyslogAgent configured for application monitor c:\scripts\scripts.log
  • A scheduled task running a script
  • Using an echo statement (or similar method) with an append redirect to write to the log file
  • Configured The Dude’s syslog Regexp to filter or notify for each alert

All of our scripts are housed in c:\scripts.  This directory contains all of our custom scripts for that server.  The log file must exist before continuing with configuring the syslogagent to monitor it.  To create one, from a command prompt, type:

Echo Test >> c:\scripts\scripts.log

Now to configure the SyslogAgent, open the configuration screen.  You should already have this setup.  If not, read my post: Windows Events to Syslog.

Click on Add Application and configure the application with the following settings:

Go to your Dude and setup the regexp for “Leka-scripts”

Note that the orders of the syslog rules are important.  To change order, click, hold, and drop the rule to a new position on the list.  Actions are important as well.  The “passthrough” action will match “Leka-scripts” and will notify but will allow the rule to continue passing through the rest of the rules.  An action of “accept” will match regexp and terminate.  A “drop” action just kills the message if it is matched.  Use these action options carefully.

Notification options are setup in the notification screen of The Dude.  A notification can be any one of the following types:

  • Beep
  • Email
  • Execute locally
  • Execute on server
  • Flash
  • Group
  • Log
  • Popup
  • Sound
  • Speak
  • Syslog

Each notification can have a schedule.  I like to setup my personal cell phone to alert me in the event some things are going very wrong.  I would setup an e-mail notify to my cell phone and set the notification schedule for this to be active only during off-hours.  This way, I can get important notifications customized for the devices I want to monitor.  Here are the notifications methods I have setup currently:

Now, I need to modify my command scripts or wsh files to append status updates when they execute.  The following is a script I’m using for one of my servers (Leka) to delete log files older than 60 days:

Forfiles /d -60 /c “CMD /C del @FILE”

I would like to be notified when this runs.  So, I added the following line above the script:

Echo START Log file delete process begin %date% %time% >> C:\scripts\scripts.log

Now, on my event viewer in The Dude I get an alert:

Now, the notification can be setup to do anything you want when triggered by a match.  It really is up to you.  For these notifications, I have them going to my Event Log.  This log identifies things that we keep a close eye on.  Notice how I’ve used environment variables %date% and %time%.  You can also use environment variables to pass errorlevel and other dynamic data should you SET them in your script.  Use the SET command to find other system environment variables or to set them when modifying your scripts.

Summary

I think the final picture shows the significance of what is possible here.  I haven’t gone much into the notifications outside of event logging, e-mail, and texting a cell phone.  There are some things that I have been reading about using text-to-speech and a modem to call and annunciate should the network be unavailable.

The possibilities are open to your imagination.  Why not comment and let me know if you used this or describe a way to accomplish the same thing?

Have fun!

Advertisements


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s