Using MONIT to check and restart system services Karl Vogel %%mtime(%a, %d %b %Y %T) + Introduction + This is an update to a [reddit posting https://www.reddit.com/r/freebsd/comments/1vf5jgp/] I made about monitoring cron and restarting it if it dies. I've only had cron fail on me once (on Solaris, if anyone cares), but I depend on cron for so many things that I set up monit to keep an eye on it. After I got it working, I added a few other checks: - host status: high load average, CPU, memory, or swap use. - is cron running? - is sshd running? - is ntpd running? - is syslogd running? - is Apache running? - does the Apache binary pass a simple integrity check? + The monit ruleset + Here are the basic rules I use - [plaintext is here monitrc]: ``` ######################################################################## ## Monit control file ## ## Revision: 2.7 Date: 2026-09-22 19:53:34-04 ## Source: /usr/local/src/m/monit/admin/RCS/monitrc,v ## UUID: 74fc1f9b-3e66-4446-8118-40cb3c9dddbd ######################################################################## ## ## Comments begin with a '#' and extend through the end of the line. ## Keywords are case insensitive. All paths MUST BE FULLY QUALIFIED, ## starting with '/'. ## Start Monit in the background (run as a daemon): set daemon 120 # check services at 120-second intervals with start delay 30 # optional: delay first check by 30 seconds ## Set logging. set log /var/log/monitlog ## Set the location of the Monit lock file. set pidfile /var/run/monit.pid ## Unique id for the Monit instance. set idfile /var/log/monit.id ## State file which saves monitoring states on each cycle. set statefile /var/log/monit.state ## Set the list of mail servers for alert delivery. set mailserver localhost ## Monit will drop alert events if no mail servers are available. ## To retry delivery, use the EVENTQUEUE statement. set eventqueue basedir /var/monit # base directory where events will be stored slots 100 # optionally limit the queue size ## Who gets alerts if/when a service has errors. set alert you@your.site.com ## Check general system resources. $HOST is set by monit. check system $HOST if loadavg (1min) > 4 then alert if loadavg (5min) > 2 then alert if cpu usage > 95% for 10 cycles then alert if memory usage > 75% then alert if swap usage > 25% then alert ## LOCAL: Make sure cron is running. check process cron with pidfile /var/run/cron.pid group system start program = "/usr/sbin/service cron start" stop program = "/usr/sbin/service cron stop" ## LOCAL: Make sure SSH is running. check process sshd with pidfile /var/run/sshd.pid group system start program = "/usr/sbin/service sshd start" stop program = "/usr/sbin/service sshd stop" ## LOCAL: Make sure NTP is running. check process ntpd with pidfile /var/db/ntp/ntpd.pid group system start program = "/usr/sbin/service ntpd start" stop program = "/usr/sbin/service ntpd stop" if failed host 127.0.0.1 port 123 type udp then alert ## LOCAL: Make sure syslogd is running. check process syslogd with pidfile /var/run/syslog.pid group system start program = "/usr/sbin/service syslogd start" stop program = "/usr/sbin/service syslogd stop" ## HTTP interface to view status. set httpd port 80 and use address localhost # only accept connection from localhost allow localhost # allow localhost to connect to the server ## LOCAL: Check Apache binary: checksum, permissions, uid and gid. check file apache_bin with path /usr/local/apache2/bin/httpd if failed checksum and expect the sum 6036a2...bf7e3 then unmonitor if failed permission 755 then unmonitor if failed uid "root" then unmonitor if failed gid "wheel" then unmonitor alert security@your.site.com on { checksum, permission, uid, gid } with the mail-format { subject: Alarm! } group server ## LOCAL: Check that Apache is running and responsive. check process apache with pidfile /usr/local/apache2/logs/httpd.pid start program = "/usr/local/bin/web.server start" with timeout 10 seconds stop program = "/usr/local/bin/web.server stop" if cpu > 60% for 2 cycles then alert if cpu > 80% for 5 cycles then restart if totalmem > 200.0 MB for 5 cycles then restart if children > 250 then restart if disk read > 500 kb/s for 10 cycles then alert if disk write > 500 kb/s for 10 cycles then alert if failed port 80 protocol http then restart if 3 restarts within 5 cycles then unmonitor group system ## EOF ``` Season to taste and copy them to /usr/local/etc/monitrc, or wherever you've configured them to live. To do a syntax test: ``` root# chown root:wheel /usr/local/etc/monitrc root# chmod 600 /usr/local/etc/monitrc root# monit -t New Monit id: 3658d...64bab30 Stored in '/var/log/monit.id' Control file syntax OK me% head /var/log/monit.id /var/log/monitlog ==> /var/log/monit.id <== 3658d5ac25a00ca5aabb9a78464bab30 ==> /var/log/monitlog <== [EDT Aug 30 03:57:01] info : New Monit id: 3658d...64bab30 Stored in '/var/log/monit.id' ``` + Start monit at system boot + I use rc.local: ``` ... # Monit server -- depends on webserver already running. svr='/usr/local/bin/monit.server' if test -x "$svr" ; then $svr start /usr/bin/logger -t $tag monit.server fi # Track reboots. x=$(uname -rs | tr 'A-Z ' 'a-z-') /bin/date "+$x %Y-%m-%d %H:%M:%S %z" >> /var/log/reboot echo '.' /usr/bin/logger -t $tag rc.local finish ``` + Monit driver script + You can easily administer monit via CLI, but I like scripts for my services that allow consistent sub-commands. Copy the driver script somewhere you can find it, like /usr/local/bin. ``` NAME: monit.server SYNOPSIS: monit.server COMMAND DESCRIPTION: Script file to start and stop Monit services. COMMAND is one of the following: start -- starts Monit service stop -- stops Monit service check -- run syntax check for Monit control file help -- this message id -- print Monit unique ID reload -- reinitialise running Monit daemon restart -- stops and restarts Monit status -- displays status of Monit service summary -- displays service states HISTORY: Based on "Life with qmail" script and https://sosheskaz.github.io/tutorial/2017/03/28/FreeBSD-rcd-Setup.html ``` Sample //summary// output: ``` root# monit.server summary Monit 5.25.3 uptime: 22d 15h 19m Service Name Status Type hairball OK System cron OK Process sshd OK Process ntpd OK Process syslogd OK Process apache OK Process apache_bin OK File up: 7 (100.0%) down: 0 (0.0%) initialising: 0 (0.0%) unmonitored: 0 (0.0%) total: 7 services ``` The full monit.server script can be [found here monit.server]. + Building from source + ++ Download ++ I got the source for the version that works for my FreeBSD installation: ``` curl https://mmonit.com/monit/dist/monit-5.25.3.tar.gz curl https://mmonit.com/monit/dist/monit-5.25.3.tar.gz.sha256 ``` Unfortunately there's no GPG signature. ++ Build ++ Change to the source directory and do: ``` export CC=cc export CFLAGS=-O2 export CPPFLAGS=-I/usr/local/include export CXX=c++ export LDFLAGS=-L/usr/local/lib dest=/usr/local ./configure \ --prefix=$dest \ --mandir=$dest/man \ --with-zlib \ --with-piddir=/var/run make ``` My results: ``` Monit Build Information: Architecture: FREEBSD SSL include directory: /usr/include SSL library directory: /usr/lib Compiler flags: -Wno-address -Wno-pointer-sign -O2 -Wall -Wunused -Wno-unused-label -funsigned-char -D_GNU_SOURCE -std=c99 -D _REENTRANT -I/usr/include -I/usr/include Linker flags: -lpam -lz -lkvm -ldevstat -lpthread -lcrypt -L/usr/lib -lssl -lcrypto -L/usr/lib pid file location: /var/run Install directory: /usr/local ``` I ran make with no surprises. ++ Install and give it a whirl ++ Copy everything into place; //make install// should do this. If you rotate the files in /var/log, you may have to mess with your syslog/rsyslog setup: ``` root# /usr/local/bin/install monit /usr/local/bin root# /usr/local/bin/install -c 744 monit.server /usr/local/bin root# /usr/local/bin/install -m 644 monit.1 /usr/local/man/man1 root# cp /dev/null /var/log/monitlog ``` Copy your desired rules to /usr/local/etc/monitrc and set the permissions: ``` root# chown root:wheel /usr/local/etc/monitrc root# chmod 600 /usr/local/etc/monitrc ``` Check your configuration and see if she starts: ``` root# monit -t root# monit.server start root# monit.server summary root# monit.server status ``` Now kill cron or syslogd and watch it restart as if by magic. + Feedback + Feel free to send [comments mailto:vogelke+unix@pobox.com]. //Generated from [%%infile %%infile] by// //[txt2tags http://txt2tags.sourceforge.net]//{br} //$Revision: 1.7 $//{br} //$Date: 2026-10-23 19:19:54-04 $//{br} //$UUID: e4e05820-8645-38ee-b2c0-f55fd3aa0325 $//