μMon - minimal host monitoring

Create a dedicated unprivileged user:

# useradd -c "uMon" -d /var/umon -s /sbin/nologin -L daemon _umon
# mkdir /var/umon
# chown _umon:_umon /var/umon

Untar the umon archive to the home directory of this user:

# doas -u _umon /bin/sh -c "cd && tar xzf /path/to/umon.tar.gz"

Install dependencies:

# pkg_add rrdtool socat

Note: socat is only needed for running the fcgi as a standalone service; for production, usage of inetd (part of the base system, hence no need to install it) is recommended.

Create a copy of examples/*.conf to the main μMon directory (one level up). You do not necessarily need to make changes to all of these files, but you are encouraged to look at them. If you decide to make changes (now or later), having the upstream version under examples/ will protect you from overwriting your local config in case you untar an updated archive on top of your μMon instance.

Configure and enable snmpd:

# cp /etc/examples/snmpd.conf /etc/
# echo "listen on 127.0.0.1 snmpv2c" >> /etc/snmpd.conf
# echo "read-only community public" >> /etc/snmpd.conf
# rcctl enable snmpd
# rcctl start snmpd

Note: you are free (and encouraged) to use a different community string. Please set SNMP_COMMUNITY in umon.conf accordingly if you do so.

Verify that snmp queries work, output should be something similar to below:

$ snmp walk -v 2c -c public localhost ifDescr
ifDescr.1 = STRING: em0
ifDescr.2 = STRING: enc0
ifDescr.3 = STRING: lo0
ifDescr.4 = STRING: pflog0

By default, RRD databases will be created under db/, with consolidation functions to yield appropriate granularity on view durations ranging from the last 1 hour to the last 2 years. You can change these by editing umon.conf, but everything should be fine as is.

Configure the plugins by editing probes.conf. This should be straightforward, but you might need to check your system to know what the appropriate device names are.

Depending on which probes you have enabled, you might need some or all of the below privilege escalations added to /etc/doas.conf:

# Permit _umon to run query commands
permit nopass _umon as root cmd doveadm args who
permit nopass _umon as root cmd smtpctl args show stats

Now run probes/sample.sh manually and observe that it creates RRD databases without errors:

$ probes/sample.sh
Creating directory for RRD files: db
...
Creating db/load.rrd
Creating db/df.rrd
Creating db/cpu.rrd
Creating db/vmstat.rrd
...

Create a crontab entry for running sample.sh once every minute:

$ crontab -l
# min hr  dom mon dow command
*   *   *   *   *   /var/umon/probes/sample.sh 2>&1 >/dev/null

Configure httpd by adding this to /etc/httpd.conf:

# uMon - rrdtool-based monitoring
server "default" {
    listen on * port 8888
    location "/*" {
        fastcgi socket tcp 127.0.0.1 3333
    }
}

Feel free to change the listen address/port appropriately, e.g. listen on 127.0.0.1 instead of listen on * if you do not wish the μMon webpage to be accessible from the network. In such case, you can still access it by setting up an ssh tunnel. Or set up http basic auth so only people who know the credentials have access to it. Etc.

Make sure your changes take effect:

# rcctl reload httpd

Compile the fastCGI server:

make -C fcgi

This creates the fcgi/umon_fcgi binary executable from its C++ sources.

If you want to develop the fastCGI server program of μMon, it is convenient to run the server standalone so you can see the stdout in the console. You can do that by executing the wrapper script fcgi/standalone.sh. This requires socat to be installed. Note: to develop or change probes, graphs or views, you do not need to touch the fastCGI server, as it only invokes the corresponding shell scripts and is itself quite generic. Hence, you most probably want to deploy as "normal".

For normal deployment, you will want to set up inetd to invoke the fastCGI server via the fcgi/inetd.sh wrapper. This is convenient for development of graphs and views as well, in the sense that nothing has to be restarted after rebuilding the umon_fcgi executable; however, stderr will go to /dev/null. Use a config similar to this:

# cat /etc/inetd.conf
127.0.0.1:3333  stream  tcp     nowait  _umon     /var/umon/fcgi/inetd.sh

Make sure you enable and start inetd as appropriate:

# rcctl enable inetd
# rcctl start inetd

Configure the views you want to access by editing views.conf.

Navigate your browser to http://your.hostname:8888. The main view should load. (Tip: you can read this document by clicking on the μMon logo in the navbar!)

If you have a new, updated source archive umon.tar.gz, you can safely untar it on top of your existing installation:

doas -u _umon /bin/sh -c "cd && tar xzf /path/to/umon.tar.gz && make -C fcgi"

Your actual config files will not be overwritten. If there were changes made to the example config files, you might want to migrate some or all of those to your actual config (*.conf in the main μMon directory).

If there is a change to the RRD format (data series names) produced by a probe, reinitialize it by deleting the corresponding RRD file(s) under db/ and any probe state files probes/*.env produced by the probe (the base name should match the probe).

Create a dedicated unprivileged user:

useradd -s /usr/sbin/nologin -r -M -d /var/umon _umon
mkdir /var/umon
chown _umon:_umon /var/umon

Untar the umon archive to the home directory of this user:

# sudo -u _umon /bin/sh -c "cd && tar xzf /path/to/umon.tar.gz"

Install dependencies:

# apt-get install rrdtool socat

Note: socat is only needed for running the fcgi as a standalone service; for production, usage of xinetd is recommended.

Create a copy of examples/*.conf to the main μMon directory (one level up). You do not necessarily need to make changes to all of these files, but you are encouraged to look at them. If you decide to make changes (now or later), having the upstream version under examples/ will protect you from overwriting your local config in case you untar an updated archive on top of your μMon instance.

Install and configure Net-SNMP:

# apt-get install snmp snmpd snmp-mibs-downloader

Edit /etc/snmp/snmpd.conf and open up access to OIDs by adding a line such as:

view   systemonly  included   .1.3.6

Restart snmpd for the changes to take effect:

# systemctl restart snmpd

Note: you are free (and encouraged) to use a different community string. Please set SNMP_COMMUNITY in umon.conf accordingly if you do so.

Verify that snmp queries work, output should be something similar to below:

$ snmpwalk -O n -v 2c -c public localhost .1.3.6.1.2.1.31.1.1.1.1
.1.3.6.1.2.1.31.1.1.1.1.1 = STRING: "lo"
.1.3.6.1.2.1.31.1.1.1.1.2 = STRING: "ens3"

By default, RRD databases will be created under db/, with consolidation functions to yield appropriate granularity on view durations ranging from the last 1 hour to the last 2 years. You can change these by editing umon.conf, but everything should be fine as is.

Configure the plugins by editing probes.conf. This should be straightforward, but you might need to check your system to know what the appropriate device names are.

Depending on which probes you have enabled, you might need to delegate the privilege of running certain commands (as root) to the _umon user. On a stock Linux using sudo, just create /etc/sudoers.d/umon with some or all of the below lines:

# Permit _umon to run query commands
_umon   ALL=(root) NOPASSWD: /usr/bin/doveadm
_umon   ALL=(root) NOPASSWD: /usr/bin/smtpctl

Now run probes/sample.sh manually and observe that it creates RRD databases without errors:

# sudo -u _umon /bin/bash
$ cd
$ probes/sample.sh
Creating directory for RRD files: db
...
Creating db/load.rrd
Creating db/df.rrd
Creating db/cpu.rrd
Creating db/vmstat.rrd
...

Create a crontab entry for running sample.sh once every minute:

$ crontab -l
# min hr  dom mon dow command
*   *   *   *   *   /var/umon/probes/sample.sh 2>&1 >/dev/null

If you are collecting or tailing syslog messages as a matter of course, you might be annoyed by the nonsensical verbosity of cronjob logging, emitting not one but three entries per minute. To stop these useless entries from flooding your logs, create /etc/rsyslog.d/umon_block.conf with the below content:

if $msg contains "pam_unix(cron:session)" or $msg contains "_umon"
then {
    stop
}

Do not forget to restart the syslog daemon for this to take effect.

The below configuration example applies to nginx on the local host. If you use a different webserver or want to access the fastCGI socket from a different host, please adapt your config accordingly.

Merge this snippet into your enabled virtual hosts configs:

server {
    fastcgi_param  CONTENT_LENGTH     $content_length;
    fastcgi_param  CONTENT_TYPE       $content_type;
    fastcgi_param  DOCUMENT_URI       $document_uri;
    fastcgi_param  QUERY_STRING       $query_string;
    fastcgi_param  REQUEST_METHOD     $request_method;
    fastcgi_param  REQUEST_URI        $request_uri;

    listen 8888;
    location / {
        fastcgi_pass 127.0.0.1:3333;
    }
}

(If you do not have any webserver installed, just apt-get install nginx and add the above snippet to /etc/nginx/sites-enabled/default.)

Make sure your changes take effect:

systemctl reload nginx

Compile the fastCGI server:

make -C fcgi

Please see above in the equivalent OpenBSD section for a discussion on running the fastCGI server for development purposes.

For normal deployment, you will want to set up xinetd to invoke the fastCGI server via the fcgi/inetd.sh wrapper. This is convenient for development of graphs and views as well, in the sense that nothing has to be restarted after rebuilding the umon_fcgi executable; however, stderr will go to /dev/null.

If you do not yet have it installed, apt-get install xinetd. Then, create a file /etc/xinetd.d/umon_fcgi with the below content:

service umon_fcgi
{
        disable         = no
        type            = UNLISTED
        socket_type     = stream
        protocol        = tcp
        interface       = 127.0.0.1
        port            = 3333
        user            = _umon
        wait            = no
        server          = /var/umon/fcgi/inetd.sh
}

Don't forget to enable/start xinetd as appropriate:

# systemctl enable xinetd
# systemctl start xinetd

Configure the views you want to access by editing views.conf.

Navigate your browser to http://your-hostname:8888. The main view should load. (Tip: you can read this document by clicking on the μMon logo in the navbar!)

The same considerations apply as with OpenBSD (see above).

The equivalent update command to use:

sudo -u _umon /bin/sh -c "cd && tar xzf /path/to/umon.tar.gz && make -C fcgi"