This graph is mostly useless:

It tells us that there is a huge spike in “load average” at around 6am. It fails (because of the y-axis scale) to give us any meaningful information about the other 20 hours of the day.If we look at the (just as useless) weekly graphs, we see this spike occurs every weekday about the same time.

As it turns out, we know what this is. And it’s not a problem. The machine in question sends out a lot of email to a lot of people once per day. Having generated all the emails and passed them to exim, it then tells exim to start a bunch of extra “queue managers”. Essentially this is like saying “quick, we need half a dozen more temporary postmen to deliver all the mail in the bag”.What we want is to be able to zoom in, and reduce the y-axis scale from 20 to 2. Munin’s guess about the correct scale is out by an order of magnitude. Fail.

How to Fix Munin’s Y-Axis Scale

First, login to the machine you are monitoring. NOT the machine that generates the graphs. This is one of the quirkier aspects of Munin. The machines you monitor are responsible for setting how the graphs are drawn. It sort of makes sense - different monitored machines may have different graphing requirements, but frankly we still reckon this is confusing at best, and probably wrong.Then, find your plugin. On Debian distro’s, you can start at:

/etc/munin/plugins/

and then follow the symlinks, probably to

/usr/share/munin/plugins/

those are likely to be little perl (or bash) scripts. Edit the one you are interested in:

sudo vi /usr/share/munin/plugins/load

And make this change:

< echo 'graph_args --base 1000 -l 0'
---
> echo 'graph_args --base 1000 -l 0 -u 2 --rigid'

The figure 2 here is what we want the maxium y-axis value to be. Our graph will now show y-axis values from 0 through to 2.

Logarithmic?

If you are a maths nerd, you might be screaming Logarithmic. Firstly, we’re not sure the results logarithmic graphs produce are actually better. Second, the munin wiki says:

—logarithmic: … Should almost never be used.

No, we’re not sure why either.[Update: Nicolai has updated the Munin wiki to further clarify that while logarithmic is generally not a good idea, in some cases it can help.]

Anyway, here’s a graph from a few days later, but the underlying data pattern matches that used for the linear graphs on this page.

Not bad, although it’s a shame the y-axis doesn’t have a better numerical scale.The problem with logarithmic graphs is when all your data has small values. Logarithmic will work best when the values cover a large range (1 to 100 say). For load average, this is is when the values are typically well below 1. For us, this is most of the weekend:

The change to

/usr/share/munin/plugins/load

to get the above two logarithmic graphs was:

< echo 'graph_args --base 1000 -l 0'
---
> echo 'graph_args --lower-limit 0.01 --logarithmic'

A couple of notes.

• The date on the logarithmic graph above is a few days later than the others. We added the bits about logarithmic graphing later, but the raw data for the period was very similar.
• The weekly, monthly and yearly graphs do improve a little in logarithmic fashion, only you can’t configure munin to use different graph_args for the daily vs the other graphs. All for one! Plus, we’re not convinced even the weekly logarithmic graphs are better than simply fixing the y-axis and keeping them linear.
• If you specify —logarithmic, you have to also specify —lower-limit. And further, the value for “—lower-limit” must be greater than 0. Or so /var/log/munin/munin-graph.log tells you, if you don’t specify it.

Time for Tea

Because of the way Munin works, you now have to wait a few minutes to see what your change is going to do. The machine that collects the data and generates the graphs will poll (by default) every five minutes. It’ll re-generate the daily graph after each poll, so within the next five minutes your daily graph should be regenerated using the new fixed scale:

The weekly, monthly and yearly graphs update even less often. There is no point regenerating the yearly graph every five minutes, because you wouldn’t typically see any new data.Vítězslav Kotrla pointed out that you can force the regeneration of all the graphs by simply deleting them and waiting for the next update:

# your web docroot may be different
# this is the default debian/ubuntu path

cd /var/www/munin/foo/
# replace 'hostname' and 'graph_type' as appropriate
sudo rm hostname-graph_type*.png

OR, Ståle Kristoffersen pointed out that you can force the regeneration of all the graphs right now using the following command:

munin-graph --force --nolazy

On Debian based systems, like Ubuntu, you’ll find munin-graph at

/usr/share/munin/munin-graph

If anyone knows a better way to do this, please let us knowThe change we’ve applied overrides Munin’s intelligent, adaptive graph scaling. It means that, once they are regenerated, the four different graph periods (daily, weekly, monthly and yearly) will all use the new fixed scale.

Can I go backwards?

Happily, the change we’ve made only alters how the graphs are displayed. It doesn’t alter the data we collect.So if you decide you’ve got the scaling wrong, or just want to flip back and let Munin work out the scale for you, simply un-do the change you made to the plugin. Wait five minutes, and your daily graph will have regenerated using the y-axis scale that Munin works out for you.

See Also

Munin Wiki page about graph_args. Note what it says about the —logarithmic switch

“Should almost never be used”