A while ago I started using mrtg, which is a nice (quite simple) graphing software for server load & such...
A while (2-3 server installs) later MRTG isn't quite fulfilling my needs anymore (it can only do integer graphing, and some other limitations), so I discovered CACTI. Cacti is pretty similar to MRTG, but more point&click (all setup is done through the web interface). It's lots more flexible, but also a bit more complicated.
Adding SNMP hosts, and tracking OID keys is pretty easy, graphing those values likewise.
However there was something I stumbled upon, which I couldn't easily fix (hence this blog entry):
in MRTG there was an option to select between a flat line graph or zero values graph. Both refer to graphing in case the data source is unavailable (e.g. host down).
I couldn't find such an option in Cacti, and later-on I learned that it doesn't need to exist, as Cacti (RRDTool actually) uses a scripting language which can do the same.
The language RRDTool uses is RPN (Reverse Polish Notation) and CDEF. There are at least 2 tutorials worth reading: CDEF tutorial and RPN tutorial.
Once I read those, I realized all I wanted to do is:
if the datasource value is unknown, use the previous value, else use the datasource value.
or in RPN notation: cdef=CURRENT_DATA_SOURCE,UN,PREV,CURRENT_DATA_SOURCE,IF