Alert based upon trend data
We want to be able to alert when a value changes dramatically over a period of time. so if a system returns 110,105,112,40,38,52
to eye ball that on a graph I can see there is a problem, however that may be at the peak time of the day and 38 might be a reasonable value for in the middle of the night, so having a threshold of a low value is not appropriate, the comparison mode of query for this situation would report a change of
-5,+7,-72,-2,+14
in here there is one large drop but then it is running stable, this means that it would not alert on a trigger that is aimed to say that there is a long term issue
The alert we are attempting to create from this is to say that our data rate has dropped by over 25% over the past 5 minutes and has stayed there, there are 8 separate metrics from our system which would greatly benefit from this, as we had a 11 hour issue yesterday where the issue wasn't noticed, until someone manually looked at the graph data