I have a test that's gone 'red' multiple times today. I know this because I've seen it in the website and alerts have been sent to me. Yet if I look in ~hobbit/data/hist, ~hobbit/data/histlogs, or use the hsitory CGIs in the web GUI, it shows that it hasn't been 'red' today, rather that it's been green for >2 days. This isn't the case though, as It's gone 'red' 6 times today alone (2 are repeated e-mails, 2h delay).
from alert e-mails: bspdm07.edc.cingular.net:sgsn_procs red [55828] red Thu Mar 6 09:43:03 2008 bspdm07.edc.cingular.net:sgsn_procs red [271768] red Thu Mar 6 07:47:06 2008 bspdm07.edc.cingular.net:sgsn_procs red [271768] red Thu Mar 6 05:51:35 2008 bspdm07.edc.cingular.net:sgsn_procs red [811858] red Thu Mar 6 05:11:22 2008 bspdm07.edc.cingular.net:sgsn_procs red [811858] red Thu Mar 6 03:15:51 2008 bspdm07.edc.cingular.net:sgsn_procs red [694717] red Thu Mar 6 02:10:28 2008 bspdm07.edc.cingular.net:sgsn_procs red [738302] red Thu Mar 6 01:45:16 2008 bspdm07.edc.cingular.net:sgsn_procs red [648888] red Thu Mar 6 01:20:08 2008
yet from the logs: ~hobbit/data/hist:# tail bspdm07,edc,cingular,net.sgsn_procs Tue Feb 26 11:01:29 2008 red 1204052489 12385 Tue Feb 26 14:27:54 2008 green 1204064874 1506 Tue Feb 26 14:53:00 2008 red 1204066380 301 Tue Feb 26 14:58:01 2008 green 1204066681 549212 Mon Mar 3 23:31:33 2008 red 1204615893 1521 Mon Mar 3 23:56:54 2008 green 1204617414 592 Tue Mar 4 00:06:46 2008 red 1204618006 302 Tue Mar 4 00:11:48 2008 green 1204618308 302 Tue Mar 4 00:16:50 2008 red 1204618610 1206 Tue Mar 4 00:36:56 2008 green 1204619816
~hobbit/data/histlogs/bspdm07_edc_cingular_net/sgsn_procs:# ls -rt | tail Tue_Feb_26_11:01:29_2008 Tue_Feb_26_14:27:54_2008 Tue_Feb_26_14:53:00_2008 Tue_Feb_26_14:58:01_2008 Mon_Mar_3_23:31:33_2008 Mon_Mar_3_23:56:54_2008 Tue_Mar_4_00:06:46_2008 Tue_Mar_4_00:11:48_2008 Tue_Mar_4_00:16:50_2008 Tue_Mar_4_00:36:56_2008
So where did my reds go? ...
Also, it's not just this test, there are >10,000 tests doing this. Other tests seem to reflect red status fine, others don't. I see no rhyme or reason to it. :-(
stephen