Infrastructure

Unfortunately not :-/ I'm still using grafana for monitoring, which of course is highly suboptimal as it lacks notifications.

Unfortunately not :-/ I'm still using grafana for monitoring, which of
course is highly suboptimal as it lacks notifications.

Then let's address that early in Q4

A topic for the next admin call would be to agree on a method to deliver notifications
E-mail for sure, but a direct push message to the mobile as well would be very much desirable

I see three choices:

• use GMail's push feature -> drawback: everyone needs a @gmail.com address and I'm not sure if certain mails can create a direct alert
• use SMS notification -> drawback: small fee, limited amount of text; advantages: works everywhere even w/o data
• use PushOver -> drawback: license costs; advantages: works on Android, iOS and browser push notification
• use a Telegram bot -> no insight into that, but could be the least expensive solution; works on desktops too
• add your proposals here

I tried NotifiyMyAndroid which seems to work as well (but also has some license fee), however it was impossible for me to reach the vendor by any means, so I'd not pursue that further.

From the Nov. 21 infra call minutes:

• SMS is the way to go, but we need to the ability to specify schedules
  so not everyone is waken up in the middle of the night
• cloph: want to revive the telegram notifications that we had in the
  past (TDF Monitoring bot)
• Norbert: need the ability to temporarily disable the rules when doing
  manual maintenance
• Norbert: it's crucial to avoid false positives (cf. infra ML…)

Cool, thanks for your work on this!

Brett volunteered to start working on the alerting system on the recently setup prometheus based monitoring - so once we have some reliable/workable alerting rules in place, we can hook those up to email and irc as first step, and then expand to sipgate sms gateway.

Brett volunteered to start working on the alerting system on the
recently setup prometheus based monitoring - so once we have some
reliable/workable alerting rules in place, we can hook those up to email
and irc as first step, and then expand to sipgate sms gateway.

Sounds great! Both PushOver and the SMS gateway accept e-mail and HTTPS,
the latter probably be preferred.

Is there any ETA?
I really would like to get the cronjob mails cleaned up, and have some very basic notification system in place

Florian Effenberger wrote:

Is there any ETA?

Was part of Brett's ‘prometheus’ branch, it's being tested since early- to mid-September. We only have email notifications for now.

Was part of Brett's ‘prometheus’ branch, it's being tested since early-
to mid-September. We only have email notifications for now.

Where do these notifications get sent? :)

Florian Effenberger wrote:

Florian Effenberger wrote:

Was part of Brett's ‘prometheus’ branch, it's being tested since early-
to mid-September. We only have email notifications for now.

Where do these notifications get sent? :)

Ping? :)

They reach cloph and I at the moment. They're intentionally not being sent to the tdf-admin list as we want to keep monitoring and infra as separated as possible.

They reach cloph and I at the moment. They're intentionally not being
sent to the tdf-admin list as we want to keep monitoring and infra as
separated as possible.

Does it make sense to ask other volunteer admins if they want to be
connected to the notification system?