Infrastructure

Copying from today's #tdf-infra IRC log:

14:11 <@cloph> this is in progress re the monitoring, although no ETA as to when there's a public facing status page

I think this could be closed as this exists now: http://status.documentfoundation.org/

I beg to disagree. The status.documentfoundation.org is nice to have, but how do I check there if e.g. the TDF sysadmins are aware that dev-downloads is done (just an example, currently it's up & running)? Check e.g. status.github.com, you can see the output of automatic monitoring there + in case something is red, they post some "we're aware of it" type messages usually by the time you notice the breakage.

Maybe something for the dashboard, then.

Is this still valid/active? Will (or is) the dashboard public facing? Has a status site been implemented?

Jean Spiteri wrote:

Is this still valid/active? Will (or is) the dashboard public facing? Has a status site been implemented?

See these:
https://fosdem.org/2017/schedule/event/development_dashboard_deployment/
https://fosdem.org/2017/schedule/event/development_dashboard/

IMHO you're talking about http://status.documentfoundation.org ;-)

Florian Effenberger wrote:

IMHO you're talking about http://status.documentfoundation.org ;-)

See Miklos's comment 4.

Correct, the status.tdf blog is nice, but if there is some private automatic monitoring already, then having part of that publicly visible would be great. This way if e.g. gerrit is down, developers can just go to that status page, and in case it's red there, then they know sysadmins are aware of the problem. (It adds to the confidence if sysadmins can write some one-liner status text next to a red item, but it's a smaller thing.)

Thanks. :-)

unfortunately the status page is on github, and as the main infra guy doesn't have github account it is not really used for setting up maintenance info.

As for service status info, we now have prometheus blackbox monitoring, but we currently lack a way to publicly share info (we don't want to expose all metrics to the public) - while grafana (as visualization for the prometheus data) supports sharing dashboards, those are either static or would need account to restrict..

We (Brett) is working on configuring alerting for prometheus, so we can create notification and status pages automatically in case something goes wrong.

unfortunately the status page is on github, and as the main infra guy
doesn't have github account it is not really used for setting up
maintenance info.

Then we should fix that ASAP. If our sysadmin can't update the page it's
totally worthless.

Can we host it on the filoo machine maybe ourselves, or what is blocking
Guilhem to get a GitHub account otherwise? We should solve this ASAP.

status.tdf doesn't fit the requirement anyway. We need an API (to at least retrieve the status codes, possibly also some time metrics) to update the page in real time, and somewhere where admins can write down their findings, whether they're working on it, etc. Cf. the Feb 20 infra call minutes.

Alternatives include https://hund.io/ (demo page https://status.lineageos.org/) for a third-party (hosted) solution and https://cachethq.io/ (demo page https://demo.cachethq.io/) or https://www.netlify.com/status-pages/ for a free software solutions we can host ourselves. We didn't have time to test these alternatives yet, but I'd much rather have something we can host ourselves.

All fine for me, and I think things can be scaled and/or extended later on
What is crucial to me is that we inform people in time about issues and
that they have a reliable page they can look things up if something goes
wrong - whether automatically via API or manually edited HTML page :)

Thanks a lot! :-)

Good news indeed! :)