Infrastructure

Upgrade to 3.0 is already scheduled and will happen shortly after FOSDEM. The only reasons why this wasn't done on Christmas days is because I didn't have time to thoroughly test the upgrade path, and accounts are easier to link (using our custom scripts) on ReviewDB (I want a grace period for those who missed the multiple announcements and are currently locked out). I'm testing the migration right now…

Upgrade to 3.1 is not scheduled yet. Might not happen in Q1, perhaps not even Q2 either.

What's the status of Java 11 support, by the way? AFAICT it works just fine on the stage 2.16 instance but I've seen https://bugs.chromium.org/p/gerrit/issues/detail?id=7843 and I'm reluctant to blindly upgrade JRE on the prod instance. The release notes for 2.14 and 2.15 read “Gerrit requires Java Runtime Environment (JRE) version 8; it is not compatible with JRE 9 or newer yet. For more information, see Issue 7843.”; release notes for 2.16 don't have an explicit minimum JRE version and doesn't mention Issue 7843.

[...]
and accounts are easier to link (using our custom scripts) on ReviewDB

Isn't accounts backend was unconditionally migrated to NoteDb in 2.15 release? So to link on 2.16.15 you would have to mutate NoteDb backend anyway. See this release notes:

https://www.gerritcodereview.com/2.15.html#release-highlights
[...]
Account data is stored in NoteDb.
[...]

What's the status of Java 11 support, by the way? [...]
For more information, see Issue 7843. [...]

It is fixed now, see: [1]. I updated the status of: [2].

Note though, that there were some issues reported, mostly
related to the JDK 11 and LDAP: [3], [4] and [5].

There is also this related mailing list thread: [6].

Also note, that we are building Gerrit on master using
JDK 11. That build would produce JDK 11 byte code and
thus JDK 11 required to run it: [7]. The relevant Bazel
options are:

$ bazel build \
  --host_javabase=@bazel_tools//tools/jdk:remote_jdk11 \
  --javabase=@bazel_tools//tools/jdk:remote_jdk11 \
  --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java11 \
  --java_toolchain=@bazel_tools//tools/jdk:toolchain_java11 \
  release

It would be highly appreciated, if more gerrit users would start using JDK 11.
What could possibly go wrong? In emergency case, admin could stop gerrit,
downgrade to JDK 8, restart gerrit, done.

Moreover, with this my pending change: [8], Gerrit master is even compatible
with JDK 13, and it should be trivial to adapt that change to 3.0.x release
line.

[1] https://gerrit.googlesource.com/gerrit.git/+/da8a2ad40293c0070502ae2818ff3685cd10cae6
[2] https://bugs.chromium.org/p/gerrit/issues/detail?id=7843
[3] https://bugs.chromium.org/p/gerrit/issues/detail?id=11567
[4] https://bugs.openjdk.java.net/browse/JDK-8217606
[5] https://bugs.openjdk.java.net/browse/JDK-8213202
[6] https://groups.google.com/d/topic/repo-discuss/zMQ9g8p22EU/discussion
[7] https://gerrit-ci.gerritforge.com/view/Gerrit/job/Gerrit-bazel-java11-master
[8] https://gerrit-review.googlesource.com/c/gerrit/+/243014

Isn't accounts backend was unconditionally migrated to NoteDb in 2.15 release? So to link on 2.16.15 you would have to mutate NoteDb backend anyway.

Sure, that's fine. I'm talking about the case when someone is locked out and create a new profile in SSO (with a different email address). Once they log to Gerrit they'll end up with a duplicate account. Reconciling accounts isn't hard in principle, but it becomes cumbersome when both have some changes attached to it. We have some SQL queries to process that case, and I'd rather avoid rewriting that to deal with NoteDB. Now that we have a single IdP we should see way less duplicates (in the past duplicate accounts were created when users logged in with an unusual/unlinked IdP).

It is fixed now, see: [1]. I updated the status of: [2].

Cool, thanks for confirming and spelling out the fixed versions.

It would be highly appreciated, if more gerrit users would start using JDK 11.

A scary warning can be found in 2.16.15 docs, and it's still present is 3.1.2's: “Java SE Runtime Environment version 1.8. Gerrit is not compatible with Java 9 or newer yet.“ — https://gerrit-documentation.storage.googleapis.com/Documentation/3.1.2/linux-quickstart.html#_before_you_start . Changing that line might help :-)

A scary warning can be found in 2.16.15 docs, and it's still present is 3.1.2's: “Java SE Runtime Environment version 1.8. Gerrit is not compatible with Java 9 or newer yet.“

Indeed. Done in: [1]. Let us see the reviewers reaction ;-)

[1] https://gerrit-review.googlesource.com/c/gerrit/+/249866

David Ostrovsky, do you know if that's more likely to cause problems with NoteDB than with ReviewDB?

I don't think so. I would guess, that those changes also cannot be reindexed with NoteDb backend, like it is already the case with ReviewDb backend. Given that those changes are broken anyway, one way forward is to remove them from the database, in that way, they will be entirely skipped/ignored during the migration process. Also in most recent Gerrit version there is new feature: delete change (Administrator permission is required to remove changes), but I think, that because those changes are broken, they cannot be probably cleanly removed in Gerrit from the UI as well.

Also FWIW the stage instance is now running 3.0, with rebased & rebuilt the custom plugins, etc.: https://gerrit-stage.documentfoundation.org

Great! Though, it would be trivial to also skip 3.0.x altogether and just jump directly to 3.1.x, in which case, Git wire protocol v2 can be activated, right?

Given that those changes are broken anyway, one way forward is to remove them from the database, in that way, they will be entirely skipped/ignored during the migration process.

OK thanks, I'll try that.

Though, it would be trivial to also skip 3.0.x altogether and just jump directly to 3.1.x, in which case, Git wire protocol v2 can be activated, right?

One thing at a time please, see msg#1. Any regressions on our instance will end up on my shoulders not yours. As an upstream dev you probably want people to upgrade often, but please accept that admins are more conservative about upgrades. And this is not FUD, despite your explicit blessing (“do this, do that, done”) and tests on the stage instance we did hit some regressions in the past. We don't have resources to follow gerrit's fast moving target anyway.

I have looking forward to wire protocol v2, by the way (learnt the hard way JGit didn't support it yet at the time, cf. https://lists.freedesktop.org/archives/libreoffice/2018-October/081249.html ). But I care about stability even more :-P. It'll come in due time, no need to rush.

Unrelated to this issue.

[...] Grimoire dashboard —

https://dashboard.documentfoundation.org/app/kibana#/dashboard/Gerrit [...]

Interesting. I asked my friends from GerritForge last year, to set up
analytics dashboard for LibreOffice projects (core and a couple of other
projects). The result is here: [1].

This pipeline to extract the analytics is 100% open source and consists of Gerrit analytics plugin, Spark ETL job and Elastic/Kibana. In this article the pipeline is described: [2].

[1] https://analytics.gerrithub.io/kibana/s/libreoffice/app/kibana#/dashboard/libreoffice-11
[2] https://gitenterprise.me/2019/02/07/gerrit-analytics-wizard-your-devops-analytics-in-one-click

Thanks for the link, wasn't aware we had external dashboards :-) Our dashboard is also entirely Free Software and ES/Kibana-based. It was first announced at https://blog.documentfoundation.org/blog/2017/08/02/dashboard-announcement/ , is at least 1 year older (was already in place when I started). The maintenance was handed over from Bitergia upstream to the infra team in 2018, cf. #1836.

Given that those changes are broken anyway, one way forward is to remove them from the database, in that way, they will be entirely skipped/ignored during the migration process.

OK thanks, I'll try that.

So I deleted these 45 broken changes on the stage ReviewDB (using DELETE FROM changes WHERE change_id IN ([…]);, so excluding comments and patchsets), and ran migrate-to-note-db again. This time it didn't produce any warning and visiting URLs such as /c/221 yields a 404 rather than 500 HTTP response code (hence no longer pollutes the error_log with large Java stacktraces). Thanks!

I'll do the same before migrating the production ReviewDB. The changes metadata won't be migrated to NoteDB, but their 101-57=44 known revisions will still reachable by ref. The metadata for these changes, which is currently visible using gerrit query change:$CHANGE (but broken in the web UI), will be lost though (beside SQL dumps). I guess that's fair.