Infrastructure

One reason to update:

Gerrit 2.11 still uses Java 7, and as such uses weak cryptographic standards.

See this discussion on Gerrit mailing list:

https://groups.google.com/d/topic/repo-discuss/AMVDwTIeDJw/discussion

And this removal notice from GitHub dev team:

https://githubengineering.com/crypto-removal-notice/

I cherry-picked JSch upgrades from latest gerrit release to 2.11 release line,
and uploaded this series upstream under this topic:

https://gerrit-review.googlesource.com/#/q/topic:gh-crypto-removal-notice

Release artifact can be fetched from GerritForge CI:

https://gerrit-ci.gerritforge.com/job/Gerrit-verifier-buck/2224/artifact/gerrit/buck-out/gen/gerrit.war

or from my server:

http://ostrovsky.org/gerrit/release-2.11.10-5-g276a209ff4.war

David Ostrovsky wrote:

I cherry-picked JSch upgrades from latest gerrit release to 2.11 release line,
and uploaded this series upstream under this topic:

https://gerrit-review.googlesource.com/#/q/topic:gh-crypto-removal-notice

Thanks, much appreciated! I picked up these and that indeed fixed the broken replication issue.

We should upgrade to 2.13.11, see this release notes: [1].

[1] https://groups.google.com/d/topic/repo-discuss/aoQ2cZ_e7iI/discussion

I talked to David on the phone about this, there are actually three Gerrit-related topics:

• make Jenkins comments visible in Gerrit by giving Jenkins user proper rights, and some more user rights to other users -> we should look into that next week
• staging server -> we need to see which data needs to be omitted. Not super urgent, but sometime during spring would be nice
• migration -> not super urgent either (given Q1 and Q2 has quite some high-prio topics for the team), summer/autumn would be fine for David, the earlier the btter of course

In general, we should also share responsibilities on Gerrit, so not everything is on one table.

We will discuss this in the next team call.

Florian Effenberger wrote:

• staging server -> we need to see which data needs to be omitted. Not super urgent, but sometime during spring would be nice

Yeah that's what this task is about with the aim of doing it in Q1.

In general, we should also share responsibilities on Gerrit, so not everything is on one table.

Yup :-)

Any update already?

Just to flag this: We should look into this no later than early June, when the most pushing GDPR bits are done. This also matches our promise to do it during springtime, which I'd like to keep :-)

Thanks for update, Florian.

Do we have a plan B, in case we also fail Q2 deadline?

I actually expect that deadline is met - it's 6 weeks from now which
should give plenty of time...

I actually expect that deadline is met - it's 6 weeks from now which
should give plenty of time...

Great, looking forward to help out with testing,
once staging instance is up and running.

I'm seeing also one inaccuracy in the issue description:

Q: according to my notes 2.11.8 was released on 2016-03-09 and 2.13.9 on 2017-07-03?

The relevant date is frozen code base, which is accounted from the releasing
of first release line.

So the announcement and thus release date of Gerrit 2.11 is from: 17.04.2015: [1],
3 years ago. The minor fixes in Gerrit never contain new features, only security
vulnerabilities and small error fixes. Major inline edit improvements and support for
new inline edit use cases, were only added in Gerrit 2.12 and 2.13.

As example, 2.11.11 was released on 27.02.2018: [2], but this of course
doesn't change the fact the 2.11 line is three year old and very outdated version.

At the time of this writing the current release is 2.15.1 and 2.15.2 is going to
be released in the next weeks.

[1] https://groups.google.com/d/topic/repo-discuss/hPbetnfBkm4/discussion
[2] https://groups.google.com/d/topic/repo-discuss/NqSp7MJ11Cs/discussion

A stage instance (Debian 9 baseline, gerrit 2.13.11) is now available at https://vm178.documentfoundation.org with a subset of projects. OAuth2 and OpenID authentication are both working, but the maintenance scripts (account merging etc.) haven't been tested yet.

Guilhem Moulin wrote:

OAuth2 and OpenID authentication are both working

I should have added that they're not working out of the box: for OAuth2 one needs to fix the URL (twice), and for OpenID one needs to temporary run git config -f ~gerrit/etc/gerrit.config gerrit.canonicalWebUrl https://vm178.documentfoundation.org/ (OAuth2 breaks if it's not https://gerrit.libreoffice.org/ so that's what it's set to normally). So while it's a bit cumbersome it's not impossible. And then one can use the site as usual.

Guilhem Moulin wrote:

Guilhem Moulin wrote:

OAuth2 and OpenID authentication are both working

I should have added that they're not working out of the box: for OAuth2 one needs to fix the URL (twice), and for OpenID one needs to temporary run git config -f ~gerrit/etc/gerrit.config gerrit.canonicalWebUrl https://vm178.documentfoundation.org/ (OAuth2 breaks if it's not https://gerrit.libreoffice.org/ so that's what it's set to normally). So while it's a bit cumbersome it's not impossible. And then one can use the site as usual.

In the past we used this URL: gerrit-test.libreoffice.org to simplify this set up. Do we have that option now and can set it up that way?

David Ostrovsky wrote:

In the past we used this URL: gerrit-test.libreoffice.org to simplify this set up. Do we have that option now and can set it up that way?

I fail to see how that would help with respect to authentication. As long as the domain isn't gerrit.libreoffice.org (or a domain enabled by the OAuth2 provider) one will need to trick the system to get in.

So why having a less generic domain name? It's fairly usual to use vmXYZ.documentfoundation.org during the testing phase.

I was trying to logn with my usual account (OpenID) that I used on production site, and git this error after successful redirect from my OpenID provider (Launchpad):

https://gerrit.libreoffice.org/#SignInFailure,SIGN_IN,Direct+signature+verification+failed.
  The page you requested was not found, or you do not have permission to view this page.

Quoting myself:

for OpenID one needs to temporary run git config -f ~gerrit/etc/gerrit.config gerrit.canonicalWebUrl https://vm178.documentfoundation.org/ (OAuth2 breaks if it's not https://gerrit.libreoffice.org/ so that's what it's set to normally).

We were not able to make OAuth2 and OpenID work simultaneously. I just did the above, meaning OpenID works while OAuth2 no longer does.

Guilhem Moulin wrote:

Quoting myself:

for OpenID one needs to temporary run git config -f ~gerrit/etc/gerrit.config gerrit.canonicalWebUrl https://vm178.documentfoundation.org/ (OAuth2 breaks if it's not https://gerrit.libreoffice.org/ so that's what it's set to normally).

We were not able to make OAuth2 and OpenID work simultaneously. I just did the above, meaning OpenID works while OAuth2 no longer does.

Thanks. I was able to login now. And my old account was preserved, with Account ID 1000020, and that is fine.

Second thing I tried is to create online change (from browser). It didn't work. I got this error:

Code Review - Error
  500 Internal server error

Can you please check the server log, what is going on there?

David Ostrovsky wrote:

Can you please check the server log, what is going on there?

[2018-06-13 21:29:31,209] [HTTP-77] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /changes/
java.io.IOException: Permission denied
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.createTempFile(File.java:2024)

Too tight ACLs? Does it try to create the temporary file in /tmp rather than $gerrit_site/tmp? I just configured the service file to mount a private /tmp a dedicated FS namespace. Should help if that's indeed the case.

Guilhem Moulin wrote:

David Ostrovsky wrote:

Can you please check the server log, what is going on there?

[...]

Too tight ACLs? Does it try to create the temporary file in /tmp rather than $gerrit_site/tmp? I just configured the service file to mount a private /tmp a dedicated FS namespace. Should help if that's indeed the case.

Nothing changed, I'm still getting: "500 Internal server error".

I guess, that the gerrit user doesn't have the ACL to right into git repository, under $gerrit_site/git. I'm trying to create change for Lode project.

David Ostrovsky wrote:

I guess, that the gerrit user doesn't have the ACL to right into git repository, under $gerrit_site/git. I'm trying to create change for Lode project.

Good guess, …/lode.git was copied without preserving the ownership/group, I just chmod'ed it. Other repositories had proper ownership (gerrit:gerrit) already.

Guilhem Moulin wrote:

David Ostrovsky wrote:

I guess, that the gerrit user doesn't have the ACL to right into git repository, under $gerrit_site/git. I'm trying to create change for Lode project.

Good guess, …/lode.git was copied without preserving the ownership/group, I just chmod'ed it. Other repositories had proper ownership (gerrit:gerrit) already.

Thanks for the quick fix! It looks good now, I was able to create change online: [1].
And all new online edit features, that I contributed back in 2015 and waiting for years,
are there: [2].

What is the ETA to bump the production version to 2.13.11?

[1] https://vm178.documentfoundation.org/#/c/38878
[2] https://imgur.com/a/RCsbqVW

Quoting floeff on comment #8:

• migration -> not super urgent either (given Q1 and Q2 has quite some high-prio topics for the team), summer/autumn would be fine for David, the earlier the btter of course

Time-wise the migration also needs to be suitable from a release engineering perspective; I assume July/August would be ideal, do you agree Christian Lohmaier? That leaves time for us to test the migration scripts.

By the way, one thing we did not do in the staging instance, was to migrate the H2 database for account patch reviews to PostgreSQL: https://www.gerritcodereview.com/releases/2.13.md#manual-schema-update-for-reviewed-flags . Would you say there is a good reason to do that, David?

Guilhem Moulin wrote:

Quoting floeff on comment #8:

• migration -> not super urgent either (given Q1 and Q2 has quite some high-prio topics for the team), summer/autumn would be fine for David, the earlier the btter of course

Time-wise the migration also needs to be suitable from a release engineering perspective; I assume July/August would be ideal, do you agree Christian Lohmaier? That leaves time for us to test the migration scripts.

SGTM.

By the way, one thing we did not do in the staging instance, was to migrate the H2 database for account patch reviews to PostgreSQL: https://www.gerritcodereview.com/releases/2.13.md#manual-schema-update-for-reviewed-flags . Would you say there is a good reason to do that, David?

It depends on how much records we do have there. On bis sites tehre are millions of rows and H2 database doesn't handle this very fast. Also note, that this migration can be done later. We wrote migration site program that can be used, so it must not be done during the migration. If you decide to do that, note though, that it must be a new database that only includes this single table. Main ReviewDb database cannot be used for this purpose.

Closing this, as we upgraded the prod instance to 2.13.11 last night.

To double check:
This means that the planned upgrade for August, post-6.1, is already
done, or is there something pending?

Florian Effenberger wrote:

This means that the planned upgrade for August, post-6.1, is already
done, or is there something pending?

The latter; issue closed hence nothing pending :-) In the Jun 19 infra call cloph said he'd rather have the upgrade before RC1, hence the deadline revision.

This means that the planned upgrade for August, post-6.1, is already
done, or is there something pending?

The latter; issue closed hence nothing pending :-) In the Jun 19 infra
call cloph said he'd rather have the upgrade before RC1, hence the
deadline revision.

I'm confused now ;-)
The latter means "something pending" - I guess you meant all is done,
ticket can be closed, right?

Florian Effenberger wrote:

This means that the planned upgrade for August, post-6.1, is already
done, or is there something pending?

The latter; issue closed hence nothing pending :-) In the Jun 19 infra
call cloph said he'd rather have the upgrade before RC1, hence the
deadline revision.

I'm confused now ;-)
The latter means "something pending" - I guess you meant all is done,
ticket can be closed, right?

Oops yeah sorry, I meant “the former”: everything in this ticket has been addressed already (there is nothing pending), and the ticket itself is closed already :-)

Great, thanks a lo! :9

Great, thanks a lo! :9

Great, thanks a lot! ;-)
is what I meant... :)