Infrastructure

Gerrit 2.11.5 was released yesterday: [1].

• [1] https://groups.google.com/d/topic/repo-discuss/FUg_rcw9jks/discussion

Cloph wanted to have a look at the staging VM, so you and Norbert can
try out the upgrade path then - I'll check with Cloph during today's
team call on this item

There is one show stopper for the upgrade for OpenStack project: [1].

• [1] https://etherpad.openstack.org/p/gerrit-2.11-upgrade

some test-runs on testing VM:

#### update test procedure ####

# stop gerrit
sudo -u gerrit /srv/gerrit/bin/gerrit.sh stop
# clean cache and index dirs
# this might not be needed for production upgrade, but with that we're on the safe side
# and also have worst-case (time-wise) szenario since reindex run is necessary...
sudo -u gerrit rm -r /srv/gerrit/index/*
sudo -u gerrit rm /srv/gerrit/cache/*

# repositories copied from production to test-vm, copy to gerrit's dir
sudo -u gerrit rsync -ahvP --delete repositories/ /srv/repositories/

# read-in database dump from production matching the repositories' state
# sudo -u postgres pg_dump -c gerrit |gzip > ~/gerrit_dump.sql.gz
# → dump created with -c, so it clears existing data before import
zcat gerrit_dump.sql.gz |sudo -u postgres psql gerrit_test

# run new gerrit's init to update database schemes
sudo -u gerrit java -jar /srv/gerrit/bin/gerrit.war init --batch -d /srv/gerrit

# recreate index
# needs manually set database.poolLimit in gerrit.config- docs seems to be 
# wrong about the default?
# threads don't seem to help much - it is working on repositories in parrallel
# it seems, but almost all LO stuff is in the core-repo, load peaks at 3...
sudo -u gerrit java -jar /srv/gerrit/bin/gerrit.war reindex --threads 8 -d /srv/gerrit

Reindexing changes: projects: 100% (26/26), 94% (18702/19792), done    
Reindexed 18702 changes in 6242.0s (3.0/s)
6910.75user 290.54system 1:44:14elapsed 115%CPU (0avgtext+0avgdata 2956548maxresident)k
653872inputs+23538768outputs (0major+2260479minor)pagefaults 0swaps

# start gerrit
sudo -u gerrit /srv/gerrit/bin/gerrit.sh start

→ seems to work OK

→ ETA for downtime: 2h, + whatever it takes to create a backup :-)
  (enough diskspace available on production gerrit to duplicate the /opt/gerrit tree, so no problem)

dry testing on vm171 on excelsior
what I got so far:

new vm, one 200GB disk, 2 network (br0 and br1)
disk patition using lvm with
root
/var
/srv
swap

initial boot

force by the nanny-installer to create a dummy 'user'

finished the install normally.. selected only sudo and default utility tools (iow not graphical env)

rebooted

ssh as the dummy user, su to root
intall minion client
edit /etc/salt/minion.d/9999users.conf as usual
restart the minion
accept it on salt server

highstate on vm171
gran for a while.. after 30 minutes or so.. investigated.. nothing moving yet the command is still hanging... usual salt behavior
kill the command on the server (told us about going to run in background and the id of a magical job file that we can query 'later'
of course no such luck
rebooted vm171
re-run highstate.. this time it runs and repport a result.. mostly green except a weird message about fqdn 'not defined'
this turned out to be a pillar issue... for now that was only impacting the 'anti-virus email scanner.. so ignore

ssh to vm171 things are usable...
delete the dummy user that the installer forced us to create

note: highstate at this point only had the basic stuff like users and sudo....

complete the pillar data using vm178 as an example
add gerrit as a state then highstate

get the irker state files from vm162, add irkerd state to vm171
higstate failed with weird message... after hours wasted figured out the game with saltenv=develop

higstate now work

sudo to gerrit, create a ssh key (with passwd) for 'migration', add en entry in .ssh/config for gerrrit_prod to that effect.
ssh to gerrit_prod install that key as authorized key and put a real shell in /etc/passwd for gerrit's user

I tried to get git-daemon under systemd with no success so I used git-daemon-run package
with a tweak in /etc/sv/git-daemon/run to reflect user and repo's home

on vm171 rsync the git repo form gerrit prod gerrit_site/git to /srv/repositories

cd /srv/repositories
  rsync -avp gerrit_prod:gerrit_site/git/ .

fyi took: 36 minutes

retore the gerrit db
log on to vm171 as nthiebaud with -A
assign nthiebaud gerrit role and in general admin for gerrit db

ssh gerrit.libreoffice.org "pg_dump -Fc gerrit" | sudo -u postgres pg_restore -c -d gerrit

retore gerrit_site from prod (excluding git repos already synced)
on vm171 as gerrit

cd /srv/gerrit
  rsync -avpP --exclude 'git' gerrit_prod:gerrit_site .

that take about 4 minutes

update gerrit.conf to reflect the new position of the git repos

diff --git a/gerrit.config b/gerrit.config
  index af37556..4c9f2e9 100644
  --- a/gerrit.config
  +++ b/gerrit.config
  @ -1,5 +1,5 @
   [gerrit]
  -       basePath = git
  +       basePath = /srv/repositories
         canonicalWebUrl = https://gerrit.libreoffice.org/
         canonicalGitUrl = git://gerrit.libreoffice.org/
         changeScreen = CHANGE_SCREEN2

wrt to dns an ip... I'm thinking the best would be to:
make sure we have the ssh_host keys transfered
and then just swap the ip adressed.. iow change the ip of gerrit prod, give the ip to the new box when we are ready to go live. that way we donot even have to do the dns dance
and there is no risk of known_host issue...