Project

General

Profile

Task #1976

Task #1316: Gluster setup

berta & antares: shares from local gluster server not mounted at boot (attempt to mount before glulsterfs-server is fully started)

Added by Christian Lohmaier over 1 year ago. Updated 4 months ago.

Status:
In Progress
Priority:
Low
Category:
Backups
Target version:
Team - Pool
Start date:
Due date:
% Done:

0%

Estimated time:
Tags:
URL:

Description

This affects only berta (and antares), as those are the only hosts that have a gluster volume that is only provided by the local machine. For antares it doesn't matter if volumes aren't mounted at reboot, but for berta it is critical that the backup-berta volume is mounted for rsnapshot.

I tired to add order in the mounts by adding corresponding systemd unit files to add dependencies, but this unfortunately not enough, the mount still is attempted too early.

what is needed nevertheless is to have glusterfs-server depend not only on network (the systemd default target), but networking (the init.d job/service) as otherwise dhcp is not done yet, and gluster might fail to resolve berta.tdf/the other gluster peers:

  • /etc/systemd/system/glusterfs-server.service.d/require_nw-online.conf
    [Unit]
    Wants=networking.service
    After=networking.service
    

This ensures that gluster is started after the internal network is brought up

Failed attemts to ensure the volume mounts succeed:

  • adding x-systemd.requires=glusterfs-server.service to options in /etc/fstab - this seems to be ignored completely by debian 8
  • adding /etc/systemd/system/srv-fileshare-mnt.mount with following content
    [Unit]
    Description=Loads the local fileshare volume
    Wants=glusterfs-server.service
    After=glusterfs-server.service basic.target
    
    [Mount]
    What=antares.tdf:fileshare-antares
    Where=/srv/fileshare/mnt
    Type=glusterfs
    
    [Install]
    WantedBy=multi-user.target
    

    while enabling that unit correctly attempts to mount after the glusterfs-server job is started, it is still too early, gluster isn't done with its initializing and the mount fails. Problem might be that the glusterfs-server is not a native systemd unit, but only a sysvinit one...

So easy workaround would be to add a @reboot cronjob with a sleep.... but of course still remains the chance of having a rsnapshot job triggered before the backup-berta volume is mounted (didn't try what happens when the /srv/rsnapshot symlink points to non-available dir and rsnapshot is run)

History

#1 Updated by Alexander Werner over 1 year ago

  • Status changed from New to Feedback

Another workaround would be to mount using another gluster server, the connection afterwards would still be local, only the management connection in the beginning is then done using the different server.

#2 Updated by Florian Effenberger over 1 year ago

  • Due date set to 2016-07-31
  • Priority changed from Normal to High
  • Target version set to Q3/2016

This should be fixed before end-July - any status update already?

#3 Updated by Florian Effenberger over 1 year ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Alexander Werner to Christian Lohmaier

#4 Updated by Florian Effenberger over 1 year ago

  • Priority changed from High to Normal

#5 Updated by Florian Effenberger about 1 year ago

Florian Effenberger wrote:

This should be fixed before end-July - any status update already?

Ping? ;-)

#6 Updated by Florian Effenberger about 1 year ago

  • Assignee changed from Christian Lohmaier to Guilhem Moulin
  • Target version changed from Q3/2016 to Pool

Assinging to Guilhem
From what I recall, it's not that urgent - maybe you can quickly sync with Cloph about the impacts/caveats so we can prioritize and plan accordingly?
Adding to pool for the moment, as it doesn't look that super-urgent (anymore)

#7 Updated by Florian Effenberger 12 months ago

  • Due date deleted (2016-07-31)

#8 Updated by Florian Effenberger 10 months ago

Is that still of relevance?

#9 Updated by Florian Effenberger 8 months ago

Florian Effenberger wrote:

Is that still of relevance?

Ping?

#10 Updated by Guilhem Moulin 8 months ago

Duno, didn't reboot berta recently :-/ I'll come back to it next time we reboot the box

#11 Updated by Florian Effenberger 5 months ago

Guilhem Moulin wrote:

Duno, didn't reboot berta recently :-/ I'll come back to it next time we reboot the box

Any updates? IMHO there were no reboots yet, but asking so we can update the ticket accordingly, as it's been open for quite a while

#12 Updated by Guilhem Moulin 4 months ago

  • Priority changed from Normal to Low

Last time we rebooted berta (june 18) we indeed had to manually mount the gluster fileshare. IMHO not a big deal as long as it doesn't interrupt the boot process and drops to a rescue shell. rsnapshot will complain loud enough (through cron mails) that the mountpoints don't exist ;-)

I'll see if I can adapt cloph's systemd.mount(5) suggested unit file next time we reboot the box, but I'm lowering the priority in the meantime.

Also available in: Atom PDF