Infrastructure

Was this better before, i.e. do we know of faster downloads from that
machine(s)?

I'm not aware of any limitation enforced by them and wonder if it might
e.g. be related to our firewall setup, or the VMs having emulated 100
Mbit/s NICS?

Can we compare IPv4 and IPv6, if that makes a difference?

Do we have the same speed problem when downloading something e.g.
directly from falco via SSH?

Is it just outgoing traffic, or is downloading some external source e.g.
from falco also slow?

I tried downloading http://speedtest.netcologne.de/test_1gb.bin to
dauntless, and it yields about 20 MB per second, Hetzner gives twice as
much.

Downloading that from manitu to Hetzner only gives about 6 MB/s (I
didn't check the network saturation on dauntless though)

For
ftp://ftp.halifax.rwth-aachen.de/opensuse/distribution/13.2/iso/openSUSE-13.2-Rescue-CD-i686.iso
it looks even more different - Hetzner giving 70 MB per second, manitu
only 20

Downloading to vm168, only connected with 100 Mbit/s, gives nearly the
full rate of 10 MB/s.

Downloading from vm168 to pumbaa seems to give more than 6 MB/s, about 9
MB/s.

So either it's related to the big boxes shaping/switch, or a possible
firewall - right now it seems the regular web server gives faster
download than the big boxes

Can that be somehow related to the bridge setup?
As some last resort, we could take down one host that only runs test
stuff (if we have it), and see how fast things are in the rescue system

it is not related to the VM configuration
as mentioned in the initial description: downloading the bibisect repo from falco itself (where the VM runs on) yields 30MB/s - so the virtual network or configuration of the VM (disk, nginx,....) is not the bottleneck here.
Also even if you take the VMs out of the question and try to e.g. copy one of the virtual disk images to another host via the external network, you get the same effect (also mentioned in the initial description :-)

it is not a new problem
I always thought that networking on the manitu hostes VMs is slow - but back then I didn't know whether it was glusterfs-syncing or other stuff interfering with the network - I never was happy with the speed I got when working with the dev-downloads' bibisect repos

only outbound affected
inbound, i.e. from falco itself, there doesn't seem to be any artificial limits (way more than 10x the outbound speed)

can confirm that the vm168 host yields better rate at around 9MB/s

the point is that using multiple transfers, gives a higher overall rate.

I installed aria2 on pumbaa to demonstrate:

aria2c http://dev-downloads.libreoffice.org/bibisect/mac/Bibisect_MacOSX10.6%2b_release_lo-3.3.0_to_lo-4.1.tar.bz2

aria2c -x 3 http://dev-downloads.libreoffice.org/bibisect/mac/Bibisect_MacOSX10.6%2b_release_lo-3.3.0_to_lo-4.1.tar.bz2 # (allow up to 3 connections per server)

If at all, we could use dauntless for a test, as there we only need to wait until a testrun is complete, and then can take it down for a short while.

Thanks for the summary!
I'm on the road the next two days, but using dauntless (ideally during
manitu's office hours, so tomorrow from ~8/9 to ~1700) for a test drive
sounds worthwhile I guess, just so we can rule out any local setup issue

Firewall doesn't seem to run on the hosts. By testing via the rescue
system we could gather new details (hopefully)

curl http://ftp-stud.hs-esslingen.de/pub/Mirrors/Mageia/iso/4.1/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD.iso >/dev/null

pumbaa:
Average Dload: 85.2M (→ hs-esslingen has plenty of bw/is not limiting factor)
dauntless:
Average Dload: 30.6M (→ kind of disappointing in comparison, but still OK)

curl http://ftp.snt.utwente.nl/pub/linux/mageia/iso/4.1/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD.iso >/dev/null

dauntless
Average Dload: 8108k (→ pathetic, not even close to what pumbaa gets, and not even close to the 30M with the other mirror, the "upper limit")

dauntless (rescue system)
Average Dload: (esslingen): 30.9M (→ within margin of error)
(utwetnte): 7314k (→ even worse, but routing/... to blame - still order of magnitude lower BW than from hetzner Server)

Other tests
rsync -ahP root@dauntless:mageia.iso ./ (on pumbaa) 15.06M bytes/sec (meh)
(on excelsior) 40.77M bytes/sec (yay)

so manitu-internal networking is not throttled, but outgoing to pumbaa/Hetzner is not even half that speed.. Network interfaces on both the manitu hosts could easily handle > 30MB/s
→ it is not the boxes themselves where the throttling occurs, it is in the following hops where bandwidth is lost.

dauntless (bridge/regular):
on excelsior: 36.84MB/s (hmm - quite some impact, but still OK, maybe excelsior was busier at the time)
on pumbaa: 5.14MB/s (wow, only third of what you get with rescue system....)

→ bridged networking has some negative impact it seems, but the difference inbound/outbound is rather large. Also difference in inbound traffic with different servers is kind of unsettling.

Either manitu's networking is saturated, or traffic-shaping is applied. Either of the two is bad.

combined networking: (i.e. multiple connections) test in /run/shm - to avoid limits imposed by disk-io/similar):

aria2c http://ftp5.gwdg.de/pub/linux/mageia/iso/4.1/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD.iso http://ftp-stud.hs-esslingen.de/pub/Mirrors/Mageia/iso/4.1/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD.iso http://ftp.uni-erlangen.de/mirrors/Mageia/iso/4.1/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD/Mageia-4.1-LiveDVD-GNOME-x86_64-DVD.iso

I.e. using three fast mirrors simultaneously → only 33.8MiB/s from dauntless (77,7MiB/s from pumbaa - less than with just esslingen mirror, as the other have lower per-connection limits, but all allow for >50MB/s individually)

Summary
Inbound bandwidth is capped at around 30MiB/s
outbound bandwidth doesn't even get close to that, fluctuating quite a bit, but on average around 5MB/s
As transfers between manitu hosts (still using the external network) reach the 30MiB/s, it is unlikely that the host's own setup is to blame, but rather manitu's outbound rules throttle the transfers.
transfer to pumbaa while in rescue system was significantly faster (with around 15MB/s), but still far from what you'd expect from a 1GB link that is otherwise idle.
Transfer is not only slow to hetzner-based servers, but also e.g. to gimli or to amazon ec2 hosts (that's where I first noticed slow outbound traffic - uploading tinderbox builds to gimli from VMs on manitu server or downloading bibisect repos to ec2 from VM on manitu) - gimli can easily handle >20MB/s from e.g. pumbaa, but from manitu (no matter whether host or vms running on the hosts) you get ~5MB/s only)

Hm, that's quite odd indeed, and we should raise this with manitu support
Thanks for checking this!
Alex, maybe let's have a quick chat first before approaching manitu

This may also be related to the random monitoring plugin timeouts - no other reason could be found by me so far, so it might be related to generic connection problems.

Some ping statistics can now be found here: https://monitoring.documentfoundation.org/smokeping/smokeping.cgi?target=manitu

Setting connection speed to 100MBit to check for driver/kernel problems.

Thanks for testing! Is that also true for the regular root servers
(galaxy, website), or is the speed constant there?

Any chance to test this also with falco and excelsior, to see if they suffer the same issues, for the big picture?

First update dauntless to debian 8.0

Notes from last infra call:

* Inspect packages from machine-machine and machine-internet
* Get stats from kernel
* Test if problem persists after upgrade to Debian 8 on dauntless

related to #995
when we have feedback, plan is to replace WAN card(s) and see how things go; most likely a switch of the NIC will help