|
Ticket ID: SIXXS #9483366 Ticket Status: Invalid PoP: gblon03 - Gyron Internet LTD - Limited UK Company (London)
Packet loss, routes flapping
![[gb]](/s/countries/gb.gif) Shadow Hawkins on Friday, 31 May 2013 13:57:36
Hi,
Since 2013-05-31 12:00 UTC+0100, some IPv6 routes to/from Gyron are flapping. I'm struggling to measure/locate any packet loss; it's as though the link abruptly goes down.
An example is between 2a01:4f8:a0:9000::1 and gblon03.sixxs.net, which is flapping between he.net and tinet.net transit every few minutes, with a few seconds of loss at every switchover.
Perhaps related, from 2013-05-30 12:00 UTC+0100, some IPv4 and IPv6 routes to/from Gyron began showing ~4% packet loss.
My own tunnel endpoint's (217.155.40.118, Zen Internet) route to gblon03.sixxs.net was initially affected by this. But a routing change at around 17:38 that day cured this.
Below is the *unaffected* route. Unfortunately I have no record of the prior route where packet loss was observed:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. xenolith.pyro.eu.org 0.0% 101 0.5 0.4 0.1 2.4 0.4
2. losubs.subs.dsl2.wh-man.zen.net.uk 0.0% 101 11.0 11.0 10.4 15.8 0.6
3. ge-2-1-0-164.cr1.wh-man.zen.co.uk 0.0% 100 11.0 14.4 10.4 70.3 11.1
4. ge-3-0-0-0.cr2.th-lon.zen.net.uk 0.0% 100 17.9 20.3 17.1 38.1 4.4
5. xe-0-0-3.border-1.sov.lon.uk.as29017.net 0.0% 100 17.7 20.3 17.4 60.6 7.9
6. xe-0-0-1.border-1.thn.lon.uk.as29017.net 0.0% 100 17.8 19.9 17.4 64.1 7.7
7. xe-1-2-2.core-1.centro.hml.uk.as29017.net 0.0% 100 18.8 20.5 18.1 63.6 6.8
8. gblon03.sixxs.net 0.0% 100 18.5 18.5 18.1 19.6 0.3
Thanks.
Packet loss, routes flapping
Shadow Hawkins on Friday, 31 May 2013 14:02:32
Smokeping graphs illustrate two IPv6 routes out of gblon03.sixxs.net flapping (2013-05-31 times are UTC+0100) and the loss at each changeover:
http://pyro.eu.org/f/5xrQJlcGSOu88opClc46JA.png
http://pyro.eu.org/f/FprOCNTcRJq7anB2vO5g-g.png
Packet loss, routes flapping
You are pinging over IPv6, which goes over a tunnel from your endpoint to the PoP.
As such, any effect in IPv4 will carry over to IPv6.
While pings are great to see that something might be wrong, it does not show where.
Please actually provide proper details.
Note that IP addresses are much more useful than hostnames (which in some cases do not map back to addresses as there is no AAAA available or because it is the wrong AAAA record for that name).
Packet loss, routes flapping
Shadow Hawkins on Friday, 31 May 2013 17:25:30
This problem cleared at approx. 17:40 UTC+0100. It was possibly related to Gyron router "core-2.centro.hml" or one of its links.
I know my report was incomplete: I had been so far unable to represent the issue clearly on an IPv6 traceroute, especially since the routes were changing frequently.
I did show a clean IPv4 traceroute between tunnel endpoints to rule that out as being a factor.
Packet loss, routes flapping
It was possibly related to Gyron router "core-2.centro.hml" or one of its links.
Why do you think that, do you have any more details about this?
, especially since the routes were changing frequently.
Even if routes change frequently, it would be great to then see those differences.
I did show a clean IPv4 traceroute between tunnel endpoints to rule that out as being a factor.
A traceroute shows the likely current path, and that those packets get through; does not tell anything about the tunneled packets getting through though.
Packet loss, routes flapping
Shadow Hawkins on Friday, 31 May 2013 18:16:23It was possibly related to Gyron router "core-2.centro.hml" or one of its links. Why do you think that, do you have any more details about this?
A hop via "core-2.centro.hml" has reappeared in IPv4 traceroutes between tunnel endpoints, which had not been used for the duration of these issues.
Coinciding with this, IPv4 pings reverted to their 'normal' values, with no loss visible now:
http://pyro.eu.org/f/aIB2221JS7OXHuor4sHhNg.png
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. xenolith.pyro.eu.org 0.0% 100 0.2 0.4 0.1 0.9 0.2
2. losubs.subs.dsl2.wh-man.zen.net.uk 0.0% 100 11.1 10.9 10.4 12.5 0.3
3. ge-2-1-0-164.cr1.wh-man.zen.co.uk 0.0% 100 11.2 13.5 10.4 69.4 9.7
4. ge-3-0-0-0.cr2.th-lon.zen.net.uk 0.0% 100 25.4 19.2 17.3 36.9 3.5
5. xe-0-0-3.border-1.sov.lon.uk.as29017.net 0.0% 100 17.9 21.4 17.4 107.6 12.7
6. xe-1-2-2.core-2.centro.hml.uk.as29017.net 0.0% 100 19.0 22.3 18.7 74.4 11.2
7. gblon03.sixxs.net 0.0% 100 19.7 19.2 18.8 28.6 1.0
Smokeping graphs are no longer showing breaks in connectivity or stepped changes in latency. This one relates to Hetzner IPv6->gblon03->Zen IPv4:
http://pyro.eu.org/f/SZt9Oel7Q_Osu2aJ8bw5Rg.png
And this relates to Zen IPv4->gblon03->Bytemark IPv6:
http://pyro.eu.org/f/9ovp2zNcSEmv5Ya3RccvJg.png
(artifact at 13:30 is unrelated to this)
It is unfortunate that the latency/loss graphs no longer show measurements between POPs, as those were much more comprehensive than anything I can measure myself:
https://www.sixxs.net/misc/latency/
Packet loss, routes flapping
Coinciding with this, IPv4 pings reverted to their 'normal' values,
As mentioned, likely your IPv4 connectivity was affecting the IPv6 connectivity. Nothing we can do about that.
It is unfortunate that the latency/loss graphs no longer show measurements between POPs, as those were much more comprehensive than anything I can measure myself:
They only showed latency between PoPs, as they did not show path they could only indicate that somewhere there was a bit more latency; nothing else could be concluded from it though.
Packet loss, routes flapping
Shadow Hawkins on Friday, 31 May 2013 21:36:45As mentioned, likely your IPv4 connectivity was affecting the IPv6 connectivity. Nothing we can do about that.
Then maybe I misunderstood the purpose of opening a ticket. I do see this in the FAQ though:
"In case of problems in the routing you should notify SixXS who will in turn notify the owners of the PoP."
You could have proved/disproved your counter-claims much more easily than I am able to: by tracerouting *from* the PoP to both my IPv4 endpoint, and out from the PoP to other locations, including the IPv6 address at another ISP I gave as an example as having intermittent issues at the time.
And ultimately Gyron are best-placed to investigate routes flapping. This probably would have been affecting their hosting customers too, so I thought they might want to know about it. Perhaps I should try to contact them directly next time.
The packet loss doesn't seem to have completely gone away, so maybe we still could see the issue return tomorrow.
Packet loss, routes flapping
As mentioned, likely your IPv4 connectivity was affecting the IPv6 connectivity. Nothing we can do about > that. Then maybe I misunderstood the purpose of opening a ticket. I do see this in the FAQ though: "In case of problems in the routing you should notify SixXS who will in turn notify the owners of the PoP."
Please note the line above it "Routing between the PoP and any other destination is not in the hands of SixXS and the responsibility lies completely at the ISP who hosts the PoP.", quoting one sentence out of context does not provide a meaningful thing.
But let me rephrase the above to: "we do not control the IPv4 connectivity at the side of your ISP nor can we affect transit connectivity towards the PoP".
There are some problems we can fix, but not everything that happens in the world.
As the problem looks to be IPv4 related given the details provided and as it also seems to go away automatically, there is little we can do.
You could have proved/disproved your counter-claims
As you have not shown a single traceroute or any other detail that showed brokeness, what do you expect us to do exactly? We really do not have time to keep tracerouting all over the world because somebody saw a blimp somewhere but can't even show that blimp from their side.
We are also not in the 'game' of 'disproving' things, hence why we are asking for details. When proper details are provided and when something indicates that something is broken we will definitely look into it and try to get it resolved.
The packet loss doesn't seem to have completely gone away, so maybe we still could see the issue return tomorrow.
If you really want us to look into it, then please actually provide proper and complete details.
Packet loss, routes flapping
some IPv6 routes to/from Gyron are flapping.
Which routes are these?
I'm struggling to measure/locate any packet loss; it's as though the link abruptly goes down.
Where is the traceroute showing this problem?
2a01:4f8:a0:9000::1
That is Hetzner, if you check their status page you would know they have IPv6 issues lately. (and apparently related to DoS towards and sometimes form their network)
Unfortunately I have no record of the prior route where packet loss was observed:
If you have no signs of where something is broken, then there is nothing to fix either, what exactly do you want us to do about something that is not broken?
Please, provide a lot more details instead of filing a basically empty ticket.
Packet loss, routes flapping
Shadow Hawkins on Wednesday, 28 August 2013 11:37:55
Today I'm seeing Gyron IPv6 routes flapping again.
Currently an IPv4 traceroute from my tunnel endpoint to gblon03.sixxs.net appears clean:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 217.155.40.118 0.0% 2569 0.2 0.1 0.1 20.7 1.0
2. 62.3.83.27 0.0% 2569 10.1 10.1 9.9 17.2 0.4
3. 62.3.80.193 0.0% 2569 10.3 12.9 9.8 103.5 11.0
4. 62.3.80.45 0.0% 2569 17.0 19.2 16.5 49.8 4.5
5. 195.66.224.141 0.0% 2569 17.2 18.2 16.5 122.2 7.3
6. 89.145.125.25 0.0% 2569 17.2 21.0 16.7 105.3 10.6
7. 89.145.125.34 0.0% 2568 18.0 18.8 17.5 91.8 7.4
8. 212.113.147.150 0.0% 2568 18.3 17.8 17.6 39.6 1.7
IPv6 pings to gblon03.sixxs.net through the tunnel also seem okay:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a00:14f0:e033:1000::1 0.0% 486 0.2 0.1 0.1 0.3 0.1
2. 2a00:14f0:e000:b5::1 0.0% 486 18.6 18.1 17.7 26.9 0.7
3. 2a00:14f0:1:2::5 0.0% 486 18.1 18.1 17.7 19.4 0.3
And for the past three hours I've measured 0.00% packet loss to my servers at Bytemark and Hetzner through IPv4. (Pings every 6 seconds).
But native IPv6 from the same Bytemark server (hostname: x2.ok24.net) to the Gyron tunnelserver shows loss (there are periods with zero loss, followed by very sudden 'drops' of a few seconds each), and traffic then takes other routes which are somewhat hard to read in mtr:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2001:41c8:51::2 0.0% 1904 0.9 2.1 0.6 183.3 6.3
2. 2001:41c8:0:104::1 0.0% 1904 78.9 5.0 0.7 200.3 21.7
3. 2001:668:0:3::4000:8d2 0.0% 1904 1.2 6.8 0.8 266.0 23.8
2001:41c8:0:10a::1
2001:41c8:0:82::2
4. 2001:668:0:3::4000:8d1 0.0% 1904 0.7 4.5 0.5 100.4 11.0
2001:7f8:17::7159:1
2001:7f8:17::1b1b:1
::ffff:91.223.58.131
2001:7f8:17::7159:2
5. 2001:668:0:2::1:3082 0.1% 1904 9.5 10.2 7.3 101.3 7.2
2a00:14f0:0:1::a
2001:470:0:3f::2
2a00:14f0:0:1::1a
6. 2001:668:0:3::4000:592 2.2% 1904 10.0 10.1 8.2 69.3 5.5
2a00:14f0:1:2::5
2001:470:0:3f::1
2a00:14f0:0:1::a
7. 2a00:14f0:1:2::5 2.2% 1903 10.1 9.5 8.2 15.8 0.7
2001:7f8:4::cb9:1
And the same from the server at Hetzner (hostname: uptime.ok24.net) - the changes of route seems to happen at the exact same moment as from Bytemark:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a01:4f8:a0:9000::1 0.0% 1060 1.0 1.5 0.6 10.0 1.3
2. 2a01:4f8:0:a:3:0:a:2 0.0% 1060 0.3 3.0 0.2 165.6 12.9
3. 2a01:4f8:0:3::35 0.0% 1060 0.3 0.3 0.2 11.2 0.7
4. 2a01:4f8:0:3::26 0.0% 1060 0.4 0.3 0.3 8.4 0.6
2a01:4f8:0:3::15
5. 2001:1620:1000::1a9 0.0% 1060 0.4 3.0 0.3 89.9 4.2
2a01:4f8:0:3::1d
6. 2001:1620:2::95 0.0% 1060 14.7 15.2 3.4 165.4 8.6
2a01:4f8:0:3::e
2a01:4f8:0:3::1
7. 2001:7f8:4::ddd:1 0.0% 1060 18.0 18.4 3.5 210.8 17.8
2001:7f8::1b1b:0:1
2001:1900:5:2:2::805
2a01:4f8:0:3::6
8. 2001:450:2002:268::2 3.4% 1060 28.6 27.1 3.7 93.8 9.2
2001:470:0:225::1
2001:7f8::cb9:0:1
2001:470:0:1d2::1
2001:1900:104:7::9
2001:1900:5:2:2::b31
9. 2a00:14f0:1:2::5 3.4% 1059 28.6 26.5 3.4 88.7 5.5
2001:7f8:1::a502:9017:1
2001:668:0:2::1:39c1
2a01:4f8:0:3::d
2001:7f8:4::cb9:1
2001:1900:5:3::12a
2001:1900:104:7::9
I don't think that is very readable. Much better are these graphs generated by Smokeping. The coloured dots indicate a dropout lasting several seconds. The step up/down in latency suggests flip/flop between two or more different routes having different latency. The times shown are UTC+0100:
To Bytemark through tunnel: http://pyro.eu.org/f/rdH7b8KMRXuaRf8kUEC8eA.png
To Hetzner through tunnel: http://pyro.eu.org/f/7EpxhpeUQnmVDL37X_zFQA.png
This problem seems unrelated to the packet loss I mentioned on 2013-05-31 - that never went away - and is often seen in the evenings (averaged, about 0.25% loss from my tunnel endpoint to the tunnel server; a further 0.25% on the onward tunnelled IPv6 traffic; it may be a congestion issue affecting Gyron's whole network, but unfortunately tunneled traffic gets doubly-affected by it).
Packet loss, routes flapping
Shadow Hawkins on Wednesday, 28 August 2013 12:26:00
By the time I wrote all of this it seems the problem has gone away.
Packet loss, routes flapping
::ffff:91.223.58.131
I find that an amazing IPv6 address that should never exist on the wire. Seems to be a Bytemark IP. Typically, those kind of packets should be dropped as they never should exist on the wire.
that never went away - and is often seen in the evenings
Sounds like an overloaded network to me, or more likely some network that is applying ratelimiting or "QoS" to their network.
As the Gyron network is not doing any of that and is heavily overprovisioned, I see little reason for it to happen in that network though, thus there is also little we can do.
|
![[ch]](/s/countries/ch.gif)
on Friday, 31 May 2013 15:03:18
Posting is only allowed when you are