|
Ticket ID: SIXXS #2666029 Ticket Status: User PoP: fihel01 - DNA Oy (Helsinki)
High latency peaks
![[fi]](/s/countries/fi.gif) Shadow Hawkins on Tuesday, 21 September 2010 19:30:24
Hi,
I have high latency issues with fihel01. Latency isn't constant, but it has some nasty peaks. I'm using latest aiccu version 2007.01.15 in Linux. Nothing has changed in my configuration. Same results are shown in your latency statistics.
Traceroute to 62.78.96.38 (62.78.96.38), 30 hops max, 38 byte packets
1 pc-128-1.ftth.aina.net (88.86.128.1) 1.371 ms 1.136 ms 1.391 ms
2 81.16.67.117 (81.16.67.117) 4.403 ms 4.347 ms 4.459 ms
3 hkikork-gw9-ge1-1.ainaip.net (84.34.64.157) 4.115 ms 4.028 ms 3.871 ms
4 hkivall-gw9-2-4.ainaip.net (84.34.65.49) 4.713 ms 3.877 ms 3.787 ms
5 hkiipas-gw8-ge1-5-955.ainaip.net (84.34.64.173) 4.639 ms 4.468 ms 4.090 ms
6 hkiipas-gw1-po-2-2121.ainaip.net (84.34.64.142) 4.280 ms 3.942 ms 3.851 ms
7 hkiipas-gw1-po-1-2121.ainaip.net (84.34.64.141) 4.581 ms 3.824 ms 3.901 ms
8 dna.ficix2-ge.ficix.fi (193.110.224.20) 4.394 ms 4.069 ms 3.981 ms
9 esp2-tr1.dnaip.fi (62.78.107.99) 4.712 ms 4.519 ms 4.478 ms
10 van1-er2.dnaip.fi (62.78.111.13) 5.903 ms 4.774 ms 5.478 ms
11 fihel01.sixxs.net (62.78.96.38) 8.392 ms 4.563 ms 4.396 ms
Traceroute from computer inside IPv6-subnet.
traceroute to 2001:14b8:100:9::1 (2001:14b8:100:9::1) from 2001:14b8:108:2:224:1ff:fe2e:1ad7, 30 hops max, 16 byte packets
1 2001:14b8:108:2::1 (2001:14b8:108:2::1) 0.69 ms 0.528 ms 0.453 ms
2 gw-10.hel-01.fi.sixxs.net (2001:14b8:100:9::1) 4.748 ms 6.515 ms 5.265 ms
IPv4 ping to POP:
...
64 bytes from 62.78.96.38: seq=65 ttl=52 time=4.529 ms
64 bytes from 62.78.96.38: seq=66 ttl=52 time=4.430 ms
64 bytes from 62.78.96.38: seq=67 ttl=52 time=4.417 ms
64 bytes from 62.78.96.38: seq=68 ttl=52 time=63.646 ms
64 bytes from 62.78.96.38: seq=69 ttl=52 time=261.099 ms
64 bytes from 62.78.96.38: seq=70 ttl=52 time=450.058 ms
64 bytes from 62.78.96.38: seq=71 ttl=52 time=4.486 ms
64 bytes from 62.78.96.38: seq=72 ttl=52 time=518.976 ms
64 bytes from 62.78.96.38: seq=73 ttl=52 time=100.700 ms
64 bytes from 62.78.96.38: seq=74 ttl=52 time=131.025 ms
64 bytes from 62.78.96.38: seq=75 ttl=52 time=381.210 ms
64 bytes from 62.78.96.38: seq=76 ttl=52 time=645.685 ms
64 bytes from 62.78.96.38: seq=77 ttl=52 time=506.877 ms
64 bytes from 62.78.96.38: seq=78 ttl=52 time=147.053 ms
64 bytes from 62.78.96.38: seq=79 ttl=52 time=10.737 ms
64 bytes from 62.78.96.38: seq=80 ttl=52 time=4.479 ms
64 bytes from 62.78.96.38: seq=81 ttl=52 time=4.756 ms
64 bytes from 62.78.96.38: seq=82 ttl=52 time=4.505 ms
64 bytes from 62.78.96.38: seq=83 ttl=52 time=4.476 ms
64 bytes from 62.78.96.38: seq=84 ttl=52 time=4.459 ms
...
--- 62.78.96.38 ping statistics ---
99 packets transmitted, 99 packets received, 0% packet loss
round-trip min/avg/max = 4.351/69.090/645.685 ms
Latency multiplies in IPv6
IPv6 ping to POP:
...
64 bytes from 2001:14b8:100:9::1: seq=13 ttl=64 time=5.111 ms
64 bytes from 2001:14b8:100:9::1: seq=14 ttl=64 time=5.117 ms
64 bytes from 2001:14b8:100:9::1: seq=15 ttl=64 time=300.899 ms
64 bytes from 2001:14b8:100:9::1: seq=16 ttl=64 time=760.305 ms
64 bytes from 2001:14b8:100:9::1: seq=17 ttl=64 time=1252.023 ms
64 bytes from 2001:14b8:100:9::1: seq=18 ttl=64 time=1322.727 ms
64 bytes from 2001:14b8:100:9::1: seq=19 ttl=64 time=1394.460 ms
64 bytes from 2001:14b8:100:9::1: seq=20 ttl=64 time=1244.990 ms
64 bytes from 2001:14b8:100:9::1: seq=21 ttl=64 time=1316.931 ms
64 bytes from 2001:14b8:100:9::1: seq=22 ttl=64 time=1802.121 ms
64 bytes from 2001:14b8:100:9::1: seq=23 ttl=64 time=1872.540 ms
64 bytes from 2001:14b8:100:9::1: seq=24 ttl=64 time=2380.169 ms
64 bytes from 2001:14b8:100:9::1: seq=25 ttl=64 time=4348.711 ms
64 bytes from 2001:14b8:100:9::1: seq=26 ttl=64 time=5052.905 ms
64 bytes from 2001:14b8:100:9::1: seq=27 ttl=64 time=4906.449 ms
64 bytes from 2001:14b8:100:9::1: seq=28 ttl=64 time=4551.679 ms
64 bytes from 2001:14b8:100:9::1: seq=29 ttl=64 time=3755.993 ms
64 bytes from 2001:14b8:100:9::1: seq=30 ttl=64 time=3193.803 ms
64 bytes from 2001:14b8:100:9::1: seq=31 ttl=64 time=2830.685 ms
64 bytes from 2001:14b8:100:9::1: seq=32 ttl=64 time=2712.515 ms
64 bytes from 2001:14b8:100:9::1: seq=33 ttl=64 time=2358.731 ms
64 bytes from 2001:14b8:100:9::1: seq=34 ttl=64 time=2018.917 ms
64 bytes from 2001:14b8:100:9::1: seq=35 ttl=64 time=1236.557 ms
64 bytes from 2001:14b8:100:9::1: seq=36 ttl=64 time=453.602 ms
64 bytes from 2001:14b8:100:9::1: seq=37 ttl=64 time=5.074 ms
64 bytes from 2001:14b8:100:9::1: seq=38 ttl=64 time=5.017 ms
64 bytes from 2001:14b8:100:9::1: seq=39 ttl=64 time=322.916 ms
...
--- 2001:14b8:100:9::1 ping statistics ---
55 packets transmitted, 55 packets received, 0% packet loss
round-trip min/avg/max = 5.017/972.675/5052.905 ms
State change: user
The state of this ticket has been changed to user
High latency peaks
That latency can occur anywhere in the network between your endpoint and that of the PoP. Very difficult to troubleshoot unless you can check what happens to the IPv6 that are encapsulated in IPv4 at every hop of the route. It could be any kind of buffering or full-links anywhere in the network.
As both your IPv4 and IPv6 latencies have this problem I would not be surprised if some hop in the network is doing "QoS" which causes this.
Nothing we can do for you though.
High latencies to fihel01 (both IPv4 and IPv6)
Shadow Hawkins on Thursday, 23 September 2010 18:25:44
High latencies are occurring in my tunnel (T10820) to fihel01 and when pinging fihel01.sixxs.net over both IPv4 and IPv6. PINGs to the last hop before fihel01 seem to be fine.
To my recollection, this has been going on for a couple of days or probably a week now.
Here is a traceroute to fihel01:
$ traceroute fihel01.sixxs.net
traceroute to fihel01.sixxs.net (62.78.96.38), 64 hops max, 40 byte packets
1 gw-1-v317.kyla.fi (82.130.18.252) 0.958 ms 1.549 ms 0.742 ms
2 funet-csc6-10ge-gw-1.tky.fi (82.130.63.253) 0.613 ms 0.615 ms 0.598 ms
3 dna.ficix1-ge.ficix.fi (193.110.226.20) 0.778 ms 0.786 ms 0.737 ms
4 esp2-tr1.dnaip.fi (62.78.107.44) 26.919 ms 1.378 ms 1.240 ms
5 van1-er2.dnaip.fi (62.78.111.13) 1.385 ms 1.242 ms 1.210 ms
6 fihel01.sixxs.net (62.78.96.38) 1364.847 ms 4118.085 ms 62.699 ms
IPv4 PING results to hop 5 (van1-er2.dnaip.fi):
$ ping van1-er2.dnaip.fi
PING van1-er2.dnaip.fi (62.78.100.162): 56 data bytes
64 bytes from 62.78.100.162: icmp_seq=0 ttl=251 time=1.580 ms
64 bytes from 62.78.100.162: icmp_seq=1 ttl=251 time=1.812 ms
64 bytes from 62.78.100.162: icmp_seq=2 ttl=251 time=1.799 ms
64 bytes from 62.78.100.162: icmp_seq=3 ttl=251 time=1.874 ms
64 bytes from 62.78.100.162: icmp_seq=4 ttl=251 time=1.793 ms
64 bytes from 62.78.100.162: icmp_seq=5 ttl=251 time=1.694 ms
64 bytes from 62.78.100.162: icmp_seq=6 ttl=251 time=1.786 ms
64 bytes from 62.78.100.162: icmp_seq=7 ttl=251 time=1.713 ms
64 bytes from 62.78.100.162: icmp_seq=8 ttl=251 time=1.756 ms
64 bytes from 62.78.100.162: icmp_seq=9 ttl=251 time=1.723 ms
64 bytes from 62.78.100.162: icmp_seq=10 ttl=251 time=2.003 ms
64 bytes from 62.78.100.162: icmp_seq=11 ttl=251 time=2.732 ms
64 bytes from 62.78.100.162: icmp_seq=12 ttl=251 time=1.817 ms
64 bytes from 62.78.100.162: icmp_seq=13 ttl=251 time=1.819 ms
64 bytes from 62.78.100.162: icmp_seq=14 ttl=251 time=1.835 ms
64 bytes from 62.78.100.162: icmp_seq=15 ttl=251 time=1.739 ms
64 bytes from 62.78.100.162: icmp_seq=16 ttl=251 time=1.737 ms
64 bytes from 62.78.100.162: icmp_seq=17 ttl=251 time=1.699 ms
64 bytes from 62.78.100.162: icmp_seq=18 ttl=251 time=1.763 ms
64 bytes from 62.78.100.162: icmp_seq=19 ttl=251 time=1.741 ms
64 bytes from 62.78.100.162: icmp_seq=20 ttl=251 time=1.894 ms
^C
----van1-er2.dnaip.fi PING Statistics----
21 packets transmitted, 21 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.580/1.824/2.732/0.225 ms
To fihel01.sixxs.net over IPv4:
$ ping fihel01.sixxs.net
PING fihel01.sixxs.net (62.78.96.38): 56 data bytes
64 bytes from 62.78.96.38: icmp_seq=0 ttl=58 time=174.355 ms
64 bytes from 62.78.96.38: icmp_seq=1 ttl=58 time=1.568 ms
64 bytes from 62.78.96.38: icmp_seq=2 ttl=58 time=35.055 ms
64 bytes from 62.78.96.38: icmp_seq=3 ttl=58 time=1.845 ms
64 bytes from 62.78.96.38: icmp_seq=4 ttl=58 time=1.578 ms
64 bytes from 62.78.96.38: icmp_seq=5 ttl=58 time=1.530 ms
64 bytes from 62.78.96.38: icmp_seq=6 ttl=58 time=188.073 ms
64 bytes from 62.78.96.38: icmp_seq=7 ttl=58 time=481.631 ms
64 bytes from 62.78.96.38: icmp_seq=8 ttl=58 time=3912.739 ms
64 bytes from 62.78.96.38: icmp_seq=9 ttl=58 time=2913.725 ms
64 bytes from 62.78.96.38: icmp_seq=10 ttl=58 time=1914.899 ms
64 bytes from 62.78.96.38: icmp_seq=11 ttl=58 time=915.207 ms
64 bytes from 62.78.96.38: icmp_seq=12 ttl=58 time=1251.317 ms
64 bytes from 62.78.96.38: icmp_seq=13 ttl=58 time=242.450 ms
^C
----fihel01.sixxs.net PING Statistics----
14 packets transmitted, 14 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.530/859.712/3912.739/1236.616 ms
To POP endpoint of my tunnel over IPv6:
$ ping6 2001:14b8:100:1b3::1
PING6(56=40+8+8 bytes) 2001:14b8:100:1b3::2 --> 2001:14b8:100:1b3::1
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=0 hlim=64 time=1.709 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=1 hlim=64 time=189.729 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=2 hlim=64 time=1.454 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=3 hlim=64 time=1.494 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=4 hlim=64 time=1.447 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=5 hlim=64 time=1.493 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=6 hlim=64 time=1.464 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=7 hlim=64 time=618.833 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=8 hlim=64 time=3629.681 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=9 hlim=64 time=4325.575 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=10 hlim=64 time=3546.410 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=11 hlim=64 time=3260.231 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=12 hlim=64 time=2473.205 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=13 hlim=64 time=2337.824 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=14 hlim=64 time=1769.668 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=15 hlim=64 time=1195.275 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=16 hlim=64 time=629.269 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=17 hlim=64 time=1.505 ms
16 bytes from 2001:14b8:100:1b3::1, icmp_seq=18 hlim=64 time=1.475 ms
^C
--- 2001:14b8:100:1b3::1 ping6 statistics ---
19 packets transmitted, 19 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.447/1262.513/4325.575/1523.228 ms
State change: user
The state of this ticket has been changed to user
High latencies to fihel01 (both IPv4 and IPv6)
The latency statistics show some bumps in the latency, nothing much that can be done about this, it is ethernet, when it is full or overloaded anywhere it is just that.
High latencies to fihel01 (both IPv4 and IPv6)
Shadow Hawkins on Saturday, 25 September 2010 17:33:27
First of all, I'm sorry for posting a duplicate of ticket #2666029. It was posted just two days before this one and describes exactly the same problem.
There are a couple of things that make me suspect there is some kind of a network problem that could possibly be addressed at dna's end.
First one is that when I PING the last router before fihel01 (reported by traceroute) on the IPv4 route, it is perfectly fine. Judging from the hostname, the router is within dna's network so they could probably track it down. I wonder if they are at all aware of this?
Also, between the latency spikes (which last for approximately 10 seconds each), there are a couple of tens of seconds where the RTTs are again very low (today 0.9 ms compared to the 5000 ms or so during the spikes), so the network doesn't seem very busy in general. There could for example be a faulty device or a misconfigured host that is causing spikes of spurious traffic on the network that is otherwise fine.
Third, there is absolutely no packet loss, which I'd expect to see if this would be caused by a network overload so bad as to cause over 5000 ms RTTs.
If you at SixXS are unable to do anything about this problem, is there anyone else that you or I could contact? This is very annoying and apparently is affecting other users of fihel01 as well.
Ping / droprate on fihel01?
Carmen Sandiego on Sunday, 26 September 2010 00:57:25
Hello!
Tunnel: T7552
Connection: 10Mbit PPPoE over 100Mbit ethernet, ISP Dna
I noticed that drop-rates and latencies for fihel01 have went up on week 37. Some graphs at my end (pop-tunnel.ip6 == 2001:14b8:100:14b::1):
http://evilspy.1g.fi/ping-week.png
http://evilspy.1g.fi/pingdrop-week.png
http://evilspy.1g.fi/ping-month.png
http://evilspy.1g.fi/pingdrop-month.png
Graphs at www.sixxs.net for this tunnel show lower latencies (have rose slightly after week 37) and no lost.
Ping / droprate on fihel01?
Carmen Sandiego on Sunday, 26 September 2010 01:11:06
Sorry, I missed the PoP selection box on 'new ticket' page.
I checked http://www.sixxs.net/misc/latency/ for fihel01 for last month. All latency graphs have latency risen after week 37. So I don't think is problem with just my link to PoP.
Ping / droprate on fihel01?
Carmen Sandiego on Saturday, 02 October 2010 19:18:03
Situation has now improved. Latencies are back to pre-week 38 levels and also droprates have fixed (I figured no packets were dropped actually, instead I have 1 sec cut off in collectd which translated longer latencies to dropped packets).
|
![[ch]](/s/countries/ch.gif)
on Tuesday, 21 September 2010 19:41:21
Posting is only allowed when you are