|
Problems with IPV6 on connect,facebook.net
![[gb]](/s/countries/gb.gif) Shadow Hawkins on Saturday, 27 September 2014 11:16:19
Having problems for the past few days connecting to connect.facebook.net
]# host connect.facebook.net
connect.facebook.net is an alias for connect.facebook.net.edgekey.net.
connect.facebook.net.edgekey.net is an alias for e3821.dspe1.akamaiedge.net.
e3821.dspe1.akamaiedge.net has address 23.214.37.44
e3821.dspe1.akamaiedge.net has IPv6 address 2a02:26f0:26:28e::eed
e3821.dspe1.akamaiedge.net has IPv6 address 2a02:26f0:26:288::eed
e3821.dspe1.akamaiedge.net has IPv6 address 2a02:26f0:26:28d::eed
]# traceroute6 2a02:26f0:26:28e::eed
traceroute to 2a02:26f0:26:28e::eed (2a02:26f0:26:28e::eed), 30 hops max, 80 byte packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 17.963 ms 17.792 ms 18.106 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 18.020 ms 17.981 ms 18.260 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 18.471 ms 19.124 ms 19.039 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 18.958 ms 18.873 ms 18.783 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 20.260 ms 20.395 ms 20.312 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 20.228 ms 19.195 ms 20.433 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 20.374 ms 20.516 ms 20.700 ms
8 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 20.634 ms 20.567 ms 19.596 ms
9 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 19.782 ms 20.247 ms 20.198 ms
10 2a02:26f0:26::5c7a:d2d2 (2a02:26f0:26::5c7a:d2d2) 20.282 ms 20.731 ms 20.681 ms
]# traceroute6 2a02:26f0:26:288::eed
traceroute to 2a02:26f0:26:288::eed (2a02:26f0:26:288::eed), 30 hops max, 80 byte packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 18.452 ms 18.644 ms 18.616 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 18.587 ms 18.561 ms 18.707 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 18.737 ms 18.896 ms 19.222 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 19.240 ms 19.458 ms 19.694 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 20.540 ms 20.732 ms 20.810 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 21.041 ms 20.206 ms 20.288 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 22.916 ms 23.075 ms 23.009 ms
8 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 33.850 ms 33.782 ms 33.700 ms
9 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 21.309 ms 21.311 ms 21.432 ms
10 2a02:26f0:26::5c7a:d2cc (2a02:26f0:26::5c7a:d2cc) 21.603 ms 21.540 ms *
]# traceroute6 2a02:26f0:26:28d::eed
traceroute to 2a02:26f0:26:28d::eed (2a02:26f0:26:28d::eed), 30 hops max, 80 byte packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 17.903 ms 18.040 ms 17.947 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 17.864 ms 18.147 ms 18.115 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 18.210 ms 18.174 ms 18.323 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 18.729 ms 18.798 ms 18.763 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 19.826 ms 20.159 ms 19.660 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 19.986 ms 19.665 ms 20.831 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 20.761 ms 20.978 ms 20.746 ms
8 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 26.891 ms 26.824 ms 26.950 ms
9 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 20.779 ms 20.964 ms 19.808 ms
10 2a02:26f0:26::5c7a:d2d1 (2a02:26f0:26::5c7a:d2d1) 20.390 ms 20.309 ms 20.222 ms
]# tcptraceroute6 2a02:26f0:26:28e::eed 80
traceroute to 2a02:26f0:26:28e::eed (2a02:26f0:26:28e::eed) from 2a00:14f0:e000:d2::2, port 80, from port 40939, 30 hops max, 60 bytes packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 17.375 ms 18.180 ms 17.768 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 18.192 ms 17.808 ms 18.014 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 18.040 ms 18.209 ms 17.704 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 59.189 ms 27.197 ms 18.206 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 20.534 ms 19.975 ms 19.375 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 20.128 ms 19.783 ms 19.488 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 20.634 ms 19.707 ms 19.695 ms
8 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 28.112 ms 23.272 ms 24.936 ms
9 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 19.863 ms 19.888 ms 19.582 ms
10 2a02:26f0:26:28e::eed (2a02:26f0:26:28e::eed) 20.013 ms [open] 20.155 ms 19.941 ms
]# tcptraceroute6 2a02:26f0:26:28e::eed 443
traceroute to 2a02:26f0:26:28e::eed (2a02:26f0:26:28e::eed) from 2a00:14f0:e000:d2::2, port 443, from port 40996, 30 hops max, 60 bytes packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 18.109 ms 17.382 ms 17.717 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 18.662 ms 17.722 ms 17.556 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 18.067 ms 17.739 ms 18.279 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 18.343 ms 18.452 ms 18.788 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 19.784 ms 19.599 ms 19.223 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 31.632 ms 19.397 ms 19.712 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 23.025 ms 23.499 ms 24.308 ms
8 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 19.679 ms 20.170 ms 20.421 ms
9 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 20.329 ms 20.557 ms 20.296 ms
10 2a02:26f0:26:28e::eed (2a02:26f0:26:28e::eed) 21.009 ms [open] 20.313 ms 19.555 ms
]# tracepath6 2a02:26f0:26:28e::eed
1?: [LOCALHOST] 0.052ms pmtu 1280
1: gw-211.lon-03.gb.sixxs.net 19.766ms
1: gw-211.lon-03.gb.sixxs.net 19.177ms
2: gblon03.sixxs.net 19.730ms asymm 1
3: 2a00:14f0:1:2::2 20.481ms asymm 2
4: ae3.core-1.maylands.hml.uk.as29017.net 19.854ms asymm 3
5: xe-0-0-0.border-1.sov.lon.uk.as29017.net 21.140ms asymm 4
6: xe-0-0-1.border-1.thn.lon.uk.as29017.net 21.616ms asymm 5
7: lonap.he.net 42.248ms asymm 6
8: 10ge5-2.core1.lon2.he.net 25.182ms asymm 5
9: 2001:7f8:4::51cc:1 22.109ms asymm 5
10: 2a02:26f0:26::5c7a:d2d2 22.257ms reached
Resume: pmtu 1280 hops 10 back 6
Other tcptraceroute6'es work as well to ports 80 and 443 as do tracepath6'es. I am able to spoof the IPV4 address in my DNS to hold off complaints. A good test site is EA's Pogo Card Games the browser hangs connecting to connect.facebook.net.
Problems with IPV6 on connect,facebook.net
You likely have a pMTU issue, especially as you can reach it with smaller packets.
Note that that issues is somewhere on the path, and it is not unlikely when that is in the remote network.
Hard to say for sure though, without having access to both sides of the network.
You might be able to see it a bit better by running a Wireshark and then trying to access the host and seeing what Wireshark reports.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Sunday, 28 September 2014 15:39:57
Ran wireshark on it and suprise suprise it is working now. I'm having the same sort of problem with platform.linkedin.com as well. Both use akamaiedge.net. and the IPV6 2a02:26f0:: network
Problems with IPV6 on connect,facebook.net
![[dk]](/s/countries/dk.gif) Shadow Hawkins on Monday, 29 September 2014 08:03:56
Charlie MacDonald wrote:
Ran wireshark on it and suprise suprise it is working now. I'm having the same sort of problem with platform.linkedin.com as well. Both use akamaiedge.net. and the IPV6 2a02:26f0:: network
Hi Charlie,
When did this start? I've been having intermittent issues over the last couple of weeks reaching sites such as www.cisco.com, www.juniper.net, which is also hosted at Akamai (same prefix as connect.facebook.net).
I've tried fiddling with MTU settings on the tunnel, but it haven't changed anything - the issue still persists.
I spent most of my time Saturday trying to figure out what is going on, and ended up concluding that my computer is fragmenting the packets as expected based on the tunnel MTU, and that my Juniper firewall is allowing traffic to pass.
/Mark
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Monday, 29 September 2014 10:47:52
Hi Mark:
Started a couple of weeks ago for me. I noticed the corrolation between the Akami and the IPV6 network they were on a couple of days ago. I was able to spoof both connect.facebook.net and platform.linkedin.com on my DNS server to their IPV4 addresses whilst I tried to find a solution. I am going to asume that there is a problem with Akami and/or something on the path to 2a02:26f0:: traceroute, tcptraceroute, and tracepath all work properly. I tried www.cisco.com just a few seconds ago and it's having the same issues and is on the 2a02:26f0:: network. Can't do much troubleshooting on this end if the usual tools aren't giving any info :( Wireshark gives a clue: "[TCP Previous Segment not Captured] [TCP Segment of a Reassembled PDU]". There are a few retransmissions and when it finally gets there it is a text-only web page. Very odd.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Monday, 29 September 2014 11:19:35
Just a quick update (maybe a red herring) packets seem to stop at 2001:7f8:4::51cc:1 depending on the size of the IPV6 packets.
]# rltraceroute6 www.cisco.com
traceroute to e144.dscb.akamaiedge.net (2a02:26f0:26:3:9900::90) from 2a00:14f0:e000:d2::2, port 33434, from port 36031, 30 hops max, 60 bytes packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 17.937 ms 18.097 ms 18.015 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 18.175 ms 18.217 ms 17.642 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 18.425 ms 18.273 ms 18.609 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 18.305 ms 18.627 ms 17.804 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 19.673 ms 19.354 ms 19.393 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 19.972 ms 19.571 ms 19.719 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 29.434 ms 24.082 ms 24.642 ms
8 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 19.778 ms 19.830 ms 20.028 ms
9 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 19.939 ms 20.175 ms 19.655 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * 2a02:26f0:26::5f64:f21d (2a02:26f0:26::5f64:f21d) 21.027 ms
]# rltraceroute6 www.cisco.com 1024
traceroute to e144.dscb.akamaiedge.net (2a02:26f0:26:3:9700::90) from 2a00:14f0:e000:d2::2, port 33434, from port 36024, 30 hops max, 1024 bytes packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 19.488 ms 18.983 ms 19.231 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 19.858 ms 19.676 ms 20.073 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 21.022 ms 20.342 ms 19.392 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 21.208 ms 20.729 ms 19.677 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 22.521 ms 22.107 ms 21.183 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 22.483 ms 21.970 ms 51.365 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 22.630 ms 22.507 ms 21.302 ms
8 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 33.288 ms 31.503 ms 23.653 ms
9 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 22.812 ms 22.700 ms 21.657 ms
10 * 2a02:26f0:26::5f64:f21b (2a02:26f0:26::5f64:f21b) 22.344 ms 21.395 ms
]# rltraceroute6 www.cisco.com 1500
traceroute to e144.dscb.akamaiedge.net (2a02:26f0:26:3:9700::90) from 2a00:14f0:e000:d2::2, port 33434, from port 36020, 30 hops max, 1500 bytes packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 19.683 ms 20.123 ms 19.878 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 21.210 ms 20.730 ms 19.615 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 22.770 ms 20.754 ms 19.922 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 21.979 ms 20.730 ms 20.337 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 23.424 ms 22.388 ms 21.686 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 23.500 ms 22.246 ms 21.645 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 28.997 ms 24.049 ms 24.375 ms
8 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 24.262 ms 23.158 ms 22.001 ms
9 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 24.454 ms 23.025 ms 22.612 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Monday, 29 September 2014 13:46:16
Charlie MacDonald wrote:
Hi Mark:
Started a couple of weeks ago for me. I noticed the corrolation between the Akami and the IPV6 network they were on a couple of days ago. I was able to spoof both connect.facebook.net and platform.linkedin.com on my DNS server to their IPV4 addresses whilst I tried to find a solution. I am going to asume that there is a problem with Akami and/or something on the path to 2a02:26f0:: traceroute, tcptraceroute, and tracepath all work properly. I tried www.cisco.com just a few seconds ago and it's having the same issues and is on the 2a02:26f0:: network. Can't do much troubleshooting on this end if the usual tools aren't giving any info :( Wireshark gives a clue: "[TCP Previous Segment not Captured] [TCP Segment of a Reassembled PDU]". There are a few retransmissions and when it finally gets there it is a text-only web page. Very odd.
This sounds quite familiar to me. I'm seeing very identical symptoms in my end. Some traffic comes through while other traffic just disappears (or at least I'm not getting the traffic back).
The packet size test you've done could indicate there is a link with a MTU issue somewhere en route. I'm not getting a lot of ICMPv6 Packet too big messages, so it may simply be that router failing to send those ICMPv6 back to the source. To the user, that would just look like a painfully slow loading of webpages.
It'll be interesting to hear if other users are seeing the same.
/Mark
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Monday, 29 September 2014 11:28:29
Now getting to www.linkedin.com doesn't work...stops at gblon03.sixxs.net
]# tcptraceroute6 www.linkedin.com
traceroute to pop-idb2-alpha.www.linkedin.com (2620:109:c007:102::5be1:f881) from 2a00:14f0:e000:d2::2, port 80, from port 35917, 30 hops max, 60 bytes packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 18.038 ms 17.601 ms 18.068 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 17.873 ms 18.109 ms 18.064 ms
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Problems with IPV6 on connect.facebook.net
traceroute6 www.linkedin.com
traceroute to www.linkedin.com (2620:109:c006:102::6cae:281), 30 hops max, 80 byte packets
1 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 0.212 ms 0.200 ms 0.191 ms
2 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 0.215 ms 0.204 ms 0.188 ms
3 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 1.617 ms 1.608 ms 1.636 ms
4 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 21.041 ms 21.078 ms 21.064 ms
5 lonap.he.net (2001:7f8:17::1b1b:1) 1.776 ms 1.864 ms 1.902 ms
6 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 1.870 ms 1.879 ms 1.869 ms
7 100ge1-1.core1.nyc4.he.net (2001:470:0:2cf::2) 71.881 ms 72.989 ms 72.978 ms
8 100ge5-1.core1.ash1.he.net (2001:470:0:299::1) 93.677 ms 93.669 ms 93.697 ms
9 * * *
10 * * *
Seems the Akamai network is a bit unstable. People on nanog are also reporting various issues with it.
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Tuesday, 30 September 2014 04:43:24
I can get to it sometimes depending on the IPV6 it resolves to. At least it's not just me :)
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Tuesday, 30 September 2014 08:00:36
The issue has returned
packetsize 1280 works fime
charlie@pc1:~$ sudo tcptraceroute6 -l 1280 connect.facebook.net
traceroute to e3821.dspe1.akamaiedge.net (2a02:26f0:26:281::eed) from 2a00:14f0:e000:80d2:cd1a::1, port 80, from port 59784, 30 hops max, 1280 bytes/packet
1 cdmlan.com (2a00:14f0:e000:80d2::1) 0.258 ms 0.360 ms 0.339 ms
2 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 19.451 ms 20.181 ms 20.027 ms
3 gblon03.sixxs.net (2a00:14f0:1:2::5) 21.283 ms 20.702 ms 19.558 ms
4 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 22.258 ms 20.681 ms 20.512 ms
5 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 21.546 ms 21.576 ms 19.946 ms
6 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 23.470 ms 23.219 ms 21.043 ms
7 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 23.647 ms 22.401 ms 21.704 ms
8 lonap.he.net (2001:7f8:17::1b1b:1) 23.313 ms 23.047 ms 23.809 ms
9 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 23.730 ms 22.886 ms 23.493 ms
10 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 23.478 ms 22.553 ms 21.860 ms
11 2a02:26f0:26:281::eed (2a02:26f0:26:281::eed) 23.694 ms [open] * 21.667 ms [open]
Packet size 1281 (and anything over 1280) doesn't
charlie@pc1:~$ sudo tcptraceroute6 -l 1281 connect.facebook.net
traceroute to e3821.dspe1.akamaiedge.net (2a02:26f0:26:28d::eed) from 2a00:14f0:e000:80d2:cd1a::1, port 80, from port 59781, 30 hops max, 1281 bytes/packet
1 cdmlan.com (2a00:14f0:e000:80d2::1) 0.403 ms 0.337 ms 0.309 ms
2 * gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 20.396 ms 19.937 ms
3 gblon03.sixxs.net (2a00:14f0:1:2::5) 20.828 ms 20.834 ms 19.950 ms
4 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 22.172 ms 49.689 ms 20.539 ms
5 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 22.105 ms 21.370 ms 20.401 ms
6 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 23.820 ms 23.171 ms 21.749 ms
7 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 23.983 ms 22.622 ms 21.462 ms
8 lonap.he.net (2001:7f8:17::1b1b:1) 23.759 ms 23.756 ms 23.979 ms
9 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 23.952 ms 23.535 ms 23.420 ms
10 2001:7f8:4::51cc:1 (2001:7f8:4::51cc:1) 23.725 ms 22.731 ms 22.231 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Packetsize 1281 to a working website
charlie@pc1:~$ sudo tcptraceroute6 -l 1281 www.cloudflare.com
traceroute to www.cloudflare.com.cdn.cloudflare.net (2400:cb00:2048:1::c629:d6a3) from 2a00:14f0:e000:80d2:cd1a::1, port 80, from port 59763, 30 hops max, 1281 bytes/packet
1 cdmlan.com (2a00:14f0:e000:80d2::1) 0.300 ms 0.305 ms 0.304 ms
2 * gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 20.099 ms 21.724 ms
3 gblon03.sixxs.net (2a00:14f0:1:2::5) 20.969 ms 23.297 ms 20.128 ms
4 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 23.498 ms 21.637 ms 20.602 ms
5 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 22.224 ms 21.398 ms 20.149 ms
6 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 23.294 ms 22.700 ms 22.068 ms
7 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 23.847 ms 23.653 ms 21.830 ms
8 lonap.he.net (2001:7f8:17::1b1b:1) 23.877 ms 22.948 ms 22.086 ms
9 10ge5-2.core1.lon2.he.net (2001:470:0:2cd::1) 24.112 ms 23.000 ms 21.890 ms
10 2001:7f8:4::3417:1 (2001:7f8:4::3417:1) 25.494 ms 24.109 ms 23.344 ms
11 2400:cb00:2048:1::c629:d6a3 (2400:cb00:2048:1::c629:d6a3) 24.304 ms [open] 23.283 ms 22.379 ms
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Tuesday, 30 September 2014 19:04:20
A quick update:
Got to Akamai's website and found a contact email address (wasn't easy without IPV6 connectivity to the site!). Just heard back from them:
Akamai Customer Care
Today at 7:08 PM
To
xxxx
Dear xxxxx,
thank you for reporting this IPv6 routing issue with our London cluster. I have forwarded to our Network team for further investigation.
Regards,
a Guardian of the Internet,
Akamai Customer Care
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Thursday, 02 October 2014 00:17:56
Set my MTU to 1280 on my ethernet and wireless interfaces and suddenly all the Akamai sites come up. Obviously their network is not treating packets > 1280 bytes correctly.
Problems with IPV6 on connect.facebook.net
Charlie MacDonald wrote:
Set my MTU to 1280 on my ethernet and wireless interfaces and suddenly all the Akamai sites come up. Obviously their network is not treating packets > 1280 bytes correctly.
With such a change you are basically doing TCP MSS clamping by configuring your local interfaces to a low MTU. The problem with Akamai thus is still not gone and you are limiting your local network because of a misconfiguration elsewhere.
Note that UDP or any other non-TCP protocol will still break...
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Thursday, 02 October 2014 03:37:45
I agree that changing the MTU is not the solution. I have found other sites that will not accept packets over 1280 bytes but still work under IPV6.
~]$ sudo tcptraceroute6 -l 1280 www.google.com
traceroute to www.google.com (2a00:1450:4013:c01::69) from 2a00:14f0:e000:d2::2, port 80, from port 57125, 30 hops max, 1280 bytes packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 19.661 ms 19.599 ms 19.888 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 20.237 ms 20.283 ms 19.740 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 21.611 ms 20.611 ms 19.628 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 21.210 ms 20.654 ms 19.669 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 22.704 ms 21.903 ms 26.821 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 41.204 ms 22.018 ms 21.431 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 23.119 ms 22.091 ms 21.532 ms
8 2001:4860:1:1:0:1b1b:0:5 (2001:4860:1:1:0:1b1b:0:5) 24.030 ms 22.352 ms 26.992 ms
9 2001:4860::1:0:15f (2001:4860::1:0:15f) 24.182 ms 22.929 ms 22.256 ms
10 2001:4860::8:0:5bb8 (2001:4860::8:0:5bb8) 23.337 ms 22.644 ms 21.461 ms
11 2001:4860::8:0:519f (2001:4860::8:0:519f) 29.765 ms 28.217 ms 27.716 ms
12 2001:4860::8:0:519e (2001:4860::8:0:519e) 32.023 ms 31.731 ms 30.495 ms
13 2001:4860::2:0:66f (2001:4860::2:0:66f) 32.818 ms 31.796 ms 31.560 ms
14 * * *
15 ea-in-x69.1e100.net (2a00:1450:4013:c01::69) 32.582 ms [open] 30.973 ms 30.074 ms
~]$ sudo tcptraceroute6 -l 1500 www.google.com
traceroute to www.google.com (2a00:1450:4013:c01::6a) from 2a00:14f0:e000:d2::2, port 80, from port 57265, 30 hops max, 1500 bytes packets
1 gw-211.lon-03.gb.sixxs.net (2a00:14f0:e000:d2::1) 19.893 ms 19.459 ms 19.686 ms
2 gblon03.sixxs.net (2a00:14f0:1:2::5) 20.603 ms 21.020 ms 19.644 ms
3 2a00:14f0:1:2::2 (2a00:14f0:1:2::2) 29.664 ms 20.949 ms 19.695 ms
4 ae3.core-1.maylands.hml.uk.as29017.net (2a00:14f0:0:1::2d) 21.905 ms 20.943 ms 19.716 ms
5 xe-0-0-0.border-1.sov.lon.uk.as29017.net (2a00:14f0:0:1::9) 23.430 ms 22.485 ms 21.154 ms
6 xe-0-0-1.border-1.thn.lon.uk.as29017.net (2a00:14f0:0:1::19) 23.300 ms 22.084 ms 21.516 ms
7 lonap.he.net (2001:7f8:17::1b1b:1) 31.503 ms 24.424 ms 21.864 ms
8 * * *
9 * * *
It's probably one or two lines of configuration somewhere in their router that is causing the fault.
Problems with IPV6 on connect.facebook.net
I have found other sites that will not accept packets over 1280 bytes but still work under IPV6.
What do you exactly mean with that? If your tunnel has an MTU of 1280, then you can only send packets with a maximum size of 1280, anything larger needs fragmentation.
tcptraceroute6 -l 1500 www.google.com
That is a nonsense test which will just cause multiple fragmented packets to be sent.
Please just use tracepath + tracepath6, they will show if PathMTU works and if not, you need to kick the network of the hop that is dropping ICMPv6.
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Thursday, 02 October 2014 13:15:15
Here is tracepath6
~]$ tracepath6 www.cisco.com
1?: [LOCALHOST] 0.018ms pmtu 1280
1: gw-211.lon-03.gb.sixxs.net 19.853ms
1: gw-211.lon-03.gb.sixxs.net 19.630ms
2: gblon03.sixxs.net 19.635ms asymm 1
3: 2a00:14f0:1:2::2 20.411ms asymm 2
4: ae3.core-1.maylands.hml.uk.as29017.net 19.837ms asymm 3
5: xe-0-0-0.border-1.sov.lon.uk.as29017.net 21.448ms asymm 4
6: xe-0-0-1.border-1.thn.lon.uk.as29017.net 21.091ms asymm 5
7: lonap.he.net 21.306ms asymm 6
8: 10ge5-2.core1.lon2.he.net 26.754ms asymm 5
9: 2001:7f8:4::51cc:1 21.198ms asymm 5
10: 2a02:26f0:26::5f64:f21c 21.806ms reached
Resume: pmtu 1280 hops 10 back 6
I was doing tcptraceroute6 to show that a fragmented 1500 byte packet would make it to the endpoint. It should even if it is fragmented (hence the connection at 1500 bytes route to cloudflare working). ICMP is working properly but TCP isn't.
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Thursday, 02 October 2014 13:31:08
Here is tracepath6 to www.linkedin.com
~]$ tracepath6 www.linkedin.com
1?: [LOCALHOST] 0.060ms pmtu 1280
1: gw-211.lon-03.gb.sixxs.net 19.881ms
1: gw-211.lon-03.gb.sixxs.net 19.676ms
2: gblon03.sixxs.net 19.785ms asymm 1
3: 2a00:14f0:1:2::2 20.054ms asymm 2
4: ae3.core-1.maylands.hml.uk.as29017.net 19.844ms asymm 3
5: xe-0-0-0.border-1.sov.lon.uk.as29017.net 21.371ms asymm 4
6: xe-0-0-1.border-1.thn.lon.uk.as29017.net 21.696ms asymm 5
7: lonap.he.net 21.597ms asymm 6
8: 10ge5-2.core1.lon2.he.net 21.837ms asymm 5
9: 100ge1-1.core1.nyc4.he.net 87.420ms asymm 6
10: 100ge5-1.core1.ash1.he.net 93.850ms asymm 7
11: no reply
12: no reply
13: no reply
14: no reply
15: no reply
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply
Too many hops: pmtu 1280
Resume: pmtu 1280
Still doesn't explain why changing my MTU to 1280 on my interface allows me to get to these sites without any problem.
Problems with IPV6 on connect.facebook.net
Still doesn't explain why changing my MTU to 1280 on my interface allows me to get to these sites without any problem.
Because of TCP MSS:
https://en.wikipedia.org/wiki/Maximum_segment_size
By setting your local network to have a low MTU, TCP knows about it too and advertises a low MSS and as that is not being blocked at the remote network, things magically work... for TCP, not for anything else.
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Thursday, 02 October 2014 14:13:04
OK makes sense now. It's just so frustrating knowing there's an issue that is probably a simple fix that I can't do anything about.
Problems with IPV6 on connect.facebook.net
It is not a simple fix as there are politics and business decisions involved, next to stupid people who do not want to understand ICMPv6 and simple things like Path MTU discovery.
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Friday, 03 October 2014 10:02:15
Jeroen Massar wrote:
It is not a simple fix as there are politics and business decisions involved, next to stupid people who do not want to understand ICMPv6 and simple things like Path MTU discovery.
I understand that you think I'm stupid...that's your right. I work in Unix R&D for a major telecoms company. If my TCP/IP knowledge is a little sparse it's because I don't work with routers, switches and in depth TCP/IP every day. Akamai has had a look at their systems made a small change and voila now www.cisco.com works.
Problems with IPV6 on connect.facebook.net
I understand that you think I'm stupid...
Not you. That part is pointed towards the provider who is blocking/dropping ICMPv6 and thus are breaking Path MTU discovery.
Akamai has had a look at their systems made a small change and voila now www.cisco.com works.
Akamai is being stupid, as they should know a lot better how to test this and that filtering ICMPv6 is a bad thing.
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Friday, 03 October 2014 11:21:31
Jeroen Massar wrote:
> I understand that you think I'm stupid...
Not you. That part is pointed towards the provider who is blocking/dropping ICMPv6 and thus are breaking Path MTU discovery.
Sorry for the misunderstanding Jeroen, I thought you were talking about me. Akamai has found the problem and it looks like it's a fix for them to sort out...maybe you misunderstood me with the tcptraceroute. I have found it to be an invaluable tool when aomeone cannot get to a website or port. ICMP may work 100% to the site but if a person cannot get to a tcp port using 1500 byte TCP packets over IPV6 (or IPV4) it's still broken. I had an instance with a broken router where ICMP was working and TCP wasn't this was the tool which told me there was a problem. (remember I'm coming from a server side not a router side).
Akamai has had a look at their systems made a small change and voila now www.cisco.com works.
Akamai is being stupid, as they should know a lot better how to test this and that filtering ICMPv6 is a bad thing.
Problems with IPV6 on connect.facebook.net
ICMP may work 100% to the site
People who are blocking ICMP should not be allowed to operate a network. They clearly do not understand what ICMP is used for.
And no, "DoS" is not a reason, there are lots and lots easier ways to cause huge traffic amplification attacks that do not require ICMP in anyway.
but if a person cannot get to a tcp port using 1500 byte TCP packets over IPV6 (or IPV4) it's still broken.
Tcptraceroute fragments packets. As such, your tests are invalid as it is not visible that that is happening.
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Friday, 03 October 2014 12:28:05
Jeroen Massar wrote:
Tcptraceroute fragments packets. As such, your tests are invalid as it is not visible that that is happening.
I understand...hopefully Akamai will have a fix in place soon....
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Thursday, 02 October 2014 15:24:55
Packets stopping at gblon03 pop
~]$ tracepath6 www.linkedin.com
1?: [LOCALHOST] 0.061ms pmtu 1280
1: gw-211.lon-03.gb.sixxs.net 19.657ms
1: gw-211.lon-03.gb.sixxs.net 21.482ms
2: gblon03.sixxs.net 19.513ms asymm 1
3: no reply
4: no reply
5: no reply
6: no reply
7: no reply
8: no reply
9: no reply
10: no reply
11: no reply
12: no reply
13: no reply
14: no reply
15: no reply
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply
Too many hops: pmtu 1280
Resume: pmtu 1280
Problems with IPV6 on connect.facebook.net
2: gblon03.sixxs.net 19.513ms asymm 1
That "asymm" line hilights an issue. What do your routing tables look like?
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Thursday, 02 October 2014 16:01:45
IPV6 Routing Table
~]$ netstat -6 -nr
Kernel IPv6 routing table
Destination Next Hop Flag Met Ref Use If
::1/128 :: U 256 0 0 lo
::/96 :: !n 1024 0 0 lo
0.0.0.0/96 :: !n 1024 0 0 lo
2002:a00::/24 :: !n 1024 0 0 lo
2002:7f00::/24 :: !n 1024 0 0 lo
2002:a9fe::/32 :: !n 1024 0 0 lo
2002:ac10::/28 :: !n 1024 0 0 lo
2002:c0a8::/32 :: !n 1024 0 0 lo
2002:e000::/19 :: !n 1024 0 0 lo
2a00:14f0:e000:d2::/64 :: Un 256 1 2 sixxs
2a00:14f0:e000:80d2::/64 :: U 256 1 0 br0
3ffe:ffff::/32 :: !n 1024 0 0 lo
fe80::/64 :: U 256 0 0 eth0
fe80::/64 :: U 256 0 0 eth1
fe80::/64 :: Un 256 0 0 sixxs
fe80::/64 :: U 256 0 0 br0
fe80::/64 :: U 256 0 0 wlan0
::/0 2a00:14f0:e000:d2::1 UG 1 1 0 sixxs
::/0 :: !n -1 117217041 lo
::1/128 :: Un 0 1 23161 lo
2a00:14f0:e000:d2::/128 :: Un 0 1 0 lo
2a00:14f0:e000:d2::2/128 :: Un 0 1151773 lo
2a00:14f0:e000:80d2::/128 :: Un 0 1 0 lo
2a00:14f0:e000:80d2::1/128 :: Un 0 2303448 lo
fe80::/128 :: Un 0 1 0 lo
fe80::/128 :: Un 0 1 0 lo
fe80::/128 :: Un 0 1 0 lo
fe80::/128 :: Un 0 1 0 lo
fe80::/128 :: Un 0 1 0 lo
fe80::5c1b:7d1/128 :: Un 0 1 0 lo
fe80::250:fcff:fe60:725f/128 :: Un 0 1 0 lo
fe80::6666:b3ff:fe07:d7ea/128 :: Un 0 1 46201 lo
fe80::6666:b3ff:fe07:d7ea/128 :: Un 0 1 0 lo
fe80::8e89:a5ff:fe50:2d0f/128 :: Un 0 1 0 lo
ff00::/8 :: U 256 0 1 eth0
ff00::/8 :: U 256 0 0 eth1
ff00::/8 :: U 256 0 0 sixxs
ff00::/8 :: U 256 2 0 br0
ff00::/8 :: U 256 0 0 wlan0
::/0 :: !n -1 117217041 lo
Problems with IPV6 on connect.facebook.net
What old distribution / kernel is this that installs all those routes?
Also, please learn to use "ip -6 route show" as the netstat commands really are not appropriate anymore on Linux.
Problems with IPV6 on connect.facebook.net
Shadow Hawkins on Thursday, 02 October 2014 16:29:40
Jeroen Massar wrote:
What old distribution / kernel is this that installs all those routes?
Fedora 20 x86 Kernel 3.16.3-200.fc20.i686+PAE
Also, please learn to use "ip -6 route show" as the netstat commands really are not appropriate anymore on Linux.
Teaching an old dog new tricks ;)
ip -6 route show
unreachable ::/96 dev lo metric 1024 error -101
unreachable ::ffff:0.0.0.0/96 dev lo metric 1024 error -101
2001:503:231d::2:30 via 2a00:14f0:e000:d2::1 dev sixxs metric 0
cache
unreachable 2002:a00::/24 dev lo metric 1024 error -101
unreachable 2002:7f00::/24 dev lo metric 1024 error -101
unreachable 2002:a9fe::/32 dev lo metric 1024 error -101
unreachable 2002:ac10::/28 dev lo metric 1024 error -101
unreachable 2002:c0a8::/32 dev lo metric 1024 error -101
unreachable 2002:e000::/19 dev lo metric 1024 error -101
2600:1401:2::2 via 2a00:14f0:e000:d2::1 dev sixxs metric 0
cache
2a00:14f0:e000:d2::1 dev sixxs metric 0
cache
2a00:14f0:e000:d2::/64 dev sixxs proto kernel metric 256
2a00:14f0:e000:80d2:cd1a::1 dev br0 metric 0
cache
2a00:14f0:e000:80d2::/64 dev br0 proto kernel metric 256
2a00:fd80:0:7::4 via 2a00:14f0:e000:d2::1 dev sixxs metric 0
cache
unreachable 3ffe:ffff::/32 dev lo metric 1024 error -101
fe80::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth1 proto kernel metric 256
fe80::/64 dev sixxs proto kernel metric 256
fe80::/64 dev br0 proto kernel metric 256
fe80::/64 dev wlan0 proto kernel metric 256
default via 2a00:14f0:e000:d2::1 dev sixxs metric 1
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Friday, 03 October 2014 10:08:53
Have heard back from Akamai. They have changed the IP address of www.cisco.com and it is now reachable under IPV6 (and 1500 byte fragmented tcp packets).
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Sunday, 05 October 2014 11:28:33
www.cisco.com is 99.9% working now. Occasionaly hangs in a browser whilst loading www.static-cisco.com. connect.facebook.net works now.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Tuesday, 07 October 2014 21:27:41
Charlie MacDonald wrote:
Have heard back from Akamai. They have changed the IP address of www.cisco.com and it is now reachable under IPV6 (and 1500 byte fragmented tcp packets).
It haven't changed anything here. It appears Akamai only changed it for your particular cluster/node. Which contact form did you use? I can only find a sales form - and their page is loading painfully slow (most likely because of the same issue as www.cisco.com etc.). :)
/Mark
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Wednesday, 08 October 2014 11:39:52
Mark Holm wrote:
It haven't changed anything here. It appears Akamai only changed it for your particular cluster/node. Which contact form did you use? I can only find a sales form - and their page is loading painfully slow (most likely because of the same issue as www.cisco.com etc.). :)
/Mark
I got an email address for Akamai's NOC from LINX and they responded within a day to my email. I'm connected to gblon03.sixxs.net
~]$ tracepath6 www.cisco.com
1?: [LOCALHOST] 0.018ms pmtu 1280
1: gw-211.lon-03.gb.sixxs.net 19.584ms
1: gw-211.lon-03.gb.sixxs.net 19.431ms
2: gblon03.sixxs.net 19.720ms asymm 1
3: 2a00:14f0:1:2::2 19.734ms asymm 2
4: ae3.core-1.maylands.hml.uk.as29017.net 19.591ms asymm 3
5: xe-0-0-0.border-1.sov.lon.uk.as29017.net 20.930ms asymm 4
6: xe-0-0-1.border-1.thn.lon.uk.as29017.net 32.937ms asymm 5
7: lonap.he.net 24.108ms asymm 6
8: 10ge5-2.core1.lon2.he.net 28.326ms asymm 5
9: 2001:7f8:4::cb9:1 22.824ms asymm 5
10: xe-4-2-0.lon11.ip6.gtt.net 21.636ms asymm 4
11: gtt-gw.ip6.gtt.net 22.221ms asymm 5
12: ae1-90g.ar1.lhr1.uk.as4436.gtt.net 24.491ms asymm 6
13: no reply
14: 2a02:26f0:27::5c7a:7e44 22.191ms reached
Resume: pmtu 1280 hops 14 back 7
note:www.linkedin.com works in a browser but not with tracepath6 and I get occasional browser hangs with www.cisco.com and www.static-cisco.com where I have to stop loading and refresh the page. I have to shut off IPV6 in my network to even get to www.akami.com -- which is still not responding over IPV6 :( I have sent them another email quoting this forum thread as well.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Thursday, 09 October 2014 10:31:26
Got another response back from Akamai
Hi Charlie,
I don't believe MTU is the problem here. Fragmentation is done by the host in IPv6 but specifying 1500 bytes in tracepath forces it to use that big of a packet. I might be wrong but I believe that flag would cancel out host fragmentation and attempt to use the a full 1500 byte packet.
Regardless, we see this same behavior on our end but have no problems connecting to our servers (we get a 403 but that proves we're reaching our servers):....
Can you try using telnet on port 80 to test connecting to www.akamai.com?
$ telnet www.akamai.com 80
Trying 2a02:26f0:b8:292::22d9...
Connected to e8921.dscx.akamaiedge.net.
Escape character is '^]'.
GET /
HTTP 1.0
HTTP/1.0 400 Bad Request
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 216
Expires: Thu, 09 Oct 2014 10:23:46 GMT
Date: Thu, 09 Oct 2014 10:23:46 GMT
Connection: close
<HTML><HEAD>
<TITLE>Bad Request</TITLE>
</HEAD><BODY>
<H1>Bad Request</H1>
Your browser sent a request that this server could not understand.<P>
Reference #7.20f31402.1412850226.0
</BODY>
</HTML>
Connection closed by foreign host.
Problems with IPV6 on connect,facebook.net
GET / HTTP/1.1
Host: www.something.com
User-Agent: telnet
<extra enter>
But your better way to check things is to check Wireshark with a real browser. Chrome, Firefox and Safari (and likely others) have a "network view" in their developer mode which shows the requests and their statuses, though wireshark will be useful too as then you see the actual packets.
Problems with IPV6 on connect,facebook.net
Playing a little bit, it seems I am getting random problems too.
One fetch will succeed and the next will not. Both happens on Linux (telnet) and Mac (Chrome).
But as it is random, and caching is involved and other such magics, it is really hard to determine what goes wrong.
Let me ping a friend of mine over their who should be a bit better at this than their NOC/support folks...
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Friday, 10 October 2014 05:47:57
That's exactly what I'm getting. One time it will work the next it won't (or it hangs after one or two different servers are contacted --- www.static-cisco.com is a good example). It was working flawlessly for a short time under both Linux and Windows but not anymore :( .
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Thursday, 09 October 2014 10:48:46
www.akamai.com is now working in a browser under ipv6 as is Cisco (www.static-cisco.com still hangs occasionally). Have tried it in Linux using Firefox and Windows under Firefox, Chrome and IE. All bring me straight to the page. Whatever Whatever Akamai did on their end worked.
Mark..are you still having problems?
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Tuesday, 14 October 2014 07:12:18
www.akamai.com and www.cisco.com are not working again. Both sites hang in a browser (Linux Firefox / Windows Chrome, IE & Firefox).
Problems with IPV6 on connect,facebook.net
![[it]](/s/countries/it.gif) Shadow Hawkins on Tuesday, 14 October 2014 09:39:51
Charlie MacDonald wrote:
Having problems for the past few days connecting to connect.facebook.net...
I was experiencing the same problem during the last month. No connect.facebook.net , no cisco.com and I was really thinking about dropping ipv6 support.
While reading this post, I came up with an idea and changed MTU on my Fortigate router.
config system interface
edit "sixxs.net"
set mtu-override enable
set mtu 1280
end
After this, everything was solved and all those sites appeared again.
If it's OK for admin, I would add this to Fortigate guide at https://www.sixxs.net/wiki/Fortigate
Problems with IPV6 on connect,facebook.net
While reading this post, I came up with an idea and changed MTU on my Fortigate router.
Tunnel MTU is per default 1280 (unless changed in the webinterface, see the FAQ), if you misconfigure it then indeed PathMTU will break and you will have big problems with connecting to sites.
The problem is though, that even folks who have correctly configured their MTU and allowing ICMPv6 etc, are having issues with Akamai-hosted sites as the remote end (at Akamai) seems to be ignoring PathMTU.
config system interface
edit "sixxs.net"
set mtu-override enable
set mtu 1280
end
After this, everything was solved and all those sites appeared again.
If it's OK for admin, I would add this to Fortigate guide at https://www.sixxs.net/wiki/Fortigate
The wiki is user supplied, one can add whatever you want (well, unless it harms somebody etc...)
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Tuesday, 14 October 2014 10:41:02
Jeroen Massar wrote:
Tunnel MTU is per default 1280 (unless changed in the webinterface, see the FAQ), if you misconfigure it then indeed PathMTU will break and you will have big problems with connecting to sites.
The problem is though, that even folks who have correctly configured their MTU and allowing ICMPv6 etc, are having issues with Akamai-hosted sites as the remote end (at Akamai) seems to be ignoring PathMTU.
You're absolutely right.
I'm using two sixxs tunnel, one at home with aiccu and one at work with static ip.
At home everything is set to default and I'm having problems accessing cisco.com and other sites, except from the router itself which is probably due to the aiccu interface MTU which is set to 1280 from your web interface.
From other PCs on the LAN, I can't access cisco.com without changing Ethernet MTU to 1280 or disabling IPv6. Obviously, I won't change MTU to all my interfaces to 1280 due to what seems an Akami misconfiguration.
On the other side, at work, every network interface is set to their default MTU (1500 for most Ethernet interfaces). I noticed that the virtual sixxs.net interface on my Fortigate router hadn't any special setting for MTU, while on the web interface this is set to 1280. I just changed the virtual sixxs.net interface MTU to 1280, which I think it's right, because it is the same as what it is on your side.
That solved my problem at work, while at home I'm waiting Akamai for doing something.
Problems with IPV6 on connect,facebook.net
![[us]](/s/countries/us.gif) Shadow Hawkins on Wednesday, 15 October 2014 01:22:08
Great thread, I've been experiencing sporadic issues with connect.facebook.net and static.licdn.com for weeks with a 1480 tunnel (chi2). Usually a shift-refresh (or 3) would fix them, so I didn't have many reproducible sites for debugging, but cisco.com and akamai.com do the trick.
Before I blindly change my tunnels to 1280, any tips on getting tracepath to reliably reproduce the timeouts I'm seeing in my browser? Right now I can get neither site to load, but tracepath6 works beautifully (also includes trying any of the IPs directly):
# tracepath6 www.cisco.com
1?: [LOCALHOST] 0.116ms pmtu 1500
1: 2001:4978:307:xxx::1 1.434ms
1: 2001:4978:307:xxx::1 1.389ms
2: 2001:4978:307:xxx::1 1.438ms pmtu 1480
2: gw-1553.chi-02.us.sixxs.net 47.132ms
3: unassigned.v6.your.org 50.249ms asymm 2
4: sixxs.ge-0.0.0-30.core1.chi.bb6.your.org 47.509ms asymm 3
5: unknown.ipv6.scnet.net 52.100ms asymm 4
6: ae3-71.cr1.ord6.scnet.net 51.856ms asymm 5
7: ae4-0.cr1.ord1.scnet.net 46.448ms asymm 6
8: ae-6.r06.chcgil09.us.bb.gin.ntt.net 49.422ms asymm 7
9: 2001:418:2401:7::c6ad:2af 48.480ms reached
Resume: pmtu 1480 hops 9 back 57
# tracepath6 www.akamai.com
1?: [LOCALHOST] 0.115ms pmtu 1500
1: 2001:4978:307:xxx::1 1.454ms
1: 2001:4978:307:xxx::1 1.384ms
2: 2001:4978:307:xxx::1 1.436ms pmtu 1480
2: gw-1553.chi-02.us.sixxs.net 44.234ms
3: unassigned.v6.your.org 45.007ms asymm 2
4: sixxs.ge-0.0.0-30.core1.chi.bb6.your.org 47.656ms asymm 3
5: unknown.ipv6.scnet.net 46.168ms asymm 4
6: ae3-71.cr1.ord6.scnet.net 52.323ms asymm 5
7: ae1-0.cr1.ord1.scnet.net 48.693ms asymm 6
8: ae-6.r06.chcgil09.us.bb.gin.ntt.net 46.411ms asymm 7
9: 2001:418:2401:7::c6ad:2af 46.183ms reached
Resume: pmtu 1480 hops 9 back 57
Problems with IPV6 on connect,facebook.net
Before I blindly change my tunnels to 1280
The MTU of your tunnel can be as big as the underlying IPv4 path allows.
Just dropping it to 1280 does not make sense and will not solve anything.
If a remote site is misconfigured/broken, nothing on your side fully fix it.
any tips on getting tracepath to reliably reproduce the timeouts I'm seeing in my browser?
Tracepath is not the tool for that.
The timeouts you are seeing are caused by some packets being lost somewhere along the path.
Likely these are the ICMPv6 Packet Too Bigs being sent back from an intermediary host to the remote host (Akamai). As long as they drop/ignore those ICMPv6 packets, there is little to be done to solve this problem.
I am busy with Akamai getting them to debug this problem though. It seems that some parts of the clusters are having this issue, likely an issue in the loadbalancer itself... thus lots of fun ;)
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Wednesday, 15 October 2014 11:47:56
Jeroen Massar wrote:
> Before I blindly change my tunnels to 1280
The MTU of your tunnel can be as big as the underlying IPv4 path allows.
Just dropping it to 1280 does not make sense and will not solve anything.
If a remote site is misconfigured/broken, nothing on your side fully fix it.
Right, a 1280 MTU does not "solve," nor is it a "fix," it's just a workaround that happens to work for my link (I can now reach connect.facebook.net and all the other Akamai-driven sites).
Thanks for your effort with Akamai.
any tips on getting tracepath to reliably reproduce the timeouts I'm seeing in my browser?
Tracepath is not the tool for that.
The timeouts you are seeing are caused by some packets being lost somewhere along the path.
Likely these are the ICMPv6 Packet Too Bigs being sent back from an intermediary host to the remote host (Akamai). As long as they drop/ignore those ICMPv6 packets, there is little to be done to solve this problem.
I am busy with Akamai getting them to debug this problem though. It seems that some parts of the clusters are having this issue, likely an issue in the loadbalancer itself... thus lots of fun ;)
Problems with IPV6 on connect,facebook.net
(I can now reach connect.facebook.net and all the other Akamai-driven sites).
You are likely just hitting the nodes that do work, it changes randomly which is why the problem is hard to diagnose.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Friday, 17 October 2014 05:05:37I am busy with Akamai getting them to debug this problem though. It seems that some parts of the clusters are having this issue, likely an issue in the loadbalancer itself... thus lots of fun ;)
Thank You Jeroen...much appreciated :)
Problems with IPV6 on connect,facebook.net
![[fr]](/s/countries/fr.gif) Shadow Hawkins on Friday, 17 October 2014 09:41:21
I support everyone's efforts, thanks.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Monday, 03 November 2014 11:35:27
www.akamai.com is now working under ipv6 www.cisco.com is not
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Monday, 03 November 2014 11:41:21
Charlie MacDonald wrote:
www.akamai.com is now working under ipv6 www.cisco.com is not
connect.facebook.net still broken though :(
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Tuesday, 04 November 2014 12:45:22
I still have problems with Akamai driven websites.
I sent an email to Akamai, unanswered.
I contacted Cisco and they investigated. They sought explanations for them and Akamai said they hoped that it will be resolved without any more clarification.
I think now Akamai is unable to supply a decent IPv6 service, or maybe, they don't want invest in suitable equipment. This is a fail.
Problems with IPV6 on connect,facebook.net
Their 'equipment' are likely just off-the-shelf PCs; hence they have full control.
Yesterday this was happening to them:
http://mailman.nanog.org/pipermail/nanog/2014-November/071090.html
Yep, returning multiple Location headers in a HTTP 302.
Seems they have quite a few broken nodes out there. That breaking might be happening because of fixes though. Here is for hoping...
The only real way things will get fixed is if their paying customers complain though; but as it is a random issue, it will be hard for them to see that there is a problem in the first place.
Problems with IPV6 on connect,facebook.net
![[be]](/s/countries/be.gif) Shadow Hawkins on Wednesday, 05 November 2014 10:21:00
Francois Moreau wrote:
I still have problems with Akamai driven websites.
Yes, I also have major issues with them too. (akamai.com itself, connect.facebook.com, linkedin, ...).
To test I use something like :
curl -v "http://\[2a02:26f0:60:289::eed\]/en_GB/all.js" -H "Host: connect.facebook.net" -o /dev/null
(so I always get the same server since I force its IP).
And sometimes it works, sometimes it doesn't (like doing it 5 times in a row, the first 3 will work, then fail twice ...). Looking at the packets, sometimes for the attempts that works I see the first MSS size set in the akamai SYN-ACK response is set lower. But sometimes I get attempts that works with the default MSS set in the initial SYN-ACK (and just decreased later).
Problems with IPV6 on connect,facebook.net
(so I always get the same server since I force its IP).
That IP address is a big load balancer with a variety of machines behind it.
Hence, even hitting the same IP does not mean you are hitting the same backend machine. Which is why 'sometimes it works, sometimes it does not'.
Which makes debugging this issue so tricky when one has absolutely no insight in how the Akamai setup is.
It seems the best thing to do is public humilation of Akamai, as their techs seem completely unresponsive for this matter.
Likely because "I checked and it worked" is what they see and it might not affect all clusters.
Seeing their F-up with the multiple Location headers though a few days ago shows that they likely have more issues than solving their IPv6 problems, hence it might just also be very low prio...
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Thursday, 06 November 2014 05:19:10
Charlie MacDonald wrote:
Charlie MacDonald wrote:
Looks like I spoke too soon. www.akamai.com, www.cisco.com and connect.facebook.net all broken again :(
www.akamai.com is now working under ipv6 www.cisco.com is not
connect.facebook.net still broken though :(
Problems with IPV6 on connect,facebook.net
Charlie MacDonald wrote:
Charlie MacDonald wrote:
You see them break and then work again as only a few of the nodes in a cluster seem to be broken. Hence one connection they work, the next they will not.
I got a statement that they are looking at it, but as their are more important things, it might take quite a while unfortunately....
Little one can do about this, completely in the hands of Akamai.
Charlie MacDonald wrote:
Looks like I spoke too soon. www.akamai.com, www.cisco.com and connect.facebook.net all broken again :(
www.akamai.com is now working under ipv6 www.cisco.com is not
connect.facebook.net still broken though :(
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Friday, 07 November 2014 03:52:40
Unofficial update: this thread is not being ignored and active work is in-progress regarding issues observed here.
Problems with IPV6 on connect,facebook.net
Erik Nygren wrote:
Unofficial update: this thread is not being ignored and active work is in-progress regarding issues observed here.
Thanks!
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Monday, 17 November 2014 21:55:08
Unofficial update: things should look better now (and should have been looking better for a few days now). If anyone is still observing any issues, please post or send me (or noc@akamai.com) details.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Wednesday, 19 November 2014 08:10:08
Looks to be working for me :)
Problems with IPV6 on connect,facebook.net
![[hu]](/s/countries/hu.gif) Shadow Hawkins on Monday, 10 November 2014 06:09:08
I just wanted to add that I also noticed this problem. Tracepath6 can't get to connect.facebook.com nor platform.linkedin.com, making user experience with certain sites horrible. I added "::1 <problematic site>" to /etc/hosts on my pc a horrible workaround.
Problems with IPV6 on connect,facebook.net
Fabian Tamas Laszlo wrote:
I just wanted to add that I also noticed this problem. Tracepath6 can't get to connect.facebook.com nor platform.linkedin.com, making user experience with certain sites horrible. I added "::1 <problematic site>" to /etc/hosts on my pc a horrible workaround.
If you are going to "block" sites, add a route to loopback (::1) instead, as that does not mess with DNS in anyway and gives proper !N - network unreachable that indicates to the browser that it can retry on a different address.
Also, you might want to look into the Disconnect.me and Ghostery plugins for avoiding the need to hit most of that kind of sites.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Monday, 10 November 2014 09:35:45
That would be much better indeed. Thank you!
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Wednesday, 12 November 2014 10:37:47
Fabian Tamas Laszlo wrote:
That would be much better indeed. Thank you!
Well, if you're on linux and OK with horrible workarounds, this works :
ip6tables -t mangle -A OUTPUT -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220
But unfortunately that only fixes it for me and not for my users ... For the time being, just globally disabling IPv6 seems the safest option, which I really hate ...
I'd much rather that Akamai just disable IPv6 on their broken clusters. Bad IPv6 is much more a brake to adaption than no IPv6. I honestly have a hard time understanding how they can have such a hard time debugging/fixing this. It takes like 5 min to reproduce, in 10 requests, I'll have about half that fails.
Problems with IPV6 on connect,facebook.net
Well, if you're on linux and OK with horrible workarounds, this works :
MSS clamping is the workaround that Google uses. The problem with it is that you are effectively restricting the MTU for the Internet to 1280.
ip6tables -t mangle -A OUTPUT -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220 But unfortunately that only fixes it for me and not for my users ...
Change OUTPUT to FORWARD and it also applies to other users. That is, if that box is the one terminating the tunnel or forwarding the traffic.
I'd much rather that Akamai just disable IPv6 on their broken clusters.
They are working on it, as have people from them commented here in this thread.
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Wednesday, 12 November 2014 14:14:54MSS clamping is the workaround that Google uses. The problem with it is that you are effectively restricting the MTU for the Internet to 1280.
Sure by my IPv6 tunnel is 1280 anyway.
Change OUTPUT to FORWARD and it also applies to other users. That is, if that > box is the one terminating the tunnel or forwarding the traffic.
But it isn't the main router, this is just on my local laptop.
I'd much rather that Akamai just disable IPv6 on their broken clusters. They are working on it, as have people from them commented here in this thread.
Yes, I saw that but:
1) If you can't fix something in a reasonable time frame (and given this thread is like 1.5 month old, I think we're clearly past that), then try to apply a work around that at least makes things less broken until you come up with the right fix. After all delivering content is like their main job.
2) I totally trust that they're working on it, but I still have a hard time imagining what kind of issue could take that long to trace down. I would be really interested to hear all the tech details of the issue and how it was tracked down and fixed once it's done, unfortunately I'm not sure we ever will :(
My current take on it:
What I observed:
1) The issue only appears when something in the path has a MTU < 1500.
2) It can be observed from different tunnel provider with very different path up to the couple last nodes.
3) When doing multiple requests to the _same_ IP, sometime it will work, sometime it won't.
To which I concluded:
- With (1), I'd say this clearly points to a PMTUD issue.
- With (2) and (3), I think we can safely exclude that the problem is somewhere else than Akamai itself.
Now the final IP is obviously some kind of load balancer, which could work in one of two ways:
- Sort of like a reverse proxy, accepting connection and forwarding them to a backend. But I think we can exclude this mode because if it was doing that, the TCP session would effectively be terminated at this machine and it would either always work or always fail.
- Like an intelligent dispatcher that dispatches the raw packet to several backend machines without really processing them. It obviously still needs to know which machine handles which connection so that all packets from 1 tcp session go to the same backend. This could just be done statelessly with using a hash(ip,port) for instance.
In this latter case, the cause of the issues could either be :
- That stateless dispatcher doesn't properly dispatch the ICMP errors and sends them to the wrong backend. Sometimes, it will randomly get it right and so that works for those.
- Some of the backend machines are misconfigured and filter out the ICMP errors.
PS: And yes, I know all/part of this is just speculation and in the end pretty useless, but I still like the intellectual exercise of imagining what could be the problem.
Cheers,
Sylvain
Problems with IPV6 on connect,facebook.net
But it isn't the main router, this is just on my local laptop.
I've considered adding a "Clamp MSS" per-tunnel user toggleable option to sixxsd.
But it just will hide so many issues and make debugging more difficult for other things as something magically changes, that I don't see it as a proper fix.
Not clamping or otherwise modifying packets reveals these kind of problems.
[..] but I still have a hard time imagining what kind of issue could take that long to trace down.
A problem like described here:
https://tools.ietf.org/html/draft-v6ops-pmtud-ecmp-problem-00
As until I got that note, I had not thought of that problem either. (Then again, I don't play with loadbalancers or other such devices that don't terminate IPv6 but do make decisions on packet flow).
It is not unlikely that just a few of their platforms that they are using have this problem. Which means that they need to escalate it to their vendors, get those vendors on site, and to debug it.
The fun part is of course that it happens once in a while and likely only with a non-1500 MTU, hence likely not tested in the lab that way and thus fun to debug when they (the vendor) do not have such a setup. Imagine a lot of shouting like happening here and other threads of "look it does no work" "it does work" repeat...
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Monday, 10 November 2014 18:20:51
Fabian Tamas Laszlo wrote:
I just wanted to add that I also noticed this problem. Tracepath6 can't get to connect.facebook.com nor platform.linkedin.com, making user experience with certain sites horrible. I added "::1 <problematic site>" to /etc/hosts on my pc a horrible workaround.
Is this an inability to reach them entirely (as opposed to a PMTUD issue)? Do you have a traceroute of this?
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Saturday, 22 November 2014 20:30:56
I realized in this last wednesday night the akamai network was quite reachable, my navigation was all right for today... Did they have find a solution ?
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Sunday, 23 November 2014 10:05:10
Francois Moreau wrote:
I realized in this last wednesday night the akamai network was quite reachable, my navigation was all right for today... Did they have find a solution ?
It's hidden about 1/4 of the way up the page. They have sorted it :)
Problems with IPV6 on connect,facebook.net
Shadow Hawkins on Wednesday, 26 November 2014 00:14:20
Reposting so this shows up on the bottom of the page...
Unofficial update: things should look better now (and should have been looking better for a few days now). If anyone is still observing any issues, please post or send me (or noc@akamai.com) details.
|
![[ch]](/s/countries/ch.gif)
on Sunday, 28 September 2014 14:26:15
Posting is only allowed when you are