SixXS::Sunset 2017-06-06

Ticket ID: SIXXS #12035109
Ticket Status: Remote Problem

PoP: gblon02 - Goscomb Technologies (London)

Incoming packet loss - possible MTU issue
[gb] Shadow Hawkins on Thursday, 24 July 2014 17:08:59
I've recently been having issues with some websites (normally SSL connections) timeing out - https://ssl.comodo.com for example. SYN & ACK packets are ok, but larger packets never arrive as expected. I've also noticed problems with whois, so I've decided to use that data here. This seems similar to Issue 9469190 I've upped my MTU to 1480 (as an attempt to resolve the problem) on the pfSense box (2.1.2-RELEASE), as well as increasing it on the tunnel page. We have a leased line, so the MTU for our WAN is 1500. The client computer: $ uname -a Linux BaylyJ5 3.11.0-24-generic #42-Ubuntu SMP Fri Jul 4 21:19:31 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux $ whois --version Version 5.0.26. cat /etc/issue Linux Mint 16 Petra \n \l If the whois result is quite small, then the packets are received. $ whois 81.144.211.211 16:44:20.496508 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428 > 2001:67c:2e8:22::c100:687.43: Flags [S], seq 2110280876, win 28800, options [mss 1440,sackOK,TS val 368133969 ecr 0,nop,wscale 7], length 0 16:44:20.524135 IP6 2001:67c:2e8:22::c100:687.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428: Flags [S.], seq 1240332740, ack 2110280877, win 14280, options [mss 1440,sackOK,TS val 715991193 ecr 368133969,nop,wscale 7], length 0 16:44:20.524403 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428 > 2001:67c:2e8:22::c100:687.43: Flags [.], ack 1, win 225, options [nop,nop,TS val 368133976 ecr 715991193], length 0 16:44:20.524497 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428 > 2001:67c:2e8:22::c100:687.43: Flags [P.], ack 1, win 225, options [nop,nop,TS val 368133976 ecr 715991193], length 25 16:44:20.551542 IP6 2001:67c:2e8:22::c100:687.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428: Flags [.], ack 26, win 112, options [nop,nop,TS val 715991221 ecr 368133976], length 0 16:44:20.552083 IP6 2001:67c:2e8:22::c100:687.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428: Flags [P.], ack 26, win 112, options [nop,nop,TS val 715991221 ecr 368133976], length 197 16:44:20.552648 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428 > 2001:67c:2e8:22::c100:687.43: Flags [.], ack 198, win 234, options [nop,nop,TS val 368133983 ecr 715991221], length 0 16:44:20.558215 IP6 2001:67c:2e8:22::c100:687.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428: Flags [FP.], seq 198:1546, ack 26, win 112, options [nop,nop,TS val 715991226 ecr 368133976], length 1348 16:44:20.558981 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428 > 2001:67c:2e8:22::c100:687.43: Flags [.], ack 1547, win 256, options [nop,nop,TS val 368133984 ecr 715991226], length 0 16:44:20.559104 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428 > 2001:67c:2e8:22::c100:687.43: Flags [F.], seq 26, ack 1547, win 256, options [nop,nop,TS val 368133984 ecr 715991226], length 0 16:44:20.586153 IP6 2001:67c:2e8:22::c100:687.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.50428: Flags [.], ack 27, win 112, options [nop,nop,TS val 715991256 ecr 368133984], length 0 However, if the whois record is quite long, it times out (the packets in this case are out of order): $ whois tipstrade.net 16:52:51.561100 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [S], seq 4128570047, win 28800, options [mss 1440,sackOK,TS val 368261735 ecr 0,nop,wscale 7], length 0 16:52:51.683034 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [S.], seq 2704334763, ack 4128570048, win 13280, options [mss 1340,sackOK,TS val 1331805842 ecr 368261735,nop,wscale 7], length 0 16:52:51.683341 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [.], ack 1, win 225, options [nop,nop,TS val 368261765 ecr 1331805842], length 0 16:52:51.683442 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [P.], ack 1, win 225, options [nop,nop,TS val 368261765 ecr 1331805842], length 16 16:52:51.804896 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [.], ack 17, win 104, options [nop,nop,TS val 1331805964 ecr 368261765], length 0 16:52:51.806365 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [F.], seq 2818, ack 17, win 104, options [nop,nop,TS val 1331805966 ecr 368261765], length 0 16:52:51.806620 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [.], ack 1, win 225, options [nop,nop,TS val 368261796 ecr 1331805964,nop,nop,sack 1 {2818:2819}], length 0 16:52:51.806761 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [P.], ack 17, win 104, options [nop,nop,TS val 1331805966 ecr 368261765], length 161 16:52:51.806886 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [.], ack 17, win 104, options [nop,nop,TS val 1331805966 ecr 368261765], length 1328 16:52:51.807210 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [.], ack 1, win 225, options [nop,nop,TS val 368261796 ecr 1331805964,nop,nop,sack 1 {2657:2819}], length 0 16:52:51.807291 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [.], ack 17, win 104, options [nop,nop,TS val 1331805966 ecr 368261765], length 1328 16:52:51.807547 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [.], ack 1329, win 248, options [nop,nop,TS val 368261796 ecr 1331805966,nop,nop,sack 1 {2657:2819}], length 0 16:52:51.807922 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [.], ack 2819, win 270, options [nop,nop,TS val 368261797 ecr 1331805966], length 0 16:52:51.808027 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [F.], seq 17, ack 2819, win 270, options [nop,nop,TS val 368261797 ecr 1331805966], length 0 16:52:51.929073 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [.], ack 17, win 104, options [nop,nop,TS val 1331806088 ecr 368261796], length 1328 16:52:51.929227 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [.], ack 17, win 104, options [nop,nop,TS val 1331806088 ecr 368261796], length 1328 16:52:51.929315 IP6 2001:503:f189:1060::74.43 > 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142: Flags [.], ack 18, win 104, options [nop,nop,TS val 1331806089 ecr 368261797], length 0 16:52:51.929821 IP6 2a01:348:2c2:100:5246:5dff:fe8f:5c6d.58142 > 2001:503:f189:1060::74.43: Flags [.], ack 2819, win 270, options [nop,nop,TS val 368261827 ecr 1331805966,nop,nop,sack 1 {1:2657}], length 0 Pinging with the maximum packet size: $ ping -nc 5 -s 1472 facebook.com PING facebook.com (173.252.110.27) 1472(1500) bytes of data. 1480 bytes from 173.252.110.27: icmp_seq=1 ttl=82 time=97.8 ms 1480 bytes from 173.252.110.27: icmp_seq=2 ttl=82 time=97.6 ms 1480 bytes from 173.252.110.27: icmp_seq=3 ttl=82 time=97.6 ms 1480 bytes from 173.252.110.27: icmp_seq=4 ttl=82 time=97.5 ms 1480 bytes from 173.252.110.27: icmp_seq=5 ttl=82 time=97.5 ms --- facebook.com ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 97.579/97.654/97.807/0.083 ms $ ping6 -nc 5 -s 1432 facebook.com PING facebook.com(2a03:2880:2110:df07:face:b00c:0:1) 1432 data bytes 1440 bytes from 2a03:2880:2110:df07:face:b00c:0:1: icmp_seq=1 ttl=51 time=98.3 ms 1440 bytes from 2a03:2880:2110:df07:face:b00c:0:1: icmp_seq=2 ttl=51 time=98.4 ms 1440 bytes from 2a03:2880:2110:df07:face:b00c:0:1: icmp_seq=3 ttl=51 time=98.2 ms 1440 bytes from 2a03:2880:2110:df07:face:b00c:0:1: icmp_seq=4 ttl=51 time=98.8 ms 1440 bytes from 2a03:2880:2110:df07:face:b00c:0:1: icmp_seq=5 ttl=51 time=98.3 ms --- facebook.com ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4005ms rtt min/avg/max/mdev = 98.209/98.437/98.870/0.361 ms Tracepath6: $ tracepath6 ssl.comodo.com 1?: [LOCALHOST] 0.059ms pmtu 1500 1: 2a01:348:2c2:100::1 1.054ms 1: 2a01:348:2c2:100::1 1.062ms 2: 2a01:348:2c2:100::1 1.046ms pmtu 1480 2: gw-1148.lon-02.gb.sixxs.net 14.203ms 3: gblon02.sixxs.net 14.310ms asymm 2 4: ge-0-0-5-20.cs0.thw.uk.goscomb.net 33.790ms asymm 3 5: xe-3-1.core00.the.uk.hso-group.net 14.713ms asymm 4 6: xe-3-1.core00.gs2.uk.hso-group.net 19.948ms asymm 5 7: xe-3-1.core00.sov.uk.hso-group.net 14.953ms asymm 6 8: ae2.edge00.sov.uk.hso-group.net 14.736ms asymm 7 9: xe-4-0-1.edge6.London1.Level3.net 14.765ms asymm 8 10: vl-52.car3.London1.Level3.net 15.276ms asymm 9 11: Internet-Connections-Ltd.car3.London1.Level3.net 15.156ms asymm 10 12: 2a01:3f8:0:c::1 15.227ms asymm 7 13: 2a01:3f8:0:8::2 22.211ms asymm 8 14: 2a01:3f8:0:108::2 29.301ms asymm 15 15: ge-1-0-7-2012.h6edccrt.hex67.lon.edge.ccanet.co.uk 21.779ms asymm 11 16: no reply <snip> 31: no reply Too many hops: pmtu 1480 Resume: pmtu 1480 $ tracepath6 google.com 1?: [LOCALHOST] 0.046ms pmtu 1500 1: domain.not.configured 1.026ms 1: domain.not.configured 1.030ms 2: domain.not.configured 1.052ms pmtu 1480 2: gw-1148.lon-02.gb.sixxs.net 14.208ms 3: gblon02.sixxs.net 14.178ms asymm 2 4: ge-0-0-5-20.cs0.thw.uk.goscomb.net 19.516ms asymm 3 5: xe-3-1.core00.the.uk.hso-group.net 26.776ms asymm 4 6: xe-3-1.core00.gs2.uk.hso-group.net 14.837ms asymm 5 7: xe-3-1.core00.sov.uk.hso-group.net 21.998ms asymm 6 8: no reply <snip> 31: no reply Too many hops: pmtu 1480 Resume: pmtu 1480 So, are there any suggestions as to what I can do? Many thanks
State change: remoteproblem Locked
[ch] Jeroen Massar SixXS Staff on Thursday, 24 July 2014 17:50:43
Message is Locked
The state of this ticket has been changed to remoteproblem
Incoming packet loss - possible MTU issue
[ch] Jeroen Massar SixXS Staff on Thursday, 24 July 2014 17:55:30
https://ssl.comodo.com for example.
Comodo is known to be filtering ICMPv6. Little we can do about. We also do not control any of the other destinations, you'll have to contact those networks directly. Also, make sure that your local firewall is sending packets too big properly and not dropping such messages to the floor.
If the whois result is quite small, then the packets are received.
You might want to check tracepath6 for each of those destinations.
$ ping6 -nc 5 -s 1432 facebook.com
As you are not blocking fragmentation, those packets can still be fragmented. See the -M option ("-M do" is what you want).
Incoming packet loss - possible MTU issue
[gb] Shadow Hawkins on Friday, 25 July 2014 11:01:10
Comodo is known to be filtering ICMPv6. Little we can do about.
I've confirmed that the same occurs on my home connection - native IPv6 over pppoe - the connection times out. What's happening now with ssl.comodo.com is this: * MTU=1280 TCP:443 Connection would timeout * MTU=1480 TCP:443 Connection is reset immediately with "no route to destination" (which is interesting as I can ping it) - the IMCPv6 "Destination Unreachable" packet comes from an IP not listed in the tracepath6 output. I've given up with comodo here - clearly they're doing something wrong here if it doesn't even work over native IPv6. I fully appreciate you have no control over other people's networks - you wouldn't believe how often I get asked why [some other website] isn't working. As for the whois, I made a mistake with my interpretation of the network capture: * MTU=1280 whois was timing out. * MTU=1480 whois capture packets with a few out of sequence packets & duplicate ACKs, and then an IPv4 connection. I had thought is was failing over to IPv4. However, the whois request to whois.crsnic.net is working on IPv6 - the subsequent IPv4 whois request was to our registrar. Many apologies for the rubbish information. So, it appears that whois.crsnic.net appears to be having some issues with lower MTUs - I'll have a play around with different values and see if I can find the point at which they start failing.
Also, make sure that your local firewall is sending packets too big properly and not dropping such messages to the floor.
I'm going to have to ask a stupid question here - what's the best way of doing this?
Incoming packet loss - possible MTU issue
[ch] Jeroen Massar SixXS Staff on Friday, 25 July 2014 11:07:19
* MTU=1480 TCP:443 Connection is reset immediately with "no route to destination" (which is interesting as I can ping it)
That is because something is doing "firewalling", hence packets it does not want are rejected with the wrong error code. Though, it could also mean that the tool that you are using for displaying the error code decodes it wrong. This as there typically is a sub-code, "admin filtered" / "admin prohibited" being one.
The IMCPv6 "Destination Unreachable" packet comes from an IP not listed in the tracepath6 output.
Which IP? Do note that asynchronous routing can cause such a behaviour, next to hidden "firewall" devices.
> Also, make sure that your local firewall is sending packets too big properly and not dropping such messages to the floor.
I'm going to have to ask a stupid question here - what's the best way of doing this?
Depends on the tools involved. Enabling logging and checking that logging (or at least counters) is a good first step.
Incoming packet loss - possible MTU issue
[gb] Shadow Hawkins on Friday, 25 July 2014 15:38:24
Which IP?
It's 2a02:1788:0:865::5001 - the tracepath6 for this IP is the exact same as that for ssl.comodo.com (shown below). I didn't mention the address because I saw that the 2a02:1788::/32 block's authority is ns0.comdodns.com baylyj@BaylyJ5 ~ $ ping6 -c 3 -n -s 1432 ssl.comodo.com PING ssl.comodo.com(2a02:1788:4fd:cd::c742:cdf2) 1432 data bytes 1440 bytes from 2a02:1788:4fd:cd::c742:cdf2: icmp_seq=1 ttl=53 time=23.2 ms 1440 bytes from 2a02:1788:4fd:cd::c742:cdf2: icmp_seq=2 ttl=53 time=22.1 ms 1440 bytes from 2a02:1788:4fd:cd::c742:cdf2: icmp_seq=3 ttl=53 time=22.6 ms --- ssl.comodo.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 22.153/22.675/23.254/0.483 ms baylyj@BaylyJ5 ~ $ tracepath6 -n ssl.comodo.com 1?: [LOCALHOST] 0.020ms pmtu 1480 1: 2a01:348:2c2:100::1 1.152ms 1: 2a01:348:2c2:100::1 1.004ms 2: 2a01:348:6:47b::1 14.109ms 3: 2a01:348:0:4:0:3:1:1 14.155ms asymm 2 4: 2a01:348:0:4:0:3:0:1 16.089ms asymm 3 5: 2a01:348::65:0:1 25.561ms asymm 4 6: 2a01:348::63:1:1 22.834ms asymm 5 7: 2a01:348::71:1:1 23.348ms asymm 6 8: 2a01:348::50:0:1 14.946ms asymm 7 9: 2001:1900:5:2:2::951 14.945ms asymm 8 10: 2001:1900:101:2::3 15.436ms asymm 9 11: 2001:4c08:2003::2 15.231ms asymm 10 12: 2a01:3f8:0:d::2 21.433ms asymm 11 13: 2a01:3f8:0:9::1 21.972ms asymm 8 14: 2a01:3f8:0:108::2 34.535ms asymm 15 15: 2a02:1788:ff:51ae::b2ff:51ae 21.701ms asymm 11 16: no reply <snip> 31: no reply Too many hops: pmtu 1480 Resume: pmtu 1480 baylyj@BaylyJ5 ~ $ curl -6 https://ssl.comodo.com curl: (7) Failed connect to ssl.comodo.com:443; Network is unreachable Here is a link to a wireshark/tcpdump capture of the above commands Wireshark describes the ICMPv6 packed for the failed https connection as "Destination unreachable (no route to destination): Internet Control Message Protocol v6 Type: Destination Unreachable (1) Code: 0 (no route to destination) Interestingly, I logged onto my machine at home and tried opening ssl.comodo.com - the ssl handshake starts, but never completes and the browser gives up after 30s. Here's a link to a capture (if you're any way interested). Finally, I tried it on a hosted server of ours - it works perfectly, but then, it's native IPv6 (MTU 1452). So... as you say - something is doing firewalling, for what reason I really don't know. As for dropping "packet too big", it wasn't - I wasn't seeing as many of the packets as I expected, until I realised the MTU for a route was being cached - but what you were asking gave me a much better idea of keywords to look for. For example, I've discovered that pfSense is dropping fragemented packets - confirmed by Bug #2762 - which doesn't help. Thanks for the pointers and suggestions. I'm still going to have a play with various MTU values - if only for academic reasons, which is pretty much the point of this tunnel.
Incoming packet loss - possible MTU issue
[ch] Jeroen Massar SixXS Staff on Friday, 25 July 2014 15:50:26
John Bayly wrote:
Which IP?
It's 2a02:1788:0:865::5001 - the tracepath6 for this IP is the exact same as that for ssl.comodo.com (shown below). I didn't mention the address because I saw that the 2a02:1788::/32 block's authority is ns0.comdodns.com
Then it is a device that is in the path, not all hosts play nicely with TTLs or filter and thus traceroute gives different results. Also, paths might be asynchronous.
baylyj@BaylyJ5 ~ $ ping6 -c 3 -n -s 1432 ssl.comodo.com
You really need to set "-M do" for this to mean anything.
[..] 14: 2a01:3f8:0:108::2 34.535ms asymm 15 15: 2a02:1788:ff:51ae::b2ff:51ae 21.701ms asymm 11 16: no reply <snip> 31: no reply
There is some kind of packet filtering happening and thus the pMTU is likely meaningless.
baylyj@BaylyJ5 ~ $ curl -6 https://ssl.comodo.com
curl: (7) Failed connect to ssl.comodo.com:443; Network is unreachable
Note that you might also be filtered already due to you probing their network a lot...
Finally, I tried it on a hosted server of ours - it works perfectly, but then, it's native IPv6 (MTU 1452).
An MTU < 1500 is not 'native', likely some kind of tunneling happening.
So... as you say - something is doing firewalling, for what reason I really don't know.
Comodo is a 'security' company, hence not odd that they do 'security' even if they break things that they clearly do not know about.
As for dropping "packet too big", it wasn't
Note that these ICMPv6 PtBs are sent by the PoP, not by your host, as that is where the MTU change happens. You should be able to see it in the Live Tunnel Status though as there the last host that got one will be listed. Do note that the other side (eg ssl.comodo.com) could send you a Packet Too Big that will reach to your host completely.
- I wasn't seeing as many of the packets as I expected, until I realised the MTU for a route was being
cached - but what you were asking gave me a much better idea of keywords to look for. For example,
I've discovered that pfSense is dropping fragemented packets - confirmed by
Bug #2762 - which doesn't help.
Fragments should NEVER exist on a network. There is no reason for them (except for making things complex that do not need to be complex). They are currently still in the IPv6 spec, but there have been various discussions of getting rid of them completely. TCP can handle the packet-cut-up and for UDP one should just send smaller packets, similar solutions exist for other protocols like TCP.
Incoming packet loss - possible MTU issue
[gb] Shadow Hawkins on Friday, 25 July 2014 16:24:31
I was supposed to be using the "-M do" switch - I probably went too far up the bash history when executing it and didn't notice as the result was as expected.
Note that you might also be filtered already due to you probing their network a lot...
The thought had crossed my mind.
> Finally, I tried it on a hosted server of ours - it works perfectly, but then, it's native IPv6 (MTU 1452). An MTU < 1500 is not 'native', likely some kind of tunneling happening.
That was a typo - I posted the maximum icmp payload by mistake <sigh> Didn't know about the "Live Tunnel Status" page - very handy.
Fragments should NEVER exist on a network. There is no reason for them (except for making things complex that do not need to be complex). They are currently still in the IPv6 spec, but there have been various discussions of getting rid of them completely. TCP can handle the packet-cut-up and for UDP one should just send smaller packets, similar solutions exist for other protocols like TCP.
Fair point - my 1st port of call was to check out whether fragments were acceptable - being as they're in the spec I assumed that they were - I know path MTU discovery is expected to be used. I hadn't realised that they may be removed.
Incoming packet loss - possible MTU issue
[ch] Jeroen Massar SixXS Staff on Friday, 25 July 2014 23:16:34
I have been able to reach out to a few folks at Comodo, hopefully they will look into and attempt to resolve the problem on their side (if it is at their place, but it looks likely).

Please note Posting is only allowed when you are logged in.
Static Sunset Edition of SixXS
©2001-2017 SixXS - IPv6 Deployment & Tunnel Broker