|
Cloudfare IPv6 routing issues (imgur, reddit and others)
![[no]](/s/countries/no.gif) Shadow Hawkins on Thursday, 12 July 2012 08:55:53
[subject changed from "What to do when I see a routing problem on the net?"]
I know this is not a SixXs problem, but I want this to get fixed and asking whom I should contact:
I am unable to reach www./i.imgur.com. Both from our native IPv6 and via my SixXS tunnel there is a problem:
user@host:~$ traceroute6 i.imgur.com
traceroute to i.imgur.com (2400:cb00:2048:1::6ca2:e809), 30 hops max, 80 byte packets
1 gw.sysrq.info (2001:14b8:16c::1) 0.243 ms 0.416 ms 0.551 ms
2 gw-468.hel-01.fi.sixxs.net (2001:14b8:100:1d3::1) 41.854 ms 42.966 ms 43.451 ms
3 fihel01.sixxs.net (2001:14b8:0:3401::6) 45.666 ms 45.859 ms 46.107 ms
4 lah2-er70.ip6.dnaip.fi (2001:14b8:0:3401::2) 46.321 ms 46.446 ms 46.497 ms
5 lah1-tr1.ip6.dnaip.fi (2001:14b8::74) 53.580 ms 54.180 ms 54.594 ms
6 hel5-tr3.ip6.dnaip.fi (2001:14b8::18) 56.643 ms 56.457 ms 56.629 ms
7 * hel2-tr2.ip6.dnaip.fi (2001:14b8::188) 51.270 ms 51.143 ms
8 * * *
9 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 51.436 ms 52.754 ms 51.604 ms
10 xe-9-1-0.bar1.Stockholm1.Level3.net (2001:1900:5:2:2::33d) 52.021 ms 54.225 ms 54.212 ms
11 2001:1900:5:3::20a (2001:1900:5:3::20a) 54.398 ms 54.389 ms 55.049 ms
12 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 55.582 ms 55.569 ms 55.555 ms
13 2001:1900:5:3::20a (2001:1900:5:3::20a) 59.857 ms 59.837 ms 59.696 ms
14 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 51.568 ms 51.683 ms 51.859 ms
15 2001:1900:5:3::20a (2001:1900:5:3::20a) 57.468 ms 57.454 ms 56.786 ms
16 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 54.544 ms 55.183 ms 55.379 ms
...
29 2001:1900:5:3::20a (2001:1900:5:3::20a) 51.416 ms 59.971 ms 52.439 ms
30 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 74.779 ms 75.005 ms 74.208 ms
vho@vho-45:~$ traceroute6 i.imgur.com
traceroute to cf-ssl11496-protected-i.imgur.com (2400:cb00:2048:1::6ca2:e80b) from 2001:4610:4:****, 30 hops max, 24 byte packets
1 2001:4610:4:****:1 (2001:4610:4:****:1) 0.449 ms 0.294 ms 0.282 ms
2 2001:4610:4::1 (2001:4610:4::1) 0.812 ms 0.824 ms 0.759 ms
3 ti0043a400-ge1-2-2-16.ti.telenor.net (2001:4600:9:100::5) 0.465 ms 0.734 ms 0.632 ms
4 * * *
5 * * *
6 * * *
7 * * *
8 ti9006b400.ti.telenor.net (2001:4600::c) 36.831 ms 32.313 ms 32.299 ms
9 decix.tge4-1.ar1.fra1.de.nlayer.net (2001:7f8::1154:0:1) 33.329 ms 32.613 ms 32.69 ms
10 xe-1-3-0.cr1.cdg1.fr.nlayer.net (2001:590::4516:8e0a) 50.856 ms 49.322 ms 111.942 ms
11 * * *
12 xe-1-3-0.cr1.cdg1.fr.nlayer.net (2001:590::4516:8e0a) 57.565 ms 57.428 ms 57.563 ms
13 * * *
...
30 xe-1-3-0.cr1.cdg1.fr.nlayer.net (2001:590::4516:8e0a) 133.917 ms 134.291 ms 134.235 ms
This has been a problem for 2 days now. Is there something I can do? Someone I should alert?
Best regards,
Vidar Hoel
What to do when I see a routing problem on the net?
The folks at Cloudfare seems the right people for that, guess for endusers the imgur forums might be a location to complain.
They decided not to announce the overlapping /32 and only announce the separate /48's and as that is PA (Provider Aggregated) space, there are lots of ISPs who properly filter those announcements.
As such, complain to Cloudfare (2400:cb00::/32) that they need to run their network properly.
See amongst others also NANOG for a discussion about this. As such Cloudfare is aware of it, they just do not understand yet how many people are noticing this.
Happy Eyeballs which is in most browsers and OSs now is covering it up nicely though.
What to do when I see a routing problem on the net?
Shadow Hawkins on Wednesday, 11 July 2012 12:13:08
Thanks for the info. Sent a bug-report to Cloudfare now, I hope they resolv it soon.
Happy Eyeballs does not seem to work. Tried latest Firefox and Chrome on my Ubuntu 12.04 LTS, and I can not access imgur.com. So I have problems contacting them via their contact-form, but it seems like I could send a email to sarah@imgur.com.
What to do when I see a routing problem on the net?
I do not think they will as they are determined that they are "right" in announcing /48s out of PA space and try to convince people they are right, even though the RIR policies clearly dictate that they are not...
As they do not have a global network interconnecting their independent sites they also cannot easily announce the /32 as it would break their whole network design.
As such, an easy fix for them there is not, except for them to make 1 site a global site and announce the /32 from there, which will be nasty.
Their 'real' solution would be to get PI space in chunks of /48 like other providers did do, but that would require renegotiations with the RIRs for new address space and then renumbering which quite kills their whole "we are doing IPv6" argument.... somebody in their network team made quite a booboo...
What to do when I see a routing problem on the net?
Shadow Hawkins on Thursday, 12 July 2012 08:19:32
Yesterday I wrote a bugreport til Cloudflare, and sent a "Technical support"-email to imugr.com. Included all traceroutes and a rephrase of what you said could be the problem.
And Cloudfare answered first, thanked me, said they where looking into the problems. Later Imgur.com responded, thanked for all the details and said they was talks with Cloudfare about this issue. And what do you know, I saw new IPv6 addresses for i.imgur.com already with small TTLs. None of them worked, so I got redirected to IPv4. So it was not 100%, but it worked.
But this morning: All IPv6, all working! I can finally browse Reddit while at work :-)
What to do when I see a routing problem on the net?
I do not think you are using IPv6 as there is a routing loop inside Level(3) and various other locations.
As long as the /32 for the Cloudfare prefix is not announced their IPv6 will be broken in a lot of places which properly filter PA address space.
$ traceroute6 i.imgur.com
traceroute to cf-ssl11496-protected-i.imgur.com (2400:cb00:2048:1::6ca2:e808) from 2001:14b8:0:3401::6, 30 hops max, 24 byte packets
1 lah2-er70.ip6.dnaip.fi (2001:14b8:0:3401::2) 0.71 ms 2.745 ms 1.308 ms
2 lah1-tr1.ip6.dnaip.fi (2001:14b8::74) 9.486 ms 14.085 ms 9.97 ms
3 hel5-tr3.ip6.dnaip.fi (2001:14b8::18) 9.45 ms 9.432 ms 9.429 ms
4 hel2-tr2.ip6.dnaip.fi (2001:14b8::188) 9.492 ms 10.491 ms 9.47 ms
5 * * *
6 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.652 ms 9.547 ms 9.46 ms
7 xe-9-1-0.bar1.Stockholm1.Level3.net (2001:1900:5:2:2::33d) 40.136 ms 9.541 ms 9.471 ms
8 2001:1900:5:3::20a (2001:1900:5:3::20a) 23 ms 9.418 ms 9.318 ms
9 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.343 ms 9.414 ms 9.346 ms
10 2001:1900:5:3::20a (2001:1900:5:3::20a) 9.454 ms 9.343 ms 21.542 ms
11 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.347 ms 9.349 ms 9.349 ms
12 2001:1900:5:3::20a (2001:1900:5:3::20a) 12.288 ms 9.417 ms 14.875 ms
13 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.384 ms 9.376 ms 9.366 ms
14 2001:1900:5:3::20a (2001:1900:5:3::20a) 9.527 ms 10.38 ms 9.385 ms
15 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 51.071 ms 9.374 ms 9.385 ms
16 2001:1900:5:3::20a (2001:1900:5:3::20a) 19.816 ms 9.432 ms 14.796 ms
17 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.447 ms 26.683 ms 9.414 ms
18 2001:1900:5:3::20a (2001:1900:5:3::20a) 12.233 ms 15.322 ms 9.595 ms
19 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 18.717 ms 9.65 ms 9.504 ms
20 2001:1900:5:3::20a (2001:1900:5:3::20a) 9.499 ms 25.044 ms 16.204 ms
21 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.631 ms 9.439 ms 9.448 ms
22 2001:1900:5:3::20a (2001:1900:5:3::20a) 20.483 ms 9.47 ms 23.55 ms
23 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.448 ms 9.448 ms 9.465 ms
24 2001:1900:5:3::20a (2001:1900:5:3::20a) 12.008 ms 9.489 ms 14.34 ms
25 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.541 ms 9.473 ms 9.733 ms
26 2001:1900:5:3::20a (2001:1900:5:3::20a) 20.335 ms 9.696 ms 14.728 ms
27 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.586 ms 9.482 ms 9.535 ms
28 2001:1900:5:3::20a (2001:1900:5:3::20a) 28.787 ms 15.393 ms 9.504 ms
29 ae16.bar1.Stockholm1.level3.net (2001:1900:5:3::209) 9.524 ms 9.559 ms 9.512 ms
30 2001:1900:5:3::20a (2001:1900:5:3::20a) 10.832 ms 9.537 ms 23.46 ms
$ traceroute6 i.imgur.com
traceroute to i.imgur.com (2400:cb00:2048:1::6ca2:e80f), 30 hops max, 80 byte packets
1 2001:788:2:117:ffff:ffff:ffff:ffff (2001:788:2:117:ffff:ffff:ffff:ffff) 2.192 ms 2.212 ms 2.298 ms
2 2001:788:2:27::42 (2001:788:2:27::42) 0.641 ms 0.801 ms 0.645 ms
3 2a02:2528:303:1::1 (2a02:2528:303:1::1) 1.617 ms 1.713 ms 1.552 ms
4 2a02:2528:2:8::1 (2a02:2528:2:8::1) 2.676 ms 2.432 ms 2.406 ms
5 ge3-2.br01.gva253.ip-max.net (2a02:2528:102:1::2) 1.981 ms 2.190 ms 1.882 ms
6 10gigabitethernet1-2.core1.zrh1.he.net (2001:7f8:c:8235:194:42:48:80) 5.777 ms 7.180 ms 7.075 ms
7 10gigabitethernet3-2.core1.fra1.he.net (2001:470:0:10d::1) 15.879 ms 15.895 ms 15.850 ms
8 10gigabitethernet1-4.core1.ams1.he.net (2001:470:0:47::1) 24.986 ms 25.008 ms 24.989 ms
9 2001:7f8:1::a500:4436:1 (2001:7f8:1::a500:4436:1) 28.177 ms 29.837 ms 29.751 ms
10 vlan-73.ar1.ams3.nl.nlayer.net (2001:590::4516:8b7e) 30.096 ms 29.905 ms 29.919 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * as13335.xe-3-0-2.ar1.ams3.nl.nlayer.net (2001:590::3f8d:df1e) 2043.317 ms !H *
$ traceroute6 i.imgur.com
traceroute to i.imgur.com (2400:cb00:2048:1::6ca2:e80f) from 2001:838:1:1:210:dcff:fe20:7c7c, port 33434, from port 55320, 30 hops max, 60 byte packets
1 ge-1-3-0.breda.ipv6.concepts-ict.net (2001:838:1:1::1) 0.389 ms 0.375 ms 0.339 ms
2 gi2-11.nikhef.ipv6.concepts-ict.net (2001:838:5:a::1) 1.942 ms 1.794 ms 1.846 ms
3 ge-0.ams-ix.amstnl02.nl.bb.gin.ntt.net (2001:7f8:1::a500:2914:1) 2.205 ms 2.186 ms 2.165 ms
4 ae-2.r03.amstnl02.nl.bb.gin.ntt.net (2001:728:0:2000::12a) 2.507 ms 2.873 ms 2.715 ms
5 2001:590:2:2f::1 (2001:590:2:2f::1) 15.571 ms 15.624 ms 15.390 ms
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
Yep, it is all broken now, that is why it "works" as you will fall back silently to IPv4.
What to do when I see a routing problem on the net?
Shadow Hawkins on Friday, 13 July 2012 13:13:56
You're right: When I use my SixXS tunnel, it's all IPv4 and I get the same traceroute as you.
When I'm on Telenors IPv6 network, it works (and I verified it via tcpdump):
vho@vho-45:~$ traceroute6 i.imgur.com
traceroute to cf-ssl11496-protected-i.imgur.com (2400:cb00:2048:1::6ca2:e809) from 2001:4610:4:xxxx, 30 hops max, 24 byte packets
1 2001:4610:4:xxxx::1 (2001:4610:4:xxxx::1) 0.422 ms 0.295 ms 0.287 ms
2 2001:4610:4::1 (2001:4610:4::1) 0.769 ms 0.715 ms 0.853 ms
3 ti0043a400-ge1-2-2-16.ti.telenor.net (2001:4600:9:100::5) 0.491 ms 0.435 ms 0.435 ms
4 * * *
5 * * *
6 * * *
7 * * *
8 ti9006b400.ti.telenor.net (2001:4600::c) 32.486 ms 32.276 ms 32.292 ms
9 decix.tge4-1.ar1.fra1.de.nlayer.net (2001:7f8::1154:0:1) 33.646 ms 32.828 ms 32.95 ms
10 xe-1-3-0.cr1.cdg1.fr.nlayer.net (2001:590::4516:8e0a) 48.815 ms 48.781 ms 48.674 ms
11 xe-4-3-0.cr1.lhr1.uk.nlayer.net (2001:590::4516:8e41) 49.289 ms 54.636 ms 49.122 ms
12 xe-2-3-0.cr1.ams2.nl.nlayer.net (2001:590::4516:8e0e) 53.719 ms 53.915 ms 109.266 ms
13 ae1-20g.ar1.ams3.nl.nlayer.net (2001:590::4516:8b3d) 55.746 ms 57.686 ms 55.478 ms
14 as13335.xe-3-0-2.ar1.ams3.nl.nlayer.net (2001:590::3f8d:df1e) 54.19 ms 54.026 ms 53.961 ms
15 2400:cb00:2048:1::6ca2:e809 (2400:cb00:2048:1::6ca2:e809) 53.835 ms 54.447 ms 53.798 ms
Anyways, I have to contact Cloudfare again, since only half my IPv6 connections work.
What to do when I see a routing problem on the net?
Shadow Hawkins on Saturday, 14 July 2012 15:16:53
Updates from Cloudfare again: They think they have solved the problem. And a traceroute6 from my tunnel (fihel01) and from my ISP at work confirms it.
Is it working at your end now?
What to do when I see a routing problem on the net?
Dropping IPv6 solves it for imgur indeed ;)
$ host -t aaaa i.imgur.com
i.imgur.com is an alias for wpc.4220.edgecastcdn.net.
wpc.4220.edgecastcdn.net is an alias for gs1.wpc.edgecastcdn.net.
$ host -t aaaa gs1.wpc.edgecastcdn.net.
gs1.wpc.edgecastcdn.net has no AAAA record
As for their own IPv6, it won't get really fixed as long as they do not announce their prefix as a /32 as they are supposed to do, it is PA.
|
![[ch]](/s/countries/ch.gif)
on Wednesday, 11 July 2012 11:26:41
Posting is only allowed when you are