|
Ticket ID: SIXXS #1174109 Ticket Status: Resolved PoP: uschi02 - Your.Org, Inc. (Chicago, Illinois)
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
![[us]](/s/countries/us.gif) Carmen Sandiego on Friday, 14 August 2009 01:26:44
Any known issues? <sigh>
No config changes on my end in the past few months, but tunnel stopped working sometime yesterday (I think). I can't get at the graphs because:
https://www.sixxs.net/home/tunnelinfo/?14277
...returns an error page reading only "An internal server error occurred. Please try again later."
v6 ping is dead:
=====================
Pinging 2001:4978:f:b8::1
from 2001:4978:f:b8::2 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 2001:4978:f:b8::1:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
=====================
v4 ping is fine:
=====================
Pinging 216.14.98.22 with 32 bytes of data:
Reply from 216.14.98.22: bytes=32 time=40ms TTL=45
Reply from 216.14.98.22: bytes=32 time=40ms TTL=45
Reply from 216.14.98.22: bytes=32 time=41ms TTL=45
Reply from 216.14.98.22: bytes=32 time=51ms TTL=45
Ping statistics for 216.14.98.22:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 40ms, Maximum = 51ms, Average = 43ms
=====================
v6 trace from there is thus irrelevant; v4 trace (ignore boneheaded-uRPF hops 12 and 13 between ATT and your.org; they've always been there):
=====================
Tracing route to sixxs.cx01.chi.bb.your.org [216.14.98.22]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 6 ms 8 ms 70.159.160.31
3 12 ms 13 ms 19 ms 70.159.176.114
4 13 ms 9 ms 8 ms 70.159.176.112
5 12 ms 11 ms 12 ms 70.159.177.96
6 11 ms 10 ms 11 ms axr01asm-so-0-1-0.bellsouth.net [65.83.238.41]
7 11 ms 9 ms 9 ms 12.83.0.194
8 13 ms 15 ms 10 ms 12.83.0.187
9 10 ms 16 ms 13 ms 65.83.238.198
10 9 ms 10 ms 14 ms cr1.attga.ip.att.net [12.122.141.50]
11 9 ms 9 ms 14 ms ggr2.attga.ip.att.net [12.122.141.97]
12 * * * Request timed out.
13 * * * Request timed out.
14 39 ms 40 ms 41 ms ae0-5.core2.chi.bb.your.org [216.14.98.30]
15 40 ms 42 ms 46 ms sixxs.cx01.chi.bb.your.org [216.14.98.22]
=====================
Config on my end reads as follows (WinXP netsh):
=====================
add v6v4tunnel interface="SixXS" localaddress=66.156.66.24 remoteaddress=216.14.98.22
add address interface="SixXS" address=2001:4978:f:b8::2
add route prefix=::/0 interface="SixXS" metric=1
add route prefix=2001:4978:f:b8::1/64 interface="SixXS" metric=0
=====================
Finally, a v6 trace from my employer's workstation to the your.org side of the tunnel (trimmed leader):
=====================
traceroute to 2001:4978:f:b8::1 (2001:4978:f:b8::1) from [...], 30 hops max, 16 byte packets
[...]
6 v104.core1.ash1.he.net (2001:470:0:40::2) 23.224 ms 22.692 ms 22.85 ms
7 10gigabitethernet1-2.core1.nyc4.he.net (2001:470:0:36::2) 29.423 ms 29.231 ms 29.031 ms
8 10gigabitethernet1-2.core1.chi1.he.net (2001:470:0:4e::1) 51.15 ms 50.669 ms 50.763 ms
9 2001:470:0:7f::2 (2001:470:0:7f::2) 52.416 ms 52.656 ms 52.68 ms
10 * * *
11 gw-185.chi-02.us.sixxs.net (2001:4978:f:b8::1) 52.504 ms 52.172 ms 52.829 ms
=====================
State change:
The state of this ticket has been changed to
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Any known issues? <sigh>
Why the "sigh", if you require an SLA, then please don't hesitate to contact your local ISP and pay for it. They will then happily support you 24/7 356 days a year.
...returns an error page reading only "An internal server error occurred. Please try again later."
I can't reproduce that.
v6 ping is dead: ===================== Pinging 2001:4978:f:b8::1 from 2001:4978:f:b8::2 with 32 bytes of data: Request timed out.
works fine from noc.sixxs.net:
64 bytes from 2001:4978:f:b8::1: icmp_seq=1 ttl=57 time=102 ms
can't ping6 the ::2 variant though.
v6 trace from there is thus irrelevant; v4 trace (ignore boneheaded-uRPF hops 12 and 13 between ATT and your.org; they've always been there):
Nothing wrong with uRPF unless it is misconfigured.
[..]
1 <1 ms <1 ms <1 ms 192.168.1.1
RFC1918. I do hope you have proper firewalling and state tracking configured and are properly forwarding packets?
Config on my end reads as follows (WinXP netsh):
That might be what you configure, but what is actually configured?
Finally, a v6 trace from my employer's workstation to the your.org side of the tunnel (trimmed leader):
How do you expect us to try and traceroute back to an address that you hide?
Not that it is relevant as it is a far remote host outside of the network.
Nevertheless, clearly as you can see you can reach the <tun>::1 address is alive, the tunnel thus is configured. Problem thus must lie between the PoP and your endpoint.
I'd suggest you recheck your local tunnel and firewalling state.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Monday, 17 August 2009 01:46:31
Oh Jeroen, always the cynic (re: SLA comment <grin>). It's just my general unhappiness with the slow uptake of IPv6 in the wild, though I'm actively working to get it into the core of my employer's network (they're a tier-2). Your devotion to SixXS is a very good thing, and I didn't mean to tick you off or anything like that. :)
Packets of protocol IPV6 (41) are properly configured to traverse the gateway transparently, and are doing so for my HE (tunnelbroker.net) tunnel. That one is normally pointed to a different site, but currently re-pointed here for comparison. That tunnel's config is identically constructed, save for the lack of a default route:
add v6v4tunnel interface="tunnelbroker.net" localaddress=66.156.66.24 remoteaddress=216.66.22.2
add address interface="tunnelbroker.net" address=2001:470:8:1fc::2
add route prefix=2001:470:7:1fc::1/64 interface="tunnelbroker.net" metric=0
Yes, the 2001:470:8:1fc::2 endpoint address is externally reachable without problems, and changing the ::/0 route to go through HE works fine. The only differences in config between this and my uschi02 connection are the v4 remote endpoint and v6 addresses of both endpoints.
As of this writing, the uschi02 endpoint is still unreachable, however.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Monday, 17 August 2009 18:55:22Nothing wrong with uRPF unless it is misconfigured.
(btw...)
No, nothing wrong with it, but much like other technologies, it's misconfigured far too much. Traceroute to anything single-homed to France Telecom/opentransit sometime and you'll see a clear example. Useless diagnostics and metrics ahoy! :-)
Can't access Tunnel info on www.sixxs.net and tunnel no longer pingable.
Shadow Hawkins on Friday, 14 August 2009 20:14:05
From Wednesday this week, 12th Aug, I can no longer access the tunnel information on the site and connectivity through the tunnel is now down.
The tunnel is automatically configured using aiccu and that setup works fine.
Aug 14 11:43:22 (none) local7.info syslog: Successfully retrieved tunnel information for T19931
Aug 14 11:43:22 (none) local7.info syslog: AICCU running as PID 396
No changes have been made to the router at my end for over 6 months.
Pop status says it is up, however it is strange that I am unable to see any detail for that tunnel on the sixxs site. Error from site is for the last 3 days is "An internal server error occurred.", but only for that tunnel, the other one I have reports details just find and is also working.
Kind regards
Andrew
Can't access Tunnel info on www.sixxs.net and tunnel no longer pingable.
Shadow Hawkins on Saturday, 15 August 2009 07:58:45
Update page is working again on sixxs and tunnel came back up.
Odd but I'm happy anyway :)
Problem resolved..
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Friday, 14 August 2009 21:15:12
I am having a similar issue with uschi02, from my tunnel endpoint I cannot ping 2001:4978:f:263::1, tcpdump of the 6in4 iface shows the ICMP6 packet leaving, tcpdump of the egress interface shows the encapsulated packet leaving to the v4 of the tunnel provider:
---
[mernisse@imladris: /var/www/www.ub3rgeek.net (5540)]$sudo tcpdump -ni em0 ho >
tcpdump: listening on em0, link-type EN10MB
^Z[1] + Suspended sudo tcpdump -ni em0 host 216.14.98.22
[mernisse@imladris: /var/www/www.ub3rgeek.net (5541)]$bg
[1] sudo tcpdump -ni em0 host 216.14.98.22
[mernisse@imladris: /var/www/www.ub3rgeek.net (5542)]$ping6 2001:4978:f:263:: >
PING6(56=40+8+8 bytes) 2001:4978:f:263::2 --> 2001:4978:f:263::1
15:06:34.862754 2001:4978:f:263::2 > 2001:4978:f:263::1: icmp6: echo request (encap)
15:06:35.864878 2001:4978:f:263::2 > 2001:4978:f:263::1: icmp6: echo request (encap)
15:06:36.854862 2001:4978:f:263::2 > 2001:4978:f:263::1: icmp6: echo request (encap)
^C
--- 2001:4978:f:263::1 ping6 statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
---
From another v6 host I have off gblon02 I cannot ping my ushci02 host, the ICMP6 packet leaves uschi01 towards me, gets de-capsulated, replied to, encapsulated and sent to uschi02 and this disappears, similar to when I ping from the host:
---
[mernisse@imladris: /var/www/www.ub3rgeek.net (5543)]$sudo tcpdump -ni em0 ho >
Password:
tcpdump: listening on em0, link-type EN10MB
15:12:13.024280 2a01:348:6:217::2 > 2001:4978:f:263::2: icmp6: echo request (encap)
15:12:13.024368 2001:4978:f:263::2 > 2a01:348:6:217::2: icmp6: echo reply (encap)
15:11:13.024279 2a01:348:6:217::2 > 2001:4978:f:263::2: icmp6: echo request (encap)
15:11:13.024367 2001:4978:f:263::2 > 2a01:348:6:217::2: icmp6: echo reply (encap)
---
[mernisse@imladris: /var/www/www.ub3rgeek.net (5544)]$i gif19122 icmp6 <
tcpdump: listening on gif19122, link-type NULL
15:12:50.743536 2a01:348:6:217::2 > 2001:4978:f:263::2: icmp6: echo request
15:12:50.743590 2001:4978:f:263::2 > 2a01:348:6:217::2: icmp6: echo reply
15:12:51.748798 2a01:348:6:217::2 > 2001:4978:f:263::2: icmp6: echo request
15:12:51.748829 2001:4978:f:263::2 > 2a01:348:6:217::2: icmp6: echo reply
15:12:52.748314 2a01:348:6:217::2 > 2001:4978:f:263::2: icmp6: echo request (encap)
---
I would expect this means something in the POP is misbehaving because as you say you can ping the gw side of my uschi02 tunnel from the world:
---
mernisse@valinor:/etc/ufw$ ping6 gw-612.chi-02.us.sixxs.net
PING gw-612.chi-02.us.sixxs.net(gw-612.chi-02.us.sixxs.net) 56 data bytes
64 bytes from gw-612.chi-02.us.sixxs.net: icmp_seq=1 ttl=58 time=101 ms
64 bytes from gw-612.chi-02.us.sixxs.net: icmp_seq=2 ttl=58 time=102 ms
--- gw-612.chi-02.us.sixxs.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 101.388/101.724/102.061/0.463 ms
---
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Friday, 14 August 2009 21:25:46
Also, fwiw, when logged in as myself (mje8-ripe) my tunnel page also returns a HTTP 500 'Internal Server Error' Page: https://www.sixxs.net/home/tunnelinfo/?19122
a screencapture (not sure how helpful that is) is available at: http://ub3rgeek.net/19122-error.jpg
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Monday, 17 August 2009 13:42:50
I have read and followed the "Reporting Problems" section on the Contact page.
I am removing these four pre-filled lines to demonstrate that I really read it.
I am providing the full details as asked on the above mentioned page.
------------------------------------------------------------------------------>8
I see the same bug. I have v4 connectivity to the pop, it replies to v4 pings, etc. The heartbeat goes out, but no packets come back from the pop.
(With one exception. Yesterday, while trying to determine what was failing, I happened to point links(1) at the pop; in return I got attacked by a tcp portscan from the pop's v4 address. (It looked consistent with nmap.) Not polite. It makes me wonder whether the box got owned.)
Traceroutes to me from other v6 sites fail at 2001:4978:1:400:b4a3:94ff:fe8a:26d8 (unassigned.v6.your.org.).
Like everyone else, this started a few days ago. (In my case, right in the middle of an active ssh session.0
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Friday, 14 August 2009 22:02:00
My name is John Morrissey and my NIC handle JWM7-RIPE. I have two tunnels terminating on uschi02, T19631 and T19632.
T19632 is working fine, but T19631 cannot ping its remote (uschi02's end) IPv6 address. This behavior began at the same point (yesterday) as other posters to this ticket have observed. This is a Linux 2.6.18 host and, while my ruleset has not changed in months, I disabled the network firewall during these tests.
Interestingly, my tunnel detail page shows my IPv4 endpoint as "unknown":
https://www.sixxs.net/home/tunnelinfo/?19631
"Your IPv4 Unknown"
The working tunnel (https://www.sixxs.net/home/tunnelinfo/?19632) also displays "Unknown" for "Your IPv4."
I do not require an SLA for my IPv6 connectivity, but when something goes wrong and three people report similar behavior in a given POP, I'm hoping that you'll take at least a passing interest in fixing the problem.
[jwm@boost:pts/4 ~> sudo tcpdump -ni eth0 host 216.14.98.22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
[1]+ Stopped sudo tcpdump -ni eth0 host 216.14.98.22
[jwm@boost:pts/4 ~> bg
[1]+ sudo tcpdump -ni eth0 host 216.14.98.22 &
[jwm@boost:pts/4 ~> ping6 2001:4978:f:2a0::
PING 2001:4978:f:2a0::(2001:4978:f:2a0::) 56 data bytes
15:26:14.305629 IP 69.55.65.181 > 216.14.98.22: IP6 2001:4978:f:2a0::2 > 2001:4978:f:2a0::: ICMP6, echo request, seq 1, length 64
15:26:15.305644 IP 69.55.65.181 > 216.14.98.22: IP6 2001:4978:f:2a0::2 > 2001:4978:f:2a0::: ICMP6, echo request, seq 2, length 64
15:26:16.305831 IP 69.55.65.181 > 216.14.98.22: IP6 2001:4978:f:2a0::2 > 2001:4978:f:2a0::: ICMP6, echo request, seq 3, length 64
15:26:17.305997 IP 69.55.65.181 > 216.14.98.22: IP6 2001:4978:f:2a0::2 > 2001:4978:f:2a0::: ICMP6, echo request, seq 4, length 64
--- 2001:4978:f:2a0:: ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms
[jwm@boost:pts/4 ~> sudo traceroute 216.14.98.22
traceroute to 216.14.98.22 (216.14.98.22), 64 hops max, 52 byte packets
1 roc-core1-dc1-65.129.netsville.com (69.55.65.129) 1 ms 1 ms 1 ms
2 roc-edge2.netsville.com (69.55.64.2) 58 ms 178 ms 3 ms
3 roc-edge4.netsville.com (69.55.64.4) 1 ms 3 ms 2 ms
4 hagg-01-t3-0-0-1-0.roch.twtelecom.net (64.132.176.133) 108 ms 144 ms 5 ms
5 66.192.242.253 (66.192.242.253) 18 ms 13 ms 13 ms
6 ae0-60g.cr1.nyc2.us.nlayer.net (69.31.95.122) [MPLS: Label 712916 Exp 0] More labels 14 ms More labels 13 ms More labels 13 ms
7 xe-4-2-0.cr1.ord1.us.nlayer.net (69.22.142.85) [MPLS: Label 499729 Exp 0] 26 ms 26 ms 26 ms
8 tge6-4.ar1.ord1.us.nlayer.net (69.31.111.134) 26 ms 26 ms 26 ms
9 as19255.ge2-47-107.ar1.ord1.us.nlayer.net (69.31.111.26) 27 ms 26 ms 27 ms
10 ae0-5.core2.chi.bb.your.org (216.14.98.30) 28 ms 26 ms 28 ms
11 sixxs.cx01.chi.bb.your.org (216.14.98.22) 27 ms 26 ms 27 ms
As others have mentioned, I can ping the uschi02 end of the malfunctioning tunnel (T19631) from another IPv6 host (in this case, my end of T19632):
[jwm@roc0:pts/0 ~> sudo tcpdump -i eth1 host 216.14.98.22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
^Z
[1]+ Stopped sudo tcpdump -i eth1 host 216.14.98.22
[jwm@roc0:pts/0 ~> bg
[1]+ sudo tcpdump -i eth1 host 216.14.98.22 &
[jwm@roc0:pts/0 ~> ping6 2001:4978:f:2a0::1
PING 2001:4978:f:2a0::1(2001:4978:f:2a0::1) 56 data bytes
64 bytes from 2001:4978:f:2a0::1: icmp_seq=1 ttl=64 time=31.3 ms
15:33:44.627126 IP cpe-72-230-143-250.rochester.res.rr.com > sixxs.cx01.chi.bb.your.org: IP6 cl-674.chi-02.us.sixxs.net > gw-673.chi-02.us.sixxs.net: ICMP6, echo request, seq 1, length 64
15:33:44.658301 IP sixxs.cx01.chi.bb.your.org > cpe-72-230-143-250.rochester.res.rr.com: IP6 gw-673.chi-02.us.sixxs.net > cl-674.chi-02.us.sixxs.net: ICMP6, echo reply, seq 1, length 64
15:33:45.628752 IP cpe-72-230-143-250.rochester.res.rr.com > sixxs.cx01.chi.bb.your.org: IP6 cl-674.chi-02.us.sixxs.net > gw-673.chi-02.us.sixxs.net: ICMP6, echo request, seq 2, length 64
15:33:45.660414 IP sixxs.cx01.chi.bb.your.org > cpe-72-230-143-250.rochester.res.rr.com: IP6 gw-673.chi-02.us.sixxs.net > cl-674.chi-02.us.sixxs.net: ICMP6, echo reply, seq 2, length 64
64 bytes from 2001:4978:f:2a0::1: icmp_seq=2 ttl=64 time=31.8 ms
^C
--- 2001:4978:f:2a0::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 31.376/31.628/31.881/0.308 ms
Trying to ping my end of T19631 from my end of T19632 does not yield an ICMP echo response, and the host on T19631 does not receive any IPv6 tunnel traffic:
[jwm@roc0:pts/0 ~> sudo tcpdump -i eth1 host 216.14.98.22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
^Z
[1]+ Stopped sudo tcpdump -i eth1 host 216.14.98.22
[jwm@roc0:pts/0 ~> bg
[1]+ sudo tcpdump -i eth1 host 216.14.98.22 &
[jwm@roc0:pts/0 ~> ping6 boost.horde.net
PING boost.horde.net(cl-673.chi-02.us.sixxs.net) 56 data bytes
15:36:27.525697 IP cpe-72-230-143-250.rochester.res.rr.com > sixxs.cx01.chi.bb.your.org: IP6 cl-674.chi-02.us.sixxs.net > cl-673.chi-02.us.sixxs.net: ICMP6, echo request, seq 1, length 64
15:36:28.536707 IP cpe-72-230-143-250.rochester.res.rr.com > sixxs.cx01.chi.bb.your.org: IP6 cl-674.chi-02.us.sixxs.net > cl-673.chi-02.us.sixxs.net: ICMP6, echo request, seq 2, length 64
15:36:29.536688 IP cpe-72-230-143-250.rochester.res.rr.com > sixxs.cx01.chi.bb.your.org: IP6 cl-674.chi-02.us.sixxs.net > cl-673.chi-02.us.sixxs.net: ICMP6, echo request, seq 3, length 64
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Saturday, 15 August 2009 16:09:54
Both of these tunnels (T19631 and T19632) have been reported as unpingable by the tunnel robot, even though T19632 is still passing IPv6 traffic. Coincidentally, the last known responding date is around the time that others have reported that their tunnels stopped functioning.
The tunnels are up and configured correctly on my end. ICMP echo request/reply are unfiltered. This problem appears to be with uschi02, not end user equipment, network paths, or configuration.
2009-08-15 01:15:16 Tunnel endpoint 2001:4978:f:2a0::2 didn't ping for 2 days -5
2009-08-15 01:15:16 Tunnel endpoint 2001:4978:f:2a1::2 didn't ping for 2 days -5
To: John Morrissey <jwm@horde.net>
Subject: [SixXS] Tunnel Downtime for JWM7-RIPE's 2001:4978:f:2a0::2
From: SixXS Staff <info@sixxs.net>
Date: Sat, 15 Aug 2009 01:15:16 +0200 (CEST)
Dear John Morrissey,
This is the tunnelrobot at SixXS, mailing you to inform you that your
tunnel to Your.Org, Inc. has not been replying to pings for a period of time
Here's the information regarding the tunnel:
Handle : JWM7-RIPE
PoP Name : uschi02 (YOUR-ORG-INC [AS19255])
Your Location: Rochester NY, us
SixXS IPv6 : 2001:4978:f:2a0::1/64
Your IPv6 : 2001:4978:f:2a0::2/64
SixXS IPv4 : 216.14.98.22
Your IPv4 : 69.55.65.181
Last known responding: 2009-08-13 10:44:34
To: John Morrissey <jwm@horde.net>
Subject: [SixXS] Tunnel Downtime for JWM7-RIPE's 2001:4978:f:2a1::2
From: SixXS Staff <info@sixxs.net>
Date: Sat, 15 Aug 2009 01:15:16 +0200 (CEST)
Dear John Morrissey,
This is the tunnelrobot at SixXS, mailing you to inform you that your
tunnel to Your.Org, Inc. has not been replying to pings for a period of time
Here's the information regarding the tunnel:
Handle : JWM7-RIPE
PoP Name : uschi02 (YOUR-ORG-INC [AS19255])
Your Location: Rochester NY, us
SixXS IPv6 : 2001:4978:f:2a1::1/64
Your IPv6 : 2001:4978:f:2a1::2/64
SixXS IPv4 : 216.14.98.22
Your IPv4 : 72.230.143.250
Last known responding: 2009-08-13 10:44:34
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Saturday, 15 August 2009 00:33:57
I have read and followed the "Reporting Problems" section on the Contact page.
I am removing these four pre-filled lines to demonstrate that I really read it.
I am providing the full details as asked on the above mentioned page.
------------------------------------------------------------------------------>8
Seeing the same issues. Pings sometimes get through, but most fail. Tunnel is staying up, but having a very hard time passing data through it. Will post more info when i time to gather it.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Saturday, 15 August 2009 02:26:18
My static tunnel (T14959) seems to be down as well and went down around 2009-08-13 early in the morning MST. I've made no config changes on my Linux router in the last couple of months and for no apparent reason it's no longer working. I can ping in over IPv4 no problem but I am unable to ping it via ipv6. I've attached an IPv4 traceroute to uschi02 from my router. I've attempted rebooting but it didn't help. I can ping my side's ipv6 (2001:4978:f:102::2) side of the tunnel from my internal network but am unable to ping to your.org's side from the router (2001:4978:f:102::1). My static IPv4 address is 216.237.73.27 on my router. If there is anything further that I can provide please let me know.
traceroute to 216.14.98.22 (216.14.98.22), 30 hops max, 60 byte packets
1 169-95-237-216.skybeam.com (216.237.95.169) 24.930 ms 24.899 ms 28.076 ms
2 vlan37-core1.sopris.net (216.237.64.169) 28.077 ms 32.777 ms 34.501 ms
3 f0-1-bdr1-rtr.sopris.net (216.237.88.5) 32.715 ms 32.730 ms 32.726 ms
4 ip-216-17-203-61.rev.frii.com (216.17.203.61) 99.856 ms 99.840 ms 99.788 ms
5 gi0-5.na31.b006467-1.den01.atlas.cogentco.com (66.250.5.253) 44.566 ms 44.581 ms 44.592 ms
6 gi1-6.3522.mpd01.den01.atlas.cogentco.com (38.20.32.53) 44.375 ms 43.515 ms 43.494 ms
7 te9-2.mpd01.mci01.atlas.cogentco.com (154.54.24.82) 125.551 ms 129.010 ms 139.574 ms
8 te8-4.mpd01.dfw01.atlas.cogentco.com (154.54.5.125) 150.212 ms * *
9 * * *
10 * * *
11 your.org.ge2-5.br02.chc01.pccwbtn.net (63.218.5.38) 164.378 ms 164.348 ms 164.180 ms
12 ae0-5.core2.chi.bb.your.org (216.14.98.30) 164.196 ms 70.494 ms 90.109 ms
13 sixxs.cx01.chi.bb.your.org (216.14.98.22) 90.009 ms 90.024 ms 91.663 ms
Two tunnels on uschi02 down
![[gb]](/s/countries/gb.gif) Shadow Hawkins on Saturday, 15 August 2009 02:46:42
NIC handle: AJS32-RIPE
Tunnels: T22268, T14824 both on uschi02 pop
Both these tunnels are not reachable from the Internet but in
slightly different ways.
"aiccu test" from my end of T22268:
$ sudo aiccu test
RTNETLINK answers: File exists
RTNETLINK answers: File exists
#######
####### AICCU Quick Connectivity Test
#######
####### [1/8] Ping the IPv4 Local/Your Outer Endpoint (209.20.91.73)
### This should return so called 'echo replies'
### If it doesn't then check your firewall settings
### Your local endpoint should always be pingable
### It could also indicate problems with your IPv4 stack
PING 209.20.91.73 (209.20.91.73) 56(84) bytes of data.
64 bytes from 209.20.91.73: icmp_seq=1 ttl=64 time=0.000 ms
64 bytes from 209.20.91.73: icmp_seq=2 ttl=64 time=0.000 ms
64 bytes from 209.20.91.73: icmp_seq=3 ttl=64 time=0.000 ms
--- 209.20.91.73 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.000/0.000/0.000/0.000 ms
######
Did this work? [Y/n] y
####### [2/8] Ping the IPv4 Remote/PoP Outer Endpoint (216.14.98.22)
### These pings should reach the PoP and come back to you
### In case there are problems along the route between your
### host and the PoP this could not return replies
### Check your firewall settings if problems occur
PING 216.14.98.22 (216.14.98.22) 56(84) bytes of data.
--- 216.14.98.22 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
######
Did this work? [Y/n] n
"aiccu test" from my end of T14824:
$ sudo aiccu test
RTNETLINK answers: File exists
RTNETLINK answers: File exists
#######
####### AICCU Quick Connectivity Test
#######
####### [1/8] Ping the IPv4 Local/Your Outer Endpoint (209.237.247.192)
### This should return so called 'echo replies'
### If it doesn't then check your firewall settings
### Your local endpoint should always be pingable
### It could also indicate problems with your IPv4 stack
PING 209.237.247.192 (209.237.247.192) 56(84) bytes of data.
64 bytes from 209.237.247.192: icmp_seq=1 ttl=64 time=0.013 ms
64 bytes from 209.237.247.192: icmp_seq=2 ttl=64 time=0.008 ms
64 bytes from 209.237.247.192: icmp_seq=3 ttl=64 time=0.011 ms
--- 209.237.247.192 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.008/0.010/0.013/0.004 ms
######
Did this work? [Y/n]
####### [2/8] Ping the IPv4 Remote/PoP Outer Endpoint (216.14.98.22)
### These pings should reach the PoP and come back to you
### In case there are problems along the route between your
### host and the PoP this could not return replies
### Check your firewall settings if problems occur
PING 216.14.98.22 (216.14.98.22) 56(84) bytes of data.
64 bytes from 216.14.98.22: icmp_seq=1 ttl=53 time=64.0 ms
64 bytes from 216.14.98.22: icmp_seq=2 ttl=53 time=64.2 ms
64 bytes from 216.14.98.22: icmp_seq=3 ttl=53 time=64.1 ms
--- 216.14.98.22 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 64.062/64.135/64.210/0.215 ms
######
Did this work? [Y/n]
####### [3/8] Traceroute to the PoP (216.14.98.22) over IPv4
### This traceroute should reach the PoP
### In case this traceroute fails then you have no connectivity
### to the PoP and this is most probably the problem
traceroute to 216.14.98.22 (216.14.98.22), 30 hops max, 40 byte packets
1 207.7.148.193 (207.7.148.193) 0.835 ms 0.654 ms 0.926 ms
2 Vlan519.br02-200p-sfo.unitedlayer.com (207.7.159.85) 1.088 ms 0.566 ms 0.486 ms
3 Vlan902.br01-paix-pao.unitedlayer.com (207.7.129.73) 0.988 ms 1.095 ms 0.963 ms
4 198.32.176.20 (198.32.176.20) 1.928 ms 2.236 ms 1.995 ms
5 10gigabitethernet4-1.core1.sjc2.he.net (72.52.92.70) 1.987 ms 2.075 ms 1.967 ms
6 10gigabitethernet1-1.core1.chi1.he.net (72.52.92.74) 88.880 ms 91.113 ms 82.440 ms
7 64.71.148.238 (64.71.148.238) 65.495 ms 65.674 ms 65.389 ms
8 ae0-5.core2.chi.bb.your.org (216.14.98.30) 64.429 ms 64.644 ms 64.246 ms
9 sixxs.cx01.chi.bb.your.org (216.14.98.22) 64.301 ms 64.167 ms 63.920 ms
######
###### [4/8] Checking if we can ping IPv6 localhost (::1)
### This confirms if your IPv6 is working
### If ::1 doesn't reply then something is wrong with your IPv6 stack
PING ::1(::1) 56 data bytes
64 bytes from ::1: icmp_seq=1 ttl=64 time=0.015 ms
64 bytes from ::1: icmp_seq=2 ttl=64 time=0.010 ms
64 bytes from ::1: icmp_seq=3 ttl=64 time=0.011 ms
--- ::1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.010/0.012/0.015/0.002 ms
######
Did this work? [Y/n]
###### [5/8] Ping the IPv6 Local/Your Inner Tunnel Endpoint (2001:4978:f:f2::2)
### This confirms that your tunnel is configured
### If it doesn't reply then check your interface and routing tables
PING 2001:4978:f:f2::2(2001:4978:f:f2::2) 56 data bytes
64 bytes from 2001:4978:f:f2::2: icmp_seq=1 ttl=64 time=0.020 ms
64 bytes from 2001:4978:f:f2::2: icmp_seq=2 ttl=64 time=0.018 ms
64 bytes from 2001:4978:f:f2::2: icmp_seq=3 ttl=64 time=0.021 ms
--- 2001:4978:f:f2::2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.018/0.019/0.021/0.005 ms
######
Did this work? [Y/n]
###### [6/8] Ping the IPv6 Remote/PoP Inner Tunnel Endpoint (2001:4978:f:f2::1)
### This confirms the reachability of the other side of the tunnel
### If it doesn't reply then check your interface and routing tables
### Don't forget to check your firewall of course
### If the previous test was succesful then this could be both
### a firewalling and a routing/interface problem
PING 2001:4978:f:f2::1(2001:4978:f:f2::1) 56 data bytes
--- 2001:4978:f:f2::1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2010ms
######
Did this work? [Y/n] n
In the case of T22268 the packets are coming in over the tunnel but they never
get out again past the POP:
ping6 from a third host 2001:ba8:1f1:f019::25 to my end of T22268,
2001:4978:f:392::2, tcpdump running on my end:
$ sudo tcpdump -s 1500 -pni sixxs 'icmp6'
tcpdump: WARNING: sixxs: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on sixxs, link-type RAW (Raw IP), capture size 1500 bytes
00:32:14.070892 IP6 2001:ba8:1f1:f019::25 > 2001:4978:f:392::2: ICMP6, echo request, seq 1, length 64
00:32:14.070892 IP6 2001:4978:f:392::2 > 2001:ba8:1f1:f019::25: ICMP6, echo reply, seq 1, length 64
00:32:15.078844 IP6 2001:ba8:1f1:f019::25 > 2001:4978:f:392::2: ICMP6, echo request, seq 2, length 64
00:32:15.078844 IP6 2001:4978:f:392::2 > 2001:ba8:1f1:f019::25: ICMP6, echo reply, seq 2, length 64
These pings are not reaching back to 2001:ba8:1f1:f019::25.
In the case of T14824 the packets are getting out through the POP
but never come back again. A ping6 from my end of T14824,
2001:4978:f:f2::2, to 2001:ba8:1f1:f019::25, with tcpdump on
2001:ba8:1f1:f019::25:
$ sudo tcpdump -pni eth0 'icmp6'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
00:39:46.136101 IP6 2001:4978:f:f2::2 > 2001:ba8:1f1:f019::25: ICMP6, echo request, seq 1, length 64
00:39:46.136129 IP6 2001:ba8:1f1:f019::25 > 2001:4978:f:f2::2: ICMP6, echo reply, seq 1, length 64
00:39:47.142461 IP6 2001:4978:f:f2::2 > 2001:ba8:1f1:f019::25: ICMP6, echo request, seq 2, length 64
00:39:47.142487 IP6 2001:ba8:1f1:f019::25 > 2001:4978:f:f2::2: ICMP6, echo reply, seq 2, length 64
So packets got through T14824 and the POP, got to
2001:ba8:1f1:f019::25 which replies, but over on my end of the
tunnel where I also run tcpdump, I just see:
00:39:46.060647 IP6 2001:4978:f:f2::2 > 2001:ba8:1f1:f019::25: ICMP6, echo request, seq 1, length 64
00:39:47.067289 IP6 2001:4978:f:f2::2 > 2001:ba8:1f1:f019::25: ICMP6, echo request, seq 2, length 64
no response packets.
Both these tunnels die on Thursday 13th. I can't think of any
changes my end which could have caused this on either of the tunnels
let alone both of them. I also see tickets 1174109 and 1174295 and
wonder if these are related.
I cannot think of anything else I can check my side, but if there is
something please let me know. Just a confirmation that there is a
problem on uschi02's side would be good, if that is the case.
Cheers,
Andy
uschi02 having package loss ?
![[dk]](/s/countries/dk.gif) Carmen Sandiego on Saturday, 15 August 2009 12:35:22
I have read and followed the "Reporting Problems" section on the Contact page.
I am removing these four pre-filled lines to demonstrate that I really read it.
I am providing the full details as asked on the above mentioned page.
------------------------------------------------------------------------------>8
Hi
My name is Flemming Danielsen RIPE-FLDA on tunnel
Got 3 problems
1. The detalied status page of the above tunnel states my ipv4 address to be unknown eventhough I have tried to correct it twich (I lost 30 points trying to fix it through a chance of IP).
2. There seems to be a package loss somewhere in the ipv6 space from uschi02. I have access to another sixxs endpoint to test the connection and it show package loss.
3. Problem 2. gives me credit loss on my account as the tunnel is reported down.
ping6 2001:4978:f:1ab::1
PING 2001:4978:f:1ab::1(2001:4978:f:1ab::1) 56 data bytes
64 bytes from 2001:4978:f:1ab::1: icmp_seq=1 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=2 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=3 ttl=55 time=121 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=4 ttl=55 time=121 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=6 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=7 ttl=55 time=121 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=10 ttl=55 time=121 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=11 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=12 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=13 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=14 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=16 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=18 ttl=55 time=121 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=19 ttl=55 time=121 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=20 ttl=55 time=122 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=26 ttl=55 time=122 ms
^C
--- 2001:4978:f:1ab::1 ping statistics ---
27 packets transmitted, 16 received, 40% packet loss, time 26048ms
rtt min/avg/max/mdev = 121.701/122.092/122.485/0.450 ms
Local ping on uschi02
ping6 2001:4978:f:1ab::1
PING 2001:4978:f:1ab::1(2001:4978:f:1ab::1) 56 data bytes
64 bytes from 2001:4978:f:1ab::1: icmp_seq=1 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=2 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=3 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=4 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=5 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=6 ttl=64 time=4.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=7 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=8 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=9 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=10 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=11 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=12 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=13 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=14 ttl=64 time=4.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=15 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=16 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=17 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=18 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=19 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=20 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=21 ttl=64 time=4.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=22 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=23 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=24 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=25 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=26 ttl=64 time=8.00 ms
64 bytes from 2001:4978:f:1ab::1: icmp_seq=27 ttl=64 time=8.00 ms
--- 2001:4978:f:1ab::1 ping statistics ---
27 packets transmitted, 27 received, 0% packet loss, time 26001ms
rtt min/avg/max/mdev = 4.000/7.555/8.001/1.261 ms
Local remote ping:
ping6 2a01:198:200:199::1
PING 2a01:198:200:199::1(2a01:198:200:199::1) 56 data bytes
64 bytes from 2a01:198:200:199::1: icmp_seq=1 ttl=64 time=9.41 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=2 ttl=64 time=9.09 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=3 ttl=64 time=9.72 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=4 ttl=64 time=9.55 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=5 ttl=64 time=9.64 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=6 ttl=64 time=9.36 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=7 ttl=64 time=9.52 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=8 ttl=64 time=9.42 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=9 ttl=64 time=9.01 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=10 ttl=64 time=9.18 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=11 ttl=64 time=9.65 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=12 ttl=64 time=9.38 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=13 ttl=64 time=9.66 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=14 ttl=64 time=9.68 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=15 ttl=64 time=9.12 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=16 ttl=64 time=9.76 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=17 ttl=64 time=9.97 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=18 ttl=64 time=9.43 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=19 ttl=64 time=9.62 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=20 ttl=64 time=9.18 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=21 ttl=64 time=9.37 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=22 ttl=64 time=9.44 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=23 ttl=64 time=9.81 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=24 ttl=64 time=10.3 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=25 ttl=64 time=9.23 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=26 ttl=64 time=9.68 ms
64 bytes from 2a01:198:200:199::1: icmp_seq=27 ttl=64 time=9.46 ms
^C
--- 2a01:198:200:199::1 ping statistics ---
27 packets transmitted, 27 received, 0% packet loss, time 26038ms
rtt min/avg/max/mdev = 9.010/9.510/10.347/0.309 ms
Local traceroute:
traceroute to ns2 (2a01:198:200:199::2) from 2001:4978:f:1ab::2, 30 hops max, 24 byte packets
1 gw-428.chi-02.us.sixxs.net (2001:4978:f:1ab::1) 8 ms 8.001 ms 8 ms
2 sixxs.ge-0.0.0-30.core1.chi.bb6.your.org (2001:4978:1:400::ffff) 8.001 ms 8 ms 8.001 ms
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
Remote Traceroute:
traceroute6 2001:4978:f:1ab::1
traceroute to 2001:4978:f:1ab::1 (2001:4978:f:1ab::1) from 2a01:198:200:199::2, 30 hops max, 24 byte packets
1 gw-410.dus-01.de.sixxs.net (2a01:198:200:199::1) 9.082 ms 9.588 ms 9.692 ms
2 sixxs-dus.dus.speedpartner.de (2a01:198:200::1) 9.497 ms 9.834 ms 9.732 ms
3 speedpartner.r1.dus1.de.opencarrier.eu (2001:7f8:3a:e101::1) 10.567 ms 10.434 ms 9.769 ms
4 oc-r1-dus1.r1.fra3.de.opencarrier.eu (2001:7f8:3a:e000::2) 13.556 ms 13.707 ms 13.605 ms
5 de-cix.he.net (2001:7f8::1b1b:0:1) 23.752 ms 23.664 ms 23.35 ms
6 10gigabitethernet1-4.core1.ams1.he.net (2001:470:0:47::1) 29.756 ms 30.807 ms 29.8 ms
7 10gigabitethernet1-4.core1.lon1.he.net (2001:470:0:3f::1) 30.434 ms 29.595 ms 29.902 ms
8 10gigabitethernet2-3.core1.nyc4.he.net (2001:470:0:3e::1) 98.996 ms 99.442 ms 98.648 ms
9 10gigabitethernet1-2.core1.chi1.he.net (2001:470:0:4e::1) 120.859 ms 129.727 ms 125.335 ms
10 2001:470:0:7f::2 (2001:470:0:7f::2) 122.119 ms 121.906 ms 122.469 ms
11 int.xe-1.0.0-5.core2.chi.bb6.your.org (2001:4978:1:505::2) 123.029 ms 121.899 ms 121.923 ms
12 * * *
13 * * *
Kind Regards,
Flemming Danielsen
pop not reachable from my endpoint IPv4s
![[nl]](/s/countries/nl.gif) Shadow Hawkins on Saturday, 15 August 2009 15:32:02
When I try to ping or traceroute the pop at 216.14.98.22 it is not reachable from my side of the tunnel (64.85.161.59). From other IPs the pop is pingable, as is being confirmed by the corenetworks.net admin:
"I'm sorry about the delay on this response. Below is my trace from my
server. I've performed traces from a few other hosts on our Data Center
network as well as our office LAN. All had the same result except for
two. They failed the trace at the same point your server did. This
seems to indicate the host at 216.14.98.30 or 216.14.98.22 has a
firewall or routing rule in place that is preventing traffic from some IPs. "
A tarceroute from the not-working IP:
[otto@vera:/tmp:157]$ traceroute 216.14.98.22
traceroute to 216.14.98.22 (216.14.98.22), 64 hops max, 40 byte packets
1 64.85.160.1 (64.85.160.1) 0.863 ms 0.784 ms 0.955 ms
2 ae0-105.cr1.ord1.us.nlayer.net (69.31.111.17) 30.591 ms 7.579 ms 7.931 ms
3 tge6-4.ar1.ord1.us.nlayer.net (69.31.111.134) 384.70 ms 8.216 ms 8.251 ms
4 as19255.ge2-47-107.ar1.ord1.us.nlayer.net (69.31.111.26) 8.900 ms 8.992 ms
9.150 ms
5 ae0-5.core2.chi.bb.your.org (216.14.98.30) 8.905 ms 8.844 ms 8.822 ms
6 * * *
7 *^C
[otto@vera:/tmp:158]$
A traceroute from a working IP:
imac:~ otto$ traceroute 216.14.98.22
traceroute to 216.14.98.22 (216.14.98.22), 64 hops max, 40 byte packets
1 vera (10.0.1.1) 1.829 ms 0.700 ms 1.242 ms
2 212-127-177-1.cable.quicknet.nl (212.127.177.1) 23.703 ms 14.863 ms 7.418 ms
3 nl-nwv1-ra-01-gi1-1.multikabel.net (212.127.252.99) 10.129 ms 7.663 ms 7.005 ms
4 nl-ams2-rb-01-10G-3-2.multikabel.net (213.132.191.41) 12.518 ms 16.117 ms 41.965 ms
5 asd-lc0006-cr101-ae6-0.core.as9143.net (213.51.158.78) 10.967 ms 10.321 ms 8.188 ms
6 ams-ix.he.net (195.69.145.150) 10.220 ms 11.069 ms 8.481 ms
7 10gigabitethernet4-1.core1.nyc4.he.net (216.66.24.153) 105.119 ms 103.031 ms 97.456 ms
8 10gigabitethernet1-2.core1.chi1.he.net (72.52.92.102) 127.627 ms 119.796 ms 135.824 ms
9 64.71.148.238 (64.71.148.238) 121.696 ms 119.121 ms 119.511 ms
10 ae0-5.core2.chi.bb.your.org (216.14.98.30) 121.090 ms 163.114 ms 127.372 ms
11 sixxs.cx01.chi.bb.your.org (216.14.98.22) 121.324 ms 122.752 ms 120.331 ms
imac:~ otto$
So indeed, 216.14.98.30 and/or 216.14.98.22 seems to be blocking my traffic from 64.85.161.59
Regards, Otto
pop not reachable from my endpoint IPv4s
Shadow Hawkins on Saturday, 15 August 2009 21:45:12
I'm seeing a similar problem here. I started experiencing problems on the 13th in the morning local time. (10 am EST roughly)
This is from 70.91.226.206 (my tunnel is on .201)
traceroute to 216.14.98.22 (216.14.98.22), 64 hops max, 40 byte packets
1 * * *
2 ge-3-1-ur02.sannarbor.mi.michigan.comcast.net (68.87.187.137) 89.800 ms 34.776 ms 41.300 ms
3 te-9-2-ur03.sannarbor.mi.michigan.comcast.net (68.87.190.166) 24.622 ms 14.198 ms 43.659 ms
4 te-0-4-0-4-ar01.pontiac.mi.michigan.comcast.net (68.87.190.45) 41.935 ms 54.806 ms 50.544 ms
5 pos-2-1-0-0-cr01.chicago.il.ibone.comcast.net (68.86.90.109) 60.540 ms 26.680 ms 28.533 ms
6 tenge13/4.br03.chc01.pccwbtn.net (68.86.89.58) 35.422 ms 29.811 ms 31.022 ms
7 your.org.ge2-5.br02.chc01.pccwbtn.net (63.218.5.38) 30.489 ms 28.792 ms 33.510 ms
8 ae0-5.core2.chi.bb.your.org (216.14.98.30) 30.475 ms 43.841 ms 42.347 ms
9 * * *
10 * * *
11 * * *
12 * *^C
Yet, if I try a traceroute from Eastern Michigan University:
traceroute 216.14.98.22 traceroute to 216.14.98.22 (216.14.98.22), 64 hops max, 40 byte packets
1 ph-rtr-205 (164.76.205.1) 5.367 ms 2.829 ms 2.398 ms
2 ph-core-249-0 (164.76.249.1) 2.016 ms 4.102 ms 2.023 ms
3 mac-phc-pp (164.76.252.133) 3.288 ms 2.168 ms 2.459 ms
4 ge1-0-12.emu4.mich.net (192.122.183.137) 2.623 ms 2.783 ms 2.644 ms
5 vlan75.hom.mich.net (198.108.92.147) 3.674 ms 3.930 ms 3.989 ms
6 tenge0-0-0-0x9.aa2.mich.net (198.108.22.185) 3.359 ms 3.608 ms 3.179 ms
7 xe-0-1-0x76.eq-chi2.mich.net (198.108.23.12) 9.066 ms 8.219 ms 7.907 ms
8 equinix-chi.he.net (206.223.119.37) 8.450 ms 8.354 ms 12.277 ms
9 64.71.148.238 (64.71.148.238) 21.397 ms 21.382 ms 21.717 ms
10 ae0-5.core2.chi.bb.your.org (216.14.98.30) 22.019 ms 22.408 ms 21.729 ms
11 sixxs.cx01.chi.bb.your.org (216.14.98.22) 21.597 ms 21.813 ms 22.133 ms
pop not reachable from my endpoint IPv4s
Shadow Hawkins on Sunday, 16 August 2009 05:59:04
I got some time to run the aiccu utility's test mode. (this is the MidnightBSD mports version of aiccu). It looks like it fails at step 6 of the test.
Tunnel Information for T14678:
POP Id : uschi02
IPv6 Local : 2001:4978:f:d9::2/64
IPv6 Remote : 2001:4978:f:d9::1/64
Tunnel Type : 6in4-static
Adminstate : enabled
Userstate : enabled
ifconfig: SIOCIFCREATE2: File exists
route: writing to routing socket: File exists
add net default: gateway 2001:4978:f:d9::1: route already in table
#######
####### AICCU Quick Connectivity Test
#######
####### [1/8] Ping the IPv4 Local/Your Outer Endpoint (70.91.226.201)
### This should return so called 'echo replies'
### If it doesn't then check your firewall settings
### Your local endpoint should always be pingable
### It could also indicate problems with your IPv4 stack
PING 70.91.226.201 (70.91.226.201): 56 data bytes
64 bytes from 70.91.226.201: icmp_seq=0 ttl=64 time=0.074 ms
64 bytes from 70.91.226.201: icmp_seq=1 ttl=64 time=0.039 ms
64 bytes from 70.91.226.201: icmp_seq=2 ttl=64 time=0.046 ms
--- 70.91.226.201 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.039/0.053/0.074/0.015 ms
######
####### [2/8] Ping the IPv4 Remote/PoP Outer Endpoint (216.14.98.22)
### These pings should reach the PoP and come back to you
### In case there are problems along the route between your
### host and the PoP this could not return replies
### Check your firewall settings if problems occur
PING 216.14.98.22 (216.14.98.22): 56 data bytes
64 bytes from 216.14.98.22: icmp_seq=0 ttl=50 time=27.496 ms
64 bytes from 216.14.98.22: icmp_seq=1 ttl=50 time=36.425 ms
64 bytes from 216.14.98.22: icmp_seq=2 ttl=50 time=26.815 ms
--- 216.14.98.22 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 26.815/30.245/36.425/4.379 ms
######
####### [3/8] Traceroute to the PoP (216.14.98.22) over IPv4
### This traceroute should reach the PoP
### In case this traceroute fails then you have no connectivity
### to the PoP and this is most probably the problem
traceroute to 216.14.98.22 (216.14.98.22), 64 hops max, 40 byte packets
1 * * *
2 ge-3-1-ur02.sannarbor.mi.michigan.comcast.net (68.87.187.137) 10.306 ms 17.027 ms 29.582 ms
3 te-9-2-ur03.sannarbor.mi.michigan.comcast.net (68.87.190.166) 31.079 ms 19.828 ms 7.944 ms
4 te-0-4-0-4-ar01.pontiac.mi.michigan.comcast.net (68.87.190.45) 16.585 ms 14.744 ms 20.314 ms
5 pos-2-1-0-0-cr01.chicago.il.ibone.comcast.net (68.86.90.109) 38.946 ms 25.299 ms 51.688 ms
6 68.86.89.58 (68.86.89.58) 33.946 ms 36.431 ms 52.546 ms
7 your.org.ge2-5.br02.chc01.pccwbtn.net (63.218.5.38) 60.053 ms 47.555 ms 48.179 ms
8 ae0-5.core2.chi.bb.your.org (216.14.98.30) 70.070 ms 44.437 ms 26.434 ms
9 sixxs.cx01.chi.bb.your.org (216.14.98.22) 93.664 ms 54.957 ms 69.543 ms
######
###### [4/8] Checking if we can ping IPv6 localhost (::1)
### This confirms if your IPv6 is working
### If ::1 doesn't reply then something is wrong with your IPv6 stack
PING6(56=40+8+8 bytes) ::1 --> ::1
16 bytes from ::1: Echo Request
16 bytes from ::1, icmp_seq=0 hlim=64 dst=::1%2 time=0.265 ms
16 bytes from ::1: Echo Request
16 bytes from ::1, icmp_seq=1 hlim=64 dst=::1%2 time=0.183 ms
16 bytes from ::1: Echo Request
16 bytes from ::1, icmp_seq=2 hlim=64 dst=::1%2 time=0.153 ms
--- ::1 ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.153/0.200/0.265/0.047 ms
######
###### [5/8] Ping the IPv6 Local/Your Inner Tunnel Endpoint (2001:4978:f:d9::2)
### This confirms that your tunnel is configured
### If it doesn't reply then check your interface and routing tables
PING6(56=40+8+8 bytes) 2001:4978:f:d9::2 --> 2001:4978:f:d9::2
16 bytes from 2001:4978:f:d9::2: Echo Request
16 bytes from 2001:4978:f:d9::2, icmp_seq=0 hlim=64 dst=2001:4978:f:d9::2%3 time=0.303 ms
16 bytes from 2001:4978:f:d9::2: Echo Request
16 bytes from 2001:4978:f:d9::2, icmp_seq=1 hlim=64 dst=2001:4978:f:d9::2%3 time=0.191 ms
16 bytes from 2001:4978:f:d9::2: Echo Request
16 bytes from 2001:4978:f:d9::2, icmp_seq=2 hlim=64 dst=2001:4978:f:d9::2%3 time=0.179 ms
--- 2001:4978:f:d9::2 ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.179/0.224/0.303/0.056 ms
######
###### [6/8] Ping the IPv6 Remote/PoP Inner Tunnel Endpoint (2001:4978:f:d9::1)
### This confirms the reachability of the other side of the tunnel
### If it doesn't reply then check your interface and routing tables
### Don't forget to check your firewall of course
### If the previous test was succesful then this could be both
### a firewalling and a routing/interface problem
PING6(56=40+8+8 bytes) 2001:4978:f:d9::2 --> 2001:4978:f:d9::1
32 bytes from 2001:4978:1fc:0:21b:63ff:fea4:b9ab: Neighbor Solicitation
24 bytes from 2001:4978:1fc:0:21b:63ff:fea4:b9ab: Neighbor Advertisement
56 bytes from fe80::20d:56ff:fef1:ab2e%em0: Router Advertisement
32 bytes from fe80::224:8cff:fecb:26d6%em0: Neighbor Solicitation
--- 2001:4978:f:d9::1 ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
######
###### [7/8] Traceroute6 to the central SixXS machine (noc.sixxs.net)
### This confirms that you can reach the central machine of SixXS
### If that one is reachable you should be able to reach most IPv6 destinations
### Also check http://www.sixxs.net/ipv6calc/ which should show an IPv6 connection
### If your browser supports IPv6 and uses it of course.
traceroute6 to noc.sixxs.net (2001:838:1:1:210:dcff:fe20:7c7c) from 2001:4978:f:d9::2, 64 hops max, 12 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 *^C
######
###### [8/8] Traceroute6 to (www.kame.net)
### This confirms that you can reach a Japanese IPv6 destination
### If that one is reachable you should be able to reach most IPv6 destinations
### You should also check http://www.kame.net which should display
### a animated kame (turtle), of course only when your browser supports and uses IPv6
traceroute6 to www.kame.net (2001:200:0:8002:203:47ff:fea5:3085) from 2001:4978:f:d9::2, 64 hops max, 12 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * *^C
######
pop not reachable from my endpoint IPv4s
Shadow Hawkins on Monday, 17 August 2009 13:21:44
Hmm no progress so far.
Would it be possible to contact the pop admin directly?
pop not reachable from my endpoint IPv4s
Shadow Hawkins on Wednesday, 19 August 2009 09:30:23
As of today, the pop is reachable again and my tunnel came back. The pop is even one less hop away!
Thanks to whoever fixed this.
pop not reachable from my endpoint IPv4s
Shadow Hawkins on Monday, 17 August 2009 14:24:16
Similar issues here;
From my endpoint to POP IPv4:
traceroute to 216.14.98.22 (216.14.98.22), 30 hops max, 40 byte packets
1 209-20-72-2.slicehost.net (209.20.72.2) 0.000 ms 0.000 ms 0.000 ms
2 209-20-79-6.slicehost.net (209.20.79.6) 0.000 ms 0.000 ms 0.000 ms
3 ge-6-10-163.car1.StLouis1.Level3.net (4.53.160.241) 0.000 ms 0.000 ms 0.000 ms
4 ae-11-11.car2.StLouis1.Level3.net (4.69.132.186) 0.000 ms 0.000 ms 0.000 ms
5 ae-4-4.ebr2.Chicago1.Level3.net (4.69.132.190) 12.000 ms 12.000 ms 12.000 ms
6 ae-23-52.car3.Chicago1.Level3.net (4.68.101.39) 4.000 ms ae-23-54.car3.Chicago1.Level3.net (4.68.101.103) 8.001 ms ae-23-52.car3.Chicago1.Level3.net (4.68.101.39) 8.000 ms
7 xe-0-3-0.cr2.ord1.us.nlayer.net (4.71.101.14) 8.000 ms 4.000 ms 40.002 ms
8 po2.ar1.ord1.us.nlayer.net (69.31.111.138) 12.001 ms 12.001 ms 12.001 ms
9 as19255.ge2-47-107.ar1.ord1.us.nlayer.net (69.31.111.26) 8.001 ms 8.001 ms 8.001 ms
10 ae0-5.core2.chi.bb.your.org (216.14.98.30) 8.001 ms 8.001 ms 8.001 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 *
From another one of my IPv4 hosts (working fine on another IPv6 POP)
traceroute to 216.14.98.22 (216.14.98.22), 30 hops max, 40 byte packets
1 81-29-85-1.servers.dedipower.net (81.29.85.1) 0.465 ms 0.950 ms 1.202 ms
2 ge-0-1.cadogan2.rtr.dedipower.net (81.29.95.53) 0.638 ms 0.675 ms 0.814 ms
3 ge-0-3.thn1.rtr.dedipower.net (81.29.95.235) 2.609 ms 2.597 ms 2.590 ms
4 cr02.ldn01.pccwbtn.net (195.66.224.167) 4.903 ms 4.979 ms 4.967 ms
5 your.org.ge2-5.br02.chc01.pccwbtn.net (63.218.5.38) 93.339 ms 93.382 ms 93.311 ms
6 ae0-5.core2.chi.bb.your.org (216.14.98.30) 91.838 ms 91.615 ms 91.661 ms
7 sixxs.cx01.chi.bb.your.org (216.14.98.22) 92.533 ms 92.734 ms 92.737 ms
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
![[no]](/s/countries/no.gif) Shadow Hawkins on Saturday, 15 August 2009 16:47:34
Also got similar issues with my tunnel on uschi02 (T19189).
The Tunnel Information page takes a long time to load, and shows "unknown" under Your IPv4. Tried to set my endpoints IPv4 again, but that didn't help.
I receive tunneled IPv6 packets from the POP, but packets in the other direction disappears (tested with ICMP6 ping).
The tunnelrobot is also mailing me. Last known responding: 2009-08-13 10:44:34
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Saturday, 15 August 2009 17:05:05
I too am having issues with my tunnel (T13100) on uschi02. The tunnel robot reported that my tunnel was last reachable at 2009-08-13 10:44:34.
Unlike the other reports here, my tunnel is still functioning, however, there does seem to be significant packet loss to certain IPv6 destinations.
Pinging the your.org side of my tunnel:
ping6 2001:4978:f:3::1
PING 2001:4978:f:3::1(2001:4978:f:3::1) 56 data bytes
64 bytes from 2001:4978:f:3::1: icmp_seq=1 ttl=64 time=51.8 ms
64 bytes from 2001:4978:f:3::1: icmp_seq=2 ttl=64 time=51.8 ms
<snip>
64 bytes from 2001:4978:f:3::1: icmp_seq=489 ttl=64 time=52.0 ms
64 bytes from 2001:4978:f:3::1: icmp_seq=490 ttl=64 time=51.9 ms
64 bytes from 2001:4978:f:3::1: icmp_seq=491 ttl=64 time=51.8 ms
--- 2001:4978:f:3::1 ping statistics ---
491 packets transmitted, 491 received, 0% packet loss, time 490049ms
rtt min/avg/max/mdev = 51.344/51.855/53.049/0.304 ms
Pinging www.sixxs.net:
ping6 www.sixxs.net
PING www.sixxs.net(noc.sixxs.net) 56 data bytes
64 bytes from noc.sixxs.net: icmp_seq=1 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=2 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=3 ttl=55 time=153 ms
64 bytes from noc.sixxs.net: icmp_seq=5 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=7 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=8 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=9 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=10 ttl=55 time=155 ms
64 bytes from noc.sixxs.net: icmp_seq=12 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=13 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=15 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=18 ttl=55 time=155 ms
64 bytes from noc.sixxs.net: icmp_seq=19 ttl=55 time=154 ms
64 bytes from noc.sixxs.net: icmp_seq=20 ttl=55 time=154 ms
--- www.sixxs.net ping statistics ---
20 packets transmitted, 14 received, 30% packet loss, time 19001ms
rtt min/avg/max/mdev = 153.895/154.505/155.406/0.648 ms
Traceroute to www.sixxs.net:
traceroute6 -n www.sixxs.net
traceroute to www.m.sixxs.net (2001:838:1:1:210:dcff:fe20:7c7c) from 2001:4978:f:3::2, 30 hops max, 16 byte packets
1 2001:4978:f:3::1 51.763 ms 51.903 ms 51.788 ms
2 2001:4978:1:400::ffff 52.539 ms 52.223 ms 52.725 ms
3 2001:4978:1:505::1 51.938 ms 52.269 ms 52.034 ms
4 2001:470:0:7f::1 55.659 ms 53.088 ms 53.091 ms
5 2001:470:0:4e::2 75.499 ms 75.382 ms 74.948 ms
6 2001:470:0:3e::2 144.113 ms 143.772 ms 144.525 ms
7 2001:470:0:3f::2 152.59 ms 152.118 ms 151.535 ms
8 2001:7f8:1::a501:2871:1 152.218 ms 152.421 ms 152.439 ms
9 2001:838:5:a::2 154.393 ms 153.968 ms 154.39 ms
10 2001:838:1:1:210:dcff:fe20:7c7c 154.044 ms 154.05 ms 154.346 ms
Another thing that seems interesting, if you go to the SixXS Latency Page and select 1 week and uschi02 you can see that all the other PoPs latency graphs also stop on August 13th.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Wednesday, 19 August 2009 01:35:19
It appears that the problem has been resolved. The tunnel robot hasn't sent it's daily email, and I see that my tunnel's latency graph has data once again.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Sunday, 16 August 2009 02:40:05
^^^^ My tunnel stopped working at almost the exact same time as this guy's reported email.
Last known responding: 2009-08-13 10:44:35
I have made no changes before or since, but I suspect the problem here can be found quite simply:
~ $ ping <local ipv4>
64 bytes from xxx: icmp_seq=1 ttl=64 time=0.029 ms
64 bytes from xxx: icmp_seq=2 ttl=64 time=0.055 ms
64 bytes from xxx: icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from xxx: icmp_seq=4 ttl=64 time=0.040 ms
--- xxx ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2997ms
rtt min/avg/max/mdev = 0.029/0.042/0.055/0.009 ms
~ $ ping 216.14.98.22
PING 216.14.98.22 (216.14.98.22) 56(84) bytes of data.
--- 216.14.98.22 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4007ms
~ $ traceroute -n 216.14.98.22
traceroute to 216.14.98.22 (216.14.98.22), 30 hops max, 40 byte packets
1 xxx 1.173 ms 1.154 ms 1.142 ms
2 64.79.223.1 1.111 ms 1.109 ms 1.098 ms
3 206.81.80.40 3.021 ms 3.007 ms 2.994 ms
4 72.52.92.49 20.010 ms 20.019 ms 20.008 ms
5 72.52.92.74 76.887 ms 77.873 ms 77.862 ms
6 64.71.148.238 70.894 ms 70.478 ms 70.466 ms
7 216.14.98.30 71.449 ms 71.440 ms 71.428 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 *
Looks like, at least in my case, some kind of routing issue at the pop end is completely preventing my access to 216.14.98.22.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Monday, 17 August 2009 01:28:05
Having the same issue with my tunnel aswell, will run a test tomorrow & report the output here
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Monday, 17 August 2009 03:40:44
Ryan, good catch, I see almost the same thing.
Pinging from my firewall to 216.14.98.22 works fine:
================
Zhadum-SSG5-> ping 216.14.98.22
Type escape sequence to abort
Sending 5, 100-byte ICMP Echos to 216.14.98.22, timeout is 1 seconds
!!!!!
Success Rate is 100 percent (5/5), round-trip time min/avg/max=30/30/32 ms
Zhadum-SSG5->
=================
Likewise, a traceroute is successful:
=================
Zhadum-SSG5-> trace-route 216.14.98.22
Type escape sequence to escape
Send ICMP echos to 216.14.98.22, timeout is 2 seconds, maximum hops are 32,
1 28ms 31ms 31ms 64.81.148.1
2 28ms 28ms 26ms 69.17.83.153
3 * * *
4 30ms 32ms 28ms 63.218.5.38
5 31ms 27ms 33ms 216.14.98.30
6 28ms 29ms 29ms 216.14.98.22
Trace complete
Zhadum-SSG5->
================
But pings from my workstation? No love:
================
C:\Users\Feren>ping 216.14.98.22
Pinging 216.14.98.22 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 216.14.98.22:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
=================
And a traceroute from my workstation shows death after 216.14.98.30, same as yours:
=================
C:\Users\Feren>tracert 216.14.98.22
Tracing route to sixxs.cx01.chi.bb.your.org [216.14.98.22]
over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 172.31.108.1
2 176 ms 64 ms 171 ms er1.chi1.speakeasy.net [64.81.148.1]
3 26 ms 25 ms 30 ms 220.ge-0-1-0.cr2.chi1.speakeasy.net [69.17.83.15
3]
4 * * * Request timed out.
5 27 ms 26 ms 25 ms your.org.ge2-5.br02.chc01.pccwbtn.net [63.218.5.
38]
6 28 ms 29 ms 26 ms ae0-5.core2.chi.bb.your.org [216.14.98.30]
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 ^C
Tunnel T18857 dropping incoming/outgoing packets
Shadow Hawkins on Sunday, 16 August 2009 09:58:26
I have been having issues for the past few days with packets being dropped on ipv6. My NIC handle is RPD-SIXXS. My connection works fine on ipv4 and I have no made any configuration changes on the router at all. So far this has registered with the automated bot that my ipv6 endpoint can't be ping so I have been penalized 10 ISK so far. I have an Asus WL500g Premium router which is IPv6 capable (no ipv6 nameservers... hope to rectify that one) and has been doing both routing and dhcp assignment for IPv6 for the past several months. In fact it had shown on my log that I was pingable for 20 weeks straight.
Since my router is the actual handler of the endpoint I will give you that info first. It is software modified which made it easier to get the info you needed.
IPTables contents
[shinji@WL-001E8C3DFF31 root]$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all -- anywhere anywhere state INVALID
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere state NEW
ACCEPT all -- anywhere anywhere state NEW
ACCEPT icmp -- anywhere anywhere
ACCEPT ipv6 -- anywhere anywhere
ACCEPT tcp -- anywhere WL-001E8C3DFF31 tcp dpt:2222
DROP all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere 192.168.1.83 udp dpt:57582
ACCEPT all -- anywhere anywhere
DROP all -- anywhere anywhere state INVALID
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
DROP all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate DNAT
DROP all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain MACS (0 references)
target prot opt source destination
Chain SECURITY (0 references)
target prot opt source destination
RETURN tcp -- anywhere anywhere tcp flags:SYN,RST,ACK/SYN limit: avg 1/sec burst 5
RETURN tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/RST limit: avg 1/sec burst 5
RETURN udp -- anywhere anywhere limit: avg 5/sec burst 5
RETURN icmp -- anywhere anywhere limit: avg 5/sec burst 5
DROP all -- anywhere anywhere
Chain logaccept (0 references)
target prot opt source destination
LOG all -- anywhere anywhere state NEW LOG level warning tcp-sequence tcp-options ip-options prefix `ACCEPT '
ACCEPT all -- anywhere anywhere
Chain logdrop (0 references)
target prot opt source destination
LOG all -- anywhere anywhere state NEW LOG level warning tcp-sequence tcp-options ip-options prefix `DROP '
DROP all -- anywhere anywhere
IPv6 configuration (sorry... it's an nvram dump but I'll notate the settings)
[shinji@WL-001E8C3DFF31 etc]$ nvram show | grep ipv6
size: 11616 bytes (21152 left)
ipv6_lan_addr=2001:4978:1e7:: <-- Lan Subnetting
ipv6_sit_ttl=64 <-- Tunnel TTL
ipv6_sit_localaddr=2001:4978:f:248::2 <-- My IPv6
ipv6_wan_router= < N/A
ipv6_lan_netsize=64 <-- LAN Subnet Size (it has to be 64 otherwise it won't work)
ipv6_wan_netsize= < N/A
ipv6_sit_remote=216.14.98.22 <-- Remote IPv4
ipv6_sit_netsize=64 <-- Remote Subnet Size (once again.. has to be 64)
ipv6_sit_remoteaddr=2001:4978:f:248::1 <-- Remote IPv6
ipv6_radvd_enable=1 <-- Radvd broadcasting (for ipv6 dhcp on network)
ipv6_wan_addr= < N/A
ipv6_sit_mtu=1280 <-- IPv6 MTU
ipv6_sit_enable=1 <-- IPv6 Enable/Disable switch
ifconfig output
br0 is bridge for network
vlan1 is wan
sixtun is tunnel adapter
[shinji@WL-001E8C3DFF31 etc]$ /sbin/ifconfig
br0 Link encap:Ethernet HWaddr 00:1E:8C:3D:FF:31
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: 2001:4978:1e7::/64 Scope:Global
inet6 addr: fe80::21e:8cff:fe3d:ff31/10 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:16973485 errors:0 dropped:0 overruns:0 frame:0
TX packets:30045885 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1954176175 (1.8 GiB) TX bytes:3848843659 (3.5 GiB)
sixtun Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet6 addr: fe80::4544:34ab/10 Scope:Link
inet6 addr: fe80::c0a8:101/10 Scope:Link
inet6 addr: 2001:4978:f:248::2/64 Scope:Global
UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1
RX packets:13762 errors:0 dropped:0 overruns:0 frame:0
TX packets:12268 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5446940 (5.1 MiB) TX bytes:1589856 (1.5 MiB)
vlan1 Link encap:Ethernet HWaddr 00:1E:8C:3D:FF:31
inet addr:69.68.52.171 Bcast:69.68.52.255 Mask:255.255.255.128
inet6 addr: fe80::21e:8cff:fe3d:ff31/10 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:29461426 errors:0 dropped:0 overruns:0 frame:0
TX packets:16870258 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3345886577 (3.1 GiB) TX bytes:2247996170 (2.0 GiB)
Now my router has ping6 but no traceroute6 so for outgoing traces I did them from my pc. I can do that since the router is successful in assinging ipv6 dhcp.
IPv4 outgoing traceroute to uschi02 endpoint
C:\Users\shinji>tracert 216.14.98.22
Tracing route to sixxs.cx01.chi.bb.your.org [216.14.98.22]
over a maximum of 30 hops:
1 1 ms 1 ms <1 ms WL-001E8C3DFF31 [192.168.1.1]
2 28 ms 27 ms 26 ms oh-69-68-52-129.sta.embarqhsd.net [69.68.52.129]
3 43 ms 44 ms 43 ms oh-65-173-85-85.dyn.embarqhsd.net [65.173.85.85]
4 44 ms 44 ms 43 ms oh-69-34-246-61.sta.embarqhsd.net [69.34.246.61]
5 56 ms 54 ms 54 ms te-3-1.car2.Cincinnati1.Level3.net [4.53.66.37]
6 54 ms 56 ms 54 ms ae-2-2.bar2.Cincinnati1.Level3.net [4.69.132.214
]
7 55 ms 54 ms 54 ms ae-0-11.bar1.Cincinnati1.Level3.net [4.69.136.20
9]
8 67 ms 54 ms 54 ms ae-10-10.ebr2.Chicago1.Level3.net [4.69.136.214]
9 55 ms 57 ms 56 ms ae-23-54.car3.Chicago1.Level3.net [4.68.101.103]
10 55 ms 55 ms 55 ms xe-0-3-0.cr2.ord1.us.nlayer.net [4.71.101.14]
11 186 ms 224 ms 95 ms po2.ar1.ord1.us.nlayer.net [69.31.111.138]
12 57 ms 56 ms 56 ms as19255.ge2-47-107.ar1.ord1.us.nlayer.net [69.31
.111.26]
13 57 ms 56 ms 57 ms ae0-5.core2.chi.bb.your.org [216.14.98.30]
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
IPv6 outgoing traceroute to uschi02 endpoint
C:\Users\shinji>tracert 2001:4978:f:248::1
Tracing route to gw-585.chi-02.us.sixxs.net [2001:4978:f:248::1]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 2001:4978:1e7::
2 57 ms 58 ms 57 ms gw-585.chi-02.us.sixxs.net [2001:4978:f:248::1]
Trace complete.
When I try to do a traceroute from uschi02 it fails on both ipv4 and ipv6 on your website.
When I do it from the noc to uschi02 I get the following
IPv4
IPv4 traceroute
IPv4 traceroute from noc.sixxs.net @ SixXS NOC, AS12871 to uschi02.sixxs.net / Your.Org, Inc., AS19255 :
Hop Node Loss% Sent Last Avg Best Worst StDev
1. 213.197.29.254 0.0% 5 0.3 0.3 0.3 0.5 0.1
2. 213.197.18.49 0.0% 5 1.8 11.8 1.8 37.1 15.5
GigabitEthernet2-11.br0.ams-nik.concepts-ict.net.
3. 195.69.145.150 0.0% 5 2.7 4.4 2.3 12.1 4.3
ams-ix.he.net.
4. 216.66.24.153 0.0% 5 93.0 88.8 87.6 93.0 2.4
10gigabitethernet4-1.core1.nyc4.he.net.
5. 72.52.92.102 0.0% 5 116.8 112.6 109.6 116.8 3.8
10gigabitethernet1-2.core1.chi1.he.net.
6. 64.71.148.238 0.0% 5 108.5 108.8 108.3 109.3 0.4
7. 216.14.98.30 0.0% 5 108.5 109.2 108.2 110.1 0.8
ae0-5.core2.chi.bb.your.org.
8. ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
IPv6
IPv6 traceroute from noc.sixxs.net @ SixXS NOC, AS12871 to uschi02.sixxs.net / Your.Org, Inc., AS19255 :
Hop Node Loss% Sent Last Avg Best Worst StDev ASN Organisation
1. 2001:838:1:1::1 0.0% 5 0.4 0.4 0.4 0.4 0.0 [.nl] Netherlands, The 12871 Concepts ICT
ge-1-3-0.breda.ipv6.concepts-ict.net.
2. 2001:838:5:a::1 0.0% 5 1.9 15.4 1.8 69.4 30.2 [.nl] Netherlands, The 12871 Concepts ICT
3. 2001:7f8:1::a500:6939:1 0.0% 5 2.5 2.4 2.3 2.5 0.1 [.nl] Netherlands, The AMS-IX-IPV6
ams-ix.he.net.
4. 2001:470:0:3f::1 0.0% 5 10.2 10.2 10.2 10.3 0.1 [.us] United States 6939 Hurricane Electric
10gigabitethernet1-4.core1.lon1.he.net.
5. 2001:470:0:3e::1 0.0% 5 79.3 79.3 79.2 79.6 0.1 [.us] United States 6939 Hurricane Electric
10gigabitethernet2-3.core1.nyc4.he.net.
6. 2001:470:0:4e::1 0.0% 5 103.8 105.8 101.0 109.3 3.4 [.us] United States 6939 Hurricane Electric
10gigabitethernet1-2.core1.chi1.he.net.
7. 2001:470:0:7f::2 0.0% 5 102.5 102.5 102.4 102.5 0.1 [.us] United States 6939 Hurricane Electric
8. 2001:4978:1:505::2 0.0% 5 102.6 102.7 102.5 102.9 0.1 [.us] United States 19255 YOUR.ORG, INC.
int.xe-1.0.0-5.core2.chi.bb6.your.org.
9. 2001:4978:1:400:202:b3ff:feb 20.0% 5 102.4 103.0 102.4 104.7 1.1 [.us] United States 19255 YOUR.ORG, INC.
Here is noc to my ipv6 router. (some packet loss here)
IPv6 traceroute from noc.sixxs.net @ SixXS NOC, AS12871 to 2001:4978:f:248::2 :
Hop Node Loss% Sent Last Avg Best Worst StDev ASN Organisation
1. 2001:838:1:1::1 0.0% 5 0.4 0.4 0.4 0.4 0.0 [.nl] Netherlands, The 12871 Concepts ICT
ge-1-3-0.breda.ipv6.concepts-ict.net.
2. 2001:838:5:a::1 0.0% 5 1.8 1.9 1.8 1.9 0.0 [.nl] Netherlands, The 12871 Concepts ICT
3. 2001:7f8:1::a500:6939:1 0.0% 5 2.4 2.4 2.3 2.4 0.0 [.nl] Netherlands, The AMS-IX-IPV6
ams-ix.he.net.
4. 2001:470:0:3f::1 0.0% 5 10.4 10.5 10.2 11.7 0.7 [.us] United States 6939 Hurricane Electric
10gigabitethernet1-4.core1.lon1.he.net.
5. 2001:470:0:3e::1 0.0% 5 79.2 80.1 79.2 83.7 2.0 [.us] United States 6939 Hurricane Electric
10gigabitethernet2-3.core1.nyc4.he.net.
6. 2001:470:0:4e::1 0.0% 5 108.5 105.7 101.0 109.3 4.3 [.us] United States 6939 Hurricane Electric
10gigabitethernet1-2.core1.chi1.he.net.
7. 2001:470:0:7f::2 0.0% 5 102.9 102.6 102.5 102.9 0.2 [.us] United States 6939 Hurricane Electric
8. 2001:4978:1:505::2 0.0% 5 102.5 102.6 102.4 102.8 0.1 [.us] United States 19255 YOUR.ORG, INC.
int.xe-1.0.0-5.core2.chi.bb6.your.org.
9. 2001:4978:1:400:b4a3:94ff:fe 40.0% 5 102.4 102.4 102.4 102.5 0.0 [.us] United States 19255 YOUR.ORG, INC.
10. 2001:4978:f:248::2 40.0% 5 159.7 159.4 159.2 159.7 0.2 [.us] United States 19255 Your.Org, Inc.
And another that is noc to ipv4.
IPv4 traceroute from noc.sixxs.net @ SixXS NOC, AS12871 to 69.68.52.171 :
Hop Node Loss% Sent Last Avg Best Worst StDev
1. 213.197.29.254 0.0% 5 0.3 0.3 0.3 0.3 0.0
2. 213.197.18.41 0.0% 5 14.9 3.6 0.6 14.9 6.3
ict.18.41.concepts.nl.
3. 213.197.18.45 0.0% 5 4.1 4.1 4.0 4.1 0.0
GigabitEthernet2-4.br0.sr-encap.concepts-ict.net.
4. 87.82.50.233 0.0% 5 4.1 4.1 4.0 4.1 0.0
ip-87-82-50-233.easynet.co.uk.
5. 87.86.77.28 0.0% 5 12.1 12.1 12.0 12.2 0.1
te5-0-0.gr11.telon.uk.easynet.net.
6. 149.6.148.65 0.0% 5 12.3 12.3 12.3 12.4 0.1
te3-1.mpd01.lon02.atlas.cogentco.com.
7. 130.117.0.225 0.0% 5 12.2 12.2 12.2 12.4 0.1
te4-4.ccr01.lon01.atlas.cogentco.com.
8. 66.28.4.189 0.0% 5 84.5 84.5 84.4 84.6 0.1
te3-2.ccr01.jfk02.atlas.cogentco.com.
9. 154.54.5.210 0.0% 5 89.2 89.2 89.1 89.2 0.0
te7-2.mpd01.jfk05.atlas.cogentco.com.
10. 4.68.111.45 0.0% 5 87.2 87.2 87.2 87.2 0.0
Te-8-1.car2.NewYork1.Level3.net.
11. 4.68.16.190 0.0% 5 84.8 88.4 84.8 96.4 5.2
vlan89.csw3.NewYork1.Level3.net.
12. 4.69.134.73 0.0% 5 84.9 85.0 84.9 85.2 0.1
ae-81-81.ebr1.NewYork1.Level3.net.
13. 4.69.132.93 0.0% 5 90.3 94.7 90.3 102.9 6.0
ae-3-3.ebr4.Washington1.Level3.net.
14. 4.69.134.186 0.0% 5 91.4 91.6 90.4 95.2 2.1
ae-84-84.csw3.Washington1.Level3.net.
15. 4.69.134.137 0.0% 5 99.8 96.7 90.7 100.2 3.8
ae-81-81.ebr1.Washington1.Level3.net.
16. 4.69.136.189 20.0% 5 98.1 98.2 98.1 98.5 0.2
ae-1-6.bar1.Cleveland1.Level3.net.
17. 4.53.192.18 0.0% 5 100.3 100.2 100.0 100.4 0.2
EMBARQ-MANA.bar1.Cleveland1.Level3.net.
18. 71.0.12.54 0.0% 5 109.9 109.7 109.6 109.9 0.1
oh-71-0-12-54.sta.embarqhsd.net.
19. 65.173.85.82 20.0% 5 127.4 127.4 127.1 127.5 0.2
oh-65-173-85-82.dyn.embarqhsd.net.
20. 69.68.52.171 20.0% 5 152.6 152.1 151.3 152.6 0.6
oh-69-68-52-171.sta.embarqhsd.net.
Now then I did several traceroutes from different sources (and providers) and they all show weird packet loss in the level 3 network so I am going to place the blame on them for my odd packet loss.
I also did a ping test from the co station (I work for my ISP) and sent 100 consecutive packets. No packet loss. Pings ranged from 67ms to 108ms
PING 69.68.52.171 (69.68.52.171): source 74.5.116.210, 36 data bytes,
timeout is 1 second
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
----69.68.52.171 PING Statistics----
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 67.966/70.487/108.942/5.121 ms
Need additional data? Let me know.
Tunnel T18857 dropping incoming/outgoing packets
Carmen Sandiego on Monday, 17 August 2009 02:21:33
I also have a tunnel on this PoP which is up and working just fine, as well as responding to pings, yet for three days in a row, I've received automated E-mails stating that it's not responding.
Handle : GNI1-SIXXS
PoP Name : uschi02 (YOUR-ORG-INC [AS19255])
Your Location: Dallas TX, us
SixXS IPv6 : 2001:4978:f:88::1/64
Your IPv6 : 2001:4978:f:88::2/64
SixXS IPv4 : 216.14.98.22
Your IPv4 : 209.144.225.87
Last known responding: 2009-08-13 10:44:34
I've changed nothing on my config and can access the v6 world just fine. Others report they can ping my end of the tunnel just fine.
Tunnel T18857 dropping incoming/outgoing packets
Shadow Hawkins on Monday, 17 August 2009 05:48:05
I believe I'm having the same problem (DNQ4-SIXXS, static tunnel T22319).
- ICMPv6 from my end of the tunnel to the remote end is fine
- ICMPv6 to elsewhere on the Internet shows ~20% packet loss
- ICMPv6 from elsewhere on the Internet to BOTH ends of the tunnel show ~20% packet loss
- ICMPv4 in all combinations is fine
It appears as though uschi02 is having some IPv6 connectivity problems, and this is being picked up as though the tunnel is down, even though it's not (though it is degraded). This doesn't appear to be something I can fix, though, as the other posters point out, we're being hit with ISK deductions.
I notice that on the tunnelinfo page, the latency graphs for this tunnel stop just before the 13th, and the loss graph doesn't seem to have any data at all. Could this be a reporting problem?
Tunnel T18857 dropping incoming/outgoing packets
Shadow Hawkins on Thursday, 20 August 2009 19:57:52
I'm glad to see that I'm not the only person here based on the amount of responses.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Monday, 17 August 2009 03:34:17
I'm experiencing the same type of issue with uschi02 as reported here and by other folks in this thread. My tunnel, T16873, was last reported alive on 2009-08-13 at 10:44:34 CEST. There have been no configuration changes on my end. Likewise, a a reboot of my router/firewall (a Juniper SSG-5) has produced no joy (my DSL bridge is fine), passing IPv4 quite happily.
My tunnel interface shows "Ready".
I can ping my tunnel interface IP (2001:4978:f:1a4::2/64) from my Vista desktop just fine, so I know traffic is working okay on the LAN segment....
=========
C:\Users\Feren>ping -6 2001:4978:f:1a4::2
Pinging 2001:4978:f:1a4::2 from 2001:4978:1a0:0:5c90:58dc:4565:f2e3 with 32 byte
s of data:
Reply from 2001:4978:f:1a4::2: time=4ms
Reply from 2001:4978:f:1a4::2: time=2ms
Reply from 2001:4978:f:1a4::2: time=2ms
Reply from 2001:4978:f:1a4::2: time=2ms
==========
My router (and my workstation) can ping the IPv4 address of uschi02 (216.14.98.22) just fine...
==========
Zhadum-SSG5-> ping 216.14.98.22
Type escape sequence to abort
Sending 5, 100-byte ICMP Echos to 216.14.98.22, timeout is 1 seconds
!!!!!
Success Rate is 100 percent (5/5), round-trip time min/avg/max=33/133/170 ms
===========
But I've got no joy otherwise.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Monday, 17 August 2009 06:35:17
I'm in the same boat, connecting to uschi02 from an unfirewalled Linux host. My graphs for tunnel T22119 die off just before 2009-08-13, my tunnel information page shows "unknown" for my IPv4 endpoint, and Last Alive time is 2009-08-13 10:44:35 CEST. I know my IPv4 internet connection and IPv6 endpoint system were online the entire time this issue occurred.
When I got home this evening from a weekend trip, I was able to ping the uschi02 IPv6 IP address from my system, but graphs do not indicate that I am online at this time.
I'm already down plenty of ISK (previous issue on my end weeks ago), so I've disabled my tunnel for now.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Tuesday, 18 August 2009 10:00:41
Much the same issue. Tunnel is up but with high packet loss. I can ping the far tunnel endpoint without loss, but beyond that things deteriate. I've been getting the daily emails warning me the tunnel will be suspended if I don't sort it out since 13 August
mtr from box to tunnel endpoint:
My traceroute [v0.72]
yogi.raggedstaff.net (::) Tue Aug 18 07:57:46 2009
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. gw-34.chi-02.us.sixxs.net 0.0% 197 7.9 7.6 7.3 8.4 0.2
mtr from box to further afield:
My traceroute [v0.72]
yogi.raggedstaff.net (::) Tue Aug 18 07:58:15 2009
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. gw-34.chi-02.us.sixxs.net 0.0% 308 8.1 7.8 7.4 12.5 0.4
2. sixxs.ge-0.0.0-30.core1.chi.bb6. 0.0% 308 8.1 8.8 7.8 43.2 3.9
3. int.xe-0.0.0-5.core1.chi.bb6.you 19.2% 308 8.1 9.8 7.8 77.8 7.9
4. gige-g2-19.core1.chi1.he.net 15.9% 308 9.0 11.4 8.7 26.2 3.4
5. 10gigabitethernet2-4.core1.nyc4. 16.2% 308 30.8 31.9 30.6 41.7 2.4
6. 10gigabitethernet1-2.core1.lon1. 15.6% 307 108.5 101.7 99.6 121.0 3.4
7. ae0-1234.rt2.the.uk.goscomb.net 10.8% 307 108.1 111.2 107.1 278.7 18.3
8. gblon02.sixxs.net 10.4% 307 107.7 107.5 107.0 115.0 0.6
9. paddington.ip6.raggedstaff.net 20.6% 307 109.2 109.1 108.6 111.4 0.3
mtr to box from further afield:
My traceroute [v0.73]
paddington.raggedstaff.net (::) Tue Aug 18 07:59:42 2009
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. gw-83.lon-02.gb.sixxs.net 0.0% 373 2.2 1.8 1.6 2.5 0.1
2. ae0-1235.rt2.the.uk.goscomb.net 0.0% 373 2.3 7.1 2.0 215.5 21.6
3. ae0-1368.rt0.nik.nl.goscomb.net 0.0% 373 20.4 11.6 7.6 123.5 17.1
4. ams-ix.he.net 0.0% 373 8.3 8.4 8.0 17.9 1.0
5. 10gigabitethernet1-4.core1.lon1. 0.0% 373 9.4 10.7 8.8 21.3 2.8
6. 10gigabitethernet2-3.core1.nyc4. 0.0% 373 78.4 79.2 78.1 90.2 2.4
7. 10gigabitethernet1-2.core1.chi1. 0.0% 373 105.7 102.4 99.9 113.5 3.4
8. 2001:470:0:7f::2 0.3% 373 101.4 102.9 101.2 168.3 7.3
9. int.xe-1.0.0-5.core2.chi.bb6.you 0.3% 373 101.6 102.5 101.2 139.4 4.9
10. unassigned.v6.your.org 13.4% 373 101.4 101.5 101.1 136.0 1.9
11. cl-34.chi-02.us.sixxs.net 18.5% 372 109.2 109.2 108.4 151.8 2.6
Unable to ping tunnel endpoint.
Shadow Hawkins on Monday, 17 August 2009 06:46:40
I have been receiving emails about my SIXXS tunnel being unpingable. I have been running this tunnel for a long time now and I have not changed anything with it since setting it up. I have verified that I am unable to ping any IPv6 address or even the IPv4 endpoint in Chicago for my tunnel. I have tried many things to get this tunnel working on my end and to ensure this is not a problem on my side.I have also looked through the forums and nothing has been helpful. I have Cacti graphs that show there has been traffic passing over this tunnel recently. So I am not sure what is going on with this.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
![[ca]](/s/countries/ca.gif) Shadow Hawkins on Monday, 17 August 2009 16:43:39
Hi, I am MBE1-SIXXS and would like to mention that the Tunnel Robot did also find that my tunnel was not pingable for a few days, even though I did not change anything on my end.
I checked for the problem this morning and found out that I can't ping the Tunnel End-Point from my location at home, but I can do it from my dedicated server.
Funny thing is that a traceroute works well from both places.
Traceroute from a machine that uschi02 endpoint works:
mbelleau@newborn:~$ traceroute 216.14.98.22
traceroute to 216.14.98.22 (216.14.98.22), 30 hops max, 40 byte packets
1 ip-70-38-37-98.static.privatedns.com (70.38.37.98) 0.739 ms 0.656 ms 0.547 ms
2 te8-4.v0707.cl-core04.mtl.iweb.com (67.205.127.121) 7.132 ms 7.114 ms 7.020 ms
3 te8-1.v0709.hd-core01.mtl.iweb.com (67.205.127.129) 0.630 ms 0.804 ms 0.633 ms
4 gi-3-6.core01.tor1.prioritycolo.com (67.223.96.9) 7.991 ms 7.966 ms 7.907 ms
5 gw-he.torontointernetxchange.net (198.32.245.112) 7.931 ms 7.879 ms 7.735 ms
6 10gigabitethernet1-2.core1.nyc5.he.net (72.52.92.165) 23.571 ms 23.524 ms 23.638 ms
7 10gigabitethernet1-4.core1.nyc1.he.net (72.52.92.153) 23.648 ms 23.572 ms 23.631 ms
8 10gigabitethernet1-1.core1.nyc4.he.net (72.52.92.45) 23.812 ms 23.707 ms 23.642 ms
9 10gigabitethernet1-2.core1.chi1.he.net (72.52.92.102) 45.776 ms 45.668 ms 45.822 ms
10 64.71.148.238 (64.71.148.238) 39.447 ms 39.433 ms 39.558 ms
11 ae0-5.core2.chi.bb.your.org (216.14.98.30) 39.486 ms 39.410 ms 39.566 ms
12 sixxs.cx01.chi.bb.your.org (216.14.98.22) 39.234 ms 39.407 ms 39.561 ms
And from one that does not work:
$ traceroute 216.14.98.22
traceroute to 216.14.98.22 (216.14.98.22), 64 hops max, 40 byte packets
1 10.18.40.1 (10.18.40.1) 8.579 ms 7.552 ms 9.925 ms
2 24.200.224.14 (24.200.224.14) 13.406 ms 14.130 ms 13.845 ms
3 24.200.250.93 (24.200.250.93) 11.396 ms 12.16 ms 13.776 ms
4 216.113.123.109 (216.113.123.109) 16.203 ms 15.206 ms 14.307 ms
5 ia-cnnu-bb04-pos5-0-0-cpe074.vtl.net (216.113.122.74) 41.784 ms 39.96 ms 39.431 ms
6 sl-gw36-chi-15-0-0.sprintlink.net (144.232.26.65) 38.832 ms 41.531 ms 40.721 ms
7 sl-st20-chi-0-0-0.sprintlink.net (144.232.19.144) 44.715 ms sl-st20-chi-12-0-0.sprintlink.net (144.232.8.219) 41.210 ms 40.660 ms
8 144.232.8.114 (144.232.8.114) 39.421 ms 40.172 ms 45.330 ms
9 ae-13-55.car3.Chicago1.Level3.net (4.68.101.135) 40.27 ms ae-13-53.car3.Chicago1.Level3.net (4.68.101.71) 39.837 ms ae-23-56.car3.Chicago1.Level3.net (4.68.101.167) 39.919 ms
10 xe-0-3-0.cr2.ord1.us.nlayer.net (4.71.101.14) 31.644 ms 30.906 ms 32.238 ms
11 po2.ar1.ord1.us.nlayer.net (69.31.111.138) 30.803 ms 31.623 ms 33.331 ms
12 as19255.ge2-47-107.ar1.ord1.us.nlayer.net (69.31.111.26) 32.194 ms 34.781 ms 33.443 ms
13 ae0-5.core2.chi.bb.your.org (216.14.98.30) 32.194 ms 36.520 ms 33.367 ms
14 sixxs.cx01.chi.bb.your.org (216.14.98.22) 33.142 ms 33.488 ms 33.28 ms
Ping from the first location (tunnel works):
mbelleau@newborn:~$ ping -c 10 216.14.98.22
PING 216.14.98.22 (216.14.98.22) 56(84) bytes of data.
64 bytes from 216.14.98.22: icmp_seq=1 ttl=53 time=39.3 ms
64 bytes from 216.14.98.22: icmp_seq=2 ttl=53 time=39.3 ms
64 bytes from 216.14.98.22: icmp_seq=3 ttl=53 time=39.4 ms
64 bytes from 216.14.98.22: icmp_seq=4 ttl=53 time=39.4 ms
64 bytes from 216.14.98.22: icmp_seq=5 ttl=53 time=39.3 ms
64 bytes from 216.14.98.22: icmp_seq=6 ttl=53 time=39.2 ms
64 bytes from 216.14.98.22: icmp_seq=7 ttl=53 time=39.4 ms
64 bytes from 216.14.98.22: icmp_seq=8 ttl=53 time=39.3 ms
64 bytes from 216.14.98.22: icmp_seq=9 ttl=53 time=39.3 ms
64 bytes from 216.14.98.22: icmp_seq=10 ttl=53 time=39.2 ms
--- 216.14.98.22 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9038ms
rtt min/avg/max/mdev = 39.225/39.349/39.498/0.229 ms
And from the one I'm trying to troubleshoot (at home, tunnel that doesn't work for a few days):
$ ping -c 10 216.14.98.22
PING 216.14.98.22 (216.14.98.22): 56 data bytes
--- 216.14.98.22 ping statistics ---
10 packets transmitted, 0 packets received, 100.0% packet loss
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Monday, 17 August 2009 16:49:43
Source: https://www.sixxs.net/tools/traceroute/?pop=noc-4&dpop=uschi02&dest=uschi02.sixxs.net
This link, from the SIXXS web site shows that there is 100% packet loss to uschi02. There might be a routing issue on their end.
IPv4 traceroute
IPv4 traceroute from noc.sixxs.net @ SixXS NOC, AS12871 to uschi02.sixxs.net :
Hop Node Loss% Sent Last Avg Best Worst StDev
1. 213.197.29.254 0.0% 5 0.3 0.3 0.3 0.3 0.0
2. 213.197.18.49 0.0% 5 1.8 1.9 1.8 1.9 0.0
GigabitEthernet2-11.br0.ams-nik.concepts-ict.net.
3. 195.69.145.150 0.0% 5 2.4 2.4 2.3 2.6 0.1
ams-ix.he.net.
4. 216.66.24.153 0.0% 5 87.7 87.7 87.6 87.9 0.1
10gigabitethernet4-1.core1.nyc4.he.net.
5. 72.52.92.102 0.0% 5 114.1 115.5 111.0 119.3 3.2
10gigabitethernet1-2.core1.chi1.he.net.
6. 64.71.148.238 0.0% 5 111.9 112.4 111.2 115.8 1.9
7. 216.14.98.30 0.0% 5 111.3 111.9 111.0 113.7 1.1
ae0-5.core2.chi.bb.your.org.
8. ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
Other locations also find this host unreachable (http://www.tera-byte.com/cgi-bin/nph-trace?216.14.98.22)
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Monday, 17 August 2009 18:37:42
I am also having problems with my tunnel endpoint on uschi02. IPv4 pings go through, aiccu reports no errors, however, I'm getting no traffic back from uschi02 from the heartbeats and the tunnel information page--which loads incredibly slowly for some reason--claims that my IPv4 is unknown. (I tried to bring my tunnel endpoint back up Friday August 14, after several weeks down as I upgraded my endpoint hardware and software; thought at first there was a problem due to version skew, but that does not appear to be the case...particularly given all these reports!)
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Monday, 17 August 2009 20:34:19
I am also having problems with uschi02 from a dedicated tunnel interface on router. IPv4 pings are successful but IPv6 pings fail. This started 8/14.
uschi02 connectivity test falsely reports my tunnel is down
Shadow Hawkins on Monday, 17 August 2009 20:39:41
For the past few days, I've been receiving automated mails from the tunnel robot claiming that my tunnel endpoint is not responding to pings:
This is the tunnelrobot at SixXS, mailing you to inform you that your tunnel to Your.Org, Inc. has not been replying to pings for a period of time Here's the information regarding the tunnel: Handle : SLN1-SIXXS PoP Name : uschi02 (YOUR-ORG-INC [AS19255]) Your Location: Moline Illinois, us SixXS IPv6 : 2001:4978:f:214::1/64 Your IPv6 : 2001:4978:f:214::2/64 SixXS IPv4 : 216.14.98.22 Your IPv4 : 64.22.192.146 Last known responding: 2009-08-13 10:44:34
We use ICMPv6 ping to reach your IPv6 endpoint every hour. Pings originate from the PoP and your box should reply from your endpoint over the tunnel, but it doesn't.
This is incorrect - my tunnel endpoint is not down, it is passing traffic correctly and I can both ping the uschi02 endpoint from my side and reach that tunnel endpoint from elsewhere on the Internet (via IPv6).
Nothing has changed at all in the networking configuration on my side, so something appears to be wrong with the ping test at the POP. Since the automated mails are threatening that the tunnel will be "user disabled" three days from now if this is not corrected, I would appreciate it if someone would look into this problem soon. (responding to the info@sixxs.net mail does not appear to have elicited any response.)
uschi02 connectivity test falsely reports my tunnel is down
Shadow Hawkins on Tuesday, 18 August 2009 01:28:01
I am experiencing the exact same issue. My tunnel endpoint configuration has not changed since I set it up, it is passing traffic to and from a variety of IPv6 networks, and both IPv4 and IPv6 pings to uschi02 and elsewhere go through just fine.
To date, I have received 4 of these notices:
This is the tunnelrobot at SixXS, mailing you to inform you that your tunnel to Your.Org, Inc. has not been replying to pings for a period of time Here's the information regarding the tunnel: Handle : RRG2-SIXXS PoP Name : uschi02 (YOUR-ORG-INC [AS19255]) Your Location: Chicago, us SixXS IPv6 : 2001:4978:f:1e0::1/64 Your IPv6 : 2001:4978:f:1e0::2/64 SixXS IPv4 : 216.14.98.22 Your IPv4 : 69.47.179.234 Last known responding: 2009-08-13 10:44:34 We use ICMPv6 ping to reach your IPv6 endpoint every hour. Pings originate from the PoP and your box should reply from your endpoint over the tunnel, but it doesn't.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Tuesday, 18 August 2009 00:19:52
Not receiving connectivity for my AYIYA tunnel via uschi02 either.
Getting notified for my site not being pingable
Shadow Hawkins on Tuesday, 18 August 2009 01:34:00
I'm receiving messages titled "Tunnel Downtime for RLS3-SIXXS's 2001:4978:f:33::2".
My machine ping6's fine from the v6 internet. My machine connects fine outward to the v6 internet.
What can I do to stop receiving these erroneous messages?
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Tuesday, 18 August 2009 02:54:20
This is the tunnelrobot at SixXS, mailing you to inform you that your tunnel to Your.Org, Inc. has not been replying to pings for a period of time Here's the information regarding the tunnel:
Handle : GNI1-SIXXS
PoP Name : uschi02 (YOUR-ORG-INC [AS19255])
Your Location: Dallas TX, us
SixXS IPv6 : 2001:4978:f:88::1/64
Your IPv6 : 2001:4978:f:88::2/64
And yet. . .
$ ping6 -q -c 100 noc.sixxs.net
PING6(56=40+8+8 bytes) 2001:4978:f:88::2 --> 2001:838:1:1:210:dcff:fe20:7c7c
--- noc.sixxs.net ping6 statistics ---
100 packets transmitted, 76 packets received, 24.0% packet loss
round-trip min/avg/max/std-dev = 126.920/129.773/138.136/2.628 ms
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Tuesday, 18 August 2009 17:07:05
Currently not able to route past the PoP. Far tunnel end still pings.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Tuesday, 18 August 2009 17:20:35
Routing is back.
Packet loss issue seems to have gone away.
No clue yet if the sixxs can ping my end of the tunnel yet.
$ ping6 -q noc.sixxs.net
PING6(56=40+8+8 bytes) 2001:4978:f:88::2 --> 2001:838:1:1:210:dcff:fe20:7c7c
^C
--- noc.sixxs.net ping6 statistics ---
32 packets transmitted, 32 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 128.321/129.815/132.884/1.119 ms
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Tuesday, 18 August 2009 05:53:42
OK, still down; set myself user-disabled pending the issue being resolved. Please feel free to re-enable and ping the tunnel if desired; it's still configured on my end. Return packets for subnets other than the tunnel itself may be asymmetric however (as my default route is temporarily via HE).
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Tuesday, 18 August 2009 16:49:14
I too am experiencing problems with the above POP. No configuration change has occured on my side for well over 10 weeks and my IPv6 tunnel appears to be working fine. I'm able to PING accross the tunnel with no issue and can ping the Chicago POP's side of the tunnel via IPv6. However the SIXXS.net ping is unable to hit me and has reported that I'm unreachable for the last several days. When I plug my IPV6 tunnel IP into the SIXXS V6 traceroute tool it shows packet loss within the Chicago POP of 40-60% which I think is the cause of their lost PINGs.
I'm getting ready to also open a separate ticket on this but wanted to chime in because I think its related. I am Sixxs Handle DSQ2-SIXXS. My IPV6 address on the my tunnel is 2001:4978:F:1A2::2.
IPv6 traceroute from noc.sixxs.net @ SixXS NOC, AS12871 to 2001:4978:f:1a2::2 :
Hop Node Loss% Sent Last Avg Best Worst StDev ASN Organisation
1. 2001:838:1:1::1 0.0% 5 19.6 4.2 0.4 19.6 8.6 [.nl] Netherlands, The 12871 Concepts ICT
2. 2001:838:5:a::1 0.0% 5 1.9 1.9 1.8 1.9 0.0 [.nl] Netherlands, The 12871 Concepts ICT
3. 2001:7f8:1::a500:6939:1 0.0% 5 2.4 2.4 2.3 2.6 0.1 [.nl] Netherlands, The AMS-IX-IPV6
4. 2001:470:0:3f::1 0.0% 5 10.3 12.4 10.3 20.5 4.6 [.us] United States 6939 Hurricane Electric
5. 2001:470:0:3e::1 0.0% 5 90.6 83.8 79.3 90.6 4.8 [.us] United States 6939 Hurricane Electric
6. 2001:470:0:4e::1 0.0% 5 101.5 107.6 101.2 130.2 12.7 [.us] United States 6939 Hurricane Electric
7. 2001:470:0:7f::2 0.0% 5 102.5 108.6 102.5 132.1 13.2 [.us] United States 6939 Hurricane Electric
8. 2001:4978:1:505::2 0.0% 5 102.7 102.7 102.5 102.8 0.1 [.us] United States 19255 YOUR.ORG, INC.
9. 2001:4978:1:400:b4a3:94ff:fe 40.0% 5 102.6 102.5 102.5 102.6 0.1 [.us] United States 19255 YOUR.ORG, INC.
10. 2001:4978:f:1a2::2 80.0% 5 135.1 135.1 135.1 135.1 0.0 [.us] United States 19255 Your.Org, Inc.
David Swafford.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Tuesday, 18 August 2009 16:59:59
Ok, I've had a chance to run aiccu test now,
I get 100% packet loss when pinging 216.14.98.22 (step 2/8)
the traceroute on step 3/8 dies after the 6th hop from me:
4 scor-01-rif1.mtld.twtelecom.net (66.192.243.134) 14.514 ms 16.733 ms 14.857 ms
5 your.org.ge2-5.br02.chc01.pccwbtn.net (63.218.5.38) 37.927 ms 36.953 ms 37.247 ms
6 ae0-5.core2.chi.bb.your.org (216.14.98.30) 34.093 ms 34.899 ms 33.382 ms
7 * * *
8 * * *
9 * * *
I get 33% packet loss on pinging the inner tunnel end point on step 6/8
The rest of the test complete without issues.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Tuesday, 18 August 2009 19:37:25I get 100% packet loss when pinging 216.14.98.22
+1
Tried to ping from multiple locations in the US.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Wednesday, 19 August 2009 02:39:10
I was docked for no ipv6 ping from the 13th to 17th and my router configuration has not change in 30+ days. I was out of town on buisness, so I didn't realize I had no icmp until I returned today. Tearing down and rebuilding the tunnel seems to have resolved the issue. I believe uschi02 may have an underlying issue.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Wednesday, 19 August 2009 02:45:51
2001:4978:F:378::2/64 is my tunnel end point and my ID is: JCF7-SIXXS
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Wednesday, 19 August 2009 09:34:16
As of today the pop is reachable by IPv4 again and my tunnel came back.
Thanks to whoever fixed this.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Wednesday, 19 August 2009 23:20:59
I can verify that this problem is resolved for my tunnel (T14277).
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Thursday, 20 August 2009 05:21:19
Likewise, my tunnel T16873 is back. Graphs show it returning sometime late in the evening on 8/18.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Thursday, 20 August 2009 13:29:58
As reported, the issues seem to be resolved.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Saturday, 22 August 2009 06:19:51
Looks like the issue is cleared on my side too. Havn't gotten a notification email in 2 days now. Let's hope that it doesn't crop back up.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Shadow Hawkins on Tuesday, 25 August 2009 21:31:11
Actually just curious here but will we be getting the removed ISK's back? This obviously wasn't an issue on our side but rather either on uschi02 or elsewhere. I have 20 ISK's that were removed during the course of this incident.
uschi02 not responding / tunnelinfo for uschi02 tunnel returns error page
Carmen Sandiego on Wednesday, 02 September 2009 04:25:36
I'd like to see my ISKs returned as well.
Username:JCF7-SIXXS
Full name:Jim Coyne
State change: resolved
The state of this ticket has been changed to resolved
|
![[ch]](/s/countries/ch.gif)
on Friday, 14 August 2009 01:34:50
Posting is only allowed when you are