Ticket ID: SIXXS #606528 Ticket Status: User PoP: nlams04 - Stipte B.V. (Amsterdam)
Tunnel outgoing-only since this morning?
Shadow Hawkins on Monday, 12 November 2007 14:45:33
I'm having trouble with my heartbeat tunnel on nlams4.
Yesterday till shutdown of the router IPv6 was working.
This morning tunnel is not working anymore. (after investigating looks like the tunnel became unidirectionnal)
From my host connected through the tunnel I can send ICMP echo request to the world but the replies never arrive (tcpdump on the target machine shows incoming echo requests and responses it sends out).
When pinging some address behind the tunnel from internet I just get the following:
From broker04.ams.nl.sixxs.net icmp_seq=10 Destination unreachable: No route
Tracepath6 from world ends with:
ge-1-3-0.pe02.intx.ipv6.network.scarlet.nl asymm 13 54.712ms
broker04.ams.nl.sixxs.net 54.948ms !N
The cisco router handling my tunnel sees no incoming packets on the tunnel.
Ping to broker endpoint of the tunnel (IPv6) does not work from either behind tunnel nor from internet.
Tunnel outgoing-only since this morning?
Jeroen Massar on Monday, 12 November 2007 17:29:38
Really. Please read the contact page and then the "Reporting Problems" section and be a lot more verbose.
Note also that you have a heartbeat tunnel, but you mention "a cisco". As such, are you running a heartbeat client? A tunnel which is not up, is not configured and thus indeed you will get a !N from the PoP.
State change: user
Jeroen Massar on Monday, 12 November 2007 17:29:41
The state of this ticket has been changed to user
Tunnel outgoing-only since this morning?
Shadow Hawkins on Monday, 12 November 2007 20:50:04
Let's describe my setup in more detail:
PC 1 \
PC 2 ------ Cisco Router (NAT) ---- Inet
PC 3 / \ PC 4
PC 1 running AICCU (version from 2007.01.15), installed using Gento Portage.
The Cisco router has IPv6 over IPv4 tunnel configured as described in the FAQ but using it's dialer interface as local endpoint (I have dynamic IPv4 address and can't use static tunnels for this reason)
I'm using my tunnel T12142, heartbeat being sent from PC1 heartbeat message looks like:
HEARTBEAT TUNNEL <tunnel endpoint> sender <timestamp> <hash>
My clock is synchronized against ntp server (pool.ntp.org)
PC 1 can send icmpv6 echo requests to computers on the internet (eg PC 4) but the reply never reaches the cisco router (incoming packet count remains zero, outgoing packet count increases)
tcpdump on PC 4 sees the incoming echo requests and also the outgoing echo replies but nothing arrives on cisco router.
My setup has not changed at all between yesterday evening (working) and today (not working anymore).
I usually shutdown my complete LAN on evening to save power over night and on the following morning it comes up without any issue.
Only explanation I would have is some change on the broker side.
If the heartbeat was not seen be the broker (nlams4) the echo requests should never reach the internet...
Tunnel information page shows some obsolete IP address for my endpoint.
From PC 4 the broker endpoint of my tunnel is not reachable (ping), aiccu test from PC 1 fails on ping to broker endpoint of my tunnel.
Tunnel outgoing-only since this morning?
Jeroen Massar on Monday, 12 November 2007 20:54:12
I have a very simple thing for you:
Last Beat : 2007-11-11 21:16:29 (77615 seconds ago)
In other words, you are not sending any (correct) heartbeats to the PoP.
As such no tunnel will be configured.
Nothing on the PoP changed.
Tunnel outgoing-only since this morning?
Shadow Hawkins on Monday, 12 November 2007 23:35:50
How may I find out what's wrong?
Is there a way to check the broker's time to make sure there is no excessive offset between it and time on my host?
At least my clock matches the time reported by tic server.
Tunnel outgoing-only since this morning?
Shadow Hawkins on Tuesday, 13 November 2007 00:46:27
I've been playing with my clock:
- time offset (ntpdate -q pool.ntp.org) == 0s
--> tunnel does not work
- time offset (ntpdate -q pool.ntp.org) == 48s
--> tunnel does work
So I guess that either pool.ntp.org is drifting (unprobable) or the nlams4 is drifting...
Tunnel outgoing-only since this morning?
Jeroen Massar on Tuesday, 13 November 2007 10:54:12
All the PoPs are ntp synced with stratum 1 servers, not with stratum 3 and up ones as provided by pool.ntp.org.
Tunnel outgoing-only since this morning?
Shadow Hawkins on Tuesday, 13 November 2007 11:41:32
Here are my ntp results in verbose:
# ntpdate -q pool.ntp.org chronos.restena.lu
server 195.160.166.150, stratum 2, offset -0.032798, delay 0.09590
server 213.249.66.35, stratum 2, offset -0.025136, delay 0.08887
server 141.40.103.101, stratum 2, offset -0.033208, delay 0.06595
server 130.226.232.145, stratum 2, offset -0.032853, delay 0.07362
server 85.91.1.164, stratum 2, offset -0.034937, delay 0.08374
server 193.40.133.142, stratum 1, offset -0.033198, delay 0.09010
server 158.64.1.15, stratum 1, offset -0.033735, delay 0.05280
13 Nov 11:36:04 ntpdate[14413]: adjust time server 158.64.1.15 offset -0.033735 sec
Note that I have seen servers with ntpd running going out of sync (synced on stratum 1 server on same subnet), as such a check on nlams4 would not hurt (also comparing to multiple stratum 1 servers)
Tunnel outgoing-only since this morning?
Jeroen Massar on Tuesday, 13 November 2007 11:45:09
Due to the importance time synchronisation of those PoPs the system already monitors that they are running perfectly in sync.
Just in case you wonder, every time that the PoPs configuration gets updated we also query the time of the PoP and compare that to noc.sixxs.net. If they are off more than 5 seconds (negative and positive) then a warning is raised. This could mean two things: noc is off or the PoP is off. Both are not the case here.
Posting is only allowed when you are logged in. |