[FQuest Notice] Internap Connectivity

[Terra]

Due to suboptimal packet transfer through their SprintLink uplink, I have decided to pull down our Internap uplink while they perform testing to try and resolve the high saturation they are seeing...

We are now fully "Riding the Light" with Qwest, and all seems to be flowing smoothly...

They will send an email to us when their testing is done...

--
Terra
sysAdmin
FutureQuest

[kitchin]

Sprint DSL had trouble here in Virginia about the same time. Slow then dead. I powered off the modem, and it came back.

[Terra]

We are now turning back up our Internap uplink...

It appears they were hit with a massive DDoS attack against their SprintLink connection, and both Internap and Sprint now have reported it to be under control...

--
Terra
sysAdmin
FutureQuest

[cjcox]

Something seems to be happening again.. not sure if it's related, but everything at futurequest, especially my server :-) seems to be VERY slow or just not responding at all.

[sheila]

Can you please post a traceroute?

See for more details:

http://www.aota.net/Server_Performance/sitedown.php4
http://www.aota.net/Server_Performance/pingtracert.php4

[cjcox]

Traceroute from my comcast IP (12.238.136.47):

# traceroute www.theendlessnow.com
traceroute to www.theendlessnow.com (63.144.246.184), 30 hops max, 40 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

Traceroute from a server on cbeyond (66.180.102.142):
(actually this one is still going on)

/usr/sbin/traceroute theendlessnow.com
traceroute to theendlessnow.com (63.144.246.184), 30 hops max, 38 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *

[sheila]

Ummm...that indicates that something such as a firewall, router or ??? is blocking your traceroutes.

You will need to remove the blocking to allow the traceroutes through...

[cjcox]

No firewall rule in place. Unless both comcast and cbeyond are intentionally blocking access to futurequest. Occasionally I do get stuff to route through though.... just takes forever and after many, many repeated attempts.

[sheila]

Can you log into your CNC and click on "Troubleshooting" and then click on "Ping and Traceroute"?

[cjcox]

Hmmm... just tried from my VPN connect to Sterling Commerce and they can see the site just fine. Weird that the problem is affecting certain backbones and not others. Something's definitely down right now, but it's outside of futurequest's control apparently.

[sheila]

I was able to traceroute from one of the FutureQuest servers to your comcast IP address:


Taceroute to 12.238.136.47
traceroute to 12.238.136.47 (12.238.136.47), 30 hops max, 38 byte packets
1 63.151.147.gw.futurequest.net (63.151.147.1) 1.044 ms 0.788 ms 1.286 ms
2 brdr-01-1 (63.239.113.254) 1.848 ms 1.600 ms 1.591 ms
3 tpa-edge-03.inet.qwest.net (63.144.0.105) 6.071 ms 5.966 ms 5.628 ms
4 tpa-core-01.inet.qwest.net (205.171.27.25) 4.863 ms 5.473 ms 4.768 ms
5 atl-core-01.inet.qwest.net (205.171.5.65) 15.793 ms 17.354 ms 17.491 ms
6 atl-core-02.inet.qwest.net (205.171.21.150) 16.399 ms 15.831 ms 15.566 ms
7 atl-brdr-03.inet.qwest.net (205.171.21.102) 14.830 ms 14.986 ms 14.828 ms
8 qwest-gw.attga.ip.att.net (192.205.32.229) 20.683 ms 20.249 ms 20.329 ms
9 tbr2-p013702.attga.ip.att.net (12.123.20.254) 21.443 ms 22.815 ms *
10 tbr1-cl1.dlstx.ip.att.net (12.122.2.89) 35.813 ms 34.285 ms 35.987 ms
11 gbr5-p20.dlstx.ip.att.net (12.122.12.66) 35.450 ms 33.910 ms 34.795 ms
12 gar2-p360.dlstx.ip.att.net (12.123.17.25) 33.476 ms 34.118 ms 35.117 ms
13 12.244.73.1 (12.244.73.1) 34.287 ms 34.761 ms 34.122 ms
14 12.244.69.70 (12.244.69.70) 35.854 ms 35.486 ms 35.037 ms
15 12.244.113.34 (12.244.113.34) 36.359 ms 35.572 ms 36.193 ms
16 12-238-136-47.client.attbi.com (12.238.136.47) 44.973 ms 96.038 ms 49.748 ms


It looks good from this end. Sure would be nice to see the traceroute going the other way, though, as they are sometimes asymetrical. :\

[cjcox]

It's just plain weird. I cannot do a traceroute at all. Just times out. Nice to see things working in one direction. I have tried both traceroute and tracepath... neither work.

Appears that comcast is blocking certain ICMP messages from reaching me. If I do a ping -R (record route) I get (from a my Linux desktop). This would account for the assymetrical differences... but it's affecting normal behavior in that just access www.theendlessnow.com or pop... etc... is not
operating correctly... ick... I hate comcast:

# ping -R www.theendlessnow.com
PING www.theendlessnow.com (63.144.246.184) 56(124) bytes of data.
64 bytes from theendlessnow.com (63.144.246.184): icmp_seq=1 ttl=49 time=149 ms
RR: behemoth.theendlessnow.com (192.168.1.44)
12.244.113.34
12.244.73.9
gar2-p360.dlstx.ip.att.net (12.123.17.25)
gbr5-p20.dlstx.ip.att.net (12.122.12.66)
tbr1-cl1.dlstx.ip.att.net (12.122.2.89)
tbr2-p012501.attga.ip.att.net (12.122.9.158)
tbr1-p012302.attga.ip.att.net (12.122.2.181)
gbr4-p00.ormfl.ip.att.net (12.123.200.238)

So ping works... just not traceroute (ttl method) or tracepath (fragment method).

[cjcox]

# /usr/sbin/traceroute www.theendlessnow.com
traceroute to www.theendlessnow.com (63.144.246.184), 30 hops max, 40 byte packets
1 10.181.48.1 9.735 ms 9.080 ms 8.792 ms
2 12.244.113.33 13.589 ms 45.217 ms 32.585 ms
3 12.244.73.10 41.560 ms 35.963 ms 19.340 ms
4 gbr5-p80.dlstx.ip.att.net (12.123.17.26) 23.793 ms 10.063 ms 10.687 ms
5 tbr1-p012401.dlstx.ip.att.net (12.122.12.65) 15.794 ms 10.408 ms 11.646 ms
6 tbr2-cl1.attga.ip.att.net (12.122.2.90) 29.235 ms 27.714 ms 50.149 ms
7 tbr1-p012501.attga.ip.att.net (12.122.9.157) 34.996 ms 40.992 ms 30.702 ms
8 gbr4-p40.ormfl.ip.att.net (12.122.2.182) 34.022 ms 38.230 ms 33.419 ms
9 gar1-p360.miufl.ip.att.net (12.123.200.237) 40.871 ms * *
10 gar1-p360.miufl.ip.att.net (12.123.200.237)(H!) 40.220 ms * *

Comcast... just say NO!

[cjcox]

Well... I give up. Things are still moving dog awful slow.
Even my cjcox@acm.org forwarder is having difficulty getting mail to theendlessnow.com. Whatever you can find out....


Something is wrong in internet space.

[LightGuide]

Running like an ice-box from here [Connecticut, via SNET/SBCGlobal and across Verio]... a classic "perfect performance" on our sustained charting utility.

Might I suggest a utility like PingPlotter (no, I don't own the site... lol) -- there is a free version here: http://www.pingplotter.com/download.html

It may perhaps help you to localize just where in the routing the problem is.

etLux

[cjcox]

I talked to comcast... which is like talking to vegetables. Obviously there were greatly offended by the details I was able to give them via Linux about traceroutes, etc. So right now, it's simply a black hole. I'm not totally convinced it's localized to a portion of the comcast network simply because last night I saw similar traceroute issues coming from a machine I have at cbeyond (yet from my office, there was no connectivity issues at all).

[LightGuide]

Originally posted by cjcox:

I talked to comcast... which is like talking to vegetables.

I find it handy to keep one of these nearby whenever dealing with ISP support:

http://shop.ronco.com/jump/dialomatic/

etLux

[asjawebmaster]

via comcast and sprintpcs 1xRTT, slow, then no response, about 10 minutes ago.

now back up.

[asjawebmaster]

ftp and web page loading are still real slow via cable connection.

[LightGuide]

I'm seeing extremely scrappy performance across all our FQ sites at the moment... slow loading, pages stalling, images not loading, etc.

Oddly, sustained trace-route charts are consistently running dead-nuts perfect, absolutely lovely, from one end to the other.

Bizarre... I'd file a trouble ticket -- but I can't see anything screwed up... lol.

EDIT ... a few minutes later

NOW it looks like pretty bad in trace-routes... shudda kept my mouth shut.

etLux

[asjawebmaster]

Just got an emal from a staff person at my site...arrived about 5 3/4 hours after it was sent.

Hmm, something between att and pnap?

1 <1 ms <1 ms <1 ms 192.168.0.1
2 8 ms 9 ms 9 ms 10.134.192.1
3 10 ms 9 ms 13 ms 12.244.81.33
4 11 ms 10 ms 12 ms 12.244.0.13
5 13 ms 11 ms 11 ms 12.244.72.22
6 46 ms 13 ms 10 ms gbr2-p20.st6wa.ip.att.net [12.123.44.150]
7 13 ms 12 ms 12 ms 12.122.12.161
8 47 ms 40 ms 25 ms tbr2-cl1.sffca.ip.att.net [12.122.12.113]
9 35 ms 58 ms 37 ms tbr1-cl3.la2ca.ip.att.net [12.122.10.26]
10 63 ms 70 ms 59 ms tbr1-p012501.dlstx.ip.att.net [12.122.10.94]
11 104 ms 76 ms 76 ms tbr2-cl1.attga.ip.att.net [12.122.2.90]
12 112 ms 76 ms 75 ms tbr1-p012501.attga.ip.att.net [12.122.9.157]
13 81 ms 116 ms 81 ms gbr4-p50.ormfl.ip.att.net [12.122.2.130]
14 91 ms 90 ms 91 ms gar1-p360.miufl.ip.att.net [12.123.200.237]
15 * * * Request timed out.
16 97 ms 113 ms 131 ms border11.ge3-0-bbnet2.mia.pnap.net [216.52.160.79]
17 112 ms 133 ms * futurequest-1.border11.mia.pnap.net [216.52.162.126]
18 130 ms 102 ms 98 ms core-01-2.futurequest.net [63.239.113.253]
19 96 ms 143 ms 102 ms x51.futurequest.net [63.151.147.51]

[LightGuide]

Yeah, lotta scrap on the last three nodes before the domain -- fairly usual problem, I think. Terra generally jumps all over things like this right quickly, though... already looks clear from here; and site performance seems to be improving.

etLux

[Evoir]

Got a call from a client in the same city as me who is having the "slow like mollassas" response from their FQ hosted site. We even use the same DSL provider. They can browse other sites no problem. Is there something going on again with the connectivity? Also, their email is sorta stuck. The way they describe it, it sounds just like what happened a few days ago to me when this issue was going on.

Unfortunately, they are on macs and have a firewall, and don't have the immediate ability to do a traceroute. I'm looking into getting them up to speed on this.

[Terra]

We will be turning down the Internap uplink shortly to switch the DS-3 cables to a backup line card... I feel there may be some intermittant issues with the current line card, though I cannot find any errors being reported by the router or line card itself...

The maintenance should be pretty much transparent, except for around 30 seconds while the BGP4 routes change to our Qwest link...

--
Terra
sysAdmin
FutureQuest

[cindik]

Originally posted by Evoir:
Got a call from a client in the same city as me who is having the "slow like mollassas" response from their FQ hosted site. We even use the same DSL provider. They can browse other sites no problem. Is there something going on again with the connectivity? Also, their email is sorta stuck. The way they describe it, it sounds just like what happened a few days ago to me when this issue was going on.

Unfortunately, they are on macs and have a firewall, and don't have the immediate ability to do a traceroute. I'm looking into getting them up to speed on this.

QWest Denver was having all kinds of trouble yesterday evening. Completely killed access from here (attbi.com in the Chicago area) to my sites, although I was chatting to people on IRC who could still access them. Your customers might have been caught up in the same storm.

The Internet backbone is like that - one vertebra skips, and numbness can appear anywhere.

This is a fun site: internet pulse (http://www.internetpulse.com)