FQuest Alert: Connectivity

[Monty]

Terra, the speed is already greatly improved.[nbsp][nbsp]I don't understand the technical issues involved but have full confidence you will get the the bottom of it, and work it all out.[nbsp][nbsp]In the mean time, I shaved a few features off our boards to lighten the load.[nbsp][nbsp]We are having our 10th gathering from our website tomorrow, and could never have done all that we have done without FQ.[nbsp][nbsp]

Mont

[Benj]

Terra, the speed is already greatly improved.
Not here :(

Benj

[Deb]

Benj,

Things should be running quite smoothly right now for everyone.[nbsp][nbsp]Can you please send a copy of a trace route to the Service Desk so that we may have a look....

Deb
[nbsp]- Do you see what I see....

[Terra]

I have been monitoring the internal network closely, and we are no longer encountering those annoying potholes anymore...[nbsp][nbsp]Everything appears to be smooting out...

If you are experiencing any slowness, the please run trace routes as I am sure it is outside of our network/realm...

--
Terra
--Asphalt + gravel + shovel, is how it was fixed--
FutureQuest

[Terra]

FutureQuest has been suffering from intermittant connectivity from the edge router...[nbsp][nbsp]These hot spots last anywhere from a few minutes, to the one where the UUnet failover line had to kick in via BGP-4 routing...[nbsp][nbsp]The hot spots have also been popping up randomly, but have almost always been spaced by 16 hours or more...[nbsp][nbsp]This, I agree, is completely unacceptable...

Currently, the network engineers are in fault isolation mode trying to track down this very frustrating and elusive cause for the dropped packets...

After we unleashed our bandwidth onto the new network, we have started to surface problems in the routers that were not known prior...[nbsp][nbsp]Since this is a maiden voyage, we are working steadfast to shake out any remaining glitches and bring full stabilization to the network...

As a side effect, we are increasing the bandwidth of our UUnet failover pipes to compensate for any problems encountered with the QWest lines...

I have high confidence that these intermittant connectivity issues will slowly dissolve as the internal network is strengthed up even further...

I wish I could give everyone an immediate solution, but problem isolation, such as this, is a very time consuming and exhausting process...[nbsp][nbsp]I can only hope that we find and solve the problem - sooner rather than later...

Our apologies for the inconvenience this is causing everyone...

--
Andrew Gillespie
CTO/Systems Administrator
FutureQuest, Inc.

[Benj]

Well right now everything works at full speed :)
I ran a tracert, and I noticed a decrease of 5 hops compared to yesterday when it was slow :)

Benj

[Dean B]

I've been experiencing a real slowness connecting to sites on my domain over the last few days or so.[nbsp][nbsp]Unlike Benj, since switching to the new servers my hops have increased from 12 to 20.[nbsp][nbsp]After the initial change connectivity was very quick though.
Current tracert:

[nbsp][nbsp]1[nbsp][nbsp] 142 ms[nbsp][nbsp] 115 ms[nbsp][nbsp] 145 ms[nbsp][nbsp]nsv1-belinda.tch.cableol.net [194.168.10.241]
[nbsp][nbsp]2[nbsp][nbsp] 147 ms[nbsp][nbsp] 140 ms[nbsp][nbsp] 136 ms[nbsp][nbsp]ltn-pleiades-e800.inet.ntl.com [194.168.10.251]

[nbsp][nbsp]3[nbsp][nbsp] 147 ms[nbsp][nbsp] 140 ms[nbsp][nbsp] 145 ms[nbsp][nbsp]ltn-core-a-pos1100.inet.ntl.com [62.252.64.145]

[nbsp][nbsp]4[nbsp][nbsp] 152 ms[nbsp][nbsp] 155 ms[nbsp][nbsp] 177 ms[nbsp][nbsp]win-core-f-atm500-3.inet.ntl.com [194.168.118.34
]
[nbsp][nbsp]5[nbsp][nbsp] 142 ms[nbsp][nbsp] 145 ms[nbsp][nbsp] 125 ms[nbsp][nbsp]above-ntl.lhr.above.net [208.185.188.29]
[nbsp][nbsp]6[nbsp][nbsp] 234 ms[nbsp][nbsp] 237 ms[nbsp][nbsp] 240 ms[nbsp][nbsp]nyc-lhr-stm4-2.above.net [208.184.231.21]
[nbsp][nbsp]7[nbsp][nbsp] 253 ms[nbsp][nbsp] 211 ms[nbsp][nbsp] 226 ms[nbsp][nbsp]core1-core2-oc48.nyc.above.net [209.133.31.201]

[nbsp][nbsp]8[nbsp][nbsp] 208 ms[nbsp][nbsp] 226 ms[nbsp][nbsp] 226 ms[nbsp][nbsp]qwest-nyc-oc12.above.net [208.185.156.26]
[nbsp][nbsp]9[nbsp][nbsp] 228 ms[nbsp][nbsp] 205 ms[nbsp][nbsp] 241 ms[nbsp][nbsp]jfk-core-02.inet.qwest.net [205.171.30.17]
[nbsp]10[nbsp][nbsp] 204 ms[nbsp][nbsp] 238 ms[nbsp][nbsp] 220 ms[nbsp][nbsp]jfk-core-01.inet.qwest.net [205.171.30.1]
[nbsp]11[nbsp][nbsp] 244 ms[nbsp][nbsp] 216 ms[nbsp][nbsp] 232 ms[nbsp][nbsp]wdc-core-02.inet.qwest.net [205.171.5.235]
[nbsp]12[nbsp][nbsp] 246 ms[nbsp][nbsp] 236 ms[nbsp][nbsp] 231 ms[nbsp][nbsp]wdc-core-03.inet.qwest.net [205.171.24.6]
[nbsp]13[nbsp][nbsp] 214 ms[nbsp][nbsp] 241 ms[nbsp][nbsp] 232 ms[nbsp][nbsp]atl-core-02.inet.qwest.net [205.171.5.243]
[nbsp]14[nbsp][nbsp] 241 ms[nbsp][nbsp] 246 ms[nbsp][nbsp] 256 ms[nbsp][nbsp]tpa-core-01.inet.qwest.net [205.171.5.65]
[nbsp]15[nbsp][nbsp] 247 ms[nbsp][nbsp] 243 ms[nbsp][nbsp] 237 ms[nbsp][nbsp]tpa-edge-03.inet.qwest.net [205.171.27.26]
[nbsp]16[nbsp][nbsp] 241 ms[nbsp][nbsp] 251 ms[nbsp][nbsp] 242 ms[nbsp][nbsp]63.144.0.106
[nbsp]17[nbsp][nbsp] 238 ms[nbsp][nbsp] 241 ms[nbsp][nbsp] 241 ms[nbsp][nbsp]dmcity.com [63.95.221.200]

---Dean---

[Justin]

As TeRRa has mentioned in previous threads, number of hops is relative. I was only 7 hops from the old NOC, and am currently > 20 hops from the new NOC. However, despite this, the speed is much faster now.

Qwest has a different outlook on routing - rather than having a small amount of very large POPs, they have many small POPs, thus reducing the possibility of SPOF (single point of failure). So there are many hops, and the route can take a different path when needed, as there are many paths to choose from. You can see this for yourself by doing trace routes throughout the day, and comparing them. You will notice that the route differs quite a lot, showing that as one POP becomes full, your packets will route around that POP through a different one.

This is a newer way of thinking. The old way is to have one large POP in each major city, and in many cases (<cough>Atlanta</cough>) when one goes down, there is no other way through. This old system worked when the network currently known as the Internet was only used for voice communications, but these days, the 'one size fits all' theory no longer works.

To summarize - don't take the number of hops to be a bad thing. Contrary to popular belief, it's not the number of hops, rather it's how long it takes to get through them collectively (adding the times isn't the result you're looking for either; each hop time is the time it takes a packet to go to that hop and back - eg, through all the previous hops - so it's the final hop time that matters).

Hope this clears any confusion.

------------------
Justin Nelson
FutureQuest (http://www.FutureQuest.net/index.php) Support

[Terra]

Dean:

That's a pretty darn good looking traceroute, as it shows good consistency and low latency...

There is one major concern though:
142 ms[nbsp][nbsp] 115 ms[nbsp][nbsp] 145 ms nsv1-belinda.tch.cableol.net

Just starting out of the gate you are already off to a bad start...

Yet overall:
[nbsp]238
-142
====
[nbsp][nbsp]96 ms rise

I just don't see a problem here no matter how hard I look...

--
Terra
--It wouldn't bother me so much if we weren't so dependent on the Internet to do the right thing--
FutureQuest

[DKG]

I'm still having problems, but I think it is the @home hop in[nbsp][nbsp]atlanta... anyway

<blushing here>

I thought there was a switch that allowed me to &quot;print&quot; the tracert or ping data to a text file, but I can't remember or find it. ? I've just been using the highlight/copy feature, which is touch n go, I seldom catch the last few hops, and I don't have copies of the recent slow ones.[nbsp][nbsp]

(The problem with using other sites to do it for me is that they use a different route, which wouldn't help me find out and report the weak hop to @home.)


[This message has been edited by DKG (edited 09-09-00@2:29 pm)]

[Justin]

In DOS:
</font><font face="Courier" size="3">
C:\> tracert www.domain.com (http://www.domain.com) >trace.txt
</font><font face="Verdana, Arial" size="2">
This will redirect (>) the output to a file (trace.txt) in the current directory (\).

Hope this helps.

------------------
Justin Nelson
FutureQuest (http://www.FutureQuest.net/index.php) Support

[JoeRT]

While my traceroutes are looking pretty good, what I'm noticing is hesitation.[nbsp][nbsp]It's like........ some of the information....... gets.......................... sent but there can be.... these........................................................... pauses of between.... a few and..........................................quite a few seconds.. up to around 15 seconds............. followed by huge bursts of data coming through all at once.[nbsp][nbsp]Bad way to demonstrate it, but the only way I could think of.[nbsp][nbsp]:D[nbsp][nbsp]I'm noticing this on HTTP and FTP.[nbsp][nbsp]For instance, I'll be doing a directory listing with FTP, it will get part way through, pause, get some more, pause, etc.[nbsp][nbsp]Same with HTTP... part of the page will load and it will just sit there for a while before it loads more, usually at least two or three pauses.[nbsp][nbsp]I didn't notice this before the move and in fact not up until about three days ago.[nbsp][nbsp]So I don't know.

For reference, here's a traceroute from beautiful downtown Sacramento to you over ISDN...

Tracing route to tiswest.com [63.95.221.40]
over a maximum of 30 hops:

[nbsp][nbsp]1[nbsp][nbsp][nbsp][nbsp]75 ms[nbsp][nbsp][nbsp][nbsp]97 ms[nbsp][nbsp][nbsp][nbsp]77 ms[nbsp][nbsp]tnt4-e1.scrm01.pbi.net [206.171.130.82]
[nbsp][nbsp]2[nbsp][nbsp][nbsp][nbsp]76 ms[nbsp][nbsp][nbsp][nbsp]75 ms[nbsp][nbsp][nbsp][nbsp]74 ms[nbsp][nbsp]core1-e3-5.scrm01.pbi.net [206.171.130.85]
[nbsp][nbsp]3[nbsp][nbsp][nbsp][nbsp]73 ms[nbsp][nbsp][nbsp][nbsp]75 ms[nbsp][nbsp][nbsp][nbsp]81 ms[nbsp][nbsp]edge1-fa0-0-0.scrm01.pbi.net [206.13.31.8]
[nbsp][nbsp]4[nbsp][nbsp][nbsp][nbsp]75 ms[nbsp][nbsp][nbsp][nbsp]77 ms[nbsp][nbsp][nbsp][nbsp]85 ms[nbsp][nbsp]sfra1sr1-3-0-0.ca.us.prserv.net [165.87.161.94]
[nbsp][nbsp]5[nbsp][nbsp][nbsp][nbsp]75 ms[nbsp][nbsp][nbsp][nbsp]74 ms[nbsp][nbsp][nbsp][nbsp]74 ms[nbsp][nbsp]sfra1sr3-ge-2-1-0-0.ca.us.prserv.net [165.87.33.187]
[nbsp][nbsp]6[nbsp][nbsp][nbsp][nbsp]92 ms[nbsp][nbsp][nbsp][nbsp]76 ms[nbsp][nbsp][nbsp][nbsp]78 ms[nbsp][nbsp]svl-brdr-01.inet.qwest.net [205.171.1.17]
[nbsp][nbsp]7[nbsp][nbsp][nbsp][nbsp]77 ms[nbsp][nbsp][nbsp][nbsp]78 ms[nbsp][nbsp][nbsp][nbsp]77 ms[nbsp][nbsp]svl-core-03.inet.qwest.net [205.171.14.109]
[nbsp][nbsp]8[nbsp][nbsp][nbsp][nbsp]97 ms[nbsp][nbsp][nbsp][nbsp]80 ms[nbsp][nbsp][nbsp][nbsp]75 ms[nbsp][nbsp]sjo-core-03.inet.qwest.net [205.171.14.102]
[nbsp][nbsp]9[nbsp][nbsp] 101 ms[nbsp][nbsp][nbsp][nbsp]78 ms[nbsp][nbsp][nbsp][nbsp]77 ms[nbsp][nbsp]sjo-core-02.inet.qwest.net [205.171.22.5]
[nbsp]10[nbsp][nbsp] 119 ms[nbsp][nbsp] 118 ms[nbsp][nbsp] 119 ms[nbsp][nbsp]hou-core-02.inet.qwest.net [205.171.5.145]
[nbsp]11[nbsp][nbsp] 124 ms[nbsp][nbsp] 119 ms[nbsp][nbsp] 118 ms[nbsp][nbsp]hou-core-03.inet.qwest.net [205.171.23.6]
[nbsp]12[nbsp][nbsp] 132 ms[nbsp][nbsp] 130 ms[nbsp][nbsp] 133 ms[nbsp][nbsp]atl-core-01.inet.qwest.net [205.171.5.193]
[nbsp]13[nbsp][nbsp] 130 ms[nbsp][nbsp] 133 ms[nbsp][nbsp] 128 ms[nbsp][nbsp]atl-core-02.inet.qwest.net [205.171.21.2]
[nbsp]14[nbsp][nbsp] 158 ms[nbsp][nbsp] 141 ms[nbsp][nbsp] 140 ms[nbsp][nbsp]tpa-core-01.inet.qwest.net [205.171.5.65]
[nbsp]15[nbsp][nbsp] 142 ms[nbsp][nbsp] 154 ms[nbsp][nbsp] 142 ms[nbsp][nbsp]tpa-edge-03.inet.qwest.net [205.171.27.26]
[nbsp]16[nbsp][nbsp] 147 ms[nbsp][nbsp] 147 ms[nbsp][nbsp] 145 ms[nbsp][nbsp]63.144.0.106
[nbsp]17[nbsp][nbsp] 159 ms[nbsp][nbsp] 158 ms[nbsp][nbsp] 154 ms[nbsp][nbsp]tiswest.com [63.95.221.40]

[jimbo]

JoeRT - I thought I was the only one.[nbsp][nbsp]That is *exactly* what's been happening on my site.[nbsp][nbsp]I've been trying to download a mysql dump for two days now.

-jim

[Dean B]

Guys,
that's exactly what I'm experiencing, just I didn't explain myself very well.[nbsp][nbsp]Fast .... pause .....fast.....pause....pause....fast

[eunice]

Yeah, me too.[nbsp][nbsp]fwiw
------------------
--Laurie

[esc]

Hi,
I see a lot of lost packets here form Vienna (Austria). For me it's not the signal runtime, but the packet loss that makes the performance so bad. Here ist an edited output of a view pings:

PING astro.futurequest.net (63.95.221.104): 56 data bytes
...
--- astro.futurequest.net ping statistics ---
20 packets transmitted, 10 packets received, 50% packet loss
round-trip min/avg/max = 185.928/187.186/189.859 ms

PING astro.futurequest.net (63.95.221.104): 56 data bytes
...
64 bytes from 63.95.221.104: icmp_seq=19 ttl=223 time=186.470 ms
--- astro.futurequest.net ping statistics ---
20 packets transmitted, 13 packets received, 35% packet loss
round-trip min/avg/max = 184.999/193.377/263.458 ms

PING astro.futurequest.net (63.95.221.104): 56 data bytes
...
--- astro.futurequest.net ping statistics ---
20 packets transmitted, 13 packets received, 35% packet loss
round-trip min/avg/max = 185.637/187.256/190.007 ms

PING astro.futurequest.net (63.95.221.104): 56 data bytes
...
--- astro.futurequest.net ping statistics ---
20 packets transmitted, 16 packets received, 20% packet loss
round-trip min/avg/max = 185.838/187.545/194.521 ms

[ubu]

OK call me a dummy..[nbsp][nbsp]but I have no idea what I am looking at.[nbsp][nbsp]My site is running fine, just curious to see what this trace route looks like.

Tracing route to openminded.com [63.95.221.23] over a maximum of 30 hops:

[nbsp][nbsp]1[nbsp][nbsp][nbsp][nbsp] 4 ms[nbsp][nbsp][nbsp][nbsp] 3 ms[nbsp][nbsp][nbsp][nbsp] 3 ms[nbsp][nbsp]24.147.88.1
[nbsp][nbsp]2[nbsp][nbsp][nbsp][nbsp] 4 ms[nbsp][nbsp][nbsp][nbsp] 5 ms[nbsp][nbsp][nbsp][nbsp] 4 ms[nbsp][nbsp]24.128.5.241
[nbsp][nbsp]3[nbsp][nbsp][nbsp][nbsp] 5 ms[nbsp][nbsp][nbsp][nbsp] 4 ms[nbsp][nbsp][nbsp][nbsp] 4 ms[nbsp][nbsp]12.125.39.9
[nbsp][nbsp]4[nbsp][nbsp][nbsp][nbsp] 5 ms[nbsp][nbsp][nbsp][nbsp] 5 ms[nbsp][nbsp][nbsp][nbsp] 5 ms[nbsp][nbsp]gbr1-p70.cb1ma.ip.att.ne[12.123.40.98]
[nbsp][nbsp]5[nbsp][nbsp][nbsp][nbsp] 5 ms[nbsp][nbsp][nbsp][nbsp] 5 ms[nbsp][nbsp][nbsp][nbsp] 5 ms[nbsp][nbsp]gbr4-p70.cb1ma.ip.att.net [12.122.5.57]
[nbsp][nbsp]6[nbsp][nbsp][nbsp][nbsp]29 ms[nbsp][nbsp][nbsp][nbsp]30 ms[nbsp][nbsp][nbsp][nbsp]29 ms[nbsp][nbsp]gbr4-p40.cgcil.ip.att.net [12.122.2.49]
[nbsp][nbsp]7[nbsp][nbsp][nbsp][nbsp]29 ms[nbsp][nbsp][nbsp][nbsp]28 ms[nbsp][nbsp][nbsp][nbsp]31 ms[nbsp][nbsp]ggr1-p370.cgcil.ip.att.net [12.123.5.149]
[nbsp][nbsp]8[nbsp][nbsp][nbsp][nbsp]30 ms[nbsp][nbsp][nbsp][nbsp]31 ms[nbsp][nbsp][nbsp][nbsp]32 ms[nbsp][nbsp]192.205.32.142
[nbsp][nbsp]9[nbsp][nbsp][nbsp][nbsp]30 ms[nbsp][nbsp][nbsp][nbsp]31 ms[nbsp][nbsp][nbsp][nbsp]30 ms[nbsp][nbsp]chi-core-02.inet.qwest.net [205.171.20.141]
[nbsp]10[nbsp][nbsp][nbsp][nbsp]35 ms[nbsp][nbsp][nbsp][nbsp]35 ms[nbsp][nbsp][nbsp][nbsp]36 ms[nbsp][nbsp]wdc-core-01.inet.qwest.net [205.171.5.225]
[nbsp]11[nbsp][nbsp][nbsp][nbsp]36 ms[nbsp][nbsp][nbsp][nbsp]36 ms[nbsp][nbsp][nbsp][nbsp]35 ms[nbsp][nbsp]wdc-core-03.inet.qwest.net [205.171.24.9]
[nbsp]12[nbsp][nbsp][nbsp][nbsp]46 ms[nbsp][nbsp][nbsp][nbsp]46 ms[nbsp][nbsp][nbsp][nbsp]47 ms[nbsp][nbsp]atl-core-02.inet.qwest.net [205.171.5.243]
[nbsp]13[nbsp][nbsp][nbsp][nbsp]56 ms[nbsp][nbsp][nbsp][nbsp]57 ms[nbsp][nbsp][nbsp][nbsp]57 ms[nbsp][nbsp]tpa-core-01.inet.qwest.net [205.171.5.65]
[nbsp]14[nbsp][nbsp][nbsp][nbsp]57 ms[nbsp][nbsp][nbsp][nbsp]56 ms[nbsp][nbsp][nbsp][nbsp]57 ms[nbsp][nbsp]tpa-edge-03.inet.qwest.net [205.171.27.26]
[nbsp]15[nbsp][nbsp][nbsp][nbsp]62 ms[nbsp][nbsp][nbsp][nbsp]62 ms[nbsp][nbsp][nbsp][nbsp]62 ms[nbsp][nbsp]63.144.0.106
[nbsp]16[nbsp][nbsp][nbsp][nbsp]62 ms[nbsp][nbsp][nbsp][nbsp]61 ms[nbsp][nbsp][nbsp][nbsp]61 ms[nbsp][nbsp]openminded.com [63.95.221.23]

Jamie

[Terra]

In the last week, a lot has happened...

Moving to a new NOC / Switching to a new network / and a lot of other minor items required to make all of this work a reality...

What I am faced with right now, is that I cannot use the tried-and-true diagnosing technique of 'What Changed?', because just about everything has... ;)

I do have some ideas as to what is causing the dropped packets, but the instances are so small that I have not gotten anything conclusive yet...

1) People bouncing between the old IP's and new IP's
2) Edge router issue with timing, as we have just pulled an ATM card that was causing jabber on the backplane
3) Tunneling is in full force
4) Foundry and Cisco switches not getting along
5) Community Servers running very advanced routing rules for the tunnels

#1) I have seen this happen as they are still in propagation and not fully settled yet as to which way to go...
#2) This was worked on yesterday afternoon, and the ATM card pulled around 1:00pm...[nbsp][nbsp]This seemed to have cleared up the large potholes...[nbsp][nbsp]:)
#3) This is causing a whole lot of extra work for all networking components (router/switch/servers) as tunneling introduces a high amount of overhead as the packets flow:
Inbound:
208.243.19_router --> tunnel-old --> 63.95.221_router --> tunnel-new --> server
Outbound:
server --> tunnel-new --> router --> tunnel-old --> router
#4) I have an email off to both Foundry and Cisco concerning interoptability
#5) We are using bleeding/cutting edge routing rulesets that make the tunnels a reality...

**Make note that TAZ could not be routed through the tunnels as the advanced routing just would not work for TAZ...[nbsp][nbsp]However, all other servers are tunneled for the old IP's...

Overall, our network right now is severely taxed to ensure that everyone can reach your sites...[nbsp][nbsp]If they do not see your new IP's and are still going to your old (208.243.19.x) IP's - then they will still make it to your domain...

I plan on starting to drop the tunnels around 9/20/00, and anyone still going to the old 208.243.19.x network will need to nudge their ISP to fix their DNS as propagation should have happened long ago since I had everyones domain set to a 7616 second TTL which had over 1 week to propagate the cutover aging...[nbsp][nbsp]Other ISP's broken DNS, we cannot control - be rest assured we have done our best to accomodate them as well by investing over 2 months of development building the whole tunnel system...

So right now, be aware that I am not turning a deaf ear to anyones report of slowness - but realize that I almost must be cautious as to what I accept as being our systems...[nbsp][nbsp]Last thing I need right now is chasing phantoms...

I firmly believe that after I remove the overhead of the tunneling system (it has served it's purpose), then everything should start smoothing out to more level loads, instead of dealing with the random spikes coming in from the old network...

If the majority rules to cutoff the old network, then I would be more than *happy* to do so...[nbsp][nbsp]Just realize that you may lose up to 3% to 5% of your visitors - until they fully propagate over...[nbsp][nbsp]**But**, once it is taken offline - I will not put it back online as there is just way too much work involved in doing so...[nbsp][nbsp]Else, we will find out if the tunnel is the culprit on 9/20...

I know everyone is accustomed to speedy solutions, but I must admit this type of situation is most agonizing timewise and very problematic to isolate and fix...[nbsp][nbsp]It really is like trying to find a needle in a haystack as of all the millions of packets that flow through our networks, potentially less than .00001% may actually be causing a problem...

--
Terra
--A fast direct connection to FutureQuest servers, if you elect me for President--
FutureQuest

[YFS200]

I have had/have the same hesitation problem too. It's mostly intermittent so it's hard to track. This seems to work. When the hesitation kicks in, I continuously ping the server. PING has a switch (don't recall it) that will make it ping forever.[nbsp][nbsp]I open up a window and just watch it ping. Every few second, it will time out. Sometimes once or twice, or even for a few minutes. When I see it time out, I quickly do a traceroute in another window. If luck is with me, it will show the router that is failing.

Most of the time, it's my ISP. Seems like they unplug the router, plug in Mr.Coffee, make a pot, then plug the router back in.

YFS200[nbsp][nbsp][nbsp]

[Dean B]

Well it's 11:30am here in the UK and TAZ is racing along like on day #1.[nbsp][nbsp]Could be it's just waiting for you US guys n gals to wake up and log on and bring us to a standstill again ;)[nbsp][nbsp]or something else has kicked in and rectified something.[nbsp][nbsp]I can say that during week days it hasn't been this fast this early in the day.
I'll report back in 8hrs :)

---Dean---

[jwbutler]

Any ISP that doesn't update their routing tables weekly is IMHO, way behind the curve.

Here's my vote to take the tunnels down.
------------------
Joseph Butler
Starfleetgames.com Webmaster

[wolfstalks]

And Mine,[nbsp][nbsp]Take the Tunnels Down!!!

TeRRa, you and the &quot;FQ&quot; team have done a great job, let's finish it!

Bob

[Benj]

Yes, it starts to get really boring.
I do not[nbsp][nbsp]complain to complain, its just I think this lag......good....lag is bad for the entire community feeling.

I do know however that FQ is is as worried than us if not more about this.

Benj

[Dan Kaplan]

I'm inclined to agree that the tunneling has served its purpose and ought to be retired.[nbsp][nbsp]However... I can already hearing the site owners screaming when a few visitors can't get through because of the collapsed tunnel...

Can a page like the temporary &quot;moving in progress&quot; page be put up on the old IP, saying something like, &quot;If you can see this page, your ISP sucks, it's not the site's fault...?&quot;[nbsp][nbsp]I don't know how all that stuff works or how much trouble is involved, so it may or may not be possible.

With all this talk of tunneling, I can't help but think Terra was really planning his escape route in case things didn't go well with the new NOC move (&quot;Please, no NOC-NOC jokes, or I'll dig, I mean it!&quot;).[nbsp][nbsp]:P

Dig

[jimbo]

I've also noticed that all my pages that rely heavily on mysql02 are slow now.[nbsp][nbsp]I was surfing BenJ's site, and his was acting the same way as mine.

Just thought I'd bring that up, though I have no idea if that has anything to do with the connectivity.

-jim

[Terra]

I have just pulled the tunnels offline, and de-tunneled the SIX server...

Due to a router problem with it's ARP cache, I am halting all further work on de-tunneling the other servers until I can be assured that the router won't go brain dead on that particular server's range of listening IP(s)...

IOW, when I brought SIX back up - only 2 IP(s) out of 35 made it through the router...[nbsp][nbsp]It took a full flush to clear out the problem and bring those last 35 IP(s) online...

This is not normal behavior by any means, and as such, will be working on the router to resolve this ARP issue...

--
Terra
--I will get to the bottom of this, as I feel a fury coming on--
FutureQuest

[Benj]

This forum seems much faster than our site, is it because it was de-tunneled ?

Benj

[jimbo]

No Benj, I think there are some unrelated issues with mysql - it's been much slower since the move, my non-mysql pages fly.

DISCLAIMER TO FQ:[nbsp][nbsp]This is not complaining, this is trying to point out a situation

-jim

[Benj]

Jimbo, on which server are hosted your non SQL pages ?

Benj

[jimbo]

My site is hosted on SEVEN - the slowness is in mysql02

-jim

[Justin]

I'm not sure what might cause problems with MySQL02, as its load average rarely peaks at 1.0... The server is less than half-full (does that make me optimistic?), and there shouldn't be any speed issues at all. I will ask TeRRa to look into it however...

------------------
Justin Nelson
FutureQuest (http://www.FutureQuest.net/index.php) Support

[jwbutler]

Man, I hate to report this, but......

I have been getting reports from the users of my discussion board at http://www.starfleetgames.com/discus that they are getting multiple timeout and authentication errors when trying to post messages.[nbsp][nbsp]I've checked the scripts and the board's diagnostics all check out.

This has only started since the move, most recently around 2:30pm EDT today.[nbsp][nbsp]Are the tunnels still up, and could they be part of the problem?[nbsp][nbsp]I'm connecting directly to the 63.xxx.xxx.xxx from my end and don't seem to have this problem.

Thanks in advance,


------------------
Joseph Butler
Starfleetgames.com Webmaster

[jimbo]

Justin, it's real minor.[nbsp][nbsp]If you browse around my forums you'll see what I mean.[nbsp][nbsp]It'll load the header with the links, the ad banner, all the typical UBB/vB 'top of screen' stuff.[nbsp][nbsp]Then it hesitates before loading the table with all of the posts in it.[nbsp][nbsp]Same thing happens when viewing a thread.[nbsp][nbsp]Again, this is only since the move.

-jim

[Benj]

Exact same thing as Jim with my forum.
However I don't think it only affect mysql, but all the site as someone reported befor: good.....lag....lag...good...goood.... it doesn't seems specific to mysql for me.

I thought this was due to tuneling and was going to be fixed ?
However there IS a problem that needs to be killed.

Benj

[Terra]

jwbutler: the tunnels are offline, but not all of the server specific configurations are pulled yet due to the ARP cache issue with the router...

Jimbo: when viewing your forum, IE is reporting problems with the page...[nbsp][nbsp]Look in the lower left hand side for a yellow warning icon...[nbsp][nbsp]MYSQL02 is running quite well with no delays that I can see...[nbsp][nbsp]AAMOF: all MYSQL engines are running at optimum levels...

Benj: we are working as quickly as possible to isolate and resolve the PGPG (pause..go..pause..go) issues...[nbsp][nbsp]We are having a CCIE coming in to dig into the edge router and resolve any issues that it may have...[nbsp][nbsp]The NOC is brand new and we are the only one on the lines...[nbsp][nbsp]It will take a fair bit of work to shake the remaining bugs out, as they did not show up in preliminary testing - but surfaced when we started to heavily and consistently load up the network with real world usage...[nbsp][nbsp]Something very difficult to replicate in a pre-rollout testing environment...

Rest assured, if we cannot get the kinks out of the network (our upstream resolving the issue), then we will have no choice but to seek out alternatives...[nbsp][nbsp]Thusfar, our upstream has been very responsive, and equally diligent in solving the connectivity problems as well...

--
Terra
--I am starting to sound like a broken record--
FutureQuest

[Benj]

FQ, thanks for the answers.
Please keep us up to date here, i'm confident you will find a solution.

Benj

[Terra]

Alas, I think we might have finally nailed the connectivity problem...

Please let us know if things speed up for you and that the PGPG problem starts to fade...

Technically, the full solution will not be completed until all tunnel configs are removed (caution on side of safety) and all servers rebooted with fresh configs and updated Linux kernel (currently being configured and built)...

I believe that with the last set of changes, we may be 90% of the way there...[nbsp][nbsp]:)

--
Terra
--Please say YES!!! -- please?!?--
FutureQuest

[Benj]

I was about to report great speed improvement before even reading your message.

So.... YES, right now my site really work at FULL speed. Its a world of difference. I hope it will stay like this ;)

Benj

[Dan Kaplan]

If current results are any indication, I'd say you definitely nailed it.[nbsp][nbsp]:)[nbsp][nbsp]Things are cooking along!

Thanks, as always.

Dan

[jojo14]

Hi there Future Questers,

Just writing in a report... I noticed my e-mail down this morning about 9am and now at 7pm it's still down.[nbsp][nbsp]

I know you all are working out the kinks but it sure would be nice to know when to expect my e-mail be up and running again.

With my luck if I go and change my e-mail accounts and notifiy everyone in my address book then all the kinks will officially be worked out of the FQ system the moment I do that... meaning I'd be going through a huge inconvience for nothing.[nbsp][nbsp]So I'm gonna sit tight and hopefully you all can simply let me know when the problem will be resolved.[nbsp][nbsp]I know you've all been working hard.

Thanks,

jojo[nbsp][nbsp]

[Terra]

jjo14:[nbsp][nbsp]Open up a DOS box and ping your domain...[nbsp][nbsp]What IP are you showing?[nbsp][nbsp]Ping with both 'www.yourdomain.com (http://www.yourdomain.com)' and without the 'www'...

If it starts with: 208.243.19.???
then read the following thread:
http://www.aota.net/ubb/Forum5/HTML/000902-1.html

Also, which domain are we talking about - as I never assume on this facet...

All email on all servers is working correctly...[nbsp][nbsp]If not, this forum would be stuffed with friendly notices that our mail servers are down... ;)

--
Terra
sysAdmin
FutureQuest

[This message has been edited by ccTech (edited 09-11-00@7:26 pm)]

[wolfstalks]

TeRRa, WOW,[nbsp][nbsp]I have been experiencing the same[nbsp][nbsp]load - pause - load as others have described, but haven't said anything as I have sites across three servers and saw that it was wide spread.[nbsp][nbsp]But now it's Really[nbsp][nbsp]Really Really fast!!!![nbsp][nbsp] :)

Great Job!!!!!

Bob

[sheila]

In addition to what wolfstalks said, I can only add:

HOLY COW!!

yes it is much faster now. :D

[Monty]

Thanks, TeRRa, Deb, Justin and everyone.[nbsp][nbsp]If you hadn't spoiled us so badly to begin with, we never would have known.

Reference, http://2coolfishing.com/cgi-bin/texas/messageboard.cgi?read=27201

Mont ;)[nbsp]

[jimbo]

Terra, the errors are being caused by my advert company - it is intermittant, and not the source of the speed problems I'm talking about, since the banner has already loaded.[nbsp][nbsp]I've shot them a few nasty emails about that already, as you can imagine ;) .[nbsp][nbsp]

But the speed issues are seemingly fixed - browsing around here I've had no real problems, but I'll check again tomorrow from work since I'm on a cheesey dial up here.[nbsp][nbsp]But they seem fast on the dial-up, which is what the majority of my users use, so that is what I care about the most.[nbsp][nbsp]Whatever you did seems to have fixed the slowness issues I was speaking of.[nbsp][nbsp]So everything is groovy as far as I'm concerned :).

-jim

[Dean B]

Well I hate to put a dampner on things but I'm still experiencing the slow slow quick quick slow :([nbsp][nbsp]I check my own email on TAZ then two others on SIX.[nbsp][nbsp]Sometimes I'm waiting anything from 30-60secs to get a response from TAZ's pop server.[nbsp][nbsp]Once it starts to downloads it's fine.[nbsp][nbsp]It then proceeds to check the SIX mail accounts without any hesitation.[nbsp][nbsp]Pages on TAZ will also start to load (inc this forum) then just simply stop, resuming numerous secs later.[nbsp][nbsp]I'm having an awful time trying to use MySQL Admin atm :(

I had put it down to my isp not catching up with the tunneling process but my 3 other isps all behave the same.
The most 'niggling' part of this is it's not consistant.[nbsp][nbsp]Fine one hour, go slow the next :(

Stick with it Terra, I'm sure you'll crack it http://www.dmcity.com/rtwubb/bounce.gif

---Dean---

[jojo14]

I'm sorry but still no e-mail.[nbsp][nbsp]I don't know how to do a ping in dos... sorry... I know it's something simple but something I never bothered myself with before.[nbsp][nbsp]

My site is showing up great, running flash fast and so on.[nbsp][nbsp]But still no e-mail and today's the second day.[nbsp][nbsp]

If you could look into this I'd appreciate it so much.[nbsp][nbsp]My domain is www.virtuosoproductions.com (http://www.virtuosoproductions.com)

Thanks,
jojo

[jojo14]

Actually Justin I'm functioning through a lan and this is the only e-mail address set up to collect my e-mail through the lan.

The 'error' I keep on getting is the password prompt... and I keep entering it and it keeps asking for it over and over again.[nbsp][nbsp]

Thanks for checking into this for me.[nbsp][nbsp]It's really important to me to get my e-mail up and running now more than ever because one of my children is in the hospital.

jojo

[Brian]

How is your LAN setup?

-Brian

[jojo14]

Ok it worked!!![nbsp][nbsp]Yippeeee.[nbsp][nbsp]I followed ccTech's advice and went to the forum posted and then went to my proxy server and took a look at that according to the info I was reading in the other forum ccTech posted.[nbsp][nbsp]

Turns out that my proxy server had a different IP address.[nbsp][nbsp]It started with 208 and when I reset it it changed to 63.[nbsp][nbsp]I really had no idea it could have possibly been anything on my side because I hadn't done anything new or different.[nbsp][nbsp]

I do have to say though that these forums have been a source to lot's , no tons!, of answers for me and I can't tell you how much I appreciate that.[nbsp][nbsp]Even though I went to the wrong forum without realizing it I still found the solution to my problem.[nbsp][nbsp]

Thank you.

Jojo

[Brian]

I am glad you are all back to normal :)

-Brian

[Justin]

Joann - I have just tested your POP mail login, and it appears to be working without problems for me. I see 218 new messages waiting for you.

What is the exact error message you are receiving? If you are simply 'unable to connect', you are still experiencing DNS propagation.

I would highly recommend taking this to email rather than the forums (certainly you have another email address you can use...) so that we can get this issue resolved as soon as possible.

------------------
Justin Nelson
FutureQuest (http://www.FutureQuest.net/index.php) Support