[FQuest Notice] Network Capacity Increase

[Terra]

When: 02/08/2004
Window: 1:00am - 6:00am EST
Duration: 1 - 30 minutes

We will be increasing capacity from our core Foundry Switch, to the switch that drives the MX servers, plus MQS servers and future MySQL engines...

The first stage will be increasing the uplink capacity to 400Mbps...

The second stage, coming soon, we will once again be increasing the capacity to 4Gbps via fiber uplinks...

We do not foresee any major problems, however when trunking together multiple switches, the spanning tree protocol can sometimes go schizophrenic causing intermittent problems... To prevent this from happening, the whole back segment to the above mentioned servers will be disconnected, reset and reconnected to help minimize any impact...

--
Terra
sysAdmin
FutureQuest

[Terra]

The total time to turn over the switches was ~30 seconds, for which there was a brief interruption in service...

The network maintenance is complete and back up to full speed...

--
Terra
sysAdmin
FutureQuest

[songdog]

We will be increasing capacity from our core Foundry Switch, to the switch that drives the MX servers, plus MQS servers and future MySQL engines...

The first stage will be increasing the uplink capacity to 400Mbps...

The second stage, coming soon, we will once again be increasing the capacity to 4Gbps via fiber uplinks...Can you say a little more about this (in a way that those of us without networking backgrounds will comprehend)?

For example, what was the uplink capacity before the upgrade?

What is "uplink" capacity used for, and was FutureQuest close to maxing it out?

Thanks!

[Terra]

For example, what was the uplink capacity before the upgrade?
200Mbps (inbound + outbound)

What is "uplink" capacity used for
Uplinks, in this context, are switch to switch connections... They are the bottleneck of a switch, since there can be up to 48 devices on the Foundry 4802 switches...

In a nutshell, they are the highway that connects one city to the next city...

was FutureQuest close to maxing it out?


FI4802 <===uplink===> FI4802
|=> TAZ ............. |=> MYSQL01
|=> SIX ............. |=> MYSQL02

If only: TAZ <-> MYSQL01, then the full 200Mbps is available
if TAZ <-> MYSQL01 + SIX <-> MYSQL02, then each server would get half of the uplink capacity - and so on and so on...

We are simply tearing down a 2 lane road, and building a 4 lane one... The next major upgrade, 40 lane freeway, is primarily to solve the problem of handling backups for a rapidly expanding data set...

We are mostly just trying to plan ahead and proactively stay ahead of the curve to ensure that our network does not become saturated now or in the future... The costs of the infrastructure gear to do this has lowered enough that it is now feasible to take this proactive step...

--
Terra
--is moving his chess pieces into position--
FutureQuest

[Randall]

If only: TAZ <-> MYSQL01, then the full 200Mbps is available
if TAZ <-> MYSQL01 + SIX <-> MYSQL02, then each server would get half of the uplink capacity So while the servers on a switch can communicate in ... well, a "switch-like" manner, once you go outside the switch it's more like running through a hub?

This is just me thinking that my weak grasp of the difference between hubs and switches can somehow be applied to uplinks, an area I have no business being in anyway. :D

Randall

[Terra]

once you go outside the switch it's more like running through a hub?
Nope, each port is still like a separate network segment and will not see another ports traffic...

One of the issues I'm trying to solve now is the abundance of ARP broadcasts, which is transmitted to every port... When you have over 8000 IPs in a flat network, the ARP traffic does become a factor as it skews network graphing software, such as MRTG... The other item is scaling up the interswitch uplink capacity, for growth...

You have to consider that a switch is sort of like a router, only the switch works at layer 2 (MAC addresses) instead of at layer 3 (IP addresses) like a router does...

For the below example, we only really care about Layer 2 & 3...

When a packet needs to go from TAZ <-> MYSQL01:
1) Lookup IP address of MYSQL01
2) send packet to network layer for routing decision (layer 3)
2a) if IP local to our segment tell layer 2 as such
2b) if IP not local to our segment tell layer 2 to send to gateway router
3) Layer 2, with passed Ethernet Destination from Layer 3, now needs to find the MAC address for the Ethernet Destination... This is where ARP comes into play... Once it has found out the MAC address for that IP via ARP, it will pass it onward down to the physical layer (Layer 1) for transmission

4) Switch port that TAZ is connected to receives the packet, and its ASIC (packet engine) does a lookup to figure out which other port it needs to send it out... Here is the crux, there are 50 ASICs in the switch (hence why they are so darn expensive), and they all work independently...

5) If it goes through the uplink port, to another switch, then the packet will most likely be pushed into a buffer and sent out on a FIFO basis... With 49 other ASICs feeding this buffer, there is only 1 ASIC(**) that is handling the uplink... If this one ASIC is slowed down by congestion on the uplink port, then the buffer grows larger leading to latency... By increasing the uplink capacity, the ASIC can push the packets out much faster and keep the buffer mostly empty...

**since these are trunked uplink connections, there are now 2 ASICS handling the uplink instead of just 1 ASIC**

6) FI4802 #2 sends to FI4802 #1 via the uplink cable
7) FI4802 #1 receives on trunk #1 (ports 5+6), does lookup on MAC and pushes packet out port #6
8) MYSQL01 network card receives the packet and handles it from there

Bare in mind that each Foundry designed ASIC is designed to handle a little over 2Gbps of packet throughput and work a pure wire speed, with the backplane bus able to handle the full capacity of all ports talking both inbound and outbound at the same time... In short, Foundry switches are intoxicating in regards to how fast they are at pushing packets around - hence their dizzying prices...

It is the same analogy, of you having to drive to work everyday on the same freeway as everyone else... By widening the highway, more traffic can get through (in a perfect world without rubber neckers)...

We have grown to the point now, that I need to start converting our flat network structure, into one that is more segmented... This has an undesired side effect of adding another router hop into our network, for which I am actively fighting to figure out a way to not do that... Site Owners are very sensitive to the number of hops it takes to reach their site, and when we added another hop to our frontside core routers - there was some backlash due to the extra hop and now some people automatically grilled us because it 'looked' slower since there was an extra hop...

In conclusion, all of the internal network reorganizing is to ensure that we do not paint ourselves into a corner with capacity... With some upcoming deployments, such as a major overhaul with our backup infrastructure, as well as future MySQL servers - we need to drill down into the network infrastructure in order to maintain the same QOS level that every site owner expects from us... To accomplish this, we need to shift to a HP ProCurve Gigabit fiber/copper core network, which is an upcoming phase of this project...

--
Terra
--all of our switches require 2nd mortgages--
FutureQuest

[dank]

Yeah, like anyone's gonna be able to respond to that. :P

Dan
-- hey, that's my sand box!

[Randall]

That's what I get for thinking I have any idea what's going on. :o

Interesting though that the router tells the switch whether the destination is local or not. I would've guessed that the switch already knows its little neighborhood and only bothers the router when it needs to send something outside.

But then, I wouldn't be asking Stupid Networking Questions* if I knew stuff like that.

Actually, the scary thing is that I more or less understand everything up until Step 7. But I think I'll shut up now before my head explodes. %)

* Not to be confused with the Stupid Accounting Questions I ask at work.

Randall

[Terra]

Interesting though that the router tells the switch whether the destination is local or not.
Steps 1, 2, and 3 are all local to the host, way before it ever hits the switch...

The host (or server) looks in its route table to see if it is a local or remote address and will package it accordingly...

Local) send out directly on the wire with the MAC address of the destination host.

Remote) send out the packet with the MAC address set to that of the gateway (router) for further packet forwarding

Here are a couple links you may be interested in studying that are not as complicated as those you may find at Cisco and Foundry...
http://computer.howstuffworks.com/lan-switch.htm

Enterprise Class switches are complicated beasts, with enough techno babble that it can even make my head dizzy trying to absorb it all... 8)

The other problem with switches, is that you can quickly shoot yourself in the foot since you are so close to the raw network... They offer a tremendous amount of power, capability, and flexibility however one wrong misstep and they can create rather large instantaneous tears in the fabric of space/time continuum that can be hard to recover quickly from... :eek:

--
Terra
--Some switch features make BGP-4 look like childs play--
FutureQuest

[Mandi]

What is the reasoning behind upgrading the uplink in phases - making sure the existing infastructure is behaving like you expect with the medium-higher volume?

<--- Worries there will be a test someday . . .

[Terra]

Mainly because the HP Gigabit switch is here at home with me, where we are working towards becoming better friends with one another... }{

--
Terra
--We are at the point now that I have to keep reminding it that I'm already married--
FutureQuest

[Randall]

Steps 1, 2, and 3 are all local to the host, way before it ever hits the switch... Now it's starting to make sense. Yep indeed. :noddy:

[This would be that point in the movie where the giant monster grabs our hapless hero and starts eating his brain. But the monster thinks better of it and wanders off to the kitchen looking for Fig Newmans.] The other problem with switches, is that you can quickly shoot yourself in the foot since you are so close to the raw network... They offer a tremendous amount of power, capability, and flexibility however one wrong misstep and they can create rather large instantaneous tears in the fabric of space/time continuum that can be hard to recover quickly from... :eek: [Slowly backing away from the SMC router/switch on the desk.] Nice switchy, niiiiice switchy...

Randall

[MPaul]

So if I've noticed that my website and the FQ website loads faster, is it an illusion, or is everything really faster?

Sorry, I read all the post, but didn't understand anything from Terra's posts. %)

[Randall]

You just need to start hallucinating more -- then it all makes perfect sense. ("No, my eyes are always glazed over like that. Why do you ask?")

Randall

[Terra]

I like to think it is a grand illusion that our competition can't quite figure out how we made the elephant disappear...

:QTwand:

--
Terra
--likes to feed his Enterprise Class switches often--
FutureQuest

[Mandi]

Mainly because the HP Gigabit switch is here at home with me, where we are working towards becoming better friends with one another...
So that's what they sell on those sites which read, "You must be over 18 to read about these products . . ." :P

(Seriously: thanks for the tutorial, verrrry interesting!)

[Wassercrats]

That's what I get for thinking I have any idea what's going on.Once I realized that Terra was using the right parenthesis as a separator, as in "Local) send out directly on the wire with the MAC address of the destination host," I felt I'd learned enough and stopped reading. Don't set your goals too high. It only complicates things.

[frankc]

Originally posted by Mandi:
<--- Worries there will be a test someday . . .Thankfully, the only test here is to pay our FQ bill(s) and refer others to FQ--both of which I happily every time the opportunity presents itself. ;)

[Monty]

our competition but clearly, there is no such thing when it comes to FutureQuest hosting and you really mean "others in the same business" Thanks buying us another house, errr..switch. I know they are in the same price range.

[Chipmunk]

Originally posted by Terra:
Mainly because the HP Gigabit switch is here at home with me, where we are working towards becoming better friends with one another... }{

As I suspected: Terra's real birthday present was a Geek Toy! Deb, you didn't really expect us to :) believe the sock story?

[DogAndPony]

I feel like a kid in the back seat of a car, having no idea where we are, asking, "Are we there yet, Daddy?"

:)

...Bob

Bob West

[Deb]

As I suspected: Terra's real birthday present was a Geek Toy! Deb, you didn't really expect us to believe the sock story? I really did wash them though and around here that's kind of like getting brand new ones :P

And then..... I feel like a kid in the back seat of a car, having no idea where we are, asking, "Are we there yet, Daddy?" Heh, beats sitting in the front passenger seat where I was told (in a most childish whining voice) "Ya know, other network engineers are not asked to put their socks away when they are trying to figure something out at work...." AS IF I was going to let him get away that silly excuse :P

Deb
- Little Johnny's Mom Doesn't Make Him Do Chores...!

[DogAndPony]

LOL!!!

Well, as a curmudgeon batchelor with a messy apartment, I have to say that having someone to remind me about my socks would be a good thing. :)

...Bob

Bob West

[Rue_dAuseil]

Thanks! This is material of the highest (very useful, reassuring -- palliative) interest and of as soothing a presentation clarity as I've been exposed to a l l d a y L O N G.

Communicators I "dig special." Yup, I feel refreshed, well-driven -- the Rudymeister torques up to inter"state" “4-lane-ness”.

--actually, thanks, Terra! – all very nicely done posts; taut, taught, crisp, clean-n-lean --"We move..." :) Have assigned myself study on about half of what you just said, even so, of course, notwithstanding articulation. But what a dull place this would be if nothing to learn and only nonComm.s to learn it from.

-- Sun Tzu, warrior autodidact.
Nah, it’s just Rue d’Auseil, still…