Until a year ago I never really thought much about NIC vendors. I figured if you bought Intel or Broadcom you were safe, maybe if I was doing near Gigabit speeds constantly or iSCSI I would have to pay more attention, but other than that I figured I was good.

Man, was I ever wrong.

Fail #1

This all started almost a year ago in our Oregon data center. In fact it was one the first things that was handed back to me when I started. On our web servers after upgrading to Windows Server 2008 R2 after about a week or two we would lose connectivity on a web server. This was happening to all the web servers, just not at the same time. After losing connectivity when you went to reboot the server it wouldn’t. You had to wait about 10 minutes for the BSOD to come and then the server would reboot (More at Windows Server 2008 R2 network adapter stops working, requires hard reboot).

So we raised a case with Microsoft and after a month of back and forth and some kernel patches we still had the problem. So we tried some Intel NICs and the problem went away.

Fail #2:

Now in our NY data center (Dell hardware instead of IBM, but still Broadcom NICs) I saw some packets being lost and various network timeouts recently. So I updated the firmware on a test server and a couple days later updated the rest of our web servers. I saw no improvements so I started to dig deeper into some tcpdump data with Wireshark and I see the following sorts of ARP requests coming from the servers:

17:03:41.187682 ARP, Request who-has 89.145.83.164 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46 
17:03:41.187684 ARP, Request who-has 74.125.91.109 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46 
17:03:41.187686 ARP, Request who-has 64.34.80.179 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46

ARP is used to find the MAC address of servers within the same network. In fact it is at the foundation of the network stack that much of the Internet is built on. There should never be ARP requests for IPs outside of your network (More at Windows 2008 R2 Servers Sending Arp Requests for IPs outside Subnet). This disappeared when I disabled failover teaming and came back when I enabled it again.

Fail #3:

Now about every week I am getting corrupted arp tables. Deleting the table with arp -d usually results in the table not being rebuilt, or if it does it shortly fails again after. The solution is to reboot the server. So I called Dell and they can’t help me because it is a software issue. So now we are right back where we were with our NICs in Oregon.

Solution:

We are replacing our Broadcom NICs with Intel on our primary production boxes. We replaced one of the NICs with an Intel NIC a couple weeks ago and have not seen either of these problems in that server so we are going to do this with the rest of our servers.

I don’t ever want to touch a Broadcom NIC again. Intel is a company that makes more sense to me anyways as their engineers are more frequently part of what I think of as the fabric of the Internet. Last issue I had with an Intel product I posted on their free mailing list and got a response from an Intel engineer within hours. The best thing to do at this point I think is to take these Broadcoms out to the field:

I will be upgrading the Stack Exchange network’s switching infrastructure on Saturday February, 26th. There should be minimal downtime while I do this, however the sites may be a little slow as I work through the Web Tier. Additionally there will be about a 10-20 minute complete site downtime while I move the DB servers and routers.

What exactly will I be doing? Why, I’ll be removing our current Dell PowerConnect 5448 switches and replacing them with Cisco 2960-S-48TS-L switches.

The plan is to start working on moving the web tier Saturday afternoon and be finished moving all services by early evening. I will post to Blog.Serverfault about one hour before the start of work.

We’ve been trying to figure out what to do about the Disk subsystem IO problems we have. We spent many hours talking to vendors, trusted advisers (ok, reaaaally smart people who were willing to help us – and put up with our questions), and each other trying to find the best solution. We think we have, for now.

Before I get into what we decided to do I want to talk a little about our thought process and the requirements we put together to shape our decision.

Requirements

It should be fast

I know this may seem silly but we pride ourselves on the fact that our sites load not just fast but really fast. We needed a solution that didn’t just get us to the point where our storage subsystem was good, but decimated that barrier and made it great. The added benefit to this approach is we should be able to handle the IO needs of our sites for at least the next year or so. (Assuming our growth rate projections are accurate, of course.)

It should be reasonably priced

The primary requirement of this upgrade was to get more speed out of our storage subsystem, so it makes no sense to spend $100k or more on a SAN with a whole bunch of features that we won’t use in the next 6-12 months. But we don’t want to spend the least amount of money we possibly can on equipment, either. Like Goldilocks, we needed to find a solution that was just right.

It should be reasonably safe

By safety, I mean our data won’t go up in smoke when there is a catastrophic failure — and it needs to keep our services running briskly without a ton of overhead, either mental or physical.  We’re not a bank, and we don’t need to over-build our systems to never lose a single bit and always be 100% accurate. But we also respect that you put a lot of effort into making this place great, and we owe it to you to treat the content you’ve contributed with care.

Putting aside all the cool stuff, all the other feature options, does spending X on storage technology Y give us the best price/performance?

Research

The first thing we did is to explore the different classes of server storage, and look at the pros and cons of each.

1. Direct Attached Storage enclosures

2. SAN technologies

3. PCI bus based flash drives (FusionIO)

4. Traditional Solid State Hard Drive (SSD) storage

Evaluation

We had grandiose plans to bring a few of the finalist options in house to do a barrage of tests, but this turned out to be harder than we thought. Storage vendors were really reluctant to allow us to bring in demo units of their hardware. When I’m looking to spend tens of thousands of dollars, I’m sorry, but I really want to either a) go to your lab and actually run test against a unit that I’m going to be buying, or b) bring a unit in house for a few weeks or a month and be able to verify that the unit will do what you claim it will. What actually happened was that we only brought in one option: the Intel X25-E solid state hard drive. The price wasn’t too expensive to bring them in on a flier, and if they didn’t work out we would be able to re-purpose them somewhere else so it wouldn’t be a total loss.

We decided to use Brent Ozar’s benchmark that he did on the FusionIO drives as our benchmark. Brent isn’t some random blogger, you understand — he is a database ninja. When he does a benchmark, he does a benchmark.

Here’s a quick comparison of our results and his:

Random Reads — 2 threads, 8 outstanding requests, 64k blocks

	FusionIO	Intel X25-E in RAID10
MB/s	1424	1064
IOs/s	22788	17023

Random Writes — 2 threads, 1 outstanding request, 64k blocks

	FusionIO	Intel X25-E in RAID10
MB/s	632	584
IOs/s	10114	9337

The numbers for the X-25′s are actually limited by the controller since the H700 has a max throughput of 600MBps. We do not know how much faster this setup could go if we put a higher throughput controller in the boxes.

We ended up not bringing in any of the SANs due to the amazing performance we got out of the Intel X25 solid state drives. Speed was all we were really after. We don’t seen the need for a SAN at this point so we’ll let that money continue to earn interest in the bank for now.

The full test that we ended up running was straight out of – you guessed it – Brent Ozar’s playbook. That us, using SQLIO to run a whole barrage of tests against the storage system.

The test system consisted of:

• Dell R710 with 96GB of memory
• 2 Xeon X5680 CPUs
• 6 Intel X25-E drives in a RAID 10 array
• H700 RAID controller with 1GB of memory

The gold standard for fast is the FusionIO drives. In our benchmarking, a RAID 10 array of Intel X25 drives got within about 25% of FusionIO performance for reads, and within 10% for writes. That exceeded everyone’s expectations and made our decision very easy. Six X25-E’s are about half the list price of one FusionIO, and putting them in a RAID array eases any concerns about reliability.

For all who are interested the raw data can be found in this archive. The archive also includes a pretty ugly Perl script that converts the raw output from SQLIO into a csv file.

> I would like to take a moment to thank all the people who helped us out looking at all these options. > > – Our vendors who put up with insane questions and flip floppery – Dell, CDW, and Fusion-IO. > – All of the regulars in the Server Fault chat room. > – Of course Brent Ozar who … is … godlike? Hmm, let’s just go with “great guy”.

If you are really observant you may have noticed in my RRD graph post that our write times on our database server are not so hot:

Also looking at this from SQLs perspective confirms this:

Our sequential log writing however is really fast (<10ms), but there is a write performance problem when it comes to our database files that we need to fix.

Too Many Variables

Figuring out what direction to go in with our storage is proving to be quite challenging. The main reason is that there are just so many variables:

• Cost and The Total Cost of Ownership (TCO)
• What will satisfy our needs and prepare us for growth?
• What gives the best performance for our workload and how is that workload going to change?
• Reliability and fail over options
• What direction is storage heading in as far as technology goes
• What fits our scaling pattern

Right now we are thinking about 3 main options:

• A PCI card storage solution like a FusionIO drive
• A SAN
• SSDs in the servers themselves

However, before looking at these options there are some fundamentals and issues to consider when it comes to storage.

The Problem with Figuring Out what is Needed for an IO Workload:

When it comes to analyzing IO workload there are some basic questions that must be answered:

How Much?

There are two main measurements used to answer the question of “how much?” with storage. There is how much data is moved within a certain amount of time (MB/s) and/or how many logical operations there are (IOPS or input/output operations per second). When looking at the workload the amount of operations in the queue is also considered for data that has a steady rate.

How Much of What?

The operations are either going to be reads or writes. The reads or writes will also be either sequential or random. So you end up with some blend of the following four possibilities:

• Sequential Reads
• Sequential Writes
• Random Reads
• Random Writes

What is the Shape Over Time?

Sometimes the shape of IO will be a steady stream of data, but often disk operations will get batched to make the IO more efficient. This means that the shape of the IO will usually be spiky on a micro level (say over several seconds). There will also be a shape on the macro level because most services have peak usage times and there are also scheduled IO intensive operations such as backups.

How Fast?

With whatever workload, operations need to be fulfilled within a certain amount of time. If the disk IO system is busy, then operations have to wait in a queue to be fulfilled.

My main point with all of this is that it is not advisable to just take an average of the amount of IOPS and Megabytes per second over a day and go buy a storage solution. This does not account for how fast these IOPS are satisfied or the shape of data over time. Even if these are taken into account, and taken into account correctly, the workload still should be tested on the actual equipment. The best an analysis can do is give a hypothesis. This leaves two possible courses of action:

1. Take an educated guess and buy something
2. Set up a demo unit or go to a demo location and load test the application

The main problem with option number 2 is that this is a large time investment, and that you have to have the capability to load test your application in a way that accurately reflects real world usage.

My Personal Gripe with Storage Vendors

My gripe is that most large or fast storage solutions can not be purchased at Micro Center. More seriously, even when talking to a sales person on the phone they won’t say what all the options are and what they really cost. Sometimes there is an option to price them out on the website, but the real cost is what the sales person will knock it down to. The sales people want the data I was just talking about (often a limited subset which will only give them a rough ballpark). I have always hated this sales method, I want to see what I get for certain costs — not tell them how much I can spend and have them tell me after that.

The other big thing is that vendors are not public with their actual performance numbers. The SPC1 benchmarks are the best effort I have seen to provide useful information, but the amount of devices in that repository is limited. To a degree this is understandable given all the different workloads as I mentioned, but some basic numbers on various workloads under a Raid 10 configuration would be nice. In other words “Give me some data, please”.

Our Particular Situation and Scaling Model

The sheer size of the stackoverflow.com database compared to our other sites at the moment is a major factor for us. Looking at the above image the amount of IOPS on the stackoverflow.com database is 30 times the amount of IOPS for the superuser.com database. Because of this treating stackoverflow.com’s database as a separate entity for the other databases does make sense.

We also don’t need that much capacity. Going off of Nick Craver’s growth analysis the SO Database will grow from 85 GB to 256 GB over the next 36 months (note: this is just a projection).

I have mentioned this before but our model has been to strike a balance between scaling up and out. We are not particularly attracted to building giant monster systems, nor do we want a bunch of cheap little boxes. We want a balanced amount of medium powered servers. In my mind this fits well with a Microsoft stack.

The Current Options

So taking the above into account here is my current thinking on what the following options might mean for us. We don’t have any demo units in hand yet but our plan is to evaluate FuisionIO as the PCI card option, Equallogic as the SAN option, and Dell approved SSDs put into our current Dell r710 database servers.

Option 1: FusionIO

Pros:

• FusionIO is going to be the fastest option out there. To quote Brent Ozar in his review of FusionIO: “The only way to outperform a Fusion-IO drive is to invest six figures in a SAN and hire a really sharp SAN admin.”
• Simplicity. There is a lot that goes into configuring a SAN correctly, with FusionIO we would copy our database file to the FusionIO drive and be done.

Cons:

• Limited single system availability. There doesn’t seem to be a simple RAID equivalent. For each single server there will likely be only one of these cards in each server. Two can be put in a server and set up to use software RAID but I wonder if that might just end up lowering the availability. In theory since these are solid state devices and not mechanical I would expect to have better reliability than hard drives, but the technology is still fairly new.
• Limited multi-system availability. Any sort of SQL clustering options are out the window and what you have left is log shipping and synchronous or asynchronous mirroring.

I think the FusionIO option fits our scaling model well. We currently have two DB servers — a primary and a fail over. We are planning on expanding to 4 servers so Stack Overflow (and maybe the rest of the original trilogy) can have its own primary and secondary server. There are different options, but one 640GB FusionIO would cover growth for the trilogy and provide the fastest speed compared to a SAN or SSDs. We could then have asynchronous mirroring to the secondary server and in a failover situation Brent Ozar estimated 90 seconds of data loss. Downtime might be around 30 minutes until we get the secondary server up and going manually. We generally favor speed over the highest possible uptime. It is not that we are glib about the uptime of our service, but we don’t have the uptime requirements of a financial institution. For our sites with higher uptime requirements such as Careers we can use the storage in the servers and possibly synchronous mirroring. I also imagine either this or SSDs in the servers themselves will be the cheapest solution — the 640GB drives were quoted at about 10k each for us and the 320GB at about 6.5k from a vendor.

Option 2: A 10GE EqualLogic San with some SSDs

Pros:

• Flexibility in growth and tiered storage. With a SAN we can add shelves as we grow and can tier our storage effectively. So for example we could have an SSD array for the trilogy and a SAS array for our smaller sites. As our sites grow we could move them accordingly. We could also use storage for logs or tempdb.
• Flexibility in availability options. Unlike a fusion drive or SSDs in the server clustering options are now open to us.
• 10GE might be useful for other things if we start to hit network bottlenecks. This is the main reason why 10GE appeals to us more than fiber.

Cons:

• The SAN as a logical unit is a single point of failure unless you buy two. I know these have lots of built-in redundancy but nonetheless our current thinking is that we would want two if we went the SAN route.
• Cost. These things are not cheap. The EqualLogic PS6010S with 8 SSDs is priced at 46k on their site. The redundant 10GE switches if we go with Dell would be about 20k. So without even factoring in other various total cost of ownership factors if we want two SANs we are talking well over 100k. That would be the same cost as getting at least 8 more of our current database servers.

The flexibility and growth options that a SAN offers are appealing. The cost could drastically change if we decided we could live with one SAN, look at different vendors, or give up on the option of having SSDs in the SAN. The performance won’t be as high as the FusionIO would be but for our workload that extra performance might not really matter.

Option 3: SSDs in the Dell Servers

This option would be pretty similar to the FusionIO option except that there will be a trade off of an increase in single server availability options but a decrease in speed. With SSDs we can use a traditional RAID configuration in the servers. These drives on Dell’s site are 4.6k for a 2.5 inch SAS 3 Gbps drive, so for a mirror of two drives it would be $9.2k and would give us 150GB capacity. This wouldn’t leave us much room for growth so we would probably want 300GB capacity. In RAID 10 that will cost about 18.4k. At the moment I don’t have any data on how this would perform but with the current cost of the Dell approved SSDs for our servers this option isn’t too appealing to me yet.

Conclusion

Right now we are still in the preliminary stages. Initially I am fond of the FusionIO option where as George is more in favor of a SAN. One of the main reasons I favor FusionIO at the moment is that it would satisfy our growth in the short term, interestingly enough this is one of the reasons George is less fond of it. George’s main reason for a SAN is that we get greater flexibility with the features a SAN offers such as the ability to fully cluster, use snapshotting, full LUN replication, and dynamic expansion (on some models). By going the SAN route earlier and not later we don’t put off solving the problem until about a year from now when it might be harder to change our infrastructure. If we were to get a SAN now we will learn how to use the advantages given to us by the SAN’s flexibility. I generally agree with this philosophy but not in this particular case. The reason I favor putting off a SAN is that I feel in a year or so the SAN options that include SSDs might be a lot cheaper and more attractive. Also the FusionIO option fits well with our current scaling model. Although the growth of our Stack Exchange sites looks very promising to me, I feel it is too early to predict how they will grow. This is not so much in terms of visitors but more in terms of the IO workload growth. Our developers could make changes that greatly effect our IO workload. So we might have a much better understanding of what we need 6 months to a year from now than we do currently.

What we really need is to get more data on the performance of these various options and get our hands on some demo units. I feel like all 3 of these paths are valid options. Also there are valid options we haven’t looked as closely at yet (for example SANs that don’t support SSDs). At this point it is clear that choosing a storage route to take is no small task.

Our main backup system for files is Bacula running on Ubuntu server. I was able to get it up and going quickly but it has taken a while to get it running fair smoothly and doing what I want. My main impression of Bacula is that it favors flexibility over ease of use. So I wanted to share some of my discoveries in this process that might help others setting up and working with Bacula for the first time.

Our Current Setup and keeping it simple

Hardware:

• Tandberg 8 Tape StorageLoader that includes one LTO-4 drive
• Dell R610 with 209 GB of storage

With these resources George and I had a couple discussions about how the backups would work. We had a long conversation about how were were going to implement Grandfather-Father-Son (GFS) with Bacula:

Grandfather-father-son backup refers to the most common rotation scheme for rotating backup media. Originally designed for tape backup, it works well for any hierarchical backup strategy. The basic method is to define three sets of backups, such as daily, weekly and monthly. The daily, or son, backups are rotated on a daily basis with one graduating to father status each week. The weekly or father backups are rotated on a weekly basis with one graduating to grandfather status each month. Often one or more of the graduated backups is removed from the site for safekeeping and disaster recovery purposes.

Once we started to get pretty deep into how this would work with the recycling of tapes we realized that we just don’t need to be doing this. Instead we are going to have full backups and daily differential backups on the disk (possibly overflowing into a tape pool) and then just copy the most recent full backups to a tape once a week and take that tape off site once a week. This is right, for us, for the time being because:

• To keep a year of offsite tapes with a one year retention isn’t that expensive
• There is less room for error or complications with reusing the tapes

The lesson for us was to KISS (Keep it Simple Stupid) and to not over engineer our backups.

Don’t Make File Storage Volumes too Big

File storage is represented in Bacula as if it were tape backup solution. Your filesystem is basically the loader and each file on the filesystem that holds backed up data is like a tape. You can make these volumes any size you want really, if there isn’t room for a backup it goes onto the next one. However, do not make them too big. When you start out making your backup selections (what you want to backup) you might grab things that are big and that you don’t want. The problem is that if these are in the middle of a volume from what I can tell there is no easy way to just erase that job or file in the middle of the volume, so you will have to wait for the volume to become available for reuse (recycling) and it will sit there taking up space. If they are small you can manually recycle the volumes with the job that grabbed something large that you didn’t want to back up.

Know the Retention Periods

Bacula has three retention Periods:

• File
• Job
• Volume

When the file retention period has passed the records of files will be purged from the catalog and you will only be able to restore whole jobs. When the job retention has passed you will only be able to restore backups by scanning the whole tape. When volume retention passed and you are recycling volumes the backup might be overwritten.

Copy Jobs

The copy job is how I replicate the full backup to the offsite tape. This took me a little while to figure out as the documentation focuses more on the migration jobs. The main two points for me were that there has to be a bunch of Null configuration items so the job passes the configuration check and that you specify the destination pool of the copy job in the pool configuration of the source pool.

Here is a configuration for copy jobs that will copy the full backups from the past 2 days to my tape pool:

Job {
        Name = "OffsiteCopyFull"
        Type = Copy
        Pool = WeeklyFile
        Schedule = "OffsiteFull"
        Client = None
        Fileset = None
        Selection Type = SQLQuery
        Selection Pattern = "select * from Job where (EndTime BETWEEN NOW() - INTERVAL 2 DAY AND NOW()) AND Level = 'F' AND (JobStatus = 'e' OR JobStatus = 'T') AND PoolID = 8;"
        Maximum Concurrent Jobs = 1
        Allow Duplicate Jobs = Yes
        Messages = Standard
        Storage = File
        Write Bootstrap = "/var/lib/bacula/offsite-copy-%c.bsr"
}


Fake fileset for copy jobs

Fileset {
  Name = None
  Include {
    Options {
      signature = MD5
    }
  }
}

Fake client for copy jobs

Client {
  Name = None
  Address = localhost
  Password = "NoNe"
  Catalog = MyCatalog
}

Pool {
        Name = WeeklyFile
        Pool Type = Backup
        Recycle = yes                       # Bacula can automatically recycle Volumes
        AutoPrune = yes                     # Prune expired volumes
        Volume Retention = 14 days         # one year
        Maximum Volume Bytes = 10G          # Limit Volume size to something reasonable
        Maximum Volumes = 15               # Limit number of Volumes in Pool
        Next Pool = WeeklyOffsiteTapes          #This is for the copy job
}

Configuration Layout

Instead of one massive file you can have includes in the main bacula director configuration:

@/etc/bacula/backupjobs.conf
@/etc/bacula/copyjobs.conf
@/etc/bacula/schedules.conf
...

I find this keeps things a little bit neater in general.

Use JobDefs

Jobdefs allow you to define a template (directives that will be defaults to all jobs that inherit that job) for jobs. This makes for a cleaner configuration in the same way using inheritance in Nagios does.

Use Different Pools for Full and Differential Backups

I think using different file pools for full and differential (or incremental) backups makes sense for at least two reasons:

• If you need space you might want to blow away some of the older full backups when you have more recent ones. So for the same reason as using small volumes having different pools can make volumes easier to recycle.
• It makes sizing a little bit clearer. All you need is the du command to see how much your full vs differential backups are taking.

To specify pools for different types of backups is simple, just put something like the following in your JobDefs or Job configuration section:

 Pool = WeeklyFile
 Full Backup Pool = WeeklyFile
 Differential Backup Pool = DailyFile

Take Advantage of the SQL Backend

Since the catalog is built on SQL you can create your own queries. I chose to use MySQL and can interact with catalog using the console. Also you can run predefined queries by adding them to the /etc/bacula/scripts/query.sql file and then using the query command from bconsole.

As an example I added the following to my query.sql to the average size of full backups per client:

:Average Size of Full Backups per Client
SELECT Client.Name, AVG(Job.JobBytes) as Avg, STDDEV(Job.JobBytes) as StandardDev
 FROM Job INNER JOIN Client ON Job.ClientId=Client.ClientId
 WHERE Level = 'F'
 GROUP BY Client.Name
 ORDER BY Avg DESC;

You could also add something like

AND (EndTime BETWEEN NOW() - INTERVAL 1 WEEK AND NOW())

to the WHERE clause if you want to limit it to say the past week.

Final Thoughts:

As always with backups:

• Run test restores. Time is always limited but you need to run at least a few to sleep well at night.
• Check the backup logs regularly.

A while ago we decided that running our own DNS system was the best approach to take. We did this for many of the same reasons that our development team decided to move off of Window Azure.  One of our big gripes with our DNS hosting provider had been ease of management, simply put – a web interface is just not that good at managing a large number of DNS entries. They did have an API that allowed us to script some things, but it really wasn’t as robust as we needed it to be. It was a time consuming, repetitive process that didn’t need to be as bad as it was. So, today I present to you our three (semi)new DNS Servers!

• ns1.serverfault.com
• ns2.serverfault.com
• ns3.serverfault.com

We have laid out our DNS servers in both of our data centers - Oregon and New York. We currently have two servers serving out of the New York data center, and have re-purposed one of the Web Tier servers in Oregon to act as our third DNS server. This layout – while not perfect (we could have a fourth server in Oregon to help us if NY goes down) – splits our servers geographically and logically.

NS1 and NS2.serverfault.com

•  Dual Intel Xeon E5640 processors
• 16 GB RAM

NS3.serverfault.com

• Intel X3360 processor
• 8 GB RAM

All three of our servers are running CentOS 5.5 and Bind 9.3.  When I did load testing before putting these servers into production I was able to get a max of about 2000 Request per second from the NY servers, and about 200 Requests per second out of the Oregon server. This is well within our growth zone, and when we do need to upgrade, it will be the Oregon server first giving us plenty of headroom to grow.

As you can see below we get just over 40 rps during peak times – well this is the holiday season, so it’s  a little low. Our normal peak is about 50-80 rps

The change went pretty well on our end, one minor hick-up with me putting a bad wild-card in place, and then a strange problem where a month after the transition we where still seeing 2-3rps on our Dynect reports. Even though we were leaving them Dynect was responsive, and they tracked the issue down to OpenDNS caching things for way too long. Once that was cleared up we were able to turn off Dynect.

We see this as one more step on the way to reaching or goal of making the servers that the Stack Exchange network runs on the best and most responsive infrastructure that is can be.