Brief Outage

[CoorsLightGuy]

I apologize for the brief outage this evening. There was some sort of "connectivity issue" at the Data Center. I think we were only offline for 15-30 minutes, but I'm not 100% sure. In any case, as you can see, everything is working now.

[fordruid]

Storm interference maybe?

[Herr Mario]

.........was a lot longer than that-durn about 1 hour=-...:BangHead: ....but i'll forgive ya...:) :grinning-thumbsup: ;)

[CoorsLightGuy]

The data center where my servers are located have nearly 200,000 domains being hosted in the facilty. My Windows servers weren't affected, only the Linux ones. I could still ping my servers, but the domain wouldn't resolve, so it had to be some sort of DNS infrastructure issue.

I'm guessing that hundreds, perhaps thousands of domains were affected, but don't know for sure. I do know that it wasn't only my servers that were affected.

[CoorsLightGuy]

.........was a lot longer than that-durn about 1 hour=-...:BangHead: ....but i'll forgive ya...:) :grinning-thumbsup: ;)

Thanks. I try to take my job seriously and do my best to make sure that all the servers and domains that I'm hosting stay online and if there is a problem to get it resolved a.s.a.p.

[CoorsLightGuy]

There was another outage today, not sure how long it affected our site as I was out in the field. I got this info from the guys at the Data Center. My guess that it was this same issue that caused the outage the day before.

"We apologize for the inconvenience of your services being offline. At approximately 12 PM Noon EST, the Data Center experienced load issues with respect to the network segment where your server resides. The switches on that segment were rebooted and the services restored. We are investigating the root cause of the network segment failure and have ruled out hardware.

We are currently working with CISCO and one of our upstream providers to determine the root cause and will keep you updated. Currently, we have added additional switches to your network segment for load balancing and failover. If your services are not restored by (2:45 PM EST) please contact us immediately. Again, thank you for your patience."

I know that all my Linux servers, including this one was up and running at 2:00pm, but all the Windows servers were still not responding at that time. They're working now thank goodness, but not sure exactly when they got back online.

The Data Center has a direct link to the backbone, so if it wasn't a harware issue inside, then it had to be something upstream having to do with access to the backbone. That being the case, it was a big outage for at least 30 minutes to an hour, affectiing a LOT of websites.

In any case, this would have set-off a "red alert" and scared the crap out of everybody there so you know they are taking it seriously and trying to find out exactly what caused the problem, etc.

If I find out anymore, I'll let you guys know.

[Herr Mario]

.....dang it....I'm afraid to try and get on now....I czeched in at around noon (lunchtime) from work,and yes-I got on for about a minute-read a post,clicked back-and got the error message...tried again and got SOS....and figgered it way screwy again...:xxrotflmao: ....looked back in about 10 minutes later and all was fine...:D

[WmRoy]

So now we know that it's all Mario's fault..........:duel: :duel: :duel: :xxrotflmao:

[Herr Mario]

....nope....its Bush's fault:rolleyes: .........EVERYTHING is Bush's fault-remember...???:xxrotflmao:

[WmRoy]

....nope....its Bush's fault:rolleyes: .........EVERYTHING is Bush's fault-remember...???:xxrotflmao:

Sorry......... I won't forget again............
:icon_redface: :icon_redface: :icon_tongue: :icon_salut: :xxrotflmao:

[CoorsLightGuy]

I'll be sure to notify the Data Center and give them Mario and Bush's IP address and be on the lookout for both of them the next time there is a problem.

[WmRoy]

I'll be sure to notify the Data Center and give them Mario and Bush's IP address and be on the lookout for both of them the next time there is a problem.

Ahhhhh......... at last we get to the ROOT of the PROBLEM!!
:devil-smiley-1: :D :devil-smiley-1: :D :devil-smiley-1: :D :devil-smiley-1: :D :devil-smiley-1:


:xxrotflmao:

[CoorsLightGuy]

I got an update from the Data Center a few minutes ago. I noticed that all my Linux servers were running slower than normal today. I couldn't tell any difference with any of the Windows servers.

"We are currently experiencing a network traffic issues coming in from one of our backbone circuits. We will have this wrapped up as soon as possible, and we apologize for the inconvienence. We thank you for being patient with us in this matter."

It sounds like they have hopefully isolated whatever the problem is and are about to get it finally fixed. I've been in the business since 1985 and sometimes this crap can drive you insane, trying to find the one variable, whether it be hardware, software, whatever that's causing a sporadic problem.

[CoorsLightGuy]

Things seem much better today, so hopefully they really did get to the bottom of things. The speed seems to be back to normal.

[CoorsLightGuy]

I had to work out in the field tonight and when I got home Roy had emailed me letting me know the site was down. I checked with the data center and they had to pull it down for preventive maintenance. As you can see it's up and running again.

I don't know if it seems faster to you guys, but it sure is for me tonight. Earlier today, before I left to go back out tonight, I noticed that the SQL server log was over 800MB in size, which is pretty huge and will cause the server to run slow as that thing has to be kept in memory and record all SQL transaction.

I renamed the log, forcing it to create a new one and after doing a fast review, I deleted it to free up space. The main point is the log that's being loaded into memory now is tiny compared to what it was before, so I'm hoping that the performance increase isn't just an illusion.

[CoorsLightGuy]

For some reason the MySQL server had "puked" and the board was down for about 20 mins. All the other servers were online and working just fine, so all I had to do was restart the MySQL server and everything seems fine now.

I'll check the logs later today or tonight to try and see if I can spot the problem, but I'm in a rush right now and about to go back out the door on an appointment.

[Herr Mario]

.....it did it again today around noon EST....I czeched in from work unt it vas kaput...

[CoorsLightGuy]

.....it did it again today around noon EST....I czeched in from work unt it vas kaput...

No.....that was the same time I was talking about (see the timestamp on my post above). It was sometime in between 12:00 and 12:30. I discovered the problem around 12:30. After I restarted the MySQL server, it's been up and running with no outage that I know of.

I'm going to take a look at the server logs now and see if I can spot any clues as to what happened.

[Herr Mario]

...old thread...didnt look at the time....just tryin' to help with feedback...;)

[CoorsLightGuy]

...old thread...didnt look at the time....just tryin' to help with feedback...;)

No problem.......I appreciate the help. I got sidetracked earlier, gonna look at the logs now.