FabSwingers.com forum: Time offline: Full explanation, supporter passes extended

By *j_mark OP Couple
Forum Mod
over a year ago

Totteridge/Whetstone

Hi,

In summary:

We had on-off server problems since Friday which we now believe are fixed. You probably saw "couldn't connect" messages or slow pages during this time. We're really sorry about this

We've added a FULL DAY to all site supporter passes to apologise and make good.

Long story:

We wanted to provide an explanation of the recent glitches on the site.

We're in the process of upgrading all our servers. The main fab webserver is being replaced by new ones which gives us plenty of scope to grow and add cool stuff (and we're continuing to grow fast; the site serves up over 7 million dynamic pages each day, i.e. not including static files and image content).

We did the upgrade last Friday (18th March) after a lot of testing. However, we started to get the odd complaint that some users were seeing occasional glitches. When you run a busy site with over 50,000 people logging in each day it's sometimes hard to understand when there's a real problem because we get a steady stream of reported issues, some of which are down to local connection issues, some aren't. Eventually we started to see some of these issues ourselves.

While we could have rolled back to the old server at any time we took the decision (possibly the wrong one in retrospect) that it would have been difficult to analyse and fix any potential problems on the new server, and our own stats showed that most (99%+) of page requests were being successfully delivered. So we tried to analyse and resolve the issue on the new servers while they were live.

Today we made the decision to flip back to the old server, while we continued to test the problem on the new ones. With the help of our network experts we finally recreated the problems using a series of test, which then allowed us to trace the issue down to a network driver which for some reason was dropping packets on our new servers.

This was fixed by updating the driver software this Wednesday afternoon (23rd March) and we've been re-testing it continuously, and subsequently gone live with it again at approx 19.15 this-evening. We had to flip the site twice today, once back to the old server (lunchtime) and again to the new server (19:15-ish). During these times the site went offline for 5-10 minutes. And while the site was on the old server we experienced the old capacity problems! Aargh!!!!

We're now back up and running on the new servers and are quietly confident the glitches are resolved. We will continue to monitor.

We've improved various procedures about how we comission and test new servers through this.

Sometimes the site is so busy that we're pushing the boundaries of our web servers, software, databases, etc.

That said, we're commited to having the highest standards of site availability and site performance and we'll always make good if we don't achieve that which is why we've added a full day to all site supporter passes.

Admin x

Reply privately

Reply in forum

Reply +quote

By *j_mark OP Couple
Forum Mod
over a year ago

Totteridge/Whetstone

Just to confirm, all site supporter passes are now extended.

Admin x

Reply privately

Reply in forum

Reply +quote

By *oantrimcpl2010Couple
over a year ago

Lisburn

thxs

Reply privately

Reply in forum

Reply +quote

By *uckoldandWifeCouple
over a year ago

Manchester

all that work and the site is still free to the majority of users, just how do you do it, thanks

Reply privately

Reply in forum

Reply +quote

By (user no longer on site)
over a year ago

Admin . . . . . . . . . BREATHE !!!!!

Seems absolutely fine now good job well done. The little I know about running such a huge database the site runs very well indeed.

Cheers

Reply privately

Reply in forum

Reply +quote

By (user no longer on site)
over a year ago

3 cheers for admin

Reply privately

Reply in forum

Reply +quote

By (user no longer on site)
over a year ago

Cool, Can I just add that all users should upgrade there browers to the latest ones too and carry out regular maintenance on there PC's/Laptops.

Servers are apian to run as I have experience from it but also a badly maintained PC can also leed to a poor performing machine!

If anyone needs a quick boring lesson on how to 'clean' a machine and I don't mean with a cloth and feather duster, then I'm more than happy to help!

Reply privately

Reply in forum

Reply +quote

By (user no longer on site)
over a year ago

Cheers admin, as always, much appreciated.

Reply privately

Reply in forum

Reply +quote

By (user no longer on site)
over a year ago

"Cool, Can I just add that all users should upgrade there browers to the latest ones too and carry out regular maintenance on there PC's/Laptops.

Servers are apian to run as I have experience from it but also a badly maintained PC can also leed to a poor performing machine!

If anyone needs a quick boring lesson on how to 'clean' a machine and I don't mean with a cloth and feather duster, then I'm more than happy to help!"

i got a google chrome browser on me desktop but I usually come through aol and it still says i got old browser confused.com i am

Reply privately

Reply in forum

Reply +quote

By (user no longer on site)
over a year ago

Have just upgraded to new firefox dont know why as I am picking up my new laptop tomz morning and this is going away for a jolly good mot and clean.

Reply privately

Reply in forum

Reply +quote

By *_n_JCouple
over a year ago

between Sheffield and Doncaster

Thanks for all your hard work guys this site is cool and so worth every penny.

Rich and Julie x

Reply privately

Reply in forum

Reply +quote

By *lscmnmytngMan
over a year ago

J13 of the M5

I noticed the outage at 19:20.

Right on the button of when you said you were changing over!

Thanks for fixing it and I look forward to some FABulous times in the forthcoming weeks and months.

It truly is a great site.

Reply privately

Reply in forum

Reply +quote

By *arambarMan
over a year ago

swindon

"Thanks for all your hard work guys this site is cool and so worth every penny.

Rich and Julie x"

Agree 100%

Reply privately

Reply in forum

Reply +quote

By *ouple1234Couple
over a year ago

BELFAST UK

50,000 people logging in each day thats some ammount of people and its growing everyday with new people joining the site.

well done admin for the hard work you put in best £5 a month i could spend

Reply privately

Reply in forum

Reply +quote

By *j_mark OP Couple
Forum Mod
over a year ago

Totteridge/Whetstone

Thanks for the kind words folks, makes the effort worthwhile

Reply privately

Reply in forum

Reply +quote

By (user no longer on site)
over a year ago

thanks admin for all efforts put in...enjoy the site and tell all about it..hmm not all thinks of mothers... H.

Reply privately

Reply in forum

Reply +quote

By *unky monkeyMan
over a year ago

in the night garden

Sounds a bit hairy!

At work we recently changed to a new server environment with a similiar sized user base as FAB so I feel your pain!

Well done! Considering what you have gone through, the change over and subsequent changes back and forward were hardly noticeable.

Respek!

Reply privately

Reply in forum

Reply +quote

By *uckscouple2007Couple
over a year ago

Bucks

is the server misbehaving this morning?

have had a few SERVICE UNAVAILABLE (HTTP Error 303) messages

Reply privately

Reply in forum

Reply +quote

By *j_mark OP Couple
Forum Mod
over a year ago

Totteridge/Whetstone

"is the server misbehaving this morning?

have had a few SERVICE UNAVAILABLE (HTTP Error 303) messages

We had to apply an update this-morning (twice); unrelated to the above

Apologies for the inconvenience

Admin x

Reply privately

Reply in forum

Reply +quote

Post new Message to Thread