Issue summary as of August 16, 2012

[proteus]

So, as best I can tell (with the help of my hosting provider), here's what's been happening:

• Sometime a couple of weeks ago, there was at least an attempt of a break-in style attack on the KVM connected to my server (for those who don't know, basically a remote access server that connects to the video card in my server so you can remotely "sit at the console"). The attack does not appear to have been successful, but...
• Around the same time, two servers -- including ours -- started having intermittent network problems. These problems were not detected by monitoring equipment, because it seems that the problem was a corrupt network driver
• The network issues were the root cause of our database errors, as they were causing large queries to be interrupted when network connectivity with the database dropped. Oddly, socket-based connections were also affected
• The issues with the database eventually let to data corruption, requiring a restore from backup and a server rebuild. We lost anything between about 7am and 2pm GMT on Wednesday, as well as the table that logs who read what (annoying, but not fatal).
• Our server itself appears to be stable, however, there is some additional network issue on the provider's side that makes the site disappear for anywhere between 30 seconds and 2 hours at a pop. The provider is working on tracking this down, and we're helping.

In short: we lost a little data, but our backup strategy worked, so it wasn't much. And we're back online from a server point of view, but the site may go up and down as the result of stuff outside my control. There's some indication that this may be an attack, but no concrete evidence of that.

Since I had to rebuild, I took the opportunity to upgrade the forum and some components; so if there is any strange behavior, it's probably related to that -- feel free to mention it here.

EDIT: the "unable to connect" messages you're seeing are generated by your browser and are the symptom of the network issues our provider is having

[Will Howard]

I'm still getting "Unable to Connect" messages for a LOT of things...

[proteus]

"Unable to connect" or "unable to connect to database"? If the former, it may have to do with the network issues.

Can you post a screenshot?

[Siliconous Skumins]

Yeah, I'm getting the "unable to connect to host" messages a lot too. Seems to be for about 10 seconds at a time, on average I get them about once an hour or so.


Have you noticed any increase in site traffic / Bad Behavior blocking since the time of the apparent attack on the remote access software?  The ongoing "network issues" seem a little odd for a hosting company, I would have thought a faulty router / switch or cable would have shown up quite quickly... 

Possibly a faulty NIC throwing spurious data / collisions across the network, I suppose.

SS

[von Corax]

Both last night and tonight I've been getting a disturbing number of 404s which disappear as soon as I reload. I don't see any pattern to them.

EDIT: I take it back; they're not 404s. They're "Could not connect to remote server" errors, but they have all lasted a scant handful of seconds. I'm surfing other sites concurrently, and having no such problems with them.

[Captain Shipton Bellinger]

I am also getting very frequent 'Unable to connect to brassgoggles.co.uk' messages. Definitely browser-generated, rather than server-generated HTTP errors. I guess our host is still having considerable network problems.


Also, I note that after the rebuild ranks appear to have reverted to being post-count based rather than on-line time based.

[proteus]

The ongoing "network issues" seem a little odd for a hosting company, I would have thought a faulty router / switch or cable would have shown up quite quickly... 

We're pretty sure it's not a hardware problem. A co-location company has a vast number of subnets; it's likely that someone has a misbehaving configuration somewhere.

[Siliconous Skumins]

The ongoing "network issues" seem a little odd for a hosting company, I would have thought a faulty router / switch or cable would have shown up quite quickly... 

We're pretty sure it's not a hardware problem. A co-location company has a vast number of subnets; it's likely that someone has a misbehaving configuration somewhere.

Yeah, that sounds a good possibility...  That or if it *was* a hack attempt on the (our box) remote server, maybe some other box got rooted and is now causing the network issues (spam bot?). 


I should add we *do* all appreciate your hard work... 

SS

[proteus]

We *think* we've narrowed it down to a packet filter problem; more-detailed logs have been turned on and we're going to try a few things to confirm this.

[Ada Thorold]

Greetings all,

A few of us have been having problems registering during the last few days, presumably related to the ongoing troubles. Although my account spontaneously activated this afternoon others are struggling.

We have been congregating over at Spare Goggles:
http://sparegoggles.forumotion.net/t631-registration-problems#7226

Many thanks,

~A~

[proteus]

I have had a handful of complaints about emails not being sent; it looks like some mail was being held in queue. I don't know why. However, I've got a clank running about and flushing the queue (which sends mails and logs errors if any); emails should be going out now, but there may sometimes be a several minute delay.