#1 Posted by mrpibb (502 posts) - - Show Bio

Hi all

We're having mysql slave issues. This will cause content to not show up (but it is getting saved). I'll update as I find out more from the ops team, but please don't file bugs on this yet.

thanks

Danny

#2 Posted by mrpibb (502 posts) - - Show Bio

We are now resolved. Sorry for the issue.

#3 Edited by BumpyBoo (10105 posts) - - Show Bio

@mrpibb:

WOOHOO!

Thank you very, very much :) This was driving a lot of us crazy! You da man *wink and finger gun* :)

Moderator Online
#4 Posted by mrpibb (502 posts) - - Show Bio

@bumpyboo: not a problem, sorry for the delay. Wasn't home when it first happened.

#5 Posted by pikahyper (12494 posts) - - Show Bio

@mrpibb: it's like way worse in the wiki now, 95% of edits error out and don't save, most of the time you can't even get into edit mode and there are constant 404 errors.

Moderator Online
#6 Posted by BumpyBoo (10105 posts) - - Show Bio

@pikahyper:@mrpibb:

Yes, the PM system is way worse too, messages to and from not even going through anymore, even if the notification says otherwise O_o

Moderator Online
#7 Posted by wmwadeii (3250 posts) - - Show Bio

@mrpibb: @pikahyper: Any update, I updated a blog entry and the changes did't stick.

#8 Posted by mrpibb (502 posts) - - Show Bio

yeah, sorry guys. Looks like the replication had problems again. We're taking those machines out of rotation. Not sure why the alerts didn't contact the on call person. 30 min to fix.

#9 Posted by BumpyBoo (10105 posts) - - Show Bio

Is it just me, or is it happening again? O_o

Moderator Online
#10 Posted by AweSam (7375 posts) - - Show Bio
@bumpyboo said:

Is it just me, or is it happening again? O_o

It's happening again.

#11 Edited by BumpyBoo (10105 posts) - - Show Bio
Moderator Online
#12 Edited by wildvine (10329 posts) - - Show Bio

@mrpibb Hey, The PM box is bugging out. Can't send, or receive. Can't make new blogs. And can't start new topics.

Moderator
#13 Posted by wildvine (10329 posts) - - Show Bio

@awesam said:
@bumpyboo said:

Is it just me, or is it happening again? O_o

It's happening again.

Yeeeeeup.

Moderator
#14 Edited by BumpyBoo (10105 posts) - - Show Bio

@wildvine: *grabs and won't let go* I don't like it!!!

Moderator Online
#15 Posted by AweSam (7375 posts) - - Show Bio

@bumpyboo: Seriously, it's even worse now. Threads keep vanishing, then reappearing. It's the Vinepocalypse!

#16 Edited by wildvine (10329 posts) - - Show Bio

@bumpyboo said:

@wildvine: *grabs and won't let go* I don't like it!!!

*Strokes your hair* There there. Wv is here.

Moderator
#17 Posted by BumpyBoo (10105 posts) - - Show Bio
Moderator Online
#18 Posted by wildvine (10329 posts) - - Show Bio

@bumpyboo

We survived the new switch over. We have survived three spam bot invasions and one spam-pocalypse since I have been here. We survived 2012! We shall persevere!!!

Moderator
#19 Posted by AweSam (7375 posts) - - Show Bio
#20 Edited by wildvine (10329 posts) - - Show Bio
Moderator
#21 Edited by BumpyBoo (10105 posts) - - Show Bio
Moderator Online
#22 Posted by Pyrogram (41141 posts) - - Show Bio
Online
#23 Edited by BumpyBoo (10105 posts) - - Show Bio

@pyrogram:

It's a nice gesture but I CAN'T SEE MY NOTIFICATIONS OR PM'S!!! *runs round in circles* It's the END OF THE WORLD!!!

Moderator Online
#25 Posted by pikahyper (12494 posts) - - Show Bio

please keep it relevant people

Moderator Online
#26 Posted by Pyrogram (41141 posts) - - Show Bio
Online
#27 Edited by pikahyper (12494 posts) - - Show Bio

@pyrogram: it looks like the problem may be fixed.

Moderator Online
#28 Edited by mrpibb (502 posts) - - Show Bio

yeah sorry guys. DB replication problems. Fortunately we found the root of the issue so they should not crop back up (short of our machines burning down).

#29 Edited by pikahyper (12494 posts) - - Show Bio
Moderator Online
#30 Edited by Ellie_Knightfall (4819 posts) - - Show Bio

@mrpibb: This is happening again.

#31 Posted by pikahyper (12494 posts) - - Show Bio

@mrpibb: yep you jinxed it, submissions keep appearing and disappearing as well in the queue

Moderator Online
#32 Posted by AweSam (7375 posts) - - Show Bio

@pikahyper: I only noticed a minor problem. Not as bad.

#33 Posted by Eagle_of_the_East (867 posts) - - Show Bio

Happening again! Chrome, Windows 7

I guess the machines burned down :-P

#34 Edited by pikahyper (12494 posts) - - Show Bio

@awesam: oh it is way worse this time

Moderator Online
#35 Posted by AweSam (7375 posts) - - Show Bio

@pikahyper: No. Earlier, I could hardly check my messages, or open thread.

#36 Posted by Ellie_Knightfall (4819 posts) - - Show Bio

@pikahyper: How's it differentiate from last time?

#37 Posted by pikahyper (12494 posts) - - Show Bio

@ellie_knightfall: before stuff (wiki, forum, pm's) would still go through, it just wouldn't always show up right away or it would show up off and on, now I can't even get things through, error messages all over, submissions that were approved won't disappear, other submissions appearing and disappearing, new and old page history disappearing, edits that can be seen when editing a page but not when viewing, 404 errors all over, deleted things not deleting, etc.

Moderator Online
#38 Posted by AweSam (7375 posts) - - Show Bio

Yeah, it's pretty bad now.

#39 Edited by pikahyper (12494 posts) - - Show Bio

and it is getting worse

Moderator Online
#40 Posted by pikahyper (12494 posts) - - Show Bio

pibbs online he will save us !!!

Moderator Online
#41 Posted by mrpibb (502 posts) - - Show Bio

I jinxed it sorry guys. Should be good.

#42 Edited by mrpibb (502 posts) - - Show Bio

just to provide some background, we've been meaning to do about 3 hours of downtime to rebuild the dbs but it's been pushed off due to proper timing. I think we'll have to do this soon but you'll have at least 48 hours notice before it occurs (the site will go readonly, probably around 10pm PST some night).

#43 Posted by pikahyper (12494 posts) - - Show Bio

@mrpibb: yay !!! just in time, I've gotta get to new releases :D

I'd recommend doing it on a Sunday night, that seems to be the slowest night for wiki activity

Moderator Online
#44 Posted by akbogert (3224 posts) - - Show Bio

So is this actually fixed for now?

#45 Edited by pikahyper (12494 posts) - - Show Bio

@akbogert: everything has worked fine for me all night.

Moderator Online
#46 Edited by akbogert (3224 posts) - - Show Bio
#47 Edited by Arkhamc1tizen (2145 posts) - - Show Bio

There are so many 404 errors sometimes when I just get onto the homepage.