#1 Posted by mrpibb (496 posts) - - Show Bio

Hi all

We're having mysql slave issues. This will cause content to not show up (but it is getting saved). I'll update as I find out more from the ops team, but please don't file bugs on this yet.

thanks

Danny

#2 Posted by mrpibb (496 posts) - - Show Bio

We are now resolved. Sorry for the issue.

#3 Edited by BumpyBoo (8741 posts) - - Show Bio

@mrpibb:

WOOHOO!

Thank you very, very much :) This was driving a lot of us crazy! You da man *wink and finger gun* :)

Moderator
#4 Posted by mrpibb (496 posts) - - Show Bio

@bumpyboo: not a problem, sorry for the delay. Wasn't home when it first happened.

#5 Posted by pikahyper (11952 posts) - - Show Bio

@mrpibb: it's like way worse in the wiki now, 95% of edits error out and don't save, most of the time you can't even get into edit mode and there are constant 404 errors.

Moderator
#6 Posted by BumpyBoo (8741 posts) - - Show Bio

@pikahyper:@mrpibb:

Yes, the PM system is way worse too, messages to and from not even going through anymore, even if the notification says otherwise O_o

Moderator
#7 Posted by wmwadeii (3249 posts) - - Show Bio

@mrpibb: @pikahyper: Any update, I updated a blog entry and the changes did't stick.

#8 Posted by mrpibb (496 posts) - - Show Bio

yeah, sorry guys. Looks like the replication had problems again. We're taking those machines out of rotation. Not sure why the alerts didn't contact the on call person. 30 min to fix.

#9 Posted by BumpyBoo (8741 posts) - - Show Bio

Is it just me, or is it happening again? O_o

Moderator
#10 Posted by AweSam (7373 posts) - - Show Bio
@bumpyboo said:

Is it just me, or is it happening again? O_o

It's happening again.

#11 Edited by BumpyBoo (8741 posts) - - Show Bio
Moderator
#12 Edited by wildvine (9112 posts) - - Show Bio

@mrpibb Hey, The PM box is bugging out. Can't send, or receive. Can't make new blogs. And can't start new topics.

Moderator
#13 Posted by wildvine (9112 posts) - - Show Bio

@awesam said:
@bumpyboo said:

Is it just me, or is it happening again? O_o

It's happening again.

Yeeeeeup.

Moderator
#14 Edited by BumpyBoo (8741 posts) - - Show Bio

@wildvine: *grabs and won't let go* I don't like it!!!

Moderator
#15 Posted by AweSam (7373 posts) - - Show Bio

@bumpyboo: Seriously, it's even worse now. Threads keep vanishing, then reappearing. It's the Vinepocalypse!

#16 Edited by wildvine (9112 posts) - - Show Bio

@bumpyboo said:

@wildvine: *grabs and won't let go* I don't like it!!!

*Strokes your hair* There there. Wv is here.

Moderator
#17 Posted by BumpyBoo (8741 posts) - - Show Bio
Moderator
#18 Posted by wildvine (9112 posts) - - Show Bio

@bumpyboo

We survived the new switch over. We have survived three spam bot invasions and one spam-pocalypse since I have been here. We survived 2012! We shall persevere!!!

Moderator
#19 Posted by AweSam (7373 posts) - - Show Bio
#20 Edited by wildvine (9112 posts) - - Show Bio
Moderator
#21 Edited by BumpyBoo (8741 posts) - - Show Bio
Moderator
#22 Posted by Pyrogram (36709 posts) - - Show Bio
#23 Edited by BumpyBoo (8741 posts) - - Show Bio

@pyrogram:

It's a nice gesture but I CAN'T SEE MY NOTIFICATIONS OR PM'S!!! *runs round in circles* It's the END OF THE WORLD!!!

Moderator
#25 Posted by pikahyper (11952 posts) - - Show Bio

please keep it relevant people

Moderator
#26 Posted by Pyrogram (36709 posts) - - Show Bio
#27 Edited by pikahyper (11952 posts) - - Show Bio

@pyrogram: it looks like the problem may be fixed.

Moderator
#28 Edited by mrpibb (496 posts) - - Show Bio

yeah sorry guys. DB replication problems. Fortunately we found the root of the issue so they should not crop back up (short of our machines burning down).

#29 Edited by pikahyper (11952 posts) - - Show Bio
Moderator
#30 Edited by Ellie_Knightfall (4818 posts) - - Show Bio

@mrpibb: This is happening again.

#31 Posted by pikahyper (11952 posts) - - Show Bio

@mrpibb: yep you jinxed it, submissions keep appearing and disappearing as well in the queue

Moderator
#32 Posted by AweSam (7373 posts) - - Show Bio

@pikahyper: I only noticed a minor problem. Not as bad.

#33 Posted by Eagle_of_the_East (867 posts) - - Show Bio

Happening again! Chrome, Windows 7

I guess the machines burned down :-P

#34 Edited by pikahyper (11952 posts) - - Show Bio

@awesam: oh it is way worse this time

Moderator
#35 Posted by AweSam (7373 posts) - - Show Bio

@pikahyper: No. Earlier, I could hardly check my messages, or open thread.

#36 Posted by Ellie_Knightfall (4818 posts) - - Show Bio

@pikahyper: How's it differentiate from last time?

#37 Posted by pikahyper (11952 posts) - - Show Bio

@ellie_knightfall: before stuff (wiki, forum, pm's) would still go through, it just wouldn't always show up right away or it would show up off and on, now I can't even get things through, error messages all over, submissions that were approved won't disappear, other submissions appearing and disappearing, new and old page history disappearing, edits that can be seen when editing a page but not when viewing, 404 errors all over, deleted things not deleting, etc.

Moderator
#38 Posted by AweSam (7373 posts) - - Show Bio

Yeah, it's pretty bad now.

#39 Edited by pikahyper (11952 posts) - - Show Bio

and it is getting worse

Moderator
#40 Posted by pikahyper (11952 posts) - - Show Bio

pibbs online he will save us !!!

Moderator
#41 Posted by mrpibb (496 posts) - - Show Bio

I jinxed it sorry guys. Should be good.

#42 Edited by mrpibb (496 posts) - - Show Bio

just to provide some background, we've been meaning to do about 3 hours of downtime to rebuild the dbs but it's been pushed off due to proper timing. I think we'll have to do this soon but you'll have at least 48 hours notice before it occurs (the site will go readonly, probably around 10pm PST some night).

#43 Posted by pikahyper (11952 posts) - - Show Bio

@mrpibb: yay !!! just in time, I've gotta get to new releases :D

I'd recommend doing it on a Sunday night, that seems to be the slowest night for wiki activity

Moderator
#44 Posted by akbogert (3218 posts) - - Show Bio

So is this actually fixed for now?

#45 Edited by pikahyper (11952 posts) - - Show Bio

@akbogert: everything has worked fine for me all night.

Moderator
#46 Edited by akbogert (3218 posts) - - Show Bio
#47 Edited by Arkhamc1tizen (2143 posts) - - Show Bio

There are so many 404 errors sometimes when I just get onto the homepage.