DB Issues - 12:13am Monday 4/29

Avatar image for mrpibb
mrpibb

513

Forum Posts

181

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Hi all

We're having mysql slave issues. This will cause content to not show up (but it is getting saved). I'll update as I find out more from the ops team, but please don't file bugs on this yet.

thanks

Danny

Avatar image for mrpibb
mrpibb

513

Forum Posts

181

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

We are now resolved. Sorry for the issue.

Avatar image for bumpyboo
BumpyBoo

14977

Forum Posts

270338

Wiki Points

0

Followers

Reviews: 10

User Lists: 20

#3  Edited By BumpyBoo  Moderator

@mrpibb:

WOOHOO!

Thank you very, very much :) This was driving a lot of us crazy! You da man *wink and finger gun* :)

Avatar image for mrpibb
mrpibb

513

Forum Posts

181

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@bumpyboo: not a problem, sorry for the delay. Wasn't home when it first happened.

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#5 pikahyper  Moderator

@mrpibb: it's like way worse in the wiki now, 95% of edits error out and don't save, most of the time you can't even get into edit mode and there are constant 404 errors.

Avatar image for bumpyboo
BumpyBoo

14977

Forum Posts

270338

Wiki Points

0

Followers

Reviews: 10

User Lists: 20

#6 BumpyBoo  Moderator

@pikahyper:@mrpibb:

Yes, the PM system is way worse too, messages to and from not even going through anymore, even if the notification says otherwise O_o

Avatar image for wmwadeii
wmwadeii

3327

Forum Posts

266306

Wiki Points

0

Followers

Reviews: 2261

User Lists: 3

@mrpibb: @pikahyper: Any update, I updated a blog entry and the changes did't stick.

Avatar image for mrpibb
mrpibb

513

Forum Posts

181

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

yeah, sorry guys. Looks like the replication had problems again. We're taking those machines out of rotation. Not sure why the alerts didn't contact the on call person. 30 min to fix.

Avatar image for bumpyboo
BumpyBoo

14977

Forum Posts

270338

Wiki Points

0

Followers

Reviews: 10

User Lists: 20

#9 BumpyBoo  Moderator

Is it just me, or is it happening again? O_o

Avatar image for awesam
AweSam

7530

Forum Posts

2261

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@bumpyboo said:

Is it just me, or is it happening again? O_o

It's happening again.

Avatar image for bumpyboo
BumpyBoo

14977

Forum Posts

270338

Wiki Points

0

Followers

Reviews: 10

User Lists: 20

#11  Edited By BumpyBoo  Moderator
Avatar image for wildvine
wildvine

15332

Forum Posts

2609

Wiki Points

0

Followers

Reviews: 3

User Lists: 50

#12  Edited By wildvine

@mrpibb Hey, The PM box is bugging out. Can't send, or receive. Can't make new blogs. And can't start new topics.

Avatar image for wildvine
wildvine

15332

Forum Posts

2609

Wiki Points

0

Followers

Reviews: 3

User Lists: 50

@awesam said:
@bumpyboo said:

Is it just me, or is it happening again? O_o

It's happening again.

Yeeeeeup.

Avatar image for bumpyboo
BumpyBoo

14977

Forum Posts

270338

Wiki Points

0

Followers

Reviews: 10

User Lists: 20

#14  Edited By BumpyBoo  Moderator

@wildvine: *grabs and won't let go* I don't like it!!!

Avatar image for awesam
AweSam

7530

Forum Posts

2261

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@bumpyboo: Seriously, it's even worse now. Threads keep vanishing, then reappearing. It's the Vinepocalypse!

Avatar image for wildvine
wildvine

15332

Forum Posts

2609

Wiki Points

0

Followers

Reviews: 3

User Lists: 50

#16  Edited By wildvine

@bumpyboo said:

@wildvine: *grabs and won't let go* I don't like it!!!

*Strokes your hair* There there. Wv is here.

Avatar image for bumpyboo
BumpyBoo

14977

Forum Posts

270338

Wiki Points

0

Followers

Reviews: 10

User Lists: 20

#17 BumpyBoo  Moderator
Avatar image for wildvine
wildvine

15332

Forum Posts

2609

Wiki Points

0

Followers

Reviews: 3

User Lists: 50

@bumpyboo

We survived the new switch over. We have survived three spam bot invasions and one spam-pocalypse since I have been here. We survived 2012! We shall persevere!!!

Avatar image for awesam
AweSam

7530

Forum Posts

2261

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Avatar image for wildvine
wildvine

15332

Forum Posts

2609

Wiki Points

0

Followers

Reviews: 3

User Lists: 50

#20  Edited By wildvine
Avatar image for bumpyboo
BumpyBoo

14977

Forum Posts

270338

Wiki Points

0

Followers

Reviews: 10

User Lists: 20

#21  Edited By BumpyBoo  Moderator
Avatar image for pyrogram
Pyrogram

46168

Forum Posts

13113

Wiki Points

0

Followers

Reviews: 10

User Lists: 1

Avatar image for bumpyboo
BumpyBoo

14977

Forum Posts

270338

Wiki Points

0

Followers

Reviews: 10

User Lists: 20

#23  Edited By BumpyBoo  Moderator

@pyrogram:

It's a nice gesture but I CAN'T SEE MY NOTIFICATIONS OR PM'S!!! *runs round in circles* It's the END OF THE WORLD!!!

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#25 pikahyper  Moderator

please keep it relevant people

Avatar image for pyrogram
Pyrogram

46168

Forum Posts

13113

Wiki Points

0

Followers

Reviews: 10

User Lists: 1

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#27  Edited By pikahyper  Moderator

@pyrogram: it looks like the problem may be fixed.

Avatar image for mrpibb
mrpibb

513

Forum Posts

181

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#28  Edited By mrpibb

yeah sorry guys. DB replication problems. Fortunately we found the root of the issue so they should not crop back up (short of our machines burning down).

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#29  Edited By pikahyper  Moderator
Avatar image for ellie_knightfall
Ellie_Knightfall

5022

Forum Posts

4

Wiki Points

0

Followers

Reviews: 0

User Lists: 1

#30  Edited By Ellie_Knightfall
Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#31 pikahyper  Moderator

@mrpibb: yep you jinxed it, submissions keep appearing and disappearing as well in the queue

Avatar image for awesam
AweSam

7530

Forum Posts

2261

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@pikahyper: I only noticed a minor problem. Not as bad.

Avatar image for eagle_of_the_east
Eagle_of_the_East

886

Forum Posts

75

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Happening again! Chrome, Windows 7

I guess the machines burned down :-P

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#34  Edited By pikahyper  Moderator

@awesam: oh it is way worse this time

Avatar image for awesam
AweSam

7530

Forum Posts

2261

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@pikahyper: No. Earlier, I could hardly check my messages, or open thread.

Avatar image for ellie_knightfall
Ellie_Knightfall

5022

Forum Posts

4

Wiki Points

0

Followers

Reviews: 0

User Lists: 1

@pikahyper: How's it differentiate from last time?

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#37 pikahyper  Moderator

@ellie_knightfall: before stuff (wiki, forum, pm's) would still go through, it just wouldn't always show up right away or it would show up off and on, now I can't even get things through, error messages all over, submissions that were approved won't disappear, other submissions appearing and disappearing, new and old page history disappearing, edits that can be seen when editing a page but not when viewing, 404 errors all over, deleted things not deleting, etc.

Avatar image for awesam
AweSam

7530

Forum Posts

2261

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Yeah, it's pretty bad now.

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#39  Edited By pikahyper  Moderator

and it is getting worse

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#40 pikahyper  Moderator

pibbs online he will save us !!!

Avatar image for mrpibb
mrpibb

513

Forum Posts

181

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

I jinxed it sorry guys. Should be good.

Avatar image for mrpibb
mrpibb

513

Forum Posts

181

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#42  Edited By mrpibb

just to provide some background, we've been meaning to do about 3 hours of downtime to rebuild the dbs but it's been pushed off due to proper timing. I think we'll have to do this soon but you'll have at least 48 hours notice before it occurs (the site will go readonly, probably around 10pm PST some night).

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#43 pikahyper  Moderator

@mrpibb: yay !!! just in time, I've gotta get to new releases :D

I'd recommend doing it on a Sunday night, that seems to be the slowest night for wiki activity

Avatar image for akbogert
akbogert

3323

Forum Posts

193

Wiki Points

0

Followers

Reviews: 5

User Lists: 31

So is this actually fixed for now?

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#45  Edited By pikahyper  Moderator

@akbogert: everything has worked fine for me all night.

Avatar image for akbogert
akbogert

3323

Forum Posts

193

Wiki Points

0

Followers

Reviews: 5

User Lists: 31

#46  Edited By akbogert
Avatar image for arkhamc1tizen
Arkhamc1tizen

2320

Forum Posts

674

Wiki Points

0

Followers

Reviews: 6

User Lists: 33

#47  Edited By Arkhamc1tizen

There are so many 404 errors sometimes when I just get onto the homepage.