Why doesn't PG have a bug bounty?

PG, Why do game breaking bugs (towers cost more than storage can hold) and re entrant bugs like moving buttons around in Android and farms having only cosmetic effects constantly keep creeping into the game?

Why not have a small pod of remote interns who are play testing pre release software? They could be 1099 at minimum wage as they create regression testing plans and then paid a bug bounty when they find one before the software goes GA? (Maybe the dev holding the bug has to put a quarter into the bug jar?)

Who knows, maybe some of the experienced players would sign up in their spare time. The quality of your software goes way up and the costs are minimal.

I wasn’t going to do this, but now I am. @PGawal

Part of the player discontent is the bug fatigue and the PX team knows it. The overall quality of the game is low, in particular for android players. Players are less apt to spend on something if it crashes 5 - 22 times a day.

Some folks have even volunteered to run debug versions of the game to find memory leaks and other issues. Something to think about as well.

8 Likes

I’ll just leave this here :upside_down_face:

2 Likes

image

While the cartoon is from the developer side, the point works for QA too… Doing pretty much anything by bug count is a terrible idea. PG would be flooded with a million legitimate bug reports and ten million “bug” reports that they’d have to filter through.

2 Likes

You mean something like BattleCry Warrior Brigade?

There’s not a great or particularly short answer to this, but I’ll mention a few of the complicating factors. Note that isn’t a list of excuses and I think there’s a lot of additional things we can do to mitigate this stuff. But it’s a pet peeve of mine when people think bugs are from a sheer lack of testing (which we, you know, do) and I like talking about this stuff so buckle up friends. I’m also technically going off topic from a bug bounty so apologies for that.

So, one factor is the size of the game. There is an enormous amount of things to check, dependencies that can cause weird interactions, and just a lot of systems and, like, stuff you can do. In theory throwing more people at this problem helps, but a little like the famous mythical man-month, having more people wandering around the game hitting things may not help much. That will generate a lot of noise and bug tickets, but there’s no guarantee any of them would, say, slap an Abyssal dragon on a perch on one device and then attack that base with another device AND notice they were getting 2-shot when they shouldn’t. Or that we would wade through all the bug tickets created to notice that one in time.

The next is iteration. We’ve done a lot of work to make it easy to change certain parts of the game so you don’t need a computer science degree to tweak a tower’s health (and I’ve been helping some small endeavors to make changes even faster and more sane) but a downside of that is that, well, stuff can change a lot and very quickly. And if you don’t tell QA you’ve changed it they may not test that area again, and so a simple lapse in communication now results in a bug going live. So sure, QA has been testing Abyssal dragons on perches every week for 2 months now as they should because that’s a common player thing to do obviously, then 3 days before we ship someone fixes what they think is an unrelated bug or makes a quick perch adjustment that they forgot to mention and voila we’ve messed up (though I don’t actually know how that bug happened).

The last thing I’ll talk about is unintended changes. This is the one we possibly hate the most and so we have a lot of protections in place for it, but sadly nothing is foolproof. An accidental change can come from a lot of places, from simply hitting a key with a file open and not noticing you changed something, to outdated scripts screwing stuff up. That one always worries me somewhat since we have to use a lot of scripts to generate, optimize, package, and do all sorts of other things to our data and assets, and we assume they all work flawlessly until we’re proven otherwise. Sometimes it’s quite obvious when they need to be updated to work with some new format or whatever, but they could very subtly add changes we don’t want and that happen to be in QA’s blindspot.

For an extended hypothetical example let’s say we have a new upcoming version that solely has a single dragon. It reuses spells so there’s no new code, everyone swears upon their soul that they won’t make any other changes, and overall it looks like a small release. QA therefore does their basic checks on the tutorial, on upgrading on gamestates from players with various progress in the game, that the new dragon works and can be acquired, etc. And the version after that one has all sorts of stuff planned, really just stuffed full of things that the whole team has been working on, so QA wisely chooses to focus most of their time on that one. And none of us realize that the script that an engineer with a PhD in mathematics wrote 8 years ago that strips out blank lines at the end of certain files instead deleted the last row of a rune file, which the game gracefully handles except in a Kingdom Wars event in which case it crashes if your third dragon in an attack has that rune. I wouldn’t blame QA or a lack of testing for that; that’s instead a problem with process along with just bad luck really. And while we’ve learned long before War Dragons to set up warnings to metaphorically yell at us if we delete a row so I’m not too worried about that particular case happening, the general idea of a helper script possibly secretly breaking something hangs over me like a Bug of Damocles.

I hope this gives a little bit of insight into the exciting challenges we tackle daily. Again I’m not giving out excuses to try to make you all feel bad for us or something, for you should hold us accountable when we introduce issues in the game you’ve invested in, but I’d like to emphasize that we try to prevent bugs in a myriad of ways and man do the bugs fight back hard.

23 Likes

This is the most in depth response I have ever seen from PG. Thank you very much!

5 Likes

I really appreciate the in-depth response.

And for what it’s worth…do you think it would be worthwhile to grab a group of players as volunteer testers, the same way that the GPF advises on gameplay balance and the Early Birds do the same for Atlas?

Just taking 20-30 experienced players who are willing to poke, prod, and try every permutation and combination they can think of, and tossing them on a test server, perhaps. Maybe even locking down the “final” version of an update for a week before release, putting the teams to work on the next update, and on fixing whatever bugs the player-testers find?

I know you mentioned that throwing more people at it might not be the solution to this sort of thing, but at the very least it seems like it’d scrape away the easier-to-find ones that pop up in the forums the day after each update. Those are less the “third-dragon-in-an-attack-has-that-specific-old-rune” bugs and more the highly noticeable ones, like weird UI issues, or the extremely predictable ones, like what the players might do with a brand new dragon or tower.

It wouldn’t catch everything, but I think a lot of the frustration stems from the fact that so many of these bugs seem less like obscure things that’d go unnoticed, and more like “It’s noticeable the instant you log in/use the new spell/etc.” bugs. So maybe it would give the players a clearer sense that there’s an effort being made to fix them.

…besides, I don’t know exactly what determines release dates - maybe PG has a schedule that has to be adhered to - but I strongly suspect that you’d get a much more positive outlook from the players if updates were a little slower, but a little less buggy.

Edit:
Sorry if I came across as criticizing - I know you’re doing a hard job, and I have just barely enough familiarity with this sort of work to understand how one little change - accidental or intentional - can cause all sorts of problems, including in seemingly unrelated code. I think unfortunately, though, the knee-jerk response from players, especially those without an understanding of the process, is often an outraged “HOW COULD THEY POSSIBLY MISS THIS? DID THEY TEST THIS AT ALL???”

1 Like

One of the biggest issues I see is that it has been noticed for years that if a bug benefits pg then we will get to it when we can. If a bug benefits the player then it always seems to get fixed in record time. If players could not purchase packs then it is all hands on deck. If players games keep crashing or get booted from all chat then it is well we will get to it on Monday when we come back to work. Sorry for ya…

2 Likes

I agree, those are the worst and really damage trust in us. I have further thoughts on them but will spare you all, because it’s time to talk about test realms.

I hadn’t put a whole lot of thought into this in the past, but as it happens I was in a brainstorming group just a few weeks ago (along with PGLawson and the original mastermind behind Atlas, among others) about prototyping and iterating in our live games. And one of the things we talked about was the feasibility of creating some sort of public or closed test environment like what you describe.

Personally I don’t think it’s worthwhile for Dragons. Not a popular opinion I imagine, but there’s a few key obstacles I see. One is the overhead. It is surprising how much goes into maintaining a separate test realm, which I learned some of last week when I was involved in helping set up something for the GPF to test (which went pear-shaped in like 3 different ways and I very much appreciated their patience and general chillness). So already I suspect the amount of time we’d end up spending on it would hurt us since it’d be the same people who’d fix bugs who are maintaining and firefighting the test builds. I’d also be more amenable to this if we were talking about a game with a fresher code-base, but WD has had a lot tacked on over the years and trying to make it do things it wasn’t designed for is much like trying to make a dragon do something it doesn’t want to.

Another obstacle is the long-term nature of the game. Take World of Warcraft for an example. Their test realm is primarily used to test raids, so they outfit everyone in endgame gear and toss them at some bosses to catch bugs. They also test the leveling experience of new expansions so again they put everyone at max level and let 'em at it. Yet not only do they not catch all bugs (and they often ship bugs people reported, which is a separate topic) but they also are only adding things very carefully to the end of the game for the most part and testing accordingly. Our new things meanwhile can often be accessed at many levels and points of progress (outside of stuff like lineage tiers of course) and the established nature of each player’s gamestate is really important. Constantly duplicating their regular game progress or other solutions are possible but bound to be really engineering-intensive and so again I don’t think it’s worth the opportunity cost.

However, with all that said I do think there’s more information we could get out of the brave souls of the GPF if we improved that test environment. I suspect we could find more of the obvious (to you all) bugs as part of their usual playtesting and feedback if we put some time and effort into that framework. But again it’s an opportunity cost and we’ll have to determine the best use of our time.

I actually agree with the people saying we are too slow in shipping quality of life fixes and other things the community wants. So I want us to go faster, though not at the cost of even more bugs of course.

3 Likes

This is why test automation and code review exist though. You write code to test your code, even your data if you rely on configuration data as much as this game appears to. You run the tests for every commit. If a bug happens, while fixing it you also write a test to make sure it doesn’t happen again. Overall you should be able to iterate faster and with greater confidence.

I know this is not news to anyone working in software development.

Edit: meant to be a reply to @Lachrymae not aeana.

Test automation and code review gets tricky if you are dealing with mixed code. Just a little personal experience.

1 Like

Point well taken; creating an effective isolated testing environment may just be too resource-intensive to really make it worth the while. We really do appreciate the information, though - even just knowing why a thing is or isn’t happening makes a world of difference (especially when it turns out to be a technical limitation).

As for this, though, I agree, but I think either I misstated, or you misunderstood. Rather, many players feel like the quality of life fixes, and other things the community wants, are being ignored in favor of pushing out updates that were neither requested nor fully-tested.

Thus, the notion is to get the quality-of-life things out faster, particularly when they should be easy to implement (concepts like scaling chest contents or tier costs, for example, already have the delivery mechanics in place and validated), and to delay less immediate concerns until they can be tested properly (rather than adding to the ongoing sense of running from one fire to the next).

2 Likes

Sorry, to clarify my random thoughts: Dragons partially uses a release process that’s often described metaphorically as trains regularly leaving a station. You put the work you got done on the train about to leave (aka the next version) and if you miss it goes into the next train (aside from time-based things like seasons of course. Can’t miss that train). So one way to fight bugs kinda as you suggest would be to stagger those trains further apart, giving more time to fix things, or to wait longer to put something into a given train/release. I would rather have more trains, more cargo, and more vigilance about what exactly we’re putting into them. Well, actually I don’t like this train model much and I’m not used to it so maybe I’m just biased. I prefer defined milestones. But yeah it sounds like we want the same end goals overall.

Yup, we use them, particularly our engineers, along with other basics like version control. Yet they aren’t silver bullets and they’re much more suited to non-game software which tend to have less moving parts, edge cases, and players interacting with everything constantly. I have a programmer friend who left the games industry to work on financial apps and he described it as far more relaxing even with the stakes being higher.

2 Likes

If you’ll allow me to slightly mangle the trains metaphor, I think more care is needed in scheduling both trains and cargo. Or perhaps this really falls into your “vigilance” category.

On one hand, we want more trains; this would provide more rapid, but presumably smaller updates, create a sense of progress, and reduce some of the issues well before the complete elimination of the concern.

On the other hand, some things are being put on the wrong train; they are either being shipped before they are ready, or they are being delayed in order to load the cargo in as few shipments as possible, creating a poor end result for both PG and the players.

So it really looks like we’re on the same page with regards to this - I just hope that…well, whoever is in charge of these processes (pod leaders?) is willing to listen, and to make some of the necessary changes.

And again, I deeply appreciate how willing you’ve been to talk to us, and how open you’ve been with your information. Perhaps more than anything, that’s something that everyone has been looking for - there’s only so much “We will pass it along,” or “It is being taken under consideration” that people can stomach without seeing results, a timeline, or at least some level of solid response or explanation. We’re desperate for even the tiny scraps of “How” and “Why,” let alone “When.”

…I should probably apologize to @Hwrd, though, and leave the conversation here instead of monopolizing his thread (and your time) any further. Thank you, again. :slightly_smiling_face:

3 Likes

Stuff like this is actually pretty fun to read about. Thanks for the long reply :smiley:

It is a bit odd when things like the Battle Cry bug return (and ended up ruining stats for the most recent Assault event :sob:) or when spells get copy pasted from an existing spell and not updated >.>; - a couple more checks and such. Still, goes back to WD’s crazy codebase.

2 Likes

If it didn’t happen with every single update I think we’d be more forgiving. A smooth update is a real rarity in WD.

Oh, and:

This. :point_up_2:t2:

This. :point_up_2:t2:

Oh, and again, this. :point_up_2:t2:

Thank you so much for always giving us honest and in depth answers. When the players have a better understanding it is easy to forgive PG for the little things.

On another note, I don’t think it should take more then a month to fix the fact that moto G6 phones can’t even load the game. (Yes I have restarted, cleared the cache, reinstalled and even a factory reset.) Not even a comment about if they are working on it or if they are just dumb founded and can’t figure out what’s wrong. I would be fine if they said this, at least I would know they tried.

I absolutely respect this, and personally I’ve always been patient and understanding when things break after an update. However, I posted this in the Letter to PG post and I feel it’s topical here:

"Something that could bring a fast and easy win in helping to ease player perception would be by avoiding known problem areas. For example, if you have a glitched spell - such as the fabled quick cast spell glitch that cripples the spell kit of a dragon (Dreth, Firefin, and Hyaku being the most recent), and you cannot fix it for whatever reason, maybe it would be a good idea to not have dragons based on that spell going forward until it IS fixed.

Continuing to base ongoing mechanics on known glitches leads to frustration (putting it lightly)."

The quick cast spell glitch has been going on since…pretty much forever. It affects the reliability of dragons with an AOE freeze/damage - and yet, three divine dragons in the very recent past have been given spell kits based around it. I understand that the issue is likely buried deep in the spaghetti code, writhing about in the guts of the game like a certain tentacled outsider submerged in the depths of an ancient ocean, but it makes it feel like the devs know it’s broken and simply don’t care since they keep pushing unreliable dragons. It’s bad enough on lineage dragons (Jul and Opes spring to mind), but it especially burns when it’s a divine dragon and you’ve gone to the trouble of spending 30k sigils on something that does no damage and dies instantly on a kill island 10% of the time :jack_o_lantern:

1 Like

Thanks for the detailed explanation, I think a lot of people here appreciate it. While software development tools to some extent help to tackle bugs there will most likely always be things missed especially in games. From your description you already try to reduce complexity, apply test driven parts and so on. Without a real insight no one could blame what you possibly do wrong.

My assumption is that beside the trains, there should be trucks as well. One example here is the broken rss progress bars, which isn’t a major problem at all, but it is visible each day since 5.02 release and therefore bothers players which results in loosing trust in overall code quality.

Also bugs which never get fixed or returning in an slightly adapted version do bother players, so never mind refactoring.

3 Likes

Wow is cheaper, fixes the bugs in a timely manner and takes credit for their mistakes.

1 Like