Guild Chat - Episode 72
Guild Chat - Episode 72
- The following is an unofficial, player-written transcript of the episode. The accuracy of this transcription has not been verified by ArenaNet.
For Fans By Fans merchandise ad [5:55]
FOR FANS BY FANS®
GUILD WARS 2 LOGO HOODIE
PATH OF FIRE T-SHIRT
SHOP NOW www.guildwars2.forfansbyfans.com
Guild Chat studio [6:26]
Rubi Bayer: (laughs) Hi guys. Welcome to Guild Chat, and happy Friday. I'm your host Rubi, and today on Guild Chat, we're going to be talking to some of the members of our QA Team. So, let's go ahead and meet them. I'll start with you and have each of you talk about what your name is, what your position is, and what you do here at ArenaNet. Nathan?
Nathan Brown: I'm a QA Embed. I mostly help out other teams when they need more help.
Rubi: Okay. That makes sense. Your first time on Guild Chat, so welcome.
Nathan: Thank you.
Rubi: Thank you for giving us some of your time. Right. Nick?
Rubi: A few.
Nick: A few. (laugh)
Rubi: Thank you. Again. Travis!
Rubi: Okay, so... Doing Embed, what does that mean exactly? I'm not—it's not a genuine question, I mean, I know, but... it's a new term.
Travis: So we're mostly responsible for the organization and management of QA processes within ArenaNet. We're responsible for like, putting in the test passes that we run, and kind of being the gatekeepers of knowledge to QA. Like, the things that tend to catch teams or catch releases off guard. We try to anticipate those and make sure that we have plans in place to test those.
Nick: On top of that, we're also pseudo-in charge of making sure that the team meets specific standards set by our leadership and also... you know, having playtests with the team and holding meetings to make sure that everything is like, well-communicated? You know what I mean?
Rubi: Yeah, yeah, yeah.
Nick: Like that kind of stuff, yeah.
Rubi: That kind of liaison between the ???,[verification requested] yeah.
yeah speaking of questions if any of you guys watching have questions we're gonna try to answer a couple at the end as time allows so if you have them throw them in chat they will pass a couple along to us and we will see if we can get to those if time allows I'm finding out how talkative you guys are going to be yeah so um you mentioned that you are QA embed for AIDS they think you kind of you kind of hop around wherever is most needed right yeah so whoever I'm generally speaking a Kevin mids role will ramp up as it gets closer to release so I'm there to kind of help out with some of the extra stress that comes on with that to alleviate that yeah that's like there's like primary is usually for each individual team like he's the rates and fractals I'm one of the living world teams and Nathan Nate like Nathan Nathan's role is like help out every single person and kind of also help keep that communication going in between each one or each different team to be fair yeah it sounds like so how do I mean I feel like especially you Nick will jump around teams like once okay episode three is out that's that's off the books we are ready to move on to the next thing so you're chipping around between teams quite a lot how do you maintain I mean how do you maintain a good work flow I I know you've very good at what you do so I feel like communication is but how do you keep track of that jumping from one thing to the other is there overlap yes there's definitely overlap so one of our other embeds her name is milena she's actually getting married so that's tomorrow that yeah that's like pretty soon and because of that like you know she has to go take care of all over wedding stuff so I got made primary for Episode four and you know lake hopping from three to four like there was a lot of responsibility stuff that they need to get taken care of and on like the ground level basically to answer your question a lot of sticky notes is just filled with sticky notes that are like I need to remember every single thing that will happen and that people have asked me to do and I have like all of them on my desk and then like this subsection that's basically the done pile and every single day I look at my done pile or like at the end of every we can clean up the done pile and I look at it I'm just like this was a really busy week err this wasn't a really busy week and you know like that's just the way I do it like Nathan has something similar except for he keeps it all in Outlook because he's a lot cleaner than I have I have tasks for each different team so that I know exactly like I can switch gears very quickly and be like what do I need to do for this team right now yeah um and then swap back and forth between that okay but like to go back to the living world as a whole thing each of the living world embeds there's three of us you know we're our primaries on our own episodes but because each of the episodes are so so large and such an undertaking because we want to make sure that each of them is the best they can be we help out each other to also keep that flow of knowledge going and to find areas that you know one of us might not have found like each of us has their own strengths so we kind of help interplay that within the teams and like Keowee is all basically about like trying to make sure that we can get things out as clean and as fun as and as balanced as possible so making sure that multiple eyes are on it is really important yeah so and when you say you're the primary that's not this is all in your shoulders I mean you've got you've got people helping you that you can communicate and work with yeah something I'm interested in because you mentioned sticky notes and you mentioned Outlook and that's like Outlook is like the thing after my own heart I have so many different folders that I'm like throwing everything into in Outlook so it's very similar for me how do your processes different differ between you guys according to what works best for you I'm actually super curious about that because it's not like okay everybody everybody has issued sticky notes and that's what you have to do yeah oh gosh and I can really only speak for raids and fractals obviously but we have a we have a very iterative designs style so we're very quickly making changes so we'll usually just have an email sent out by our producer at the end of a play test which will contain a list of all the things that we've from the play test so I mentioned earlier that's kind of our job to write the tests and kind of be the keepers of QA knowledge so how that ends up playing out at least four raids and fractals is I'm not the only one that's testing the raids and the fractals I test it with the team so developers can go in and be like well I don't really like all that boss is acting or I don't really like how this particular hallway leads into it so what we then do is those kinds of notes get put into an email from our producer that gets sent to us and then we produce those in a we put them into the bug writing software that we have which is a Wow total mind blank therefore say we put them into JIRA which is one of the software games that we use and then from those bugs we can triage those bugs out as this is something that's a high priority fix as in we need it fixed by tomorrow for the next play test or it can be something it's a little bit later on like do we really want to fix this what kind of gameplay opportunities does it offer and then it's I guess I'd have to say like from as far as like I'm not entirely sure how it is on living world but I feel like raids and fractals is very fluid in how we do things it's very in fact when it comes time to regress the bugs which for those of you who may not know regressing is when you take a bug someone's made a fix to it and then you go back and run that bug a whole bunch of times over and over again to make sure it doesn't happen again our regressions sometimes aren't terribly detailed so they could just say things like boss stalled during fight which could mean a lot of things at the time that we may at the time that we write the bug we know what it means and we know what do we make the fix but later on if you weren't actually there to see the bug you probably couldn't regress it so I usually have to do and regress all of the bugs that we have okay it makes sense and to kind of go like to dovetail off of what Tavis is talking about I don't know having conversations which each of the other embeds because it's not just us you know we have we have a bunch and it we aren't like even that bunch isn't the entirety of our QA department because we also like you know like test ones and tests to you isn't thing yeah but like having conversations leader of us embeds it's pretty consistently clear at least to me of that like a lot of us do things very differently like with Travis you know a lot of that progression iteration progress process and scheduling is balanced between him and his producer whereas with living world because we're so large you know we have to make use of our test 1 and test 2 is a lot more because of the fact that like I couldn't do it myself and like yeah myself and like the other at living world and beds and even Nathan we wouldn't be able to do it by ourselves and on the question of like how do we how do we differ like I said earlier each of us has different strengths and it's all dependent on like how we do things what's cool buttery Internet is like we are allowed to introduce process and kind of pitch process and play to our strengths when it comes to our it rolls as long as we get as long as we get the work done and get it done well yeah so what I do is like you know I go into our engine and I will like review all like a lot of the scripts you know I'll review a lot of the content I'll actually look at the scripts and create passes based around my interpretation of how those you know of how that stuff is built out because for me that's how I can find where all the paint where I received the paint points to be and yeah like like true what Travis is saying like you said that like your producer schedules those play tests and kind of creates those triage lists what me and my producer do is like we split it between team and iterative play tests with alpha is like I handle that stuff and then like stakeholder reviews are handled by our producer and it's like I'm usually the one that's like writing up and sending out all of those notes okay yeah so it's iterative and different between each team and kind of divide and conquer yeah so how does that work for and you can just pick one of the last things you work on please pick something that's already released I need this job how did the process work for one of the recent things that you helped out with um well one of the more recent things that I helped out with was the I would say three would be what I worked the most on um and so you have to what I'm doing is I I kind of I work under the embed that you know the primary mode yeah so I have to learn their process and I've worked with them um and so each one's different some of them like Nick uses on the tester ones a lot more and so we end up like sending them the information and like how to run the tests and so I will like assist with that but other times it will be like if I'm working on like an arch pass for like a wedding or something I will just like the end it will be like hey could you just check this out and I'll just test that myself um essentially just like uh it really depends on like who you know who they are right um so yeah okay let me our pests for a weapon seem what are you looking for uh in general you're looking to make sure that the I mean the weapon looks good yeah we're using it um so it doesn't vanish yeah I mean or like when someone like turns invisible like what does it look all right alright um that's something we notice a lot is like there be a checkerboard pattern so oh yeah yeah um so you're just you're looking for stuff like that you're looking to see if the weapon like will stretch something if the weapon is too long if you're holding it correctly um yeah that's what that's what I do more for like um what we'd be releasing when we release empty Exide um okay that makes sense yeah but like with like it's like Nick it's it's much more like a like make sure that this content actually you know you can run through it and sometimes that can take a long time because you have to you know go through it and see if it will read you know set correctly and other things so that's more like a I asked someone else to do it and then I I review what they've they've got okay alright that makes sense yeah I'm kind of curious about how the process starts like our last like our last trade wing came out so they come to you and are we're gonna start we're gonna do this raid wing what do they need from you what's the first thing they need from you so one of the things that I'm pretty I'm fairly certain about living role this involved in this as well but QA actually sits in with the designers during the very beginning of the design because it allows us to see for instance if they want to do something like say moving platforms which are a huge QA risk no we've definitely made some some good strides and making that less scary than it used to be thanks to that way early on we can identify hey these are gonna be some large QA pain points we're gonna want to test around a lot we're gonna test around this a lot so we may not test it specifically but we know that things that interact with it might go badly so by being there at the earliest of the design phase we're able to identify as though sooner sometimes that can for instance let them know that maybe they have to look at other alternatives because it's just going to be too much of a QA lift and then sometimes it allows them to go ok we love it we're like we really want to do this idea so we're gonna focus on it and we're actually gonna make it better so like that for instance is where those changes came in from Ben Arnold is like he saw how bad moving platforms were for QA to test and he's like well I really want to do them so we're just gonna make them better for you to test ok so you're not just in the back of the room going what if we didn't right it's it's definitely a collaborative work we let them know that hey this is gonna it's we're kind of like a warning sign ok hey this could be true this could be trouble we're willing to take it on but just you know it might take a lot of time away from something else or it I mean it also allows us to be able to be to say something like yes please do that it's really easy it's super fast for us to check and it's super fun for the players so do that Travis is saying like we are usually this like funnel of tribal knowledge that we try best a document but you know there's there's just so much that we that we like through through our careers as I'm beds learn and know so like I was saying like we are included pretty early on in the process we go to all of the like the scope meetings the design meetings the implementation planning meetings because through those things like we can make sure that people avoid specific pitfalls you know we don't we try to make sure that we don't make the same mistakes twice yeah because it's important to make sure that everything is as clean as efficient and as balanced as possible so we want to make sure that we don't waste time retreading old you know all the issues you've worked on so many different bits of content and been involved in so many of those things all of those pieces come in and you can remember hey this is something that is relevant to what I'm working on now yeah there are often times really you'll go up to a designer and be like hey this is something that I'm worried about because this is broken in the past um could you could you maybe like could we look into this right look into maybe a better way of doing this or you know something that would be less risky and it enables you to not only it sounds like it enables you to not only offer a warning sign but maybe offer potential solutions based on past knowledge and also like be able to point them in direction of like who might know the the right answer who the other designer was that ran through these issues already and has have of course more tribal knowledge on how do they solve those issues well it sounds like Ben there's that alright so I have a slightly more specific question what are some of your favorite work stories about things that you guys have found and/or caught before it went live answer each of those okay I will I will take whatever because I love I've seen sort of like these bizarre completely bonkers QA stories from the trenches that never make it on to life and I want you guys to see these so I read like one of the things is that I do in addition to being the rates and fractals is and also the automation meant as then we have automation that goes through and checks the things of code that you you generally don't want to do because it takes up a lot of time like logging into every single map or making sure that when you do an attack a particular like things take damage yeah um so as a result of doing those those automated testings I ran across a thing we weren't quite ready to talk about yet that might have gone to live and so you know I kept that from going to life because I mean obviously since we're not ready to talk about it yet I can't oh what did you do I just picture you're running across the bay it was I happened to actually sit right next to the people that working that are working on this so I just kind of turned around and I was like so this thing failed this usually fails when this thing happens is that related to what you're doing yes that led to the designers a particular going yeah we should investigate man what about you guys oh man aye aye I think so I've been here for quite a while um but all of his like blended together into like a hodgepodge of different bugs um Nick do you have any I'm still trying to think I'm thinking of one that you had mentioned to me in particular about an NPC where you could turn him hostile or friendly oh yeah there was so there was a on the plains of Ashford there was an event that had been broken for a while and I was I was scrubbing through some of the player submitted bugs and yeah thank you very much um and I found one that was it was basically saying like this event they never got they never saw it again and I was like that seems strange what's going on so I looked at the event and it was working fine for me but apparently like people would only ever see in the first week after patch came out and so I went and investigated couldn't figure out what was wrong with this thing I brought a friend in and they then we looked it together and it just so happens that we both interacted with the same NPC at the same time it both chose different choices because the way this event worked you had to choose to pacify the the char and if you chose the wrong choice they turned hostile okay so what happened is I chose to pacify and my coworker chose to make my hostel and so they became both pacified and hostile at the same time but you had to make those two opposite choices at the very same instance right um once that happened the they would they would turn hostile forever you could kill them the event wouldn't start up again and it's one of those things where you're just you you'd never think really to do this yes um but once once you figured out it worked you know we were able to fix it yeah um and that's the part that I really liked liked doing is going in and figuring out like something that it's like a real large pain point in figuring out like how can we you know fix that yeah and that is that's interesting to me because it seems like a lot of this testing is what is the absolute weirdest most unlikely edge case situation we could create that would cause a bug and let's think of them all it actually it actually comes up when you're not playing game then you're black playing games at home and you're like I wonder if and then you break the game you're like no I got it I got a turn off my QA brain for a little while but write that down first right and then we'll go back to work and we'll test that this is for like other games yeah yeah and just this seems weird let's see if we can apply it to work right exactly how about you Nick for me I mean honestly it's all relative like thing is when I started out that seemed crazy egregious now we're like okay that's that that'll happen she would do that there are so like I can't I can think of one specific example I'll give that but before that like there are there's a slew of issues that we catch that I'm so thankful that players never receive like we've run into issues and then I ask you to talk about them so sorry no it's like we've ran into things that would delete your characters you know we've ran into things that would make it's that way you can never log in again you know more specifically there was so let me go to the fun one before I go to the all the painful ones the fun one was that my stomach hurts now episode 2 of season 3 so that was back during a fire island like that area when you could turn into a more SOT you have the ability to attack other players and yeah and like stop them so as you can tell you know while it was fun we didn't want to make sure I didn't get alive yeah I have like stressed out thinking about that and accidentally going live and yes just wow yeah the grief ability of that was was crazy yeah like and there are other ones where it's like you would log in to your character and it would immediately crash because of the fact that our entire player like data is tracked through like progress bits basically they get set and unset and if a progress bit gets if a progress deaf gets shrunk at any point and you already have a bit set on the side that got deleted the game will basically crash constantly so finding those is both edge Kacie and really important okay so go back for non QA people if your progress if your progress death gets shrunk yes now uncute that okay if you were to do something right in an achievement let's say and then we were to change that achievement in some fashion like and push it to live if we weren't attentive to what things could have been set by the players what happens is whenever you log into that character you will constantly crash because the game is basically like this doesn't exist I don't assert I'm dead I'm it just I'm out no yeah so it's it's like I there is no like hard key example that I can like really good for those because there's it's happened a lot of times it's basically like yeah we make sure stuff like that doesn't happen I think so much so during pacifier we had an achievement for reading all of the books in Coors library that used to be a longer list and ultimately just because of various time we couldn't make it as long as long as we wanted to so we had a short in it and that's exactly what mixed talking where it used to have more entries than it did so if you had already completed one of those entries like an okay like as I'm obsessed characters yeah hadn't it would try and check that when you logged in because you were still working on the achievement and but it's checking for things that doesn't exist and it just gets confused and knope's out yeah I thought you were gonna talk about it via the inventory one that happened oh that's okay boy am i scared now no though this this went live I'm Olivia some of you may remember I think remember some people may have blocked this you may remember that a while ago a couple years ago we did our very first ever on Guild Wars 2 rollback live server rollback that was a result of us on expec 2 we had created code for our bigger bags mm-hmm so how bags are stored is in an array and we created a new code that allowed those arrays to extend farther what we didn't account for is that our inventory code determined soul binding based on its index within that array so if you extended an arrays length instead of making space for it it would just spill over into the next array so if you're like if your Guardians code ended at this point in the array and your Mesmer's code started in this array it would get pushed over and anything that used to have belonged to your guardian the game would now think belong - your Mesmer Oh which cost a lot of issues as you remember that now yes yeah like like going off of that like you know these are things like when issues like that occur it's always like breaking ground issues that we would have never been able to catch you know like like with episode three as an example that memory issue that we had where like for the first day everybody was unable to was unable to go past a second instance that was some like that like we would have never been able to cash before and but we've we've like I said earlier like we always push to make sure that we don't make the same mistakes twice yeah so we have created process and developed and or iterated on our current tools and or made new tools to make sure that something like that never happens again yeah the episode three thing I think it was a you and I talked to Z about that a little bit to us and it was fascinating after the fact I remember talking to you guys and learning so much I had no idea the boundaries that we pushed at that point so that was fascinating and the processes that we put in place after the fact to be like hey what if we didn't do that again yeah cool it's it's always when you try to break new ground and we were always trying to push the envelope in some fashion to make sure that players get the best experience possible yeah yeah well yes weirdly wacky fun things that come out of the bug is it not yeah account destroying madness so for instance the very first thing I worked on when I came to ArenaNet actually I worked externally for a while but the very first thing I did when I came over here was I was part of the heart of thorns clean up that big patch way back in April yeah at the time we happened to be I believe they were prototyping Mountain stuff on the some mount code yeah they were gonna come out with expansion too and they wanted to do mount so they're playing around with it what could they do we've seen some mount weirdness on yes nothing to do with mounts themselves though oh really what had happened is they were playing around with the camera because we knew we'd have to have different camera setups for mounts and players and somehow all of the corpses got tied to the got tied to that camera so we were in the middle of doing heart of thorns clean up and we get a patch which is always scary for us it's cute hey because I mean someone put something new in what happened and what are we load back in we go through we test things which of course requires killing things and as we kill things that we turn to do something else all of the dead corpses would turn with us they would all face the same direction the character was this pack of dead chalk around your feet that would follow your camera as it moved around another odd happenstance if that was because I happened to be doing one of the warrior animations which causes you to spin when I killed one of them yeah it actually took it actually took the corpse and attached it to me no so so I had a little chopped corpse orbiting me around as I move through the mouth did you fraps that or something oh absolutely I have a gif of it I can totally share with you see that because that's amazing oh my gosh speaking of seeing that Nathan shared a whole bunch of like QA gifts with me that we should watch because it's like a best-of things that didn't make it to life can we look at those mark and can we actually see them on the monitor briefly so we know what we're talking about we also have a slew more but these are like yes we'd be here like all literally all day we love taking so this one oh okay okay that one yeah okay we can talk about this one okay you know more about this one I mean like it's it's basically it's sideways boat yeah so if you all remember in episode 2 you have this boat back and path arounds [Laughter] okay for this one basically what happens is the way that these moving platforms were built out and you know episode 2 was kind of like the point where we were like alright it seems like we're gonna be using moving platforms and it's using 4 so we should really push like to get these things fixed the way that's built out is basically there's a path variable what the much points that this thing moves down but the way that it was built out and this was built out by Cameron rich who was a designer on the episode 2 team was fantastic he's awesome and I watched him do it a few times so it's it's a little hazy exactly on like every point but I'll try my best explain it basically happens is each of the points has a different like rotation and what happens is it will do this thing called interpolation where it takes the 1 point and another point and then kind of like hands in between each one okay so what would happen is after it's circled around a few times something got messed up with like the interpolation data or I guess they like each one kind of like slowly shifts and adds on so it got to a point where it added on too much and then it would just get stuck in a specific like in a specific stance will continue to move down that path so that's what you're seeing here and the cat frantically trying to keep up okay all boats go sideways this one was about in episode 3 this is during the four hours for warned ya and essentially Kanak would start just throwing bombs at allies ya figure out why so um so when so I like so some of these I haven't actually like seen through okay um so I don't know what the dissolution obviously was that he no worker does that yeah so for him what was actually happening here is that Kevin Freeman built this out also a fantastic designer for my team was axe happening as Kanak was running mm-hmm he would be choosing random targets and then firing off this skill when he was going to go plant seeds on his own but what would happen is that never really ended you know so like something something happened and that handler didn't get interrupted or something along those lines so that's he would keep on choosing like random spots and just throwing out these bombs constantly and that's pretty much how it worked because he had he had to run while doing it which is like two different things that have to be focusing on time so one was turned off he was he was done running but the part where he's throwing ball bombs never got turned off this also reminds me of something that I really love when I'm looking at a bug report sometimes it will have the issue and then the expected behavior which for some reason is so funny to me because it'll be like Kanak is throwing bugs at Kanak is throwing bombs at allies expect a behavior Kanak does not throw bombs which I mean I get that they have to be filled out I would like him to stop that very much please don't hurt my friends there is not a traitor you know it's like there is an inescapable pit here and the expected behavior is there are no inescapable pits yes stop it okay this is in the exact same instance exact same area except for you are the one using the bomb and what we found out is that we defined what what team's supposed to affect you know so so like it could hurt everyone and not just your enemies but hey it's finally quiet seriously the NPC chatter at that instant is perfect it makes me so happy yeah so how did you read just like tossing bombs at everyone see what happened I mean probably I mean yeah like that's I think each person kind of like does their own thing you know everybody works in a different way including like everybody like all of our testers yeah yeah and for this one it's just like it's a fun job you know like you get you get to play the game that you enjoy and you get to try out a bunch of dumb stuff that will be to stuff like this so it's just hey I'm gonna drop bombs everywhere and see what happens yeah it's usually an I wonder if moment yeah okay um we we do we do usually sound like guided like like this is what you should look for especially if we have like a new tester but like oftentimes these are just things you learn while you're going through like generally speaking you know like when you give me something like this I want to see if I can use it on my allies personality type cuz like I have like my friend mark his name is Mark or skis or Chester he's fantastic during I think it was episode 1 or 2 I can't remember there was a reward he was episode 3 there was a reward where it's a Canada you can place on the ground 3 that was 3 okay thank you it was there was a reward that you could place on the ground it was the cannon but the cannon had collision so what you could do is you could put another cannon on top of it and then no they came on top of that and then you could jump on top of that and then you could jump and put another one top of that so you could basically have like at but you could only ever have I think like two so it would just keep on stacking on top of each other until you hit the skybox yeah and it's just like what I'm always amazed by these things cuz you know like when these things all come through me and Nathan at least for my team and then we triage them all out so I get to see all these things and sometimes I'm like did you how did you do that you know cuz it's just it's like that it's like you have to want to be Sufi with this stuff you know there's a there is an issue actually when we did Fire Island um that it was actually a live live submitted bug um it was if you go into the instance with the mursaat yeah um and you if you went into the instance you left you came back you jumped up on a platform he would teleport up with you become invulnerable and you couldn't do anything anymore and if you went back in it was it was also still broken but it was like this this thing we're like why why did you do this why would you do that why would you go through all this effort yeah - yeah it's er it broke it but it's like who would have thought of doing that oh my yeah that's and that's what we were talking about earlier like what is the weirdest most convoluted bizarre edge case thing you could think of all right let's try that sliding so that one so that one's was an issue with the the Halloween puzzle and if you fell just at the right time and then you did the and failed it would teleport you back to the beginning but you would get stuck in this animation stage and sliding around on the floor the cape flapping is the best for me it's the animation so good and I remember we had that for a while like that was a thing yeah mm-hmm I mean like well like what's what's unique about things that could happen to the player is that this what looks like the same bug can also be different bugs like you can also get your character stuck in the state or you used to be able to by getting CCD like at a very specific moment during your animations and you would this would happen to you you'd just be like bouncing off man at while laying down and that's some of the difficulty that comes with it is you're like like I don't know why this is happening and sometimes you get bugs back it's like the designers like I don't know how you got this to reproduce and you're like I'm not really sure either and you you have to figure out like what what specific things all came together to try to make this bug happen yeah if you basically it's constant puzzle solving yeah it's like that's what that's what being a QA really boils down to is like creating puzzles and solving puzzles that are like handed to you so if like three different things are causing the same bug you say okay so I was I was falling this happened and now I'm sliding around on the ground they say okay I fixed that thing where it's falling and they hand it back but it's still happening so now you have to figure out what else is causing it yeah right and you have to like kind of like you know first make sure it does it still happen is it happening in the exact same way what happened changed like sometimes it's like I have to change one variable each time being like alright so it's still happening when I'm doing it what happens if I try doing it you know when I'm not map loading or you know what I mean like yeah and kind of like trying to tweak it to see what's actually breaking this is this is more than one puzzle yeah oh gosh that sounds super fun it is stressful okay let's look at the moonwalking teddy bear I know that's not at all what's happening but that's what it looks like to me it does yeah basically for this one the bear like the agent got stuck in its swimming animation so even though it's on land it thinks it's swimming so like you know those corgis are like those dogs that you hold over the water or like when they just hear the water and then they're just like doggy paddling he's just being lowered into a tub okay what else we got oh yes what happened why this is actually a this is found by Brett one of our other embeds yeah um I'm actually not quite sure what's going on it has something to do with dismounting gems are you know great this is when they were testing mounts right yeah it was from the the mounting our testing sorry from when we were testing dismounting on Mount gosh I'm not sure I can save it might be a little bit too much okay behind the curtain yeah I mean basically what's happening is like as you know that mystic toilet brand-new phrase yeah the the mystic forage like will spit you out so basically here is like you jump you move your characters over here but then the thing that the mystic forage is like nope you're in still inside of me and I need to kick you out so what happens is your characters then pulled back into it because the game was like wait no you shouldn't be in there so you like oh no so much invention and then somewhere it's like no I want you out then you're just like BAM back out so that's basically what's happening with itself is it's like so morose and and the the game server are having a debacle like a dispute about fantastic I'm also just like watching the characters fly yeah yeah alright what's next there's a lot of character flying around stuff right that's that's my favorite yeah right this one yeah it's it's it's the same issue to an external sensor up in a different manner where it's basically the client and the server are talking to each other and like this gadget is like I should be doing something to the player agent but it's happening after like the player is already out of that area so that's what causes things like this gotcha yeah cuz with mounts you know our game was built around no mounts so when mounts introduced it was like you can now move way faster and with everything being built out to go like things not being built out to go as fast as players we're doing like it caused issues like this where we had to now account for the fact that like hey this thing can enter and leave this trigger volume rapidly you know faster than we've ever been able to do it which as you can see costs problem yeah this one's also dismounting causing you to have infinite jumps I failed to see the problem good places to be there's some times where you get a bug and you're just like no I don't I you're like okay yes I need to report it but you're like but it's like I remember I'm for this one when we were making legendary armor should I should I talk about the that's too late now okay yeah whatever so with all armor sets you know we have to we have to worry about ESRB concerns and what was yeah we just can't show that yeah that was only on you right now of course so what was happening is that we test which we die and we do specific things to make sure that we don't like go outside of Ras RV so what happened with this one was that I was dying the the male heavy armor it's only specific to him a male heavy armor by the way and you could die the pants to be nude toned so it looked like the surah the Norn and the like human safari men all weren't wearing pants so you'd to see you late it's like crazy heavy armor and then their butt cheeks fantastic and I felt so sad riding up that bug because I was like but can't we just okay that's awful okay so yeah we can't show okay so what happened yeah what happened here that's no you you'd know more about that that's actually that's pretty simple in this case what ended up happening is the when you go onto a mount you've noticed the camera adjusts to see that's actually moving into a separate camera at that point in this case it just didn't attach it like maybe the camera attaches to your mount as an agent and this time it just didn't really care achter after disc after dismounting so we're done here yeah I'm done and then but they're too bad but then once you're dead it runs a refresh on that camera and reattached it and like going off what I said before like you know when we break new ground it will surface and you issues mounts where the biggest ground we've kind of like the biggest undertaking that we've really done and as you can see it there come out of bedroom how terrifying was that it it's more fun than anything else like I mean Joel has the best stories more than likely good Oh Joel always had such good but yeah like at least when I was working on the road beetle like so many of you camera things came into play and like so many do you like physics things because it's just like oh this dude goes egregiously fast now you know and it's the the game just breaks in so many new ways when it comes to mounts and I don't know I love it it's it's great you look like you've got a new story yes I did um I don't know if Joel's short yet has he shown you the rapid fire has he shown the rapid fire raptor yet I don't think so so um during design yeah so yeah what are we working amounts we they're when we were still working on the mount dismount button the mounted or when you created the mount it obviously creates a existence of the be agent that you hate well when you were dismount we hadn't yet set it up to where it wouldn't be spawn the instance of that agent like 40 Raptors yes so just so you could spam the dismount keen it with launch Raptors Raptor gun oh and another version of that when he hadn't quite gotten to work yet it wouldn't it wouldn't dismount you but would keep it in the right place so you'd some of the nuts some of them that you try to bream out and it would some of the next Raptor on top of the previous one yeah giant stack of 40 Raptors on top of one another but you are still at the top and controlling them so the holsters that was my favorite yeah we showed that on a previous kill chat and it was just like this giant stock Raptors lurching around them in half that was one of my favorites that was awesome yeah yeah I don't know how or what the justification would be but I wish we'd figured out a way like huge stack of Raptors that was that was fantastic okay what's this one oh yeah in this one it's uh so there was a kill field around the around the boss when so players couldn't get in after the boss had already started um but it just so happened that it also didn't remove your gliding so you just down yeah you just kind of hang in there that's awful okay we just couldn't bear to see dead Ferren so that's actually lets me but that's you yes regards a char um so if Ferren was Ferren would help you out with squashing the bugs if you just let him do his thing he would get to the last one never squash it and just turn sideways and just start jumping up and down oh you tried I mean it's the most we can hope for okay so this is something that we've talked about before where different types of enemies will also fight each other Nathan you should totally talk about yes please um so this one is a is the this the shark is a just an event in the in Episode three tomato corner yeah domain of corner and uh and essentially you could just kite him throughout the entire map and he wasn't the same team as everybody else so he just fights two for you which was really convenient like it and I had to watch it a couple times I was like are they fighting each day uh I mean it's what we all hope for right so how did you how did you fix that you just put him on a shorter leash or be like okay now he fights no other enemy um I mean it would just be too if I understand correctly to just swap the team to be on the right the right team yeah okay so they wouldn't fight each other I love Detroit like come back no you get back here comes Bobby slow-mo by the way just in case you're wondering he just takes that shot right upside the head this guy's like okay that's like this tiny tiny dude like you get back here Oh [Laughter] God and then levitating Choya is happening it's an issue with our navmesh um where they just it it thinks that's where the ground is yeah basically the way like a little little peek under the curtains the way that our engine works is that we have this blanket and or like look it of paint basically for lack of better terms that we dumped over the map I know or lay over the map that has all non player agents like NPCs and like also enemies and things like in many pets yeah all those things adhere to that like blanket on where they can and cannot move okay so what's happening here is basically that the navmesh didn't get built correctly in that area so enemies would be able to like basically run fly over things and what happens sometimes is that the character is will be on one part of navmesh and then there'll be a breakage but they can basically like ET fly off you know then just keep on running within that specific lake path and that's that's pretty much what's happening here follow your dreams locating Choya because if you notice like the other people around him are like sit down like they're stuck to the navmesh so this is just he walked off and it's not flying off into sunset just kept going mm-hmm okay and and then there's these yeah it's it's a similar issue um they just can't figure out how to get around the walls my favorite part about this is if you zoom out a little bit there's an enormous opening about two inches to the left and there's like how did doors work so is it just like a pathing thing for this one i don't specifically know it could be a pathing issue it could be like there there could be a point within that path that's stuck inside the wall because what happens is like we'll build out this content at the same time that the map artists are working on their stuff so sometimes props will get put on top of specific points so what could be happening here is that that prop got placed on top of one of the points and they're hitting that point okay and it's like the next points on the other side of that wall but they're trying to follow the exact path they're just like I don't know how to get past like these like obstacle that one is just a like perfect storm kind of thing where it was just like the NPCs just met up in just such a funny way and also that camera angle too right yeah I mean those types of things can just happen um while you're you know it's like an NPCs in the wrong spot or something yeah oh my god though I'm pretty sure that was in real time - that's not slowed down yeah I think the the character was walking basically okay and just trying to interact yeah so what like systemically what's happening is that the one that's walking has the other character targeted and then when that happens your head has the ability yeah move around to face whatever it is so as you're walking closer your heads turning further and further perfect yeah all right well do you guys want to answer a couple of questions I think we have some time it's just about one o'clock we have we can go a little bit longer if you guys yeah we have a little bit of time so I will kind of let you guys look at those I did see at the beginning that apparently sunday is international software testers day so everybody get a cake Oh something so alright guys oh I can so the for the how the expansion stuff is so the question is from immortality X how is it how is it to test expansion stuff and then play it release is it still fun and does it provide an advantage it definitely does provide an advantage because you know exactly like the content that you're going to be working on like I can go through a lot of the content with my eyes closed it probably feels really smooth - yeah and I mean it does take away a little bit of luster like cuz you know you know what's coming you know how things work I'll you seen behind the curtain but it's still it's quite fun to like be able to like while I've been here I've actually gotten a lot better at just playing Guild Wars because there's some times where it's just like they're like we need you to do this naturally and you're like but this is a really hard fight it's like oh well you got to figure out how to play the game at speed yeah like with the hearts and thorns one oh my god he put me on a on a path back when I was a tester and it was to run through hearts and minds the final instance of challenge mode and do it naturally you're mean because we had to get done because we were going over the the content um when we were doing the the reworks okay and so I had to learn how to fight that that fight in childhood with five other people without any I knew what you meant I had a pet okay people to Nick yes and I had to figure out like what were the you know what were the meta builds I am here like how to do damage you know and how to avoid all the stuff on the ground that is some in the deep end stuff right there yeah I cannot do that really easily to okay but at the same time though it like it does get repetitive because we do have to play these things throughout multiple iterations so like you know I it's also our jobs to know the content like the back of our hands so when we enter live its the enjoyment that I personally get out of it at least now isn't me playing the game but it's seeing all this play the game and how they react to it and if they enjoy it or if they don't enjoy it and if they don't enjoy it why do they why do they not enjoy it and how can we better serve those wants to shooter you know how can we process a feedback yeah but that's a good thing to watch yeah cuz at a certain point like playing through the content least once again for me this might be subjective but it can get like okay the game is out now I'm just gonna run through it like I did wait yeah every other time yep yeah that's awesome what else I'm letting you guys decide I'm not directing you so said you like Howell you're like Nathan me Travis sure which one what test was it ArenaNet yeah you can still talk about like the fair yeah so within our game we've got some external tools I guess internal tools for testing mainly what we use is slash these things called slash documents which is basically what we do is we put in a console command into the chat box in the game and it will cause specific things to happen on the backend code side depending on what it is so for example like I don't want to die so I'm gonna like type slash dot nd which means notice yes so my character will never drop below one health or I want to be invisible and vis will make your character invisible let me just run through the map without getting aggro though yeah one thing I didn't realize before I was testing here is because I had Manchester somewhere else I was like do I have to make a level 80 character before I start testing and so we have commands for things like that - yeah turn you into a level 80 character yeah and we're constantly iterating on these commands like for that it's Slashdot Omega reset and as like mounts came out like we've we made us level you can unlock mounts and it's like a couple weeks ago I got the feedback like hey can we get come in or tags on those things so we do is then we'll just go into the engine italy futz with it and it's like okay now when you use it again you'll get come here tiger automatically so it's just okay like all of these things are getting constantly iterated upon um we have this thing called slash out examine which you know you used to be massive so we found ways to cool it down to like the most important information or we have this thing called /t we'll just lashed out teleport it used to be that you'd have to put in the exact string of like a like a name of something well we don't know that yeah and it was very tedious and painful like y'all know that but yeah so what we did was I think it was Ben Newell um one of our early programmers he changed the command so that way does partial matching which is basically like if there are two thousand things and you type in like a part of a word it's gonna look at all those things and find whichever one has that specific subset of letters and then push out that one and if it's if it's more than that one it'll populate a list of like a at minimum five all rights alright most five and it's just like okay here are things that match that that like partial input and then you could choose which one of those comes narrow down yeah yeah and that like that change has helped us immensely I'm sure yes yeah it'll at least start narrowing it down well and you had mentioned like you're testing weapons and stuff and you can create items instead oh yeah - yeah I can create you can create all the items you can um detach your camera off your buddies - like look at the items a little bit closer um you can slow down the the the animations so that you can kind of like see how it is as somebody's like yeah backing slow down time on the map after you check to see if someone else is on the map okay see what else I'll go ahead and take from Rizal what he and I is a programmer do to make you a job easier from our point of view do you work here I think there are really three things that are probably the most important things you can do the first being to communicate that's probably the most important thing to do let QA know when you're testing the ground or when you've made a slight tweak do something sometimes those slight tweaks you don't think of them as anything big because you're looking at the the code at the time and um you you're like there's no way that anything could possibly happen with this and then you find out six months later that no one has their soul bound stuff correctly so yeah just communicating letting QA know when you're doing when you're doing something before and after before and after you've done something and to go off of that like I think the most important point but not not just for Q it I think every team in general is communication is like the most important thing because at least for us like we try our best to communicate to all the other departments and the more everybody knows the more empowered we are to make good decisions because we have all the breadth of knowledge that's necessary and like going off what Travis was saying you know like if I if I'm somebody that like fixed a bug right from one designer that fix a bug and it's like a C bug to fix this one thing so you bug by the way severity yes severity sorry yeah officer verities go DC be a and basically what it is is a is like this needs to get fixed no matter what V is like this needs to get shipped thanks before ship C is like this is bad but we can like we can still ship with it and then D is like if you want to fix it weird yeah yeah sometimes they can be seen this feedback sometimes there are real bugs but they're like small enough you'd never worry about them okay yeah but basically it's like I could fix AC bug in my like black box content right and I know it's gonna fix it here but what I might not be considering is the fact that that thing then changed to this thing that then go over here and that thing is now hasn't like a bee bug because I fixed this like C or D bug on an actor you know so it's just like oh over communication in my opinion is really important because it's like the more everybody knows the more everybody is educated on things that are going on yeah the more the more we can prevent issues from happening and the easier it'll be to tackle to something to happen and that severity the dcba yeah allows everyone to see that and make decisions on the use of their time because you only have so much time right right and okay what do we need to do with the amount of resources and the amount of time that we have so everyone and it's a communications tool yeah so everyone working on this stuff can see okay what what do I need to triage like you were talking about earlier yeah okay second C what is debug commands anything that allows your testers to save time helps so much because for instance if you have a long series of events that you need to test but the bug is always in the fifth event if there's no way to easily skip to that fifth event and have all the actors in the right place and all the right triggers set you have to manually go through all of those and it doesn't seem like a lot but 10 minutes every time you're running something when you got to run it here 20 or 30 times adds up super fast so any kind of debug code and what's really important is that it can't just like skip to it it has to put everything in the right state for it to or it has to call you like it has to act like you've played through that content right because otherwise you could end up getting false positives there and then probably the the third most important thing after communication and debug commands is and that goes my train of thought we might come back to so we have time for each of you to do one more question okay cool um I so Charlie's asking what advice do you have for someone who's new in QA and wants to yeah so my my thing is don't be afraid to ask questions like when I first started working here I was very afraid to ask questions because I didn't want to like take up somebody's time but you have to realize that your time is also valuable and and the more questions you ask the extra the faster you will learn about how something works because that's really what it comes down to is you need to you need to learn how the game works internally in order to be able to like point out bugs even before they might happen so if you don't ask questions you won't be able to get that um yeah also in my opinion following a proper formatting for things like bug writing is really useful should we finish up this question first yeah I'd say this is this is like definitely useful for queue everything it's also useful for kind of like breaking into the industry and career in general is fail fast and fail forward like do not be afraid to make mistakes you know because a lot of what we do is make mistakes you know a lot like and not just playing the game to find bugs but like like I said like we we missed that that memory thing in episode 3 that was a mistake like granted it was it would have been a super difficult thing to find and we didn't have a tool set but it was still a mistake that was made and we failed so we learned from it and we're gonna move forward with it yeah and really that's kind of what it boils down to and being QA is just like learn from your mistakes and a mistakes that others make be communicative and just kind of have fun with it because at the end of the day like if you don't love your job what like seriously like like to be a QA you need to enjoy doing this specific subset of things because if you don't you're not gonna have a good time yeah it's just plotting in here doing the same thing over and over and yeah and that's like you said it's you're putting a puzzle together and you're loving it yeah and that makes a huge difference yeah I think that oh gosh I'd say probably just get creative I mean oh yeah it's I mean it's it's not just the things you never think of it's the thing is you don't even think to think of thinking of mean it's you there's never a wrong thing to try I mean the worst case that happens is nothing happens and then you've done your job yeah I mean it's I mean as long as you're not it's you do need to be kind of acceptable to some level of monotony if you are gonna have to run through the same content dozens hundreds or even thousands of times and you have to be aware of that that's something that's gonna be part of it and you don't necessarily have to love that yeah but you should be aware that you're going to have to do it um but it's it's definitely something where the more creative you can be with it the easier that kind of thing is and the more you get to try out weird things that you wouldn't ever think about trying oh yeah and dev environment is a great place to do it because there's no consequences so the third thing as programmer designer or whatever to help QA is life is know how it can go wrong whenever you're doing something get like at least have an idea of some of the things that could cause a failure on it okay that way you at least has a vague idea of where to start like for instance if someone were to tell me hey I made some changes to fractals about how to load in surprise and I'm like okay what am I looking for like did you change loading times so like should it be faster to load in um is it a method and how you load in how you select different things to load in because it's not having an idea of where to start I mean there's a I mean technically there's only one way to loan into a fractal but if I'm not sure if I know what I'm supposed to be looking for I don't know if I failed this change it's just I made a change to this thing find it good luck it's really nice when they say something like I I changed this and I'm worried this might happen like oh I can go look at that for you I can go I can go like you know look at it like from a holistic point of view where like from all sides and say okay I feel comfortable with it I don't think it's gonna break right rather than just being like yeah the change you made looks good you don't mean like I'm not sure what to look for so yeah yeah I totally agree with that all right Nick so I'm gonna butcher this name probably so I do apologize Dena read Tibet asked without spoilers can you give concrete examples of what you've done to keep things coordinated and preserve continuity ask you this is it okay to talk about things that we ended up not doing so I'm mostly talking that's like saying I made a change in this fractal remember that I think we decided not to if it's what I'm thinking of okay sorry no it's okay I happen to have already had okay yeah no that's that's good to know sorry sorry but basically things that we do to make sure that we keep things coordinated and preserve continuity is really looking at the time box that we have and figuring out what we can best do to get a return on investment for the time allotted like for example I use the alphas a lot you know like because we want their feedback on things and to preserve continuity like these are people our office or people that have played the game like more more than anybody else honestly yeah and they love it and they love everything that goes into it and they want to be a part of it and they want to help like you know make it the best that it can be yeah definitely so of course like having them participate in our play tests and having play tests that are tailored for them and kind of having these conversations about what's going on with like the story and the content that were presenting and the political character presenting helps to really preserve the continuity and make sure that everything that the feedback that we're getting is is wholesome you know it's things that people who play our game enjoy whilst also making sure that we are always being cognizant of of making sure that we don't care to a specific sub-sect of everything that's also important yeah yes kind of dangerous to do yeah that's that's a good answer for this question and just concrete as we can get yeah that makes sense all right and Travis huh hmm last question pressures on um so I will actually go ahead and answer two of these at the same time ooh you rebel yes okay so uh as Nick mentioned I am going to slaughter this name yeah terribly but Eduardo ped Ray pedrera asks why can't mini pets be as fast as ranger pets minions is there a I too sensitive to tweak and Matt sessions asks what do you do to help keep your day fresh while playing the same content over and over so by keeping my day fresh I believe I have an answer for the question about any pets okay awesome so part about keeping your day fresh is all about finding new ways to do things you've already done a hundred times um so this is part of that whole being creative thing that I mentioned earlier so a lot of times you can do things like maybe you don't like Nick also mentioned earlier about us having certain commands that don't exist in a live environment one of these is a command allows us to change the species of our character so all right perhaps you don't feel like running the event as a char today you would rather run the event as a whale or maybe you would rather run the array be you would perhaps rather do the heart 200 times larger as a character than you actually are mm-hmm or maybe 10 percent of the size of a normal character so okay oh there's a time for that that's those exist um but I mean we can go well beyond what the tonics can do in a development environment um so one of the things that you can so because of that the you know we chose of changing my size and doing all kinds of other things you notice specifically the difference between say when an orang moves and when an asura moves an orang takes very large deliberate steps and a Asura takes lots of tiny frantic ones but they actually move at the same speed in the game okay um it is important very yes they move at the same speed it's just their animations are increased so if I understand it correctly the reason that many pets can't be as fast as mmm Ranger pets or minions is because they're very very very small so their legs are very short so as a result a ranger pet will look normal as a normal gait running but if you were taking mini pet and make it move that fast it would just be zipping along and a super fast rate those legs are like hummingbird wings right exactly it looks terrible okay so that's I could be wrong but that's my understanding and how I've seen it work so it's it's just a presentation thing we don't want the mini pets to look like their poor little hearts are going to explode catch up to your character so I'm pretty sure we've we've written a bug's boat many pets going too fast we have this I'm too slow yes low end too fast at the same time everything is wrong that explains what like you know your character will stop and like five seconds later your mini pet comes for me put me in your backpack something anything okay that makes some sense all right we are yeah it's like a quarter after one so it's we're and I know you guys have to go back to work so thank you guys very very much for helping us kind of uncover I mean I talked to Nick about this a couple weeks ago and it was like you know I know you guys find bug and you test the game and that's kind of it why don't we kind of look behind the curtain and let the players see how you guys do your work so I really appreciate you guys hanging out with us and letting us chat about QA for about an hour so I really appreciate you guys taking some time thank you so thank you I will you guys get to work and I will see you guys on the next skilled chat thank you very much y'all have a wonderful weekend have a great weekend you get some rest
- The episode title may be a pun on the 2010 film The A-Team.