I’m sure anyone who has spent a lot of time working on a big project knows what this is like, and if they’re reading this, they’re already sagely nodding / snickering / cringing from PTSD due to the title of this post.
While EXU2 concerns itself with Hell quite a bit considering its setting, there are still some kinds of hell that even I, a card-carrying Hell Enthusiast, am not comfortable with.
First, let’s look at a brief history of 10 of EXU2’s most notable Development Hell cycles. There have been many more. This is mostly because EXU2 began as a sort of ~Babbys First Mod~ experience for me in 2004, and served as my introduction to scripting. I only tinkered with various classes’ default properties for the first couple of years of development, but eventually started actually coding stuff, much to Dean’s (and my future-self’s) dismay.
You gotta start somewhere, of course, but wading through your infantile shit as a more experienced developer is not fun, especially when mistakes made years in the past come back to fuck with you when you REALLY don’t have the patience to deal with them. And especially when you have nearly 2000 classes to handle. But that’s what Development Hell is all about!
[1.] The Not-So-Great Schism:
Originally, EXU.u was just a small collection of scripts with only a handful of external assets. I basically imported all the custom sounds and textures I needed into EXU.Stuff.<asset>, because CLEARLY there was no way I’d ever have more than maybe a dozen assets total. I have over 1,700 textures and 200+ custom sounds in use now. WHOOPS.
At some point I realized I needed to split EXU.u into three files: EXU-Sounds.u, EXU-Textures.u, EXU.u. This was done a really long time ago; I can’t remember if it was before Demo 2 or 1. Anyway, I already had a good hundred or so assets to manage by the time this went down, which meant I had to fix A LOT of maps that had references to EXU.Stuff.* instead of the actual assets. Cue countless hours spent fixing actors by hand in UED.
[2.] The Not-So-Great Schism, Again:
Yes, again. I created EXU2BI.u to handle EXU2: Batshit Insane-specific assets for the campaign, putting them in their own package. More fun times spent fixing level asset references.
[3.] The Accessed None Removal Spree:
Accessed Nones are small, stupid little errors that get spammed all over the log when your code sucks balls. (My code sucks balls.) I had tons and tons of functions that spawned an actor and then applied changes to said actor without checking to see if that actor actually existed in the level. Assigning variables to “none” = Accessed None. I had a lot of this.
[4.] The Map 15 Clusterfuck:
Literally hours before Demo 3’s original deadline, Map 15 fucking exploded. This was also the final map in the demo, so having severe problems in it was DEFINITELY not acceptable. Basically, the server would crash online (and sometimes even offline) with some error message referring to a pawn doing some dumb crap. I found find the pawn responsible, deleted it, rebuilt paths, and retested. Just when I thought everything was peachy, IT CRASHED AGAIN. I repeated the process many times. My blood pressure skyrocketed during this phase of debugging and I probably almost died from supercritical quantities of ragelets plowing through my neurons.
This was all caused by instantiation bugs brought about by copy/pasting pawns to duplicate them instead of using UED’s built-in duplicate function. I did this to save time since pasted pawns’ events/tags get retained, while duplicating resets them. Time-saving MY FUCKING ASS. I had to revert to an older version of the map and redo all my changes to get it stable again.
[5.] The Netcode Shitstorm of 2010:
At some point I learned that basically 90% of EXU2’s netcode was atrocious. I went on a massive optimizing spree a couple of weeks before launching Demo 4, a spree that soaked up probably 12 hours in total over the course of two days, and de-shittified almost EVERYTHING, making the game 1000x more playable in coop. Lots of testing, lots of keyboard hammering.
[6.] The MakeNoise() Fixing Assault:
Dean recently explained MakeNoise() to me. It’s an AI function and doesn’t have anything to do with making sound. I used to just slap that stuff all over the place without really knowing why because I saw it used in Epic’s code, which is where I learned all my bad habits through misinterpretation. Anyway, when debugging the source of some seriously bad and nonsensical lag, Dean discovered that the number you put in the parentheses (typically 1.0 or less) is a multiplier for the radius of the MakeNoise() call. I thought it was volume. Nope.
Putting in a REALLY BIG number, like, say, 100, means your MakeNoise() call is so massive it can span the entire map. Things like the LRPC would make such a loud noise that every pawn would “hear” it simultaneously. Sounds good, right? I mean when a huge artillery cannon goes off, you’d expect everything to acknowledge THAT, right? Of course! From a design perspective, this makes sense.
Technically, though, this means iterating through a HUUUUUGE radius and applying a CPU-taxing AI function to ALL the pawns within it. On large maps, this translates into SEVERE LAG every time you fire the gun – we’re talking slideshow speeds, here – because every single pawn will spend a moment trying to locate the player regardless of distance or visibility.
I also later learned that calling MakeNoise() if Instigator==None in a projectile wastes processing power, so I spent a good bit of time adding instigator checks to every MakeNoise() call in a projectile.
[7.] The SpawnWhenTriggered Netcode Debacle:
SpawnWhenTriggered is a wonderful, wonderful thing I designed for EXUScriptedPawns that makes level editing way easier in a lot of ways. It causes a pawn to be invisible until triggered, whereupon it then “spawns” into the map. Technically, though, it exists in the map even before it has been triggered – it is just dormant, invisible, and has no collision or awareness of its surroundings. Everything about this system worked just fine offline, with one exception: creatures that used overlay actors.
If a monster has some kind of special glowy skin or something, it spawns an overlay effect in PostBeginPlay(), which gets called as soon as the creature is spawned in the level (i.e. when the game first loads). The problem: SpawnWhenTriggered assignment happens AFTER PostBeginPlay(), so as a result, you’d have invisible pawns with VISIBLE overlay effects.
I fixed this by making a separate function that was only called in PostBeginPlay() IF the orders were not set to SpawnWhenTriggered, and it worked just fine. I called that same function in SpawnWhenTriggered while bringing the pawn out of stasis mode. But then we discovered one critical flaw: none of this shit worked online.
Long story short, Dean created a system that fixed it, but it required creating special overlay proxy classes to load the overlay effects. I had to go and modify EVERY pawn that had overlay effects to work with this system. It took me a few days to get them all.
[8.] The Decal Crisis:
EXU2 uses a really badass decal management system that Dean wrote. It keeps track of decal overlap, max decals in the map, and other cool stuff. At one point, this system totally fucked up and started crashing singleplayer. Of course, I only found this out because I was in the middle of a SP playtest, but finding and reporting/eliminating bugs is just one of the many things I do every day, so whatever. The crashing happened whenever a decal was removed from the map, either due to a timer-based expiration event or a “too many decals, removing old ones” sort of thing.
Fixing this involved a much more complicated and convoluted workaround than expected. I spent a couple of weeks focusing on MP gameplay development and balance testing instead of SP since that was hosed, but the Bawss Man eventually solved the problem. Then:
[9.] The Decal Crisis, Again:
Yes, again. A couple weeks after the first decal shitstorm, I noticed the decal manager had stopped working mid-way through my second attempt at replaying the EXU2 campaign with the latest features and balance. My MaxDecals of 1000 turned into infinity, and entire maps were plastered with blood, explosion marks, etc. While this looked hilarious, it was not appreciated by my old video card with only 128MB of video ram. The maps were virtually unplayable if I didn’t “killall decal” every so often.
Again, a convoluted debugging process ensued, but together we figured out what was going on: the DecalManager instance, which was always supposed to be EXUDecalManager0, was turning into multiple instances, sometimes as many as 100 or more. Dean thought it might be caused by a memory leak, and for a while we had to simply disable the decal management system. For a few dark days, it looked like we’d have to scrap the whole system due to technical instability, but we snatched victory from the jaws of defeat yet again and found a workaround that’s been functioning fine ever since. Huzzah! At long last, all decal problems had been permanently resolved!
[10.] The Magic Number 1116:
Finally: this is the latest bullshit engine limitation Dean and I just figured out over the weekend! It started with decals. Yes, again. But it wasn’t caused by them; rather, the decals turned out to be a symptom of a much deeper, much stupider problem.
First of all, I got a new video card a while ago because I retardedly managed to fry my old one by forgetting to plug the GPU fan back in after cleaning the case out. RIP, faithful old 7600GT. Anyway, I decided that with eight times as much video RAM as before, I could afford to crank my MaxDecals up from 1000 to 2000 or so. It was after doing this that I started to experience the most fantastic bullshit of all: a straight-up crash to desktop with no error message and nothing in the logs to indicate what the fuck had happened. It took me a few hours to even figure out the crash was decal related; I ended up able to save when I did a “killall decal” first.
I narrowed down the problem to setting my MaxDecals higher than what it had been in a previous save – or at least I thought I did. I knew that if I set my MaxDecals above 1000, then loaded a save where the MaxDecals had previously been only 1000, it would crash when I saved if I hit 2000 or 1500 decals. Setting MaxDecals back down to 1000 fixed it, thank fuck, so I could continue playtesting. And I did. Until Map 19, Fuckercleave Gorge, which happens to be the biggest map in the game by far – not just due to its enormous size, but also from a technical perspective; it’s absolutely packed with scripting.
About 30 minutes in to my playtest, I saved the game in a bunker before engaging a big swarm of enemies… where I proceeded to crash to desktop. AGAIN. What the fuck? My maxdecals were definitely not above 1000, and I’d been able to save on this map before just fine. And that was way before I made some major optimizations to the level! Some serious bullshit was going on. I documented it in as much detail as possible and even quarantined a save file just before the bullshit kicked in.
We spent days just trying to isolate what classes caused the crash, because we knew at this point that it wasn’t necessarily caused by decals. Dean spent hours trying to figure out which classes worked and which didn’t though some Bawss Magic process he invented, but didn’t come up with anything conclusive. We still didn’t have an answer to the maxdecals thing, either. A memory leak was looking increasingly likely.
Then Dean started testing stuff outside of EXU2 and was able to reproduce the crash WITHOUT EXU2 assets. You know what this means? ENGINE FUCKERY. After a few harrowing weeks of trying to find out what the hell the deal was, he eventually came up with a delightful theory: the engine cannot handle saving a linked list with EXACTLY 1116 items (or more) in it. What the shit?
It was true, too. If you saved the game with exactly 1115 decals, it would save. If you added ONE MORE, bam, crash to desktop upon saving.
EXU’s decal management system uses a decal list to keep track of decals in order to remove old ones when new ones are spawned and prevent MaxDecals from being exceeded, which is why decals could cause the crash. But why was Map 19 crashing? There was no fucking way I had over 1000 decals in there! God fucking damn it, this fucking engine, FUCK.
I wrote an ActorDumpster script which, when summoned, would count every single actor in a map and output it to the log file. I also had it count up how many of those actors were Decorations, how many were Inventory items, Pawns, Navigation Points, etc.
Navigation Points. These all get added to a Navigation Point list, just like Pawns get added to a Pawn List and Inventory gets added to its own list as well. When I looked at the dumpster output for Map 19, I had over 1500 fucking Navigation Points. (Tons of EXUSpawnpoints because I built Map 19 before GenericSpawner existed.) TERRIFIC.
We confirmed Dean’s theory by creating a non-pathed Spawnpoint, ThingFactory, CreatureFactory, and MultiCreatureFactory replacement meant to be used solely in emergencies (like Map 19). I swapped all the actors, rebuilt paths, checked the dumpster, and blam: just over 900 Navigation Points. AND THE MAP SAVED. HALLELUJIAH.
So, in the end, we spent countless hours trying to figure out what on earth our code / this engine was doing, only to eventually find out that UT just shits itself if you save a linked list with more than EXACTLY 1116 items in it. Why 1116? Who the fuck knows. I don’t even try to understand this stuff anymore. The Unreal Engine has figured out the most novel ways to crash, I swear. Of course, I’m the one who’s actually adding billions of actors to my maps, so I’m not totally innocent, but come on!
So anyway, since we can’t have linked lists greater than 1116, this means that MaxDecals has to be capped at 1000, or set up so that >1000 MaxDecals = infinite decals (no linked list) in order for saves to work. Dean may be able to create multiple lists on the fly whenever one list fills up with over 1000, which would enable us to have MaxDecals set to 2000 or some other number, but we’ll just have to see. I’m just glad that the fucking game actually works again…
…for now. >:E