What do you think they are doing? Tp is shutdown while they remove the items and people involved and patch the bug. AGS is doing the right thing that everyone gets so upset about it would be 10x worse if they didn’t.
Your response is fair.
Unsolvable is a weird word. It implies that there is one bug that is not solvable. The answer is no, this bug is not unsolvable. If it were, then the game would be over. I have implied and will continue to that there will continue to be exploits that people will attempt and eventually find. There is no lock that man has made that can not be broken. That going after exploits is not something that is done once, then it never occurs again. Never underestimate the ability of people to think of a condition that you did not.
Architecturally, yes, this game is interesting in design. From what it looks like, there are numerous ‘instanced’ type places. These instanced typed places would be expeditions, OPR, etc. I also think that they instance the TP, housing, sheds, and crafting tables. Somehow, the shed has to interact with the TP, the shed has to interact with the crafting table. Housing has to interact with the shed. This, to me, and this is 100% fully opinion, is probably where some of the issues are occurring. If you look at how they are handling inventory, it looks like the table holding the values consists of “ItemID, ItemCategory, ItemQty.” If you notice how they do the display of your inventory, it is a constant sort. Again, this is how I think they have done it, I do not know I think they are grabbing the info, sorting, displaying it. Then they are getting info from the client, the client is moving fast and sending data back to the server that something has happened to the inventory. As data i/o is expensive, the write to the server that something has happened with your inventory is queued. It is not written to disk on every transaction. Something is happening in that queue before it is written. This is where I think the issue is occurring.
I think the latest exploit has to do with how they are now buffering keyboard commands. If you notice in the past, there was no keyboard queuing. You could hit left click 10 times, hit 1, then left click 2 times. Well, if the left click 10 times was NOT finished, the 1 key was not queued up. The 1 key had to wait until the left click was finished. Now, we have the ability to queue commands. This is now a new buffer that was not there in the past. This is where I think they will start to find more and more issues.
Now, before people scream that I am an AGS apologist. For some of this stuff, yeah, this shit be hard to solve! For some of this other stuff, yeah . . . rookie mistakes. I personally believe that AGS has tried to do something that is seriously biting them in the ass. They are going for SPEED. Pushing new content every 30 days is VERY hard to do. It leaves a very short window for development (code, graphics, music, writing, etc.) as well as a very short window for testing. Because of that speed, we, as players, are paying the price. I would much prefer they spend 60 days with content releases and there be no issues, than what we have now.
I suspect you’re pretty much spot on. My own guess is that this game has a microservice architecture, which fits well with running on Amazon Web Services and is why they were able to add loads of new servers on demand (as opposed to other games where they had to wait for deliveries of new physical hardware to data centres to do so). Having lots of separate services doing different jobs and communicating back and forth has the disadvantage of having lots of opportunities for errors in orchestrating them, which is what exploiters look for.
To repeat what I said before: this shit is hard, which doesn’t excuse AGS because they need to produce a game that works and if this is the path they’ve chosen it’s on them to make it work. However, I have some sympathy with them because I’ve had to deal with similar problems myself, and it really isn’t as simple as some of the armchair architects here seem to think.
It’s possible, because they have programists repleaced with the trolls, not all ofc, but there is someone without brain who can’t do the most important things as much simple as possible to avoid those situations. For example, this guy is doing it that way. We want to achieve as a result number 2 and the only way should be 1 + 1 = 2, no other posibilities, but this dude is allowing for 4-2=2 for 3-1=2 then something what was letting/using 4-2=2 is modiffied by another person and do not to remove that posibility from the core place which was allowing for it. With trade should be just 1 simple line anything else ended as a return;
The fact they cant simply run a check on each persons inventory to see if items magically appear without some type of legitimate transaction shouldn’t be possible. But this is Relentless studios were talking about.
Spaghetti code definition:
Often during the development of software there is a loss of intellectual continuity. People quit or are fired and their institutional knowledge is lost. As these individuals are replaced they often are not afforded the opportunity to master the code base and are forced to do changes ASAP. This leads to sub-optimal programmatic changes or as I like to say “hack jobs”. Eventually these get layered on top of each other making maintenance difficult and new modifications challenging. Everything becomes a work around and no one develops a holistic understanding of the modules.
Voila . . . spagetti
they have been working on this game since 2016… I could make some pretty damn good spaghetti from scratch in 6+ years.
One way that games combat magically appearing items is individual item identifiers and timestamps/item creation method data, but with the vast amounts of crap items this game generates that would have a very high resource cost and with stacks of 10k items it would be too cumbersome.
They really need a single core system that handles all item and gold interactions across every single feature in the game and that can be called by those modules, sent the relevant item information and from there, it handles all transactions securely on the server side, only ever creating an item in it’s new place after the existing one is destroyed in it’s old place.
With the way a lot of the duping in this game has happened, it seems like each feature/module/part of the game is doing this independently and this creates vast numbers of potential duping issues, since an error in the code of any one of those (like a missed function call to delete that original item) creates a duplicate item. It also means in a lot of cases that it is either doing it on the client side or doing it on the server end while overly trusting the client inputs and without error checks that 1 item was destroyed for every 1 created.
We have too many individual systems doing the same thing, when one would do it better and you could quickly and easily pin down duping issues because you aren’t having to search the entire world to find the causes.
This is one of the main advantages of object oriented programming methods. Instead of having a different “banker” function all over the place handling transactions, and causing potential issues in many places, you could just write one that does them all and call it from everywhere and you only have to look at one function to find your culprit.
In the worst case scenario if you don’t find the dupe issue there, you could search for every call to that one function to see who/what sent it bad data.
I’ve always thought of spaghetti code as an encapsulation issue as well. All the steps you’ve listed leading to classes etc that have too much intertwined functionality. Hence the seemingly small changes made in one spot having unintended cascading effects.
The armchair people think that this is easy, but have no concept because they have never had to do it. They, because they stayed at a Holiday Inn Express the night before, think that they know more than those that have focused their studies, their livelihood on these things.
Now from a nerd perspective, I still think their unit tests are weak. Whoever is writing the tests is not putting in enough permutations on possible negative tests. It looks like they are testing for positives mainly and some negative. This would make sense if their code was really OOP and compartmentalized!
OOP doesn’t catch network issues
I would bet . . . that their code is OOP. I would bet that they are doing exactly what you are stating. I would also bet that the issues are the handoffs between the client, the server, and the queuing systems.
The key input queuing seems like a very good guess, it makes sense. All the architectural and QA suggestions on this thread make sense. It makes you wonder just how much of a beast this code base is. 6 years with a large every-changing dev team is a very long time. Have never worked on anything this scale, tough to imagine how tangled a knot you could create under those circumstances.
This is a really good and understandable possible explanation. Thank you.
I don’t think they have client authority on anything important in the game. They have built a distributed system without a single source of truth for the data of what items you own. This is why we see duping bugs.
So when you put an item into one area of the game from another there may be some service that that has to communicate that the item is leaving service A and now in service B. Client authority is what caused the housing dupes, problems between the communication where both services think they have the item looks like they caused others. This one clearly the input queue they have built has information other than the keystroke stored within the message.
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.