How/Why duping works, and why it is so hard to correct

General Discussion General Discussion
#1

I wanted to help the community, and maybe some of the devs, out by explaining how dupes happen and why they are so complicated to find and fix.

If you are looking for instructions, you won’t get those. Sorry, you are out of luck. This won’t teach you how to dupe items in the game.

Rather, I am a developer who works at the hardware level and I am going to teach you why the software can sometimes copy things. This is fairly specialized knowledge, so it may take a second, but at the end hopefully everyone can group together with some knowledge about how this happens and how it can be tracked/fixed.

First we need to go over how software actually modifies values. A computer can never work with a value in memory directly. This is the root of the problem.

Let me explain how a computer actually works with values using an example of gold. Let’s say you have 100 gold and that is represented by a memory address with the value of 100 in it. Now if the game wants to subtract 1 gold from you the following process must occur.

The value in the memory address is COPIED into a register on the processor.
The processor then subtracts 1 from the value in the register resulting in 99 IN THE REGISTER.
The computer then COPIES the value from the register back into the memory address.

This all happens blazingly fast, often in less than 1 millionth of a second, but it does take time. And, the game is not a single thread. So let’s consider the following.

A thread in the game wants to subtract 1 gold from your total, and another thread wants to add 1 gold to your total AT THE EXACT SAME TIME.

In theory 100 - 1 + 1 = 100 Right?

So thread “A” COPIES the value of 100 from the memory location with your total into a register.
Right after that thread “B” COPIES the value of 100 from the memory location with your total into a different register.
Now thread “A” subtracts 1 from the register it is working with resulting in a value of 99 in that register.
Thread “B” adds 1 to the register it is working with resulting in a value of 101 in that different register.
Remember both threads COPIED the original value of 100.

Thread “A” finishes its calculations and then COPIES the value of 99 back into the memory location for your total. Your total is now 99.
But, then Thread “B” finishes its calculations and then COPIES the value of 101 back into the memory location for your total.

COPIES always OVERWRITE any values already there…

So now your total is 101 in memory. 100 - 1 + 1 = 101?

Lets ponder on how this works in game terms. Lets say you open a trade with a friend. you have 100 gold and you put a trade of 1 gold in the trade window to give that to your friend. You then click confirm AND cancel on the trade window and those go to the server at the same time.

If things are unlucky then one thread will handle the trade giving 1 gold to your friend and leaving you with 99 THEN the cancel will get handled returning your gold value to 100. You have 100 gold AND your friend has 1 gold.

This all happens because memory can not be worked with directly and it is possible for 2 different parts of the game code to be working with the memory location at the same time.

Now there are ways to prevent this, none of them are easy. This is REALLY REALLY HARD and why really good database developers are worth every cent.

The scary part is this is a potential problem every time you modify the database. What happens at a crafting table, when you upgrade a city, when you trade, when you put something on the ah, … This is why we are having a little pause here as they look.

It gets even more complicated when we consider there are likely thousands of cores running each with their own prefetch and cache.

So this has slipped through. We all need to give a little time for a fix to be implemented and understand it isn’t easy. However, there is some good news.

Dupes of this sort always have 2 modifications of the value back to back. IF the database of your gold logs all changes that happen due to trading windows then the log will show a change from 100 to 99 then to 101 all within a fraction of a second. In theory this log could be searched for this pattern and 100% of duplications can be found. So this issue may be 100% traceable.

Don’t Dupe, you DO leave footprints.

If anyone has any questions, feel free to ask and I will try and clarify however I can.

Shameless plug here, if AGS needs expertise to find this stuff I will gladly donate some time because I love the game, and If you need a lot of time accommodations can be made.

-Z

5 Likes
#2

This makes sense in the hardware context you are coming from, specially if you do Assembler or such a super low level language.

On such levels as Databases, the operations are put into transactions. Which doesn’t seem to be the case here. You pack your critical operations into a transaction and if ALL steps inside it ain’t a “Success” the whole transaction is rolled back and none of the operations get committed to the database. Almost every single database out there supports transactions nowadays and have the tools to avoid this kind of issues from happening. And if they don’t, the first step is always to look for 3rd party support or use another database.

We actually use them a lot at work and are specially careful when implementing them for processes that manage integral data for the systems we make.

1 Like
#3

As I said, there are ways to prevent this. The issue is, obviously they didn’t.

This isn’t that straightforward. What defines success or fail? there have to be multiple handlers for the throughput they have. AND, almost all of those fixes open up other windows for problems or slow things down significantly. Game databases are kinda wonky because they don’t operate like most real life databases. We are not dealing with failures to modify the database, on the contrary we are dealing with too many successes.

I have no doubt they will eventually fix it, but they didn’t protect against this in the first place and doing it retroactively is not going to be easy.

At a very minimum a client should not be able to call the handlers at anywhere near the speed it currently can. There should be an input timing filter on the server side receiving and processing data. I can only imagine the issues that might show up when people actually have outside processes changing the data stream not just a lag switch.

#4

That’s up to the devs. Transactions are per definition atomic operations, so for example in this case, you would pack in there “remove money from A”, “save money to B”. Both operations get pseudo committed. If both are successful, the operation is committed for real to the database. After the transaction got marked as successful it doesn’t matter if the client drops the connection, crashes, lags. The transaction will be committed and both entities will be updated correctly. If they use a lag switch somewhere in the middle, one of the operations will failed and the whole transaction will be rolled back. These things are used in most if not all systems involving money and sensitive data, they work and basically don’t fail.

The issue here, is that we can only speculate what they did or didn’t do and have no idea what they are really using.

Databases like MongoDB can handle what AGS do and more, its absolutely insane how fast it is and Graph DBs are even faster. Volume of data and number of operations ain’t really an issue with these modern DBs.

From all the issues I’ve seen in the past couple weeks, I just think they don’t have enough senior devs in there and most of the work is being done by junior devs (Not sanitized html user input, WHAT?!)

I’m not sure I want to know either, hopefully no CheatEngine experts start “testing” stuff with the game and on the network side… I seriously hope the packets being sent/received are AT LEAST encrypted.

#5

This is very good from your perspective as an hardware level programmer, however for their cases, all actual databases have solved this problem DECADES ago (PostgreSQL, MongoDB, ES, Cassandra, etc…) all support atomic operations and ACID transactions, no work from AGS is required here.

Problem is the game is client-sided and no databases is involved other than sync-ing

#6

It’s an issue between the integration of the client and server most likely.

Most of them involve using a lag switch from what i’ve read. It’s sad that people are abusing exploits.

#7

Not necessarily, just playing devils advocate here: But it could also be that the different things in game (Tradepost, Storage, Characters, etc) are served on different Microservices (For what AWS is pretty good at) and they implemented some kind of distributed transaction on their backends, which is failing to sync all services databases.

#8

The item/inventory database is not client side. Very little of the game is actually client side.

There is zero evidence that it is client side and you can actually see it fail to download the info from the database if you open your inventory the first time after logging in and have any kind of traffic slowdown.

1 Like
#9

of course the database is not client side, changing files in your local computer would result in duping in that case, there’s just synchronization happening after each in-game request/transaction (and at the beginning to retrieve the inventory from the remote copy), allowing the client to send garbage to server without decents checks, and THAT is a client-authoritative pattern

#10

Not at all, check my other answer to your post :slight_smile:

If we had a real client authoritative game, you would have seen already all kind of crazy stuff happening, from people editing the inventory, characters, weapons, etc using CheatEngine.

This actually happened in ESO in the early release days. PvP was almost impossible cause the skills were ALL done on the client (WHAT?!). There are some hilarious videos of all kind of absurd stuff happening at keep sieges.

1 Like
#11

This is not as black and white as you believe, a game is not fully client sided or fully server sided

In the case of new world, the problem is it is not completely server authoritative (and by all means IT SHOULD BE), as Kay mentioned in his previous post, server has to receive client confirmation in some cases to proceed (he said this referring to the hatchet bug), this is exactly what it means to have authority handled to client.

Those mistakes are allowing dupes and exploit to be discovered every day, in most game engine nowadays the networking stack is not even ALLOWING client to have any forms of authority (eg, disallow the use of network synchronizer and only allow RPCs (remote procedural calls) for inputs and intents of actions)

#12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.