Welcome back to the weekly VV Eekly Update, posted every VV Ednesday. It was a real honor when the people in charge decided that we would have a whole day every single week dedicated to Vyn and Verdan! I’m glad you’re here celebrating with me.
This week, we’re going to do a real deep dive into some terrible technical headaches. When I first decided on the general direction for Vyn and Verdan, I knew that this particular aspect would be a real challenge, but I naively sallied forth into the great unknown. Now, I emerge from the depths, bearing some tales to share with the tavern crowd… of dealing with NETWORKED MULTIPLAYER!!! DUM DUM DUMMM lightning crashes
Multiplayer
Networked multiplayer seems to be the great white whale of today’s indie devs. Godot recently hosted their 2024 community poll, and a whopping ~75% of ~10,000 respondents said that they wanted to use networking in some aspect and 37% said that they wanted to use real-time networking.
That’s a ton of people! However, while multiplayer may increase a game’s fun by ten-fold, multiplayer might also increases the difficulty of creating said game by hundred-fold.
A Single Bug
Let’s dive into a bug that I’ve been smashing my head on for a few months.
If you look closely, you can see the bug in this gif. The left side is the server hosting the game, and the right side is a client that’s connected to the server. They should be the same! When the fight starts, there’s a subtle difference between the two – some of the bees on the right side don’t rotate and get angry!
Here’s some simplified code for how this works.
enemy.gd
var active := false
var global_id := 0
var body
var behavior
# These get synced across clients after enemies spawn.
var syncing := ["active", "global_id", "body", "behavior"]
func init():
if is_server():
global_id = Globals.get_id()
set_up_body()
set_up_behavior()
func anger():
active = true
set_starting_state()room.gd
# Maps IDs to enemies for each client.
var enemies := {}
# Called when a body enters into the room.
func body_entered(body: Body):
if body.parent is Enemy:
var enemy := body.parent as Enemy
enemies[enemy.global_id] = enemy
func anger():
for enemy : Enemy in enemies:
enemy.anger()encounter.gd
func start_encounter():
room.anger()The whole thing starts by calling Encounter.start_encounter(). Take a look, and guess which line the bug is on. I’ll give you an imaginary cookie if you’re right.
This bug has been appearing in-game for months now. It’s not an absolutely terrible bug – it doesn’t happen all the time, and I can still test other aspects of the game while this is happening. Some play-testers don’t even notice when it happens, since it only happens sometimes on rooms with many enemies. The chaos camouflages the fact that some of the enemies aren’t really doing anything. However, it’s definitely game-breaking, so I decided to take some time to debug it.
The real difficulty with debugging this is that it only happens sometimes when hosting multiple clients in large encounters. I would attempt to reproduce this, and the bug would only appear on the third or fourth attempt. There would need to be at least forty enemies on the screen, and I would have to scroll through all the debug data to see what’s happening.
Eventually, I pinpointed this to the “anger” section. I added these logging statements.
enemy.gd
func anger():
Log.debug("Angering enemy " + name())
active = true
set_starting_state()room.gd
func body_entered(body: Body):
if body.parent is Enemy:
var enemy := body.parent as Enemy
Log.debug("Adding enemy " + body + " to room " + room)
enemies[enemy.global_id] = enemyThese statements showed that the enemies were being added to the room, but that some of them somehow weren’t being angered.
Adding enemy Enemy1 to room BossRoom
Adding enemy Enemy2 to room BossRoom
Adding enemy Enemy3 to room BossRoom
[...]
Adding enemy Enemy39 to room BossRoom
Adding enemy Enemy40 to room BossRoom
# Why are enemies 1 through 24 not being angered??
Angering enemy Enemy25
Angering enemy Enemy26
Angering enemy Enemy27
[...]
Angering enemy Enemy39
Angering enemy Enemy40If you want another chance to figure out which line needs fixing… here’s your last chance to scroll up and look! Take a good guess!
…
Here’s the broken line!
room.gd
func body_entered(body: Body):
if body.parent is Enemy:
var enemy := body.parent as Enemy
enemies[enemy.global_id] = enemy # <-- THIS LINE!The global ID is one of the attributes that get synced across clients after enemies spawn. However, it was possible that the body was being added to the room before the global ID was synced! Here’s some possible timelines – the only difference is that the last two lines of each are swapped in order.
A Good Timeline!
On the server:
Enemy30 added to the server. (Server global ID starts at 0)
Server sets Enemy30's global ID to 30.
Server adds Enemy30 to the room.
On the client:
Enemy30 added to the client. (Client global ID starts at 0)
Server tells client that Enemy30's global ID is 30!
Client adds Enemy30 to the room. Everything is good.A Bad Timeline :(
On the server:
Enemy30 added to the server. (Server global ID starts at 0)
Server sets Enemy30's global ID to 30.
Server adds Enemy30 to the room.
On the client:
Enemy30 added to the client. (Client global ID starts at 0)
Client adds Enemy30 to the room - but wait, global ID is still 0!
Server tells client that Enemy30's global ID is 30, but it's too late!This only happens sometimes for some of the enemies, since it depends on whether the client gets around to adding enemies to the room before the global ID is synced. Usually, the global ID is synced quickly, but it sometimes takes more time if there are more enemies in a single room. This explains why it only happens sometimes in large rooms!
The fix here is to wait for the global ID to be set.
enemy.gd
func get_global_id():
if global_id == 0: # A global_id of 0 is invalid.
await global_id_set
return global_idWow, that was easy! (not)
Now, imagine having to understand your global server and client state constantly, always having to account for syncing delays, and always remembering which fields are synced and which are not. This requires a discipline that’s difficult even for experienced backend programmers like myself! There are paradigms that I have found helped me, which I will go over in a future posting, but even those don’t cover everything. Networked multiplayer is hard!!
Thanks for reading all the way to the end. Please let me know if this post was interesting for you, or if you’d rather me go back to game design! Any feedback is appreciated, and, as always, can be timely posted into the Discord.