Hello everyone, and welcome back to the VV Eekly Update! It has been a very long time since the last update – a whole two weeks! In case anyone missed it, I decided to turn this into a VV Bi Eekly Update, only updating every two weeks. Here’s the message I posted last week:
Hey @Eekly Updates! Quick update – I’m now planning on doing biweekly updates for VV Eekly Update. Two big reasons for that: First, the release date being moved back to the summer in 2025 means that there’s a much longer runway. I’m afraid that I just don’t have that much content to share! Second, the work is getting to the point where I need to do little tweaks, fix little bugs, and put a lot of time into tasks. There’s not too much that I can show off anymore week-to-week.
I’m sorry to everyone who has been looking forward to reading the VV Eekly Update mid-week every week, but I think this is the right decision to make sure I can stay productive on getting Vyn and Verdan out the door!
Anyway, today, I’m going to be telling you a bit about a networking issue that we discovered in a playtest, and what I’m doing to investigate and mitigate the issue.
The Problem
Playtests have been going fairly well! As my head playtester-in-chief said, it’s starting to feel like Vyn and Verdan is a real game instead of a bug-laden tech demo. That’s very exciting! However, there’s definitely still issues that we come across. One issue that we found only occurs in certain rooms.
Essentially, in rooms with a lot of enemies, the ping jumps up really high! You can see in the top right corner of this picture that the ping says it’s 4232 ms (milliseconds). We had been testing with computers in the same room, and the ping had been a mere 32 ms before. This means that the non-hosting player will feel a lag of 4 whole seconds when moving, dodging, and using abilities. Obviously, this isn’t great.
How did I approach debugging this problem? First, some background: ping measures the round-trip time for a packet to be sent from one player to the other. I implemented a fairly simple system in Vyn and Verdan that just sends a ping from the host to all connected players that then waits for the ping to be returned.
When I say it’s “fairly simple” – I mean it. There’s not much to it! It doesn’t interact with the rest of the game at all! That was pretty confusing to me – how could this possibly be impacted by what room the players are in?
With some further investigation, I found the answer. The game engine, Godot, actually processes all incoming RPCs within a single queue in the order that they arrive. You can imagine that, for example, in a really busy room with lots of enemies, that queue might take longer to process than there’s time between frames, meaning that the queue gets overloaded. I’m assuming that Godot automatically purges non-mandatory requests when the queue gets too long, which is why the ping didn’t keep increasing to infinity. You can split incoming RPCs into different queues to make sure that one queue getting clogged doesn’t clog different RPCs, which is what I needed to do to fix the ping measurement!
However, there was still the issue of the lag in busy rooms.
Network Clog
After some investigation, I boiled down the problem to “there’s too much data being sent”. Clearly, the solution is “reduce the amount of data being sent”. But how?
Thankfully, Godot has some pretty nifty tools to help debug network usage.
You can see here the Network Profiler that Godot has embedded. It’s not exactly the same as what gets sent on the network – there’s protocol headers, protocol compression, other lower level stuff – but this definitely can show you what’s the biggest offenders and whether your changes help.
The problem manifested only in rooms with at least 20 enemies. Therefore, although I’m sure it would help to reduce some of the other load that’s not enemy-related, I should focus on reducing enemy usage.
You can see in the screenshot above that enemies will send ~116 bytes in every RPC. That includes where the enemy is located, what the enemy’s label is, what the enemy’s health is at, and what the enemy’s max health is at. Enemies will also send this data every frame.
But wait… the enemy’s max health never changes, and I decided against showing the enemy’s label! If I remove those, then the enemies only send ~84 bytes in every RPC! Wow, that’s a ~25% improvement!
I’m in the middle of this process so far – finding what small improvements I can make here or there to make a huge improvement to the lag issue. After compressing the data a little more, next on my list is only sending the data every few frames rather than every frame. Theoretically, that should cut down on the data usage by a LOT.
If you have any comments about today’s VV Eekly Update, please ping them my way in an appropriate number of bytes in the regular Discord!