[Server] Massive timed out from clients then server hang connections

FXServer version : 1552-8e2468eb15caacdf739ff893d7e2cd83dc6f6e0a
Physical server info : Xeon E3-1245v5 - 16GB RAM - SoftRaid 2x2To - 1Gb/s ethernet connection
Resources : Custom resources (no ESX/VRP) and essentialmode.

Since our community start growing and we have switch to onesync 64players, we have each day this problem.
After some times (variable) with 50+ players connected, players start to receive timed out ( 60s timed out ) and get disconnected one by one from server. After that, they can’t reconnect without a FXServer restart.
The problem is, there is no error log on server side, nor client side ( can send if needed ).
I can connect to info.json/players.json from the server during the hang.

I have done some ETW during the problem but nothing appears ( no anormal hitch ). Can send it if needed

Here is the procdump ( Warning : 3.5Gb) just before FXServer restart : https://www.transfernow.net/3139a8j5phht

Anyone got a clue ?

Thanks a lot !

1 Like

Hello, this is a friendly reminder because this is your first time creating a topic (or it has been a while since your last topic) in this category.

Please note that most of the support is provided by the FiveM community on a voluntary basis. We ask you to be patient; there is no guarantee we have a solution to your problem(s). To avoid unnecessary/duplicate topics, please browse the forums before creating a topic.

To improve your chances of your issue(s) being solved, please provide as much information as possible about the issue(s) you are having. Also —whenever possible— please use the template given to you when creating a topic.

Thanks for keeping these forums tidy!
:mascot:

Adding some netgraph screen taken during the “crash”

Edit : Crash happend at 10.20PM tonight >> Updated to artifact 1570… Will see tomorrow

Yes, we have the same problem since we have Onesync. From 10 players the server is feeling so hangs behind but the server is not so rather onesync because it is the way it is now and then we lose the connection

A good idea for next time: compress the dump - it easily gets 5-10x smaller if you do so.

At least someone’s finally uploaded a dump of an actual hung(?) server.

… ouch, the dump seems to be corrupted? It seems to be missing half of the file. :frowning:

1 Like

ah shit ! i will try to compresd it before upload !

Same file from server, but zipped this time. Maybe the transfer corrupt the file :

I’m getting the file to on a PC to open it with WinDBG in order to check if corrupted too.

Edit : It seems to be ok with the zipped version. Can open it with WinDBG !

Otherwise, i will try to do another ProcDump tonight if we reach 50+ players

Yeah, but this server build seems to be a weird one which for some reason didn’t have symbols saved, so it’s still practically impossible to debug - or VS is acting up. :confused:

In addition to that mess, the dump shows some Mono-side hang on a background thread. That’s… interesting, but also hard to diagnose postmortem since one can’t invoke mono_pmip on a dump. Due to the lack of symbols and use of Windows before 1703, it’s hard to conclusively say what went on with this dump, as nothing seems too out of the ordinary (all 3 core threads are just busy waiting for requests).

Oh, so it could be related to Windows Server version (2016 Datacenter up to date )?
We are using a dedicated server from OVH ( 2 years old offer : Serveur SP-16-S).

We update FXServer to 1570 yesterday and have a crash, but the admin didn’t save the procdump.

Can you give me any informations in order to help you to diagnose something ?

BTW, thanks for the time your spend on this issue

No, but newer versions add some extra info to .dmp files that makes it a bit easier to figure out what causes a hang.

This dump you uploaded, however, does not show any real signs of a hang…

Can people who did not get timed out still play/connect? It might be these people are temporarily getting blocked by OVH’s UDP mitigation, which is a different, yet still unusual concern.

Ok. I have asked to the physical server owner to call OVH about this problem (thinking about TCP/UDP limitation ). Maybe they have some monitoring apps that can help us.

All connected people are disconnected. But i will try to wait this issue and connect in order to check if new incomming connection still pass.

Thanks

if needed, a long ETW trace https://we.tl/t-rrhNtFWd5V when these little hangs happened 2-3 times

My problem is probably due to OVH udp Mitigation… We are trying to change our dedicated server offer to a “better” one, with more control on incoming traffic.
I will try to provide more test when switching to another “gaming” offer.

I found something.

For me, it only happens after the console is spammed with this

INFO: Requesting voice channel crypt resync

So basically is based on voice channel this issue, but I think we can not solve it and we have to wait for FiveM to update the artifact.

and again you think wrong.

as said on Discord, something is causing packet loss or delay, or a server hang.

that then results in BOTH that voice channel message (as it tries to recover from packet loss) as well as clients disconnecting as they lose too many reliable packets.

the fitting analogy stated there as well: you’re saying “ah i finally know why i bleed! it is because i hurt. someone needs to make me stop hurting!” when you actually are walking through prickly thorns causing a wound that both hurts and bleeds… the unknown scenario that leads to packet loss is the equivalent of the prickly thorns, the hurting is your theory of voice resync and the bleeding is the reliable packet dropping.

1 Like

does my trace posted above or what I sent in DM have provide anything useful?

What’s the server specs?

We are not using IG voice channel in our server so our error isn’t linked to that issue. No news from server admin atm about ovh offer migration. Issue occurs 2 times yesterday, 10 minutes delay between connections timeout.

2 dumps related to the same problem.

OS : Windows server 2019 Standard
Players ± 70
Artifact 1610

795bd986-1e85-4bfd-85ea-37593e9d56ca.dmp (636.9 KB) ea2b0357-bfeb-48ab-9575-0f4f0dd8581c.dmp (630.2 KB)