Server Crash on New User (Linux/MySQL)

I have been working on setting up a dedicated server on a Linux VPS. I have tried (and prefer) CentOS7 and Ubuntu 16.04.

Once I setup EssentialMode, EssentialMode Extended, and MySQL-Async/Async, the server crashes when a NEW player joins. If the player joins once and crashes the server, that same player can join once the server restarts. I have looked through all logs and do not see much on the server or client consoles. The server console simply stops, and the SSH session freezes.

If you look at the database, the user gets created and imported to the database immediately before the server crashes. Iā€™ve looked at all the OS log files, and the only suspicious activity I see is:

Feb  5 21:14:40 gta-fivem kernel: TCP: request_sock_TCP: Possible SYN flooding on port 30120. Sending cookies.  Check SNMP counters.

I have disabled TCP SYN cookies, and that did not help. Iā€™ve disabled iptables and/or firewall-cmd to ensure it is not a host firewall issue, and it did not help. Iā€™ve dumped the database and started from scratch, wiped the OS, switched OS, and it keeps coming back to this problem.

Iā€™m not new to server administration and have never experienced a log-less crash like this. Has anyone else seen this before? Is there something Iā€™m missing? Iā€™d be happy to provide any logs necessary. Is there any way to enable more verbose logging that I may be able to see more details from? Iā€™m truly at a loss here.

EDIT: It looks like if I use tmux and run it in the background, the server may stay running when it would typically knock my SSH session out. Iā€™m going to keep testing. I can replicate this over and over again by purging any database entries associated with my Steam ID and rejoining the server.

I was wrong about having the server run in the background fixing the issue, after about 4 tries I was able to replicate it again. I opened up a bunch of SSH sessions and actively monitored all of my log files, and found the the MySQL error log gets 4-6 error messages at the time of the disconnect:

2018-02-06T10:58:56.298767Z 39 [Note] Aborted connection 39 to db: 'essentialmode' user: 'fivemadmin' host: 'localhost' (Got an error reading communication packets)
2018-02-06T10:58:56.299064Z 38 [Note] Aborted connection 38 to db: 'essentialmode' user: 'fivemadmin' host: 'localhost' (Got an error reading communication packets)
2018-02-06T10:58:56.299252Z 37 [Note] Aborted connection 37 to db: 'essentialmode' user: 'fivemadmin' host: 'localhost' (Got an error reading communication packets)
2018-02-06T10:58:56.299444Z 36 [Note] Aborted connection 36 to db: 'essentialmode' user: 'fivemadmin' host: 'localhost' (Got an error reading communication packets)
2018-02-06T10:58:56.299635Z 35 [Note] Aborted connection 35 to db: 'essentialmode' user: 'fivemadmin' host: 'localhost' (Got an error reading communication packets)
2018-02-06T10:58:56.299823Z 34 [Note] Aborted connection 34 to db: 'essentialmode' user: 'fivemadmin' host: 'localhost' (Got an error reading communication packets)
2018-02-06T10:58:56.300010Z 33 [Note] Aborted connection 33 to db: 'essentialmode' user: 'fivemadmin' host: 'localhost' (Got an error reading communication packets)

Iā€™ve already increased the max_allowed_packet for MySQL, Iā€™m apparently having another issue somewhere. Still open to ideas, hoping to get this straightened out today.

EDIT:

Turned on Mono debugging with:

export MONO_OPTIONS=--trace=E:all
export MONO_LOG_MASK=asm
export MONO_LOG_LEVEL=debug

Server crashes with no additional output.

Update #3:

Started with a fresh server, no modified scripts, blank database. Installed EssentialMode and ESX, restarted and purged my player from the database and was eventually faced with the same problem.

If I purge the database and then restart the MySQL service with:

service mysql restart

I am able to join successfully. Iā€™m not sure how long it takes for the stability issues to resume. Perhaps I will need to restart the MySQL service periodically.

Still looking for a root cause and any advice to try to narrow it down.

Completely. Stumped.

Update #4:

Not sure if multiple posts is allowed, mods please merge if not.

The chronicles continue.

I decided to try using a MySQL server that Iā€™m hosting on my primary web host, and have yet to be able to reproduce the problem. Iā€™ve loaded in as a fresh character 10 times now without so much as a hiccup. Iā€™m wondering if A) this VPS Iā€™m hosting on was underpowered for hosting a database server and FiveM, or B) if Iā€™m being stupid during the MySQL setup and am missing something.

Regardless, Iā€™m going to keep testing it with having the database running on a separate server (which has plenty of resources to handle anything thrown at it). Iā€™m hopeful this will work??

Still open to ideas for testing :slight_smile:

We had that connection aborted issue as well with our customers databases.
In our case it was a failing disk, caused i/O waits.
Run the command ā€œtop -cā€ in Linux:
Screenshot_81
Check the wa value. More than 1% is very bad. Make sure your VPS has a SSD. Will make database performance much better.

I recommend using MariaDB on Linux:
https://downloads.mariadb.org/mariadb/repositories/
After that download mysqltuner for mariaDB:


It is a small perl script. U can run it with:
perl mysqltuner.pl
It checks ur database config and usages and recommends new settings for your /etc/mysql/my.cnf

2 Likes

Excellent, thank you for the advise. Iā€™m on SSD hosting. I checked top and have 0,0 wa, I will run the MySQL tuner to see what I should change, thanks for the tip - I havenā€™t seen this script before. Iā€™ll run top and monitor it as I connect (and have some friends connect). Iā€™ve not used MariaDB before, but Iā€™ll consider making that switch as well.

Out of curiosity, what are the specs you guys typically allocate to a FiveM server?

less than 2 GB Ram also for full servers with many resources. CPU is not really much too, depending on the clockrate of the server, but bandwidth is very heavy with many players.

1 Like

Hi,

Can someone please help me?
I have the same issue and I use nfoservers.

-Ralf

I have the same problem. New user crash server, can walk but chat/garage/admin panel not working. New user have skater ped and cant register charakter.

How to solve it?

My server VPS KVM, 6GB RAM, 120GB
Debian 9.3

Never found a definitive reason. Offloading the database server to another server fixed mine. I now have them both in the same VPS and it works fine. I wish I could help.