[FXServer] Random error(s)

Zoddl · July 20, 2017, 3:22pm

Hello, I have a problem with my FXServer, when I restart my server, no error, I join, I spawn and
I have this error often :

System.ArgumentException: Destination array was not long enough. Check destIndex and length, and the array's lower bounds
  at System.Array.Copy (System.Array sourceArray, System.Int32 sourceIndex, System.Array destinationArray, System.Int32 destinationIndex, System.Int32 length) [0x000c1] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at System.Collections.Generic.List`1[T].ToArray () [0x0000c] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x0008d] in /src/code/client/clrcore/CitizenTaskScheduler.cs:120
  at CitizenFX.Core.InternalManager.TriggerEvent (System.String eventName, System.Byte[] argsSerialized, System.String sourceString) [0x000ff] in /src/code/client/clrcore/InternalManager.cs:210

I tell myself it’s nothing, I reboot …

And for the first time I had this :

Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.Tick () [0x00076] in /src/code/client/clrcore/InternalManager.cs:174
Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.Tick () [0x00076] in /src/code/client/clrcore/InternalManager.cs:174
Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.Tick () [0x00076] in /src/code/client/clrcore/InternalManager.cs:174
Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.Tick () [0x00076] in /src/code/client/clrcore/InternalManager.cs:174

(infinitely)

I reboot and now no problem… This errors are normaly ?

Thefoxeur54 · July 21, 2017, 6:38am

I have the same error randomly and after a few one the server crash with infinite of the same error
https://cdn.discordapp.com/attachments/337510130407047179/337803091338592256/error2.png
https://cdn.discordapp.com/attachments/337510130407047179/337811657742876673/error3.png
https://cdn.discordapp.com/attachments/337510130407047179/337845009246191627/error5.png

Zoddl · July 21, 2017, 7:15am

At least I’m not the only one but it’s not good

Thefoxeur54 · July 21, 2017, 7:21am

are you using couchDB or MYSQL-Async ?

Thefoxeur54 · July 21, 2017, 9:15am

and this error create instance , or is due to instance , I try recreate it

Thefoxeur54 · July 21, 2017, 7:34pm

Do you use VDK_Inventory ?

Supa · July 22, 2017, 9:41am

I’m not experienced in C# however a quick google search yielded: https://stackoverflow.com/questions/10362023/destination-array-not-long-enough

“Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object”

This means you’re trying to use a null object for SOMETHING. Assuming the syntax doesn’t cause the code to crash, it’ll just behave weird like yours when something is null. You should review your code, possibly with the thread I just supplied, and see if that solves the issue.

If not, please reply again so someone can better help you

justcfx2u · July 26, 2017, 4:58am

I have opened a mysql-async Issue with some of my findings. I think the problem might be within FiveM itself, though.

justcfx2u · July 26, 2017, 10:29am

Using artifact 295 seems to have helped, but not resolved the issue. I normally get 1 or 2 nulls on startup, and a few scattered ones before the big waterfall of System.NullReferenceException's which grind everything to a halt.

Artifact 295 has improved this as I receive no System.NullReferenceException on startup, but fivem still ultimately entered a deathball state (infinite System.NullReferenceExceptions) after 30 minutes of operation using my test script.

xenon · July 26, 2017, 10:33am

Care to provide such a test script that can operate without players? It’d be helpful so we can figure out exactly what happens causing null tasks to be enqueued.

justcfx2u · July 26, 2017, 10:35am

I’ll try to put together a 100% vanilla FXServer test case for you.

justcfx2u · July 26, 2017, 11:33am

This test-case does not require players, and barely requires MySQL (it uses SELECT statements which do not require any specific database or tables). I am able to get a crash using this setup in 20-90 seconds fairly reliably.

Though we would not normally be querying the database quite this rapidly in game, the randomness of the delays and ordering seen in a live game does seem to make this more reproducible and is roughly simulated by including the MySQL sleep()s included in the resource.

Download FXServer artifact 295.
Extract it
Download and deposit the following configuration into the new directory
Edit server.cfg to point to a valid, running MySQL instance, user, and password
Start the server in the usual way (e.g., bash run.sh exec +server.cfg)
Wait for the flood of System.NullReferenceExceptions.

fivem-295-mysql-async-vanilla-crash.tar.gz (149.7 KB)

Edit: Using fivem-mysql-async v2.0.0-beta1 as seen at https://github.com/brouznouf/fivem-mysql-async/releases/tag/v2.0.0-beta-1 (included in the .tar.gz for convenience). I am able to reproduce this using the latest fivem-mysql-async and a freshly-built replacement MySqlConnector v0.23.0 at https://github.com/mysql-net/MySqlConnector/releases/tag/0.23.0 so later versions do not appear to reduce the frequency of this crash.

phosphorus · July 26, 2017, 11:49am

On Windows this scenario seems to run as expected (without errors), other than sometimes causing main thread hitching for 30 seconds (it seems you exposed another unknown issue there?).

We’ll still have to try on Linux and with a typical ‘release’-type server build, as we currently used a debug build to experiment with this.

justcfx2u · July 26, 2017, 11:53am

Using 5.7.18-0ubuntu0.16.04.1 on a separate server/machine/IP for MySQL version, if it helps. On local LAN, <1ms latency.

flerovium · July 26, 2017, 11:59am

Oddly this doesn’t seem to reproduce on Windows at all in this scenario, local MariaDB instance however; on my current LAN there seems to be a configuration issue that makes queries have a 800ms response time no matter what.

Thing is this issue (as shown above in @Thefoxeur54’s post) does occur on Windows, however the test case does not repro there at all.

We’ll try the Linux artifact build now as a final attempt to cause the exact issues.

tantalum · July 26, 2017, 12:04pm

Ah, it does repro on Linux with the provided code, at least. Interesting. This does make it a bit harder to debug, however, as there’s no interactive development setup currently for the Linux server…

justcfx2u · July 26, 2017, 12:05pm

Thanks for being so incredibly responsive, team. I hope it is an easy fix in the end.

americium · July 26, 2017, 12:14pm

It seems like a lock statement was missing around one use of m_runningTasks in the commit that was intended to fix this issue.

Currently, we’ve been running the test case for a few minutes now with this missing lock statement added, and have not encountered this issue yet.

It has been committed as 854f4ad2, and a build will follow 10-25 minutes later (look for pipeline 298 or higher).

(minor edit)

Running!

… and passed, even!

bndzorro · July 26, 2017, 12:29pm

Thanks alot for the great support on this, impressed by how fast you guys were able to work on this and solve it!

We will try it out in a few minutes!

justcfx2u · July 26, 2017, 1:31pm

We’ve been running a test instance for a while now using our full game (plus the test code to hammer the database) and haven’t seen a single instance of this bug. I’m going to call it resolved for us as well.

Thank you VERY much for the prompt attention! This makes all the difference to our porting efforts. Things are looking up!