[RESOLVED]Multiplayer desyncing, warping etc killing missions..

Hey there fellas, First of all TACVIEW and Server Track is below.

I cannot for the life of me get any large multiplayer missions completed without desync issues destroying my missions. I've tried it multiple times, no matter what, we get F18's that cause major FPS issues when they get close to anyone else. Then we get about an hour into the mission and then they warp and rubber band not just taking themselves out of the fight but snowplowing the entire flight of 20+ people from 300 miles away...

We have a dedicated server hosted at a data center with a 1gb/s up/down connection, I7 8700K, 32 gb of ram. Yet, with all this that we pay lots of money for, we cant run a mission with our org together because of these issues.

Can we please get some kind of answer as to what's going on, all I need is a "yes, we are aware of it" or "No, thank you for reporting this." It does not matter if we have a clean client or not we've tested this so many times trying to figure out what's going on.

We're at a point where we cannot operate without this stuff happening... I'm pulling my hair out and guys are angry because we're wasting weekends on this stuff..

We fully understand its Open Beta and consider this a bug report because that's exactly what it is.

This is not a rant nor is it a hit on ED. We feel ED is doing a great job and we love all of the changes. I just feel it needs some context as it is really hurting the larger online community.

Thanks!

Tacview

Track

Track from Same mission

Here is my Track from the same mission , I have never seen anything like this.

https://drive.google.com/file/d/1QaghDvPtn7wg9vEBngB57v5-4m80SJKk/view?usp=sharing

SoW server reporting similar problems. From 15 players upwards things get unplayable.

Here's the log... the mission starts at 01:20 in this log.

https://drive.google.com/open?id=12zNpTBrYgGIYhJkVasluB3qXcvEFo_aw

Server Specs.

i7-8700K

MSI- Carbon Gaming motherboard

32gb DDR4 3200 RAM

1tb Samsung NVMe Drive

Located at a data center in the central United States with a 1 gb up and down connection. This issue started prior to all the quarantines fyi.

Thanks BN,

Strike posted the server specs, and Spiceman posted the log.

Also our server seems affected by this problem on last OB.

Example Youtube video

Our server is:

CPU: Intel Xeon E5-1630v3 ( 4C/8T @ 3.7Ghz)

RAM: 64GB @ 2133MHz ECC REG

Storage: SSD 480GB NVMe

Connection: 250Mbps Up/Down Datacenter OVH

When rubber banding occurs not even reached nominal bandwidth, DCS machine takes only 2MB/sec ( Less then 20Mbps)

Observed an high CPU usage on single core, even not full load of 100%, but reach 70% on higher used core. The other remains at really low load like 15%

@IceFire, for start... you should try it without LotAtc, SRS, Tacview, etc. Than you will see if problem is DCS or some third party mod.

Btw you have LotAtc 1.2.3 ... we had performance issues caused by it on our server, it was fixed by update to LotAtc 2.0.0.

2020-03-22 01:03:35.186 INFO    LuaNET: [LOTATC] ---- LotAtc is ready and correctly LOADED ----
2020-03-22 01:03:35.186 INFO    LuaNET: [LOTATC] ---- Version is	1.2.3-3a348a97

SRS... you have old AutoConnect script...it can be fine, not sure if there were changes between 1.7.1.5 and 1.7.7.0.

GameGUI and Overlay GameGUI are client side scripts so it should not be installed on server. (I may be wrong.)

2020-03-22 01:03:35.083 INFO    LuaNET: Loaded - DCS-SRS-AutoConnect 1.7.1.5
2020-03-22 01:03:35.091 INFO    LuaNET: Loading - DCS-SRS GameGUI - Ciribob: 1.7.7.0
2020-03-22 01:03:35.092 INFO    LuaNET: Loaded - DCS-SRS GameGUI - Ciribob: 1.7.7.0
2020-03-22 01:03:35.096 INFO    LuaNET: Loading - DCS-SRS Overlay GameGUI - Ciribob: 1.7.7.0 
2020-03-22 01:03:35.097 INFO    LuaNET: Loaded - DCS-SRS Overlay GameGUI - Ciribob: 1.7.7.0 

Tacview: You have profilling info from TacView wrote to log so often, it coud cause issues too. (Every 10 ms?)

2020-03-22 01:06:31.143 INFO    TACVIEW.DLL: During the last 0.0s an average of 0 units objects and 0 ballistic objects have been active per frame
2020-03-22 01:06:31.151 INFO    TACVIEW.DLL: During the last 0.0s an average of 0 units objects and 0 ballistic objects have been active per frame

I was asked to post the server specs for the 132nd Server here:

CPU: Intel i7 4790K

RAM: 32GB

Storage: All SATA SSDs (no NVMe)

WAN: 500/500Mbit FTTH

TGW servers having the same issues since the last Open Beta patch.. rubberb anding, desync etc is really bad, seems to be especially worse when a ground unit moves at all, in any shape or form on a mission that's literally not changed in 3 months except to update a spelling error.

Servers only using 4gig of 32gb ram and 20% cpu i7 7600k , maximum send got to 11kbps on a 250mbps/1tb datacenter pipe, somethings messed up. Don't know if it's related but there are also transport.dll and weapon.dll crashes happening occasionally.

Yeah I’ve noticed units in groups greater than one seem to cause it.

Just to be clear and exclude any doubts regarding server overhead hardware performance, if you have multiple instances on the same server, you can notice that one high populated instances can start to do crazy things like rubberbanding, and desync, while the other play smooth as butter, so there is nothing related to server hardware/bandwidth usage exceeding free resources ;)

Yeah I’ve noticed units in groups greater than one seem to cause it.

This seems a hot tip!

I also experienced many problems with ground troops in the last weeks (most of the groups have more than one unit).

Additionally, the LGBs are not synchronized always. A friend of mine and me are doing buddy-lasing all the time, and he (the client) is not always seeing the LGBs following the laser, while I (server) can see them following.

The new build today, 2.5.6.45915 has not fixed this serious desyncing issue.

It seems that we are unable to run missions any where near the complexity since 2.5.6.44266 was updated to 2.5.6.45317.

Am I to spend many hours rebuilding missions with 1 x unit moving groups instead of with multiple units? Or should I wait for this to be properly addressed for the next release?

I could maybe add some valuable feedback: Desync is exactly the same whether it on LAN or WAN. Rolling the server only back to 2.5.6.44266 did not alleviate the issue. So it seems to be client side? The number of clients need only be 1 on LAN to reproduce.

Thanks ED for your hard work.

Same issue with our missions/servers. It is like the AI does not see one another, or more accurately they see enemy that is not there.

having this happen for the last 2 patches watching and only seems to have actually gotten WORSE this patch :( on stuff that was running smooth as in 2.5.5.xx :( When it happens with the new Net information there is a massive spike in the RTT each time which I'm going to presume is the round trip time, the ISSUE however is these spikes are not network related as you can have a ping running constantly to the server showing a very very different set of numbers, case in point in this image, RTT time is almost 300ms (to a dedicated server) while the ping time at the exact same moment is average sub 50.

So what gives? why is there suddenly this jump in RTT times which don't correlate to any other network jump in RTT's at all so it's gotta be the DCS server itself taking time on the packets not network congestion and why can't DCS handle interpolation of positioning better when desync occurs rather then snap rubber banding things and 'breaking' stuff?

Somethings really off here.

Edit: Actually you can nix some of the above, because i just had the whole dynsc happen with sub 60ms RTT's. and a sudden jump in packet loss though again if you sit and do a set of pings etc packet loss is reported at 0% while dcs repots 5% etc. Worse you can have the exact same happen over a LAN connection at times not just a internet one :/

yep, been running our server on 2.5.5 for a while now without issues

tried yesterdays 2.5.6 patch and the rubber banding is insane one poor infantry guy warped into my helicopter and got splattered (was trying to rescue him)

Hopefully this gets fixed before super carrier release.

Hopefully this gets fixed before super carrier release.

Yeah I'm also worried about the experience that can we offer to our players in Multiplayer.

SuperCarrier has super cool features but I think nowday it will be almost impossible to experience all the best of it in public servers.

A "little" sliding effects that we're usual to see on DCS since many years, are now really "amplyfied" making carrier unusuable on highly populated server. It will be a pain miss this stuff maybe for lags and desync.

I'm keeping finger crossed ;) :thumbup:

I have conducted a large amount of testing regarding the Server starting to drop packets.

AEF Server #5 is the server we are using. Internet connection is Business grade 250mbps/100mbps connection.

Server Spec is a virtual machine running on ESXi 6.7+:

3.5Ghz Dual Core

64GB RAM

1TB NVMe Drive

ESXi Host CPU runs at about 40% when the DCS Server is running with approximately 400 units on it.

At server and mission load there is zero latency, no lag, no (DCS reported) dropped packets.

After approximately 24 hours of constant running there is an increase of packet loss to less than 1%. At approximately 5 days the packet loss situation is at around 10%. This includes my client connection that is in the same LAN segment so the dropped packets are not ISP related but rather DCS related.

I have conducted extensive testing on my LAN utilizing winPcap and only have retrans from the DCS server. (i.e. the network is solid)

The mission has multiple groups spawning via Moose functions, as well as 2x AWACS, 6x A2A Refuellers and Combined Arms units to help capture Maykop airfield and an Aircraft Carrier Group. LotATC, Tacview and SRS are running on the server and are all up to date.

Findings at this stage are inconclusive as to why the server gradually grows in packet drop. We did find that the first aircraft affected seem to be JF-17's. It appears the simulation is pausing due to the fact that in the Tacview recording units stop moving when descync occurs.

##New finding## At server Start the Aircraft Carrier and supporting group elements (all the boats), seem to be pausing about once a second. I am going to remove the Aircraft Carrier Group and see if this reduces the Lag experienced.

Add me to the list of server admins who report EXACTLY what you see. This is absurd and damn frustrating. How do non-moving units even cause rubberbanding lol, twilight zone stuff.

+1 My group is experiencing this as well. Crazy rubber banding everywhere

DCS OB 2.5.6.47224

Carrier in MP is worse now (ping<40, players<10)

Lots of desync and deck sliding, ship was not turning.

A huge improvement ED. Thank you for your efforts.