Jump to content

CPU Resource optimizations & tips for large multiplayer dynamic / or persistent missions


Recommended Posts

Hi,

I've been working on a large (full map) style multiplayer semi-dynamic mission that may run for days (or weeks), and one of the problems I have identified is running into CPU issues and managing performance in a way that isn't over taxing with the mission editor.

I'm trying to work through it at the moment, and I'm working on the following - but I'd be very keen to know if anyone has any additional tips or tricks that they use in order to allow for a large scale mission to operate without hitting CPU issues.

(I am pretty green with both DCS mission editing, and LUA - so I'm not sure whether there are better ways of achieving this or not).

The main thing I am working on at the moment is using Late Activation & Uncontrolled in the mission editor, and then setup many zones on my mission, and many units. I figure if they're not activated - then they shouldn't be using CPU resources? I then create tables of the zones and units, and then have a generic scheduled function that runs that does checks to see if the units need to be activated. 

I use this approach instead of the mission editor as it makes it more efficient to add new units and zones later without having to do a lot of triggers, all I need to do is add them to the table.

Here is an example of the table and function I'm planning on using. (I hope I haven't reinvented the wheel here - but open if there is a better way to achieve this!) 

activationunits = {
   -- begin zone section
   {zonename = 'zone1', activated = false,       zonedesc = 'yeah, whatever', activationtype = 'bluforall', groupinfo = {
                                                  {groupname = 'TestGroup1', intel = false},
                                                  {groupname = 'TestGroup2', intel = false},
                                                                     } --group info
   }, --zonename
   -- end zone section
   {zonename = 'zone2', activated = false, zonedesc = 'somewhere', activationtype = 'bluforair', groupinfo = {
                                                  {groupname = 'SAMGroup1', intel = true, inteldesc = 'SAM Alert!'},
                                                  {groupname = 'TestVehicle2-1', intel = false},
                                                                     } --group info
   }, --zonename
  
-- ... etc
  
} -- activationunits


zonesched = SCHEDULER:New(nil,
function()
   for i = 1 , #activationunits do
      activateflag = false
      if activationunits[i].activated ~= true then
         if activationunits[i].activationtype == 'bluforair' then
            if bluforair:AnyInZone(ZONE:FindByName(activationunits[i].zonename)) then   --blufor declared in InitFlags
               activateflag = true
            end
         else -- if activationunits[i].activationtype == 'bluforall' then
            if bluforall:AnyInZone(ZONE:FindByName(activationunits[i].zonename)) then   --blufor declared in InitFlags
               activateflag = true
            end
         end

         if activateflag == true then
            k = 1
            thisactivationgroupinfo = activationunits[i].groupinfo
            for j = 1 , #thisactivationgroupinfo do
               myactivategroup = GROUP:FindByName(activationunits[i].groupinfo[j].groupname)
               if myactivategroup ~= nil then
                  if activationunits[i].groupinfo[j].startuncontrolled == true then
                     myactivategroup:StartUncontrolled(k)
                  else
                     myactivategroup:Activate(k)
                  end
                  k = k + 1

                  if activationunits[i].groupinfo[j].intel == true then
                     MESSAGE:New('Enemy Activity located near '..activationunits[i].zonedesc, 10):ToAll()
                  end --intel
               end --myactivegroup not nil
            end --for j
            activationunits[i].activated = true   -- Does not work :-(
         end -- if activateflag
      end -- Not activated
   end

   end, {}, 1, 15) --zonetrigger Scheduler

A better solution maybe (although I'm not sure if it would create it's own problems) is to scan through each group and check for any activity within xxx distance from that unit and activate or deactivate accordingly. This would eliminate the need for zones, and less maintenance with mission designing - but I'm not sure whether or not that is practical. Very keen to hear if someone has done something else better than the above.

 

Another option I'm toying with is dynamically spawning random units within zones but I haven't yet perfected script for that yet. I know dynamic spawning with Moose is easy - that's not the issue. Figuring out how to decide what units in what general areas that would be practical for the mission is what I'm working through.

Another small thing I'm doing is making sure wake turbulence is OFF. I believe (from reading on threads here) -that can have an impact with performance.

Lastly - I'm looking at using script to save the mission so the server can be reset every 24hrs. It seems that if I leave a mission running for days on end - that performance can end up being an issue, so it's my current understanding that restarting the server helps with CPU and resource management.

 

I'm not sure if anyone has any additional tips over these that they use that they're willing to share - or if you have any better solutions to the above thoughts that I'm looking at using, but I'd be keen to hear.


Edited by Dangerzone
Link to comment
Share on other sites

I have a 24/7 mission usually 1000+ ground forces on the Syria map, 20 clients, 70 ish air assets, upto 120 units on the ground moving over a max of 90km 75000 lines of code and my CPU is normally around 60-70% loaded (when the mission is forced to a single core). The only way to achieve that is some clever code and sadly not with the mission editor. Im on the road at the minute but can reply proper when i get home, feel free to hit me up on Discord, there are some other fundamentals that you can be doing with your hosting env and configuration I’ll post in a bit. But the key is to understand what’s expensive and how it affects the netsync for clients to know how to go about scheduling your actions. Regular reboots and other things floating about are a bit of a myth. 


Edited by PravusJSB
  • Thanks 1

Developer of XSAF :: A PVE with a custom GOAP/Learning AI model that wants to kill you

Two 24/7 servers and a regular structured op on the weekend

Discord: PravusJSB#9484   twitch.tv/pravusjsb  https://www.patreon.com/XSAF  https://discord.gg/pC9EBe8vWU

Link to comment
Share on other sites

There also may be some implicit points to look at:

  • Obviously, if you run the mission on a dedicated server instead of self-hosting mission, you'd have much greater reserves. 
  • Also, look at the frameworks you are using for mission scripting. Some of these (and I won't point any fingers since they are all great, and they all can be mis-used 🙂) may impart a significant performance drain because they poll unit positions every 10th or a second.
  • When using frameworks, be sure you understand how their accessor functions work, and cache results (and check viability via isExist() ) rather than using the accessor if that entitles searching an internal DB. 
  • When spawning units, especially ground units, be advised that issuing move orders can (but doesn't have to) result in CPU spikes, especially if you order them to follow roads, or re-issues too often. 

Cheers,

-ch

Link to comment
Share on other sites

On 11/25/2021 at 11:25 PM, PravusJSB said:

I have a 24/7 mission usually 1000+ ground forces on the Syria map, 20 clients, 70 ish air assets, upto 120 units on the ground moving over a max of 90km 75000 lines of code and my CPU is normally around 60-70% loaded (when the mission is forced to a single core). The only way to achieve that is some clever code and sadly not with the mission editor. Im on the road at the minute but can reply proper when i get home, feel free to hit me up on Discord, there are some other fundamentals that you can be doing with your hosting env and configuration I’ll post in a bit. But the key is to understand what’s expensive and how it affects the netsync for clients to know how to go about scheduling your actions. Regular reboots and other things floating about are a bit of a myth. 

 

Hi PravusJSB,

Thanks for that. I'd be really keen to hear about what code to use and what code to avoid plus what to set up environmental wise. 

Interesting re the reboots. That may make hosting more simplistic. At this stage almost all scheduling is done exclusively in Moose.

Link to comment
Share on other sites

On 11/26/2021 at 12:20 AM, cfrag said:

There also may be some implicit points to look at:

  • Obviously, if you run the mission on a dedicated server instead of self-hosting mission, you'd have much greater reserves. 
  • Also, look at the frameworks you are using for mission scripting. Some of these (and I won't point any fingers since they are all great, and they all can be mis-used 🙂) may impart a significant performance drain because they poll unit positions every 10th or a second.
  • When using frameworks, be sure you understand how their accessor functions work, and cache results (and check viability via isExist() ) rather than using the accessor if that entitles searching an internal DB. 
  • When spawning units, especially ground units, be advised that issuing move orders can (but doesn't have to) result in CPU spikes, especially if you order them to follow roads, or re-issues too often. 

Cheers,

-ch

Hi CH,

Thanks for the tips. yes - I have a dedicted machine for hosting. My main concern is learning the best techniques in allowing for a large map with many assets without being a burden on netcode and/or client CPU cycles. (As well as the server). 

I'm using mist and moose, as well as a few other small addons. Moose is very large. I'm aware that some things have more of an impact than others but not sure exactly what they are. I'm trying to avoid trial and error as I've spent weeks setting up a scenario in Moose just to then ditch it because of performance issues which I'm sure is me asking too much of it and/or  misusing it as you say (making things too complicated). I don't blame moose - just looking for techniques on what to do and what to avoid at the moment for large scale maps.

Re your last point - are you saying it's best to get vehicles to ignore roads and just go offroad for both tracking as well as CPU?

Cheers

DZ

Link to comment
Share on other sites

10 hours ago, Dangerzone said:

Thanks for the tips. yes - I have a dedicted machine for hosting. My main concern is learning the best techniques in allowing for a large map with many assets without being a burden on netcode and/or client CPU cycles. (As well as the server). 

Understood. One item I left out that I think is blindingly obvious - but then promptly forget myself is that the map you choose also has tremendous impact on your performance. In more ways than one. We'll get to that later on when we discuss roads, but as a rule of thumb, Caucasus is the map to beat in terms of performance, while drop-dead beautiful Syria can also be drop-dead for your performance.

Also (and I'm sure you know this, I'm merely making sure we have the same baseline), the lua code that goes into a .miz file is executed on all clients (and self-hosted servers), while the code deployed to /Hooks only executes on the server. Since currently there is very little heavy lifting that can be done server-side, that's not much relevant now, but (here's hoping) later versions of DCS may allow server-side offloading, so you may want to design your code with both points in mind: your .miz is executed on all clients (design .miz code for the weakest client) and modularize your code for possible server-side extensions.

10 hours ago, Dangerzone said:

I'm aware that some things have more of an impact than others but not sure exactly what they are.

MIST is one of the best ways to learn about lua mission coding, and chock full of some of the greatest, meanest tricks this side of the DCS executable. Whenever I don't know how to go about something, my first reference is MIST, and I try to figure out "how Grimes does it" 🙂. And MIST is a framework that is designed for broad usability, not necessarily peak performance (although it does have effective built-in functionality to spread CPU load): its goal seems to be to be as massively helpful to as many people as possible. Those who have progressed far enough to require peak performance in large missions should strive to know enough about it to be able to either curtail MIST itself, or roll their own. For example, MIST installs an update loop that runs 100 times a second. If you understand what it does in that update loop and why, you can use that to your advantage: increase invocation interval, or eliminate it altogether if you don't need that functionality. And you should also understand that out-of-the-box, if you have 1000 units on your map, Mist iterates them at a certain rate (IIRC updateAlive() iterates 1000/20 = 50 units every 1/5 second, or 250 units per second with 1000 units on the map). This is incredibly helpful for 99% of all missions - except those outlier missions that require thousands of units, and can thus overwhelm a lesser client (since it's part of the .miz it is run on every client). This is not a knock against Mist - far from it, it's the best DCS mission framework around - it's a reminder of the fact that for peak performance we need to understand the tools we use, and use them accordingly. TBH, dissecting Mist was one of the most fun and enlightening bits of learning DCS mission scripting. Highly addictive, so beware.

10 hours ago, Dangerzone said:

Re your last point - are you saying it's best to get vehicles to ignore roads and just go offroad for both tracking as well as CPU?

It seems that in order for units to follow a route, DCS creates a polyline that follows the shape of the road on the map. This can result in routes that have a very high number of vertices. Obviously, if one wishes to traverse from A to B, this highly depends on a multitude of parameters: the complexity of interconnecting road networks, the curvature of roads, shape of terrain, and if the road systems are connected. In any event, a high vertex point count induces high AI workload both for pathing and tracking, and can be a significant drag on performance. 

Consider:

image.png  and image.png

aren't the same in terms of complexity for pathing if you want your troops to path from top to bottom while they are following roads. Telling your troops to simply use (mostly) straight lines will be much more performant for the example on the right. As always, it's our fun and challenge to find a good way to handle this in an efficient manner.

You can try to optimize issues like this by adding points to the map from hand (perhaps as zones) and add code to make units to gravitate towards these points instead of using roads. It'll often greatly reduce path complexity while still giving the semblance of troops following roads. Alas, there is no one best approach, and I have found that the results are highly dependent on where on the map your troops are placed. Consequently, I have an option in my path creation routines to use roads or straight lines. Additionally, I can place Zones in ME that can override that by adding a "pathing" property to the zone. The path creating routine looks up if a group is inside one such zone, and creates the route according to the property's value. Note that this is only my own implementation, you'll probably come up with much better ideas.

image.png

In any way, welcome to your newest addiction 🙂 - I hope you are having as much fun writing missions as I have.

-ch


Edited by cfrag
  • Thanks 2
Link to comment
Share on other sites

7 hours ago, Dangerzone said:

Hi PravusJSB,

Thanks for that. I'd be really keen to hear about what code to use and what code to avoid plus what to set up environmental wise. 

Interesting re the reboots. That may make hosting more simplistic. At this stage almost all scheduling is done exclusively in Moose.

Sorry I've been struggling to come back to you. cfrag has given you lots of things to think about there for sure. I agree that pathfinding is hugely expensive and there are many ways to mitigate this but I feel those might be far advanced at the minute. I would however, like to point out that sending units over long distances off road (and to be honest indeed on road in Syria depending on the unit and the road) will cause a seemingly infinite number of 'dead' events as the unit/s kills scenery objects on its way to Z. This is a particularly bad situation for a MP server with aspirations of grandeur as all those events will need to be communicated with clients, and unless your code is robust it may destroy the net sync.

There as some really quick wins and some fundamentals to consider;

  • Make sure your server is running in a streamlined OS env.
  • Find a way to lock the affinity of the server to 3 physical cores only, and if possible tell everything else to use whatever else is left on the CPU instead.
  • Avoid using tracks, SRS hooks, tacview hooks. At least until you get a benchmark on your performance limitations.
  • Get a benchmark on your performance, this much is very important.
  •  I recommend not to use a framework at all. If you must, then refer to MIST, its a great bit of code. If you have to, use MIST. Otherwise, learn the SSE, code things for yourself. This way you learn Lua and DCS SSE/API, using a framework is learning a framework and if you do this you will thank me in a year or two, else if you're still here you'll avoid telling me I was right whilst you secretly try to dig yourself out lol.
  • Learn Lua, there are some great resources to be found with a google search.
  • Localise, re-use and avoid new closures and tables where possible. If you don't get this point see the above point.
  • Have a robust logging system to know what your code is doing.
  • Make a github for your code and branch test and release, test 'test' locally and have extra log outputs for this build. This will avoid pain when clients are connected.
  • Simulate your code and think about fringe cases before deploying to release, be cautious in your exception handling and understand that DCS will stop execution of your code if it throws, and also DCS sometimes doesn't throw and you'll never know there's a problem if you didn't write your own catches.

That should give you enough to think about. Honestly I could talk you to sleep with the knowledge I have but it would be too much at this point I think. Go and try things, it won't break beyond what a reboot won't fix, you need to make mistakes to learn for sure, so deff please do think about point 5.

I'm actually writing something at the minute for the community which is very much along the lines of this post's theme, I can share with you a draft copy if you hit me up on Discord, it might help you and it might help me getting feedback from you as you are my target audience 🙂

If you have any questions just ask.

  • Thanks 2

Developer of XSAF :: A PVE with a custom GOAP/Learning AI model that wants to kill you

Two 24/7 servers and a regular structured op on the weekend

Discord: PravusJSB#9484   twitch.tv/pravusjsb  https://www.patreon.com/XSAF  https://discord.gg/pC9EBe8vWU

Link to comment
Share on other sites

On 11/27/2021 at 11:57 PM, PravusJSB said:

Sorry I've been struggling to come back to you. cfrag has given you lots of things to think about there for sure. I agree that pathfinding is hugely expensive and there are many ways to mitigate this but I feel those might be far advanced at the minute. I would however, like to point out that sending units over long distances off road (and to be honest indeed on road in Syria depending on the unit and the road) will cause a seemingly infinite number of 'dead' events as the unit/s kills scenery objects on its way to Z. This is a particularly bad situation for a MP server with aspirations of grandeur as all those events will need to be communicated with clients, and unless your code is robust it may destroy the net sync.

There as some really quick wins and some fundamentals to consider;

  • Make sure your server is running in a streamlined OS env.
  • Find a way to lock the affinity of the server to 3 physical cores only, and if possible tell everything else to use whatever else is left on the CPU instead.
  • Avoid using tracks, SRS hooks, tacview hooks. At least until you get a benchmark on your performance limitations.
  • Get a benchmark on your performance, this much is very important.
  •  I recommend not to use a framework at all. If you must, then refer to MIST, its a great bit of code. If you have to, use MIST. Otherwise, learn the SSE, code things for yourself. This way you learn Lua and DCS SSE/API, using a framework is learning a framework and if you do this you will thank me in a year or two, else if you're still here you'll avoid telling me I was right whilst you secretly try to dig yourself out lol.
  • Learn Lua, there are some great resources to be found with a google search.
  • Localise, re-use and avoid new closures and tables where possible. If you don't get this point see the above point.
  • Have a robust logging system to know what your code is doing.
  • Make a github for your code and branch test and release, test 'test' locally and have extra log outputs for this build. This will avoid pain when clients are connected.
  • Simulate your code and think about fringe cases before deploying to release, be cautious in your exception handling and understand that DCS will stop execution of your code if it throws, and also DCS sometimes doesn't throw and you'll never know there's a problem if you didn't write your own catches.

That should give you enough to think about. Honestly I could talk you to sleep with the knowledge I have but it would be too much at this point I think. Go and try things, it won't break beyond what a reboot won't fix, you need to make mistakes to learn for sure, so deff please do think about point 5.

I'm actually writing something at the minute for the community which is very much along the lines of this post's theme, I can share with you a draft copy if you hit me up on Discord, it might help you and it might help me getting feedback from you as you are my target audience 🙂

If you have any questions just ask.

Thanks for that. It looks like I've been doing it all wrong right from the beginning:

  • I've been using Moose instead of coding direct - Check
  • I've been using Syria instead of Georgia - Check
  • I've been routing vehicles via roads to use CPU cycles - Check
  • I've been using SRS hooks - Check
  • I've been using tables (a lot) - Check
  • I've been adding all my units to the map (although most inactive), but looks like they'll still take up cpu cycles.

 

Although I did disable tacview and tracks so I guess I managed to get something right. 😉

So I guess if I continue down this path I could get myself into a lot of hurt. Thanks for the tips. I'll have to try and reevaluate how I'm going to do everything - but I'm beginning to think it's going to be beyond me. Even something as simple as having a squadron of planes take off when there's threats in the area, be able to respawn x times if shot down within y minutes, and land when there's no air activity is something I was so grateful to find MOOSE - and even then am struggling to get it to behave the way I want... I think I'm out of my depth if I get rid of libraries altogether and try and code directly. 

 As for I'll be thanking you in a year or two... I'm thanking you now... but I'm also kinda hoping my missions won't be needed then - and we'll have dynamic campaigns. 😉

On 11/27/2021 at 9:48 PM, cfrag said:

Understood. One item I left out that I think is blindingly obvious - but then promptly forget myself is that the map you choose also has tremendous impact on your performance. In more ways than one. We'll get to that later on when we discuss roads, but as a rule of thumb, Caucasus is the map to beat in terms of performance, while drop-dead beautiful Syria can also be drop-dead for your performance.

Also (and I'm sure you know this, I'm merely making sure we have the same baseline), the lua code that goes into a .miz file is executed on all clients (and self-hosted servers), while the code deployed to /Hooks only executes on the server. Since currently there is very little heavy lifting that can be done server-side, that's not much relevant now, but (here's hoping) later versions of DCS may allow server-side offloading, so you may want to design your code with both points in mind: your .miz is executed on all clients (design .miz code for the weakest client) and modularize your code for possible server-side extensions.

Thanks cfrag! As for the lua code, I've done mine a bit differently. I have it executing external code - it's not in the mission file (or .miz file) at all.  I don't know if this means it only runs on the server, and whether or not that is an advantage?

(The reason I'm executing it externally and not including it in the .miz file is simply so I can make changes very quickly and restart the mission to continue testing, so I don't have to open up the mission, remove the unit, re-add it again, and then re-start the mission for testing). 

So - since the .lua files are server-side only - does that mean only the server is executing them, and will that be helping to reduce CPU hits on the clients? 

On 11/27/2021 at 9:48 PM, cfrag said:

MIST is one of the best ways to learn about lua mission coding, and chock full of some of the greatest, meanest tricks this side of the DCS executable. Whenever I don't know how to go about something, my first reference is MIST, and I try to figure out "how Grimes does it" 🙂. And MIST is a framework that is designed for broad usability, not necessarily peak performance (although it does have effective built-in functionality to spread CPU load): its goal seems to be to be as massively helpful to as many people as possible. Those who have progressed far enough to require peak performance in large missions should strive to know enough about it to be able to either curtail MIST itself, or roll their own. For example, MIST installs an update loop that runs 100 times a second. If you understand what it does in that update loop and why, you can use that to your advantage: increase invocation interval, or eliminate it altogether if you don't need that functionality. And you should also understand that out-of-the-box, if you have 1000 units on your map, Mist iterates them at a certain rate (IIRC updateAlive() iterates 1000/20 = 50 units every 1/5 second, or 250 units per second with 1000 units on the map). This is incredibly helpful for 99% of all missions - except those outlier missions that require thousands of units, and can thus overwhelm a lesser client (since it's part of the .miz it is run on every client). This is not a knock against Mist - far from it, it's the best DCS mission framework around - it's a reminder of the fact that for peak performance we need to understand the tools we use, and use them accordingly. TBH, dissecting Mist was one of the most fun and enlightening bits of learning DCS mission scripting. Highly addictive, so beware.

OK - that's a bit of a concern too. I was planning on having thousands of units on the map, but not activating them until they were 'needed' hoping that would help. I'm guessing this is going to be a bad idea, because with MIST (which is also in my mission as I use some modules such as CTLD and CSAR which depend on it) - it's going to create performance issues even though the units are inactive.

One alternatively I've thought about is recoding so that I just 'spawn' or 'dump' units dynamically onto the map. This will work OK for some basic stuff - but will limit what I'm wanting to do by customizing the units and their functions depending on the locations of the map. I'm beginning to think that the idea's I've had are bigger than Ben Hurr and may not be achievable with DCS now. 

Neat idea re the pathing property in the zones though to control how vehicles will behave in the said area. I'll have to give this some more thought, but in the end - I'm beginning to think that each direction I take, I'm going to hit walls:

If I go down the route of adding late-activated units (which I thought I was being cute with and would help)- I hit a problem with MIST and cycles once I get into the multiple thousands of units.

If I go down the route of dynamically spawning units in when I need them - I hit a problem with not being able to be 'particular' in what they're doing easily, plus needing to learn how to do this all again without using MOOSE to optimize CPU. 

And then there's the problem of having more A.I. controlled air units that can take off, land, refuel, etc without using Moose (so no AUFTRAG or AI_A2A_DISPATCHER, etc)

So maybe to start with I need to limit which units will actually move to avoid pathing CPU issues, and then limit what units have customised functions and just dump the same 'template' unit at various parts of the map and forget about doing the more involved individual customising stuff that I was planning on doing through the mission editor. 

Thanks so much for your replies. I was kinda hoping it would be easier to avoid CPU pitfalls but it seems as though it's far deeper than I first imagined and will be difficult to avoid unless I really become an expert in lua coding directly as opposed to using MOOSE.

Link to comment
Share on other sites

6 hours ago, Dangerzone said:

I have it executing external code - it's not in the mission file (or .miz file) at all.  I don't know if this means it only runs on the server, and whether or not that is an advantage?

This sounds slightly wrong to me. How do you invoke that external code? From inside the mission? If your code is referencing external .lua files from within a mission, than these missions are not portable, meaning they will only run on that exact computer they were put together on; they will not run as MP missions on clients other than your own. This is because the code inside the .miz runs on all clients, and when theses clients come to a part that wants to call mission-external code, it simply fails because that code doesn't exist on their machine. DCS has no arbitrary remote procedure call abilities (and you really don't want that either 🙂 ). A tell-tale if you are running mission or server code is this: are you invoking anything that involves Object, Unit, Group or trigger (*)? If so, that code is mission code and MUST be inside the .miz if you want it to run on other computers.

(*) with the exception of trigger.misc.xxxUserFlag() - these are available on the server side as well

If, on the other hand, you invoke code inside other .lua files from within code that runs in your /hooks folder, that's fine - because that code only runs on your machine - and therefore only needs to exist there.

Since it's convenient to develop with outside files, your way of building missions is quite common. It simply involves an additional 'build' step where you include all external files into an internal build (and change all external references to internal - this is usually done inside the mission with a variable)

 

6 hours ago, Dangerzone said:

So - since the .lua files are server-side only - does that mean only the server is executing them, and will that be helping to reduce CPU hits on the clients? 

We first need to establish that these really are server-side only files, and not merely external references that are cut when any other client is trying to execute them. If any invocation to those external files happens from within the .miz, they are not server-side-only. They are merely only-available-on-your-machine, similar to mods that everyone else must also install in the same place prior to running the mission. Currently, I think your mission will simply not run on other computers. That's simple to test, though: simply host the mission and have a friend connect and play together.

 

6 hours ago, Dangerzone said:

OK - that's a bit of a concern too. I was planning on having thousands of units on the map, but not activating them until they were 'needed'

If you have thousands of hand-placed units in your mission, you may want to re-visit the method how you create missions: you are likely quite inefficient during design, and definitely during execution. Spawning units dynamically not only makes mission design more efficient (I use zones that I place in ME for spawn zones, and have a script that spawns units inside the zone), it also takes significant load off the CPU for units that are idling or inactive (I have no knowledge of DCS's internal working, but simple expediency tells me that DCS regularly iterates all units that exist in the game). I also have implemented a spawn limiter into my mission scripts that simply stop (and queue) spawn requests until the number of living groups comes back into a manageable zone.

 

6 hours ago, Dangerzone said:

One alternatively I've thought about is recoding so that I just 'spawn' or 'dump' units dynamically onto the map.

Yes! Spend some time with this idea. DCS is quite unwieldly when it comes to spawning units, and you will lose much of your hair. But thinking spawning through, and developing a smart spawning and unit tracking system may not only be worth it, it can also be fun!

 


Edited by cfrag
  • Thanks 1
Link to comment
Share on other sites

15 hours ago, cfrag said:

This sounds slightly wrong to me. How do you invoke that external code? From inside the mission? If your code is referencing external .lua files from within a mission, than these missions are not portable, meaning they will only run on that exact computer they were put together on; they will not run as MP missions on clients other than your own.

Hi Cfrag,

Firstly - thanks so much for the time you've invested in getting back to me with your thoughts and help. 

Just to confirm - it definitely works on multiplayer, and they're definitely not included in the .miz file (nor installed on the clients side). 

I invoke the code simply with a single trigger that calls "DoScriptFile", and I have a load script file.

The script file that it imports has:

missionpath = "c:\\missionscripts\\activemission\\"
env.info("Loading Dependencies and scripts")
dofile(missionpath .. "Moose.lua")
dofile(missionpath .. "mist.lua")
dofile(missionpath .. "ctld.lua")
dofile(missionpath .. "csar.lua")
dofile(missionpath .."missionscheduler.lua")
dofile(missionpath .."missionevents.lua")
env.info("Scripts loaded")

... with the above files being available on the dedicated server only. This actually works a treat. I have a dedicated server which this runs on. It definitely only runs on the server (clients don't have access to any of these), and anything spawned in, triggers, etc definitely show on the clients that are connected. I've tested with over 5 different clients simultaneously connected to the dedicated server, and it works 'as expected'. 

The only problem I have identified (which is fully understandable, but a non-issue for me) is if clients try and replay their own track files. They simply won't work because they haven't got the scripts available locally to replicate - but as far as the actual live mission/gameplay goes - I can confirm this definitely works.

These scripts definitely use  Objects, Units, Groups as well. 

That's why I was wondering if this helps CPU usage by denying clients access, yet the events still execute on the server, and the end fruit of it's calculations still appear on the clients. (ie, new units spawned in, events trigger, etc).

I'm aware this would not work if I was to distribute the mission files for others to use (unless I distribute in such a way that they have access to all the files, and setup their servers the same), but at this stage my focus is more on trying to develop a large scale mission that by the end will have many thousand units that have populated the full map without CPU hits. If executing code externally from the .miz file on the server alone assists with this, I'm happy to use this method. 

Maybe I'm onto something here? 🙂  (Although I can't take credit for this - I've seen this used elsewhere, I just didn't think until I saw your comment that this could be a potential CPU saver on the client side)

15 hours ago, cfrag said:

If you have thousands of hand-placed units in your mission, you may want to re-visit the method how you create missions: you are likely quite inefficient during design, and definitely during execution. Spawning units dynamically not only makes mission design more efficient (I use zones that I place in ME for spawn zones, and have a script that spawns units inside the zone), it also takes significant load off the CPU for units that are idling or inactive (I have no knowledge of DCS's internal working, but simple expediency tells me that DCS regularly iterates all units that exist in the game). I also have implemented a spawn limiter into my mission scripts that simply stop (and queue) spawn requests until the number of living groups comes back into a manageable zone.

This is what I was afraid of. I think I'm going to have to really change the way I develop. This approach is OK for non-moving units such as non-directional SAM sites in the open- because I can create a template and then spawn them in places out in the open when I want. I guess I can also add some randomness to it too - as to where it spawns inside those zones. I've also been toying with both A2ADispatcher and AUFTRAG for spawning airborne units (although now wondering if I should revisit due to not using Moose for CPU optimization). But even so, I'm struggling to figure out to:

  • Handle spawning in units that I want specific pathing done (driving down a road, with turns at particular intersections, etc) in a particular manner,
  • Precision spawning. (Where I want a group where each unit is placed in a precise location such as a corner of a building, or under a tree, etc)
  • Direction of spawning. Such as Patriots or NASAM's that only have a 270° view
  • Spawning in units with specific parameters/attributes assigned. (Such as going back to waypoints, changing attributes at certain waypoints, etc) 

... basically all the more precision stuff that is far easier to just dump on a map. It seems like these would be a lot of extra work to do via script than via the mission editor, and I'd likely end up with hundreds (if not thousands) of zones to have that level of precision.

I'm beginning to think that what I'm wanting to achieve just isn't going to be possible with DCS at the moment, at least not without putting far more time development into it than I currently have available.  The glimmer of hope I have is that maybe the way I'm doing scripting means that it only executes on the server as it iterates through all the groups - and that may allow me to get away with a lot more than I normally could for now until I can come up with something better.

But all in all - it looks like you've confirmed for me - putting units down that are inactive is not going to help me with CPU cycles. (It may help, but it won't eliminate the problem) so I can't just cut sick with units in the mission editor. 


Edited by Dangerzone
Link to comment
Share on other sites

Hi guys, I don't have time to fully read the few post's above but want to clarify that it doesn't matter where the codebase lives, if it needs to be shared with clients it will be. Having the code outside the miz file is prob best practice in my opinion, it keeps the miz file size down and negates the need to constantly update code inside it. I and many other servers keep the code seperate to the miz file.

  • Thanks 1

Developer of XSAF :: A PVE with a custom GOAP/Learning AI model that wants to kill you

Two 24/7 servers and a regular structured op on the weekend

Discord: PravusJSB#9484   twitch.tv/pravusjsb  https://www.patreon.com/XSAF  https://discord.gg/pC9EBe8vWU

Link to comment
Share on other sites

7 hours ago, Dangerzone said:

But all in all - it looks like you've confirmed for me - putting units down that are inactive is not going to help me with CPU cycles. (It may help, but it won't eliminate the problem) so I can't just cut sick with units in the mission editor. 

 

Or can I....

I don't know if this is even possible, but I'm just thinking out loud.

Would it be possible to create a script that does the following:

  1. Iterates through every zone that is on the map, and obtains data for every unit that has been placed there (from the mission editor).
  2. Create a table and inserts ALL attributes of each group that's in each zone. (Unit name, type, attributes, waypoints... basically everything needed to recreate it).
  3. Destroy every group that has been added into the table
  4. Have a once off trigger for each zone that when a player enters that zone - all groups will be spawned using the table created in (2).

I have no idea if this is possible, but the idea would be to allow for very simple mission designing with thousands of units that don't impact the CPU much. The designer simply designs a mission like they would now, and then adds zones in various areas.  My idea is to do the hard work (scripting) once, and then have it as simple as possible for designing scenario's later on, being able to re-use the script as much as possible.

Does anyone know if this is possible (or better still), if it's been done before and being successful, or if this is a worthy thought worth pursuing?


Edited by Dangerzone
Link to comment
Share on other sites

1 hour ago, Dangerzone said:

I have no idea if this is possible, but the idea would be to allow for very simple mission designing with thousands of units that don't impact the CPU much. The designer simply designs a mission like they would now, and then adds zones in various areas.  My idea is to do the hard work (scripting) once, and then have it as simple as possible for designing scenario's later on, being able to re-use the script as much as possible.

That's certainly possible. There's an easier way, though. If you look at Mist, you'll see that it also reads the info on each and every unit in the mission. Mist, however (as always 🙂 )  goes about it smartly, and goes to the source: the mission info directly. It reads all unit info, including path, and stores it in a table. No reason you can't do it and adapt it to your needs. 

Another way is using a mixed approach: I usually place the "precise stuff", what is time- or location critical, by hand. And then I add 'mob' zones that spawn the rabble randomly inside zones. This keeps the mission randomized and still is held erect my the 'spine' of precisely placed units.

  • Thanks 1
Link to comment
Share on other sites

There's a lot of info in here now, and some of it is not quite right or misleading, I don't have time to point it all out but about spawning, some important info;

Dynamically spawning an aircraft is expensive, it's expensive at a moment in time but in terms of load on the mission it's transient and is not too much to worry about unless you're spawning more than one-two groups of 4 however, Ground Units for some reason are vastly more so, and they 'reverberate' CPU load for about 20-30 seconds as the units are shared with the clients, spawning 1 or 2 is OK sure, but it gets exponentially more expensive the higher you go and the more clients connected. Especially if there is more than a few units in the group. Consider, it's not only the units, their task, route etc that needs to be shared, but every single radar/datalink etc alive in your mission now needs to pick these up and figure out if they have LOS and know the them etc.

Secondly, you don't need much info to spawn a group so it's not hard to build up a DB of info to make a group on the fly.

Finally, the Mist framework is good for sure, but why make that DB of existing units when it already exists and isn't hard to iterate for example this is part of a tiny little class I have to retrieve ME declared assets;

        spmCore.side = { "neutrals", "red", "blue" }

        -- ME object name, side as an enum
        -- cats are string, plane, helicopter, ship, vehicle, static
        function spmCore.find(name,side,cat)
            if not side then return false end
            local data = env.mission.coalition[spmCore.side[side+1]]
            for j = 1,#data.country do
                if data.country[j][cat] then
                    local Data = data.country[j][cat].group
                    for i = 1,#Data do
                        if Data[i].name == name then
                            return deepCopy(Data[i])
                        end
                    end
                end
            end
            return false
        end

 

  • Thanks 1

Developer of XSAF :: A PVE with a custom GOAP/Learning AI model that wants to kill you

Two 24/7 servers and a regular structured op on the weekend

Discord: PravusJSB#9484   twitch.tv/pravusjsb  https://www.patreon.com/XSAF  https://discord.gg/pC9EBe8vWU

Link to comment
Share on other sites

15 hours ago, cfrag said:

That's certainly possible. There's an easier way, though. If you look at Mist, you'll see that it also reads the info on each and every unit in the mission. Mist, however (as always 🙂 )  goes about it smartly, and goes to the source: the mission info directly. It reads all unit info, including path, and stores it in a table. No reason you can't do it and adapt it to your needs. 

Another way is using a mixed approach: I usually place the "precise stuff", what is time- or location critical, by hand. And then I add 'mob' zones that spawn the rabble randomly inside zones. This keeps the mission randomized and still is held erect my the 'spine' of precisely placed units.

Oooh - I didn't realise that mist had all this information stored. I'll have to dig deeper into it. Thanks very much! This gives me some hope that I may be able to have my big missions after all. 🙂

I'll have to also look at how you do the mob spawning. IIRC I think you have some missions on the dcs website in the files section?

Link to comment
Share on other sites

6 hours ago, PravusJSB said:

There's a lot of info in here now, and some of it is not quite right or misleading, I don't have time to point it all out but about spawning, some important info;

Dynamically spawning an aircraft is expensive, it's expensive at a moment in time but in terms of load on the mission it's transient and is not too much to worry about unless you're spawning more than one-two groups of 4 however, Ground Units for some reason are vastly more so, and they 'reverberate' CPU load for about 20-30 seconds as the units are shared with the clients, spawning 1 or 2 is OK sure, but it gets exponentially more expensive the higher you go and the more clients connected. Especially if there is more than a few units in the group. Consider, it's not only the units, their task, route etc that needs to be shared, but every single radar/datalink etc alive in your mission now needs to pick these up and figure out if they have LOS and know the them etc.

Secondly, you don't need much info to spawn a group so it's not hard to build up a DB of info to make a group on the fly.

Finally, the Mist framework is good for sure, but why make that DB of existing units when it already exists and isn't hard to iterate for example this is part of a tiny little class I have to retrieve ME declared assets;

        spmCore.side = { "neutrals", "red", "blue" }

        -- ME object name, side as an enum
        -- cats are string, plane, helicopter, ship, vehicle, static
        function spmCore.find(name,side,cat)
            if not side then return false end
            local data = env.mission.coalition[spmCore.side[side+1]]
            for j = 1,#data.country do
                if data.country[j][cat] then
                    local Data = data.country[j][cat].group
                    for i = 1,#Data do
                        if Data[i].name == name then
                            return deepCopy(Data[i])
                        end
                    end
                end
            end
            return false
        end

 

Hi PravusJSB,

Thanks for your reply once again.

If I understand you correctly - spawning dynamically could actually be working against me because it's expensive for ground units? Does this mean I'd be better off having them all on the map at the start of the mission after all, but just set to inactive? (Knowing that they will still be included in any iterations). 

The idea behind making the DB of existing units is to have a cache, because the units would be destroyed.

ie:

At mission start:

1) Create a database / Cache ground units on the map (could be a few thousand)

2) Destroy ground units from map, so that clients connecting have very few ground units - good performance

3) When client flies into certain zone - recreate ground units for that zone only, using the database created in #1.

If I'm not mistaken (and I quite well could be - I'm very much a noob here 🙂), the code above would only obtain the units that still exist in the mission editor, which wouldn't be available when I get to step #3?

 

The reason why I was toying with this idea was to make it easier to design large missions with the ME but still 'cheat' and have it CPU friendly. I could use the mission editor almost exclusively. The script would automate the destroying and recreating of units based on what I did in the mission editor. (Maybe even to the point if I could make it 'good enough' that I could distribute the code so other people that don't know scripting could use it to make large missions friendly on the CPU cycles).

However - it seems from what you've said this will only move the burden on the CPU from one area to another, and not fix the issue.  If lots of units alive in the mission create problems, and dynamically spawning units in is also expensive on CPU, I'm not sure there's any other option remaining to achieve what I'm wanting to do.

Thanks for saving me from spending countless days (or weeks) on trying to get this to work and then finding that it doesn't help with CPU performance. I'll have to rethink and see if there's yet another approach. 

Link to comment
Share on other sites

No you get me all wrong, I'm not suggesting what to do, because looking at what you're taking away you're over complicating it. I'm just giving you the information about what's expensive to do. It doesn't mean don't do it, just be mindful and if you need to do it in excess then think about how to phase/schedule/cache/stack it out into the world for the least performance hit.

You should really avoid ME as much as possible, make your assets in there is you want/not sure how to make them on the fly without a reference, late activate them and then if you can design with code calling the assets when needed or If you don't know how to do that yet then for sure, design in the ME. Have it all late activated (spawning everything and deleting is mental) and then activate when needed.

 

EDIT: That code wasnt MIST data it was the mission data which is stored in 'env.mission', the same data MIST copies from to store again.


Edited by PravusJSB
  • Thanks 1

Developer of XSAF :: A PVE with a custom GOAP/Learning AI model that wants to kill you

Two 24/7 servers and a regular structured op on the weekend

Discord: PravusJSB#9484   twitch.tv/pravusjsb  https://www.patreon.com/XSAF  https://discord.gg/pC9EBe8vWU

Link to comment
Share on other sites

8 hours ago, Dangerzone said:

I'll have to also look at how you do the mob spawning. IIRC I think you have some missions on the dcs website in the files section?

Try one from the "pushback" series. This mission mixes ME pre-placed/pathed units (a few), and spawn zones (what I call 'owned zones' - they spawn units depending on who owns the zone, and path to the nearest enemy/neutral owned zone) to populate the map. The relevant modules are 'Commander', 'GroundTroops', and 'OwnedZones'. 

 

  • Thanks 1
Link to comment
Share on other sites

 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...