Jump to content

Designing a common Export.lua file


FSFIan

Recommended Posts

This thread exists to continue a discussion that started at post #14 in the DCS Integrator thread.

 

The goal is to get the developers of software that relies on an Export.lua file to come up with a single, shared Export.lua implementation.

 

The current situation

Currently, we have several pieces of software that use an Export.lua file to connect real hardware to DCS or to implement a glass cockpit:

(and I probably forgot a few, PM me and I'll add them to this list. Does anyone know if there is anything in the Russian community?)

 

Each of these comes with its own Export.lua file. In theory, the end user can mix and match multiple programs to get the best of all the worlds. In practice, this would cause a significant performance hit as the same data is exported several times and executing Lua blocks the simulation. It would also probably break down at some point because not all Export.lua files get all of the basics right, such as respecting previous callback functions and avoiding the use of non-unique global variables.

 

Where we want to go

1. Build a single Export.lua file that talks a protocol we can all agree on

2. Adapt our respective software packages to work with that

3. Happy end users!

 

If you are interested in working on this, please post your requirements and design ideas here.

 

I propose that we start gathering ideas in this thread and see how many devs we get on board. After about two weeks, we should try to get together on Skype or TS because some things can be solved much faster that way.

 

Later today, I'll post the requirements from my point of view and some lessons I learned from DCS-BIOS.


Edited by [FSF]Ian
Link to comment
Share on other sites

Ian;2508785']... not all Export.lua files get all of the basics right' date=' such as respecting previous callback functions and avoiding the use of non-unique global variables.[/quote']

 

I'm guilty I'm sure... doing the export stuff for DCS Integrator has been my first (and hopefully last!) experience of LUA - God knows why it's so popular! Horrendous script... ;) I've done the bare minimum to get something working, though I did take a few tips from DCS-BIOS in terms of chaining the export callbacks. :book:

 

My design principles would be:

1) Export only the information that the cockpit requires

2) Put the minimum load on the DCS host - per simulation-frame lua processing & coms in/out

3) Export the information in an efficient way - just because we shouldn't be wasteful

4) Inform target controllers (software/boards/etc) when an aircraft session starts & stops

5) Not require config files for aircraft - use what DCS already has to offer. I admit this may be naïve as I've only got 5 aircraft (one of which doesn't have a clickable cockpit) to try out.

 

I see points 1-3 as all related to the same aim - being as efficient as possible for both DCS (to protect frame rates) and the target controllers (so as not to swamp underpowered MCUs with loads of unnecessary data).

 

Things I've shamelessly *not* considered :music_whistling:

1) Coexistence with other solutions.

2) Exporting data to multiple end-points (as in my case, DCS Integrator acts as the marshaller to direct updates between DCS and a number of controllers (RS232,Ethernet) and indirectly RS485, and then, only the data that each controller is "subscribed" to.

 

That's my POV off the top of my head!

Link to comment
Share on other sites

I'll start with some lessons learned from DCS-BIOS:

  • Don't try to keep things simple enough for non-programmers to understand. Most of the design decisions I got wrong with DCS-BIOS were influenced by my overall design goal to keep it as simple as possible. I wanted to teach people how to fish.
    Turns out that people who are building a simpit are not interested in learning how everything works. Ain't nobody got time for that! (Especially if you are learning woodworking, machining, soldering, CAD, etc. at the same time and also, you know, building a simpit.)
    The reason I have any users at all is because of a happy accident of programmer laziness. I implemented the example code generation in the reference docs so I would have to do less typing. Suddenly you could get something going without understanding a thing about programming.
  • Don't try to make Export.lua do too much. I wanted to avoid a custom middleware to reduce the amount of programming languages that were involved (see above). I ended up with an export protocol that is good for microcontrollers to consume, but requires a moderately complex parser (as opposed to a simple key=value design that can be parsed with a few lines of C#/Python/insert your favorite high-level language here). The abstraction from DCS implementation details is a good thing to have, but also does not belong in Export.lua.
  • Documentation is important. I would have no users at all if it weren't for the step-by-step user guide (and WarHog's feedback on that to make it understandable to non-programmers). ArturDCS's Flightpanels software (to connect Saitek ProFlight panels) and BrassEm's VRI2DCSBIOS (VRInsight devices) would not exist without the protocol documentation.
  • Export.lua is not only about argument values. While most of the values we need to export are cockpit arguments, some special cases absolutely require custom Lua code (best example: the CDU display in the A-10C). This also means that we need a way to export string values (and it should be able to handle things other than ASCII, because we also have Russian planes).

 

For a common Export.lua, I propose the following architecture:

 

Transport layer

I think DCS-BIOS got this one right. Multicast UDP takes care of the common case of sending one data stream to multiple programs running on the local machine without requiring any additional processing in the DCS process. A TCP server provides an even easier way to connect at a slight performance penalty (I never tried to measure it, but I suspect it's not noticeable if you have less than a dozen clients or so), with the disadvantage that DCS has to be running before the consuming application can connect. Through a configuration file, you can optionally define a list of targets that should receive a unicast UDP stream.

 

Protocol

The Export.lua file keeps track of several values, each being identified by a key. Keys are strings. A cockpit argument is identified by its decimal representation, i.e. the Master Caution Light in the A-10C would have the key "404" and get the value GetDevice(0):get_argument_value(404). By writing an aircraft-specific extension to the Export.lua file, we can add other values with their own keys (e.g. add a function that knows how to read the A-10C CDU and export it as the "CDU" key, or as one key per line on the display).

 

By default, no data except the current aircraft type is being exported. When a new aircraft is entered, connected applications are notified. A connected application can send a list of keys it is interested in, which will result in those keys being added to the export data (i.e. a standard Observer pattern, with the difference that if there is at least one subscriber, the data goes to every connected subscriber so we can use multicast UDP).

 

I am still unsure how to handle an application that connects later on. One solution would be to constantly export the aircraft type even if it does not change, have the new application send its list of keys it wants exported, and re-send a value whenever a new application subscribes to its key.

 

Another problem here is when to stop exporting a certain key. Naturally there will be a reference count for each key and we can stop exporting a value as soon as no subscribers are left. If an application connects via TCP, we can unsubscribe it automatically. For UDP, it would have to explicitly unsubscribe. On the other hand, maybe this problem doesn't need to be solved (perfectly): in the worst case, we end up exporting some extra data until the next aircraft is entered.

 

For the input side, we should provide an API to call LoSetCommand(), GetDevice(dev):performClickableAction(cmd, arg), and GetDevice(dev):SetCommand(cmd, arg). For anything else (can't think of a use case right now), you'd need to write a Lua extension; the protocol should provide some way (maybe a generic event mechanism) to talk to such a custom extension.

 

Representation of export data

I would suggest using JSON, as there are fast parsers available for just about any programming language you can think of and it handles things like strings which have quotes or spaces in them.

 

I am not sure about the precision of floating point values. I'd suggest to export all floats with full precision.

 

Deciding on a case-by-case basis whether to use string.format("%f"), string.format("%.1f"), string.format("%.4f), or something else would reduce the amount of data we send to the network, but as we are dealing with a local network here, I assume it won't be a significant performance gain. It would also require all applications to agree on the right precision. We'll have to experiment with that.

 

License

Some variant of the GPL should work well for this. As you can see in the DCS-BIOS License, I am a fan of the Simple Public License, but I'd be happy with GPL (v2 or v3) too.

 

 

 

 

@ohall: I think that covers 1), 2) and 4) on your list. Efficient representation: I'd prefer saving CPU cycles in DCS.exe over saving network bandwidth. We are either sending data to the same machine or over a local network, so bandwidth shouldn't be an issue. Whenever bandwidth is restricted (such as on a serial port), there will be another program in between to translate to a more efficient representation anyway.

 

Regarding 5) -- not requiring configuration files for each aircraft -- this is mostly covered. If all you need is an argument number, simply subscribe to the key tostring(arg_number) and you'll get it. Same for the input side if all you need is to call LoSetCommand, performClickableAction and SetCommand. For the advanced stuff you'd need to write an add-on module in Lua. We could allow doing that dynamically over the network connection, but that would only lead to several applications adding their own way to export the FOO display, so we'd be wasting performance again.

 

Of course, the consuming application still needs to know which argument numbers are interesting and what they mean. IMO this is out of scope for this project, but as I previously mentioned, coming up with a machine-readable format to document this stuff would make a good sister project.

 

 

@everyone:

Please tell me what you think. I wrote a proposal for a concrete implementation instead of a list of requirements because I think this makes it easier to talk about by answering the question "is there anything I would do differently and why?". The answer to that "why" is then a requirement I have missed.

 

We'll also need a name at some point. I'd suggest "DCS-BIOS 2.0", because that is what DCS-BIOS was intended to be all along, but it would probably cause too much confusion because most people seem to equate "DCS-BIOS" with "The DCS-BIOS Arduino Library".

Link to comment
Share on other sites

Hello everybody,

I am Michael, the lead programmer from DCS ExportScript.

 

First, my English is not so good, so there is the information only in short form.

 

Our software package is more than a export script.

We have a export script, that support HawgTouch or HELIOS, D.A.C, a middleware that support the Arcaze USB controller and moduls, and it is open to other software.

D.A.C. is a middleware for Arcaze USB controller and modules, how LED driver 2/3 and Display Driver 32 (7 segment displays).

HT, a extended Gauges Pack DLL for HawgTouch.

 

Our requirements:

- simple Key/Value format, with reproducible key name (potential also JSON)

- send the data via UDP to many clients (UDP Multicast?)

- easy to expand for new DCS modules (without programming experience)

- receive and process data for input side

- compatible as possible with specialized export scripts for Aris Radio System, DCS Simple Radio, Tac View or without export script as UltraMFCD

- ...

 

Idea:

For a simple key name I would suggest the argument ID of DCS

The ID is always in mainpanel_init.lua described and per modul explicit.

 

Pro receiver it should be possible to send your own / specific data.

 

A modular design of the script, so it can be easily expanded.

Export.lua (as basic file) + config.lua (for config ;-)) + basic.lua (for functions for processing the data) + a special file per modul (all exportable ID + special functions for the module). Then, every so hopefully create a file when it looks at the files of the module (mainpanel_init.lua, device.lua, clickable.lua).

 

For input side, GetDevice(dev):performClickableAction(cmd, arg) for clickable cockpits, LoSetCommand() for Flaming Cliffs Airplanes or similar modules

 

License:

clearly opensource

But it must be the LGPL, the export script is integrated into DCS and DCS is not unter the GPL license.

 

 

@Ian:

what is the difference between GetDevice(dev):SetCommand(cmd, arg) to GetDevice(dev):performClickableAction(cmd, arg)

 

Form ohall's list

1) We do export all cockpit data pro module and all needed data pro Flaming Cliff Airplane.

2) We work with LUA coroutines for a minimum load.

3) We will send the data when the have changed.

4) We will send the name of the module when the module is selected and a stop signal when the simulation is terminated.

This also works in multiplayer, if you between the aircraft changes.

5) If you wish to have the exact dates of the cockpit, then you must also accurately read the data from the instruments.

It only works with specific config scripts.

Simpit Software by SDA "SIMPIT DEVELOPER ASSOCIATION"

  • DCS ExportScript
  • D.A.C. DCS to Arcaze Communicator
  • Ikarus a new Virtual Cockpit Software

Deutscher Forums Thread

English Forums Thread

 

Hard/Software: AMD Ryzen 7 1800X, 32 GiB RAM, extra SSD for Windows 10 and DCS World, AMD Vega Frontier Edition with 16 GiB VRAM

Link to comment
Share on other sites

Hello all !

I just see this thread and i found it very interesting.

 

As you may know, i create Helios profiles for diferents airplanes.

Due to the limitations of Helios avaiable interfaces, (a10c, ka50 and FC) i need to work in the export.lua everytime i make a new profile for a specific airplane, like P51, Mi-8, Su27 etc.

Till now, i am trying to mantaing a export.lua capable of work width all my profiles, but everyday this export.lua is growing in size.

 

Normaly i check what kind of airplane we are flying, and then i run a specific function for that airplane, where i collect all the data i need to send to Helios.

 

One of the things i notice is the fact that several info is captured several times for different purposes, for example, i need the information of the radios freq and channels, so i get this info from the devices, but later, Ptars ( a comunications sofware we use in our scuadron) do exactly the same. So this info is captured two times.

 

 

if you need my help for something just let me know. My background and career is a Lead Artist, but i have some programming experience aswell.

Link to comment
Share on other sites

  • 3 weeks later...

I have written a first prototype: https://github.com/jboecker/dcs-export-core

 

See the README on GitHub for setup and usage instructions.

 

It doesn't do custom aircraft-specific Lua functions yet, that's what I will add next (it will load an aircraft-specific Lua file and if that file contains a "foo" function, you will get the output of that function through the "lFoo" key or something like that).

 

You can also export draw arguments for the external aircraft model. This allows you to get the angles of all control surfaces, and AFAIK also the status of external lights and information about damage to individual parts. To see what different draw model arguments do, open the aircraft model (e.g. "Bazar\World\Shapes\A-10.edm") in the ED Model Viewer, select the argument numer you want in the "Arg #" box and move the slider around.

Link to comment
Share on other sites

  • 1 year later...

Just poking around on the forums doing some searches today and I ran across this. I'm the developer of iControl DCS (iOS) and DCS Virtual Cockpit (Android)

 

I would certainly be interested and willing to contribute. Especially with upcoming aircraft that everyone is going to want to use.

 

Unfortunately GPL licensing would kill it for me since I'm not going to open source my apps.

 

Although I suppose I could only open source my server application and that's all that'd have to distribute the common lua.

 

I like what you're doing with DCS-BIOS. Are you planning on sticking with that or is there something else on the horizon?

iControl DCS/DCS Virtual Cockpit - Full featured iPad Cockpit - Now with Android support!

A10 Virtual Cockpit Free - Free limited functionality version of iControl DCS!

DCS Virtual Cockpit - Android version!

 

Follow on Twitter for all the latest news

Link to comment
Share on other sites

You don't have to open source your app just because it's talking to someting GPL-licensed over the network. Otherwse you couldn't use Microsoft Edge to talk to a GPL-licensed web server. That said, I have no problems with changing the license to MIT.

 

I do plan to continue work on DCS-BIOS (version 2.x will be based on DCS Export Core). Development progress has slowed to a crawl right now as I am studying to get a CCNA and looking for a new job, but I hope to pick up speed again in a few months.

 

For DCS Export Core, here is what I want to add to get to a stage where aircraft with clickable cockpits are fully supported:

 

  • An action to call LoSetCommand(x) or LoSetCommand(x, y)
  • An action to call GetDevice(id):performClickableAction(command, arg)
  • Export keys to get the output of list_indication(i), e.g. "i7" would be list_indication(7)

 

I haven't thought about FC3 aircraft that much yet. One option would be to add a key type beginning with "f" to call a function and get the resuts (e.g. "fLoGetIndicatedAirspeed") and then have a whitelist of allowed functions.

 

The design goal here is that DCS export core should contain only as much knowledge about DCS internals as it absolutely needs to have and should do a minimum amount of processing or interpretation of data. When a new clickable cockpit aircraft is released, it should not require an update to DCS Export Core.

 

We also have to think about how to distribute the DCS Export Core. We need an installer that can be bundled with other applications that depend on DCS Export Core, that will not install over a newer, already existing version, and that allows the user to select one or multiple DCS installations to install to.

 

This means we also need a detection mechanism to find DCS installations. Looking for folders that match the pattern %USERPROFILE%\Saved Games\DCS(\..*)? and checking if they look like a DCS user data folder (e.g. have "Missions" and "Logs" subfolders) is probably good enough.

 

At least with the new GameGui* mechanism, the installer won't have to edit any files, it only has to copy them to the right place (and to uninstall, just delete DcsExportCoreGameGui.lua and the DcsExportCore subfolder).

 

Note that I am also planning to drop support for UDP. It's a great idea in theory and has the advantage that the order of starting DCS and the client application does not matter, but in practice it caused too many problems with firewalls. Also, with the JSON data format, an update that includes the complete state (such as after jumping into a new aircraft) will be too large to fit into a single UDP packet.

If a client application has been started before DCS, it will just have to periodically try to reconnect.

Link to comment
Share on other sites

Well I just lost a big reply since my session timed out while I was typing it and then the browser didn't redirect properly :( Let's try this again.

 

If I distributed the script with my server installer I think I'd be bound by the license, no? Either way, MIT would totally solve the issue completely.

 

I'm not real familiar with what changes are happening for 2.x since I've been out of the loop for quite a while and now I'm ramping back up to make some pretty big changes to my apps. Is there a good resource for what's new/changed in 2.x?

 

Versioning of scripts is interesting especially since multiple could be installed at the same time as the end user might have several applications that depend on different versions of the protocol, etc. You'd have to have a pretty lightweight listener mode until a client connected and then do all the heavy lifting.

 

I'm not a master at LUA, I've only absorbed what I needed to in order to get what I needed to done. I do have lots of winforms/wpf .net experience and could definitely help write some sort of script version manager if you wanted to go that route for managing versions of scripts that are installed. I already have written code for detecting and validating save games/dcs install directories in my server application which could be useful.

 

If you're not going to use UDP what protocol are you planning on? TCP?

 

The overall concept of DCS-BIOS and/or what you're talking about here would be a huge boon for me so I'm totally willing to contribute where I can rather than perpetuating yet another script that does the same thing as others!

iControl DCS/DCS Virtual Cockpit - Full featured iPad Cockpit - Now with Android support!

A10 Virtual Cockpit Free - Free limited functionality version of iControl DCS!

DCS Virtual Cockpit - Android version!

 

Follow on Twitter for all the latest news

Link to comment
Share on other sites

  • 2 weeks later...

+1 for json

[sIGPIC][/sIGPIC]

 

Creator of:

 

F-18C VFA-195 "Dambusters" 1998 CAG Livery

https://forums.eagle.ru/showthread.php?t=213788

 

F-18C VFA-195 "Dambusters" July 2001 CAG Livery

https://forums.eagle.ru/showthread.php?t=215950

 

Pilot avatars for DCS Logbook

https://forums.eagle.ru/showthread.php?t=221160

 

How to make a DCS A-10C Panel

http://forums.eagle.ru/showthread.php?t=65998

Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...