derammo Posted June 20, 2013 Share Posted June 20, 2013 Hi all. I am new to this platform and so I am soliciting ideas/experience. I was trying to diff two mission files to see what had gone wrong, breaking the product. My first approach was this: expand miz files convert each LUA table to a canonical form, meaning all fields are sorted and pretty printed the same way That looked something like this: #!/bin/bash function canonicalize { lua - $1 <<EOF local serpent = require("serpent") local text = assert(loadfile(arg[1]), "failed to read "..arg[1]) local value = string.gsub(arg[1], '.*/', '') text() local formatted = assert(serpent.block(_G[value], {comment = false})) print("rewriting "..arg[1].." in canonical form") io.output(arg[1]) io.write(value.." = ") io.write(formatted) EOF } for MISSION in *.miz ; do BASE=${MISSION%.miz} mkdir -p "$BASE" pushd "$BASE" unzip -o "../$MISSION" for FILE in `find . -type f -not -name "*.*"` ; do canonicalize $FILE ; done popd done Then I was able to recursively diff. Now here is my problem: normal LUA pretty printers (I used the LUA rock 'serpent') require the table you want to pretty print be in a table variable. So that means my script reads the tables from the files. This is equivalent to executing LUA on my machine that I didn't write myself. Note the invocation of text() to get the table file contents into a variable. Anyone could put anything they want in their mission file and I would end up running it on my machine. You see, in this scenario, the LUA is executed in an unrestricted LUA interpreter, not some embedded one in the game. How do people deal with this? Is there a non-LUA parser/formatter for LUA tables that doesn't require executing the source? I realize this is a direct result of using a script LANGUAGE to store DATA, and I am taking short cuts by just running it. But there must have been other projects that handled this. I have seen some LUA code that attempts to read a table without executing anything, is that what you use? Cheers and thanks for reading this long-@$$ post ;) Link to comment Share on other sites More sharing options...
derammo Posted June 20, 2013 Author Share Posted June 20, 2013 ok I got a pointer from somewhere else, so here is a new proposed script (simply removing ALL LUA functions from the calling environment): requirements: LUA 5.1 luarocks lua rock "serpent" bash unzip #!/bin/bash function canonicalize { lua - $1 <<EOF local serpent = require("serpent") local text = assert(loadfile(arg[1]), "failed to read "..arg[1]) local value = string.gsub(arg[1], '.*/', '') setfenv(text, {}) text() local formatted = assert(serpent.block(_G[value], {comment = false})) print("rewriting "..arg[1].." in canonical form") io.output(arg[1]) io.write(value.." = ") io.write(formatted) EOF } for MISSION in *.miz ; do BASE=${MISSION%.miz} mkdir -p "$BASE" pushd "$BASE" unzip -o "../$MISSION" for FILE in `find . -type f -not -name "*.*"` ; do canonicalize $FILE ; done popd done Do you think this is safe? It worked fine in a simple test but I am no LUA expert. Link to comment Share on other sites More sharing options...
daribouca Posted June 20, 2013 Share Posted June 20, 2013 I am not sure I got your point exactly, but when I needed to parse and then alter "mission" files LUA table, I pushed them into an ordered Python dictionary, without running any piece of LUA code. Then changes can easily be made programmatically, and re-injected into the mission file, or spawn any number of new tweaked *.MIZ as necessary. But you could do pretty much whatever you want with the resulting data, very easily. I chose Python because of it being widely available on almost every machine/server. If you're interested, I already made the parser, along with all the *.MIZ file handling. You'd just need to tailor it down to your specific needs. Link to comment Share on other sites More sharing options...
derammo Posted June 20, 2013 Author Share Posted June 20, 2013 Daribouca: Let me try to re-state my point then: I was trying to canonicalize the LUA tables, so that tables from two different miz files can be compared, regardless of the order of fields and escaping/quoting/formatting used originally. Sucking the whole thing into python dictionaries and then formatting back out in canonical form would presumably accomplish this. But you are then relying on your own implementation of any escaping that LUA might use. So yeah, you could do that. But I was asking about a more direct way in LUA, and whether the answer I gave for that (see the LUA code embedded in the bash script above?) is sane from a security perspective. Link to comment Share on other sites More sharing options...
Speed Posted June 20, 2013 Share Posted June 20, 2013 (edited) Anyone could put anything they want in their mission file and I would end up running it on my machine. You see, in this scenario, the LUA is executed in an unrestricted LUA interpreter, not some embedded one in the game. How do people deal with this? Is there a non-LUA parser/formatter for LUA tables that doesn't require executing the source? I realize this is a direct result of using a script LANGUAGE to store DATA, and I am taking short cuts by just running it. But there must have been other projects that handled this. I have seen some LUA code that attempts to read a table without executing anything, is that what you use? If you need to dostring on untrusted code, you make a sandboxed environment for that code, and run it in there. do local untrustedLua = <some string data from where ever, that you don't necessarily trust> local env = {} --[[ now, either add to env all the functions you think are safe, or copy everything from _G into env except those functions/libs you think are unsafe. It is probably better to manually add references to safe functions/libs to env rather than copy over _G and then remove unsafe refereneces. That way, you are not dependent on what else _G might or might not contain. ]] --[[in this example, I copy a few SAFE funcitons/libraries from _G into env that are safe. Do not consider this a complete set of safe functions! Google "Lua Sandboxing" for a more complete list!]] env.print = print env.string = string env.table = table env.math = math env.loadstring = loadstring -- etc. local f = loadstring(untrustedLua) --now, compile the untrusted code if f then -- the untrusted Lua compiled successfully. setfenv(f, env) -- Switches the environment the compiled, untrusted code will run in from _G to our new, safe environment. pcall(f) -- now, run the unsafe code in our safe environment, where it cannot do any damage. -- Also note the use of pcall. pcall prevents runtime errors in the untrusted Lua code from causing a -- Lua error. end end Edited June 20, 2013 by Speed Intelligent discourse can only begin with the honest admission of your own fallibility. Member of the Virtual Tactical Air Group: http://vtacticalairgroup.com/ Lua scripts and mods: MIssion Scripting Tools (Mist): http://forums.eagle.ru/showthread.php?t=98616 Slmod version 7.0 for DCS: World: http://forums.eagle.ru/showthread.php?t=80979 Now includes remote server administration tools for kicking, banning, loading missions, etc. Link to comment Share on other sites More sharing options...
derammo Posted June 20, 2013 Author Share Posted June 20, 2013 (edited) [edit] sorry Speed, I just realized that you replied to my first post, which didn't have the sandbox. I believe that we are on the same page.[/edit] I already do that in the code I presented above. I didn't pcall because I WANT to abort if anything errors in the untrusted code, so I don't overwrite the file in that case. Edited June 20, 2013 by derammo Link to comment Share on other sites More sharing options...
Speed Posted June 20, 2013 Share Posted June 20, 2013 (edited) [edit] sorry Speed, I just realized that you replied to my first post, which didn't have the sandbox. I believe that we are on the same page.[/edit] I already do that in the code I presented above. I didn't pcall because I WANT to abort if anything errors in the untrusted code, so I don't overwrite the file in that case. Ummm... is that pseudocode you posted? Anyway, without pcall, the Lua calling the untrusted code will crash. I donno why you would ever want to do that. It's better if you just test whether or not the first value returned by pcall was true or false. pcall returns true (plus any values returned by the function) if the function it calls executes without error. If there is an error, pcall returns false, plus an error message string that tells what the error was. Edited June 20, 2013 by Speed Intelligent discourse can only begin with the honest admission of your own fallibility. Member of the Virtual Tactical Air Group: http://vtacticalairgroup.com/ Lua scripts and mods: MIssion Scripting Tools (Mist): http://forums.eagle.ru/showthread.php?t=98616 Slmod version 7.0 for DCS: World: http://forums.eagle.ru/showthread.php?t=80979 Now includes remote server administration tools for kicking, banning, loading missions, etc. Link to comment Share on other sites More sharing options...
derammo Posted June 21, 2013 Author Share Posted June 21, 2013 (edited) I see our difficulty at communicating now :). This is definitely not pseudo code, it runs fine. I work on Linux in bash shell, and this is a bash script that contains and runs a LUA script. The bash script iterates all the .miz files and creates a directory for each MIZ into which it is unzipped. For each such directory, it iterates all the files that have no file extension (these are the tables, rather than the LUA scripts.) Then it calls a bash function for each such file. This function is declared in the same file. It simply calls the command line lua interpreter, feeding it a LUA script that is also embedded in the script. If anything goes wrong in reading the table, lua exits with an error and it does not get to rewrite the file. If it correctly formats pretty printed and sorted table output, then it overwrites the original table file with that. When its all done you have a directory for each .miz file in the current directory, containing canonical form tables, which can therefore be compared since all the fields are in the same order. Here's the code again with markup to follow along [edited to correct LUA script, previous version was not doing anything]: #!/bin/bash # this comment means linux automatically runs this script in bash if executed # a bash shell function function canonicalize { # run LUA command interpreter, taking script from standard input (-) and passing name of table # << EOF means replace standard input with contents of this file from here until # you see EOF again lua - $1 <<EOF -- beginning of LUA code -- load "serpent" LUA rock, which is a pretty printer local serpent = require("serpent") -- load the table or crash, returning to iteration of files in bash local text = assert(loadfile(arg[1]), "failed to read "..arg[1]) -- remove path to get name of table variable we expect in the file local value = string.gsub(arg[1], '.*/', '') -- disable ALL LUA functions setfenv(text, {}) -- invoke table code, which will just define that one table variable text() -- retrieve expected variable from sandbox environment local loaded = getfenv(text)[value] -- make sure we loaded something, so we don't serialize nil assert(loaded) -- serialize canonical LUA local formatted = assert(serpent.block(loaded, {comment = false})) -- comment print("rewriting "..arg[1].." in canonical form") -- since we didn't crash, replace the original file io.output(arg[1]) -- add the assignment back since this table file is supposed to create this table var io.write(value.." = ") -- write the pretty table io.write(formatted) -- end of code piped into LUA EOF # and now we are back in bash } # iterate all *.miz files for MISSION in *.miz ; do # get the name of the miz file without the extension BASE=${MISSION%.miz} # use this as a directory name mkdir -p "$BASE" # change to that directory, remembering where we come from pushd "$BASE" # unzip the MIZ unzip -o "../$MISSION" # for all files without a file extension for FILE in `find . -type f -not -name "*.*"` ; do # call canonicalize function to rewrite it, assuming it is a LUA table canonicalize $FILE ; done # return to directory we came from popd done Edited June 28, 2013 by derammo code was incorrect Link to comment Share on other sites More sharing options...
Recommended Posts