I'm trying to convert a saved Lua table into something I can parse more easily for inclusion on a web page. I'm using Lua for windows from code.google's luaforwindows. It has included in it this harningt's luajson for handling this conversion. I've been able to figure out how to load in the contents of the file. I get no errors, but the "json" it produces is invalid. it just encloses the entire thing in quotes and adds \n and \t. The file I'm reading is a .lua file, which follows the format:
MyBorrowedData = {
["first"] = {
["firstSub"] = {
["firstSubSub"] = {
{
["stuffHere"]="someVal"
},
{
["stuffHere2"]="some2Val"
},
},
},
},
}
Note the , following the final item in each "row" of the table, is that the issue? Is it valid Lua data? I feel like given the output, Lua is unable to parse the table when I read it in. I believe this even more when I try to just require the lua data file, and I seem to be unable to iterate through the table manually.
Can anyone tell me if it's a bug in the code or poorly formatted data that's causing the issue?
The Lua lifting is easy:
local json = require("json")
local file = "myBorrowedData.lua"
local jsonOutput = "myBorrowedOutput.json"
r = io.input(file)
t = io.read('*all')
u = io.output(jsonOutput)
s = json.encode(t)
io.write(s)
You're reading the file as plain text, and not loading the Lua code contained inside of it. t becomes a string with the Lua file's contents, which of course serializes to a plain JSON string.
To serialize the data in the Lua file, you need to run it.
local func = loadstring(lua_code_string) -- Load Lua code from string
local data = {} -- Create table to store data
setfenv(func, data) -- Set the loaded function's environment, so any global writes that func does will go to the data table instead.
func() -- Run the loaded function.
local serialized_out = json.encode(data)
Also, ending the last item of a table with a comma (or semicolon) is perfectly valid syntax. In fact, I recommend it; you don't need to worry about adding a comma to the former last object when adding new items to the table.
Related
I'm setting up a snowpipe to load data from s3 bucket to snowflake schema.
S3 contains files in NDJOSN format. One file can contain multiple records and I want to process all of them. Even if one record is broken.
To do so, I need to add on_error='continue' option to pipe creation and use csv file format as stated in official snowflake docs here.
That way I receive raw strings of JSON that I need to parse to access data. And since snowpipes do not support nested selects the only way to do that is to parse it for each column individually.
Resulting in this copy statement:
copy into MY_TABLE
from (select parse_json($1):id, parse_json($1):name, parse_json($1):status
from #MY_STAGE_CONNECTED_TO_S3)
on_error = 'continue'
This code needs to parse json 3 times for each row.
I have a table with ~40 col's and for it, this query works ~5 times slower than a more straightforward solution that uses file format option to parse JSON but unfortunately does not support on_error=continue option.
copy into HQO_DEVELOPMENT.PUBLIC.DIM_TENANT
from (select $1:id, $1:name, $1:status
from #HQO_DEVELOPMENT.PUBLIC.DIM_TENANT_STAGE_NN)
file_format = (type = 'json')
What was tried
Using nested select like this: : is not supported
copy into HQO_DEVELOPMENT.PUBLIC.DIM_TENANT from
(select $1:id, $1:name, $1:status from (
select parse_json($1) from
#HQO_DEVELOPMENT.PUBLIC.DIM_TENANT_STAGE))
on_error = 'continue'
Using JSON type on stage and omit on the pipe: won't help the issue
Is there a way to have the benefit of on_error=continue and don't parse JSON for every column?
Snowflake documentation says: Both CSV and semi-structured file types are supported; however, even when loading semi-structured data (e.g. JSON), you should set CSV as the file format type (default value). You can use the corresponding file format (e.g. JSON), but any error in the transformation will stop the COPY operation, even if you set the ON_ERROR option to continue or skip the file.
https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html
On the other hand, I see on_error option works for NDJSON files, at least when you set file type on stage level. For testing,
I created the following NDJSON file for testing:
{ "id":1, "name":"Gokhan", "location":"Netherlands" }
{ "id":2, "name":"Hans", location:Germany -- broken json #1 }
{ "id":3, "name":"Joe", "location":"UK" }
{ broken json #2 }
{ "id":4, "name":"Mike", "location":"US" }
I created a file type object, and created a stage using this file type (you can alter your existing stage and set a file type):
CREATE FILE FORMAT myformat TYPE = json;
CREATE STAGE mystage FILE_FORMAT = (FORMAT_NAME = myformat);
I uploaded my sample NDJSON file to this stage, created a snowpipe to load it:
CREATE PIPE mypipe AS
COPY INTO mytable
FROM (SELECT $1:id, $1:name, $1:location FROM #mystage)
ON_ERROR = CONTINUE;
When I refreshed the pipe, it successfully loaded the "valid" (3) records from the file.
After writing a few files for saving in my JSON file in Godot. I saved the information in a variable called LData and it is working. LData looks like this:
{
"ingredients":[
"[KinematicBody2D:1370]"
],
"collected":[
{
"iname":"Pineapple",
"collected":true
},{
"iname":"Banana",
"collected":false
}
]
}
What does it mean when the file says KinematicBody2D:1370? I understand that it is saving the node in the file - or is it just saving a string? Is it saving the node's properties as well?
When I tried retrieving data - a variable that is assigned to the saved KinematicBody2D.
Code:
for ingredient in LData.ingredients:
print(ingredient.iname)
Error:
Invalid get index name 'iname' (on base: 'String')
I am assuming that the data is stored as a String and I need to put some code to get the exact node it saved. Using get_node is also throwing an error.
Code:
for ingredient in LData.ingredients:
print(get_node(ingredient).iname)
Error:
Invalid get index 'iname' (on base: 'null instance')
What information is it exactly storing when it says [KinematicBody2D:1370]? How do I access the variable iname and any other variables - variables that are assigned to the node when the game is loaded - and is not changed through the entire game?
[KinematicBody2D:1370] is just the string representation of a Node, which comes from Object.to_string:
Returns a String representing the object. If not overridden, defaults to "[ClassName:RID]".
If you truly want to serialize an entire Object, you could use Marshalls.variant_to_base64 and put that string in your json file. However, this will likely bloat your json file and contain much more information than you actually need to save a game. Do you really need to save an entire KinematicBody, or can you figure out the few properties that need to be saved (postion, type of object, ect.) and reconstruct the rest at runtime?
You can also save objects as Resources, which is more powerful and flexible than a JSON file, but tends to be better suited to game assets than save games. However, you could read the Resource docs and see if saving Resources seems like a more appropriate solution to you.
I need to convert a .CSV to a specific .JSON format, and I decided to use the csvtojson package from NPM since it seemed to be designed for this sort of thing.
First, a little background on my problem. I have .CSV data that looks similar to this:
scan_type, date/time, source address, source-lat, source-lng, dest address, dest-lat, dest-lng
nexpose,2016-07-18 18:21:44,1008,40.585260,-10.124120,10.111.131.4,10.844880,-10.933360
I have a plain csv-to-json converter here , but there is a problem. It outputs a rather plain file, and I need to split the source/destination, and add some special formatting that I will show later on.
[
{
"scan_type": "nexpose",
"date/time": "2026-07-28 28:22:44",
"source_address": 2008,
"source-lat": 10.58526,
"source-lng": -105.08442,
"dest_address": "11.266.282.0",
"dest-lat": 11.83388,
"dest-lng": -111.82236
}
]
The first thing I need to do is be able to separate the "source" values, from the "destination" values. Here is an example of what I want the "source" values to look like:
(var destination will be exactly the same format)
var source={
id: "99.58926-295.09492",
"source-lat": 49.59926,
"source-lng": -209.98942,
"source_address": 2009,
x: {
valueOf: function() {
var latlng=[
49.58596,
-209.08442
];
var xy = map.FUNCTION_FOR_CONVERTING_LAT_LNG_TO_X_Y(latlng);
return xy[0]; //xy.x
},
y: {
valueOf: function(){
varlatlng=[
49.58596,
-209.08442
];
var xy = map.FUNCTION_FOR_CONVERTING_LAT_LNG_TO_X_Y(latlng);
return xy[1]; //xy.y
}
}
So, my question is, how should I approach converting my data? Should I convert everything with csvtojson? Or should I convert it from the plain .JSON file I generated?
Does anyone have any advice, or similar examples, they could share on how to approach this problem?
I do a lot of work with parsing CSV data and as I am sure you have seen, CSV is very hard to parse and work with correctly as there are a huge number of edge cases that can break even the most rugged of parsers (although it looks like your dataset is fairly plain so that isn't a huge concern). Not to mention you could potentially run into corruption by performing operations while reading from disk, so it is a much better idea to get the data from CSV into a JSON file and then make any manipulations to a JSON object loaded from that "plain" JSON file.
tl;dr: convert your data from the plain .JSON file
I'm using webwrite to post to an api. One of the field names in the json object I'm trying to setup for posting is odata.metadata. I'm making a struct that looks like this for the json object:
json = struct('odata.metadata', metadata, 'odata.type', type, 'Name', name,);
But I get an error
Error using struct
Invalid field name "odata.metadata"
Here's the json object I'm trying to use in Matlab. All strings for simplicity:
{
"odata.metadata": "https://website.com#Element",
"odata.type": "Blah.Blah.This.That",
"Name": "My Object"
}
Is there a way to submit this json object or is it a lost cause?
Field names are not allowed to have dots in them. The reason why is because this will be confused with accessing another nested structure within the structure itself.
For example, doing json.odata.metadata would be interpreted as json being a struct with a member whose field name is odata where odata has another member whose field name is metadata. This would not be interpreted as a member with the combined field name as odata.metadata. You're going to have to rename the field to something else or change the convention of your field name slightly.
Usually, the convention is to replace dots with underscores. An automated way to take care of this if you're not willing to manually rename the field names yourself is to use a function called matlab.lang.makeValidName that takes in a string and converts it into a valid field name. This function was introduced in R2014a. For older versions, it's called genvarname.
For example:
>> matlab.lang.makeValidName('odata.metadata')
ans =
odata_metadata
As such, either replace all dots with _ to ensure no ambiguities or use matlab.lang.makeValidName or genvarname to take care of this for you.
I would suggest using a a containers.Map instead of a struct to store your data, and then creating your JSON string by iterating over the Map filednames and appending them along with the data to your JSON.
Here's a quick demonstration of what I mean:
%// Prepare the Map and the Data:
metadata = 'https://website.com#Element';
type = 'Blah.Blah.This.That';
name = 'My Object';
example_map = containers.Map({'odata.metadata','odata.type','Name'},...
{metadata,type,name});
%// Convert to JSON:
JSONstr = '{'; %// Initialization
map_keys = keys(example_map);
map_vals = values(example_map);
for ind1 = 1:example_map.Count
JSONstr = [JSONstr '"' map_keys{ind1} '":"' map_vals{ind1} '",'];
end
JSONstr =[JSONstr(1:end-1) '}']; %// Finalization (get rid of the last ',' and close)
Which results in a valid JSON string.
Obviously if your values aren't strings you'll need to convert them using num2str etc.
Another alternative you might want to consider is the JSONlab FEX submission. I saw that its savejson.m is able to accept cell arrays - which can hold any string you like.
Other alternatives may include any of the numerous Java or python JSON libraries which you can call from MATLAB.
I probably shouldn't add this as an answer - but you can have '.' in a struct fieldname...
Before I go further - I do not advocate this and it will almost certainly cause bugs and a lot of trouble down the road... #rayryeng method is a better approach
If your struct is created by a mex function which creates a field that contains a "." -> then you will get what your after.
To create your own test see the Mathworks example and modify accordingly.
(I wont put the full code here to discourage the practice).
If you update the char example and compile to test_mex you get:
>> obj = test_mex
obj =
Doublestuff: [1x100 double]
odata.metadata: 'This is my char'
Note: You can only access your custom field in Matlab using dynamic fieldnames:
obj.('odata.metadata')
You need to use a mex capability to update it...
I am having problems with the JSON file in my Corona game. Basically, the game gives you trophies (cards) when you reach certain points. The card information is then written into a JSON file. When you start the game, it checks if the file "playerCards.json" already exists, and if not, it creates such file with the following structure:
{"common":[],"uncommon":[],"rare":[]}
Later in the game, the player finally receives one card. The (simplified) code below runs:
local category = "common"
local rdm = math.random(1,20)
array = loadFile("playerCards.json")
array[category][rdm] = collection[category][rdm]
writeFile("playerCards.json", array)
Collection is a preloaded Lua table structured like this: {"common" = {"001", "002", "003",..., "020"}}. For the sake of the question, I've restricted the cards to a certain category (common). Let's suppose the player won card number 3, so the code should run like this:
array["common"][3] = collection["common"][3]
And the resulting table array would be:
array = {"common" = {nil, nil, "003"}}
When I use the function writeFile("playerCards.json", array) I am encoding the table above into the file playerCards.json. For now, this code works perfectly, and the resulting JSON file is as follows:
{"common":[null,null,"003"],"uncommon":[],"rare":[]}
The problem comes when the player gets a card above 9, for example, 15. When written, the JSON file becomes this:
{"common":{"3":"003","15":"015"},"uncommon":[],"rare":[]}
How can the same code produce such different results? Can you help me solve this problem? If you need it, here is the code for the load and write functions:
local loadFile = function(name)
local data = nil
local path = system.pathForFile(name, system.DocumentsDirectory)
handle = io.open(path, "r")
if handle then
data = json.decode(handle:read("*a"))
io.close(handle)
end
return data
end
local writeFile = function(name, data)
local path = system.pathForFile(name, system.DocumentsDirectory)
local handle = io.open(path, "w+")
if handle then
handle:write(json.encode(data))
io.close(handle)
end
end
The problem is that Lua does not distinguish between list and mapping objects (tables are used for both) while JSON does.
This creates ambiguity when serializing to JSON; the serializer has to determine whether a Lua table should serialize to an array or an object. Most serializers do this by checking if the keys of the table are roughly sequential integers, and only serializing to an array if so. If your array is too 'sparse', as in your second example, then the serializer thinks its a map with numerical keys instead.
I don't know about Corona, but some Lua JSON libraries I have seen include a method to mark tables explicitly as arrays. Alternatively, you can fill empty slots of the array with a placeholder value like false instead of nil.