Determining the equality of JSON-formatted objects in node.js - json

What is the most efficient way to compare two JSON-formatted objects data in node.js?
These objects do not contain "undefined" or functions and their propotype is Object.
I've heard there is a good support if JSON in node.js

Friends! I'm really sorry if I don't understand something, or explain question in wrong terms. All I wanna do is to compare the equality of two pieces of JSON-standardized data like this:
{"skip":0, "limit":7, "arr": [1682, 439, {"x":2, arr:[]}] }
{"skip":0, "limit":7, "arr": [1682, 450, "a", ["something"] }
I'm 100% sure there will be no functions, Date, null or undefined, etc. in these data. I want to say I don't want to compare JavaScript objects in the most general case (with complex prototypes, circular links and all this stuff). The prototype of these data will be Object. I'm also sure lots of skillful programmers have answered this question before me.
The main thing I'm missing is that I don't know how to explain my question correctly. Please feel free to edit my post.
My answer:
First way: Unefficient but reliable. You can modify a generic method like this so it does not waste time looking for functions and undefined. Please note that generic method iterates the objects three times (there are three for .. in loops inside)
Second way: Efficient but has one restriction. I've found JSON.stringify is extremely fast in node.js. The fastest solution that works is:
JSON.stringify(data1) == JSON.stringify(data2)
Very important note! As far as I know, neither JavaScript nor JSON don't matter the order of the object fields. However, if you compare strings made of objects, this will matter much. In my program the object fields are always created in the same order, so the code above works. I didn't search in the V8 documentation, but I think we can rely on the fields creation order. In other case, be aware of using this method.
In my concrete case the second way is 10 times more efficient then the first way.

I've done some benchmarking of the various techniques, with underscore coming out on top:
> node nodeCompare.js
deep comparison res: true took: 418 (underscore)
hash comparison res: true took: 933
assert compare res: true took: 2025
string compare res: true took: 1679
Here is the source code:
var _ = require('underscore');
var assert = require('assert');
var crypto = require('crypto');
var large = require('./large.json');
var large2 = JSON.parse(JSON.stringify(large));
var t1 = new Date();
var hash = crypto.createHash('md5').update(JSON.stringify(large)).digest('hex');
var t2 = new Date();
var res = _.isEqual(large, large2);
var t3 = new Date();
var res2 = (hash == crypto.createHash('md5').update(JSON.stringify(large2)).digest('hex'));
var t4 = new Date();
assert.deepEqual(large, large2);
var t5 = new Date();
var res4 = JSON.stringify(large) === JSON.stringify(large2);
var t6 = new Date();
console.log("deep comparison res: "+res+" took: "+ (t3.getTime() - t2.getTime()));
console.log("hash comparison res: "+res2+" took: "+ (t4.getTime() - t3.getTime()));
console.log("assert compare res: true took: "+ (t5.getTime() - t4.getTime()));
console.log("string compare res: "+res4+" took: "+ (t6.getTime() - t5.getTime()));

On the face of it, I believe this is what you're looking for:
Object Comparison in JavaScript
However, the more complete answer is found here:
How do you determine equality for two JavaScript objects?
These answers are from 2 and 3 years ago, respectively. It's always a good idea to search the site for your intended question before posting, or search Google more broadly -- "javascript compare two JSON objects" in Google returns a lot. The top 4 hits are all StackOverflow answers, at the moment.

You can use rename/unset/set diff for JSON objects - https://github.com/mirek/node-rus-diff
npm install rus-diff
Your example JSON objects:
a = {"skip":0, "limit":7, "arr": [1682, 439, {"x":2, arr:[]}] }
b = {"skip":0, "limit":7, "arr": [1682, 450, "a", ["something"]] }
var rusDiff = require('rus-diff').rusDiff
console.log(rusDiff(a, b))
...generate the following diff:
{ '$set': { 'arr.1': 450, 'arr.2': 'a', 'arr.3': [ 'something' ] } }
...which is MongoDB compatible diff format.
If documents are the same, rusDiff(a, b) returns false.

Related

convert FiX message to JSON using JAVA

i have extracted the fix message as below from Unix server and now need to convert this message into JSON. how can we do this?
8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111
Note: I tried to make this a comment on the answer provided by #selbie, but the text was too long for a comment, so I am making it an answer.
#selbie's answer will work most of the time, but there are two edge cases in which it could fail.
First, in a tag=value field where the value is of type STRING, it is legal for value to contain the = character. To correctly cope with this possibility, the Java statement:
pair = item.split("=");
should be changed to:
pair = item.split("=", 2);
The second edge case is when there are a pair of fields, the first of which is of type LENGTH and the second is of type DATA. In this case, the value of the LENGTH fields specifies the length of the DATA field (without the delimiter), and it is legal for the value of the DATA field to contain the delimiter character (ASCII character 1, but denoted as | in both the question and Selbie's answer). Selbie's code cannot be modified in a trivial manner to deal with this edge case. Instead, you will need a more complex algorithm that consults a FIX data dictionary to determine the type of each field.
Since you didn't tag your question for any particular programming language, I'll give you a few sample solutions:
In javascript:
let s = "8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111"
let obj = {};
items = s.split("|")
items.forEach(item=>{
let pair = item.split("=");
obj[pair[0]] = pair[1];
});
let jsonString = JSON.stringify(obj);
Python:
import json
s = "8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111"
obj = {}
for item in s.split("|"):
pair = item.split("=")
obj[pair[0]] = pair[1]
jsonString = json.dumps(obj)
Porting the above solutions to other languages is an exercise for yourself. There's comments below about semantic ordering and handling cases where the the = or | chars are part of the content. That's on you to explore if you need to support those scenarios.

How to work with data returned by mysql select query in nodejs

I am working on a discord bot written in nodejs, the bot utilises a mysql database server to store information. The problem I have run into is that I cannot seem to retrieve the data from the database in a neat way, every single thing I try seems to run into some issue or another.
The select query returns an object called RowDataPacket. When googling every single result will reference this solution: Object.values(JSON.parse(JSON.stringify(rows)))
It postulates that I should get the values back, but I dont I get an array back that is as hard to work with as the rowdatapacket object.
This is a snippet of my code:
const kenneledMemberRolesTableName = 'kenneled_member_roles'
const kenneledMemberKey = 'kenneled_member'
const kenneledMemberRoleKey = 'kenneled_member_role_id'
const kenneledStaffMemberKey = 'kenneled_staff_member'
const kenneledDateKey = 'kenneled_date'
const kenneledReturnableRoleKey = 'kenneled_role_can_be_returned'
async function findKenneledMemberRoles(kenneledMemberId) {
let sql = `SELECT CAST(${kenneledMemberRoleKey} AS Char) FROM ${kenneledMemberRolesTableName} WHERE ${kenneledMemberKey} = ${kenneledMemberId}`
let rows = await databaseAccessor.runQuery(sql)
let result = JSON.parse(JSON.stringify(rows)).map(row => {
return row.kenneled_member_role_id
})
return result
}
This seemed to work, until I had to do a type conversion on the value, now the dot notations requires me to reference row.CAST(kenneled_member_role_id AS Char), this cannot work, and I have found no other way to retrieve the data than through dot notation. I swear there must be a better way to work with mysql rowdatapackets but the solution eludes me
I figured out something that works, however I still feel like this is an inelegant solution, I would love to hear from others if I am misunderstanding how to work with mysql code in nodejs, or if this is just a consequence of the library:
let result = JSON.parse(JSON.stringify(rows)).map(row => {
return row[`CAST(${kenneledMemberRoleKey} AS CHAR)`];
})
So what I did is I access the value through brackets instead of dot notation, this seems to work, and at least makes me able to store part of or the whole expression in a constant variable, hiding the ugliness.

Better way to send data to server

I have two ways in which i can send data(json string) to server.
1)map<key,list>
eg. {"components":["ab","bc","cd"],"values":[1,2,3]}
This represents value of component "ab" is 1.
2) map<key,values>
eg. {"ab":1,"bc":2,"cd":3}
Now above example is of three components but I want to send thousands of components so which one is more scalable in terms of size and performance.
If you feel I missed some details feel free to ask.
Server side you will have to do this if you want value for key "bc":
Option 1: {"components":["ab","bc","cd"],"values":[1,2,3]}
String json = [...] // got json from client
MyJsonObject jo = new Gson().fromJson(json, MyJsonObject.class);
int index = jo.getComponents().indexOf("bc"); // O(n)
int value = jo.getValuess().get(index); // O(1)
Option 2: {"ab":1,"bc":2,"cd":3}
String json = [...] // got json from client
MyJsonObject jo = new Gson().fromJson(json, MyJsonObject.class);
int value = jo.get("bc"); // O(1) ~ O(n)
I believe the option 2 is better because of better performance. Moreover it is simpler and more readable than option 1. Differences in the bandwidth can be neglected.
But you should decide what solution is better for your application.

Fast JSON Parser for Matlab

Do you know a very fast JSON Parser for Matlab?
Currently I'm using JSONlab, but with larger JSON files (mine is 12 MB, 500 000 lines) it's really slow. Or do you have any tips' for me to increase the speed?
P.S. The JSON file is max. 3 levels deep.
If you want to be fast, you could use the Java JSON parser.
And before this answer gets out of hand, I am going to post the stuff I put down so far:
clc
% input example
jsonStr = '{"bool1": true, "string1": "some text", "double1": 5, "array1": [1,2,3], "nested": {"val1": 1, "val2": "one"}}'
% use java..
javaaddpath('json.jar');
jsonObj = org.json.JSONObject(jsonStr);
% check out the available methods
jsonObj.methods % see also http://www.json.org/javadoc/org/json/JSONObject.html
% get some stuff
b = jsonObj.getBoolean('bool1')
s = jsonObj.getString('string1')
d = jsonObj.getDouble('double1')
i = jsonObj.getJSONObject('nested').getInt('val1')
% put some stuff
jsonObj = jsonObj.put('sum', 1+1);
% getting an array or matrix is not so easy (you get a JSONArray)
e = jsonObj.get('array1');
% what are the methods to access that JSONArray?
e.methods
for idx = 1:e.length()
e.get(idx-1)
end
% but putting arrays or matrices works fine
jsonObj = jsonObj.put('matrix1', ones(5));
% you can get these also easily ..
m1 = jsonObj.get('matrix1')
% .. as long as you dont convert the obj back to a string
jsonObj = org.json.JSONObject(jsonObj.toString());
m2 = jsonObj.get('matrix1')
If you can afford to call .NET code, you may want to have a look at this lightweight guy (I'm the author):
https://github.com/ysharplanguage/FastJsonParser#PerfDetailed
Coincidentally, my benchmark includes a test ("fathers data") in the 12MB ballpark precisely (and with a couple levels of depth also) that this parser parses into POCOs in under 250 ms on my cheap laptop.
As for the MATLAB + .NET code integration:
http://www.mathworks.com/help/matlab/using-net-libraries-in-matlab.html
'HTH,
If you just want to read JSON files, and have a C++11 compiler, you can use the very fast json_read mex function.
Since Matlab 2016b, you can use jsondecode.
I have not compared its performance to other implementations.
From personal experience I can say that it is not horribly slow.

How to use joins in HTML5 Indexed Db?

I do want to join two table in HTML5 indexed db.
I found lot samples to add, Update, Delete and listing record but can't found any samples to join multiple table.
Sample URL:
*http://users.telenet.be/kristofdegrave/*
Was wondering the same question when just generally looking at these technologies. Did not check your particular example, but here is an example from Mozilla on how to do a join if you need it:
https://hacks.mozilla.org/2010/06/comparing-indexeddb-and-webdatabase/
It is quite a lot of code compared to SQL, but I think the idea is that abstracting these using JavaScript later is quite easy.
There is no join, but you can open multiple object stores and join during cursor iterations.
Using my own ydn-db library, you can query SELECT * FROM Supplier, Part WHERE Supplier.CITY = Part.CITY by
var iter_supplier = new ydn.db.IndexValueIterator('Supplier', 'CITY');
var iter_part = new ydn.db.IndexValueIterator('Part', 'CITY');
var req = db.scan(function(keys, values) {
var SID = keys[0];
var PID = keys[1];
console.log(SID, PID);
if (!SID || !PID) {
return []; // done
}
var cmp = ydn.db.cmp(SID, PID); // compare keys
if (cmp == 0) {
console.log(values[0], values[1]);
return [true, true]; // advance both
} else if (cmp == 1) {
return [undefined, SID]; // jump PID cursor to match SID
} else {
return [PID, undefined]; // jump SID cursor to match PID
}
}, [iter_supplier, iter_part]);
See more detail on Join query article.
As far as I know, IndexedDB does not have an API for doing JOINs, yet. The solutions I adopted involved opening a cursor, looping through the results and doing the join manually. It's the dreadful RBAR approach, but I couldn't find a better solution.
Instead of trying to join, redesign how you store the data so that a join is not required. You do not need to follow the same normalization constraints as SQL when you are using a no-sql approach. No-sql supports storing data redundantly when it is appropriate.
IndexedDB is in fact an object store and normaly in an object store there is less need for joins because you can just save the nested structure.
In the case you show, data from a codetable gets joined with real data. The case to solve this without a join is just fetch your code table into memory (normally this data never changes) and make the join in code.