How can i make a new single json object by extracting particular fields from realtime json data using node.js - json

I have the following code which publishes the json data in the specified url using mqtt.The initial data is retrieved from http.
var request = require('request');
var JSONStream = require('JSONStream');
var es = require('event-stream');
var mqtt = require('mqtt');
request({url: 'http://isaacs.couchone.com/registry/_all_docs'})
.pipe(JSONStream.parse('rows.*'))
.pipe(es.mapSync(function (data) {
console.info(data);
var client = mqtt.createClient(1883, 'localhost');
client.publish('NewTopic', JSON.stringify(data));
client.end();
return data;
}))
The following is the subscriber code which subscribes the data that is published (in the above code) through mqtt
var mqtt = require('mqtt');
var client = mqtt.createClient();
client.subscribe('NewTopic');
client.on('message', function(topic, message) {
console.info(message);
});
In the above code, I get all json data in the specified url in 'message'.I need to extract 'id' and 'value' from the received data and make it as a single JSON object and need to publish it to mqtt,so that another client can subscribe only the 'id' and 'value' as json data.

To convert a JSON text into an object, you can use the eval() function. eval() invokes the JavaScript compiler. Since JSON is a proper subset of JavaScript, the compiler will correctly parse the text and produce an object structure. The text must be wrapped in parens to avoid tripping on an ambiguity in JavaScript's syntax.
var myObject = eval(message);
The eval function is very fast. However, it can compile and execute any JavaScript program, so there can be security issues. The use of eval is indicated when the source is trusted and competent. It is much safer to use a JSON parser. In web applications over XMLHttpRequest, communication is permitted only to the same origin that provide that page, so it is trusted. But it might not be competent. If the server is not rigorous in its JSON encoding, or if it does not scrupulously validate all of its inputs, then it could deliver invalid JSON text that could be carrying dangerous script. The eval function would execute the script, unleashing its malice.
To defend against this, a JSON parser should be used. A JSON parser will recognize only JSON text, rejecting all scripts. In browsers that provide native JSON support, JSON parsers are also much faster than eval.
var myObject = JSON.parse(message);
And use it as a Object:
myObject.id;
myObject.value;
Create a object with just id and value :
var idAndValueObj = {};
idAndValueObj.id = myObject.id;
idAndValueObj.value = myObject.value;
Convert to JSON string:
var jsonStr = JSON.stringify(idAndValueObj);

Related

Ballerina, Using Json Response from REST-API

My professor wants me to write a little tutorial on how to deploy Ballerina services. So I'm trying to learn it. I'm using Version 1.2 and I'm a bit overwhelmed by the concept of taint checking and the variable types...
I'm trying to write a minimal REST-Service with an endpoint that requests json data from another api and then uses that JSON to do stuff.
What's working so far is the following:
service tutorial on new http:Listener(9090) {
// Resource functions are invoked with the HTTP caller and the incoming request as arguments.
resource function getName(http:Caller caller, http:Request req) {
http:Client clientEP = new("https://api.scryfall.com/");
var resp = clientEP->get("/cards/random");
if (resp is http:Response) {
var payload = resp.getJsonPayload();
if (payload is json) {
var result = caller->respond(<#untainted>(payload));
} else {
log:printError("");
}
} else {
log:printError("");
}
}
That responds with the JSON that is returned from https://api.scryfall.com/cards/random
But lets now say, that I want to access a single value from that JSON. e.G. "name".
If I try to access it like this: payload["name"]
I get: invalid operation: type 'json' does not support indexing
I just figured out that it works if I create a map first like that:
map mp = <map>payload;
If I then access mp["name"] it works. BUT WHY? What is the json type good for if you still have to create a map and then cast the payload? And how would I access json inside the json? For example mp["data"][0]... invalid operation: type 'json' does not support indexing again...
And I'm still trying to understand the concept of taint checking....
do I just cast everything that is tainted to <#untainted> after checking the content?
Sometimes I really do not get what the documentation is trying to tell me....
I would recommend you to use Ballerina Swan Lake versions. Swan Lake versions contain enhancements to various language features. Here is a sample code that covers your use case. You can download Swan Lake Alpha2 at https://ballerina.io/
import ballerina/io;
import ballerina/http;
service tutorial on new http:Listener(9090) {
resource function get payload() returns json|error {
http:Client clientEP = check new ("https://api.scryfall.com/");
json payload = <json> check clientEP -> get("/cards/random", targetType = json);
// Processing the json payload
// Here the type of the `payload.name` expression is json | error
// You can eliminate with check: it returns from this resource with this error
json nameField = check payload.name;
io:println(nameField);
// You can traverse the json tree as follows
json standardLegality = check payload.legalities.standard;
io:println(standardLegality);
// colors is an array
// The cast is necessary because `check payload.colors` gives you a json
json colors = <json[]> check payload.colors;
io:println(colors);
// Responding with the complete payload recived from api.scryfall.com
return payload;
}
}
Taint analysis helps you to write code with no security vulnerabilities. However, we've disabled taint analysis in Swan Lake versions. If you want to enable it, you can use the option --taint-check with bal build

Can you use 'require' in react to import a library?

In my react project, I'm trying to convert XML data from an API call into JSON (using a library called xml-js).
As per the documentation, I'm importing the library in my parent component as follows
const convert = require('xml-js')
and then attempting the convert the API data as follows
const beerList =
'<Product>
<Name>Island Life IPA</Name>
<Volume>300ml/473ml</Volume>
<Price>$10/$13</Price>
<ABV>6.3%</ABV>
<Handpump>No</Handpump>
<Brewery>Eddyline</Brewery>
<IBU/>
<ABV>6.3%</ABV>
<Image>islandlife.png</Image>
<Country>New Zealand</Country>
<Description>Fruited IPA</Description>
<Pouring>Next</Pouring>
<IBU/>
<TapBadge/>
<Comments/>
</Product>'
const beerJs = convert(beerList,{compact: true, spaces: 4})
The errors are telling me that 'convert' is not a function, which tells me that the library isn't being imported. So is the issue with using 'require' syntax, and if so, what alternative would work in react?
which tells me that the library isn't imported
No. If that were the case, you wouldn't even get that far, your require call would throw an error.
Instead, it tells you that convert is not a function - which it isn't! Look at it in a debugger or log it, and you'll see it's an object with several functions inside. You can't call an object like a function.
Take a look at the xml-js docs again:
This library provides 4 functions: js2xml(), json2xml(), xml2js(), and xml2json(). Here are the usages for each one (see more details in the following sections):
var convert = require('xml-js');
result = convert.js2xml(js, options); // to convert javascript object to xml text
result = convert.json2xml(json, options); // to convert json text to xml text
result = convert.xml2js(xml, options); // to convert xml text to javascript object
result = convert.xml2json(xml, options); // to convert xml text to json text
So the solution is to call convert.xml2json and not convert:
const beerJs = convert.xml2json(beerList, {compact: true, spaces: 4})
Or maybe you want an actual object and not a JSON string, then you'd use convert.xml2js (in which case the spaces option is useless):
const beerJs = convert.xml2js(beerList, {compact: true})

JSON String parsing each character as an object

I have a JSON file that contains what I believe to be a correct JSON string:
{"title": "exampleTitle", "tipTitle": "exampleTipTitle", "tip": "exampleTip"}
I'm trying to parse said file and take out the 3 values then store them in variables, however currently, it parses each individual character as a separate object, therefore:
JSONobj[1] = "
and so on. Assuming that currentLocation = the directory location of the json file.
Code
var jsonLocation = currentLocation + "json.txt";
var request = new XMLHttpRequest();
request.open("GET", jsonLocation, false);
request.send(null);
var returnValue = request.responseText;
var JSONobj = JSON.parse(JSON.stringify(returnValue));
var headerTitle = JSONobj[0];
A few clarifications, the stringify is in because it was throwing an unexpected token error. I've tried changing the file tile to .json instead but that also makes no difference. "It also gives off a XMLHttpRequest on the main thread is deprecated" but I'm not particularly sure how to solve that issue. Any help would be appreciated.
var returnValue = request.responseText;
Here returnValue is a string of JSON.
"{\"title\": \"exampleTitle\", \"tipTitle\": \"exampleTipTitle\", \"tip\": \"exampleTip\"}
var JSONobj = JSON.parse(JSON.stringify(returnValue));
Here you convert the string of JSON to JSON. So you have a JSON string representing a string, and that string is a representation of a data structure in JSON.
"\"{\\"title\\": \\"exampleTitle\\", \\"tipTitle\\": \\"exampleTipTitle\\", \\"tip\\": \\"exampleTip\\"}"
Then you parse it and convert it back to the original string of JSON.
"{\"title\": \"exampleTitle\", \"tipTitle\": \"exampleTipTitle\", \"tip\": \"exampleTip\"}
So you end up back where you start.
Just don't use JSON.stringify here, and you'll convert your JSON to a JavaScript object:
var javascript_object = JSON.parse(returnValue);
Then you have an object, but it doesn't have a 0 property so it doesn't make sense to access it with javascript_object[0]. The properties have names, such as javascript_object.title.
Your JSON doesn't describe an array, so indexing into it with an index like 0 doesn't make sense. Your JSON describes an object, which will have properties with the names title, tipTitle, and tip.
Additionally, you're overdoing your parsing: You just want to parse, not stringify (which is the opposite of parsing):
var JSONobj = JSON.parse(returnValue);
So:
var JSONobj = JSON.parse(returnValue);
var headerTitle = JSONobj.title;
console.log(headerTitle); // "exampleTitle"
Side note: By the time you've assigned it to the variable you've called JSONobj, it isn't JSON anymore, it's just a normal JavaScript object, so that name is a bit misleading. If you're writing source code, and you're not dealing with a string, you're not dealing with JSON anymore. :-)

Response JSON object or JSON.stringify?

Suppose I want to return JSON content
var content = {
a: 'foo',
b: 'bar'
};
What is the best practice to return my JSON data?
A) Return object as is; i.e res.end(content)?
B) JSON.stringify(content) and then call JSON.parse(content) on the client?
The client must always send a string. That's what the protocol says. After all, HTTP is a wide-ranging protocol, and not all languages support JSON objects, let alone JavaScript data.
If you don't convert it to a JSON string, chances are that pure Node will just send it as [object Object], and i'm sure that's not your intention.
As mentioned previously, Express lets you send an actual JS object, and does the JSON string converting for you. Alternately, you can manually convert it.
If you send the response with express's res.json you can send the Object directly as application/json encoded response.
app.get('/route/to/ressource', function(req, res){
var oMyOBject = {any:'data'};
res.json(oMyOBject);
});

how to parse a large, Newline-delimited JSON file by JSONStream module in node.js?

I have a large json file, its is Newline-delimited JSON, where multiple standard JSON objects are delimited by extra newlines, e.g.
{'name':'1','age':5}
{'name':'2','age':3}
{'name':'3','age':6}
I am now using JSONStream in node.js to parse a large json file, the reason I use JSONStream is because it is based on stream.
However,both parse syntax in the example can't help me to parse this json file with separated JSON in each line
var parser = JSONStream.parse(**['rows', true]**);
var parser = JSONStream.parse([**/./**]);
Can someone help me with that
Warning: Since this answer was written, the author of the JSONStream library removed the emit root event functionality, apparently to fix a memory leak.
Future users of this library, you can use the 0.x.x versions if you need the emit root functionality.
Below is the unmodified original answer:
From the readme:
JSONStream.parse(path)
path should be an array of property names, RegExps, booleans, and/or functions. Any object that matches the path will be emitted as 'data'.
A 'root' event is emitted when all data has been received. The 'root' event passes the root object & the count of matched objects.
In your case, since you want to get back the JSON objects as opposed to specific properties, you will be using the 'root' event and you don't need to specify a path.
Your code might look something like this:
var fs = require('fs'),
JSONStream = require('JSONStream');
var stream = fs.createReadStream('data.json', {encoding: 'utf8'}),
parser = JSONStream.parse();
stream.pipe(parser);
parser.on('root', function (obj) {
console.log(obj); // whatever you will do with each JSON object
});
JSONstream is intended for parsing a single huge JSON object, not many JSON objects. You want to split the stream at newlines, then parse them as JSON.
The NPM package split claims to do this splitting, and even has a feature to parse the JSON lines for you.
If your file is not enough large here is an easy, but not performant solution:
const fs = require('fs');
let rawdata = fs.readFileSync('fileName.json');
let convertedData = String(rawdata)
.replace(/\n/gi, ',')
.slice(0, -1);
let JsonData= JSON.parse(`[${convertedData}]`);
I created a package #jsonlines/core which parses jsonlines as object stream.
You can try the following code:
npm install #jsonlines/core
const fs = require("fs");
const { parse } = require("#jsonlines/core");
// create a duplex stream which parse input as lines of json
const parseStream = parse();
// read from the file and pipe into the parseStream
fs.createReadStream(yourLargeJsonLinesFilePath).pipe(parseStream);
// consume the parsed objects by listening to data event
parseStream.on("data", (value) => {
console.log(value);
});
Note that parseStream is a standard node duplex stream.
So you can also use for await ... of or other ways to consume it.
Here's another solution for when the file is small enough to fit into memory. It reads the whole file in one go, converts it into an array by splitting it at the newlines (removing the blank line at the end), and then parses each line.
import fs from "fs";
const parsed = fs
.readFileSync(`data.jsonl`, `utf8`)
.split(`\n`)
.slice(0, -1)
.map(JSON.parse)