Regex for fetching JSON data - json

I have a JSON file abc.json
{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[],"value4":{"value41":{"value411":5,"value412":"hey"}}}
I can get value2 using this regex
sed -E 's/."value2":"?([^,"])"?.*/\1/' abc.json
I want to know how I can get values of value411 and value412
I don't want to use jq or any other tool as my requirement is to use regex for this.

You should always try to use an existing parser, depending on what platform you work on there should be one that can interpret the data model for you.
I refer to this famous answer about parsing HTML with regex

var jData = {
"value1": 5.0,
"value2": 2.5,
"value3": "2019-10-24T15:26:00.000Z",
"modifier": [],
"value4": {
"value41": {
"value411": 5,
"value412": "hey"
}
}
};
If You try with regex then use this:
JSON.stringify(jData).match(/(?<=(\"(value411|value412)\"\:))[\"\w+]+/g)
// Output: ['5', '"hey"']
Demo: https://regex101.com/r/c3K4cH/1
Limitation: You have to put only the key name, don't try to fetch the full object
You create a javascript function get any non-empty value from a JSON, as follows:
function getData(obj, fKey, fData = null) {
for (const prop of Object.getOwnPropertyNames(obj)) {
if (!fData) {
if (typeof obj[prop] === 'object' && Object.keys(obj[prop]).length > 0 && prop !== fKey) {
return getData(obj[prop], fKey, fData)
} else {
if(prop === fKey) {
fData = obj[prop];
return fData
}
}
} else {
return fData;
}
}
}
console.log(getData(jData, 'value411'));
console.log(getData(jData, 'value412'));

Related

Unable to parse JSON data due to errors or undefined data

I apologize if this seems similar to other questions asked but I have not been able to find any posts that have resolved this issue for me. Basically, I am getting a JSON object and I am trying to parse it but I can't parse it correctly. Mainly the WordDetails section that I am getting from a Word API. I am able to get everything outside the results section under WordDetails. Basically, when I get to results, I am not able to parse it correctly. Below is an example of the format.
{
"LastIndex": 133,
"SRDWords": [
{
"Domain": {
"URL": "abactinal.com",
"Available": true
},
"WordDetails": "{\"word\":\"abactinal\",\"results\":[{\"definition\":\"(of radiate animals) located on the surface or end opposite to that on which the mouth is situated\",\"partOfSpeech\":null,\"antonyms\":[\"actinal\"]}],\"syllables\":{\"count\":4,\"list\":[\"ab\",\"ac\",\"ti\",\"nal\"]}}"
},
{
"Domain": {
"URL": "aaronical.com",
"Available": true
},
"WordDetails": "{\"word\":\"aaronical\",\"syllables\":{\"count\":4,\"list\":[\"aa\",\"ron\",\"i\",\"cal\"]},\"pronunciation\":{\"all\":\"ɜ'rɑnɪkəl\"}}"
},
...
Here is my code below. Basically, I am getting to the results section of WordDetails but if I try to parse the results section it fails and if I try object.entries on it, it will not return a response according to the alert messages I used. I know there must be a better way but not sure what. Most articles say just JSON.parse then map it but that does not work. Any help would be appreciated!
data.Words.map(word => {
//get data
for (let [key, value] of Object.entries(word)) {
if (key === "Domain") {
url = value.URL;
availability = value.Available;
} else if (key.trim() === "WordDetails") {
alert("value " + value);
wDetails = JSON.parse(value);
for (let [key2, value2] of Object.entries(wDetails)) {
if (key2 === "word") {
//store word
} else if (key2.toString().trim() === "results") {
let test = JSON.parse(value2);
test = Object.entries(value2);
test.map(t => {
alert(t.definition);
});
}
}
}
}
});
You did JSON.parse above, no need to parse value2 again.
And value for results is an array, so no need for Object.entries.
...
} else if (key2.toString().trim() === 'results') {
let test = JSON.parse(value2); // this should be remove
test = Object.entries(value2); // this should be remove, value2 should be an array
// map value2 directly
value2.map(t => {
alert(t.definition);
});
}
...

Restructuring JSON Response in Angular 2+

I'm currently receiving a web api JSON response that is heavily nested and I want to restructure it plucking out only the data I need and making it more simple.
How would I do this in Angular 2+/Typescript?
Any help would be really appreciated.
Say I have something like this:
"O1": {
"P1": "Something",
"A1": [{
"P2": "Something",
"A2": [{
"P3": "Something"
}]
}]
"P4": "Something"
}
and I want to restructure it to this :
"O1": {
"P1": "Something",
"P2": "Something",
"P3": "Something"
}
Can I rebuild the model in a class in the constructor? If so, I haven't been able to find anything online as of yet.
Let me know if I need to provide any further info.
Thanks
Let's flatten the initial object using the snippet which was proposed. And then we can filter those properties which we need to include in the final object:
function filterObj(obj: any, propsForFilter: string[]) {
const result = {};
for (let prop in obj) {
const filteredProp = propsForFilter.find(p => prop.includes(p));
if (filteredProp) {
result[filteredProp] = obj[prop];
}
}
return result;
}
const flattened = flattenObject(obj);
console.log(flattened, filterObj(flattened, ['P1', 'P2']));
This method filterObj accepts the array of properties in order to include them in the final result. Hope this will be helpful for you.
Found this snippet in github:
https://gist.github.com/penguinboy/762197
var flattenObject = function(ob) {
var toReturn = {};
for (var i in ob) {
if (!ob.hasOwnProperty(i)) continue;
if ((typeof ob[i]) == 'object') {
var flatObject = flattenObject(ob[i]);
for (var x in flatObject) {
if (!flatObject.hasOwnProperty(x)) continue;
toReturn[i + '.' + x] = flatObject[x];
}
} else {
toReturn[i] = ob[i];
}
}
return toReturn;
}

Postman test for empty dictionary value of json name

How can I write a test for an empty value of a specific JSON name pair. For example I have this JSON:
{
"data": {
"sectionGroupName": "PPConfig:APIMethod",
"sections": {}
},
"success": true,
"errorMessage": ""
}
I want to check if sections is empty, like it is in this case. I have other successful tests written like this:
tests["Status code is 200"] = responseCode.code === 200;
var body = JSON.parse(responseBody);
tests["Success Response"] = body.success === true;
tests["No Error message"] = body.errorMessage === "";
tests.Data = body.data.sectionGroupName === "PPConfig:APIMethod";
But I haven't been able to find successful test code for checking if the value of a specific name is an empty dictionary. Can someone help me with this as an example please?
You can get the list of properties of sections and test its length.
let sectionKeys = Object.keys(body.data.sectionGroupName)
if(sectionKeys.length){
//Proceed with section
} else {
//Proceed when it's empty
}
See Object.keys()
from this link
to check if it's a dictionary (use your 'sections' as v)
function isDict(v) {
return !!v && typeof v==='object' && v!==null && !(v instanceof Array) && !(v instanceof Date) && isJsonable(v);
}
Then check that it is empty (from this other link) use:
function isEmpty(obj) {
for (var x in obj) { return false; }
return true;
}
That should work

Formatting DynamoDB data to normal JSON in AWS Lambda

I'm using AWS Lambda to scan data from a DynamoDB table. This is what I get in return:
{
"videos": [
{
"file": {
"S": "file1.mp4"
},
"id": {
"S": "1"
},
"canvas": {
"S": "This is Canvas1"
}
},
{
"file": {
"S": "main.mp4"
},
"id": {
"S": "0"
},
"canvas": {
"S": "this is a canvas"
}
}
]
}
My front-end application is using Ember Data Rest Adapter which does not accepts such response. Is there any way I can get normal JSON format? There is this NPM module called dynamodb-marshaler to convert DynamoDB data to normal JSON. I'm looking for a native solution if possible.
Node.js
Use the unmarshall function from AWSJavaScriptSDK:
const AWS = require("aws-sdk");
exports.handler = function( event, context, callback ) {
const newImages = event.Records.map(
(record) => AWS.DynamoDB.Converter.unmarshall(record.dynamodb.NewImage)
);
console.log('Converted records', newImages);
callback(null, `Success`);
}
Python
Use TypeDeserializer.deserialize from boto3.dynamodb.types:
import json
from boto3.dynamodb.types import TypeDeserializer
def ddb_deserialize(r, type_deserializer = TypeDeserializer()):
return type_deserializer.deserialize({"M": r})
def lambda_handler(event, context):
new_images = [ ddb_deserialize(r["dynamodb"]["NewImage"]) for r in event['Records'] ]
print('Converted records', json.dumps(new_images, indent=2))
I know is a bit old but I had the same problem processing stream data from dynamoDB in node js lambda function. I used the proposed by #churro
import sdk and output converter
var AWS = require("aws-sdk");
var parse = AWS.DynamoDB.Converter.output;
use the parse function with a small hack
exports.handler = function( event, context, callback ) {
var docClient = new AWS.DynamoDB.DocumentClient();
event.Records.forEach((record) => {
console.log(record.eventID);
console.log(record.eventName);
console.log('DynamoDB Record:', parse({ "M": record.dynamodb.NewImage }));
});
callback(null, `Successfully processed ${event.Records.length} records.`);
}
Hope it helps
AWS JavaScript SDK was recently updated with Document Client which does exactly what you need. Check the announce and usage examples here: http://blogs.aws.amazon.com/javascript/post/Tx1OVH5LUZAFC6T/Announcing-the-Amazon-DynamoDB-Document-Client-in-the-AWS-SDK-for-JavaScript
Javascript: AWS SDK provides the unmarshall function
Python: use TypeDeserializer from boto3.dynamodb.types:
from boto3.dynamodb.types import TypeDeserializer, TypeSerializer
def from_dynamodb_to_json(item):
d = TypeDeserializer()
return {k: d.deserialize(value=v) for k, v in item.items()}
## Usage:
from_dynamodb_to_json({
"Day": {"S": "Monday"},
"mylist": {"L": [{"S": "Cookies"}, {"S": "Coffee"}, {"N": "3.14159"}]}
})
# {'Day': 'Monday', 'mylist': ['Cookies', 'Coffee', Decimal('3.14159')]}
Here you can find gist which does that:
function mapper(data) {
let S = "S";
let SS = "SS";
let NN = "NN";
let NS = "NS";
let BS = "BS";
let BB = "BB";
let N = "N";
let BOOL = "BOOL";
let NULL = "NULL";
let M = "M";
let L = "L";
if (isObject(data)) {
let keys = Object.keys(data);
while (keys.length) {
let key = keys.shift();
let types = data[key];
if (isObject(types) && types.hasOwnProperty(S)) {
data[key] = types[S];
} else if (isObject(types) && types.hasOwnProperty(N)) {
data[key] = parseFloat(types[N]);
} else if (isObject(types) && types.hasOwnProperty(BOOL)) {
data[key] = types[BOOL];
} else if (isObject(types) && types.hasOwnProperty(NULL)) {
data[key] = null;
} else if (isObject(types) && types.hasOwnProperty(M)) {
data[key] = mapper(types[M]);
} else if (isObject(types) && types.hasOwnProperty(L)) {
data[key] = mapper(types[L]);
} else if (isObject(types) && types.hasOwnProperty(SS)) {
data[key] = types[SS];
} else if (isObject(types) && types.hasOwnProperty(NN)) {
data[key] = types[NN];
} else if (isObject(types) && types.hasOwnProperty(BB)) {
data[key] = types[BB];
} else if (isObject(types) && types.hasOwnProperty(NS)) {
data[key] = types[NS];
} else if (isObject(types) && types.hasOwnProperty(BS)) {
data[key] = types[BS];
}
}
}
return data;
function isObject(value) {
return typeof value === "object" && value !== null;
}
}
https://gist.github.com/igorzg/c80c0de4ad5c4028cb26cfec415cc600
If you are using python in the lambda you can utilise the dynamodb-json library.
Install library
pip install dynamodb-json
and use the below snippet
from dynamodb_json import json_util as util
def marshall(regular_json):
dynamodb_json = util.dumps(reular_json)
def unmarshall(dynamodb_json):
regular_json = util.loads(dynamodb_json)
Reference
https://pypi.org/project/dynamodb-json/
I think it's just a custom transformation exercise for each app. A simple conversion from DynamoDB's item format to you application format might look like this:
var response = {...} // your response from DynamoDB
var formattedObjects = response.videos.map(function(video) {
return {
"file": video.file.S,
"id": video.id.S,
"canvas": video.canvas.S
};
});
If you want to build a generic system for this, you would have to handle DynamoDB's various AttributeValue types. A function like the one below would do the job, but I've left out the hard work of handling most of DynamoDB's more complex attribute value types:
function dynamoItemToPlainObj(dynamoItem) {
var plainObj = {};
for (var attributeName in dynamoItem) {
var attribute = dynamoItem[attributeName];
var attributeValue;
for (var itemType in attribute) {
switch (itemType) {
case "S":
attributeValue = attribute.S.toString();
break;
case "N":
attributeValue = Number(attribute.N);
break;
// more attribute types...
default:
attributeValue = attribute[itemType].toString();
break;
}
}
plainObj[attributeName] = attributeValue;
}
return plainObj;
}
var formattedObjects = response.videos.map(dynamoItemToPlainObj);
I tried several solutions here but none worked with multi-level data, such as if it includes a list of maps e.g.
{
"item1": {
"M": {
"sub-item1": {
"L": [
{
"M": {
"sub-item1-list-map": {
"S": "value"
Below, adapted from #igorzg's answer (which also has that drawback), fixes that.
Example usage:
dynamodb.getItem({...}, function(err, data) {
if (!err && data && data.Item) {
var converted = ddb_to_json(data.Item);
Here's the conversion function:
function ddb_to_json(data) {
function isObject(value) {
return typeof value === "object" && value !== null;
}
if(isObject(data))
return convert_ddb({M:data});
function convert_ddb(ddbData) {
if (isObject(ddbData) && ddbData.hasOwnProperty('S'))
return ddbData.S;
if (isObject(ddbData) && ddbData.hasOwnProperty('N'))
return parseFloat(ddbData.N);
if (isObject(ddbData) && ddbData.hasOwnProperty('BOOL'))
return ddbData.BOOL;
if (isObject(ddbData) && ddbData.hasOwnProperty('NULL'))
return null;
if (isObject(ddbData) && ddbData.hasOwnProperty('M')) {
var x = {};
for(var k in ddbData.M)
x[k] = convert_ddb(ddbData.M[k])
return x;
}
if (isObject(ddbData) && ddbData.hasOwnProperty('L'))
return ddbData.L.map(x => convert_ddb(x));
if (isObject(ddbData) && ddbData.hasOwnProperty('SS'))
return ddbData.SS;
if (isObject(ddbData) && ddbData.hasOwnProperty('NN'))
return ddbData.NN;
if (isObject(ddbData) && ddbData.hasOwnProperty('BB'))
return ddbData.BB;
if (isObject(ddbData) && ddbData.hasOwnProperty('NS'))
return ddbData.NS;
if (isObject(ddbData) && ddbData.hasOwnProperty('BS'))
return ddbData.BS;
return data;
}
return data;
}
If you need online editor try this
https://2json.net/dynamo

Can I stop Angular.js’s json filter from excluding properties that start with $?

Angular.js has a handy built-in filter, json, which displays JavaScript objects as nicely formatted JSON.
However, it seems to filter out object properties that begin with $ by default:
Template:
<pre>{{ {'name':'value', 'special':'yes', '$reallyspecial':'Er...'} | json }}</pre>
Displayed:
{
"name": "value",
"special": "yes"
}
http://plnkr.co/edit/oem4HJ9utZMYGVbPkT6N?p=preview
Can I make properties beginning with $ be displayed like other properties?
Basically you can't. It is "hard-coded" into the filter's behaviour.
Nonetheless, it is quite easy to build a custom JSON filter that behaves identically with the Angular's one but not filtering out properties starting with '$'.
(Scroll further down for sample code and a short demo.)
If you take a look at the 1.2.15 version source code, you will find out that the json filter is defined like this:
function jsonFilter() {
return function(object) {
return toJson(object, true);
};
}
So, it uses the toJson() function (the second parameter (true) means: format my JSON nicely).
So, our next stop is the toJson() function, that looks like this:
function toJson(obj, pretty) {
if (typeof obj === 'undefined') return undefined;
return JSON.stringify(obj, toJsonReplacer, pretty ? ' ' : null);
}
This function makes use of the "native" JSON.stringify() function, passing a custom replacer function (toJsonReplacer).
The toJsonReplacer() function handles some special cases: It checks if the key starts with $ and ignores it if it does (this is what we want to change) and it checks if the value is either a Window, a Document or a Scope object (in which case it converts it to a descriptive string in order to avoid "Converting circular structure to JSON" errors).
function toJsonReplacer(key, value) {
var val = value;
if (typeof key === 'string' && key.charAt(0) === '$') {
val = undefined;
} else if (isWindow(value)) {
val = '$WINDOW';
} else if (value && document === value) {
val = '$DOCUMENT';
} else if (isScope(value)) {
val = '$SCOPE';
}
return val;
}
For the sake of completeness, the two functions that check for Window and Scope look like this:
function isWindow(obj) {
return obj && obj.document && obj.location && obj.alert && obj.setInterval;
}
function isScope(obj) {
return obj && obj.$evalAsync && obj.$watch;
}
Finally, all we need to do is to create a custom filter that uses the exact same code, with the sole difference that our toJsonReplacer() won't filter out properties starting with $.
app.filter('customJson', function () {
function isWindow(obj) {
return obj &&
obj.document &&
obj.location &&
obj.alert &&
obj.setInterval;
}
function isScope(obj) {
return obj &&
obj.$evalAsync &&
obj.$watch;
}
function toJsonReplacer(key, value) {
var val = value;
if (isWindow(value)) {
val = '$WINDOW';
} else if (value && (document === value)) {
val = '$DOCUMENT';
} else if (isScope(value)) {
val = '$SCOPE';
}
return val;
}
function toJson(obj, pretty) {
if (typeof obj === 'undefined') { return undefined; }
return JSON.stringify(obj, toJsonReplacer, pretty ? ' ' : null);
}
return function(object) {
return toJson(object, true);
};
});
See, also, this short demo.
* The downside is that your custom JSON filter will not benefit from further improvement/enhancement of Angular's json filter, so you'll have to re-define your's to incorporate changes. Of course, for such a basic and simple filter like this, one should'nt expect frequent or extensive changes, but that doesn't mean there aren't going to be any.