JSON.stringify add a carriage return - json

I am using postman for batch API calls and want a new line between each records (for the ease of others copy and pasting into csv/excel)
let responses = pm.collectionVariables.get('collectionResponses')
if(responses) {
responses = JSON.parse(responses);
} else {
responses = []
}
responses.push(pm.response.json());
pm.collectionVariables.set('collectionResponses', JSON.stringify(responses));
I tried
JSON.stringify(responses, '},', '},\n')
and that did not work
This is what the output looks like
{"success":true,"data":"0.5950391865600001\t0.49508964727322147\t193.383783","id":"2ec0a50f-862e-11ec-a41f-06c185e97372"},
{"success":true,"data":"0.5950391865600001\t0.49508964727322147","id":"410113f9-8630-11ec-a41f-06c185e97372"},

The simplest, most reliable way of doing this is to stringify each array element individually rather than all together:
const json = theArray.map(el => JSON.stringify(el)).join(",\n");
Live Example:
const theArray = [
{id: 1},
{id: 2,},
{id: 3},
];
const json = theArray.map(el => JSON.stringify(el)).join(",\n");
console.log(json);
I've left off the [] there because you seemed not to want them, but of course you could add them (`"[" + theArray.map(/.../) + "\n]").
Although JSON.stringify accepts a third parameter you can use for indentation (which triggers pretty-printing), it's awkard to use it for this purpose. You could provide a single space and then replace "\n " since a literal newline won't appear within a JSON value:
const json = JSON.stringify([1, 2, 3], null, 1)
.replace(/\n /g, "\n");
console.log(json);
...but it's a bit hacky.

Related

Bad JSON escape sequence: \\1. Path 'sections[0].facts[0].value', line 1, position 196."

Am getting getting Bad JSON escape sequence: \\1. Path 'sections[0].facts[0].value', line 1, position 196." while parsing json string with string.Format to add data based on placeholder.
Below is the code :
string jsonFormatString = #"{{""title"": ""{0}"",""sections"": [{{""facts"": [{{""name"": ""TransactionNo"",""value"": ""{1}""}}]}}]}}";
string formattedJson= string.Format(jsonFormatString , message, transactionNo ?? "");
var result = JsonConvert.DeserializeObject<dynamic>(formattedJson);
The string values for placeholders were message = "Transaction Success", transactionNo = "1920\01\ABC";
Looks like escape character transactionNo field is creating problem.
I tried with string.Replace(#"\\", #"\") and others nothing helped in resolving the error.
If i added string.Replace(#"\", #"\\", it works but result json contains transactionNo as "1920\\01\\ABC" which is also wrong.
Am i missing something or should i add anything more
I have seen folks do this in the past and even had to maintain code that looks like this... and as you are experiencing: It is a huge PITA.
You can easily lay out your objects using anonymous types so they are simple to maintain and understand for the folks who may have the pleasure to work with your code in the future.
Take a look at this:
var myObj = new {
title = "<input value>",
sections = new[]
{
new
{
facts = new[]
{
new
{
name = "TransactionNo",
value = "<input value>"
}
}
}
}
};
var result = JsonConvert.SerializeObject(myObj, Formatting.Indented);
Result will look like this:
{
"title": "<input value>",
"sections": [
{
"facts": [
{
"name": "TransactionNo",
"value": "<input value>"
}
]
}
]
}
Now, keep in mind that this is easier to maintain than formatting a string, but at the end of the day it's better to just bite the bullet and create some concrete models.

Getting image URL's from amazon product page

I'm trying to scrape image URL's from Amazon products, for example, this link.
In the page source code, there is a section which contains all the urls for images of different sizes (large, medium, hirez, etc). I can get that part of the script by doing, with scrapy,
imagesString = (response.xpath('//script[contains(., "ImageBlockATF")]/text()').extract_first())
Which gives me a string that looks like this,
P.when('A').register("ImageBlockATF", function(A){
var data = {
'colorImages': { 'initial': [{"hiRes":"https://images-na.ssl-images-amazon.com/images/I/81FED1p-sTL._SL1500_.jpg","thumb":"https://images-na.ssl-images-amazon.com/images/I/31HoKqtljqL._SS40_.jpg","large":"https://images-na.ssl-images-amazon.com/images/I/31HoKqtljqL.jpg","main":{"https://images-na.ssl-images-amazon.com/images/I/81FED1p-sTL._SX355_.jpg":[308,355],"https://images-na.ssl-images-amazon.com/images/I/81FED1p-sTL._SX450_.jpg":[390,450],"https://images-na.ssl-images-amazon.com/images/I/81FED1p-sTL._SX425_.jpg":[369,425],"https://images-na.ssl-images-amazon.com/images/I/81FED1p-sTL._SX466_.jpg":[404,466],"https://images-na.ssl-images-amazon.com/images/I/81FED1p-sTL._SX522_.jpg":[453,522],"https://images-na.ssl-images-amazon.com/images/I/81FED1p-sTL._SX569_.jpg":[494,569],"https://images-na.ssl-images-amazon.com/images/I/81FED1p-sTL._SX679_.jpg":[589,679]},"variant":"MAIN","lowRes":null},{"hiRes":"https://images-na.ssl-images-amazon.com/images/I/81e8905DlhL._SL1500_.jpg","thumb":"https://images-na.ssl-images-amazon.com/images/I/31Y%2B8oE5DtL._SS40_.jpg","large":"https://images-na.ssl-images-amazon.com/images/I/31Y%2B8oE5DtL.jpg","main":{"https://images-na.ssl-images-amazon.com/images/I/81e8905DlhL._SX355_.jpg":[308,355],"https://images-na.ssl-images-amazon.com/images/I/81e8905DlhL._SX450_.jpg":[390,450],"https://images-na.ssl-images-amazon.com/images/I/81e8905DlhL._SX425_.jpg":[369,425],"https://images-na.ssl-images-amazon.com/images/I/81e8905DlhL._SX466_.jpg":[404,466],"https://images-na.ssl-images-amazon.com/images/I/81e8905DlhL._SX522_.jpg":[453,522],"https://images-na.ssl-images-amazon.com/images/I/81e8905DlhL._SX569_.jpg":[494,569],"https://images-na.ssl-images-amazon.com/images/I/81e8905DlhL._SX679_.jpg":[589,679]},"variant":"PT01","lowRes":null},{"hiRes":null,"thumb":"https://images-na.ssl-images-amazon.com/images/I/51rORrvh0hL._SS40_.jpg","large":"https://images-na.ssl-images-amazon.com/images/I/51rORrvh0hL.jpg","main":{"https://images-na.ssl-images-amazon.com/images/I/51rORrvh0hL._SX355_.jpg":[236,355],"https://images-na.ssl-images-amazon.com/images/I/51rORrvh0hL._SX450_.jpg":[300,450],"https://images-na.ssl-images-amazon.com/images/I/51rORrvh0hL._SX425_.jpg":[283,425],"https://images-na.ssl-images-amazon.com/images/I/51rORrvh0hL._SX466_.jpg":[310,466],"https://images-na.ssl-images-amazon.com/images/I/51rORrvh0hL.jpg":[333,500]},"variant":"PT02","lowRes":null},{"hiRes":null,"thumb":"https://images-na.ssl-images-amazon.com/images/I/41L2OU5rPyL._SS40_.jpg","large":"https://images-na.ssl-images-amazon.com/images/I/41L2OU5rPyL.jpg","main":{"https://images-na.ssl-images-amazon.com/images/I/41L2OU5rPyL._SX355_.jpg":[236,355],"https://images-na.ssl-images-amazon.com/images/I/41L2OU5rPyL._SX450_.jpg":[300,450],"https://images-na.ssl-images-amazon.com/images/I/41L2OU5rPyL._SX425_.jpg":[283,425],"https://images-na.ssl-images-amazon.com/images/I/41L2OU5rPyL._SX466_.jpg":[310,466],"https://images-na.ssl-images-amazon.com/images/I/41L2OU5rPyL.jpg":[333,500]},"variant":"PT03","lowRes":null},{"hiRes":null,"thumb":"https://images-na.ssl-images-amazon.com/images/I/51%2BsCYjx6OL._SS40_.jpg","large":"https://images-na.ssl-images-amazon.com/images/I/51%2BsCYjx6OL.jpg","main":{"https://images-na.ssl-images-amazon.com/images/I/51%2BsCYjx6OL._SX355_.jpg":[236,355],"https://images-na.ssl-images-amazon.com/images/I/51%2BsCYjx6OL._SX450_.jpg":[300,450],"https://images-na.ssl-images-amazon.com/images/I/51%2BsCYjx6OL._SX425_.jpg":[283,425],"https://images-na.ssl-images-amazon.com/images/I/51%2BsCYjx6OL._SX466_.jpg":[310,466],"https://images-na.ssl-images-amazon.com/images/I/51%2BsCYjx6OL.jpg":[333,500]},"variant":"PT04","lowRes":null}]},
'colorToAsin': {'initial': {}},
'holderRatio': 1.0,
'holderMaxHeight': 700,
'heroImage': {'initial': []},
'heroVideo': {'initial': []},
'spin360ColorData': {'initial': {}},
'spin360ColorEnabled': {'initial': 0},
'spin360ConfigEnabled': false,
'spin360LazyLoadEnabled': false,
'playVideoInImmersiveView':'false',
'tabbedImmersiveViewTreatment':'T2',
'totalVideoCount':'0',
'videoIngressATFSlateThumbURL':'',
'mediaTypeCount':'0',
'atfEnhancedHoverOverlay' : true,
'winningAsin': 'B00XLSS79Y',
'weblabs' : {},
'aibExp3Layout' : 1,
'aibRuleName' : 'frank-powered',
'acEnabled' : false
};
A.trigger('P.AboveTheFold'); // trigger ATF event.
return data;
});
My goal is to get into a Json dictionary the data inside colorImages, so then I can easily get each URL.
I tried doing something like this:
m = re.search(r'^var data = ({.*};)', imagesString , re.S | re.M)
data = m.group()
jsonObj = json.loads(data[:-1].replace("'", '"'))
But it seems that imagesString does not work well with re.search, I keep getting errors regarding imagesString not being a string when it actually is.
I got similar data from an amazon page by using re.findall, something like this (script is a chunk of text i got from the page).
variationValues = re.findall(r'variationValues\" : ({.*?})', ' '.join(script))[0]
and then
variationValuesDict = json.loads(variationValues)
But my knowledge of regular expressions is not that great.
From the string I pasted above, I erased the start and end so only the data remained, so I was left with this:
https://jsoneditoronline.org/?id=9ea92643044f4ac88bcc3e76d98425fc
I can't figure out how to get colorImages with re.findall() (or the data in the json editor) so I can then load it into Json and use it like a dictionary, any ideas on how to achieve this?
You just need to initially convert the var data to the correct markup json. It is easy ))) Just replace all chars ' to " and delete SPACES. And you will get json object:
(It's your right json)

Add Headers to CSV and Return Row with Matching String in Node.js

I am trying to return a row value on a matching string.
I begin with a csv that doesn't have header details. After loading the csv into memory, I think I need to first add headers and then convert the data to json, then loop through the data to find the correct object in the array.
I have used highland to create a read stream and output to objects. However, the line break characters \r and \r\n are not getting parsed out and are injected into the value strings.
This approach also doesn't seem to parse the entire file, it is outputting 3 lines.
highland(fs.createReadStream('example.csv', 'utf8'))
.map(line => line.split(','))
.map(parts => ({
a: parts[0],
b: parts[1],
c: parts[2]
}))
.each(x => console.log(x))
It would be nice to structure the CSV as JSON and then use the .filter() method to match the record.
Actual output of 1000+ line file
{ a: '', b: 'CD53110' }
{ a: '\nRD40115', b: 'CD40315' }
{ a: '', b: '63\r\nRE15468' }
{ a: '96798', b: '5\r\nRR60899' }
Example input snippet
,CD510,13
,T9069,65
RCM22,TC633,101
RC023,87693,16
M2024,T7636,109
Note: first few rows only contain 'b' and 'c' columns.
You can write the headers out as a string, set the csv to memory, and concatenate the two.
Then with the csvtojson module you can map the csv to json and use the native filter function to return the matching row.
const fs = require('fs')
const csv = require('csvtojson')
let headers = 'a,b,c\n'
let file = fs.readFileSync(__dirname + 'example.csv', 'utf8')
let strToMatch = 'def123'
let c = headers + file
csv().fromString(c)
.then((json) => {
let arr = json.filter(el => el.a == strToMatch)
res.json({
a: arr[0]['a'],
b: arr[0]['b'],
c: arr[0]['c']
})
})

Regular expression to extract a JSON array

I'm trying to use a PCRE regular expression to extract some JSON. I'm using a version of MariaDB which does not have JSON functions but does have REGEX functions.
My string is:
{"device_types":["smartphone"],"isps":["a","B"],"network_types":[],"countries":[],"category":["Jebb","Bush"],"carriers":[],"exclude_carriers":[]}
I want to grab the contents of category. I'd like a matching group that contains 2 items, Jebb and Bush (or however many items are in the array).
I've tried this pattern but it only matches the first occurrence: /(?<=category":\[).([^"]*).*?(?=\])/g
Does this match your needs? It should match the category array regardless of its size.
"category":(\[.*?\])
regex101 example
JSON not a regular language. Since it allows arbitrary embedding of balanced delimiters, it must be at least context-free.
For example, consider an array of arrays of arrays:
[ [ [ 1, 2], [2, 3] ] , [ [ 3, 4], [ 4, 5] ] ]
Clearly you couldn't parse that with true regular expressions.
See This Topic:
Regex for parsing single key: values out of JSON in Javascript
Maybe Helpful for you.
Using a set of non-capturing group you can extract a predefined json array
regex answer: (?:\"category\":)(?:\[)(.*)(?:\"\])
That expression extract "category":["Jebb","Bush"], so access the first group
to extract the array, sample java code:
Pattern pattern = Pattern.compile("(?:\"category\":)(?:\\[)(.*)(?:\"\\])");
String body = "{\"device_types\":[\"smartphone\"],\"isps\":[\"a\",\"B\"],\"network_types\":[],\"countries\":[],\"category\":[\"Jebb\",\"Bush\"],\"carriers\":[],\"exclude_carriers\":[]}";
Matcher matcher = pattern.matcher(body);
assertThat(matcher.find(), is(true));
String[] categories = matcher.group(1).replaceAll("\"","").split(",");
assertThat(categories.length, is(2));
assertThat(categories[0], is("Jebb"));
assertThat(categories[1], is("Bush"));
There are many ways. One sloppy way to do it is /([A-Z])\w+/g
Please try it on your console like
var data = '{"device_types":["smartphone"],"isps":["a","B"],"network_types":[],"countries":[],"category":["Jebb","Bush"],"carriers":[],"exclude_carriers":[]}',
res = [];
data.match(/([A-Z])\w+/g); // ["Jebb", "Bush"]
OK the above was pretty sloppy however a solid single regex solution to extract every single element regardless of the number, one by one and to place them in an array (res) is the following...
var rex = /[",]+(\w*)(?=[",\w]*"],"carriers)/g,
str = '{"device_types":["smartphone"],"isps":["a","B"],"network_types":[],"countries":[],"category":["Jebb","Bush","Donald","Trump"],"carriers":[],"exclude_carriers":[]}',
arr = [],
res = [];
while ((arr = rex.exec(str)) !== null) {
res.push(arr[1]); // <- ["Jebb", "Bush", "Donald", "Trump"]
}
Check it out # http://regexr.com/3d4ee
OK lets do it. I have come up with a devilish idea. If JS had look-behinds this could have been done simply by reversing the applied logic in the previous example where i had used a look-forward. Alas, there aren't... So i decided to turn the world the other way around. Check this out.
String.prototype.reverse = function(){
return this.split("").reverse().join("");
};
var rex = /[",]+(\w*)(?=[",\w]*"\[:"yrogetac)/g,
str = '{"device_types":["smartphone"],"isps":["a","B"],"network_types":[],"countries":[],"category":["Jebb","Bush","Donald","Trump"],"carriers":[],"exclude_carriers":[]}',
rev = str.reverse();
arr = [],
res = [];
while ((arr = rex.exec(rev)) !== null) {
res.push(arr[1].reverse()); // <- ["Trump", "Donald", "Bush", "Jebb"]
}
res.reverse(); // <- ["Jebb", "Bush", "Donald", "Trump"]
Just use your console to confirm.
In c++ you can do it like this
bool foundmatch = false;
try {
std::regex re("\"([a-zA-Z]+)\"*.:*.\\[[^\\]\r\n]+\\]");
foundmatch = std::regex_search(subject, re);
} catch (std::regex_error& e) {
// Syntax error in the regular expression
}
If the number of items in the array is limited (and manageable), you could define it with a finite number of optional items. Like this one with a maximum of 5 items:
"category":\["([^"]*)"(?:,"([^"]*)"(?:,"([^"]*)"(?:,"([^"]*)"(?:,"([^"]*)")?)?)?)?
regex101 example here.
Regards.

Node-red SQL output object / Array conversion

I'm doing a SQL query in Node-Red to output a load of time/value data. This data is then passed to a web page for display in a graph.
Previously I've used php to do the SQL query, which I'm trying to replace. However SQL queries in php are delivered in a different format.
With Node-Red, I get:
[
{
"Watts": 1018,
"Time": 1453825454
},
{
"Watts": 1018,
"Time": 1453825448
},
{
"Watts": 1010,
"Time": 1453825442
}]
With PHP, I get:
[
[1453819620000,962],
[1453819614000,950],
[1453819608000,967],
[1453819602000,947]
]
I think I'm getting an array from php and an array of JSON objects from Node-Red. How do I convert the Node-Red object to be output from Node-Red in the same format as the PHP is? (Ie: I want to handle the processing at the server, rather than the client.)
A function node can be used to generate something in the same format.
var array = msg.payload;
var phpFormat = "[";
for (var i=0; i<array.length; i++) {
phpFormat += "[" +
// time format differ, NodeJS is in seconds
// php is in milliseconds
(array[i].Time * 1000 ) +
"," +
array[i].Watts + "],";
}
//take the last "," off
phpFormat = phpFormat.substring(0,phpFormat.lenght - 1);
phpFormat += "]";
msg.payload = phpFormat;
return msg;
I've had a bit of help from a chap at work and here is what he's come up with, modified for node-red by me:
var outputArray = [];
for(var i in msg.payload){
var entryData = [msg.payload[i]['Time']];
for(var attr in msg.payload[i]) {
if(attr!='Time') {
entryData.push(msg.payload[i][attr])}
};
outputArray.push(entryData); }
var returnMsg={"payload":outputArray};
return returnMsg;
I know, I know, this question is over 2 years old... however, for the next 500 people seeking an answer to a similar problem, I'd like to highlight the new JSONata expression feature built-in to the change node. Using this simple expression:
payload.[Time, Watts]
transforms your JS objects into the requested output of an array of arrays. In fact, much of my old repetitive looping through arrays has been replaced with some simpler (to me) expressions like this.
The magic of the lambda syntax evaluator is documented on the JSONata site. There you will also find the online exerciser where you can build an expression against your own data and immediately see the resulting structure.
Note: in order to use a jsonata expression in your change node, be sure to select the J: pulldown next to the input field (not the {} JSON option)... two totally different things!