Filter out duplicates from a JSON array - json

Lets say I have a JSON file with an array in it like this:
[
{
"distinct_id": "abc",
"properties": {
"bookID": "123",
"userID": "abc",
}
},
{
"distinct_id": "abc",
"properties": {
"bookID": "123",
"userID": "abc",
}
},
{
"distinct_id": "ced",
"properties": {
"bookID": "123",
"userID": "ced",
}
},
{
"distinct_id": "abc",
"properties": {
"bookID": "456",
"userID": "ced",
}
}
]
I am trying to figure out how I would loop through this and return a new array just one instance of each distinct_id (basically removing duplicates) - think of it like a an array of unique users.
I am using TypeScript. I was trying to use Set but that wasn't working.

interface Item {
distinct_id: string;
properties: {
bookID: string;
userID: string;
};
}
const getDistinctIds = (data: Item[]) => data.map(item => item.distinct_id);
const getUniqueDistinctIds = (data: Item[]) => Array.from(new Set(getDistinctIds(data)));
const getItemByDistinctId = (data: Item[], distinct_id: string) => data.find(item => item.distinct_id === distinct_id);
const getUniqueItems = (data: Item[]) => getUniqueDistinctIds(data).map(distinct_id => getItemByDistinctId(data, distinct_id));
const data: Item[] = [
{
"distinct_id": "abc",
"properties": {
"bookID": "123",
"userID": "abc",
}
},
{
"distinct_id": "abc",
"properties": {
"bookID": "123",
"userID": "abc",
}
},
{
"distinct_id": "ced",
"properties": {
"bookID": "123",
"userID": "ced",
}
},
{
"distinct_id": "abc",
"properties": {
"bookID": "456",
"userID": "ced",
}
}
];
const uniqueItems = getUniqueItems(data);
console.log(uniqueItems);
Here is link to TS PLAYGROUND

Related

how to merge multiple json files with same structure into one json file with same structure (combined all into one with keeping same structure))

I need to merge file1.json file2.json (could be more) into onefile.json.
version is always the same value in all files. however vulnerabilities array and dependency_files array values different but there might be duplicate/which I want to remove if any after the merge
file1.json:
{
"version": "x.x.x",
"vulnerabilities": [
{
"id": "0000"
},
{
"id": "11111"
},
{
"id": "2222"
}
],
"dependency_files": [
{
"name": "name0000"
},
{
"name": "name1111"
},
{
"name": "name2222"
}
]
}
file2.json:
{
"version": "x.x.x",
"vulnerabilities": [
{
"id": "2222"
},
{
"id": "3333"
}
],
"dependency_files": [
{
"name": "name2222"
},
{
"name": "name3333"
}
]
}
onefile.json:
{
"version": "x.x.x",
"vulnerabilities": [
{
"id": "0000"
},
{
"id": "11111"
},
{
"id": "2222"
},
{
"id": "3333"
}
],
"dependency_files": [
{
"name": "name0000"
},
{
"name": "name1111"
},
{
"name": "name2222"
},
{
"name": "name3333"
}
]
}
I tried a lot with no luck
You could have a reduce on all files, initialized with the first, hence no need for the -n option:
jq '
reduce inputs as {$vulnerabilities, $dependency_files} (.;
.vulnerabilities = (.vulnerabilities + $vulnerabilities | unique_by(.id))
| .dependency_files = (.dependency_files + $dependency_files | unique_by(.name))
)
' file*.json
{
"version": "x.x.x",
"vulnerabilities": [
{
"id": "0000"
},
{
"id": "11111"
},
{
"id": "2222"
},
{
"id": "3333"
}
],
"dependency_files": [
{
"name": "name0000"
},
{
"name": "name1111"
},
{
"name": "name2222"
},
{
"name": "name3333"
}
]
}
Demo
Using this python code
import json
def merge_dicts(*dicts):
r = {}
skip = 'version'
for item in dicts:
for key, value in item.items():
if (key == skip):
r[skip] = value
else:
r.setdefault(key, []).extend(value)
unique = []
for obj in r[key]:
if obj not in unique:
unique.append(obj)
r[key] = unique
return r
with open("file1.json") as file_1:
data_1 = json.load(file_1)
with open("file2.json") as file_2:
data_2 = json.load(file_2)
with open('data.json', 'w') as merge_file:
json.dump(merge_dicts(data_1, data_2), merge_file, indent = 4)
Result
{
"version": "x.x.x",
"vulnerabilities": [
{
"id": "0000"
},
{
"id": "11111"
},
{
"id": "2222"
},
{
"id": "3333"
}
],
"dependency_files": [
{
"name": "name0000"
},
{
"name": "name1111"
},
{
"name": "name2222"
},
{
"name": "name3333"
}
]
}
This code is multiple json files support
import os, json
def merge_dicts(*dicts):
r = {}
skip = 'version'
for item in dicts:
for key, value in item.items():
if (key == skip):
r[skip] = value
else:
r.setdefault(key, []).extend(value)
unique = []
for obj in r[key]:
if obj not in unique:
unique.append(obj)
r[key] = unique
return r
json_files = [pos_json for pos_json in os.listdir('./') if pos_json.endswith('.json')]
a = []
print(type(a))
for json_file in json_files:
with open(json_file) as file_item:
read_data = json.load(file_item)
a.append(read_data)
file_item.close()
with open('data.json', 'w') as merge_file:
json.dump(merge_dicts(*tuple(a)), merge_file, indent = 4)

Flutter: parsing json response that has multiple keys

I'm trying to get the values of these keys from this json response:
{
"pro": {
"groups": [
"1": {
"name": "Base",
"fields": [
{
"id": 3,
"value": {
"raw": "Name",
}
},
{
"id": 4,
"value": {
"raw": "avatar",
}
},
]
},
"2": {
"name": "Base",
"fields": [
{
"id": 6,
"value": {
"raw": "Name",
}
},
{
"id": 7,
"value": {
"raw": "avatar",
}
},
]
}
]
}
}
I could get the values "name": "Base"
json['pro']['groups']["1"]['name'],
But I can't get the values of key "raw".
How can I get the values of key "raw"?
The values of fields are a list, so you will get a list of raw values:
List<String> raw = json['pro']['groups']['1']['fields'].map((v) => v['value']['raw'];
Also, it seems like groups is an array but as an object? then you can do something like this:
List<String> raw = [];
Map<String, dynamic> groups = json['pro']['groups'];
for (var key in groups.keys) {
raw.add(groups[key]['fields'].map((v) => v['value']['raw']);
}
or
List<String> raw = groups.keys.map((key) => groups[key]['fields'].map((v) => v['value']['raw']);
I haven't tested the code, but hopefully, it works as expected :)
first thing first. your json is invalid.
try to paste your json here it will show why is your json is invalid
after you fix the json the new structure json will be looked like this
{
"pro": {
"groups": [
{
"name": "Base",
"fields": [
{
"id": 3,
"value": {
"raw": "Name",
}
},
{
"id": 4,
"value": {
"raw": "avatar",
}
},
]
},
{
"name": "Base",
"fields": [
{
"id": 6,
"value": {
"raw": "Name",
}
},
{
"id": 7,
"value": {
"raw": "avatar",
}
},
]
}
]
}
}
and then in order to grab the value of raw, you have to parse the json first using jsonDecode(), then you can use something like this:
Map<String, dynamic> groupOne = json['pro']['groups'][0];
Map<String, dynamic> groupOneFieldOne = groupOne['fields'][0];
print(groupOneFieldOne['value']['raw']);
but that's just an example. if you want to access them easily you can use .map() like this:
List<Map<String, dynamic>> groups = json['pro']['groups'];
groups.map(
(Map<String, dynamic> group) => (group['fields'] as List<dynamic>).map(
(dynamic field) => field['value']['raw'],
),
);
that's it! if you want to ask anything just put a comment ;)
you can copy and paste on dartpad
List<Map<String, dynamic>> groups = json['pro']['groups'];
print(groups.map(
(Map<String, dynamic> group) => (group['fields'] as List<dynamic>).map(
(dynamic field) => field['value']['raw'],
),
));
}
Map<String, dynamic> json = {
"pro": {
"groups": [
{
"name": "Base",
"fields": [
{
"id": 3,
"value": {
"raw": "Name",
}
},
{
"id": 4,
"value": {
"raw": "avatar",
}
},
]
},
{
"name": "Base",
"fields": [
{
"id": 6,
"value": {
"raw": "Name",
}
},
{
"id": 7,
"value": {
"raw": "avatar",
}
},
]
}
]
}
};

Modify JSON after getting a response from sequelize

I get a JSON from my sequelize function. This i have to modify, because i have to send it to a database importer, who needs it in a fixed form.
Is there a way to customize this function so that I get back the desired result ?
models:
User.associate = function (models) {
User.hasMany(models.SurveyResult)
}`
SurveyResult.associate = function (models) {
SurveyResult.belongsTo(models.User)
The function:
async mediImport (req, res) {
try {
const transaction = await User.findAll({
where: { released: true },
// Select forename as Vorname, name as Nachname
attributes: [
['forename', 'PAPPS286'],
['name', 'Nachname'],
['birthdate', 'PADPS60']
],
include: [{ model: SurveyResult, attributes: ['result'] }]
}).map(user => user.toJSON())
res.send({
transaction
}
)
}
this is the JSON which i get from the function:
{
"transaction": [
{
"PAPPS286": "Tes",
"Nachname": "Josef",
"PADPS60": null,
"SurveyResults": [ {
"result": {
"name": "blau",
"email": "mail",
"birthdate": "01.02.1990"
}
}]
},
{
"PAPPS286": "Dampf",
"Nachname": "Hans",
"PADPS60": null,
"SurveyResults": [
{
"result": {
"name": "blau",
"email": "mail",
"birthdate": "01.02.1990"
}
}
]
},
]
},
This is the JSON form i need:
{
"transaction": [
PAD{
"PAPPS286": "Tes",
"Nachname": "Josef",
"PADPS60": null,
"MH": {
"name": "blau",
"email": "mail",
"birthdate": "01.02.1990"
},
PAD{
"PAPPS286": "Dampf",
"Nachname": "Hans",
"PADPS60": null,
"MH": {
"name": "blau",
"email": "mail",
"birthdate": "01.02.1990"
}
},
]
},
Perhaps there is a way to modify the JSON after i get it back. But i dont Knoe how this can be work.
The below map does what I think you are looking for. You may need to modify it, especially around accessing the variables you need, based on the key. (t.PAPPS286 and t.PADPS60)
Additionally, I am only grabbing the first survey result SurveyResults. Unsure what you want if there are none or if there are more than 1.
var obj = {
"transaction": [{
"PAPPS286": "Tes",
"Nachname": "Josef",
"PADPS60": null,
"SurveyResults": [{
"result": {
"name": "blau",
"email": "mail",
"birthdate": "01.02.1990"
}
}]
},
{
"PAPPS286": "Dampf",
"Nachname": "Hans",
"PADPS60": null,
"SurveyResults": [{
"result": {
"name": "blau",
"email": "mail",
"birthdate": "01.02.1990"
}
}]
},
]
}
obj.transaction = obj.transaction.map((t) => {
return Object.assign({
"PAPPS286": t.PAPPS286,
"Nachname": t.Nachname,
"PADPS60": t.PADPS60,
"MH": {
"name": t.SurveyResults[0].result.name,
"email": t.SurveyResults[0].result.email,
"birthdate": t.SurveyResults[0].result.birthdate
}
})
})
console.log(obj);

Re-arrange JSON file (using adjacency matrix)

I have a json file that looks like this:
[
{
"id": 1,
"country": "Greece",
"names": [
"Alex",
"Betty",
"Catherine",
"Dave",
"Edward",
"Frank",
"George",
"Helen",
"Irene"
]
},
{
"id": 2,
"country": "US",
"names": [
"John",
"Alex",
"Edward",
"Kate",
"Robert",
"Irene",
"Tim",
"Sam"
]
},
{
"id": 3,
"country": "France",
"names": [
"Helen",
"Kate",
"Louise",
"Tim",
"Catherine",
"Arthur",
"Frank",
"Natalie",
"Dave"
]
},
{
"id": 4,
"country": "India",
"names": [
"Ritesh",
"Alex",
"Betty",
"Robert"
]
},
{
"id": 5,
"country": "India",
"names": [
"Nafeez",
"Tom",
"Natalie",
"Gunar",
"Louise",
"Arthur"
]
}
]
I want it to be "name centered" and look like this:
{
"groups": [
{
"gr_id":1
"name":"Alex",
"country":"Greece"
},
.........
{
"gr_id":1
"name":"Irene",
"country":"Greece"
},
{
"gr_id":2
"name":"John",
"country":"US"
..........
{
"gr_id":2
"name":"Sam",
"country":"US"
},
{
"gr_id":3
"name":"Helen",
"country":"France"
},
.........
{
"gr_id":3
"name":"Dave",
"country":"France"
},
{
"gr_id":4
"name":"Ritesh",
"country":"India"
},
........
{
"gr_id":4
"name":"Robert",
"country":"India"
},
{
"gr_id":5
"name":"Nafeez",
"country":"India"
},
...........
{
"gr_id":5
"name":"Arthur",
"country":"India"
}
],
"links": [
{
"source":"Alex"
"target":"Irene",
"count":1
"country":"Greece"
},
...
{
"source":"Alex"
"target":"Arthur",
"count":0
"country":"India"
},
...
]
}
For count in Links I have an adjacency matrix for each country/name (csv format) like this :screenshot of csv file (ad. matrix for India)
This json is just an example. I have much bigger file (I need it for D3 graph visualization)
Reduce() and map() work perfectly for this. This basically takes each item and then maps over the names, appending the results of map() to an array:
let obj = {}
obj.groups = json.reduce(
(acc, curr) => acc.concat(curr.names.map(
item => ({gr_id: curr.id, country: curr.country, name: item})
)), [])
console.log(obj)
// { groups:
// [ { gr_id: 1, country: 'Greece', name: 'Alex' },
// { gr_id: 1, country: 'Greece', name: 'Betty' },
// ...etc
// ]
// }

How to check if a key exists in a nested JSON object in node?

I've got the following JSON being sent to the server from the browser:
{
"title": "Testing again 2",
"abstract": "An example document",
"_href": "http://google.com",
"tags": [ "person" ],
"attributes": [ {
"id": 1,
"type": "TEXT",
"data": "test"
} ],
"sections": [ {
"id": 1,
"type": "LIST",
"data": [ {
"revision": 124,
"text": "test"
} ]
} ]
}
I need to make sure that the keys "_href", "id" and "revision" are not in the object anyplace at any level.
I found this but it doesn't quite work.
I searched npms.io and found has-any-deep which you can use after JSON.parse ing the JSON.
you need to parse json then check into the data
var str = '{
"title": "Testing again 2",
"abstract": "An example document",
"_href": "http://google.com",
"tags": [ "person" ],
"attributes": [ {
"id": 1,
"type": "TEXT",
"data": "test"
} ],
"sections": [ {
"id": 1,
"type": "LIST",
"data": [ {
"revision": 124,
"text": "test"
} ]
} ]
}';
var jsonObj = JSON.parse(str);
if ( typeof jsonObj._href == 'undefined') {
// check
}
A simple but not 100% foolproof solution would be to parse the JSON to string, and just search for your keys:
var a = JSON.stringify(JSONObject);
var occurs = false;
['"_href"', '"id"', '"version"'].forEach(function(string) {
if(a.indexOf(string) > -1) occurs = true;
});
The issue of course, is if there are values that match
'_href', 'id', 'version' in your JSON. But if you want to use native JS, I guess this is a good bet.
var a = {
"title": "Testing again 2",
"abstract": "An example document",
"tags": [ "person" ],
"attributes": [ {
"type": "TEXT",
"data": "test"
} ],
"sections": [ {
"type": "_href asdad",
"data": [ {
"text": "test"
} ]
} ]
},
b = {
"title": "Testing again 2",
"abstract": "An example document",
"_href": "http://google.com",
"tags": [ "person" ],
"attributes": [ {
"id": 1,
"type": "TEXT",
"data": "test"
} ],
"sections": [ {
"id": 1,
"type": "LIST",
"data": [ {
"revision": 124,
"text": "test"
} ]
} ]
},
aJson = JSON.stringify(a),
bJson = JSON.stringify(b);
var occursa = false, occursb = false;
['"_href"', '"id"', '"version"'].forEach(function(string) {
if(aJson.indexOf(string) > -1) { occursa = true};
});
['"_href"', '"id"', '"version"'].forEach(function(string) {
if(bJson.indexOf(string) > -1) { occursb = true};
});
console.log("a");
console.log(occursa);
console.log("b");
console.log(occursb);
You could use the optional second reviver parameter to JSON.parse for this:
function hasBadProp(json) {
let badProp = false;
JSON.parse(json, (k, v) => {
if ([_href", "id", "revision"].includes(k)) badProp = true;
return v;
});
return badProp;
}