Related
I'm new in the community and I'm not a dev, but I have a task I need to find a solution for. I hope I can get your ideas.
I have a set of JSON files. I want to be able to use jq or a command line that can help me unify the files into one single file.
For example:
File 1, has the following format:
{
"Interactions": [
{
"ID": "ispring://presentations/F7385CB7-DFDC-4D05-90FA-B927DB3D170D/quizzes/",
"Type": "2",
"TimestampUtc": "8/27/2020 12:09:54 PM",
"Timestamp": "8/27/2020 12:09:54 PM",
"Weighting": "",
"Result": "1",
"Latency": "1000",
"Description": "What is the purpose of Summarizing next steps?\n\nSelect the correct box or boxes",
"LearnerResponse": "0_correct_answer[,]1_Rectangle_2",
"ScormActivityId": "12392705",
"InteractionIndex": "3",
"AULMRID": "38093846"
},
]
}
File 2:
{
"Interactions": [
{
"ID": "ispring://presentations/CAA34147-7B48-40C6-84FD-5CE8077DB2BF/quizzes/",
"Type": "2",
"TimestampUtc": "12/8/2020 6:19:12 PM",
"Timestamp": "12/8/2020 6:19:12 PM",
"Weighting": "",
"Result": "1",
"Latency": "1300",
"Description": "'Can't do' language tends to relay this impression...\n\nSelect one.",
"LearnerResponse": "4_All_of_the_above",
"ScormActivityId": "13334358",
"InteractionIndex": "3",
"AULMRID": "40715598"
},
]
}
And this is my expected result in a third file:
{
"Interactions": [
{
"ID": "ispring://presentations/F7385CB7-DFDC-4D05-90FA-B927DB3D170D/quizzes/",
"Type": "2",
"TimestampUtc": "8/27/2020 12:09:54 PM",
"Timestamp": "8/27/2020 12:09:54 PM",
"Weighting": "",
"Result": "1",
"Latency": "1000",
"Description": "What is the purpose of Summarizing next steps?\n\nSelect the correct box or boxes",
"LearnerResponse": "0_correct_answer[,]1_Rectangle_2",
"ScormActivityId": "12392705",
"InteractionIndex": "3",
"AULMRID": "38093846"
},
{
"ID": "ispring://presentations/CAA34147-7B48-40C6-84FD-5CE8077DB2BF/quizzes/",
"Type": "2",
"TimestampUtc": "12/8/2020 6:19:12 PM",
"Timestamp": "12/8/2020 6:19:12 PM",
"Weighting": "",
"Result": "1",
"Latency": "1300",
"Description": "'Can't do' language tends to relay this impression...\n\nSelect one.",
"LearnerResponse": "4_All_of_the_above",
"ScormActivityId": "13334358",
"InteractionIndex": "3",
"AULMRID": "40715598"
},
]
}
Any ideas on how to unify them ?
Thank you!
RG
Try something like the following:
jq -n '{ Interactions: [ inputs.Interactions ] | add }' file1.json file2.json
This assumes that you have made both input files valid JSON by stripping that trailing comma at the end of each object in the Interactions array.
For input file1.json:
{
"Interactions": [
{
"ID": "file1",
"Type": "1",
"Timestamp": "8/27/2020 11:11:11 PM"
}
]
}
and input file2.json:
{
"Interactions": [
{
"ID": "file2",
"Type": "2",
"Timestamp": "8/27/2020 22:22:22 PM"
}
]
}
this results in:
{
"Interactions": [
{
"ID": "file1",
"Type": "1",
"Timestamp": "8/27/2020 11:11:11 PM"
},
{
"ID": "file2",
"Type": "2",
"Timestamp": "8/27/2020 22:22:22 PM"
}
]
}
I'm trying to extract a specific value from JSON file.
the key value is: "info": "this is an example" (The key is unique)
I want to extract only the value: "this is an example"
My code:
cat 9.json | jq '.info'
result:
null
JSON file example:
{
"Event": {
"id": "13",
"orgc_id": "1",
"org_id": "1",
"date": "2019-01-09",
"threat_level_id": "3",
"info": "test9",
"published": false,
"uuid": "5c35d180",
"attribute_count": "2",
"analysis": "0",
"timestamp": "1547044733",
"distribution": "1",
"proposal_email_lock": false,
"locked": false,
"publish_timestamp": "1547034089",
"sharing_group_id": "0",
"disable_correlation": false,
"extends_uuid": "",
"event_creator_email": "o#cyhgfnt.com",
"Org": {
"id": "1",
"name": "Cygfdgfdnt",
"uuid": "5b9f938d-e3a0-4ecb-83b3-0bdeac1b41bc"
},
"Orgc": {
"id": "1",
"name": "Cyhgfgft",
"uuid": "5b9f938d-e3a0-4ecb-83b3-0bdeac1b41bc"
},
"Attribute": [{
"id": "292630",
"type": "domain",
"category": "Network activity",
"to_ids": true,
"uuid": "5c35dd94-cccc-4086-b386-682823717aa5",
"event_id": "1357",
"distribution": "5",
"timestamp": "1547034584",
"comment": "This is a comment",
"sharing_group_id": "0",
"deleted": false,
"disable_correlation": false,
"object_id": "0",
"object_relation": null,
"value": "dodskj.com",
"Galaxy": [],
"ShadowAttribute": [],
"Tag": [{
"id": "223",
"name": "kill-chain:Exploitation",
"colour": "#a80079",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}]
}, {
"id": "292631",
"type": "ip-dst",
"category": "Network activity",
"to_ids": true,
"uuid": "5c35dd94-fe90-4ef6-b3a9-682823717aa5",
"event_id": "1357",
"distribution": "5",
"timestamp": "1547044733",
"comment": "comment example",
"sharing_group_id": "0",
"deleted": false,
"disable_correlation": false,
"object_id": "0",
"object_relation": null,
"value": "8.8.6.6",
"Galaxy": [],
"ShadowAttribute": [],
"Tag": [{
"id": "247",
"name": "maec-malware-capabilities:maec-malware-capability=\"anti-removal\"",
"colour": "#3f0004",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}, {
"id": "465",
"name": "osint:lifetime=\"perpetual\"",
"colour": "#006ebe",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}]
}],
"ShadowAttribute": [],
"RelatedEvent": [],
"Galaxy": [{
"id": "3",
"uuid": "698774c7-8022-42c4-917f-8d6e4f06ada3",
"name": "Threat Actor",
"type": "threat-actor",
"description": "Threat actors are characteristics of malicious actors (or adversaries) representing a cyber attack threat including presumed intent and historically observed behaviour.",
"version": "3",
"icon": "user-secret",
"namespace": "misp",
"GalaxyCluster": [{
"id": "6397",
"collection_uuid": "7cdff317-a673-4474-84ec-4f1754947823",
"type": "threat-actor",
"value": "Sofacy",
"tag_name": "misp-galaxy:threat-actor=\"Sofacy\"",
"description": "The Sofacy Group (also known as APT28, Pawn Storm, Fancy Bear and Sednit) is a cyber espionage group believed to have ties to the Russian government. Likely operating since 2007, the group is known to target government, military, and security organizations. It has been characterized as an advanced persistent threat.",
"galaxy_id": "3",
"source": "MISP Project",
"authors": ["Alexandre Dulaunoy", "Florian Roth", "Thomas Schreck", "Timo Steffens", "Various"],
"version": "82",
"uuid": "5b4ee3ea-eee3-4c8e-8323-85ae32658754",
"tag_id": "608",
"meta": {
"cfr-suspected-state-sponsor": ["Russian Federation"],
"cfr-suspected-victims": ["Georgia", "France", "Jordan", "United States", "Hungary", "World Anti-Doping Agency", "Armenia", "Tajikistan", "Japan", "NATO", "Ukraine", "Belgium", "Pakistan", "Asia Pacific Economic Cooperation", "International Association of Athletics Federations", "Turkey", "Mongolia", "OSCE", "United Kingdom", "Germany", "Poland", "European Commission", "Afghanistan", "Kazakhstan", "China"],
"cfr-target-category": ["Government", "Military"],
"cfr-type-of-incident": ["Espionage"],
"country": ["RU"],
"refs": ["https:\/\/en.wikipedia.org\/wiki\/Sofacy_Group", "https:\/\/aptnotes.malwareconfig.com\/web\/viewer.html?file=..\/APTnotes\/2014\/apt28.pdf", "http:\/\/www.trendmicro.com\/cloud-content\/us\/pdfs\/security-intelligence\/white-papers\/wp-operation-pawn-storm.pdf", "https:\/\/www2.fireeye.com\/rs\/848-DID-242\/images\/wp-mandiant-matryoshka-mining.pdf", "https:\/\/www.crowdstrike.com\/blog\/bears-midst-intrusion-democratic-national-committee\/", "http:\/\/researchcenter.paloaltonetworks.com\/2016\/06\/unit42-new-sofacy-attacks-against-us-government-agency\/", "https:\/\/www.cfr.org\/interactive\/cyber-operations\/apt-28", "https:\/\/blogs.microsoft.com\/on-the-issues\/2018\/08\/20\/we-are-taking-new-steps-against-broadening-threats-to-democracy\/", "https:\/\/www.bleepingcomputer.com\/news\/security\/microsoft-disrupts-apt28-hacking-campaign-aimed-at-us-midterm-elections\/", "https:\/\/www.bleepingcomputer.com\/news\/security\/apt28-uses-lojax-first-uefi-rootkit-seen-in-the-wild\/"],
"synonyms": ["APT 28", "APT28", "Pawn Storm", "PawnStorm", "Fancy Bear", "Sednit", "TsarTeam", "Tsar Team", "TG-4127", "Group-4127", "STRONTIUM", "TAG_0700", "Swallowtail", "IRON TWILIGHT", "Group 74"]
}
}]
}],
"Object": [],
"Tag": [{
"id": "608",
"name": "misp-galaxy:threat-actor=\"Sofacy\"",
"colour": "#12e000",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}, {
"id": "118",
"name": "gdpr:special-categories=\"health\"",
"colour": "#3ce600",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}]
}
}
I suppose you are trying to get the .info field inside .Event which should have been written as below. Use -r for without quotes
jq '.Event.info'
I'm want to make my json to csv so that i can upload it on google sheets and make it as json api. Whenever i have change data i will just change it on google sheets. But I'm having problems on converting my json file to csv because it changes the variables whenever i convert it. I'm using https://toolslick.com/csv-to-json-converter to convert my json file to csv.
What is the best way to convert json nested to csv ?
JSON
{
"options": [
{
"id": "1",
"value": "Jumbo",
"shortcut": "J",
"textColor": "#FFFFFF",
"backgroundColor": "#00000"
},
{
"id": "2",
"value": "Hot",
"shortcut": "D",
"textColor": "#FFFFFF",
"backgroundColor": "#FFFFFF"
}
],
"categories": [
{
"id": "1",
"order": 1,
"name": "First Category",
"active": true
},
{
"id": "2",
"order": 2,
"name": "Second Category",
"shortcut": "MT",
"active": true
}
],
"products": [
{
"id": "03c6787c-fc2a-4aa8-93a3-5e0f0f98cfb2",
"categoryId": "1",
"name": "First Product",
"shortcut": "First",
"options": [
{
"optionId": "1",
"price": 23
},
{
"optionId": "2",
"price": 45
}
],
"active": true
},
{
"id": "e8669cea-4c9c-431c-84ba-0b014f0f9bc2",
"categoryId": "2",
"name": "Second Product",
"shortcut": "Second",
"options": [
{
"optionId": "1",
"price": 11
},
{
"optionId": "2",
"price": 20
}
],
"active": true
}
],
"discounts": [
{
"id": "1",
"name": "S",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "2",
"name": "P",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "3",
"name": "G",
"type": 2,
"amount": 5,
"active": true
}
]
}
Using python, this can be easily done or almost done. Maybe this code will help you in some way to understand that.
import json,csv
data = []
with open('your_json_file_here.json') as file:
for line in file:
data.append(json.loads(line))
length = len(data)
with open('create_new_file.csv','w') as f:
writer = csv.writer(f)
writers = csv.DictWriter(f, fieldnames=['header1','header2'])
writers.writeheader()
for iter in range(length):
writer.writerow((data[iter]['specific_col_name1'],data[iter]['specific_col_name2']))
f.close()
I wanted to sort the file which have json data on basis of a key(fare) in a shell script (.sh) file.Is there a way to acheive the same ?
{ "route":[
{"match":"true","column":"10","fare":"120.0","source":"false","length":"1","name":"41","row":"4","width":"1","zIndex":"0"},
{"match":"true","column":"9","fare":"110.0","source":"false","length":"1","name":"37","row":"3","width":"1","zIndex":"0"},
{"match":"true","column":"8","fare":"500.0","source":"false","length":"1","name":"33","row":"2","width":"1","zIndex":"0"},
{"match":"true","column":"7","fare":"510.0","source":"false","length":"1","name":"29","row":"1","width":"1","zIndex":"0"},
{"match":"true","column":"6","fare":"50.0","source":"false","length":"1","name":"29","row":"0","width":"1","zIndex":"0"}
]
};
jq '.route = (.route | sort_by(.fare))' file.json
{
"route": [
{
"zIndex": "0",
"match": "true",
"column": "9",
"fare": "110.0",
"source": "false",
"length": "1",
"name": "37",
"row": "3",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "10",
"fare": "120.0",
"source": "false",
"length": "1",
"name": "41",
"row": "4",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "6",
"fare": "50.0",
"source": "false",
"length": "1",
"name": "29",
"row": "0",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "8",
"fare": "500.0",
"source": "false",
"length": "1",
"name": "33",
"row": "2",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "7",
"fare": "510.0",
"source": "false",
"length": "1",
"name": "29",
"row": "1",
"width": "1"
}
]
}
Note that the sorting is lexical ("50" is in the middle). This is because the values are strings not numbers. To get numeric sorting, we need to cast as a number
jq '.route = (.route | sort_by(.fare | tonumber))' file.json
{
"route": [
{
"zIndex": "0",
"match": "true",
"column": "6",
"fare": "50.0",
"source": "false",
"length": "1",
"name": "29",
"row": "0",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "9",
"fare": "110.0",
"source": "false",
"length": "1",
"name": "37",
"row": "3",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "10",
"fare": "120.0",
"source": "false",
"length": "1",
"name": "41",
"row": "4",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "8",
"fare": "500.0",
"source": "false",
"length": "1",
"name": "33",
"row": "2",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "7",
"fare": "510.0",
"source": "false",
"length": "1",
"name": "29",
"row": "1",
"width": "1"
}
]
}
I have a json data as follows
[{
"_id": "52cbafc5afcec877d925d116",
"id": "52",
"url": "172.34.56.7"
}, {
"_id": "52c255e5cf0060b8cce84b4a",
"id": "13",
"url": "172.18.2.114"
}, {
"_id": "52cbd018afcec877d925d131",
"id": "49",
"url": "172.16.2.120"
}, {
"_id": "52d60c9eafcec877d925d17a",
"id": "100",
"url": "156.45.34.56"
}]
I want to print the data as below.(last data comes first)
[{
"_id": "52d60c9eafcec877d925d17a",
"id": "100",
"url": "156.45.34.56"
}, {
"_id": "52cbd018afcec877d925d131",
"id": "49",
"url": "172.16.2.120"
}, {
"_id": "52c255e5cf0060b8cce84b4a",
"id": "13",
"url": "172.18.2.114"
},
[{
"_id": "52cbafc5afcec877d925d116",
"id": "52",
"url": "172.34.56.7"
}]
How it is possible
Implement a Comparator and then sort your Json array with it.
This can be done using
data=data.reverse()