Related
I have an api which I would like to use directly within Vegalite, in order for the chart to update automatically.
https://stats.oecd.org/SDMX-JSON/data/REVGBR/TOTALTAX.NES/all?startTime=1965&endTime=2021&dimensionAtObservation=allDimensions
However the api doesn't seem to be in a pleasant format.
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.2.json",
"data": {
"url": "https://stats.oecd.org/SDMX-JSON/data/REV/NES.TOTALTAX.TAXGDP.OAVG?contentType=json",
"format": {
"type": "json",
"property": "dataSets[0].series['0:0:0:0'].observations"
},
"transform": [
{
"type": "formula",
"as": "year",
"expr": "parseInt(datum.key)"
},
{
"type": "formula",
"as": "Tax_revenue",
"expr": "datum.value[0]"
}
]
},
"mark": "line",
"encoding": {
"x": {"field": "year", "type": "temporal"},
"y": {"field": "Tax_revenue", "type": "quantitative"}
}
}
When looking at the data viewer, all that is shown is the values, without the years.
The values however are not clean, e.g. [24.855,null]
Is it even possible to get this api to work within vegalite?
It is possible but you need to make assumptions. As you know the query to the API starts with 1965, you can use something like this assuming the years are continuous.
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.2.json",
"data": {
"name": "api",
"url": "https://stats.oecd.org/SDMX-JSON/data/REV/NES.TOTALTAX.TAXGDP.OAVG?contentType=json",
"format": {
"type": "json",
"property": "dataSets[0].series['0:0:0:0'].observations"
}
},
"transform": [
{"fold": ["0", "1", "2", "3","4","5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "50", "51", "52", "53", "54", "55"]},
{"calculate": "parseInt( datum.key) + 1965", "as": "year"},
{"calculate": "datum.value[0]", "as": "Tax_revenue"}
],
"mark": "line",
"encoding": {
"x": {"field": "year", "type": "nominal"},
"y": {"field": "Tax_revenue", "type": "quantitative"}
}
}
I'm trying to convert JSON to CSV using the ConvertRecord processor but the only error I'm getting back is Could not parse incoming data. As this is not very descriptive, I'm at a loss as to how to diagnose the issue.
I know that my avro schema is valid because A) NiFi doesn't throw an error regarding the schema when I insert it into the Schema Registry and B) I tested my schema on here and it didn't give me an issue.
I also know that my JSON is valid because I can load it in Python using json.loads() and it doesn't give me any problems.
I'm just not quite sure where I've gone wrong, nor how to fix it.
JSON
{
"DOC": {
"DOCID": "1234",
"Subjects": {
"Subject_xref": ["2233"]
},
"TXT": {
"COUNTRY": ["United States"],
"ESTATE": ["Mount Vernon"],
"PERSON": ["George Washington"]
},
"RAW_TXT": "George Washington lived in his family home, Mount Vernon, located in the United States.",
"RELINFO": [
{"ID" : "REL-1234-100",
"RELTYPE" : "PER-PROP",
"PERID" : "PER-1234-009",
"PROPID" : "PROP-1234-001",
"SENTID" : "1234-SENT-001",
"PROP_NORM" : "Mount Vernon",
"PROP_MENTION" : "Mount Vernon",
"PER_NORM" : "George Washington",
"PER_MENTION" : "George Washington"}
],
"ENTINFO": [
{"ID": "PER-1234-009", "TYPE": "PERSON", "NORM": "George Washington", "REFID": "PER-1234-009", "MENTION": "George Washington"},
{"ID": "CTRY-1234-003", "TYPE": "COUNTRY", "NORM": "United States", "REFID": "CTRY-1234-003", "MENTION": "United States."},
{"ID": "PROP-1234-001", "TYPE": "ESTATE", "NORM": "Mount Vernon", "REFID": "PROP-1234-001", "MENTION": "Mount Vernon"}
]
}
}
Avro
{
"type": "record",
"namespace": "name.space",
"name": "nlp_output",
"fields": [
{"name": "DOC", "type": {
"name": "DOCDocument", "type": "record", "namespace": "doc.name.space", "fields": [
{"name": "DOCID", "type": ["long","null"], "default": null},
{"name": "Subjects", "type": {
"name": "Subjects", "type": "record", "namespace": "subjects.name.space", "fields": [
{"name": "SubjectIdentificationID", "aliases": ["Subject_xref"], "type": ["long","null"], "default": null}
]
}},
{"name": "TXT", "type": {
"name": "TXT", "type": "record", "namespace": "text.name.space", "fields": [
{"name": "COUNTRY", "type": {"type": "array", "items": ["string", "null"]}, "default": null, "doc": ""},
{"name": "ESTATE", "type": {"type": "array", "items": ["string", "null"]}, "default": null, "doc": ""},
{"name": "PERSON", "type": {"type": "array", "items": ["string", "null"]}, "default": null, "doc": ""}
]
}},
{"name": "RAW_TXT", "type": ["string","null"], "default": null},
{"name": "RELINFO", "type": {
"name": "RelatedEntities", "type": "record", "namespace": "relent.name.space", "fields": [
{"name": "ID", "type": ["string", "null"], "default": null},
{"name": "RELTYPE", "type": ["string", "null"], "default": null},
{"name": "PERID", "type": ["string", "null"], "default": null},
{"name": "PROPID", "type": ["string", "null"], "default": null},
{"name": "SENTID", "type": ["string", "null"], "default": null},
{"name": "PROP_NORM", "type": ["string", "null"], "default": null},
{"name": "PROP_MENTION", "type": ["string", "null"], "default": null},
{"name": "PER_NORM", "type": ["string", "null"], "default": null},
{"name": "PER_MENTION", "type": ["string", "null"], "default": null}
]
}},
{"name": "ENTINFO", "doc": "Sentences stripped of tags for ease of reading", "type": {
"name": "Entities", "type": "record", "namespace": "entities.name.space", "fields": [
{"name": "ID", "type": ["string", "null"], "default": null},
{"name": "TYPE", "type": ["string", "null"], "default": null},
{"name": "NORM", "type": ["string", "null"], "default": null},
{"name": "REFID", "type": ["string", "null"], "default": null},
{"name": "MENTION", "type": ["string", "null"], "default": null}
]
}}
]
}}
]
}
Your schema doesn't match your JSON. You have SubjectIdentificationID defined as long or null but in the JSON Subject_xref is an array.
{
"type": "record",
"namespace": "name.space",
"name": "nlp_output",
"fields": [
{"name": "DOC", "type": {
"name": "DOCDocument", "type": "record", "namespace": "doc.name.space", "fields": [
{"name": "DOCID", "type": ["long","null"], "default": null},
{"name": "Subjects", "type": {
"name": "Subjects", "type": "record", "namespace": "subjects.name.space", "fields": [
{"name": "SubjectIdentificationID", "aliases": ["Subject_xref"], "type": {"type": "array", "items": ["long", "null"]}, "default": null}
]
}},
{"name": "TXT", "type": {
"name": "TXT", "type": "record", "namespace": "text.name.space", "fields": [
{"name": "COUNTRY", "type": {"type": "array", "items": ["string", "null"]}, "default": null, "doc": ""},
{"name": "ESTATE", "type": {"type": "array", "items": ["string", "null"]}, "default": null, "doc": ""},
{"name": "PERSON", "type": {"type": "array", "items": ["string", "null"]}, "default": null, "doc": ""}
]
}},
{"name": "RAW_TXT", "type": ["string","null"], "default": null},
{"name": "RELINFO", "type": {
"name": "RelatedEntities", "type": "record", "namespace": "relent.name.space", "fields": [
{"name": "ID", "type": ["string", "null"], "default": null},
{"name": "RELTYPE", "type": ["string", "null"], "default": null},
{"name": "PERID", "type": ["string", "null"], "default": null},
{"name": "PROPID", "type": ["string", "null"], "default": null},
{"name": "SENTID", "type": ["string", "null"], "default": null},
{"name": "PROP_NORM", "type": ["string", "null"], "default": null},
{"name": "PROP_MENTION", "type": ["string", "null"], "default": null},
{"name": "PER_NORM", "type": ["string", "null"], "default": null},
{"name": "PER_MENTION", "type": ["string", "null"], "default": null}
]
}},
{"name": "ENTINFO", "doc": "Sentences stripped of tags for ease of reading", "type": {
"name": "Entities", "type": "record", "namespace": "entities.name.space", "fields": [
{"name": "ID", "type": ["string", "null"], "default": null},
{"name": "TYPE", "type": ["string", "null"], "default": null},
{"name": "NORM", "type": ["string", "null"], "default": null},
{"name": "REFID", "type": ["string", "null"], "default": null},
{"name": "MENTION", "type": ["string", "null"], "default": null}
]
}}
]
}}
]
}
I'm trying to extract a specific value from JSON file.
the key value is: "info": "this is an example" (The key is unique)
I want to extract only the value: "this is an example"
My code:
cat 9.json | jq '.info'
result:
null
JSON file example:
{
"Event": {
"id": "13",
"orgc_id": "1",
"org_id": "1",
"date": "2019-01-09",
"threat_level_id": "3",
"info": "test9",
"published": false,
"uuid": "5c35d180",
"attribute_count": "2",
"analysis": "0",
"timestamp": "1547044733",
"distribution": "1",
"proposal_email_lock": false,
"locked": false,
"publish_timestamp": "1547034089",
"sharing_group_id": "0",
"disable_correlation": false,
"extends_uuid": "",
"event_creator_email": "o#cyhgfnt.com",
"Org": {
"id": "1",
"name": "Cygfdgfdnt",
"uuid": "5b9f938d-e3a0-4ecb-83b3-0bdeac1b41bc"
},
"Orgc": {
"id": "1",
"name": "Cyhgfgft",
"uuid": "5b9f938d-e3a0-4ecb-83b3-0bdeac1b41bc"
},
"Attribute": [{
"id": "292630",
"type": "domain",
"category": "Network activity",
"to_ids": true,
"uuid": "5c35dd94-cccc-4086-b386-682823717aa5",
"event_id": "1357",
"distribution": "5",
"timestamp": "1547034584",
"comment": "This is a comment",
"sharing_group_id": "0",
"deleted": false,
"disable_correlation": false,
"object_id": "0",
"object_relation": null,
"value": "dodskj.com",
"Galaxy": [],
"ShadowAttribute": [],
"Tag": [{
"id": "223",
"name": "kill-chain:Exploitation",
"colour": "#a80079",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}]
}, {
"id": "292631",
"type": "ip-dst",
"category": "Network activity",
"to_ids": true,
"uuid": "5c35dd94-fe90-4ef6-b3a9-682823717aa5",
"event_id": "1357",
"distribution": "5",
"timestamp": "1547044733",
"comment": "comment example",
"sharing_group_id": "0",
"deleted": false,
"disable_correlation": false,
"object_id": "0",
"object_relation": null,
"value": "8.8.6.6",
"Galaxy": [],
"ShadowAttribute": [],
"Tag": [{
"id": "247",
"name": "maec-malware-capabilities:maec-malware-capability=\"anti-removal\"",
"colour": "#3f0004",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}, {
"id": "465",
"name": "osint:lifetime=\"perpetual\"",
"colour": "#006ebe",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}]
}],
"ShadowAttribute": [],
"RelatedEvent": [],
"Galaxy": [{
"id": "3",
"uuid": "698774c7-8022-42c4-917f-8d6e4f06ada3",
"name": "Threat Actor",
"type": "threat-actor",
"description": "Threat actors are characteristics of malicious actors (or adversaries) representing a cyber attack threat including presumed intent and historically observed behaviour.",
"version": "3",
"icon": "user-secret",
"namespace": "misp",
"GalaxyCluster": [{
"id": "6397",
"collection_uuid": "7cdff317-a673-4474-84ec-4f1754947823",
"type": "threat-actor",
"value": "Sofacy",
"tag_name": "misp-galaxy:threat-actor=\"Sofacy\"",
"description": "The Sofacy Group (also known as APT28, Pawn Storm, Fancy Bear and Sednit) is a cyber espionage group believed to have ties to the Russian government. Likely operating since 2007, the group is known to target government, military, and security organizations. It has been characterized as an advanced persistent threat.",
"galaxy_id": "3",
"source": "MISP Project",
"authors": ["Alexandre Dulaunoy", "Florian Roth", "Thomas Schreck", "Timo Steffens", "Various"],
"version": "82",
"uuid": "5b4ee3ea-eee3-4c8e-8323-85ae32658754",
"tag_id": "608",
"meta": {
"cfr-suspected-state-sponsor": ["Russian Federation"],
"cfr-suspected-victims": ["Georgia", "France", "Jordan", "United States", "Hungary", "World Anti-Doping Agency", "Armenia", "Tajikistan", "Japan", "NATO", "Ukraine", "Belgium", "Pakistan", "Asia Pacific Economic Cooperation", "International Association of Athletics Federations", "Turkey", "Mongolia", "OSCE", "United Kingdom", "Germany", "Poland", "European Commission", "Afghanistan", "Kazakhstan", "China"],
"cfr-target-category": ["Government", "Military"],
"cfr-type-of-incident": ["Espionage"],
"country": ["RU"],
"refs": ["https:\/\/en.wikipedia.org\/wiki\/Sofacy_Group", "https:\/\/aptnotes.malwareconfig.com\/web\/viewer.html?file=..\/APTnotes\/2014\/apt28.pdf", "http:\/\/www.trendmicro.com\/cloud-content\/us\/pdfs\/security-intelligence\/white-papers\/wp-operation-pawn-storm.pdf", "https:\/\/www2.fireeye.com\/rs\/848-DID-242\/images\/wp-mandiant-matryoshka-mining.pdf", "https:\/\/www.crowdstrike.com\/blog\/bears-midst-intrusion-democratic-national-committee\/", "http:\/\/researchcenter.paloaltonetworks.com\/2016\/06\/unit42-new-sofacy-attacks-against-us-government-agency\/", "https:\/\/www.cfr.org\/interactive\/cyber-operations\/apt-28", "https:\/\/blogs.microsoft.com\/on-the-issues\/2018\/08\/20\/we-are-taking-new-steps-against-broadening-threats-to-democracy\/", "https:\/\/www.bleepingcomputer.com\/news\/security\/microsoft-disrupts-apt28-hacking-campaign-aimed-at-us-midterm-elections\/", "https:\/\/www.bleepingcomputer.com\/news\/security\/apt28-uses-lojax-first-uefi-rootkit-seen-in-the-wild\/"],
"synonyms": ["APT 28", "APT28", "Pawn Storm", "PawnStorm", "Fancy Bear", "Sednit", "TsarTeam", "Tsar Team", "TG-4127", "Group-4127", "STRONTIUM", "TAG_0700", "Swallowtail", "IRON TWILIGHT", "Group 74"]
}
}]
}],
"Object": [],
"Tag": [{
"id": "608",
"name": "misp-galaxy:threat-actor=\"Sofacy\"",
"colour": "#12e000",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}, {
"id": "118",
"name": "gdpr:special-categories=\"health\"",
"colour": "#3ce600",
"exportable": true,
"user_id": "0",
"hide_tag": false,
"numerical_value": null
}]
}
}
I suppose you are trying to get the .info field inside .Event which should have been written as below. Use -r for without quotes
jq '.Event.info'
I wanted to sort the file which have json data on basis of a key(fare) in a shell script (.sh) file.Is there a way to acheive the same ?
{ "route":[
{"match":"true","column":"10","fare":"120.0","source":"false","length":"1","name":"41","row":"4","width":"1","zIndex":"0"},
{"match":"true","column":"9","fare":"110.0","source":"false","length":"1","name":"37","row":"3","width":"1","zIndex":"0"},
{"match":"true","column":"8","fare":"500.0","source":"false","length":"1","name":"33","row":"2","width":"1","zIndex":"0"},
{"match":"true","column":"7","fare":"510.0","source":"false","length":"1","name":"29","row":"1","width":"1","zIndex":"0"},
{"match":"true","column":"6","fare":"50.0","source":"false","length":"1","name":"29","row":"0","width":"1","zIndex":"0"}
]
};
jq '.route = (.route | sort_by(.fare))' file.json
{
"route": [
{
"zIndex": "0",
"match": "true",
"column": "9",
"fare": "110.0",
"source": "false",
"length": "1",
"name": "37",
"row": "3",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "10",
"fare": "120.0",
"source": "false",
"length": "1",
"name": "41",
"row": "4",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "6",
"fare": "50.0",
"source": "false",
"length": "1",
"name": "29",
"row": "0",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "8",
"fare": "500.0",
"source": "false",
"length": "1",
"name": "33",
"row": "2",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "7",
"fare": "510.0",
"source": "false",
"length": "1",
"name": "29",
"row": "1",
"width": "1"
}
]
}
Note that the sorting is lexical ("50" is in the middle). This is because the values are strings not numbers. To get numeric sorting, we need to cast as a number
jq '.route = (.route | sort_by(.fare | tonumber))' file.json
{
"route": [
{
"zIndex": "0",
"match": "true",
"column": "6",
"fare": "50.0",
"source": "false",
"length": "1",
"name": "29",
"row": "0",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "9",
"fare": "110.0",
"source": "false",
"length": "1",
"name": "37",
"row": "3",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "10",
"fare": "120.0",
"source": "false",
"length": "1",
"name": "41",
"row": "4",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "8",
"fare": "500.0",
"source": "false",
"length": "1",
"name": "33",
"row": "2",
"width": "1"
},
{
"zIndex": "0",
"match": "true",
"column": "7",
"fare": "510.0",
"source": "false",
"length": "1",
"name": "29",
"row": "1",
"width": "1"
}
]
}
I have a JSON file something like below:
{
"name": "flare",
"children": [
{
"name": "analytics",
"children": [
{
"name": "cluster",
"children": [
{"name": "AgglomerativeCluster", "size": 3938},
{"name": "CommunityStructure", "size": 3812},
{"name": "HierarchicalCluster", "size": 6714},
{"name": "MergeEdge", "size": 743}
]
}, ....
As you can see, there is a "size" attribute at the end nodes.
I want to add the "size" attribute to other nodes as well (i.e. to "analytics", and "cluster") so what I did was:
{
"name": "flare",
"children": [
{
{"name": "analytics", "another_size":345},
"children": [
{
{"name": "cluster", "another_size":345},
"children": [
{"name": "AgglomerativeCluster", "size": 3938},
{"name": "CommunityStructure", "size": 3812},
{"name": "HierarchicalCluster", "size": 6714},
{"name": "MergeEdge", "size": 743}
]
}, ....
But it doesn't compile this way.
How do I go about achieving this? I want to colour my nodes differently for those other nodes under "another_size".
Any help appreciated!!
Try it like this:
{
"name": "flare",
"children": [
{
{"name": "analytics",
"another_size":345,
"children": [
{
{"name": "cluster",
"another_size":345,
"children": [
{"name": "AgglomerativeCluster", "size": 3938},
{"name": "CommunityStructure", "size": 3812},
{"name": "HierarchicalCluster", "size": 6714},
{"name": "MergeEdge", "size": 743}
]
}, ....
FULL EXAMPLE
{
"name": "flare",
"children": [
{
"name": "analytics",
"another_size": 345,
"children": [
{
"name": "cluster",
"another_size": 345,
"children": [
{
"name": "AgglomerativeCluster",
"size": 3938
},
{
"name": "CommunityStructure",
"size": 3812
},
{
"name": "HierarchicalCluster",
"size": 6714
},
{
"name": "MergeEdge",
"size": 743
}
]
},
{
"name": "cluster",
"another_size": 345,
"children": [
{
"name": "AgglomerativeCluster",
"size": 3938
},
{
"name": "CommunityStructure",
"size": 3812
},
{
"name": "HierarchicalCluster",
"size": 6714
},
{
"name": "MergeEdge",
"size": 743
}
]
}
]
}
]
}