Add array into json for each row of a dataframe - json

I would like to populate the following json array for each row of the data frame
Json Array
"techniques": [
{
"techniqueID": "",
"tactic": "",
"color": "",
"comment": "",
"enabled": true,
"metadata": [],
"links": [],
"showSubtechniques": false
}
]
Dataframe
techniqueID Value color tactic
0 T1078 13 #74c476 Defense-Evasion
1 T1078 13 #74c476 Initial-Access
2 T1078 13 #74c476 Persistence
3 T1078 13 #74c476 Privilege-Escalation
4 T1110 5 #74c476 Credential-Access
5 T1070 3 #a1d99b Defense-Evasion
6 T1059 3 #a1d99b Execution
7 T1114 3 #a1d99b Collection
8 T1098 3 #a1d99b Persistenc
I have tried to convert the json into a data frame and combine the two but this does not then create a array for each row.


It seems that you are looking for to_dict('records') (possibly need to add missing columns):
res_dict = {
'techniques': df.to_dict('records')
}

Related

Array of objects to grid/table in Angular

I have data as an array of objects like this:
[{
"x":1,
"y":10,
"val": 13
},
{
"x":1,
"y":20,
"val": 15
},
{
"x":2,
"y":10,
"val": 12
},
{
"x":2,
"y":20,
"val": 16
}
]
I want this data to be displayed in an HTML table like this:
1 2
10 | 13 12
20 | 15 16
How to proceed?

I would parse the data so you have a 2d array. Something like this:
[
[13,12],
[15,16],
]
and then use two nested ng-for loops to render the content:
<div *ngFor="let row of matrix">
<span *ngFor="let item of row">{{item}}</span>
</div>
and place the headers appart.

Extracting multiple values having same path in json using json map in sas

Dose anyone can help me get multiple values in a json having same path using a json map. Any help is appreciated. Thank you.
JSON
{
"totalCount": 2,
"facets": {},
"content": [
[
{
"name": "customer_ID",
"value": "1"
},
{
"name": "customer_name",
"value": "John"
}
]
]
}
JSON MAP
{
"DATASETS": [
{
"DSNAME": "customers",
"TABLEPATH": "/root/content",
"VARIABLES": [
{
"NAME": "name",
"TYPE": "CHARACTER",
"PATH": "/root/content/name"/*output as customer_ID*/
},
{
"NAME": "name",
"TYPE": "CHARACTER",
"PATH": "/root/content/name"/*output as customer_name*/
},
{
"NAME": "value",
"TYPE": "CHARACTER",
"PATH": "/root/content/value"/*output as 1*/
},
{
"NAME": "value",
"TYPE": "CHARACTER",
"PATH": "/root/content/value"/*output as John*/
}
]
}
]
}
When i use the above json map I get the output for name as only "customer_name", but i need both "customer_ID" and "customer_name" in the output.
Similarly i need both values of "value"

JSON is a hierarchy of name-value pairs. The JSON engine in SAS will take the "name" and assign as a variable name, and then populate with the value. In your JSON, there are two sets of name-values, one being the name of an intended variable, and another being its value. This is a common output scheme we find in GraphQL responses -- and these require a little manipulation to turn into 2-D data sets.
For your example, you could use PROC TRANSPOSE:
libname j json fileref=test;
proc transpose
data=j.content
out=want;
id name;
var value;
run;
Output:
customer_ customer_
Obs _NAME_ ID name
1 value 1 John
You can also do more manual seek/assignment by using DATA step to process what you see in the ALLDATA member in the JSON libname. In your example, SAS sees that as:
Obs P P1 P2 V Value
1 1 totalCount 1 2
2 1 facets 0
3 1 content 0
4 1 content 0
5 2 content name 1 customer_ID
6 2 content value 1 1
7 1 content 0
8 2 content name 1 customer_name
9 2 content value 1 John
Processing the ALLDATA member is not as friendly as using the relational data that the JSON engine can create, but I find with GraphQL responses that's what you need to do to get more control over the name, length, and type/format for output variables.

Parse JSON to CSV using jq but split sub list in multiple records

I am parsing a JSON result which i get from the Azure RateAPI and want to convert it into a standard CSV file.
The following line is what i am using to convert it into CSV and it works but as one of the attributes is a list, it does not provide me the result i am seeking. For every item in the "sub list", i would need to create another record in my csv file.
cat myfile.json | jq -r '.Meters[] | [ .EffectiveDate, .IncludedQuantity, .MeterCategory, .MeterId, .MeterName, .MeterRates[], .MeterRegion, .MeterStatus, .MeterSubCategory, .MeterTags[], .Units] | #csv'
Here are 3 records I am trying to parse. I am having trouble with record 2 because MeterRates is actually the list where i need both, the attribute and the value. I would need record 2, once parsed, to correspond to 3 records in the CSV file where each record contains one item of the list in the MeterRates. An example of expected result is at the end
"OfferTerms": [],
"Meters": [
{
"EffectiveDate": "2019-03-01T00:00:00Z",
"IncludedQuantity": 0,
"MeterCategory": "Virtual Machines",
"MeterId": "d0bf9053-17c4-4fec-8502-4eb8376343a7",
"MeterName": "F2/F2s Low Priority",
"MeterRates": {
"0": 0.0766
},
"MeterRegion": "US West 2",
"MeterStatus": "Active",
"MeterSubCategory": "F/FS Series Windows",
"MeterTags": [],
"Unit": "1 Hour"
},
{
"EffectiveDate": "2014-11-01T00:00:00Z",
"IncludedQuantity": 0,
"MeterCategory": "Azure DevOps",
"MeterId": "c4d6fa88-0df9-4680-867a-b13c960a875f",
"MeterName": "Virtual User Minute",
"MeterRates": {
"0": 0.0004,
"1980000": 0.0002,
"9980000": 0.0001
},
"MeterRegion": "",
"MeterStatus": "Active",
"MeterSubCategory": "Cloud-Based Load Testing",
"MeterTags": [],
"Unit": "1/Month"
},
{
"EffectiveDate": "2017-04-01T00:00:00Z",
"IncludedQuantity": 0,
"MeterCategory": "SQL Database",
"MeterId": "cb770eab-d5c8-45fd-ac56-8c35069f5a29",
"MeterName": "P4 DTUs",
"MeterRates": {
"0": 68.64
},
"MeterRegion": "IN West",
"MeterStatus": "Active",
"MeterSubCategory": "Single Premium",
"MeterTags": [],
"Unit": "1/Day"
}
]
}
Actual results using the code i provided is the following:
"2019-03-01T00:00:00Z",0,"Virtual Machines","d0bf9053-17c4-4fec-8502-4eb8376343a7","F2/F2s Low Priority",0.0766,"US West 2","Active","F/FS Series Windows",
"2014-11-01T00:00:00Z",0,"Azure DevOps","c4d6fa88-0df9-4680-867a-b13c960a875f","Virtual User Minute",0.0004,0.0002,0.0001,"","Active","Cloud-Based Load Testing",
"2017-04-01T00:00:00Z",0,"SQL Database","cb770eab-d5c8-45fd-ac56-8c35069f5a29","P4 DTUs",68.64,"IN West","Active","Single Premium",
but the result i would expect is (record 2 to correspond to 3 records in the CSV file based on MeterRates):
"2019-03-01T00:00:00Z",0,"Virtual Machines","d0bf9053-17c4-4fec-8502-4eb8376343a7","F2/F2s Low Priority",0,0.0766,"US West 2","Active","F/FS Series Windows",
"2014-11-01T00:00:00Z",0,"Azure DevOps","c4d6fa88-0df9-4680-867a-b13c960a875f","Virtual User Minute",0,0.0004,"","Active","Cloud-Based Load Testing",
"2014-11-01T00:00:00Z",0,"Azure DevOps","c4d6fa88-0df9-4680-867a-b13c960a875f","Virtual User Minute",1980000,0.0002,"","Active","Cloud-Based Load Testing",
"2014-11-01T00:00:00Z",0,"Azure DevOps","c4d6fa88-0df9-4680-867a-b13c960a875f","Virtual User Minute",9980000,0.0001"","Active","Cloud-Based Load Testing",
"2017-04-01T00:00:00Z",0,"SQL Database","cb770eab-d5c8-45fd-ac56-8c35069f5a29","P4 DTUs",0,68.64,"IN West","Active","Single Premium",
Thank you for your help.

You'll want to add a step between getting Meters items and outputting rows, to output the various combinations of Meters items with different rates. As you have it right now, you're outputting the rates as other items for the row which isn't really what you want.
In this case, you could just add a new property to hold the value of the corresponding MeterRate.
.Meters[] | .MeterRate = (.MeterRates | to_entries[])
| [.EffectiveDate, .IncludedQuantity, .MeterCategory, .MeterId , .MeterName,
.MeterRate.key, .MeterRate.value,
.MeterRegion, .MeterStatus, .MeterSubCategory, .MeterTags[], .Units]
| #csv
You may want to consider doing something similar for the MeterTags items so you don't end up with potentially random column counts.

Add elements within array with same field name in jq

Need to add array elements with same field name
Input:
[
{
"all": 1,
"sys": "bus"
},
{
"all": 14,
"sys": "bus"
}
]
I have tried like below:
.[] | (.all +.all)
but got result like
2
28
Expected result: 15 (1 + 14)

First map, then add:
map(.all) | add
C'est tout.

How to split multiple values of a parameter in JSON response?

My JSON response has multiple values for a single attribute, as copied below.
{
"version": "10.2.2.48",
"systemMessages": [ {
"code": -8010,
"error": "",
"type": "error",
"module": "BROKER"
}],
"locations": [ {
"id": "10101102",
"name": "Bus Stop",
"disassembledName": "Bus",
"coord": [
3755258,
4889121
],
"type": "stop",
"matchQuality": 1000,
"isBest": true,
"modes": [
1,
5,
11
],
"parent": {
"id": "95301001|1",
"name": "Sydney",
"type": "locality"
},
"assignedStops": [ {
"id": "10101102",
"name": "Bus Stop",
"type": "stop",
"coord": [
3755258,
4889121
],
"parent": {
"name": "Sydney",
"type": "locality"
},
"modes": [
1,
5,
11
],
"connectingMode": 100
}]
}]
}
Observe that "modes" has 3 values. If I try to extract the value of modes through JsonSlurper script assertion, it gives the value as [1,5,11] and count as just 1. I want to split them into 3 array elements or variables and need the count as 3. What's the possible script assertion code?
Assertion:
import groovy.json.JsonSlurper
def resp = messageExchange.response.responseContent;
def jsl = new JsonSlurper().parseText(resp);
def modes = jsl.locations.modes
log.info modes
log.info modes.size()
Result:
Wed Feb 13 10:50:49 AEDT 2019:INFO:[[1, 5, 11]]
Wed Feb 13 10:50:49 AEDT 2019:INFO:1

What you are dealing with in this example is a shorthand version of Groovy's spread operator and your code returns a valid result. When you call jsl.locations you actually access a list of all locations objects (a singleton list in your example). When you call
jsl.locations.modes
you use a shorthand version of
jsl.locations*.modes
which is an equivalent of the following code:
jsl.locations.collect { it.modes }
This code means: iterate locations and transform a list of locations into a list of lists of modes of these locations - [[1,5,11]].
Applying the correct solution depends on a few more factors. For instance, you need to consider locations list containing multiple locations - in this case, transformation jsl.locations.modes may produce a result like [[1,5,11],[1,5,11],[2,4,9]] - a list of 3 modes lists.
If you assume that there is always a single location returned then you can simply flatten the final list like:
def modes = jsl.locations.modes.flatten()
assert modes == [1,5,11]
assert modes.size() == 3
However, if locations contains another JSON object, let's say, with exactly the same modes, then it will produce a completely different result:
def modes = jsl.locations.modes.flatten()
assert modes == [1,5,11,1,5,11]
assert modes.size() == 6
In this case, it might be better to use assertions like:
def modes = jsl.locations.modes
assert modes == [[1,5,11],[1,5,11]]
assert modes*.size() == [3,3]
which means:
modes stores 2 lists [1,5,11] and [1,5,11],
and the size of the first list is 3, and the size of the second list is also 3.