I am trying to list all keys with parent keys from a dictionary using python 3. How can I achieve this goal?
Here is so far I did using a recursive function (so that I can use this with any depth of dictionaries).
Here, if I do not use header_prefix, I get all the keys without parent keys. However, when I use header_prefix, it keeps adding parent keys incorrectly to the keys. Basically, I cannot reset header_prefix in the appropriate location.
from pprint import pprint
#%%
data = {
"AWSTemplateFormatVersion": "2010-09-09" ,
"Description": "Stack for MyProject 01",
"Resources": {
"elb01": {
"Type": "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties": {
"CrossZone" : "false",
"HealthCheck" : {
"Target" : "TCP:80",
"Interval" : "20"
},
"ConnectionSettings": {
"IdleTimeout": "120"
}
}
},
"lc01": {
"Type": "AWS::AutoScaling::LaunchConfiguration" ,
"Properties": {
"ImageId" : "ami-01010105" ,
"InstanceType" : "t2.medium"
}
},
"asg01": {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"HealthCheckGracePeriod" : 300,
"HealthCheckType" : "EC2"
}
}
}
}
pprint(data)
#%%
def get_headers(json_data, headers, header_prefix):
for key, value in json_data.items():
if type(value) == dict:
header_prefix = header_prefix + key + '.'
get_headers(value,headers,header_prefix)
else:
headers.append(header_prefix+key)
return(headers)
#%%
header_list = []
prefix = ''
data_headers = get_headers(data, header_list, prefix)
pprint(data_headers)
From the above code, I get the following output:
['AWSTemplateFormatVersion',
'Description',
'Resources.elb01.Type',
'Resources.elb01.Properties.CrossZone',
'Resources.elb01.Properties.HealthCheck.Target',
'Resources.elb01.Properties.HealthCheck.Interval',
'Resources.elb01.Properties.HealthCheck.ConnectionSettings.IdleTimeout',
'Resources.elb01.lc01.Type',
'Resources.elb01.lc01.Properties.ImageId',
'Resources.elb01.lc01.Properties.InstanceType',
'Resources.elb01.lc01.asg01.Type',
'Resources.elb01.lc01.asg01.Properties.HealthCheckGracePeriod',
'Resources.elb01.lc01.asg01.Properties.HealthCheckType']
My expected output is like below:
['AWSTemplateFormatVersion',
'Description',
'Resources.elb01.Type',
'Resources.elb01.Properties.CrossZone',
'Resources.elb01.Properties.HealthCheck.Target',
'Resources.elb01.Properties.HealthCheck.Interval',
'Resources.elb01.Properties.ConnectionSettings.IdleTimeout',
'Resources.lc01.Type',
'Resources.lc01.Properties.ImageId',
'Resources.lc01.Properties.InstanceType',
'Resources.asg01.Type',
'Resources.asg01.Properties.HealthCheckGracePeriod',
'Resources.asg01.Properties.HealthCheckType']
It seems to be a scoping issue. When you modify header_prefix inside the if statement, it modifies it in the function scope and so for all iterations of the loop, leading to the incorrect version being passed to get_headers in later iterations of the loop
In short:
Change
header_prefix = header_prefix + key + '.'
get_headers(value,headers,header_prefix)
To
pfx = header_prefix + key + '.'
get_headers(value,headers,pfx)
This way a new local variable will be created and passed, rather than the header_prefix being updated within the function scope.
(any variable name that's not used within the get_headers function will do
Related
PostgreSQL 13
The goal is to rename all src keys in the photos array to image.
I have a table plans which has a column json with a simplified structure similar to the below sample.
{
"id": "some id",
"name": "some name",
"tags": [
{
"keyId": 123,
"valueId": 123
},
{
"keyId": 123,
"valueId": 123
}
],
"score": 123,
"photos": [
{
"src": "someString"
},
{
"src": "someString"
}
],
"payment": true
}
The number of objects in the photos array varies, but in general, it is less than 10, so a non-iterating method would be fine, too.
I tried something like this, but it is only good for modifying the value of a key, not the name of the key itself.
UPDATE
plans
SET
json = jsonb_set(json::jsonb, '{photos, 0, src}', '"image"')
;
With the following attempt, I was actually able to rename the key but it overwrites everything else, so only an object with {"image": "someUrl"} is left:
UPDATE
plans
SET
json = (json -> 'photos' ->> 0)::jsonb - 'src' || jsonb_build_object ('image',
json::jsonb -> 'photos' -> 0 -> 'src')
WHERE json::jsonb ? 'photos' = true;
Is there a way to rename keys as expected?
So in the end I used a variation of my initial jsonb_set method. The solution isn't elegant or efficient, but since it is a one-time operation, it was only important to work:
UPDATE
plans
SET
json = jsonb_set(json::jsonb, '{photos, 0, imageUrl}', (json->'photos'->0->'src')::jsonb)
WHERE
json->'photos'->0->'src' IS NOT NULL
;
This query would add the imageUrl key with the existing value of the src key for the first object (position 0) in the photos array. So it left me with src and imageUrl key.
To remove the src key, I ran the following query
UPDATE
plans
SET
json = json::jsonb #- '{ photos, 0, src}'
;
Repeating this as many times as the maximum number of elements in a photos array eventually solved the issue for me.
I'm creating an AWS Step Function definition in Dhall. However, I don't know how to create a common structure they use for Choice states such as the example below:
{
"Not": {
"Variable": "$.type",
"StringEquals": "Private"
},
"Next": "Public"
}
The Not is pretty straightforward using mapKey and mapValue. If I define a basic Comparison:
{ Type =
{ Variable : Text
, StringEquals : Optional Text
}
, default =
{ Variable = "foo"
, StringEquals = None Text
}
}
And the types:
let ComparisonType = < And | Or | Not >
And adding a helper function to render the type as Text for the mapKey:
let renderComparisonType = \(comparisonType : ComparisonType )
-> merge
{ And = "And"
, Or = "Or"
, Not = "Not"
}
comparisonType
Then I can use them in a function to generate the record halfway:
let renderRuleComparisons =
\( comparisonType : ComparisonType ) ->
\( comparisons : List ComparisonOperator.Type ) ->
let keyName = renderComparisonType comparisonType
let compare = [ { mapKey = keyName, mapValue = comparisons } ]
in compare
If I run that using:
let rando = ComparisonOperator::{ Variable = "$.name", StringEquals = Some "Cow" }
let comparisons = renderRuleComparisons ComparisonType.Not [ rando ]
in comparisons
Using dhall-to-json, she'll output the first part:
{
"Not": {
"Variable": "$.name",
"StringEquals": "Cow"
}
}
... but I've been struggling to merge that with "Next": "Sup". I've used all the record merges like /\, //, etc. and it keeps giving me various type errors I don't truly understand yet.
First, I'll include an approach that does not type-check as a starting point to motivate the solution:
let rando = ComparisonOperator::{ Variable = "$.name", StringEquals = Some "Cow" }
let comparisons = renderRuleComparisons ComparisonType.Not [ rando ]
in comparisons # toMap { Next = "Public" }
toMap is a keyword that converts records to key-value lists, and # is the list concatenation operator. The Dhall CheatSheet has a few examples of how to use both of them.
The above solution doesn't work because # cannot merge lists with different element types. The left-hand side of the # operator has this type:
comparisons : List { mapKey : Text, mapValue : Comparison.Type }
... whereas the right-hand side of the # operator has this type:
toMap { Next = "Public" } : List { mapKey : Text, mapValue : Text }
... so the two Lists cannot be merged as-is due to the different types for the mapValue field.
There are two ways to resolve this:
Approach 1: Use a union whenever there is a type conflict
Approach 2: Use a weakly-typed JSON representation that can hold arbitrary values
Approach 1 is the simpler solution for this particular example and Approach 2 is the more general solution that can handle really weird JSON schemas.
For Approach 1, dhall-to-json will automatically strip non-empty union constructors (leaving behind the value they were wrapping) when translating to JSON. This means that you can transform both arguments of the # operator to agree on this common type:
List { mapKey : Text, mapValue : < State : Text | Comparison : Comparison.Type > }
... and then you should be able to concatenate the two lists of key-value pairs and dhall-to-json will render them correctly.
There is a second solution for dealing with weakly-typed JSON schemas that you can learn more about here:
Dhall Manual - How to convert an existing YAML configuration file to Dhall
The basic idea is that all of the JSON/YAML integrations recognize and support a weakly-typed JSON representation that can hold arbitrary JSON data, including dictionaries with keys of different shapes (like in your example). You don't even need to convert the entire the expression to this weakly-typed representation; you only need to use this representation for the subset of your configuration where you run into schema issues.
What this means for your example, is that you would change both arguments to the # operator to have this type:
let Prelude = https://prelude.dhall-lang.org/v12.0.0/package.dhall
in List { mapKey : Text, mapValue : Prelude.JSON.Type }
The documentation for Prelude.JSON.Type also has more details on how to use this type.
Below is the JSON response I receive when I am hitting a particular web service:
[
{
"sId" : "0001",
"sName" : "abc1",
"sPlace" : "abc11"
}, {
"sId" : "0002",
"sName" : "abc2",
"sPlace" : "abc12"
}, {
"sId" : "0003",
"sName" : "abc3",
"sPlace" : "abc13"
}, {
"sId" : "0004",
"sName" : "abc4",
"sPlace" : "abc14"
}
]
I don't know which index has my expected values (I need to validate multiple values after identifying which has sId == '0003'), this is dynamic. Don't want to user hard coded value.
And match response.[3].sId == '0003'
because this will be changed next time.
I have two questions regarding this:
How can I pass response to java code and get the array index which having sId == '0003' so that I can use this index to validate?
How can I pass a variable value as an array index in response?
The code below is not working.
def ind = Java.type('karate.Utility.FindIndex')
response.['#ind'].sId == '0003'
karate uses json-path which allows writing conditions to read data from JSON.
example:
* def sId = "0003"
* def sValue = karate.jsonPath(response, "$[?(#.sId == '" + sId + "')]")
* match sValue[0] == {"sId" : "0003","sName" : "abc3","sPlace" : "abc13"}
now if there is a match in sId on the response JSON array, all such matches will be returned.
No need to do * match sValue[0].sId == "0003" as this is your filter
criteria
More about JSON path
online JSON path evaluator
karate doc refernce
I have following code to create a JSON for making a call to Adobe Analytics API (method segment.save)
item <-
list(definition = list(
container = list (
type = "hits",
operator = "or",
rules=I(list(
list(value= "test1 test2",
operator = "contains_any",
element = "page")))
)
),
owner="test",
reportSuiteID="test",
description="API Generated Segment",
name="test segment"
)
Once prettyfied and auto-unboxed, the result is:
> jsonlite::toJSON(item, pretty = T, auto_unbox= T)
{
"definition": {
"container": {
"type": "hits",
"operator": "or",
"rules": [
{
"value": "test1 test2",
"operator": "contains_any",
"element": "page"
}
]
}
},
"owner": "test",
"reportSuiteID": "test",
"description": "API Generated Segment",
"name": "test segment"
}
Good for creating new segments, but not so good for editing them
The JSON structure is valid, as I am able to create the new segment. However, I would like to check if the segment already exists (using f.i. the GetSegments() function from randyzwitch RSiteCatalyst package and check if name coincides already with a created segment). If the segment already exists, I want to pass the id to the API call, which is the method used for editing already existing segments. It should then look like:
> jsonlite::toJSON(item, pretty = T, auto_unbox= T)
{
"definition": {
...
},
"owner": "test",
"reportSuiteID": "test",
"description": "API Generated Segment",
"name": "test segment",
"id": "s1982XXXXXXXXX_XXXXX_XXXXX",
}
It is possible to make an if alike statement within the list() definition provided in the first piece of code? I would like to reach a solution that does not need an if statement that checks if segmentID exists and, depending on it, generates a call with id or a call without id.
Once a "JSON alike structure" is created using list function:
item <-
list(definition = list(
container = list (
type = "hits",
operator = "or",
rules=I(list(
list(value= "test1 test2",
operator = "contains_any",
element = "page")))
)
),
owner="test",
reportSuiteID="test",
description="API Generated Segment",
name="test segment"
)
We can push new elements to this list using the needed conditions. For example, if we have our segment IDs in a dataframe with name segments, we can push this ID to item this way:
if (!is.na(segments$segmentID[i])) {
item <- c(item, id=segments$segmentID[i])
}
I have a raw json in following format-
"luns": [
{
"numReadBlocks": 15444876,
"numWriteBlocks": 13530714,
"blockSizeInBytes": 512,
"writeIops": 495344,
"readIops": 312702,
"serialNumber": "aaaaaaa",
"uuid": "id",
"shareState": "none",
"usedBytes": 6721716224,
"totalSizeBytes": 16106127360,
"path": "/vol/lun_23052014_025830_vol/lun_23052014_025830"
},
{
"numReadBlocks": 15444876,
"numWriteBlocks": 13530714,
"blockSizeInBytes": 512,
"writeIops": 495344,
"readIops": 312702,
"serialNumber": "aaaaaaa",
"uuid": "id",
"shareState": "none",
"usedBytes": 6721716224,
"totalSizeBytes": 16106127360,
"path": "/vol/lun_23052014_025830_vol/lun_23052014_025830"
}]
The luns may contains list.
I want to process above json and form output as following-
"topStorageLuns": [
{
"name": "Free (in GB)",
"data": [7.79,7.79]
},
{
"name": "Used (in GB)",
"data": [7.21,7.21]
}]
I tried following in order to get output-
val storageLuns = myRawJson
val topStorageLuns = storageLuns.map { storageLun =>
val totalLunsSizeOnStorageDevice = storageLun.luns.foldLeft(0.0) {
case (totalBytesOnDevice, lun) =>
totalBytesOnDevice + lun.usedBytes.getOrElse(0.0).toString.toLong
}
val totalAvailableLunsOnStorageDevice = storageLun.luns.foldLeft(0.0) {
case (totalBytesOnDevice, lun) =>
totalBytesOnDevice + lun.usedBytes.getOrElse(0.0).toString.toLong
}
Json.obj("name" -> storageLun.hostId, "data" -> "%.2f".format(totalLunsSizeOnStorageDevice / (1024 * 1024 * 1024)).toDouble)
}
Can anybody help me to get desired output please???
The key lesson I want to impart is that your algorithm should reflect the shape of the output you want. Work backward from the result you want to build the algorithm.
It looks to me like you want to create an array of length 2, where each entry has a corresponding algorithm (spaced used, space free). Within each of these elements, you want a nested array with an element for each item in your input array, calculated using the algorithm from the outer array. Here's how I would approach the problem:
1) Define your algorithms
val dfAlgorithm: (Seq[(String, JsValue)] => Double) = _.foldLeft(0.0) { (acc, item) =>
/* whatever logic you need to do */
}
val duAlgorithm: (Seq[(String, JsValue)] => Double) = _.foldLeft(0.0) { (acc, item) =>
/* whatever logic you need to do */
}
2) Create a data structure to map over to build your final output
val stats = Seq("Free (in GB)" -> dfAlgorithm, "Used (in GB)" -> duAlgorithm)
3) Map over your input data within your mapping over your algorithms (the logic here reflects the shape of the result you want)
stats.map { case (name, algorithm) =>
Json.obj("name" -> name, "data" -> storageLuns.map { storageLun => algorithm(storageLun) }
}
This isn't going to be a turnkey solution, since I don't know how your free/used algorithms are supposed to work, but this overall scheme should get you there.