How to Change the defaults in Great Expectations DataDoc HTML Report? - html

Great Expectations provides the ability to produce Html reports using DataDocs as shown in the folloiwng example:
I would like the change the following defaults in the header - see image
The report is generated using the following
validation_results_stg = ge_stg_update.validate(expectation_suite='stg_expectations.html', only_return_failures=False)
I believe I can make changes because if I run the code validatation_results_stg
I get the following output
{
"evaluation_parameters": {},
"meta": {
"great_expectations_version": "0.13.38",
"expectation_suite_name": "default",
"run_id": {
"run_time": "2021-11-18T14:59:49.831733+00:00",
"run_name": null
},
"batch_kwargs": {
"ge_batch_id": "2c157d12-4880-11ec-8b5e-000d3ad66fea"
},
"batch_markers": {},
"batch_parameters": {},
"validation_time": "20211118T145949.831530Z",
"expectation_suite_meta": {
"great_expectations_version": "0.13.38"
}
},
"results": [
{
"result": {
"element_count": 14539,
"missing_count": 0,
"missing_percent": 0.0,
"unexpected_count": 0,
"unexpected_percent": 0.0,
"unexpected_percent_total": 0.0,
"unexpected_percent_nonmissing": 0.0,
"partial_unexpected_list": []
},
"exception_info": {
"raised_exception": false,
"exception_message": null,
"exception_traceback": null
},
"meta": {},
"success": true,
"expectation_config": {
"kwargs": {
"column": "SERVICE",
"value_set": [
"CMC",
"Divorce",
"Probate",
"SSCS"
]
},
"expectation_type": "expect_column_values_to_be_in_set",
"meta": {},
"ge_cloud_id": null
}
},
{
"result": {
"observed_value": 14539
},
"exception_info": {
"raised_exception": false,
"exception_message": null,
"exception_traceback": null
},
"meta": {},
"success": true,
"expectation_config": {
"kwargs": {
"value": 14539
},
"expectation_type": "expect_table_row_count_to_equal",
"meta": {},
"ge_cloud_id": null
}
}
],
"success": true,
"statistics": {
"evaluated_expectations": 2,
"successful_expectations": 2,
"unsuccessful_expectations": 0,
"success_percent": 100.0
}
}
As you can see the defaults are loaded e.g. you will notice "expectation_suite_name": "default",
Can someone show me how you can the default to something else?

Related

How to configure everything into one or multiple configure files

I haven't used this since 2 years ago. I'm trying to figure out how to put everything into one json or it has to be mulitple jsons.
Using v11.9.46. The documentation is not clear as it references things that are for one version or another.
I'm moving items from ADOorg1 to ADOorg2. In this order I was going to setup the json
Areas and Iterations
Team setup
Shared Queries
Work Items (using a query to find select area)
This is what I had so far as I tried to figure out how to put it into one:
{
"ChangeSetMappingFile": null,
// "Endpoints": {
// "TfsTeamSettingsEndpoints": [
// {
// "Name": "TeamSettingsSource",
// "AccessToken": "",
// "Query": {
// "Query": "SELECT [System.Id], [System.Tags] FROM WorkItems WHERE [System.TeamProject] = #TeamProject AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan') ORDER BY [System.ChangedDate] desc"
// },
// "Organisation": "https://dev.azure.com/test1/",
// "Project": "test1",
// "ReflectedWorkItemIdField": "ReflectedWorkItemId",
// "AuthenticationMode": "AccessToken",
// "AllowCrossProjectLinking": false,
// "LanguageMaps": {
// "AreaPath": "Area",
// "IterationPath": "Iteration"
// }
// },
// {
// "Name": "TeamSettingsTarget",
// "AccessToken": "",
// "Query": {
// "Query": "SELECT [System.Id], [System.Tags] FROM WorkItems WHERE [System.TeamProject] = #TeamProject AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan') ORDER BY [System.ChangedDate] desc"
// },
// "Organisation": "https://dev.azure.com/test2/",
// "Project": "test2",
// "ReflectedWorkItemIdField": "ReflectedWorkItemId",
// "AuthenticationMode": "AccessToken",
// "AllowCrossProjectLinking": false,
// "LanguageMaps": {
// "AreaPath": "Area",
// "IterationPath": "Iteration"
// }
// }
// ]
// },
"Source": {
"$type": "TfsTeamProjectConfig",
"Collection": "https://dev.azure.com/test1/",
"Project": "test1",
"ReflectedWorkItemIDFieldName": "ReflectedWorkItemId",
"AllowCrossProjectLinking": false,
"AuthenticationMode": "Prompt",
"PersonalAccessToken": "",
"LanguageMaps": {
"AreaPath": "Area",
"IterationPath": "Iteration"
}
},
"Target": {
"$type": "TfsTeamProjectConfig",
"Collection": "https://dev.azure.com/test2/",
"Project": "Test2",
"ReflectedWorkItemIDFieldName": "ReflectedWorkItemId",
"AllowCrossProjectLinking": false,
"AuthenticationMode": "Prompt",
"PersonalAccessToken": "",
"LanguageMaps": {
"AreaPath": "Area",
"IterationPath": "Iteration"
}
},
"Endpoints": {
"InMemoryWorkItemEndpoints": [
{
"Name": "Source",
"EndpointEnrichers": null
},
{
"Name": "Target",
"EndpointEnrichers": null
}
]
},
"GitRepoMapping": null,
"LogLevel": "Information",
"Processors": [
{
"$type": "TfsAreaAndIterationProcessorOptions",
"Enabled": true,
"PrefixProjectToNodes": false,
"NodeBasePaths": null,
"ProcessorEnrichers": null,
"SourceName": "Source",
"TargetName": "Target"
},
{
"$type": "TfsTeamSettingsProcessorOptions",
"Enabled": false,
"MigrateTeamSettings": true,
"UpdateTeamSettings": true,
"PrefixProjectToNodes": false,
"Teams": null,
"ProcessorEnrichers": null,
"SourceName": "Source",
"TargetName": "Target"
},
{
"$type": "WorkItemMigrationConfig",
"Enabled": false,
"ReplayRevisions": true,
"PrefixProjectToNodes": false,
"UpdateCreatedDate": true,
"UpdateCreatedBy": true,
"BuildFieldTable": false,
"AppendMigrationToolSignatureFooter": false,
"WIQLQueryBit": "AND [Microsoft.VSTS.Common.ClosedDate] = '' AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan')",
"WIQLOrderBit": "[System.ChangedDate] desc",
"LinkMigration": true,
"AttachmentMigration": true,
"AttachmentWorkingPath": "E:\\temp\\WorkItemAttachmentWorkingFolder\\",
"FixHtmlAttachmentLinks": false,
"SkipToFinalRevisedWorkItemType": true,
"WorkItemCreateRetryLimit": 5,
"FilterWorkItemsThatAlreadyExistInTarget": true,
"PauseAfterEachWorkItem": false,
"AttachmentMaxSize": 480000000,
"CollapseRevisions": false,
"LinkMigrationSaveEachAsAdded": false,
"GenerateMigrationComment": true,
"NodeBasePaths": [
"Product\\Area\\Path1",
"Product\\Area\\Path2"
],
"WorkItemIDs": null
}
],
"Version": "0.0",
"workaroundForQuerySOAPBugEnabled": false,
"WorkItemTypeDefinition": {
"sourceWorkItemTypeName": "targetWorkItemTypeName"
}
// "Endpoints": {
// "InMemoryWorkItemEndpoints": [
// {
// "Name": "Source",
// "EndpointEnrichers": null
// },
// {
// "Name": "Target",
// "EndpointEnrichers": null
// }
// ]
// }
}
If you use this Azure DevOps Migration Tools, it can migrate Work Items, TestPlans & Suits, Teams, Shared Queries, & Pipelines from one Organization to another. You could refer to Processors: TfsAreaAndIterationProcessor, TfsTeamSettingsProcessor, TfsSharedQueryProcessor and Work Item Tracking Processor to create corresponding configuration Json files one by one.

Iterate over objects and print select values

Question: I would like to know how I can iterate over an object to print out the name of each subnet and its cidr range in the subnets object.
I know how I can print out the vnet name and cidr range:
jq '.[] | select(.vnet) | {name: .vnet.name, cidr: .vnet.address_space[0]}'
I have looked online and found that you can convert the object to an array, but I am unsure how I can look through that array to print out the values:
jq '.[] | to_entries | map_values(.value)'
Essentially the desired OUTPUT should look something like this:
{
"name": "asdf1",
"cidr": "10....."
},
{
"name": "asdf2",
"cidr": "10....."
}
//...and so forth
Sample INPUT:
{
"route_tables": {
"asdf": {
"disable_bgp_route_propagation": true,
"name": "az_afw",
"resource_group_name": "vnet-spoke",
"route_entries": {
"re1": {
"name": "rt-rfc-10-8",
"next_hop_in_ip_address": "10.0.0.0",
"next_hop_type": "VirtualAppliance",
"prefix": "10.0.0.0/8"
},
"re2": {
"name": "rt-rfc-172-12",
"next_hop_in_ip_address": "10.0.0.0",
"next_hop_type": "VirtualAppliance",
"prefix": "172.16.0.0/12"
}
}
}
},
"vnet_peering_settings": {
"peer1": {
"peer_to_source": {
"allow_forwarded_traffic": true,
"allow_gateway_transit": false,
"allow_virtual_network_access": true,
"use_remote_gateways": true
},
"source_to_peer": {
"allow_forwarded_traffic": true,
"allow_gateway_transit": true,
"allow_virtual_network_access": true,
"use_remote_gateways": false
}
}
},
"vnet_spoke_object": {
"specialsubnets": {},
"subnets": {
"objectAsdf1": {
"cidr": "10.0.0.1/24",
"enforce_private_link_endpoint_network_policies": false,
"enforce_private_link_service_network_policies": false,
"name": "asdf1",
"nsg_creation": true,
"nsg_inbound": [],
"nsg_outbound": [],
"route": null,
"service_endpoints": []
},
"objectAsdf2": {
"cidr": "10.0.0.1/24",
"enforce_private_link_endpoint_network_policies": false,
"enforce_private_link_service_network_policies": false,
"name": "asdf2",
"nsg_creation": true,
"nsg_inbound": [],
"nsg_outbound": [],
"route": "asdf",
"service_endpoints": [
"Microsoft.EventHub"
]
},
"objectAsdf3": {
"cidr": "10.0.0.1/24",
"enforce_private_link_endpoint_network_policies": false,
"enforce_private_link_service_network_policies": false,
"name": "asdf3",
"nsg_creation": true,
"nsg_inbound": [],
"nsg_outbound": [],
"route": "asdf",
"service_endpoints": []
},
"objectAsdf4": {
"cidr": "10.0.0.1/24",
"enforce_private_link_endpoint_network_policies": false,
"enforce_private_link_service_network_policies": false,
"name": "asdf4",
"nsg_creation": true,
"nsg_inbound": [],
"nsg_outbound": [],
"route": "asdf",
"service_endpoints": []
}
},
"vnet": {
"address_space": [
"10.0.0.0/16"
],
"ddos_id": "placeholder",
"dns": [
"10.0.0.1",
"10.0.0.1"
],
"enable_ddos_std": false,
"name": "asdf"
}
}
}
Start with the root path and traverse down and use to_entries to get past the varying key names and use the .value field only
jq '.vnet_spoke_object.subnets | to_entries[].value | { name, cidr }'
or collect it as an array of objects
jq '.vnet_spoke_object.subnets | to_entries | map(.value | { name, cidr })'
jqplay - Demo

How to transcode MP4 video with SRT subtitle on AWS Elemental MediaConvert

I have a MP4 video with SRT captions and I need to transcode them with media convert. In media convert I set automatic ABR and I specified the SRT origin path.
At the moment, I have tested the following:
I set SRT file in one output and video/audio in another
I set SRT, video and audio in the same output
For the first test, the job finish successfully, but on the S3 bucket there isnt any .SRT file. For the second test, the job fails with "aption destination type [SRT] requires a raw muxer." message
This is my JSON for the first test
{
"Queue": "arn:aws:mediaconvert:us-east-1:{{ACCOUNT-NUMBER}}:queues/Default",
"UserMetadata": {},
"Role": "arn:aws:iam::{{ACCOUNT-NUMBER}}:role/{{MY-ROLE-NAME}}",
"Settings": {
"TimecodeConfig": {
"Source": "ZEROBASED"
},
"OutputGroups": [
{
"Name": "DASH ISO",
"Outputs": [
{
"ContainerSettings": {
"Container": "MPD"
},
"VideoDescription": {
"ScalingBehavior": "DEFAULT",
"TimecodeInsertion": "DISABLED",
"AntiAlias": "ENABLED",
"Sharpness": 50,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"InterlaceMode": "PROGRESSIVE",
"ScanTypeConversionMode": "INTERLACED",
"NumberReferenceFrames": 3,
"Syntax": "DEFAULT",
"Softness": 0,
"GopClosedCadence": 1,
"GopSize": 90,
"Slices": 1,
"GopBReference": "DISABLED",
"SlowPal": "DISABLED",
"EntropyEncoding": "CABAC",
"FramerateControl": "INITIALIZE_FROM_SOURCE",
"RateControlMode": "QVBR",
"CodecProfile": "MAIN",
"Telecine": "NONE",
"MinIInterval": 0,
"AdaptiveQuantization": "AUTO",
"CodecLevel": "AUTO",
"FieldEncoding": "PAFF",
"SceneChangeDetect": "ENABLED",
"QualityTuningLevel": "MULTI_PASS_HQ",
"FramerateConversionAlgorithm": "DUPLICATE_DROP",
"UnregisteredSeiTimecode": "DISABLED",
"GopSizeUnits": "FRAMES",
"ParControl": "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames": 2,
"RepeatPps": "DISABLED",
"DynamicSubGop": "STATIC"
}
},
"AfdSignaling": "NONE",
"DropFrameTimecode": "ENABLED",
"RespondToAfd": "NONE",
"ColorMetadata": "INSERT"
},
"AudioDescriptions": [
{
"AudioTypeControl": "FOLLOW_INPUT",
"AudioSourceName": "Audio Selector 1",
"CodecSettings": {
"Codec": "AAC",
"AacSettings": {
"AudioDescriptionBroadcasterMix": "NORMAL",
"Bitrate": 96000,
"RateControlMode": "CBR",
"CodecProfile": "LC",
"CodingMode": "CODING_MODE_2_0",
"RawFormat": "NONE",
"SampleRate": 48000,
"Specification": "MPEG4"
}
},
"StreamName": "latino",
"LanguageCodeControl": "FOLLOW_INPUT",
"LanguageCode": "SPA"
}
]
},
{
"ContainerSettings": {
"Container": "MPD"
},
"CaptionDescriptions": [
{
"CaptionSelectorName": "Captions Selector 1",
"DestinationSettings": {
"DestinationType": "SRT"
},
"LanguageCode": "SPA",
"LanguageDescription": "latino"
}
]
}
],
"OutputGroupSettings": {
"Type": "DASH_ISO_GROUP_SETTINGS",
"DashIsoGroupSettings": {
"SegmentLength": 30,
"MinFinalSegmentLength": 0,
"Destination": "s3://{{BUCKET-NAME}}/streaming15/dash-iso/",
"FragmentLength": 2,
"SegmentControl": "SINGLE_FILE",
"MpdProfile": "ON_DEMAND_PROFILE",
"HbbtvCompliance": "NONE"
}
},
"AutomatedEncodingSettings": {
"AbrSettings": {
"MaxAbrBitrate": 8000000,
"MinAbrBitrate": 600000
}
}
}
],
"AdAvailOffset": 0,
"Inputs": [
{
"AudioSelectors": {
"Audio Selector 1": {
"Offset": 0,
"DefaultSelection": "DEFAULT",
"ProgramSelection": 1
}
},
"VideoSelector": {
"ColorSpace": "FOLLOW",
"Rotate": "DEGREE_0",
"AlphaBehavior": "DISCARD"
},
"FilterEnable": "AUTO",
"PsiControl": "USE_PSI",
"FilterStrength": 0,
"DeblockFilter": "DISABLED",
"DenoiseFilter": "DISABLED",
"InputScanType": "AUTO",
"TimecodeSource": "ZEROBASED",
"CaptionSelectors": {
"Captions Selector 1": {
"SourceSettings": {
"SourceType": "SRT",
"FileSourceSettings": {
"SourceFile": "s3://{{BUCKET-NAME}}/PROMO_CAP_01.srt"
}
}
}
},
"FileInput": "s3://{{BUCKET-NAME}}/PROMO_CAP_01.mp4"
}
]
},
"AccelerationSettings": {
"Mode": "DISABLED"
},
"StatusUpdateInterval": "SECONDS_60",
"Priority": 0
}
What I am missing?
According to the AWS Elemental MediaConvert user guide, SRT is not a supported output for a DASH-ISO output group when the input caption type is SRT.
Here's a link to that guide (reference page 176):
https://docs.aws.amazon.com/mediaconvert/latest/ug/mediaconvert-guide.pdf
The supported caption outputs for SRT input in DASH-ISO are:
Burn in
IMSC (as sidecar .fmp4)
IMSC (as sidecar .xml)
TTML (as sidecar .fmp4)
TTML (as sidecar .ttml)
Additionally, there is a gap in the documentation. SRT->DASH-ISO+WebVTT is supported, even though it is not listed. The documentation will be corrected, but I wanted to share that with you in case it helps.
If you must send SRT to the output destination, then you could create a separate output group where the caption is in a track with no container (see pages 192-196 in the document).

How to build a parent key based on a sub-string of another value using jq?

I'm re-shaping a JSON file which base content is the output of an AWS API call. From the raw output, I'm currently extracting the fields that I'm interested in the format I want except for a small detail that motivated this question.
Specifically, my input comes from the output of aws rds describe-db-instances command and contains the info of multiple RDS instances. Taking an example with 2 instances, this is how it looks:
{
"DBInstances": [
{
"DBInstanceIdentifier": "db1-name",
"DBInstanceClass": "db.m5.xlarge",
"Engine": "oracle-ee",
"DBInstanceStatus": "available",
"MasterUsername": "user",
"DBName": "RANDOM",
"Endpoint": {
"Address": "some-endpoint.rds.amazonaws.com",
"Port": 5698,
"HostedZoneId": "GHDSFHFSDHSDH"
},
"AllocatedStorage": 4000,
"InstanceCreateTime": "2018-07-23T23:21:42.361000+00:00",
"PreferredBackupWindow": "09:30-07:00",
"BackupRetentionPeriod": 14,
"DBSecurityGroups": [],
"VpcSecurityGroups": [
{
"VpcSecurityGroupId": "sg-xxxxxxxxxxxxxxxxx",
"Status": "active"
},
{
"VpcSecurityGroupId": "sg-xxxxxxxxxxxxxxxxx",
"Status": "active"
}
],
"DBParameterGroups": [
{
"DBParameterGroupName": "DB1-parameter",
"ParameterApplyStatus": "in-sync"
}
],
"AvailabilityZone": "ZONE1",
"DBSubnetGroup": {
"DBSubnetGroupName": "dbsubnetgroup-1",
"DBSubnetGroupDescription": "dbsubnetgroup-1",
"VpcId": "vpc-xxxxxxxxxxxxxxxxx",
"SubnetGroupStatus": "Complete",
"Subnets": [
{
"SubnetIdentifier": "subnet-xxxxxxxxxxxxxxxxx",
"SubnetAvailabilityZone": {
"Name": "az1"
},
"SubnetStatus": "Active"
},
{
"SubnetIdentifier": "subnet-xxxxxxxxxxxxxxxxx",
"SubnetAvailabilityZone": {
"Name": "az2"
},
"SubnetStatus": "Active"
}
]
},
"PreferredMaintenanceWindow": "sat:07:00-sat:07:30",
"PendingModifiedValues": {},
"LatestRestorableTime": "2020-03-27T18:54:25+00:00",
"MultiAZ": false,
"EngineVersion": "X.X.X",
"AutoMinorVersionUpgrade": false,
"ReadReplicaDBInstanceIdentifiers": [],
"LicenseModel": "bring-your-own-license",
"Iops": 5000,
"OptionGroupMemberships": [
{
"OptionGroupName": "optiongroupName",
"Status": "in-sync"
}
],
"CharacterSetName": "WE8ISO8859P15",
"PubliclyAccessible": false,
"StorageType": "io1",
"DbInstancePort": 0,
"StorageEncrypted": true,
"KmsKeyId": "someKey",
"DbiResourceId": "db-xxxxxxxxxxxxxxxxxxxxxxxxx",
"CACertificateIdentifier": "rds-ca-2019",
"DomainMemberships": [],
"CopyTagsToSnapshot": true,
"MonitoringInterval": 0,
"DBInstanceArn": "someARN",
"IAMDatabaseAuthenticationEnabled": false,
"PerformanceInsightsEnabled": false,
"DeletionProtection": false,
"AssociatedRoles": []
},
{
"DBInstanceIdentifier": "db2-name",
"DBInstanceClass": "db.m5.large",
"Engine": "oracle-ee",
"DBInstanceStatus": "available",
"MasterUsername": "user2",
"DBName": "XXXX",
"Endpoint": {
"Address": "endpoint2.rds.amazonaws.com",
"Port": 8974,
"HostedZoneId": "FASDFDS54FSA"
},
"AllocatedStorage": 100,
"InstanceCreateTime": "2020-04-23T21:38:53.023000+00:00",
"PreferredBackupWindow": "01:00-05:30",
"BackupRetentionPeriod": 35,
"DBSecurityGroups": [],
"VpcSecurityGroups": [
{
"VpcSecurityGroupId": "sg-xxxxxxxxxxxxxxxxx",
"Status": "active"
}
],
"DBParameterGroups": [
{
"DBParameterGroupName": "default",
"ParameterApplyStatus": "in-sync"
}
],
"AvailabilityZone": "AZ-2",
"DBSubnetGroup": {
"DBSubnetGroupName": "subnet-group",
"DBSubnetGroupDescription": "",
"VpcId": "vpc-xxxxxxxxxxxxxxxxx",
"SubnetGroupStatus": "Complete",
"Subnets": [
{
"SubnetIdentifier": "subnet-xxxxxxxxxxxxxxxxx",
"SubnetAvailabilityZone": {
"Name": "AZ-1"
},
"SubnetStatus": "Active"
},
{
"SubnetIdentifier": "subnet-xxxxxxxxxxxxxxxxx",
"SubnetAvailabilityZone": {
"Name": "AZ-2"
},
"SubnetStatus": "Active"
},
{
"SubnetIdentifier": "subnet-xxxxxxxxxxxxxxxxx",
"SubnetAvailabilityZone": {
"Name": "AZ-3"
},
"SubnetStatus": "Active"
}
]
},
"PreferredMaintenanceWindow": "sun:08:39-sun:09:09",
"PendingModifiedValues": {},
"LatestRestorableTime": "2020-07-27T18:53:18+00:00",
"MultiAZ": false,
"EngineVersion": "X.X.X",
"AutoMinorVersionUpgrade": false,
"ReadReplicaDBInstanceIdentifiers": [],
"LicenseModel": "bring-your-own-license",
"Iops": 2000,
"OptionGroupMemberships": [
{
"OptionGroupName": "optiongroup-name",
"Status": "in-sync"
}
],
"CharacterSetName": "AL32UTF8",
"PubliclyAccessible": false,
"StorageType": "io1",
"DbInstancePort": 0,
"StorageEncrypted": true,
"KmsKeyId": "someARN",
"DbiResourceId": "db-xxxxxxxxxxxxxxxxx",
"CACertificateIdentifier": "rds-ca-2019",
"DomainMemberships": [],
"CopyTagsToSnapshot": false,
"MonitoringInterval": 0,
"DBInstanceArn": "someARN",
"IAMDatabaseAuthenticationEnabled": false,
"PerformanceInsightsEnabled": false,
"DeletionProtection": false,
"AssociatedRoles": []
}
]
}
This is my current output:
[
{
"DBInstancePrefix": {
"DBInstanceClass": "db.m5.xlarge",
"DBInstanceIdentifier": "db1-name",
"DBName": "RANDOM",
"DBParameterGroupName": "DB1-parameter",
"DBSubnetGroupName": "dbsubnetgroup-1",
"KmsKeyId": "someKey",
"OptionGroupName": "optiongroupName",
"VpcSecurityGroupIds": [
"sg-xxxxxxxxxxxxxxxxx",
"sg-xxxxxxxxxxxxxxxxx"
]
}
},
{
"DBInstancePrefix": {
"DBInstanceClass": "db.m5.large",
"DBInstanceIdentifier": "db2-name",
"DBName": "XXXX",
"DBParameterGroupName": "default",
"DBSubnetGroupName": "subnet-group",
"KmsKeyId": "someARN",
"OptionGroupName": "optiongroup-name",
"VpcSecurityGroupIds": [
"sg-xxxxxxxxxxxxxxxxx"
]
}
}
]
This is my current JQ filter:
. | [.[] | map(.) | .[] | {DBInstancePrefix: {DBInstanceClass: .DBInstanceClass, DBInstanceIdentifier: .DBInstanceIdentifier, DBName: .DBName, DBParameterGroupName:.DBParameterGroups[].DBParameterGroupName, DBSubnetGroupName: .DBSubnetGroup.DBSubnetGroupName, KmsKeyId:.KmsKeyId, OptionGroupName: .OptionGroupMemberships[].OptionGroupName, VpcSecurityGroupIds: [.VpcSecurityGroups | .[] | .VpcSecurityGroupId] }}]
You can verify it on this snippet on jqplay.org.
What I need is to turn the parent key "DBInstancePrefix" dynamic based on a substring from "DBInstanceIdentifier" key. So for the example names I wrote would be:
"db1-name" >>> "db1"
"db2-name" >>> "db2"
So, my desired output would be:
[
{
"db1": {
"DBInstanceClass": "db.m5.xlarge",
"DBInstanceIdentifier": "db1-name",
"DBName": "RANDOM",
"DBParameterGroupName": "DB1-parameter",
"DBSubnetGroupName": "dbsubnetgroup-1",
"KmsKeyId": "someKey",
"OptionGroupName": "optiongroupName",
"VpcSecurityGroupIds": [
"sg-xxxxxxxxxxxxxxxxx",
"sg-xxxxxxxxxxxxxxxxx"
]
}
},
{
"db2": {
"DBInstanceClass": "db.m5.large",
"DBInstanceIdentifier": "db2-name",
"DBName": "XXXX",
"DBParameterGroupName": "default",
"DBSubnetGroupName": "subnet-group",
"KmsKeyId": "someARN",
"OptionGroupName": "optiongroup-name",
"VpcSecurityGroupIds": [
"sg-xxxxxxxxxxxxxxxxx"
]
}
}
]
Any ideas or suggestions? Thanks for reading.
To manipulate the object key-name as you desire, you can apply the filter operation inside (..). Any operation done inside it, the result is preserved "literally".
Your case demands the DBInstanceIdentifier to be split by - and using the first element in the resultant array.
With that and few other trivial modifications, you need
.DBInstances |
map
(
{
( .DBInstanceIdentifier | split("-")[0] ): {
DBInstanceClass,
DBInstanceIdentifier,
DBName,
DBParameterGroupName:.DBParameterGroups[].DBParameterGroupName,
DBSubnetGroupName: .DBSubnetGroup.DBSubnetGroupName,
KmsKeyId,
OptionGroupName: .OptionGroupMemberships[].OptionGroupName,
VpcSecurityGroupIds: [.VpcSecurityGroups[] | .VpcSecurityGroupId ]
}
}
)
jqplay - Demo
I took the long path and since I couldn't resolve it on the same filter I did another one that takes the output of the first one and thus achieved the desired result. But it is awful compared with the answer selected as the solution.
Based on Inian answer, I did a small modification since my real DBInstanceIdentifier values have the following format:
<name>-db-<environment> and I need DBInstancePrefix becomes <name>-db.
So, my final filter is:
.DBInstances |
map
(
{
( .DBInstanceIdentifier|split("-")[0:2] | join("-") ): {
DBInstanceClass,
DBInstanceIdentifier,
DBName,
DBParameterGroupName:.DBParameterGroups[].DBParameterGroupName,
DBSubnetGroupName: .DBSubnetGroup.DBSubnetGroupName,
KmsKeyId,
OptionGroupName: .OptionGroupMemberships[].OptionGroupName,
VpcSecurityGroupIds: [.VpcSecurityGroups[] | .VpcSecurityGroupId ]
}
}
)

how to pass params as json arrays in postman having key and value pair

I am not able to pass json array in postman with key and value pair.
when using like this working
[
{
"serviceTypeId": 40,
"serviceName": "standard Cut2",
"active": true,
},
{
"serviceTypeId": 44,
"serviceName": "Special Cut1",
"active": true,
},
{
"serviceTypeId": 46,
"serviceName": "Feather xxxy",
"active": true,
}
]
but when i add a key its not(i.e like below)
{"serviceType": [
{
"serviceTypeId": 40,
"serviceName": "standard Cut2",
"active": true,
},
{
"serviceTypeId": 44,
"serviceName": "Special Cut1",
"active": true,
},
{
"serviceTypeId": 46,
"serviceName": "Feather xxxy",
"active": true,
}
]
}
Your Json is not correctly formatted, there is an extra comma after active attribute. Remove it. It should be :
[
{
"serviceTypeId":40,
"serviceName":"standard Cut2",
"active":true
},
{
"serviceTypeId":44,
"serviceName":"Special Cut1",
"active":true
},
{
"serviceTypeId":46,
"serviceName":"Feather xxxy",
"active":true
}
]
Try this in array :
{
"serviceType":[
{
"serviceTypeId":40,
"serviceName":"standard Cut2",
"active":true
},
{
"serviceTypeId":44,
"serviceName":"Special Cut1",
"active":true
},
{
"serviceTypeId":46,
"serviceName":"Feather xxxy",
"active":true
}
]
}
When your not sure about your JSON you can use tools on the internet like this : https://jsonformatter.curiousconcept.com/ .
You will be able to see quickly if your json is correct.