JQ edit file in place after using JQ Select - json

to edit a json file inplace with JQ like -i in sed I have found many solutions like
jq ... input.json > tmp.json && mv tmp.json input.json
This works, but I need to filter on my file, add some data, and place it back in the original. (ie. Update a specific object in a file)
My file contains over 1000 objects with different Element. I will always be using a filter on Element, I shortened for question. I have a sample file original.json ...
{
"Element": "acton",
"objectName": "contacts",
"Path": "/contacts",
"Scenario": "",
"serviceLevel": "",
"IBM": "",
"Gap": "",
"clientPhase": "",
"parentPhase": "",
"existsToday": ""
}
{
"Element": "acton",
"objectName": "optouts",
"Path": "/optouts",
"Scenario": "",
"serviceLevel": "",
"IBM": "",
"Dependency": "",
"Gap": "",
"clientPhase": "",
"parentPhase": "",
"existsToday": ""
}
{
"Element": "acton",
"objectName": "subscriptionTypes",
"Path": "/subscription-types",
"Scenario": "",
"serviceLevel": "",
"IBM": "",
"Dependency": "",
"Gap": "",
"clientPhase": "",
"parentPhase": "",
"existsToday": ""
}
I want to filter by objectName for contacts add some data to empty IBM field and save to the file
Applying the same logic to inplace edit the IBM field to "Y" on object containing "objectName": "contacts"
jq 'select(.objectName == "contacts") | .IBM = "Y"' original.json > tmpjson.json && mv tmpjson.json original.json
Now my file original.json shows
{
"Element": "acton",
"objectName": "contacts",
"Path": "/contacts",
"Scenario": "",
"serviceLevel": "",
"IBM": "Y",
"Dependency": "",
"Gap": "",
"clientPhase": "",
"parentPhase": "",
"existsToday": ""
}
Expected Result
{
"Element": "acton",
"objectName": "contacts",
"Path": "/contacts",
"Scenario": "",
"serviceLevel": "",
"IBM": "Y",
"Gap": "",
"clientPhase": "",
"parentPhase": "",
"existsToday": ""
}
{
"Element": "acton",
"objectName": "optouts",
"Path": "/optouts",
"Scenario": "",
"serviceLevel": "",
"IBM": "",
"Dependency": "",
"Gap": "",
"clientPhase": "",
"parentPhase": "",
"existsToday": ""
}
{
"Element": "acton",
"objectName": "subscriptionTypes",
"Path": "/subscription-types",
"Scenario": "",
"serviceLevel": "",
"IBM": "",
"Dependency": "",
"Gap": "",
"clientPhase": "",
"parentPhase": "",
"existsToday": ""
}
It seems after using select I can no longer use the provided solution https://github.com/stedolan/jq/wiki/FAQ#general-questions

Your jq filter is not the one you need because select selects. That is, it filters out the objects that don't match the selection criterion.
This jq filter should do what you want:
if (.objectName == "contacts") then .IBM = "Y" else . end
As for overwriting the file, many people who have access to sponge (part of the moreutils collection of CLI utilities) use it, e.g.
jq 'if (.objectName == "contacts") then .IBM = "Y" else . end' input.json |
sponge input.json

Related

how to give a json with about 1000 fields random values

I have a json with about 1000 fields like below
{
"Order": {
"AdjustmentInvoicePending": "",
"AllAddressesVerified": "",
"AllocationRuleID": "",
"AuthorizationExpirationDate": "",
"BillToKey": "",
"CarrierAccountNo": "",
"ChainType": "",
"ChargeActualFreightFlag": "",
"ComplimentaryGiftBoxQty": "",
"ContactKey": "",
"CustomerAge": "",
"CustomerEMailID": "",
"CustomerFirstName": "",
"CustomerLastName": "",
"CustomerPhoneNo": "",
"CustomerZipCode": "",
"Division": "",
"DocumentType": "",
"DraftOrderFlag": "",
"EnterpriseCode": "",
"EntryType": "",
"FreightTerms": "",
"HasDeliveryLines": "",
"HasDerivedChild": "",
"HasDerivedParent": "",
"HasProductLines": "",
"HoldFlag": "",
"HoldReasonCode": "",
"InvoiceComplete": "",
"isHistory": "",
"MaxOrderStatus": "",
"MaxOrderStatusDesc": "",
"MinOrderStatus": "",
"MinOrderStatusDesc": "",
"NoOfAuthStrikes": "",
"NotifyAfterShipmentFlag": "",
"OrderClosed": "",
"OrderDate": "",
"OrderHeaderKey": "",
"OrderNo": "",
"OrderType": "",
"OriginalTax": "",
"OriginalTotalAmount": "",
"OtherCharges": "",
"PaymentRuleId": "",
"PaymentStatus": "",
"PendingTransferIn": "",
"PriorityNumber": "",
"ReqCancelDate": "",
"ReturnByGiftRecipient": "",
"SaleVoided": "",
"ScacAndService": "",
"ScacAndServiceKey": "",
"SellerOrganizationCode": "",
"ShipToKey": "",
"SourcingClassification": "",
"Status": "",
"Tax": "",
"TotalAdjustmentAmount": "",
"AdditionalAddresses": {
"AdditionalAddress": [
{
"AddressType": "",
"EntityKey": "",
"PersonInfoKey": "",
"PersonInfo": {
"AddressID": "",
"AddressLine1": "",
"AddressLine2": "",
"AddressLine3": "",
"AddressLine4": "",
"AddressLine5": "",
"AddressLine6": "",
"AddressType": "",
"AlternateEmailID": "",
"Beeper": "",
"City": "",
"Company": "",
"Country": "",
"DayFaxNo": "",
"DayPhone": "",
"Department": "",
"EMailID": "",
"EveningFaxNo": "",
"EveningPhone": "",
"FirstName": "",
"IsAddressVerified": "",
"IsCommercialAddress": "",
"JobTitle": "",
"LastName": "",
"Latitude": "",
"Longitude": "",
"MiddleName": "",
"MobilePhone": "",
"OtherPhone": "",
"PersonID": "",
"PersonInfoKey": "",
"State": "",
"Suffix": "",
"TaxGeoCode": "",
"Title": "",
"ZipCode": ""
}
}
]
}
I want an output like "AdjustmentInvoicePending": "WQEMQAMCDQ", With all null spots replace with random unique values which changes each time i run the code
I tried using python and coverted json and assigned it to variable but am stuck from their i am new to programming and first time posting on slack please go easy on me if you are not able to understand and please forgive any typo
I recommend just using nested loops to iterate through and add random values. A snippet is below. You'll need to change it to work precisely for your problem. With that said, you can't guarantee randomness and uniqueness. For most uses, you can get by with just using a pseudorandom generator, but there is always the chance for collisions when dealing with randomness.
Also, this will fail with the list embedded in the dictionary, so you'll need to add some extra control-flow logic to deal with that circumstance, but this should help with getting you started.
import string
import random
N = 10 # length of the string
for i in json_var.keys(): # json_var is your json
for j in i.keys():
json_var[i][j].join(random.choices(string.ascii_uppercase, k=N))

jq: error: syntax error, unexpected if, while checking for a object to be present and if not the adding default value

Getting the error
jq: error: syntax error, unexpected if (Unix shell quoting issues?) at <top-level> while parsing kubernetes kubectl command,
kubectl get pv -o=json | jq -c '.items[] | {name: .metadata.name, claimName: .spec | if has("claimRef") then .claimRef.name else empty end }'
Trying to get the PV's which have PVC and if there are no PVC can be null of empty.
For it uilding a map of name and their respective PVC if there otherwise empty or null.
Below is a sample input to the jq command or the out put of the command kubectl get pv -o=json
{
"apiVersion": "v1",
"items": [{
"apiVersion": "v1",
"kind": "PersistentVolume",
"metadata": {
"annotations": {
"meta.helm.sh/release-name": "",
"meta.helm.sh/release-namespace": ""
},
"creationTimestamp": "",
"finalizers": [
""
],
"labels": {
"app.kubernetes.io/managed-by": ""
},
"name": "",
"resourceVersion": "",
"uid": ""
},
"spec": {
"accessModes": [
"ReadWriteMany"
],
"capacity": {
"storage": ""
},
"claimRef": {
"apiVersion": "",
"kind": "",
"name": "",
"namespace": "",
"resourceVersion": "",
"uid": ""
},
"csi": {
"driver": "",
"volumeHandle": ""
},
"persistentVolumeReclaimPolicy": "",
"volumeMode": ""
},
"status": {
"phase": ""
}
},
{
"apiVersion": "v1",
"kind": "PersistentVolume",
"metadata": {
"annotations": {
"meta.helm.sh/release-name": "",
"meta.helm.sh/release-namespace": ""
},
"creationTimestamp": "",
"finalizers": [
""
],
"labels": {
"app.kubernetes.io/managed-by": ""
},
"name": "",
"resourceVersion": "",
"uid": ""
},
"spec": {
"accessModes": [
"ReadWriteMany"
],
"capacity": {
"storage": ""
},
"claimRef": {
"apiVersion": "",
"kind": "",
"name": "",
"namespace": "",
"resourceVersion": "",
"uid": ""
},
"csi": {
"driver": "",
"volumeHandle": ""
},
"persistentVolumeReclaimPolicy": "",
"volumeMode": ""
},
"status": {
"phase": ""
}
}
],
"kind": "List",
"metadata": {
"resourceVersion": "",
"selfLink": ""
}
}
For data security have removed the data and only kept the keys
Please help
Put parentheses around the value of the claimName field:
{
name: .metadata.name,
claimName: (.spec | if has("claimRef") then .claimRef.name else empty end)
}
Depending on how much you simplified your example, you may consider reducing the the if statement to:
{
name: .metadata.name,
claimName: (.spec.claimRef.name? // empty)
}
Note that in both cases not just .claimName but the whole object will disappear if the respective test is going to fail.

Create Table in Athena From Nested JSON

I have nested JSON of type
[{
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}],
"licenses": [{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}, {
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}]
}, {
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}],
"licenses": [{
"allocated": "2016-04-26 01:46:26",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}]
}]
which is not able to be converted to athena table.
I have tried to update it to list of objects also
{
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}
],
"licenses": [{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
},{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}
]
}
{
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}
],
"licenses": [{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}
]
}
{
"emails": [{
"label": "",
"primary": "",
"relationdef_id": "",
"type": "",
"value": ""
}
],
"licenses": [{
"allocated": "",
"parent_type": "",
"parentid": "",
"product_type": "",
"purchased_license_id": "",
"service_type": ""
}
]
}
with Query:
CREATE EXTERNAL TABLE `test_orders1`(
`emails` array<struct<`label`: string, `primary`: string,`relationdef_id`: string,`type`: string, `value`: string>>,
`licenses` array<struct<`allocated`: string, `parent_type`: string, `parentid`: string, `product_type`: string,`purchased_license_id`: string, `service_type`: string>>)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ( 'ignore.malformed.json' = 'true')
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
but only 1 row is formed.
Is there a way where i can use Nested json of type JSONArray into Athena table?
Or how can I change Nested Json that will work for me?
When querying JSON data Athena requires the files to be formatted with one JSON document per line. It's unclear from your question if this is the case or not, the examples you give are multiline, but perhaps that's only to make the question more clear.
The table DDL you include looks like it should work on the second example data, provided that it is formatted as one document per line, e.g.
{"emails": [{"label": "", "primary": "", "relationdef_id": "", "type": "", "value": ""}], "licenses": [{"allocated": "", "parent_type": "", "parentid": "", "product_type": "", "purchased_license_id": "", "service_type": ""}, { "allocated": "", "parent_type": "", "parentid": "", "product_type": "", "purchased_license_id": "", "service_type": ""}]}
{"emails": [{"label": "", "primary": "", "relationdef_id": "", "type": "", "value": ""}], "licenses": [{"allocated": "", "parent_type": "", "parentid": "", "product_type": "", "purchased_license_id": "", "service_type": ""}]}
{"emails": [{"label": "", "primary": "", "relationdef_id": "", "type": "", "value": ""}], "licenses": [{"allocated": "", "parent_type": "", "parentid": "", "product_type": "", "purchased_license_id": "", "service_type": ""}]}

Create JSON.NET structure with JTokenWriter

Hey all I have the following json output that I would like to create:
{
"scheduleName": "",
"firstName": "",
"lastName": "",
"theRole": "",
"linker": "",
"Schedule": {
"ID": "",
"totalHrs": "",
"Mon": "",
"Tue": "",
"Wed": "",
"Thu": "",
"Fri": "",
"Sat": ""
},
"empInfo": {
"ID": "",
"Email": "",
"Phone": "",
"Active": "",
"Img": "",
"Badge": ""
},
"availability": {
"ID": "",
"Mon": "",
"Tue": "",
"Wed": "",
"Thu": "",
"Fri": "",
"Sat": ""
},
"training": {
"name": "",
"id": ""
}
}
Using the newtonsoft Create JSON with JTokenWriter I am wondering how to create the "Schedule", "empInfo", etc in my json output since there are no examples on the page of those types.
The only example it shows is structured like so:
{
"name1": "value1",
"name2": [
1,
2
]
}
The first few values are easy to create:
Dim jsonWriter As New JTokenWriter()
jsonWriter.WriteStartObject()
jsonWriter.WritePropertyName("scheduleName")
jsonWriter.WriteValue("value1")
jsonWriter.WritePropertyName("firstName")
jsonWriter.WriteValue("value2")
jsonWriter.WritePropertyName("lastName")
jsonWriter.WriteValue("value3")
jsonWriter.WritePropertyName("theRole")
jsonWriter.WriteValue("value4")
jsonWriter.WritePropertyName("linker")
jsonWriter.WriteValue("value5")
'"?": {
' "?": "?",
' "?": "?",
' etc....
'?
jsonWriter.WriteEndObject()
But that's where I have to stop since I do not know how to go about making the other structure.
To write a nested object as the value of a property, write the property name, then do a nested WriteStartObject(), followed by the properties to be written, and finally a nested WriteEndObject(). E.g.:
Dim jsonWriter As New JTokenWriter()
jsonWriter.WriteStartObject() 'Start the root object
jsonWriter.WritePropertyName("scheduleName")
jsonWriter.WriteValue("value1")
jsonWriter.WritePropertyName("Schedule") 'Write the "Schedule" property name
jsonWriter.WriteStartObject() 'Start the nested "Schedule" object
jsonWriter.WritePropertyName("ID")
jsonWriter.WriteValue("ID Value")
jsonWriter.WriteEndObject() 'End the Schedule object
jsonWriter.WriteEndObject() 'End the root object
Sample fiddle.

SoapUI JSON Parsing - Get Parent Node using Child value

I have a JSON response like below from SoapUI.
{
"-1": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 1,
"build": "",
"projectKey": "TEST",
"started": ""
},
"0": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 2,
"build": "",
"projectKey": "BEST",
"started": ""
},
"2": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 3,
"build": "",
"projectKey": "WORST",
"started": ""
}
}
My requirement is, I have to get the value/node "0" which has the projectkey="BEST" using JsonSurpler or Groovy Script Test Step. The projectkey is now under the node "0". Maybe it will be under "10" or "1000" or "-500".
How to get the parent node using a child node value?
There are different ways to achieve that. These are the simplest I can recollect now:
import groovy.json.JsonSlurper
String string = '''{
"-1": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 1,
"build": "",
"projectKey": "TEST",
"started": ""
},
"0":{
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 2,
"build": "",
"projectKey": "BEST",
"started": ""
},
"2": {
"startDate": "",
"modifiedBy": "",
"endDate": "",
"projectId": 3,
"build": "",
"projectKey": "WORST",
"started": ""
}
}'''
def json = new JsonSlurper().parseText(string)
assert json.find { it.value?.projectKey == 'BEST' }?.key == '0'
assert json.findResult { k, v -> v?.projectKey == 'BEST' ? k : null } == '0'
This will find the first item which satisfies the condition. If you need all the items which satisfy the condition then use findAll or findResults accordingly.