Getting JSON data in Groovy - json

I need to get some datas from JSON, I could manage to transform it into String. For example, I need to get the amount value if team role id is 4.( The last scope in the JSON.) When I run the code below, the "result" output is
{id=1, effectiveDate=2003-01-01, currencyCode=USD, rates=[{id=1, rateTable={id=1, effectiveDate=2003-01-01, currencyCode=USD, name=Tempo Default Price Table, defaultTable=false}, amount=0.0, link={type=DEFAULT_RATE}}], name=Tempo Default Price Table, defaultTable=true}
How can I get the whole data?
Thanks.
http.request(Method.GET) {
response.success = { resp, json ->
arrayDen = JsonOutput.toJson(json).substring(1, JsonOutput.toJson(json).length()-1)
}
}
def slurper = new groovy.json.JsonSlurper()
def result = slurper.parseText(arrayDen)
log.warn(result)
[
{
"id": 1,
"rateTable": {
"id": 1,
"effectiveDate": "2003-01-01",
"currencyCode": "USD",
"name": "Tempo Default Price Table",
"defaultTable": false
},
"amount": 0.0,
"link": {
"type": "DEFAULT_RATE"
}
},
{
"id": 2,
"rateTable": {
"id": 3,
"effectiveDate": "2022-03-21",
"currencyCode": "USD",
"name": "Rate",
"defaultTable": false
},
"amount": 0.0,
"link": {
"type": "DEFAULT_RATE"
}
},
{
"id": 3,
"rateTable": {
"id": 3,
"effectiveDate": "2022-03-21",
"currencyCode": "USD",
"name": "Rate",
"defaultTable": false
},
"amount": 200.0,
"link": {
"type": "TEAM_ROLE",
"id": 8
}
},
{
"id": 4,
"rateTable": {
"id": 3,
"effectiveDate": "2022-03-21",
"currencyCode": "USD",
"name": "Rate",
"defaultTable": false
},
"amount": 500.0,
"link": {
"type": "TEAM_ROLE",
"id": 5
}
},
{
"id": 5,
"rateTable": {
"id": 3,
"effectiveDate": "2022-03-21",
"currencyCode": "USD",
"name": "Rate",
"defaultTable": false
},
"amount": 1000.0,
"link": {
"type": "TEAM_ROLE",
"id": 4
}
}
]
`

Hmmm. The following worked fine for me:
def arrayDen = '<YOUR_URL_GOES_HERE>'.toURL().text
def slurper = new groovy.json.JsonSlurper()
def result = slurper.parseText(arrayDen)
def desiredData = result.find { it.id == 4 }
println desiredData.amount
You might give it a try.

Related

Check if a key exists and return another key

I need help with jq syntax on how to return the Gitlab job ID if it contains an artifact. The JSON output looks like this (removed a lot of unrelated info from it and added [...]):
[{
"id": 3219589880,
"status": "success",
"stage": "test",
"name": "job_with_no_artifact",
"ref": "main",
"tag": false,
"coverage": null,
"allow_failure": false,
"created_at": "2022-10-24T18:21:25.119Z",
"started_at": "2022-10-24T18:21:25.986Z",
"finished_at": "2022-10-24T18:21:38.464Z",
"duration": 12.478682,
"queued_duration": 0.499786,
"user": {
"id": 123456789,
[...]
},
"commit": {
"id": "5e0e1f287d20daf2036a3ca71c656dce55999265",
[...]
"pipeline": {
"id": 123456789,
[...]
"project": {
"ci_job_token_scope_enabled": false
},
"artifacts": [],
"runner": {
"id": 12270859,
[...]
},
"artifacts_expire_at": null,
"tag_list": []
}, {
"id": 3219589878,
"status": "success",
"stage": "test",
"name": "create_artifact_job_2",
"ref": "main",
"tag": false,
"coverage": null,
"allow_failure": false,
"created_at": "2022-10-24T18:21:25.111Z",
"started_at": "2022-10-24T18:21:25.922Z",
"finished_at": "2022-10-24T18:21:39.090Z",
"duration": 13.168405,
"queued_duration": 0.464364,
"user": {
"id": 123456789,
[...]
},
"commit": {
"id": "5e0e1f287d20daf2036a3ca71c656dce55999265",
[...]
},
"pipeline": {
"id": 675641982,
[...],
"project": {
"ci_job_token_scope_enabled": false
},
"artifacts_file": {
"filename": "artifacts.zip",
"size": 223
},
"artifacts": [{
"file_type": "archive",
"size": 223,
"filename": "artifacts.zip",
"file_format": "zip"
}, {
"file_type": "metadata",
"size": 153,
"filename": "metadata.gz",
"file_format": "gzip"
}],
"runner": {
"id": 12270845,
[...]
},
"artifacts_expire_at": "2022-10-25T18:21:35.859Z",
"tag_list": []
}, {
"id": 3219589876,
"status": "success",
"stage": "test",
"name": "create_artifact_job_1",
"ref": "main",
"tag": false,
"coverage": null,
"allow_failure": false,
"created_at": "2022-10-24T18:21:25.103Z",
"started_at": "2022-10-24T18:21:25.503Z",
"finished_at": "2022-10-24T18:21:41.407Z",
"duration": 15.904028,
"queued_duration": 0.098837,
"user": {
"id": 123456789,
[...]
},
"commit": {
"id": "5e0e1f287d20daf2036a3ca71c656dce55999265",
[...]
},
"pipeline": {
"id": 123456789,
[...]
},
"web_url": "WEB_URL",
"project": {
"ci_job_token_scope_enabled": false
},
"artifacts_file": {
"filename": "artifacts.zip",
"size": 217
},
"artifacts": [{
"file_type": "archive",
"size": 217,
"filename": "artifacts.zip",
"file_format": "zip"
}, {
"file_type": "metadata",
"size": 152,
"filename": "metadata.gz",
"file_format": "gzip"
}],
"runner": {
"id": 12270857,
},
"artifacts_expire_at": "2022-10-25T18:21:37.808Z",
"tag_list": []
}]
I've been trying to do either of the following using jQ:
Either:
Check if artifacts_file key exists in each iteration and if it does return the (job) id (so .[].id)
Check if artifacts array is empty in each iteration and if it is empty return the (job) id.
In both cases I'm able to do the first part but I am not sure how to return the .id key.
Related stackoverflow questions that I've been trying to utilize and adapt to my case:
jq - return array value if its length is not null
How to check for presence of 'key' in jq before iterating over the values
What I have so far: jq '[.[].artifacts[]|select(length > 0)] | .[]' which returns all the artifacts found (but it doesn't contain the .id of the job).
Checking the existence of a field using has:
.[] | select(has("artifacts_file")).id
3219589878
3219589876
Demo
Checking if a field is an empty array by comparing it to []:
.[] | select(.artifacts == []).id
3219589880
Demo

Convert nested json to multiple dataframes - dynamically

i have a question:
Is it possible read a json file and convert to dataframe dynamically?
My example is the next code:
Having this json file, i need 3 table dataframes:
{
"date_time": "01-03-2022, 15:18:32",
"regions": {
"Home Region": "Madrid",
"Primary Region": "Barcelona",
"Secondary Region": "Rio"
},
"customers": [
{
"name": "campo santo",
"address": "rua trebal 1",
"phone": 987456321,
"parking": true
},
{
"name": "santo da silva",
"address": "rua sama 6",
"phone": 654321987,
"parking": false
},
{
"name": "roger campos",
"address": "av casal 10",
"phone": 684426654,
"parking": true
}
],
"office": [
{
"location": "madrid",
"co_working_spaces": 25,
"kitchen": false,
"food_track": 2,
"internal_staff": [
{
"id": 123,
"name": "pablo"
},
{
"id": 874,
"name": "saul"
},
{
"id": 741,
"name": "maria"
}
]
},
{
"location": "rio",
"co_working_spaces": 55,
"kitchen": true,
"food_track": 4,
"internal_staff": [
{
"id": 784,
"name": "raquel"
},
{
"id": 874,
"name": "pedro"
},
{
"id": 145,
"name": "maria"
},
{
"id": 365,
"name": "rocio"
}
]
},
{
"location": "barcelona",
"co_working_spaces": 5,
"kitchen": false,
"food_track": 1,
"internal_staff": [
]
},
{
"location": "la",
"co_working_spaces": 5,
"kitchen": true,
"food_track": 4,
"internal_staff": [
{
"id": 852,
"name": "maria"
},
{
"id": 748,
"name": "sara"
}
]
}
]
}
this is my python code:
import pandas as pd
# from pandas.io.json import json_normalize
import json
with open('offices.json') as f:
dt = json.load(f)
# df = pd.json_normalize(dt)
df1 = pd.json_normalize(dt, 'customers', 'date_time')[['name', 'address', 'phone', 'parking', 'date_time']]
print(df1)
df2 = pd.json_normalize(dt, 'office', 'date_time')[['location', 'co_working_spaces', 'kitchen', 'food_track']]
print(df2)
df3 = pd.json_normalize(dt['office'], 'internal_staff', 'location')
print(df3)
With this code, i got my 3 table dataframes. But i have to know the json structure to create the dataframes.
So is it possible to do it dynamically ?
Regards

Nested json - store values in csv

I am trying to convert a nested json file into csv. It's data from a darts API and the structure is always the same. Nevertheless I got some problems flattening and storing the values in a csv because of the nested structure.
json:
{
"summaries": [{
"sport_event": {
"id": "sr:sport_event:12967512",
"start_time": "2017-11-11T13:15:00+00:00",
"start_time_confirmed": true,
"sport_event_context": {
"sport": {
"id": "sr:sport:22",
"name": "Darts"
},
"category": {
"id": "sr:category:104",
"name": "International"
},
"competition": {
"id": "sr:competition:597",
"name": "Grand Slam of Darts"
},
"season": {
"id": "sr:season:47332",
"name": "Grand Slam of Darts 2017",
"start_date": "2017-11-11",
"end_date": "2017-11-20",
"year": "2017",
"competition_id": "sr:competition:597"
},
"stage": {
"order": 1,
"type": "league",
"phase": "stage_1",
"start_date": "2017-11-11",
"end_date": "2017-11-15",
"year": "2017"
},
"round": {
"number": 1
},
"groups": [{
"id": "sr:league:29766",
"name": "Grand Slam of Darts 2017, Group G",
"group_name": "G"
}]
},
"coverage": {
"live": true
},
"competitors": [{
"id": "sr:competitor:35936",
"name": "Smith, Michael",
"abbreviation": "SMI",
"qualifier": "home"
}, {
"id": "sr:competitor:83895",
"name": "Wilson, James",
"abbreviation": "WIL",
"qualifier": "away"
}]
},
"sport_event_status": {
"status": "closed",
"match_status": "ended",
"home_score": 5,
"away_score": 3,
"winner_id": "sr:competitor:35936"
}
}, {
"sport_event": {
"id": "sr:sport_event:12967508",
"start_time": "2017-11-11T13:40:00+00:00",
"start_time_confirmed": true,
"sport_event_context": {
"sport": {
"id": "sr:sport:22",
"name": "Darts"
},
"category": {
"id": "sr:category:104",
"name": "International"
},
"competition": {
"id": "sr:competition:597",
"name": "Grand Slam of Darts"
},
"season": {
"id": "sr:season:47332",
"name": "Grand Slam of Darts 2017",
"start_date": "2017-11-11",
"end_date": "2017-11-20",
"year": "2017",
"competition_id": "sr:competition:597"
},
"stage": {
"order": 1,
"type": "league",
"phase": "stage_1",
"start_date": "2017-11-11",
"end_date": "2017-11-15",
"year": "2017"
},
"round": {
"number": 1
},
"groups": [{
"id": "sr:league:29764",
"name": "Grand Slam of Darts 2017, Group F",
"group_name": "F"
}]
},
"coverage": {
"live": true
},
"competitors": [{
"id": "sr:competitor:70916",
"name": "Bunting, Stephen",
"abbreviation": "BUN",
"qualifier": "home"
}, {
"id": "sr:competitor:191262",
"name": "de Zwaan, Jeffrey",
"abbreviation": "DEZ",
"qualifier": "away"
}]
},
"sport_event_status": {
"status": "closed",
"match_status": "ended",
"home_score": 5,
"away_score": 4,
"winner_id": "sr:competitor:70916"
}
}
So for each sport_event I would like to store the variables:
"start_time"
from "season" the variable "name"
from "competitors" both "id" and "name"
from "sport_event_status" the "winner_id"
I have already tried to flatten the json file with this code:
import json
f = open(r'path of file.json')
data = json.load(f)
def flatten(data):
for key,value in data.items():
print (str(key)+'->'+str(value))
if type(value) == type(dict()):
flatten(value)
elif type(value) == type(list()):
for val in value:
if type(val) == type(str()):
pass
elif type(val) == type(list()):
pass
else:
flatten(val)
flatten(data)
print(data)
This actually prints out the following:
id->sr:season:47332
name->Grand Slam of Darts 2017
start_date->2017-11-11
end_date->2017-11-20
year->2017
competition_id->sr:competition:597
Now my question is how to store the values I mentioned above in a csv file.
Thanks in advance for your support.
Using jq, you basically just have to transcribe your specification, adding a bit of context and taking care of an embedded array:
.summaries[]
| .sport_event # Your specification:
| [.start_time, # start_time
.sport_event_context.season.name] # from "season" the variable "name"
+ [.competitors[] | .id, .name] # from "competitors" both "id" and "name"
+ [.sport_event_status.winner_id] # from "sport_event_status" the "winner_id"
| #csv
Invocation
E.g.
jq -rf program.jq my.json

How to set alias instead of id , i need category_id without using forloop

$categoryData = $this->compCatObj->find()->select(['CompanyCategory.id', 'CompanyCategory.name','CompanyCategory.restricted'])->contain(['CompanyItems'=>['fields'=>['CompanyItems.id','CompanyItems.company_category_id','CompanyItems.name']]])->toArray();
Response
{
"status": "success",
"message": "List of company categories",
"data": [
{
"id": 1,
"name": "Breakfast gdfgedfgdf",
"restricted": "no",
"company_items": []
},
{
"id": 2,
"name": "Breakfast",
"restricted": "yes",
"company_items": []
}
]
}
i need category_id instead of id. Is there any way to do this without using forloop.
yes you can do this by a small change in your code,
$categoryData = $this->compCatObj->find()->select(['category_id' => 'CompanyCategory.id', 'CompanyCategory.name','CompanyCategory.restricted'])->contain(['CompanyItems'=>['fields'=>['CompanyItems.id','CompanyItems.company_category_id','CompanyItems.name']]])->toArray();
Response
{
"status": "success",
"message": "List of company categories",
"data": [
{
"category_id": 1,
"name": "Breakfast gdfgedfgdf",
"restricted": "no",
"company_items": []
},
{
"category_id": 2,
"name": "Breakfast",
"restricted": "yes",
"company_items": []
}
]
}

How to exclude specific fields from JSON using groovy

I would like to exclude the items which don't have productModel property in the below JSON. How can we achieve this in groovy
I tried using hasProperty but not worked for me as expected. If possible can I get some sample snippet
I tried below code - but didn't work as I expected.
response.getAt('myData').getAt('data').getAt('product').hasProperty('productModel').each { println "result ${it}" }
Any help would be really appreciated.
{
"myData": [{
"data": {
"product": {
"productId": "apple",
"productName": "iPhone",
"productModel": "6s"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
},
{
"data": {
"product": {
"productId": "apple",
"productName": "iPhone",
"productModel": "7"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
},
{
"data": {
"product": {
"productId": "apple",
"productName": "Macbook"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
}
],
"metadata": {
"count": 3,
"offset": 0
}
}
If you want to exclude specific fields from JSON object then you have to recreate it using filtered data. The crucial part takes these two lines (assuming that json variable in the below example stores your JSON as text):
def root = new JsonSlurper().parseText(json)
def myData = root.myData.findAll { it.data.product.containsKey('productModel') }
What happens here is we access root.myData list and we filter it using findAll(predicate) method and predicate in this case says that only objects that have key productModel in path data.product are accepted. This findAll() method does not mutate existing list and that is why we store the result in variable myData - after running this method we will end up with a list of size 2.
In next step you have to recreate the object you want to represent as a JSON:
def newJsonObject = [
myData: myData,
metadata: [
count: myData.size(),
offset: 0
]
]
println JsonOutput.prettyPrint(JsonOutput.toJson(newJsonObject))
In this part we create newJsonObject and in the end we convert it to a JSON representation.
Here is the full example:
import groovy.json.JsonOutput
import groovy.json.JsonSlurper
def json = '''{
"myData": [{
"data": {
"product": {
"productId": "apple",
"productName": "iPhone",
"productModel": "6s"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
},
{
"data": {
"product": {
"productId": "apple",
"productName": "iPhone",
"productModel": "7"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
},
{
"data": {
"product": {
"productId": "apple",
"productName": "Macbook"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
}
],
"metadata": {
"count": 3,
"offset": 0
}
}'''
def root = new JsonSlurper().parseText(json)
def myData = root.myData.findAll { it.data.product.containsKey('productModel') }
def newJsonObject = [
myData: myData,
metadata: [
count: myData.size(),
offset: 0
]
]
println JsonOutput.prettyPrint(JsonOutput.toJson(newJsonObject))
And here is the output it produces:
{
"myData": [
{
"data": {
"product": {
"productId": "apple",
"productName": "iPhone",
"productModel": "6s"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [
{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {
}
},
{
"data": {
"product": {
"productId": "apple",
"productName": "iPhone",
"productModel": "7"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [
{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {
}
}
],
"metadata": {
"count": 2,
"offset": 0
}
}