removing an entry from JSON using powershell - json

I am using GCP's BigQuery. I want to remove access of someone. Per https://cloud.google.com/bigquery/docs/dataset-access-controls#bq_1, I run the command
bq show \
--format=prettyjson \
project1:dataset1 > /tmp/myjson.json
and here is the JSON I get
{
"access": [
{
"role": "WRITER",
"specialGroup": "projectWriters"
},
{
"role": "OWNER",
"specialGroup": "projectOwners"
},
{
"role": "OWNER",
"userByEmail": "employee1#myco.com"
},
{
"role": "READER",
"specialGroup": "projectReaders"
}
],
"creationTime": "1528762487037",
"datasetReference": {
"datasetId": "dataset1",
"projectId": "project1"
},
"description": "My Data",
"etag": "<redacted>",
"friendlyName": "dataset1",
"id": "project1:dataset1",
"kind": "bigquery#dataset",
"lastModifiedTime": "1528762487037",
"location": "US",
"selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/project1/datasets/dataset1"
}
I an new to powershell and am trying to remove the entry for employee1, and put the json back for the
bq update
command per the link above.
Any ideas how to get the index of the entry.
I have tried to just get the "access" part of the json as an Array object. But the challenge is to find the employee1 entry index and remove that Array member and put it back into the json.
Any help is appreciated.

Using the convertfrom-json cmdlet, you get an object you can edit. Then you can convert it back using convertto-json
$x=#'
{
"access": [
{
"role": "WRITER",
"specialGroup": "projectWriters"
},
{
"role": "OWNER",
"specialGroup": "projectOwners"
},
{
"role": "OWNER",
"userByEmail": "employee1#myco.com"
},
{
"role": "READER",
"specialGroup": "projectReaders"
}
],
"creationTime": "1528762487037",
"datasetReference": {
"datasetId": "dataset1",
"projectId": "project1"
},
"description": "My Data",
"etag": "<redacted>",
"friendlyName": "dataset1",
"id": "project1:dataset1",
"kind": "bigquery#dataset",
"lastModifiedTime": "1528762487037",
"location": "US",
"selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/project1/datasets/dataset1"
}
'# | ConvertFrom-json
$x.Access=$x.Access.Where({$_.userByEmail -ne 'employee1#myco.com'})
$x | convertto-json

Related

Combining the bookmark JSON files from Chome and Edge Chromium into one file using PowerShell?

We are migrating from Chrome to Edge Chromium (and when Microsoft puts the final nail into the IE 11 coffin from IE 11 as well). This is being done when PCs are being replaced.
We are using USMT to copy users files over. So the old computer may or may not have Chrome, but if it does, I'd like to copy the bookmark files. If it has Edge installed too (most do), I'd like to copy those bookmarks too. When restoring the USMT data, I'd like to add a step to our restore script that merges the results of both of the bookmark files into one JSON file and import it into Edge Chromium (I think I can just plop the "Bookmarks" file into "C:\Users$UserID\AppData\Local\Microsoft\Edge\User Data\Default").
I have these three functions that get called in other parts of the script:
Function Backup-ChromeBookmarks {
$pathToChromeJsonFile = -join("C:\Users\", $UserID, "\AppData\local\Google\Chrome\User Data\Default\Bookmarks")
$global:chromeBookmarkExists = Test-Path $pathToChromeJsonFile
if ($chromeBookmarkExists -eq $true){
Copy-Item -Path $pathToChromeJsonFile -Destination $BackupStore/Bookmarks-chrome.json
}
}
Function Backup-EdgeBookmarks {
$pathToEdgeJsonFile = -join("C:\Users\", $UserID, "\AppData\Local\Microsoft\Edge\User Data\Default\Bookmarks")
$global:edgeBookmarkExists = Test-Path $pathToEdgeJsonFile
if ($edgeBookmarkExists -eq $true){
Copy-Item -Path $pathToEdgeJsonFile -Destination $BackupStore/Bookmarks-edge.json
}
}
Function Combine-Bookmarks {
# referenced https://www.jonathanmedd.net/2020/04/combine-two-json-files-with-powershell.html
#https://web.archive.org/web/20200517182000/https://www.jonathanmedd.net/2020/04/combine-two-json-files-with-powershell.html
if($chromeBookmarkExists -eq $true -And $edgeBookmarkExists -eq $true){
$data1 = Get-Content $BackupStore/Bookmarks-chrome.json -Raw | ConvertFrom-Json
$data2 = Get-Content $BackupStore/Bookmarks-edge.json -Raw | ConvertFrom-Json
$data1.psobject.Properties | ForEach-Object {
$data2 | Add-Member -MemberType $_.MemberType -Name $_.Name -Value $_.Value -Force
}
#($data2) | ConvertTo-Json | Out-File $BackupStore/Bookmarks
}
}
It works to copy the "Bookmarks" files from Chrome and Edge, but when it combines them and writes the file, this is all the contents are:
"checksum": "0f9bf8e97b9ac6cd3654c15c673b8cb8",
"roots": {
"bookmark_bar": {
"children": " ",
"date_added": "13252082948039886",
"date_modified": "13268156238031255",
"guid": "0bc5d13f-2cba-5d74-951f-3f233fe6c908",
"id": "1",
"name": "Bookmarks bar",
"type": "folder"
},
"other": {
"children": "",
"date_added": "13252082948040297",
"date_modified": "0",
"guid": "82b081ec-3dd3-529c-8475-ab6c344590dd",
"id": "2",
"name": "Other bookmarks",
"type": "folder"
},
"synced": {
"children": "",
"date_added": "13252082948040301",
"date_modified": "0",
"guid": "4cf2e351-0e85-532b-bb37-df045d8f8d0f",
"id": "3",
"name": "Mobile bookmarks",
"type": "folder"
}
},
"version": 1
}
Does anyone have any idea what I am doing wrong with merging the two JSON files? I don't have a lot of experience with this and would appreciate any feedback.
Please let me know if I'm missing any info. Thanks!
Here are some random bookmark lists from Chrome and Edge for sample data:
Chrome
"checksum": "0e3450f30154cec188275de0e1eed2a5",
"roots": {
"bookmark_bar": {
"children": [ {
"date_added": "13282342589274138",
"guid": "dff3def7-9693-443c-a44c-2b1047687b33",
"id": "5",
"name": "Google News",
"type": "url",
"url": "https://news.google.com/topstories?hl=en-CA&gl=CA&ceid=CA:en"
}, {
"children": [ {
"date_added": "13282342602339055",
"guid": "3a6114ed-d13a-491e-a049-d47a174a349b",
"id": "7",
"name": "reddit - Google Search",
"type": "url",
"url": "https://www.google.com/search?q=reddit&oq=reddit&aqs=chrome..69i57j46i131i199i433i465i512j0i131i433i512l3j0i131i433j0i131i433i512l3.1205j0j7&sourceid=chrome&ie=UTF-8"
}, {
"date_added": "13282342613505854",
"guid": "0da478c1-a91b-4af9-80db-cae8dc6927c0",
"id": "8",
"name": "Reddit - Dive into anything",
"type": "url",
"url": "https://www.reddit.com/"
}, {
"date_added": "13282342620306591",
"guid": "b0311e2e-a41f-47fa-a1ef-214dd3964729",
"id": "9",
"name": "Facebook - Log In or Sign Up",
"type": "url",
"url": "https://www.facebook.com/"
}, {
"date_added": "13282342631549806",
"guid": "a06637af-50be-458e-95bc-730ffc1cda36",
"id": "10",
"name": "Instagram",
"type": "url",
"url": "https://www.instagram.com/?hl=en"
} ],
"date_added": "13282342597429656",
"date_modified": "13282342644382518",
"guid": "a5b18126-11f7-40df-809f-2d96f311f1f6",
"id": "6",
"name": "Test Folder",
"type": "folder"
}, {
"date_added": "13282342644382518",
"guid": "39ef0ac5-b70c-406e-acea-96c3f90935ef",
"id": "11",
"name": "Dogpile.com",
"type": "url",
"url": "https://www.dogpile.com/"
}, {
"date_added": "13282342655028948",
"guid": "d152f027-2aae-47b1-bae6-f9d7e617b639",
"id": "12",
"name": "Ask.com - What's Your Question?",
"type": "url",
"url": "https://www.ask.com/"
} ],
"date_added": "13282342511050735",
"date_modified": "13282342655028948",
"guid": "0bc5d13f-2cba-5d74-951f-3f233fe6c908",
"id": "1",
"name": "Bookmarks bar",
"type": "folder"
},
"other": {
"children": [ ],
"date_added": "13282342511050739",
"date_modified": "0",
"guid": "82b081ec-3dd3-529c-8475-ab6c344590dd",
"id": "2",
"name": "Other bookmarks",
"type": "folder"
},
"synced": {
"children": [ ],
"date_added": "13282342511050740",
"date_modified": "0",
"guid": "4cf2e351-0e85-532b-bb37-df045d8f8d0f",
"id": "3",
"name": "Mobile bookmarks",
"type": "folder"
}
},
"version": 1
}
Edge
{
"checksum": "c26640ae81258f8bb73935ccd2bb1e91",
"roots": {
"bookmark_bar": {
"children": [ {
"date_added": "13282342518391441",
"guid": "00f514ec-8d15-45b1-9c26-ba258d59c688",
"id": "9",
"name": "Google News",
"show_icon": false,
"source": "user_add",
"type": "url",
"url": "https://news.google.com/topstories?hl=en-CA&gl=CA&ceid=CA:en"
}, {
"date_added": "13282342527623442",
"guid": "f690f411-6f29-4e22-bee4-d57b225a9287",
"id": "10",
"name": "Stack Overflow - Where Developers Learn, Share, & Build Careers",
"show_icon": false,
"source": "user_add",
"type": "url",
"url": "https://stackoverflow.com/"
}, {
"children": [ {
"date_added": "13282342544239807",
"guid": "bf6d87ae-9cd1-4d02-b2de-5083e9b487f1",
"id": "12",
"name": "Amazon.ca: Low Prices – Fast Shipping – Millions of Items",
"show_icon": false,
"source": "user_add",
"type": "url",
"url": "https://www.amazon.ca/"
}, {
"date_added": "13282342551951302",
"guid": "d504da5d-4217-4a79-998c-f3dd5b0c95e0",
"id": "13",
"name": "Best Buy: Shop Online For Deals & Save | Best Buy Canada",
"show_icon": false,
"source": "user_add",
"type": "url",
"url": "https://www.bestbuy.ca/en-ca"
}, {
"date_added": "13282342564797804",
"guid": "cad2d5d7-c72e-477e-854a-d58df397574c",
"id": "15",
"name": "Laptops, Desktops, Tablets, Computer Components, Printers, TVs, Video Games & Appliances - Canada Computers & Electronics",
"show_icon": false,
"source": "user_add",
"type": "url",
"url": "https://www.canadacomputers.com/"
} ],
"date_added": "13282342538649786",
"date_modified": "13282342564797804",
"guid": "3616a33e-7bf6-47de-8142-673cc19f6d64",
"id": "11",
"name": "Shopping",
"source": "unknown",
"type": "folder"
} ],
"date_added": "13282342417969383",
"date_modified": "13282342544239807",
"guid": "0bc5d13f-2cba-5d74-951f-3f233fe6c908",
"id": "1",
"name": "Favorites bar",
"source": "unknown",
"type": "folder"
},
"other": {
"children": [ ],
"date_added": "13282342417969428",
"date_modified": "0",
"guid": "82b081ec-3dd3-529c-8475-ab6c344590dd",
"id": "2",
"name": "Other favorites",
"source": "unknown",
"type": "folder"
},
"synced": {
"children": [ ],
"date_added": "13282342417969430",
"date_modified": "0",
"guid": "4cf2e351-0e85-532b-bb37-df045d8f8d0f",
"id": "3",
"name": "Mobile favorites",
"source": "unknown",
"type": "folder"
}
},
"version": 1
}
As #zetta42 points out, it is likely that you just need to add more depth:
#($data2) | ConvertTo-Json -Depth 10 | Out-File $BackupStore/Bookmarks
as well as overwriting the roots property (not actually merging).
However, from a user experience you may want to consider another update. Right now if the two contain bookmarks of the same name, your intended merge logic will overwrite one with the other. A user (you) may be a whole lot happier knowing which browser a given bookmark came from and can easily re-organize from within the new browser (or, since I am not familiar with it, hopefully it does) if not desired. So I would suggest grabbing the the two bookmarks and putting them under their own folders in the new bookmarks. Something like the following:
Function Combine-Bookmarks {
# referenced https://www.jonathanmedd.net/2020/04/combine-two-json-files-with-powershell.html
#https://web.archive.org/web/20200517182000/https://www.jonathanmedd.net/2020/04/combine-two-json-files-with-powershell.html
if($chromeBookmarkExists -eq $true -And $edgeBookmarkExists -eq $true){
$data1 = Get-Content $BackupStore/Bookmarks-chrome.json -Raw | ConvertFrom-Json
$data2 = Get-Content $BackupStore/Bookmarks-edge.json -Raw | ConvertFrom-Json
$dataNew = #{
'roots'=#{'Chrome'=$data1.Roots; 'IE'=$data2.Roots}
#whatever the structure of the new json file should be
}
$dataNew | ConvertTo-Json -Depth 10 | Out-File $BackupStore/Bookmarks_new.json
}
}
Also you can either leave a folder blank if they don't have that browser, or you can exclude it from the $dataNew (by using a bit of different syntax I don't recall). Right now it will only attempt a merge if both exist.

Parsing Git Json with Regular Express

I am taking a Github json file and parsing it with Java's regular expression library JsonPath. I am having a problem parsing arrays that do not have labels.
I need to send a email every time a particular file is changed in our repository.
Here is the Git Json:
{
"trigger": "push",
"payload": {
"type": "GitPush",
"before": "xxxxxxxx",
"after": "yyyyyyyy",
"branch": "branch-name",
"ref": "refs/heads/branch-name",
"repository": {
"id": 42,
"name": "repo",
"title": "repo",
"type": "GitRepository"
},
"beanstalk_user": {
"type": "Owner",
"id": 42,
"login": "username",
"email": "user#example.org",
"name": "Name Surname"
},
"commits": [
{
"type": "GitCommit",
"id": "ffffffff",
"message": "Important changes.",
"branch": "branch-name",
"author": {
"name": "Name Surname",
"email": "user#example.org"
},
"beanstalk_user": {
"type": "Owner",
"id": 42,
"login": "username",
"email": "user#example.org",
"name": "Name Surname"
},
"changed_files": {
"added": [
"NEWFILE",
],
"deleted": [
"Gemfile",
"NEWFILE"
],
"modified": [
"README.md",
"NEWFILE"
],
"copied": [
]
},
"changeset_url": "https://subdomain.github.com/repository-name/changesets/ffffffff",
"committed_at": "2014/08/18 13:30:29 +0000",
"parents": [
"afafafaf"
]
}
]
}
}
This is the expression I am using: to get the commits
$..changed_files
This return the whole changed files part but I can not explicitly choose the name "NEWFILE"
I tried
$..changed_files.*[?(#.added == "NEWFILE")]
$..changed_files.*[?(#.*== "NEWFILE")]
It just returns a empty array.
I just want it to return Newfile and what type of change. Any Ideas?
You can use the following JsonPath to retrieve the commits which list "NEWFILE" as an added file :
$.payload.commits[?(#.changed_files.added.indexOf("NEWFILE") != -1)]

Use of ResultTemplate in vsts extension tasks

I created a task that talks to a REST API to retreive the values for 2 picklists.
Filling the first dropdown box works fine, when using just the jsonpath.
Based on the first picklist I'd like to retreive values of the second list.
I've tried some variations and I'm trying something like this:
The json which I receive in the first rest call is similar to:
{
"id": "45",
"href": "https://selfservice/api/configurations/45/",
"name": "Type",
"description": "",
"version": "0.0.4",
"attributes": [
{
"value": {
"sdk-object": {
"type": "ConfigurationElement",
"id": "56"
}
},
"type": "ConfigurationElement",
"name": "win"
},
{
"value": {
"sdk-object": {
"type": "ConfigurationElement",
"id": "57"
}
},
"type": "ConfigurationElement",
"name": "lin"
}
]
}
I try to show the attributes name in the list and need the id of the attribute in the second picklist.
I created the following datasourcebindings in the task.json. Of course, the targets exist in the task.
task.json:
{
"id": "GUID",
"name": "Spinup",
"friendlyName": "Create environment",
"description": "Starts Workflow to create an environment. ___ The following placeholders are created deplending on the template and can be used in the rest of the release:**XLDEnvironment** and **testAgentHostname**",
"category": "Deploy",
"author": "***",
"version": {
"Major": 0,
"Minor": 0,
"Patch": 33
},
"minimumAgentVersion": "1.95.3",
"inputs": [
{
"label": "Endpoint",
"name": "connectedServiceName",
"required": true,
"type": "connectedService:server",
"helpMarkDown": "endpoint to connect to."
},
{
"name": "stage",
"type": "string",
"label": "Stage",
"defaultValue": "$(Release.EnvironmentName)",
"required": true,
"helpMarkDown": "Stage of the release, default value is based on the pipeline."
},
{
"name": "releaseName",
"type": "string",
"label": "Environment name",
"defaultValue": "$(Release.ReleaseName)",
"required": true,
"helpMarkDown": "Name of the environment that will be created."
},
{
"name": "owner",
"type": "string",
"label": "Owner of the environment",
"defaultValue": "***",
"required": true,
"helpMarkDown": "It is common to use the initials of the product owner in this field. There has to be a valid AD user owner of each environment. This can also be set by using a variable through the pipeline."
},
{
"name": "group",
"type": "string",
"label": "Group of the owner",
"defaultValue": "",
"required": true,
"helpMarkDown": "group that owns the environment. groups can be requested by the infrastructure department. Example: ****"
},
{
"name": "platform",
"type": "pickList",
"label": "Platform (OS)",
"defaultValue": "",
"required": true,
"helpMarkDown": "Choose the type of the target platform."
},
{
"name": "size",
"type": "pickList",
"label": "Environment Template",
"defaultValue": "",
"required": true,
"helpMarkDown": "Size of the environment to create."
}
],
"dataSourceBindings": [
{
"dataSourceName": "GetCEPlatformType",
"endpointId": "$(connectedServiceName)",
"target": "platform",
"selector": "jsonpath:$.attributes[*].name",
"keySelector": "jsonpath:$.attributes[*].value.sdk-object.id"
},
{
"dataSourceName": "GetCEPlatformSize",
"endpointId": "$(connectedServiceName)",
"target": "size",
"parameters": {
"platformTypeId": "$(platform)"
}
}
],
"instanceNameFormat": "Spin up $(size) $(platform) environment",
"execution": {
"PowerShell3": {
"target": "$(currentDirectory)\\task.ps1",
"argumentFormat": "",
"workingDirectory": "$(currentDirectory)"
}
}
}
vss-extension.json:
{
"manifestVersion": 1,
"id": "*****",
"name": "Release Tasks",
"version": "1.0.1",
"publisher": "***",
"targets": [
{
"id": "Microsoft.VisualStudio.Services"
}
],
"description": "Tools to contact ***. Includes a task to spin up a platform and a task to remove the platform.",
"categories": [
"Build and release"
],
"icons": {
"default": "images/extension-icon.png"
},
"files": [
{
"path": "RemoveEnvironment"
},
{
"path": "SpinUpEnvironment"
}
],
"contributions": [
{
"id": "******",
"type": "ms.vss-distributed-task.task",
"targets": [
"ms.vss-distributed-task.tasks"
],
"properties": {
"name": "RemoveEnvironment"
}
},
{
"id": "*********",
"type": "ms.vss-distributed-task.task",
"targets": [
"ms.vss-distributed-task.tasks"
],
"properties": {
"name": "SpinUpEnvironment"
}
},
{
"description": "Service Endpoint type for all connections",
"id": "endpoint-type",
"properties": {
"authenticationSchemes": [
{
"inputDescriptors": [
{
"description": "Username",
"id": "username",
"inputMode": "textbox",
"name": "Username",
"validation": {
"dataType": "string",
"isRequired": true
}
},
{
"description": "Password",
"id": "password",
"inputMode": "passwordbox",
"isConfidential": true,
"name": "Password",
"validation": {
"dataType": "string",
"isRequired": false
}
}
],
"type": "ms.vss-endpoint.endpoint-auth-scheme-basic"
}
],
"dataSources": [
{
"endpointUrl": "api/configurations/*****/",
"name": "GetCEPlatformType",
"resultSelector": "jsonpath:$.attributes[*].name"
},
{
"endpointUrl": "api/configurations/$(platformTypeId)/",
"name": "GetCEPlatformSize",
"resultSelector": "jsonpath:$.attributes[*].name"
}
],
"displayName": "*****",
"helpMarkDown": "Enter the url and credentials to connect to the endpoint.",
"name": "server",
"url": {
"displayName": "Server URL",
"helpText": "Url for the server to connect to."
}
},
"targets": [
"ms.vss-endpoint.endpoint-types"
],
"type": "ms.vss-endpoint.service-endpoint-type"
}
]
}
Sadly I can't get this to work. The datasources exist in the endpoint, but I'm not sure what to do with the resultSelector.
Does anyone have an idea on how to get this to work? The documentation on this isn't too good.
I'm not familiar with mustache, but in the mustache test tool this seems to work.
Thoughts are appreciated!
Try with this in the first data source:
{
"dataSourceName": "dsList1",
"endpointId": "$(connectedServiceName)",
"target": "list1",
"selector": "jsonpath:$.attributes[*].name",
"keySelector": "jsonpath:$.attributes[*].value.sdk-object.id"
}
Update:
You were trying to use "platformTypeId" in vss-extension.json file while it was defined in task.json, this cause the error message you mentioned in the comment.
By the way, I just noticed that you are using "dataSourceBindings", then you cannot use "selector" and "KeySelector" to parse the result since they are used for "sourceDefinitions".
To achieve the feature you want with "dataSourceBindings", you can definition the endpoint url in the task.json directly instead of defining datasource in endpoint contribution.
So you can move the datasources section in the vss-extension.json file first, and then in the task.json file, change to following:
"dataSourceBindings": [
{
"endpointId": "$(connectedServiceName)",
"endpointURL": "api/configurations/*****/",
"target": "platform",
"resultselector": "jsonpath:$.attributes[*]",
"resultTemplate": "{ \"Value\" : \"{{{value.id}}}\", \"DisplayValue\" : \"{{{name}}}\" }"
},
{
"endpointId": "$(connectedServiceName)",
"endpointURL": "api/configurations/$(platformTypeId)/",
"target": "size"
}
],

Parsing JSON with arrays and structures with Coldfusion

I need to output the JSON result in HTML so that I can use it in Coldfusion. I am getting the data but I do not know how to parse the returned data so that I can use it. I am a newby so be easy on me. I don't know JSON at all and this is the first time I am trying to work work this.
<cfset qpxFields = {
"request": {
"passengers": {
"adultCount": "1"
},
"slice": [
{
"origin": "JNB",
"destination": "MRU",
"date": "2016-12-19"
}
],
"solutions": "1"
}
}/>
<cfhttp url="https://www.googleapis.com/qpxExpress/v1/trips/search?key=AIzaSyB0NN9WYmrkc2Ikq9TVGzzAD_gGoSBSbP4" method="post" result="httpResp" timeout="60">
<cfhttpparam type="header" name="Content-Type" value="application/json" />
<cfhttpparam type="body" value="#serializeJSON(qpxFields)#">
</cfhttp>
<cfset getResult = deserializeJSON(httpResp.filecontent)>
<cfoutput>#httpResp.filecontent#</cfoutput>
The result I am getting when outputting the filecontent is the following:
{ "kind": "qpxExpress#tripsSearch", "trips": { "kind": "qpxexpress#tripOptions", "requestId": "ylROS8N2kR4vI8ktC0PQVj", "data": { "kind": "qpxexpress#data", "airport": [ { "kind": "qpxexpress#airportData", "code": "JNB", "city": "JNB", "name": "Johannesburg O.R. Tambo International" }, { "kind": "qpxexpress#airportData", "code": "MRU", "city": "MRU", "name": "Mauritius Sir S. Ramgoolam International" } ], "city": [ { "kind": "qpxexpress#cityData", "code": "JNB", "name": "Johannesburg" }, { "kind": "qpxexpress#cityData", "code": "MRU", "name": "Mauritius" } ], "aircraft": [ { "kind": "qpxexpress#aircraftData", "code": "343", "name": "Airbus A340" } ], "tax": [ { "kind": "qpxexpress#taxData", "id": "EV_001", "name": "South Africa Passenger Safety Charge" }, { "kind": "qpxexpress#taxData", "id": "YQ_F", "name": "MK YQ surcharge" }, { "kind": "qpxexpress#taxData", "id": "YR_I", "name": "MK YR surcharge" }, { "kind": "qpxexpress#taxData", "id": "ZA_001", "name": "South Africa Passenger Service Charge" }, { "kind": "qpxexpress#taxData", "id": "UM_001", "name": "South Africa Asc Charge" }, { "kind": "qpxexpress#taxData", "id": "WC_001", "name": "South Africa Air Passenger Tax" } ], "carrier": [ { "kind": "qpxexpress#carrierData", "code": "MK", "name": "AIR MAURITIUS LTD" } ] }, "tripOption": [ { "kind": "qpxexpress#tripOption", "saleTotal": "ZAR5398.23", "id": "bGXAC5TFGRFO06xEFBzMaA001", "slice": [ { "kind": "qpxexpress#sliceInfo", "duration": 240, "segment": [ { "kind": "qpxexpress#segmentInfo", "duration": 240, "flight": { "carrier": "MK", "number": "852" }, "id": "GB3DFc4zHshnkF7H", "cabin": "COACH", "bookingCode": "T", "bookingCodeCount": 9, "marriedSegmentGroup": "0", "leg": [ { "kind": "qpxexpress#legInfo", "id": "LwsDBsjHUFZrYe5Y", "aircraft": "343", "arrivalTime": "2016-12-19T19:35+04:00", "departureTime": "2016-12-19T13:35+02:00", "origin": "JNB", "destination": "MRU", "originTerminal": "B", "duration": 240, "mileage": 1904, "meal": "Meal" } ] } ] } ], "pricing": [ { "kind": "qpxexpress#pricingInfo", "fare": [ { "kind": "qpxexpress#fareInfo", "id": "AdGb1tIdX7T9uIyT9qTU9OFHTXhaVL0C0ojKd7a7use6", "carrier": "MK", "origin": "JNB", "destination": "MRU", "basisCode": "TAANOWMU" } ], "segmentPricing": [ { "kind": "qpxexpress#segmentPricing", "fareId": "AdGb1tIdX7T9uIyT9qTU9OFHTXhaVL0C0ojKd7a7use6", "segmentId": "GB3DFc4zHshnkF7H" } ], "baseFareTotal": "ZAR3690.00", "saleFareTotal": "ZAR3690.00", "saleTaxTotal": "ZAR1708.23", "saleTotal": "ZAR5398.23", "passengers": { "kind": "qpxexpress#passengerCounts", "adultCount": 1 }, "tax": [ { "kind": "qpxexpress#taxInfo", "id": "ZA_001", "chargeType": "GOVERNMENT", "code": "ZA", "country": "ZA", "salePrice": "ZAR346.00" }, { "kind": "qpxexpress#taxInfo", "id": "EV_001", "chargeType": "GOVERNMENT", "code": "EV", "country": "ZA", "salePrice": "ZAR20.23" }, { "kind": "qpxexpress#taxInfo", "id": "UM_001", "chargeType": "GOVERNMENT", "code": "UM", "country": "ZA", "salePrice": "ZAR24.00" }, { "kind": "qpxexpress#taxInfo", "id": "WC_001", "chargeType": "GOVERNMENT", "code": "WC", "country": "ZA", "salePrice": "ZAR190.00" }, { "kind": "qpxexpress#taxInfo", "id": "YQ_F", "chargeType": "CARRIER_SURCHARGE", "code": "YQ", "salePrice": "ZAR1090.00" }, { "kind": "qpxexpress#taxInfo", "id": "YR_I", "chargeType": "CARRIER_SURCHARGE", "code": "YR", "salePrice": "ZAR38.00" } ], "fareCalculation": "JNB MK MRU 260.69TAANOWMU NUC 260.69 END ROE 14.15458 FARE ZAR 3690.00 XT 20.23EV 24.00UM 190.00WC 346.00ZA 1090.00YQ 38.00YR", "latestTicketingTime": "2016-12-19T06:34-05:00", "ptc": "ADT", "refundable": true } ] } ] } }
I am using CF 10.
Instead of trying to deal with the HTTP response (httpResp.filecontent) look at your deserialized content (getResult). You cannot simply output that variable as ColdFusion creates a structure from the JSON data.
Dump out your deserialzed JSON like this:
<cfdump var="#getResult#">
Looking at that output you will see how ColdFusion has already parsed all of the information for you. You are going to have some structure values and some array values. You will need to use appropriate methods (looping over an array for example) to extract each individual piece of data.
Reply to your comment
Looping in ColdFusion is well documented. To get you started you can look at the Adobe documentation itself - cfloop. The documentation on that page shows how to link over arrays, structures, etc.
Another good reference is Learn CF In a Week - looping.
Looping works when you have to traverse one-dimensional or two-dimensional structs or arrays. When the data combines structs and arrays in 3 or more dimensions, as in this particular problem, then looping is often insufficient as a solution. The complexity will certainly overwhelm most beginners.
A good solution is to play dumb. Start with the result, the structure getResult. Do a dump, as suggested by others earlier. Then simply run through it by eye and drill down to the values.
The list of values is:
getResult.kind
getResult.trips.data.aircraft[1].code
getResult.trips.data.aircraft[1].kind
getResult.trips.data.aircraft[1].name
getResult.trips.data.airport[1].city
getResult.trips.data.airport[1].code
getResult.trips.data.airport[1].kind
getResult.trips.data.airport[1].name
getResult.trips.data.airport[2].city
getResult.trips.data.airport[2].code
getResult.trips.data.airport[2].kind
getResult.trips.data.airport[2].name
getResult.trips.data.carrier[1].code
getResult.trips.data.carrier[1].kind
getResult.trips.data.carrier[1].name
getResult.trips.data.city[1].code
getResult.trips.data.city[1].kind
getResult.trips.data.city[1].name
getResult.trips.data.city[2].code
getResult.trips.data.city[2].kind
getResult.trips.data.city[2].name
getResult.trips.data.kind
getResult.trips.data.tax[1].id
getResult.trips.data.tax[1].kind
getResult.trips.data.tax[1].name
getResult.trips.data.tax[2].id
getResult.trips.data.tax[2].kind
getResult.trips.data.tax[2].name
getResult.trips.data.tax[3].id
getResult.trips.data.tax[3].kind
getResult.trips.data.tax[3].name
getResult.trips.data.tax[4].id
getResult.trips.data.tax[4].kind
getResult.trips.data.tax[4].name
getResult.trips.data.tax[5].id
getResult.trips.data.tax[5].kind
getResult.trips.data.tax[5].name
getResult.trips.data.tax[6].id
getResult.trips.data.tax[6].kind
getResult.trips.data.tax[6].name
getResult.trips.kind
getResult.trips.requestId
getResult.trips.tripOption[1].id
getResult.trips.tripOption[1].kind
getResult.trips.tripOption[1].saletotal
getResult.trips.tripOption[1].pricing[1].basefareTotal
getResult.trips.tripOption[1].pricing[1].fare[1].basiscode
getResult.trips.tripOption[1].pricing[1].fare[1].carrier
getResult.trips.tripOption[1].pricing[1].fare[1].destination
getResult.trips.tripOption[1].pricing[1].fare[1].id
getResult.trips.tripOption[1].pricing[1].fare[1].kind
getResult.trips.tripOption[1].pricing[1].fare[1].origin
getResult.trips.tripOption[1].pricing[1].fareCalculation
getResult.trips.tripOption[1].pricing[1].kind
getResult.trips.tripOption[1].pricing[1].latestTicketingTime
getResult.trips.tripOption[1].pricing[1].passengers.adultCount
getResult.trips.tripOption[1].pricing[1].passengers.kind
getResult.trips.tripOption[1].pricing[1].ptc
getResult.trips.tripOption[1].pricing[1].refundable
getResult.trips.tripOption[1].pricing[1].saleFareTotal
getResult.trips.tripOption[1].pricing[1].saletaxTotal
getResult.trips.tripOption[1].pricing[1].saleTotal
getResult.trips.tripOption[1].pricing[1].segmentPricing[1].fareId
getResult.trips.tripOption[1].pricing[1].segmentPricing[1].kind
getResult.trips.tripOption[1].pricing[1].segmentPricing[1].segmentId
getResult.trips.tripOption[1].pricing[1].tax[1].chargeType
getResult.trips.tripOption[1].pricing[1].tax[1].code
getResult.trips.tripOption[1].pricing[1].tax[1].country
getResult.trips.tripOption[1].pricing[1].tax[1].id
getResult.trips.tripOption[1].pricing[1].tax[1].kind
getResult.trips.tripOption[1].pricing[1].tax[1].salePrice
getResult.trips.tripOption[1].pricing[1].tax[2].chargeType
getResult.trips.tripOption[1].pricing[1].tax[2].code
getResult.trips.tripOption[1].pricing[1].tax[2].country
getResult.trips.tripOption[1].pricing[1].tax[2].id
getResult.trips.tripOption[1].pricing[1].tax[2].kind
getResult.trips.tripOption[1].pricing[1].tax[2].salePrice
getResult.trips.tripOption[1].pricing[1].tax[3].chargeType
getResult.trips.tripOption[1].pricing[1].tax[3].code
getResult.trips.tripOption[1].pricing[1].tax[3].country
getResult.trips.tripOption[1].pricing[1].tax[3].id
getResult.trips.tripOption[1].pricing[1].tax[3].kind
getResult.trips.tripOption[1].pricing[1].tax[3].salePrice
getResult.trips.tripOption[1].pricing[1].tax[4].chargeType
getResult.trips.tripOption[1].pricing[1].tax[4].code
getResult.trips.tripOption[1].pricing[1].tax[4].country
getResult.trips.tripOption[1].pricing[1].tax[4].id
getResult.trips.tripOption[1].pricing[1].tax[4].kind
getResult.trips.tripOption[1].pricing[1].tax[4].salePrice
getResult.trips.tripOption[1].pricing[1].tax[5].chargeType
getResult.trips.tripOption[1].pricing[1].tax[5].code
getResult.trips.tripOption[1].pricing[1].tax[5].id
getResult.trips.tripOption[1].pricing[1].tax[5].kind
getResult.trips.tripOption[1].pricing[1].tax[5].salePrice
getResult.trips.tripOption[1].pricing[1].tax[6].chargeType
getResult.trips.tripOption[1].pricing[1].tax[6].code
getResult.trips.tripOption[1].pricing[1].tax[6].id
getResult.trips.tripOption[1].pricing[1].tax[6].kind
getResult.trips.tripOption[1].pricing[1].tax[6].salePrice
getResult.trips.tripOption[1].saletotal
getResult.trips.tripOption[1].slice[1].duration
getResult.trips.tripOption[1].slice[1].kind
getResult.trips.tripOption[1].slice[1].segment[1].bookingCode
getResult.trips.tripOption[1].slice[1].segment[1].bookingCodeCount
getResult.trips.tripOption[1].slice[1].segment[1].cabin
getResult.trips.tripOption[1].slice[1].segment[1].duration
getResult.trips.tripOption[1].slice[1].segment[1].flight.carrier
getResult.trips.tripOption[1].slice[1].segment[1].flight.number
getResult.trips.tripOption[1].slice[1].segment[1].id
getResult.trips.tripOption[1].slice[1].segment[1].kind
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].aircraft
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].arrivalTime
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].departureTime
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].destination
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].duration
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].id
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].kind
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].meal
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].mileage
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].origin
getResult.trips.tripOption[1].slice[1].segment[1].leg[1].originTerminal
getResult.trips.tripOption[1].slice[1].segment[1].marriedSegmentGroup

Extract values in json object with awk/sed, but cannot get it to work

I have a file with the return of a curl statement in it, in the form of json. Each object has a set of values, but the parameters for these values are all called the same names. See code below.
These objects are part of a larger object called workflow. The Cleaning up object is the last process that runs in our workflow. For every video that passes through the workflow, a json file in this format is created. (There are more than only these three objects, this is just for illustrative purposes)
I want to take the value of completed of the object with "description": "Cleaning up" and store it as a variable $end_time. Then I want to take the value of completed of the object with "description": "Ingest" and store it as a variable $start_time. These two values are then subtracted to give me an integer time in milliseconds so I can calculate the time it took for the video to go through this part of the process. With the maths part I am fine, and know how to do it. It is the extraction of the values that I am struggling with.
I hope this makes sense? ANY help would be appreciated. Thank you in advance!
EDIT: Had to delete original code in post, due to character limitations
Here is a proper example of the file that I have to work with:
{
"workflows": {
"count": "20",
"searchTime": "1",
"startPage": "0",
"totalCount": "1",
"workflow": {
"configurations": {
"configuration": [
{
"$": "1409750880000",
"key": "schedule.start"
},
{
"$": "1409755980000",
"key": "schedule.stop"
},
{
"$": "Capture_agent",
"key": "schedule.location"
},
{
"$": "false",
"key": "trimHold"
},
{
"$": "true",
"key": "archiveOp"
},
{
"$": "false",
"key": "captionHold"
},
{
"$": "false",
"key": "videoPreview"
}
]
},
"creator": {
"organization": "mh_default_org",
"roles": [
"76b1bdde-a080-40a4-b929-bde89af6a0a8_Instructor",
"ROLE_ADMIN",
"ROLE_ANONYMOUS",
"ROLE_USER"
],
"userName": user_name
},
"description": "This workflow definition defines the steps involved in scheduling a recording, capturing it, and\n ingesting it, after which processing operations may be added.\n ",
"errors": "",
"id": "15518",
"mediapackage": {
"attachments": "",
"creators": {
"creator": "Name"
},
"id": "2d25ed19-2978-458d-a4a0-c9c56d791c68",
"license": "Creative Commons 3.0: Attribution-NonCommercial-NoDerivs",
"media": "",
"metadata": "",
"publications": {
"publication": {
"channel": "engage-player",
"id": "b7b68f91-2c33-4673-ba7c-2e9b891788f9",
"mimetype": "text/html",
"tags": "",
"url": "http://some.url.com:80/engage/ui/watch.html?id=2d25ed19-2978-458d-a4a0-c9c56d791c68"
}
},
"series": "76b1bdde-a080-40a4-b929-bde89af6a0a8",
"seriestitle": "Recording_Title_user_name",
"start": "2014-09-03T13:28:00Z",
"title": "Recording_Title"
},
"operations": {
"operation": [
{
"abortable": "false",
"completed": 1409750882092,
"configurations": {
"configuration": [
{
"$": "1409750880000",
"key": "schedule.start"
},
{
"$": "1409755980000",
"key": "schedule.stop"
},
{
"$": "Capture_agent",
"key": "schedule.location"
}
]
},
"continuable": "false",
"description": "Scheduled",
"execution-history": "",
"execution-host": "http://some.url.com:8080",
"fail-on-error": "true",
"failed-attempts": "0",
"hold-action-title": "View schedule",
"holdurl": "/workflow/hold/org.opencastproject.workflow.handler.scheduleworkflowoperationhandler",
"id": "schedule",
"job": "15519",
"max-attempts": "1",
"retry-strategy": "none",
"started": 1409750881745,
"state": "SUCCEEDED",
"time-in-queue": 0
},
{
"abortable": "false",
"configurations": "",
"continuable": "false",
"description": "Capture",
"execution-history": "",
"execution-host": "http://some.url.com:8080",
"fail-on-error": "true",
"failed-attempts": "0",
"hold-action-title": "Monitor capture",
"holdurl": "/workflow/hold/org.opencastproject.workflow.handler.captureworkflowoperationhandler",
"id": "capture",
"job": "42894",
"max-attempts": "1",
"retry-strategy": "none",
"started": 1409750884085,
"state": "SKIPPED",
"time-in-queue": 0
},
{
"completed": 1409756171224,
"configurations": "",
"description": "Ingest",
"execution-history": "",
"fail-on-error": "true",
"failed-attempts": "0",
"id": "ingest",
"max-attempts": "1",
"retry-strategy": "none",
"state": "SUCCEEDED"
},
{
"completed": 1409854379552,
"configurations": {
"configuration": {
"key": "preserve-flavors"
}
},
"description": "Cleaning up",
"execution-history": "",
"execution-host": "http://some.url.com:8080",
"fail-on-error": "false",
"failed-attempts": "0",
"id": "cleanup",
"job": "45113",
"max-attempts": "1",
"retry-strategy": "none",
"started": 1409854378128,
"state": "SUCCEEDED",
"time-in-queue": 0
}
]
},
"organization": {
"adminRole": "ROLE_ADMIN",
"anonymousRole": "ROLE_ANONYMOUS",
"id": "mh_default_org",
"name": "Opencast Project",
"properties": {
"property": [
{
"$": "true",
"key": "adminui.i18n_tab_episode.enable"
},
{
"$": "false",
"key": "adminui.i18n_tab_users.enable"
},
{
"$": "/engage/ui/img/mh_logos/OpencastLogo.png",
"key": "logo_small"
},
{
"$": "http://opencast.org/matterhorn/",
"key": "engageui.link_mobile_redirect.url"
},
{
"$": "false",
"key": "engageui.annotations.enable"
},
{
"$": "true",
"key": "engageui.links_media_module.enable"
},
{
"$": "2024",
"key": "adminui.chunksize"
},
{
"$": "false",
"key": "adminui.series_prepopulate.enable"
},
{
"$": "true",
"key": "engageui.link_download.enable"
},
{
"$": "false",
"key": "engageui.link_mobile_redirect.enable"
},
{
"$": "For more information have a look at the official site.",
"key": "engageui.link_mobile_redirect.description"
},
{
"$": "/engage/ui/img/mh_logos/MatterhornLogo_large.png",
"key": "logo_large"
}
]
},
"servers": {
"server": {
"name": "localhost",
"port": "8080"
}
}
},
"parent": {
"nil": "true"
},
"state": "SUCCEEDED",
"template": "full",
"title": "Scheduled Workflow"
}
}
}
Here is a jq example that should point you to getting what you want:
#!/bin/bash
# Assuming the json is in a file workflow.json
end_time=$( jq '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .completed' < workflow.json )
start_time=$( jq '.workflows.workflow.operations.operation[] | select(.description == "Ingest") | .completed' < workflow.json )
This is assuming the input you have is in an JSON array called workflow at the top level. Here's this on the command line:
$ jq '.workflows.workflow.operations.operation[] | select(.description == "Ingest") | .completed' < workflow.json
1406051539118
$ jq '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .completed' < workflow.json
1406051695440