JOLT remove all but first two fields - json

Trying to use JOLT within Nifi to remove some fields from an API call, but having trouble getting it to work. Below is what is returned from the GitHub API:
[
{
"login": "sample name",
"id": "000001",
"node_id": "ID#",
"avatar_url": "RETURNED_URL",
"gravatar_id": "",
"url": "RETURNED_URL",
"html_url": "RETURNED_URL",
"followers_url": "RETURNED_URL",
"following_url": "RETURNED_URL",
"gists_url": "RETURNED_URL",
"starred_url": "RETURNED_URL",
"subscriptions_url": "RETURNED_URL",
"organizations_url": "RETURNED_URL",
"repos_url": "RETURNED_URL",
"events_url": "RETURNED_URL",
"received_events_url": "RETURNED_URL",
"type": "MEMBER_TYPE",
"site_admin": false
},
...
}
]
This is the JOLT Spec I have that is raising errors:
[
{
"operation": "remove",
"spec": {
"node_id":"",
"avatar_url":"",
"gravatar_id":"",
"url":"",
"html_url":"",
"followers_url": "",
"following_url": "",
"gists_url": "",
"starred_url": "",
"subscriptions_url": "",
"organizations_url": "",
"repos_url": "",
"events_url": "",
"received_events_url": "",
"type": "User",
"site_admin": false
}
}
]
There are about 70 results returned and i need it to do this for each one. None of them have a higher level key, so I thought it was that, but everything I've tried seems to throw a " is invalid because Specification not valid for the selected transformation

Instead or removing long list of fields, you can just pick the fields you need. In your case you can use Jolt Spec like this:
[
{
"operation": "shift",
"spec": {
"*": {
"login": "[&1].login",
"id": "[&1].id"
}
}
}
]
You can also use this jolt demo site to check yourself. This is how i checked it:

Related

Select elements from json array based on 2 criteria

I am fetching the list of issue comments on a specific PR with the following curl command
curl -sH "Accept: application/vnd.github+json" -H "Authorization: token XXXXXXXXXX" https://api.github.com/repos/org/repo/issues/12/comments
The response format is according to the documentation as follows:
[
{
"id": 1,
"node_id": "MDEyOklzc3VlQ29tbWVudDE=",
"url": "https://api.github.com/repos/octocat/Hello-World/issues/comments/1",
"html_url": "https://github.com/octocat/Hello-World/issues/1347#issuecomment-1",
"body": "Me too",
"user": {
"login": "octocat",
"id": 1,
"node_id": "MDQ6VXNlcjE=",
"avatar_url": "https://github.com/images/error/octocat_happy.gif",
"gravatar_id": "",
"url": "https://api.github.com/users/octocat",
"html_url": "https://github.com/octocat",
"followers_url": "https://api.github.com/users/octocat/followers",
"following_url": "https://api.github.com/users/octocat/following{/other_user}",
"gists_url": "https://api.github.com/users/octocat/gists{/gist_id}",
"starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/octocat/subscriptions",
"organizations_url": "https://api.github.com/users/octocat/orgs",
"repos_url": "https://api.github.com/users/octocat/repos",
"events_url": "https://api.github.com/users/octocat/events{/privacy}",
"received_events_url": "https://api.github.com/users/octocat/received_events",
"type": "User",
"site_admin": false
},
"created_at": "2011-04-14T16:00:49Z",
"updated_at": "2011-04-14T16:00:49Z",
"issue_url": "https://api.github.com/repos/octocat/Hello-World/issues/1347",
"author_association": "COLLABORATOR"
}
]
I know that I can get the last comment's author's username by piping the output to jq '.[-1].user.login'
How can I get from ALL the comments matching user.login=foobar the most recent one?
select is the function that filters by criteria.
Use with map to retain the array structure
jq 'map(select(.user.login == "foobar"))'
Or iterate and get just the matching items
jq '.[] | select(.user.login == "foobar")'

jq: error: syntax error, unexpected if, while checking for a object to be present and if not the adding default value

Getting the error
jq: error: syntax error, unexpected if (Unix shell quoting issues?) at <top-level> while parsing kubernetes kubectl command,
kubectl get pv -o=json | jq -c '.items[] | {name: .metadata.name, claimName: .spec | if has("claimRef") then .claimRef.name else empty end }'
Trying to get the PV's which have PVC and if there are no PVC can be null of empty.
For it uilding a map of name and their respective PVC if there otherwise empty or null.
Below is a sample input to the jq command or the out put of the command kubectl get pv -o=json
{
"apiVersion": "v1",
"items": [{
"apiVersion": "v1",
"kind": "PersistentVolume",
"metadata": {
"annotations": {
"meta.helm.sh/release-name": "",
"meta.helm.sh/release-namespace": ""
},
"creationTimestamp": "",
"finalizers": [
""
],
"labels": {
"app.kubernetes.io/managed-by": ""
},
"name": "",
"resourceVersion": "",
"uid": ""
},
"spec": {
"accessModes": [
"ReadWriteMany"
],
"capacity": {
"storage": ""
},
"claimRef": {
"apiVersion": "",
"kind": "",
"name": "",
"namespace": "",
"resourceVersion": "",
"uid": ""
},
"csi": {
"driver": "",
"volumeHandle": ""
},
"persistentVolumeReclaimPolicy": "",
"volumeMode": ""
},
"status": {
"phase": ""
}
},
{
"apiVersion": "v1",
"kind": "PersistentVolume",
"metadata": {
"annotations": {
"meta.helm.sh/release-name": "",
"meta.helm.sh/release-namespace": ""
},
"creationTimestamp": "",
"finalizers": [
""
],
"labels": {
"app.kubernetes.io/managed-by": ""
},
"name": "",
"resourceVersion": "",
"uid": ""
},
"spec": {
"accessModes": [
"ReadWriteMany"
],
"capacity": {
"storage": ""
},
"claimRef": {
"apiVersion": "",
"kind": "",
"name": "",
"namespace": "",
"resourceVersion": "",
"uid": ""
},
"csi": {
"driver": "",
"volumeHandle": ""
},
"persistentVolumeReclaimPolicy": "",
"volumeMode": ""
},
"status": {
"phase": ""
}
}
],
"kind": "List",
"metadata": {
"resourceVersion": "",
"selfLink": ""
}
}
For data security have removed the data and only kept the keys
Please help
Put parentheses around the value of the claimName field:
{
name: .metadata.name,
claimName: (.spec | if has("claimRef") then .claimRef.name else empty end)
}
Depending on how much you simplified your example, you may consider reducing the the if statement to:
{
name: .metadata.name,
claimName: (.spec.claimRef.name? // empty)
}
Note that in both cases not just .claimName but the whole object will disappear if the respective test is going to fail.

Can you get an issue's priority in a Github milestone with the REST API?

Github lets you prioritize issues in a milestone. Is there any way to get that information with the Github REST API?
For example, this query
https://api.github.com/repos/criesbeck/workit/issues
for the issues for this milestone returns no clue that I can see that the issue "User can control with voice commands" is the sixth issue in the milestone. Am I missing it?
[ ...
{
"url": "https://api.github.com/repos/criesbeck/workit/issues/3",
"repository_url": "https://api.github.com/repos/criesbeck/workit",
"labels_url": "https://api.github.com/repos/criesbeck/workit/issues/3/labels{/name}",
"comments_url": "https://api.github.com/repos/criesbeck/workit/issues/3/comments",
"events_url": "https://api.github.com/repos/criesbeck/workit/issues/3/events",
"html_url": "https://github.com/criesbeck/workit/issues/3",
"id": 852867677,
"node_id": "MDU6SXNzdWU4NTI4Njc2Nzc=",
"number": 3,
"title": "User can control with voice commands",
"user": {
"login": "criesbeck",
"id": 1717574,
"node_id": "MDQ6VXNlcjE3MTc1NzQ=",
"avatar_url": "https://avatars.githubusercontent.com/u/1717574?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/criesbeck",
"html_url": "https://github.com/criesbeck",
"followers_url": "https://api.github.com/users/criesbeck/followers",
"following_url": "https://api.github.com/users/criesbeck/following{/other_user}",
"gists_url": "https://api.github.com/users/criesbeck/gists{/gist_id}",
"starred_url": "https://api.github.com/users/criesbeck/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/criesbeck/subscriptions",
"organizations_url": "https://api.github.com/users/criesbeck/orgs",
"repos_url": "https://api.github.com/users/criesbeck/repos",
"events_url": "https://api.github.com/users/criesbeck/events{/privacy}",
"received_events_url": "https://api.github.com/users/criesbeck/received_events",
"type": "User",
"site_admin": false
},
"labels": [
{
"id": 2893844412,
"node_id": "MDU6TGFiZWwyODkzODQ0NDEy",
"url": "https://api.github.com/repos/criesbeck/workit/labels/enhancement",
"name": "enhancement",
"color": "a2eeef",
"default": true,
"description": "New feature or request"
},
{
"id": 2893865786,
"node_id": "MDU6TGFiZWwyODkzODY1Nzg2",
"url": "https://api.github.com/repos/criesbeck/workit/labels/size:%205",
"name": "size: 5",
"color": "E96C70",
"default": false,
"description": "5 story points"
}
],
"state": "open",
"locked": false,
"assignee": null,
"assignees": [
],
"milestone": {
"url": "https://api.github.com/repos/criesbeck/workit/milestones/1",
"html_url": "https://github.com/criesbeck/workit/milestone/1",
"labels_url": "https://api.github.com/repos/criesbeck/workit/milestones/1/labels",
"id": 6637701,
"node_id": "MDk6TWlsZXN0b25lNjYzNzcwMQ==",
"number": 1,
"title": "Final demo",
"description": "",
"creator": {
"login": "criesbeck",
"id": 1717574,
"node_id": "MDQ6VXNlcjE3MTc1NzQ=",
"avatar_url": "https://avatars.githubusercontent.com/u/1717574?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/criesbeck",
"html_url": "https://github.com/criesbeck",
"followers_url": "https://api.github.com/users/criesbeck/followers",
"following_url": "https://api.github.com/users/criesbeck/following{/other_user}",
"gists_url": "https://api.github.com/users/criesbeck/gists{/gist_id}",
"starred_url": "https://api.github.com/users/criesbeck/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/criesbeck/subscriptions",
"organizations_url": "https://api.github.com/users/criesbeck/orgs",
"repos_url": "https://api.github.com/users/criesbeck/repos",
"events_url": "https://api.github.com/users/criesbeck/events{/privacy}",
"received_events_url": "https://api.github.com/users/criesbeck/received_events",
"type": "User",
"site_admin": false
},
"open_issues": 6,
"closed_issues": 0,
"state": "open",
"created_at": "2021-04-08T01:00:22Z",
"updated_at": "2021-04-08T01:03:44Z",
"due_on": "2021-04-29T07:00:00Z",
"closed_at": null
},
"comments": 0,
"created_at": "2021-04-07T22:16:01Z",
"updated_at": "2021-04-08T01:01:54Z",
"closed_at": null,
"author_association": "OWNER",
"active_lock_reason": null,
"body": "",
"performed_via_github_app": null
},
...]

Postman: receiving syntax errors from postman POST request

{
"order": {
"operationTimes": {
"start": "",
"end": ""
},
"forCustomer": "87778",
"type": "8787",
"stand": "",
"plannedStartAt": "",
"begin": {
"number": "9445",
"destination": "8787",
"suffix": "",
"originDate": "21\/02\/2021",
"isNew": true,
"origin": "989898",
"existingId": "",
"company": "98998"
},
"private": false,
"loc": "",
"types": "test",
"date": "21\/02\/2021",
"transactionType": "",
"testSupplier": "88881d335b647351068b0000",
"cust": "343434"
},
"forNew": false
}
However, when I post this as raw in Postman, I'm getting the following syntax error:
Failed to parse data (org.json.JSONException: Expected a ',' or '}' at 650 [character 45 line 26]
This points to the testSupplier line but I cannot see where I'm missing anything here.I don't want to be closing the object with } at this point as I have more properties which are part of the order object. And I already have a ,. Obviously this error is preventing the full body from posting and therefore I am receiving a 400 response.
Any help would be greatly appreciated.
Update:
When I check the call through Charles proxy, this is what is actually being posted in the body:
{
"fuelOrder": {
"operationTimes": {
"onBlocksAtScheduled": "",
"offBlocksAtScheduled": ""
},
"forCustomer": "3255",
"fuelType": "23523",
"stand": "",
"plannedStartAt": "",
"outbound": {
"number": "9445",
"destination": "235235",
"suffix": "",
"originDate": "21\/02\/2021",
"isNew": true,
"origin": "235235",
"existingId": "",
"airline": "52323523"
},
"privateFlight": false,
"terminal": "",
"type": "ref",
"date": "21\/02\/2021",
"transactionType": "",
"fuelSupplier": "23523"
So it is not posting anything after fuelSupplier
This is how the error is showing in Charles

JSON path extractor

I have the following JSON response and I wanted to build the JSON expression to extract the "connectionId" based on given "systemId" and "connectionConfig"."type" value. For example, extract "connectionId" for "systemId"= 'DHB_K8S_HADOOP_SYS' and "connectionConfig"."type"='S3'
{
"browse": [
{
"name": "DHB_K8S_HDFS_Conn1",
"type": "connection",
"connectionId": "DHB_K8S_HDFS_Conn1",
"systemId": "DHB_K8S_HADOOP_SYS",
"uri": "hdfs:///",
"connectionConfig": {
"type": "HDFS",
"description": "",
"connection": "",
"protocol": "",
"authentication": {
"type": "",
"user": "vora"
},
"systemId": "DHB_K8S_HADOOP_SYS",
"zoneId": "DHB_Zone"
},
"children": [
{}
]
},
{
"name": "DHB_K8S_HDFS_S3_Conn1",
"type": "connection",
"connectionId": "DHB_K8S_HDFS_S3_Conn1",
"systemId": "DHB_K8S_HADOOP_SYS",
"uri": "s3a:/",
"connectionConfig": {
"type": "S3",
"description": "",
"connection": "",
"protocol": "",
"authentication": {},
"systemId": "DHB_K8S_HADOOP_SYS",
"zoneId": "DHB_Zone"
},
"children": [
{}
]
},
{
"name": "DHB_K8S_VORA_S3",
"type": "connection",
"connectionId": "DHB_K8S_VORA_S3",
"systemId": "DHB_K8S_VORA_SYS",
"uri": "s3a:/",
"connectionConfig": {
"type": "S3",
"description": "",
"connection": "",
"protocol": "",
"authentication": {},
"systemId": "DHB_K8S_VORA_SYS",
"zoneId": "DHB_Zone"
},
"children": [
{}
]
}
]
}
Any help is appreciated