Info
I have a terraform state file (json) with some deprecated attributes.
I would like to remove theses deprecated attributes.
I try to use jq and select() && del() but did not succeed to get back my full json without the deprecated attribue timeouts.
Problem
How to get my full json without the attribute timeouts for only one type of resources google_dns_record_set.
Data
{
"version": 4,
"terraform_version": "1.0.6",
"serial": 635,
"lineage": "6a9c2392-fdae-2b54-adcc-7366f262ffa4",
"outputs": {"test":"test1"},
"resources": [
{
"module": "module.resources",
"mode": "data",
"type": "google_client_config"
},
{
"module": "module.xxx.module.module1[\"cluster\"]",
"mode": "managed",
"type": "google_dns_record_set",
"name": "public_ip_ic_dns",
"provider": "module.xxx.provider[\"registry.terraform.io/hashicorp/google\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"id": "projects/xxx-xxx/managedZones/xxx--public/rrsets/*.net1.cluster.xxx--public.net.com./A",
"managed_zone": "xxx--public",
"name": "*.net1.cluster.xxx--public.net.com.",
"project": "xxx-xxx",
"rrdatas": [
"11.22.33.44"
],
"timeouts": null,
"ttl": 300,
"type": "A"
},
"sensitive_attributes": [],
"private": "xxx",
"dependencies": [
"xxx"
]
}
]
}
]
}
Command
jq -r '.resources[] | select(.type=="google_dns_record_set").instances[].attributes | del(.timeouts)' data.json
Pull the del command up front to include the whole selection as its own filter
del(.resources[] | select(.type=="google_dns_record_set").instances[].attributes.timeouts)
Demo
Related
I am using aws ecs query to get list of properties being used by the current running task.
command -
cft = "aws ecs describe-tasks --cluster arn:aws:ecs:us-west-2:4984314772:cluster/secrets --tasks arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b
I am storing this in an output variable
output= $( eval $cft)
Output:
"tasks": [
{
"attachments": [
{
"id": "da8a1312-8278-46d5-8e3b-6b6a1d96f820",
"type": "ElasticNetworkInterface",
"status": "ATTACHED",
"details": [
{
"name": "subnetId",
"value": "subnet-0a151f2eb959ad4"
},
{
"name": "networkInterfaceId",
"value": "eni-081948e3666253f"
},
{
"name": "macAddress",
"value": "02:2a:9i:5c:4a:77"
},
{
"name": "privateDnsName",
"value": "ip-172-56-17-177.us-west-2.compute.internal"
},
{
"name": "privateIPv4Address",
"value": "172.56.17.177"
}
]
}
],
"availabilityZone": "us-west-2a",
"clusterArn": "arn:aws:ecs:us-west-2:4984314772:cluster/secrets",
"containers": [
{
"taskArn": "arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b",
"name": "nginx",
"image": "nginx",
"lastStatus": "PENDING",
"networkInterfaces": [
{
"attachmentId": "da8a1312-8278-46d5-6b6a1d96f820",
"privateIpv4Address": "172.31.17.176"
}
],
"healthStatus": "UNKNOWN",
"cpu": "0"
}
],
"cpu": "256",
"createdAt": "2020-12-10T18:00:16.320000+05:30",
"desiredStatus": "RUNNING",
"group": "family:nginx",
"healthStatus": "UNKNOWN",
"lastStatus": "PENDING",
"launchType": "FARGATE",
"memory": "512",
"overrides": {
"containerOverrides": [
{
"name": "nginx"
}
],
"inferenceAcceleratorOverrides": []
},
"platformVersion": "1.4.0",
"tags": [],
"taskArn": "arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b",
"taskDefinitionArn": "arn:aws:ecs:us-west-2:4984314772:task-definition/nginx:17",
"version": 2
}
],
"failures": []
}
now if do an echo of $output.tasks[0].containers[0] nothing happens it prints the entire thing again, i want to store the result in output variable and refer different parameter like we do in json format.
You will need to use a json parser such as jq and so:
eval $cft | jq '.tasks[].containers[]'
To avoid using eval you could simple pipe the aws command into jq and so:
aws ecs describe-tasks --cluster arn:aws:ecs:us-west-2:4984314772:cluster/secrets --tasks arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b | jq '.tasks[].containers[]'
or:
cft=$(aws ecs describe-tasks --cluster arn:aws:ecs:us-west-2:4984314772:cluster/secrets --tasks arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b | jq '.tasks[].containers[]')
echo $cft | jq '.tasks[].containers[]'
I am taking a Github json file and parsing it with Java's regular expression library JsonPath. I am having a problem parsing arrays that do not have labels.
I need to send a email every time a particular file is changed in our repository.
Here is the Git Json:
{
"trigger": "push",
"payload": {
"type": "GitPush",
"before": "xxxxxxxx",
"after": "yyyyyyyy",
"branch": "branch-name",
"ref": "refs/heads/branch-name",
"repository": {
"id": 42,
"name": "repo",
"title": "repo",
"type": "GitRepository"
},
"beanstalk_user": {
"type": "Owner",
"id": 42,
"login": "username",
"email": "user#example.org",
"name": "Name Surname"
},
"commits": [
{
"type": "GitCommit",
"id": "ffffffff",
"message": "Important changes.",
"branch": "branch-name",
"author": {
"name": "Name Surname",
"email": "user#example.org"
},
"beanstalk_user": {
"type": "Owner",
"id": 42,
"login": "username",
"email": "user#example.org",
"name": "Name Surname"
},
"changed_files": {
"added": [
"NEWFILE",
],
"deleted": [
"Gemfile",
"NEWFILE"
],
"modified": [
"README.md",
"NEWFILE"
],
"copied": [
]
},
"changeset_url": "https://subdomain.github.com/repository-name/changesets/ffffffff",
"committed_at": "2014/08/18 13:30:29 +0000",
"parents": [
"afafafaf"
]
}
]
}
}
This is the expression I am using: to get the commits
$..changed_files
This return the whole changed files part but I can not explicitly choose the name "NEWFILE"
I tried
$..changed_files.*[?(#.added == "NEWFILE")]
$..changed_files.*[?(#.*== "NEWFILE")]
It just returns a empty array.
I just want it to return Newfile and what type of change. Any Ideas?
You can use the following JsonPath to retrieve the commits which list "NEWFILE" as an added file :
$.payload.commits[?(#.changed_files.added.indexOf("NEWFILE") != -1)]
I have a JSON file abc.json containing text:
{
"size": 3,
"limit": 25,
"isLastPage": true,
"values": [
{
"slug": "docker_apache_customised",
"id": 234889,
"name": "docker_apache_customised",
"scmId": "git",
"state": "AVAILABLE",
"statusMessage": "Available",
"forkable": true,
"project": {
"key": "UFD",
"id": 36239,
"name": "UF_docker",
"public": false,
"type": "NORMAL",
"links": {
"self": [{
"href": "https://rndwww.abc.xxx.net/git/projects/UFD"
}]
}
},
"public": false,
"links": {
"clone": [{
"href": "https://rndwww.abc.xxx.net/git/scm/ufd/docker_apache_customised.git",
"name": "http"
}, {
"href": "ssh://git#git.rnd.xxx.net/ufd/docker_apache_customised.git",
"name": "ssh"
}],
"self": [{
"href": "https://rndwww.abc.xxx.net/git/projects/UFD/repos/docker_apache_customised/browse"
}]
}
},
{
"slug": "web-software",
"id": 241533,
"name": "web-software",
"scmId": "git",
"state": "AVAILABLE",
"statusMessage": "Available",
"forkable": true,
"project": {
"key": "UFD",
"id": 36239,
"name": "UF_docker",
"public": false,
"type": "NORMAL",
"links": {
"self": [{
"href": "https://rndwww.abc.xxx.net/git/projects/UFD"
}]
}
},
"public": false,
"links": {
"clone": [{
"href": "https://rndwww.abc.xxx.net/git/scm/ufd/web-software.git",
"name": "http"
}, {
"href": "ssh://git#git.rnd.xxx.net/ufd/web-software.git",
"name": "ssh"
}],
"self": [{
"href": "https://rndwww.abc.xxx.net/git/projects/UFD/repos/web-software/browse"
}]
}
},
{
"slug": "web-loy-conf",
"id": 240959,
"name": "web-loy-conf",
"scmId": "git",
"state": "AVAILABLE",
"statusMessage": "Available",
"forkable": true,
"project": {
"key": "UFD",
"id": 36239,
"name": "UF_docker",
"public": false,
"type": "NORMAL",
"links": {
"self": [{
"href": "https://rndwww.abc.xxx.net/git/projects/UFD"
}]
}
},
"public": false,
"links": {
"clone": [{
"href": "ssh://git#git.rnd.xxx.net/ufd/web-loy-conf.git",
"name": "ssh"
}, {
"href": "https://rndwww.abc.xxx.net/git/scm/ufd/web-loy-conf.git",
"name": "http"
}],
"self": [{
"href": "https://rndwww.abc.xxx.net/git/projects/UFD/repos/web-loy-conf/browse"
}]
}
}
],
"start": 0
}
This text contains three repositories(named docker_apache_customised, web-software, web-loy-conf) in a git project. There may be more repos containing web as substring.
I want to perform some operation on the repositories which has web as substring, and for that I think I have to apply a for loop in shell script. I don't want to use jq tool
I wrote a script using external tool jq, but I want to do it with Linux in-built tools only. The script using jq is working fine:
for k in $(jq '.values | keys | .[]' abc.json); do
value=$(jq -r ".values[$k]" abc.json);
name=$(jq -r '.name' <<< "$value");
if [[ $name == *"web"* ]]; then
#MYLOGIC
done
done
Expected result are names (web-software, web-loy-conf) and to be able to loop through that
You can run jq from its current path in your git repository, there's no need to copy it to a directory in the PATH. After adding execution permissions:
value=$(<path to jq in git dir>/jq -r ".values[$k]" abc.json);
You can make it relative to git repository root
value=$(./<path to jq from git repo root>/jq -r ".values[$k]" abc.json);
Also, you can set the path to it in a variable
jqbin='./<path to jq from git repo root>/jq'
value=$($jqbin -r ".values[$k]" abc.json);
I have a deep json. Sometimes, I need to look for the json path for a key containing certain word.
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"creationTimestamp": "2019-03-28T21:09:42Z",
"labels": {
"bu": "finance",
"env": "prod"
},
"name": "auth",
"namespace": "default",
"resourceVersion": "2786",
"selfLink": "/api/v1/namespaces/default/pods/auth",
"uid": "ce73565a-519d-11e9-bcb7-0242ac110009"
},
"spec": {
"containers": [
{
"command": [
"sleep",
"4800"
],
"image": "busybox",
"imagePullPolicy": "Always",
"name": "busybox",
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "default-token-dbpcm",
"readOnly": true
}
]
}
],
"dnsPolicy": "ClusterFirst",
"nodeName": "node01",
"priority": 0,
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "default",
"serviceAccountName": "default",
"terminationGracePeriodSeconds": 30,
"tolerations": [
{
"effect": "NoExecute",
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"tolerationSeconds": 300
},
{
"effect": "NoExecute",
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"tolerationSeconds": 300
}
],
"volumes": [
{
"name": "default-token-dbpcm",
"secret": {
"defaultMode": 420,
"secretName": "default-token-dbpcm"
}
}
]
},
"status": {
"conditions": [
{
"lastProbeTime": null,
"lastTransitionTime": "2019-03-28T21:09:42Z",
"status": "True",
"type": "Initialized"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2019-03-28T21:09:50Z",
"status": "True",
"type": "Ready"
},
{
"lastProbeTime": null,
"lastTransitionTime": null,
"status": "True",
"type": "ContainersReady"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2019-03-28T21:09:42Z",
"status": "True",
"type": "PodScheduled"
}
],
"containerStatuses": [
{
"containerID": "docker://b5be8275555ad70939401d658bb4e504b52215b70618ad43c2d0d02c35e1ae27",
"image": "busybox:latest",
"imageID": "docker-pullable://busybox#sha256:061ca9704a714ee3e8b80523ec720c64f6209ad3f97c0ff7cb9ec7d19f15149f",
"lastState": {},
"name": "busybox",
"ready": true,
"restartCount": 0,
"state": {
"running": {
"startedAt": "2019-03-28T21:09:49Z"
}
}
}
],
"hostIP": "172.17.0.37",
"phase": "Running",
"podIP": "10.32.0.4",
"qosClass": "BestEffort",
"startTime": "2019-03-28T21:09:42Z"
}
}
Currently If i need the podIP, then I do that this way to find the object which has the search keyword and then I build the path
curl myson | jq "[paths]" | grep "IP" --context=10
Is there any nice shortcut to simplify this? What I really need is - all the paths which could have the matching key.
spec.podIP
spec.hostIP
select paths containing keyword in their last element, and use join(".") to generate your desired output.
paths
| select(.[-1] | type == "string" and contains("keyword"))
| join(".")
.[-1] returns the last element of an array,
type == "string" is required because an array index is a number and numbers and strings can't be checked for their containment.
You may want to specify -r option.
As #JeffMercado implicitly suggested you can set the query from command line without touching the script:
jq -r 'paths
| select(.[-1] | type == "string" and contains($q))
| join(".")' file.json --arg q 'keyword'
You can stream the input in, which provides paths and values. You could then inspect the paths and optionally output the values.
$ jq --stream --arg pattern 'IP' '
select(length == 2 and any(.[0][] | strings; test($pattern)))
| "\(.[0] | join(".")): \(.[1])"
' input.json
"status.hostIP: 172.17.0.37"
"status.podIP: 10.32.0.4"
shameless plug
https://github.com/TomConlin/json_to_paths
because sometime you do not even know the component you want to filter for before you see what is there.
json2jqpath.jq file.json
.
.apiVersion
.kind
.metadata
.metadata|.creationTimestamp
.metadata|.labels
.metadata|.labels|.bu
.metadata|.labels|.env
.metadata|.name
.metadata|.namespace
.metadata|.resourceVersion
.metadata|.selfLink
.metadata|.uid
.spec
.spec|.containers
.spec|.containers|.[]
.spec|.containers|.[]|.command
.spec|.containers|.[]|.command|.[]
.spec|.containers|.[]|.image
.spec|.containers|.[]|.imagePullPolicy
.spec|.containers|.[]|.name
.spec|.containers|.[]|.resources
.spec|.containers|.[]|.terminationMessagePath
.spec|.containers|.[]|.terminationMessagePolicy
.spec|.containers|.[]|.volumeMounts
.spec|.containers|.[]|.volumeMounts|.[]
.spec|.containers|.[]|.volumeMounts|.[]|.mountPath
.spec|.containers|.[]|.volumeMounts|.[]|.name
.spec|.containers|.[]|.volumeMounts|.[]|.readOnly
.spec|.dnsPolicy
.spec|.nodeName
.spec|.priority
.spec|.restartPolicy
.spec|.schedulerName
.spec|.securityContext
.spec|.serviceAccount
.spec|.serviceAccountName
.spec|.terminationGracePeriodSeconds
.spec|.tolerations
.spec|.tolerations|.[]
.spec|.tolerations|.[]|.effect
.spec|.tolerations|.[]|.key
.spec|.tolerations|.[]|.operator
.spec|.tolerations|.[]|.tolerationSeconds
.spec|.volumes
.spec|.volumes|.[]
.spec|.volumes|.[]|.name
.spec|.volumes|.[]|.secret
.spec|.volumes|.[]|.secret|.defaultMode
.spec|.volumes|.[]|.secret|.secretName
.status
.status|.conditions
.status|.conditions|.[]
.status|.conditions|.[]|.lastProbeTime
.status|.conditions|.[]|.lastTransitionTime
.status|.conditions|.[]|.status
.status|.conditions|.[]|.type
.status|.containerStatuses
.status|.containerStatuses|.[]
.status|.containerStatuses|.[]|.containerID
.status|.containerStatuses|.[]|.image
.status|.containerStatuses|.[]|.imageID
.status|.containerStatuses|.[]|.lastState
.status|.containerStatuses|.[]|.name
.status|.containerStatuses|.[]|.ready
.status|.containerStatuses|.[]|.restartCount
.status|.containerStatuses|.[]|.state
.status|.containerStatuses|.[]|.state|.running
.status|.containerStatuses|.[]|.state|.running|.startedAt
.status|.hostIP
.status|.phase
.status|.podIP
.status|.qosClass
.status|.startTime
I am starting with elasticsearch now and i don't know anything about it.
I have folowing .JSON:
[
{
"label": "Admin Law",
"tags": [
"#admin"
],
"owner": "generalTopicTagText"
},
{
"label": "Judicial review",
"tags": [
"#JR"
],
"owner": "generalTopicTagText"
},
{
"label": "Admiralty/Shipping",
"tags": [
"#shipping"
],
"owner": "generalTopicTagText"
}
]
My mapping is this:
{
"topic_tax": {
"properties": {
"label": {
"type": "string",
"index": "not_analyzed"
},
"tags": {
"type": "string",
"index_name": "tag"
},
"owner": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
I need to put the first .Json into Elasticsearch, but it does not work.
All I know is that i am defining only 1 of this:
{
"label": "Judicial review",
"tags": [
"#JR"
],
"owner": "generalTopicTagText"
}
So when i try to put all of them with my elasticsearch.init, it will not work.
But I really don't know how to declare the mapping.Json to put the all .Json, it is like i need something like a for there.
You have to insert them json after json. But what you should do is use the bulk api of elasticsearch to insert multiple documents in one request. Check this api doc to see how it works
You can do something like this
curl -XPUT 'localhost:9000/es/post/1?version=2' -d '{
"text" : "your test message!"
}'
here is the documentation for index json with elasticsearch