I'm trying to concatenating an output from oc get in openshift, I'm using -o=custom-columns but it's not joining 2 equal values when I get the normal output...
{
"roleRef":{
"kind": "ClusterRole",
"name": "view"
}
}
The result I wanted is that in the same column the values of "kind" and "name" come out like this:
ROLE
ClusterRole/view
If I give an "oc get rolebinding" in openshift it already gives me this return, but if I use -o=custom-columns to edit the columns as I prefer, I didn't find a way to convert this, can you help me?
Unfortunately kubectl/oc only supports simplistic jsonpath expressions for -o custom-columns. If you really want to display this as a single column (as opposed to just displaying two separate columns), you'll have to rely on -o jsonpath. What this would look like:
$ oc get rolebinding -o jsonpath='{range .items[*]}{.roleRef.kind}{"/"}{.roleRef.name}{"\n"}{end}'
It's obviously a bit more verbose, but infinitely more customizable. Hope this helps!
For what it's worth, if you want/need to get more fancy with column headers, etc, it would look something like
$ oc get rolebinding -o jsonpath='{"NAME"}{"\t"}{"ROLE"}{"\n"}{range .items[*]}{.metadata.name}{"\t"}{.roleRef.kind}{"/"}{.roleRef.name}{"\n"}{end}'
Related
apiVersion: v1
data:
backupscript:
service.properties: |
agent.download.location = /home/bnr
script.execution.time.in.minutes = 1
Need to fetch value in "script.execution.time.in.minutes".
I am using ..-o "jsonpath={.data['service\.properties'].'script\.execution\.time\.in\.minutes'}"
It is giving me empty result.
How do we use the Escape on the end field's filter.
Sometimes I find it helpful when working with jsonpath expressions, to start smaller and build up.
For example, does this work and if so, what does it return?
-o "jsonpath={.data['service\.properties']"
Then add the next part on.
That being said, I think the problem you're going to run into is that these are not yaml properties, but instead is a string, as indicated by the |:
service.properties: |
agent.download.location = /home/bnr
script.execution.time.in.minutes = 1
So I don't think you're going to be able to use jsonpath alone to query the value of script.execution.time.in.minutes.
You can probably do what you want with a combination of jsonpath and awk+sed, something like this:
kubectl get pod foo -o jsonpath="{.data['service\.properties']}" | awk -F'=' '$1 ~ /script\.execution\.time\.in\.minutes/ {print $2}' | sed 's/ //'
This does the following:
Get the service properties using jsonpath
Use awk to extract the number from the line that has script.execution.time.in.minutes
Use sed to remove spaces
There might be a more elegant way to accomplish this, but hopefully this can at least help you get an idea of one way to do it.
I'm trying to get usable json from the docker cli, however it seems it will only produce json for individual items, and not the complete result, as a whole.
For example, running docker container ls -a --format="{{ json .Names }}" produces:
"hopeful_payne"
"trusting_turing"
"stupefied_morse"
"unruffled_noyce"
"pensive_fermi"
"objective_neumann"
"confident_bhaskara"
"unruffled_cray"
"epic_newton"
"boring_bartik"
"priceless_sinoussi"
"naughty_grothendieck"
"hardcore_bose"
"sad_jones"
"optimistic_napier"
"trusting_stallman"
"xenodochial_dijkstra"
"pedantic_cocks"
The above is not json.
How can I produce a result that is, ideally, a json array?
I think you cannot do this using docker only.
The command-line's format function is effectively taking each input line (one for each container) and applying the Go template to it. So you need another tool to aggregate the lines into a JSON array.
One way that you can achieve your goal is using the excellent jq tool:
docker container ls --format="{\"name\":\"{{.Names}}\"}" --all | jq --slurp
This generates each container line as a JSON string: {"name": "[VALUE]"} and then uses jq to slurp them into a JSON array.
A challenge doing this directly in bash is JSON's stricture that the last element in a list can't be terminated with a ,. So, the following simple bash script generates invalid JSON and you'd need extra logic to remove it (or better yet, not add the last one):
echo "[$(for CONTAINER in $(docker container ls --format="{{.Names}}" --all); do echo "{\"name\":\"${CONTAINER}\"},"; done;)]"
What are you trying to do with these JSON responses? It might be easier just to talk directly to the Docker API, which will give you JSON responses directly. E.g., to get a list of containers:
curl --unix-socket /var/run/docker.sock http://localhost/v1.24/containers/json
You can, as DazWilkin suggested, use jq for filtering JSON on the command line. E.g., if we want a list of container names:
curl --unix-socket /var/run/docker.sock http://localhost/v1.24/containers/json |
jq '[.[]|.Names]'
You can find Docker API documentation here.
One way to think of the output is that it's JSONL: http://jsonlines.org/
This Docker output is JSON, per line. Since you asked for a single attribute -- just the name -- you're simply getting a string back. But, notice it's quoted. It's technically JSON. It may make more sense if you update your format to {{ json . }}, which will then output lines that look more like the JSON you're expecting.
However, it's still a JSON document per line, so you'd have to process each line as its own document.
I am trying to update a simple JSON file (consists of one object with several key/value pairs) and I am using the same command yet getting different results (sometimes even having the whole json wiped with the 2nd command). The command I am trying is:
cat ~/Desktop/config.json | jq '.Option = "klay 10"' | tee ~/Desktop/config.json
This command perfectly replaces the value of the minerOptions key with "klay 10", my intended output.
Then, I try to run the same process on the newly updated file (just value is changed for that one key) and only get interactive terminal with no result. ps unfortunately isn't helpful in showing what's going on. This is what I do after getting that first command to perfectly change the value of the key:
cat ~/Desktop/config.json | jq ‘.othOptions = "-epool etc-eu1.nanopool.org:14324 -ewal 0xc63c1e59c54ca935bd491ac68fe9a7f1139bdbc0 -mode 1"' | tee ~/Desktop/config.json
which I would have expected would replace the othOptions key value with the assigned result, just as the last did. I tried directly sending the stdout to the file, but no result there either. I even tried piping one more time and creating a temp file and then moving it to change to original, all of these, as opposed to the same identical command, just return > and absolutely zero output; when I quit the process, it is the same value as before, not the new one.
What am I missing here that is causing the same command with just different inputs (the key in second comes right after first and has identical structure, it's not creating an object or anything, just key-val pair like first. I thought it could be tee but any other implementation like a passing of stdout to file produces the same constant > waiting for a command, no user.
I genuinely looked everywhere I could online for why this could be happening before resorting to SE, it's giving me such a headache for what I thought should be simple.
As #GordonDavisson pointed out, using tee to overwrite the input file is a (well-known - see e.g. the jq FAQ) recipe for disaster. If you absolutely positively want to overwrite the file unconditionally, then you might want to consider using sponge, as in
jq ... config.json | sponge config.json
or more safely:
cp -p config.json config.json.bak && jq ... config.json | sponge config.json
For further details about this and other options, search for ‘sponge’ in the FAQ.
I need to automatically move new cases (TheHive-Project) to LimeSurvey every 5 minutes. I have figured out the basis of the API script to add responses to LimeSurvey. However, I can't figure out how to add only new cases, and how to parse the Hive case data for the information I want to add.
So far I've been using curl to get a list of cases from hive. The following is the command and the output.
curl -su user:pass http://myhiveIPaddress:9000/api/case
[{"createdBy":"charlie","owner":"charlie","createdAt":1498749369897,"startDate":1498749300000,"title":"test","caseId":1,"user":"charlie","status":"Open","description":"testtest","tlp":2,"tags":[],"flag":false,"severity":1,"metrics":{"Time for Alert to Handler Pickup":2,"Time from open to close":4,"Time from compromise to discovery":6},"updatedBy":"charlie","updatedAt":1498751817577,"id":"AVz0bH7yqaVU6WeZlx3w","_type":"case"},{"createdBy":"charlie","owner":"charlie","title":"testtest","caseId":3,"description":"ddd","user":"charlie","status":"Open","createdAt":1499446483328,"startDate":1499446440000,"severity":2,"tlp":2,"tags":[],"flag":false,"id":"AV0d-Z0DqHSVxnJ8z_HI","_type":"case"},{"createdBy":"charlie","owner":"charlie","createdAt":1499268177619,"title":"test test","user":"charlie","status":"Open","caseId":2,"startDate":1499268120000,"tlp":2,"tags":[],"flag":false,"description":"s","severity":1,"metrics":{"Time from open to close":2,"Time for Alert to Handler Pickup":3,"Time from compromise to discovery":null},"updatedBy":"charlie","updatedAt":1499268203235,"id":"AV0TWOIinKQtYP_yBYgG","_type":"case"}]
Each field is separated by the delimiter },{.
In regards to parsing out specific information from each case, I previously tried to just use the cut command. This mostly worked until I reached "metrics"; it doesn't always work for metrics because they will not always be listed in the same order.
I have asked my boss for help, and he told me this command might get me going in the right direction to adding only new hive cases to the survey, but I'm still very lost and want to avoid asking too much again.
curl -su user:pass http://myhiveIPaddress:9000/api/case | sed 's/},{/\n/g' | sed 's/\[{//g' | sed 's/}]//g' | awk -F '"caseId":' {'print $2'} | cut -f 1 -d , | sort -n | while read line; do echo '"caseId":'$line; done
Basically, I'm in way over my head and feel like I have no idea what I'm doing. If I need to clarify anything, or if it would help for me to post what I have so far in my API script, please let me know.
Update
Here is the potential logic for the script I'd like to write.
get list of hive cases (curl ...)
read each field, delimited by },{
while read each field, check /tmp/addedHiveCases to see if caseId of field already exists
--> if it does not exist in file, add case to limesurvey and add caseId to /tmp/addedHiveCases
--> if it does exist, skip to next field
why are you thinking that the fields are separated by a "},{" delimiter?
The response of the /api/case API is a valid JSON format, that lists the cases.
Can you use a Python script to play with the API? If yes, I can help you write the script you need.
I'm creating a Bash script to parse the air pollution levels from the webpage:
http://aqicn.org/city/beijing/m/
There is a lot of stuff in the file, but this is the relevant bit:
"iaqi":[{"p":"pm25","v":[59,21,112],"i":"Beijing pm25 (fine
particulate matter) measured by U.S Embassy Beijing Air Quality
Monitor
(\u7f8e\u56fd\u9a7b\u5317\u4eac\u5927\u4f7f\u9986\u7a7a\u6c14\u8d28\u91cf\u76d1\u6d4b).
Values are converted from \u00b5g/m3 to AQI levels using the EPA
standard."},{"p":"pm10","v":[15,5,69],"i":"Beijing pm10
(respirable particulate matter) measured by Beijing Environmental
Protection Monitoring Center
I want the script to parse and display 2 numbers: current PM2.5 and PM10 levels (the numbers in bold in the above paragraph).
CITY="beijing"
AQIDATA=$(wget -q 0 http://aqicn.org/city/$CITY/m/ -O -)
PM25=$(awk -v FS="(\"p\":\"pm25\",\"v\":\\\[|,[0-9]+)" '{print $2}' <<< $AQIDATA)
PM100=$(awk -v FS="(\"p\":\"pm10\",\"v\":\\\[|,[0-9]+)" '{print $2}' <<< $AQIDATA)
echo $PM25 $PM100
Even though I can get PM2.5 levels to display correctly, I cannot get PM10 levels to display. I cannot understand why, because the strings are similar.
Anyone here able to explain?
The following approach is based on two steps:
(1) Extracting the relevant JSON;
(2) Extracting the relevant information from the JSON using a JSON-aware tool -- here jq.
(1) Ideally, the web service would provide a JSON API that would allow one to obtain the JSON directly, but as the URL you have is intended for viewing with a browser, some form of screen-scraping is needed. There is a certain amount of brittleness to such an approach, so here I'll just provide something that currently works:
wget -O - http://aqicn.org/city/beijing/m |
gawk 'BEGIN{RS="function"}
$1 ~/getAqiModel/ {
sub(/.*var model=/,"");
sub(/;return model;}/,"");
print}'
(gawk or an awk that supports multi-character RS can be used; if you have another awk, then first split on "function", using e.g.:
sed $'s/function/\\\n/g' # three backslashes )
The output of the above can be piped to the following jq command, which performs the filtering envisioned in (2) above.
(2)
jq -c '.iaqi | .[]
| select(.p? =="pm25" or .p? =="pm10") | [.p, .v[0]]'
The result:
["pm25",59]
["pm10",15]
I think your problem is that you have a single line HTML file that contains a script that contains a variable that contains the data you are looking for.
Your field delimiters are either "p":"pm100", "v":[ or a comma and some digits.
For pm25 this works, because it is the first, and there are no occurrences of ,21 or something similar before it.
However, for pm10, there are some that are associated with pm25 ahead of it. So the second field contains the empty string between ,21 and ,112
#karakfa has a hack that seems to work -- but he doesn't explain very well why it works.
What he does is use awk's record separator (which is usually a newline) and sets it to either of :, ,, or [. So in your case, one of the records would be "pm25", because it is preceded by a colon, which is a separator, and succeeded by a comma, also a separator.
Once it hits the matching content ("pm25") it sets a counter to 4. Then, for this and the next records, it counts this counter down. "pm25" itself, "v", the empty string between : and [, and finally reaches one when hitting the record with the number you want to output: 4 && ! 3 is false, 3 && ! 2 is false, 2 && ! 1 is false, but 1 && ! 0 is true. Since there is no execution block, awk simply prints this record, which is the value you want.
A more robust work would probably be using xpath to find the script, then use some json parser or similar to get the value.
chw21's helpful answer explains why your approach didn't work.
peak's helpful answer is the most robust, because it employs proper JSON parsing.
If you don't want to or can't use third-party utility jq for JSON parsing, I suggest using sed rather than awk, because awk is not a good fit for field-based parsing of this data.
$ sed -E 's/^.*"pm25"[^[]+\[([0-9]+).+"pm10"[^[]+\[([0-9]+).*$/\1 \2/' <<< "$AQIDATA"
59 15
The above should work with both GNU and BSD/OSX sed.
To read the result into variables:
read pm25 pm10 < \
<(sed -E 's/^.*"pm25"[^[]+\[([0-9]+).+"pm10"[^[]+\[([0-9]+).*$/\1 \2/' <<< "$AQIDATA")
Note how I've chosen lowercase variable names, because it's best to avoid all upper-case variables in shell programming, so as to avoid conflicts with special shell and environment variables.
If you can't rely on the order of the values in the source string, use two separate sed commands:
pm25=$(sed -E 's/^.*"pm25"[^[]+\[([0-9]+).*$/\1/' <<< "$AQIDATA")
pm10=$(sed -E 's/^.*"pm10"[^[]+\[([0-9]+).*$/\1/' <<< "$AQIDATA")
awk to the rescue!
If you have to, you can use this hacky way using smart counters with hand-crafted delimiters. Setting RS instead of FS transfers looping through fields to awk itself. Multi-char RS is not available for all awks (gawk supports it).
$ awk -v RS='[:,[]' '$0=="\"pm25\""{c=4} c&&!--c' file
59
$ awk -v RS='[:,[]' '$0=="\"pm10\""{c=4} c&&!--c' file
15