Meteorologist trying to create forecast pages - json

Let me preface this by saying my last computer class was in high school 1968-69. I'm sure I'm not using best practices and I always appreciate help there. Everything I do is self taught and this is the first truly original piece of code I've written.
In this case I'm trying to produce weather forecast pages. Here are samples for Honolulu. The data comes from NWS NDFD (national digital forecast database) via api.weather.gov in json. I pluck those variables and plug them into ImageMagick.
Two problems. The only way I could accommodate the forecast, which is a different length every time, was to use the caption command. But SUNNY, next to a three or four line forecast is jarring. Is there a better way or at least a way to limit the upper font size?
Also, this takes a lot longer than I expected. Is there a way for me to speed the process?
Thanks in advance for your help. I learn a lot here.
#!/bin/bash
#process forecast json
#8Oct2019
##geofffox
cd /tmp/json
#curl -o kofk https://api.weather.gov/gridpoints/OAX/31,93/forecast
#curl -o kofk https://api.weather.gov/gridpoints/AFG/381,359/forecast
#curl -o kofk https://api.weather.gov/gridpoints/APX/36,23/forecast
#curl -o kofk https://api.weather.gov/gridpoints/HFO/153,144/forecast
curl -o kofk https://api.weather.gov/gridpoints/OKX/66,65/forecast
counter=0
while [ $counter -le 13 ]
do
number["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].number')
name["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].name')
start["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].startTime')
end["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].endTime')
swch["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].isDaytime')
temp["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].temperature')
wind["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].windSpeed')
wdir["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].windDirection')
shrt["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].shortForecast')
long["$counter"]=$(cat kofk | jq -r '.properties.periods['$counter'].detailedForecast')
echo $counter
((counter++))
done
innerLoop=0
rm /var/www/html/output/json/kofk/*.png
while [ $innerLoop -le 13 ]
do
echo $innerLoop
convert -size 1920x1080 xc:blue PNG32:/var/www/html/output/json/kofk/kofk.png
convert -background rgba\(0,0,0,0.001\) -fill white -stroke black -strokewidth 3 -gravity west -font Open-Sans-Extrabold -size 700x400 caption:"${shrt["$innerLoop"]^^}" \( +clone -shadow 70x12+5+5 \) +swap \
-flatten -trim +repage /var/www/html/output/json/kofk/shrt["$innerLoop"].png
convert /var/www/html/output/json/kofk/kofk.png -gravity northwest -pointsize 50 -fill white -font Open-Sans-Bold -stroke black -strokewidth 2 -draw "text 950,115 '${name["$innerLoop"]^^}'" /var/www/html/output/json/kofk/kofk["$innerLoop"].png
if ${swch["$innerLoop"]} == false; then
convert /var/www/html/output/json/kofk/kofk["$innerLoop"].png -pointsize 50 -fill white -font Open-Sans-Bold -stroke black -strokewidth 2 -draw "text 950 700 'DAYTIME HIGH:'" /var/www/html/output/json/kofk/kofk["$innerLoop"].png
else
convert /var/www/html/output/json/kofk/kofk["$innerLoop"].png -pointsize 50 -fill white -font Open-Sans-Bold -stroke black -strokewidth 2 -draw "text 950 700 'OVERNIGHT LOW:'" /var/www/html/output/json/kofk/kofk["$innerLoop"].png
fi
convert /var/www/html/output/json/kofk/kofk["$innerLoop"].png -pointsize 200 -fill black -font Open-Sans-Extrabold -draw "text 1405 705 '${temp["$innerLoop"]^^}°'" -fill white -stroke black -strokewidth 5 -draw "text 1400 700 '${temp["$innerLoop"]^^}°'" /var/www/html/output/json/kofk/kofk["$innerLoop"].png
convert /var/www/html/output/json/kofk/kofk["$innerLoop"].png -pointsize 50 -fill white -font Open-Sans-Bold -stroke black -strokewidth 2 -draw "text 950 750 'WIND: ${wdir["$innerLoop"]}"" ${wind["$innerLoop"]^^}'" /var/www/html/output/json/kofk/kofk["$innerLoop"].png
convert -composite -gravity west -geometry +950-175 /var/www/html/output/json/kofk/kofk["$innerLoop"].png /var/www/html/output/json/kofk/shrt["$innerLoop"].png /var/www/html/output/json/kofk/kofk["$innerLoop"].png
rm /var/www/html/output/json/kofk/shrt["$innerLoop"].png
((innerLoop++))
done
exit

Related

Shell script for Multiple Linux Servers Health report in html with column color change based on condition

This code will fetch the data from Multiple servers and store in CSV file.
I am trying to get same data into HTML table format with the condition base columns, like if disk free space is less in 20% then columns of that server becomes yellow if less then 10% then becomes red.
rm -f /tmp/health*
touch /tmp/health.csv
#Servers File Path
FILE="/tmp/health.csv"
USR=root
#Create CSV File Haeader
echo " Date, Hostname, Connectivity, Root-FreeSpace, Uptime, OS-version, Total-ProcesCount, VmToolsversion, ServerLoad, Memory, Disk, CPU, LastReboot-Time, UserFailedLoginCount, " > $FILE
for server in `more /root/servers.txt`
do
_CMD="ssh $USR#$server"
Date=$($_CMD date)
Connectivity=$($_CMD ping -c 1 google.com &> /dev/null && echo connected || echo disconnected)
ip_add=`ifconfig | grep "inet addr" | head -2 | tail -1 | awk {'print$2'} | cut -f2 -d:`
RootFreeSpace=$($_CMD df / | tail -n +2 |awk '{print $5}')
Uptime=$($_CMD uptime | sed 's/.*up \([^,]*\), .*/\1/')
OSVersion=$($_CMD cat /etc/redhat-release)
TotalProcess=$($_CMD ps axue | grep -vE "^USER|grep|ps" | wc -l)
VmtoolStatus=$($_CMC vmtoolsd -v |awk '{print $5}')
ServerLoad=$($_CMD uptime |awk -F'average:' '{ print $2}'|sed s/,//g | awk '{ print $2}')
Memory=$($_CMD free -m | awk 'NR==2{printf "%.2f%%\t\t", $3*100/$2 }')
Disk=$($_CMD df -h | awk '$NF=="/"{printf "%s\t\t", $5}')
CPU=$($_CMD top -bn1 | grep load | awk '{printf "%.2f%%\t\t\n", $(NF-2)}')
Lastreboottime=$($_CMD who -b | awk '{print $3,$4}')
FailedUserloginCount=$($_CMD cat /var/log/secure |grep "Failed" | wc -l)
#updated data in CSV
echo "$Date,$HostName,$Connectivity,$RootFreeSpace,$Uptime,$OSVersion,$TotalProcess,$VmtoolStatus,$ServerLoad,$Memory,$Disk,$CPU,$Lastreboottime,$FailedUserloginCount" >> $FILE
done

How can I minify JSON in a shell script?

I've been looking for a way to uglify some JSON while in my bash console. This help using it afterward in another command (for example, to pass json inline to httpie)
Giving:
{
"foo": "lorem",
"bar": "ipsum"
}
I want to obtain:
{"foo":"lorem","bar":"ipsum"}
NOTE: this question is intentionnaly greatly inspired by it's pretty-print counterpart. However, googling for bash minify json didn't give me a proper result, hence this questions for the minify/uglify.
You can use jq -c (compact) option.
jq -c . < input.json
TL;DR
no install
python -c 'import json, sys;json.dump(json.load(sys.stdin), sys.stdout)' < my.json
very fast (with jj)
jj -u < my.json
Perf benchmark
Here's the script, using hyperfine:
#!/usr/bin/env bash
tmp=$(mktemp json.XXX)
tmp_md=$(mktemp md.XXX)
trap "rm $tmp $tmp_md" EXIT
cat <<JSON > $tmp
{
"foo": "lorem",
"bar": "ipsum"
}
JSON
hyperfine \
--export-markdown $tmp_md \
--warmup 100 \
"jj -u < $tmp" \
"yq eval -j -I=0 < $tmp" \
"xidel -s - -e '\$json' --printed-json-format=compact < $tmp" \
"jq --compact-output < $tmp" \
"python3 -c 'import json, sys;json.dump(json.load(sys.stdin), sys.stdout)' < $tmp" \
"ruby -r json -e 'j JSON.parse \$stdin.read' < $tmp"
pbcopy < $tmp_md
The result on my mac — MacBook Air (M1, 2020), 8 GB:
Command
Mean [ms]
Min [ms]
Max [ms]
Relative
jj -u < json.p72
1.3 ± 0.2
0.9
2.7
1.00
yq eval -j -I=0 < json.p72
4.4 ± 0.4
3.8
7.8
3.37 ± 0.65
xidel -s - -e '$json' --printed-json-format=compact < json.p72
5.5 ± 0.3
5.0
6.5
4.19 ± 0.77
python3 -c 'import json, sys;json.dump(json.load(sys.stdin), sys.stdout)' < json.p72
14.0 ± 0.4
13.4
15.0
10.71 ± 1.89
jq --compact-output < json.p72
14.4 ± 2.0
13.2
33.6
11.02 ± 2.45
ruby -r json -e 'j JSON.parse $stdin.read' < json.p72
47.3 ± 0.6
46.1
48.5
36.10 ± 6.32
Result for a large JSON file (14k lines):
http https://france-geojson.gregoiredavid.fr/repo/regions.geojson | jj -p > $tmp
Command
Mean [ms]
Min [ms]
Max [ms]
Relative
jj -u < json.wFY
3.4 ± 0.7
2.7
12.2
1.00
jq --compact-output < json.wFY
35.1 ± 0.4
34.5
36.1
10.24 ± 2.23
python3 -c 'import json, sys;json.dump(json.load(sys.stdin), sys.stdout)' < json.wFY
47.4 ± 0.5
46.3
48.7
13.82 ± 3.01
xidel -s - -e '$json' --printed-json-format=compact < json.wFY
55.5 ± 1.2
54.7
63.5
16.17 ± 3.53
ruby -r json -e 'j JSON.parse $stdin.read' < json.wFY
94.9 ± 0.7
93.8
96.8
27.65 ± 6.02
yq eval -j -I=0 < json.wFY
3087.0 ± 26.6
3049.3
3126.8
899.63 ± 195.81
And here is the pretty print counterpart benchmark
yq worked for me, via utilization of input file (containing the prettified JSON)
yq eval -j -I=0 uglify-test.txt
Docs link: https://mikefarah.gitbook.io/yq/usage/convert
With xidel:
xidel -s input.json -e '$json' --printed-json-format=compact
#or
xidel -s input.json -e 'serialize-json($json)'
{"foo": "lorem", "bar": "ipsum"}
Interesting "benchmark", Ulysse BN.
I couldn't test jj, but on my old cpu these are my results:
var='{
"foo": "lorem",
"bar": "ipsum"
}'
time (for i in {1..100}; do python -c 'import json, sys;json.dump(json.load(sys.stdin), sys.stdout)' <<< "$var" >& /dev/null; done)
real 0m10.813s
user 0m7.532s
sys 0m5.798s
time (for i in {1..100}; do jq --compact-output <<< "$var" >& /dev/null; done)
real 0m10.500s
user 0m1.835s
sys 0m0.769s
time (for i in {1..100}; do xidel -se '$json' --printed-json-format=compact <<< "$var" >& /dev/null; done)
real 0m2.250s
user 0m1.692s
sys 0m0.889s
jq-minify
Here is a bash script that will write back to the file minified
works with bash v3.2+ and jq v1.6+
#!/usr/bin/env bash
set -eu
path=
options=()
# change -c to -r to get pretty-print
set -- "$#" -c .
for arg; do
if [ -f "$arg" ]; then
if [ -n "$path" ]; then
echo "Cannot specify multiple paths to jq-minify" >&2
exit 1
fi
path="$arg"
else
options+=("$arg")
fi
done
tmp=$(mktemp)
jq "${options[#]}" "$path" >"$tmp"
cat "$tmp" >"$path"

Tk/Tcl scale widget interval issue

I'm building a scale widget going from 10 to 100 with a spacing (interval) of 10.
When the resolution=1 the result is:
scale widget with resolotion=1
But when the resolution=3 the scale is going from 9 to 99 and no more 10->100,
the results is showed here:
scale widget with resolution=3
The code is:
#!/bin/sh
####################################################################### \
exec /sw/freetools/tk/8.5.6/Linux/rh5/x86_64/bin/wish8.5 "$0" ${1+"$#"}
package require Tk
font create FONT1 -family {DejaVu Sans Mono} -size 7 -weight normal
font create FONT2 -family {DejaVu Sans} -size -8 -weight normal -slant roman
ttk::style configure rule1.TCombobox -background lavender \
-foreground black \
-fieldbackground mistyrose2 \
-padding 0.5
ttk::style map rule1.TCombobox -fieldbackground [list readonly gray98]
frame .frame_F1 -background gray98 -highlightbackground black -highlightthickness 1
frame .frame_F1.top -background gray98 -highlightbackground red -highlightthickness 0
frame .frame_F1.bottom -background gray98 -highlightbackground blue -highlightthickness 0
pack .frame_F1 -padx 10 -pady 10
pack .frame_F1.top -side top -anchor w
pack .frame_F1.bottom -side bottom -anchor w
label .frame_F1.bottom.scale_text1 -text "scale" -foreground black -background gray98 -font FONT1 -justify left
entry .frame_F1.bottom.scale_value -textvariable ::SCALE -foreground black -width 5 -justify right -state readonly
scale .frame_F1.top.scale -orient horizontal \
-variable ::SCALE \
-length 210 \
-from 10 -to 100 \
-tickinterval 10 \
-resolution 1 \
-showvalue false \
-background gray98 \
-font FONT2 \
-foreground black \
-width 12 \
-highlightthickness 0 \
-borderwidth 2 \
-sliderrelief groove \
-sliderlength 18
pack .frame_F1.top.scale
grid .frame_F1.bottom.scale_text1 -column 0 -row 1
grid .frame_F1.bottom.scale_value -column 1 -row 1
How to correct this and to have from 10 to 100 ?

extract token from curl result by shell script

I write this script
#!/bin/bash
# cm.sh
curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "admin",
"domain": { "id": "default" },
"password": "secret"
}
}
}
}
}' \
"http://localhost/identity/v3/auth/tokens" ; echo
echo $tokenizer1
echo $tokenizer2
But all of them(awk or sed) it's the same
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 540 100 312 100 228 312 228 0:00:01 --:--:-- 0:00:01 5142
My goal is to put the token in a variable for later.
Thanks guys in advance.
Instead of using the direct result of cURL, you could save the result in a file, and use your grep command on it.
Something like this maybe :
curl -o boulou.txt http://localhost/identity/v3/auth/tokens && cat boulou.txt | grep "X-Subject-Token" | awk '{printf $2}'
Edit, if you just want you desired output, add the --silent to the cURL command :
curl -o boulou.txt http://localhost/identity/v3/auth/tokens --silent && cat boulou.txt | grep "X-Subject-Token" | awk '{printf $2}'
Edit 2: If you want to export it, and delete your file, you could use something like this :
export OS_TOKEN=$(curl -o billy.txt hhttp://localhost/identity/v3/auth/tokens --silent && cat billy.txt | grep "X-Subject-Token" | awk '{printf $2}') && rm billy.txt
"How do I use grep/awk to extract a header in a field from curl when I pass it a JSON document when the contents is stored in a variable?" is a very tricky and unique problem indeed.
However, if you gradually mock out every part of your code to narrow it down, you'll discover that this is the much easier question you could have researched or asked instead:
How do I use grep/awk on contents from a variable?
I have a variable containing HTTP headers, and I want to extract the value of one of them. Here's an example:
variable='Foo: 1
Bar: 2
Baz: 3'
This is what I've tried to get 2 from Bar:
# Just hangs
tokenizer1=$variable `grep "Bar" | awk '{printf $2}'`
# Is empty
tokenizer2=$variable | `grep "Bar" | awk '{printf $2}'`
The answer here is to use echo to pipe the contents so that grep can read it on stdin:
tokenizer3=$(echo "$variable" | grep "Bar" | awk '{printf $2}')
This is easily applied to your example:
tokenizer3=$(echo "$token" | grep "X-Subject-Token" | awk '{printf $2}')
echo "The value is $tokenizer3"

Understanding MLT melt mixer luma mix (fade) duration?

Related to Understanding/controlling MLT melt slideshow?, I am trying to understand how does melt's luma mixer work, especially in context of short (small frame number) durations.
For instance, if I use something like this specification tmppics/pic_01.jpg length=6 -mix 2 -mixer luma I would have expected 2 frames of fade of pic_01, then 2 frames full pic_01, then 2 frames of fade again - however, I've both experienced results that are like this, and those that are not.
To clarify this, I've developed a bash script, which uses ImageMagick convert to generate test images, then melt to make a "slideshow" with fades out of these images, ffmpeg to convert that to animated .gif, then ImageMagick convert and montage to obtain a film strip (sprite sheet) of the video .gif (Ubuntu 18.04, melt 6.6.0, ffmpeg version 3.4.4-0ubuntu0.18.04.1, ImageMagick 6.9.7-4 Q16 x86_64 20170114).
Here is the bash script, melt-test-strip.sh:
#!/usr/bin/env bash
FRAMERATE=25
echo "
description=DV PAL
frame_rate_num=$FRAMERATE
frame_rate_den=1
width=720
height=576
progressive=0
sample_aspect_num=59
sample_aspect_den=54
display_aspect_num=4
display_aspect_den=3
colorspace=601
" > my-melt.profile
mkdir tmppics
convert -background lightblue -fill blue -size 3840x2160 -pointsize 200 -gravity center label:"Test A" tmppics/pic_01.jpg
convert -background lightblue -fill blue -size 2160x3840 -pointsize 200 -gravity center label:"Test B" tmppics/pic_02.jpg
if [ -z "$IMGDURATIONF" ]; then
IMGDURATIONF=6 # picture duration, frames
fi
if [ -z "$FADEDURATIONF" ]; then
FADEDURATIONF=2 # single-end fade duration, frames
fi
melt -verbose -profile ./my-melt.profile \
tmppics/pic_01.jpg length=$IMGDURATIONF \
tmppics/pic_02.jpg length=$IMGDURATIONF -mix $FADEDURATIONF -mixer luma \
colour:black length=$IMGDURATIONF -mix $FADEDURATIONF -mixer luma \
-consumer avformat:meltout.mp4 vcodec=libx264 an=1
# auxiliary: just for creating sprite sheet/film strip:
melt -verbose -profile ./my-melt.profile tmppics/pic_01.jpg length=$IMGDURATIONF -consumer avformat:meltout-01.mp4 vcodec=libx264 an=1
melt -verbose -profile ./my-melt.profile tmppics/pic_02.jpg length=$IMGDURATIONF -consumer avformat:meltout-02.mp4 vcodec=libx264 an=1
melt -verbose -profile ./my-melt.profile colour:black length=$IMGDURATIONF -consumer avformat:meltout-b.mp4 vcodec=libx264 an=1
# convert to gif to obtain sprite sheet/film strip from:
ffmpeg \
-i meltout.mp4 \
-r 25 \
-vf scale=256:-1 \
-y meltout.gif
# auxiliary: just for creating sprite sheet/film strip:
ffmpeg -i meltout-01.mp4 -r $FRAMERATE -vf scale=256:-1 -y meltout-01.gif
ffmpeg -i meltout-02.mp4 -r $FRAMERATE -vf scale=256:-1 -y meltout-02.gif
ffmpeg -i meltout-b.mp4 -r $FRAMERATE -vf scale=256:-1 -y meltout-b.gif
convert meltout.gif -coalesce meltoutc.gif
convert meltout-01.gif -coalesce meltoutc-01.gif
convert meltout-02.gif -coalesce meltoutc-02.gif
convert meltout-b.gif -coalesce meltoutc-b.gif
FRAMETHICK=5
#~ montage temp.gif -tile x1 -geometry '1x1+0+0<' -border 5 -bordercolor "rgb(200, 200, 200)" -label 'Image' -quality 100 meltout.png
# "%p index of image in current image list" is "t=> index of current image (s) in list" in fx:
montage -label 'Frame %[fx:t+1]/%n' meltoutc.gif -tile x1 -geometry '1x1+0+0<' -frame $FRAMETHICK -bordercolor "rgb(200, 200, 200)" -quality 100 meltout.png
# here from the .gif - otherwise for replicating images: montage in.jpg +clone +clone +clone -tile x4 -geometry +0+0 out.jpg
montage -label 'Frame %[fx:t+1]/%n' meltoutc-01.gif -tile x1 -geometry '1x1+0+0<' -frame $FRAMETHICK -bordercolor "rgb(200, 200, 200)" -quality 100 meltout-01.png
montage -label 'Frame %[fx:t+1]/%n' meltoutc-02.gif -tile x1 -geometry '1x1+0+0<' -frame $FRAMETHICK -bordercolor "rgb(200, 200, 200)" -quality 100 meltout-02.png
montage -label 'Frame %[fx:t+1]/%n' meltoutc-b.gif -tile x1 -geometry '1x1+0+0<' -frame $FRAMETHICK -bordercolor "rgb(200, 200, 200)" -quality 100 meltout-b.png
# for offsetting:
# (gif) frame width/height
fw=$(convert meltoutc-01.gif[0] -format "%w" info:)
fh=$(convert meltoutc-01.gif[0] -format "%w" info:)
# strip width/height
sw=$(convert meltout.png -format "%w" info:)
sh=$(convert meltout.png -format "%h" info:)
echo fw $fw fh $fh sw $sw sh $sh
convert -size $(( (IMGDURATIONF-FADEDURATIONF)*(fw+2*FRAMETHICK) ))x$sh xc:white meltout-02.png +append meltout-02B.png
# IMGDURATIONF-FADEDURATIONF to get to the start of second clip; +IMGDURATIONF from there to get to end of second clip, and -FADEDURATIONF from there to get to start of third clip
convert -size $(( (IMGDURATIONF-FADEDURATIONF+IMGDURATIONF-FADEDURATIONF)*(fw+2*FRAMETHICK) ))x$sh xc:white meltout-b.png +append meltout-bB.png
montage -geometry '+0+0' meltout.png meltout-01.png meltout-02B.png meltout-bB.png -tile 1x meltout-all.png
eog meltout-all.png
So, if you call bash melt-test-strip.sh, you get the defaults, IMGDURATIONF=6 and FADEDURATIONF=2, for which the output is this (click for full size):
The only way the luma-mixed result (on top) makes sense to me, is if the starting frame of the new clip participates in the mix with 0% - which is why, for the second image clip, we observe 1 frame fade + 3 frames full + 1 frame fade (instead of 2 frames fade + 2 frames full + 2 frames fade, which I'd expect).
Is this correct?
Reading https://www.mltframework.org/docs/melt/#mixes I cannot really tell if this interpretation is correct.
If I run the script with other parameters, like IMGDURATIONF=8 FADEDURATIONF=3 bash melt-test-strip.sh, then the output is:
... in which case the interpretation holds (if every new clip's first frame participates in the mix with 0%, that explains why we're seeing 2 frames fade + 3 frames full + 2 frames fade for second clip = 7 frames in all, instead of the requested 8) - but now I'm not sure if this is just an artefact of my script (as opposed to the true behavior of melts mix).
Can anyone confirm if this is how melts mixer works - and if not, explain how can it be understood?