How to get JSON from cURL as array in a batch file - json

Hi I'm very new in batch files. I try to do something simular to my bash script. So here is my problem:
I want to get all versions/tags from a github repository to generate a configuration file for the php documentor sami. But how can I write the JSON into a variable in batch to get the versions? In my bash script I did this and it's working fine:
function jsonDecode() {
json=$1
key=$2
echo ${json} | jq -r ${key}
}
ghUser="MisterMarlu"
ghRepo="sentence"
json=$(curl "https://api.github.com/repos/${ghUser}/${ghRepo}/tags")
versions=$(echo "${json}" | jq -c ".[]")
for version in ${versions[#]}; do
versionNumber=$(jsonDecode ${version} ".name")
echo " ->add( '${versionNumber}', '${versionNumber}' )" >> ${config}
done
# Here comes alot of code below this for loop..
This will output "v0.0.1" and "v0.0.2". Bus how can I do this in a batch file?
EDIT
Here is the JSON response where I need just the "name" as array:
[
{
"name": "v0.0.2",
"zipball_url": "https://api.github.com/repos/MisterMarlu/sentence/zipball/v0.0.2",
"tarball_url": "https://api.github.com/repos/MisterMarlu/sentence/tarball/v0.0.2",
"commit": {
"sha": "82c4b6d74cc16816104934114766f0328e77ee66",
"url": "https://api.github.com/repos/MisterMarlu/sentence/commits/82c4b6d74cc16816104934114766f0328e77ee66"
},
"node_id": "MDM6UmVmMTMzMDM1MDMxOnYwLjAuMg=="
},
{
"name": "v0.0.1",
"zipball_url": "https://api.github.com/repos/MisterMarlu/sentence/zipball/v0.0.1",
"tarball_url": "https://api.github.com/repos/MisterMarlu/sentence/tarball/v0.0.1",
"commit": {
"sha": "0cf1a83a51716da3f42915c9eab571166845bb0b",
"url": "https://api.github.com/repos/MisterMarlu/sentence/commits/0cf1a83a51716da3f42915c9eab571166845bb0b"
},
"node_id": "MDM6UmVmMTMzMDM1MDMxOnYwLjAuMQ=="
}
]

To process the output of another program you need a for /f
parsing the lines filtered by findstr
:: Q:\Test\2018\06\12\SU_50811698.cmd
#Echo off & SetLocal EnableDelayedExpansion
Set "ghUser=MisterMarlu"
Set "ghRepo=sentence"
Set "Version="
For /f "tokens=1,2 delims=:, " %%U in ('
curl "https://api.github.com/repos/%ghUser%/%ghRepo%/tags" 2^>Nul ^| findstr /i "\"name\""
') do Set "Version=!Version!,%%~V"
If defined Version (set "Version=%Version:~1%") Else (Set "Version=n/a")
Set Version
Sample output:
> Q:\Test\2018\06\12\SU_50811698.cmd
Version=v0.0.2,v0.0.1
You are aware that batch has no real arrays?
Just an alternative in PowerShell:
$ghUser="MisterMarlu"
$ghRepo="sentence"
$URL = "https://api.github.com/repos/$ghUser/$ghRepo/tags"
$Json=(curl.exe $URL)|ConvertFrom-json
$Json | Select name
name
----
v0.0.2
v0.0.1

With Xidel it's simple as:
xidel -s "https://api.github.com/repos/MisterMarlu/sentence/tags" -e "join($json()/name,',')"
This puts out: v0.0.2,v0.0.1.
To export this as $config/%config%...
Bash:
eval "$(xidel -s "https://api.github.com/repos/MisterMarlu/sentence/tags" -e '
config:=join(
$json()/name,
","
)' --output-format=bash
)"
Batch:
FOR /F "delims=" %%A IN ('xidel.exe -s "https://api.github.com/repos/MisterMarlu/sentence/tags" -e ^"
config:^=join^(
$json^(^)/name^,
'^,'
^)^" --output-format^=cmd
') DO %%A
or...
FOR /F "delims=" %%A IN ('xidel.exe -s "https://api.github.com/repos/MisterMarlu/sentence/tags" -e "config:=join($json()/name,',')" --output-format=cmd') DO %%A

Related

Correct quote settings for nested objects with jq in windows cmd for special characters

I have the following json input file( I know it's not a good structure, but I cannot change it)
{
"TEST1": {
"ONE": "value",
"TWO": "value-1",
},
"TEST2": [
{
"A": [
"test": "test"
],
"B-B":[
"test": "test"
]
}
]
}
And I want to process it in batch loop. But sometimes there are special characters inside which needs to be escaped.
for /f %%i in ('type %LOCAL_PATH_FILE% ^| %JQ% -r ".TEST2[] | keys[]"') do (
set SUB=%%i
ECHO sub is !SUB!
for /f %%j in ('type %LOCAL_PATH_FILE% ^| %JQ% -r ".TEST2[].!SUB![] | keys[]"') do (
set CONTENT=%%j
ECHO content path is !CONTENT!
)
)
The current result looks like:
jq: error: syntax error, unexpected IDENT, expecting $end (Windows cmd shell quoting issues?) at <top-level>, line 1:
.TEST2[].B-B[] | keys[]
jq: 1 compile error
The process tried to write to a nonexistent pipe.
I know it should somehow possible to escape it via quota, but I'm not able to find the correct syntax for it.
Maybe someone can help me :)

Windows Batch - Extract values for ffmpeg processing

I have dozens of json files, and I am trying to find two values in each of them and assign the results to two separate variables for ffmpeg processing.
An example json file looks like this:
{
"year": "2018",
"track": "12",
... other data omitted
}
I wish to extract 2018 and 12 so that I can use them in the following ffmpeg command:
ffmpeg -i "same_file_name_as_json.m4a" -metadata:s:a:0 year=2018 --metadata:s:a:0 track=12 -acodec libmp3lame "same_file_name_as_json.mp3"
Is it possible to write a single batch file to achieve the desired result? Any help would be greatly appreciated as I am a complete novice at findstr and setting variables. Thank you.
EDITED:
set "year=" & set "track="
for %%i in (*.json) do (
for /f "usebackq tokens=1,2 delims={:}, " %%a in ("%%i") do (
set "%%~a=%%~b"
if defined year if defined track goto :CONT
)
:CONT
C:\ffmpeg -i "%%~ni.m4a" -metadata:s:a:0 year=%year% -metadata:s:a:0 track=%track% -acodec libmp3lame "%%~ni.mp3"
)
pause
Windows batch scripting does not understand the JSON file format, so it is better to use a language natively supports it. It is not the best idea to treat JSON as "normal" text, because only a slight change (for instance, added, deleted, or moved line-breaks) that do not violate the JSON format can still make big troubles then.
That said, given that the JSON file exactly appears as you have shown it and it features Unix- or DOS/Windows-style line-breaks (that is, a carriage-return character followed by a line-feed character), this code could work for you:
for /F "usebackq tokens=1,2 delims={:}, " %%M in ("file.json") do set "%%~M=%%~N"
echo year = %year%
echo track = %track%
If you have got a huge JSON file you do not want to unnecessarily fully process, you could use this code instead:
set "year=" & set "track="
for /F "usebackq tokens=1,2 delims={:}, " %%M in ("file.json") do (
set "%%~M=%%~N"
if defined year if defined track goto :CONT
)
:CONT
echo year = %year%
echo track = %track%
If the (non-array) values you want to extract may also contain one of the defined delimiters ({, :, }, ,, SPACE), you could extend the code to this, given that the values do not contain the characters *, ?, <, >:
set "year=" & set "track="
for /F "usebackq tokens=1,* delims={:}, " %%M in ("file.json") do (
for %%K in (%%N) do set "%%~M=%%~K"
if defined year if defined track goto :CONT
)
:CONT
echo year = %year%
echo track = %track%
To prevent the script from assigning unwanted superfluous variables, you may try this:
for /F "usebackq tokens=1,2 delims={:}, " %%M in ("file.json") do (
if "%%~M"=="year" (set "%%~M=%%~N") else if "%%~M"=="track" set "%%~M=%%~N"
)
echo year = %year%
echo track = %track%
Or this, which prepreocesses the data by the findstr command and filters out the desired lines:
for /F "tokens=1,2 delims={:}, " %%M in ('
findstr /R /C:"^ *\"year\" *:" /C:"^ *\"track\" *:" "file.json"
') do set "%%~M=%%~N"
echo year = %year%
echo track = %track%
Based on your edit, let me suggest to use the last of the above methods, because there is no goto :CONT, which cannot be used within loops as it breaks the block context, and it does not assign additional unwanted variables. Since variables are written and read within the loop body, you have to enable and apply delayed variable expansion. I would do all that the following way:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem /* Iterate over the `*.json` files in the current working directory (`%CD%`);
rem to use the parent directory of this script, use `%~dp0*.json` instead: */
for %%I in ("*.json") do (
rem // Store name of current JSON file in variable:
set "name=%%~nI"
rem // Clear variables for later check for availability:
set "year=" & set "track="
rem // Process the current JSON file:
for /F "tokens=1,2 delims={:}, " %%M in ('
findstr /R /C:"^ *\"year\" *:" /C:"^ *\"track\" *:" "%%~I"
') do (
rem // Assign year and track variables:
set "%%~M=%%~N"
rem // Check of both year and track are available:
if defined year if defined track (
rem // Toggle delayed expansion to avoid troubles with `!`:
setlocal EnableDelayedExpansion
rem // Eventually execute `ffmpeg` tool using all the derived data:
ffmpeg -i "!name!.m4a" -metadata:s:a:0 year=!year! -metadata:s:a:0 track=!track! -acodec libmp3lame "!name!.mp3"
endlocal
)
)
)
endlocal
exit /B
I have dozens of json files...
Windows' cmd doesn't support JSON, so you'd have to resort to PowerShell, or use an external tool that does. You might find xidel interesting.
To extract the value for "year" and "track":
xidel -s input.json -e "$json/(year,track)"
#or
xidel -s input.json -e "$json/year,$json/track"
2018
12
To export to a variable %year% and %track%:
FOR /F "delims=" %A IN ('xidel -s input.json -e "$json/(year:=year,track:=track)" --output-format^=cmd') DO %A
#or
FOR /F "delims=" %A IN ('xidel -s input.json -e "year:=$json/year,track:=$json/track" --output-format^=cmd') DO %A
You don't however need variables to create the strings (ffmpeg commands) you want. xidel can do that too.
You could use a FOR-loop to iterate over all your JSON-files...
FOR %A IN (*.json) DO #xidel -s %A -e "$json/concat('ffmpeg -i \"%~nA.m4a\" -metadata:s:a:0 year=',year,' --metadata:s:a:0 track=',track,' -acodec libmp3lame \"%~nA.mp3\"')"
ffmpeg -i "name-of-json-file.m4a" -metadata:s:a:0 year=2018 --metadata:s:a:0 track=12 -acodec libmp3lame "name-of-json-file.mp3"
...but to call xidel for each and every JSON-file is very inefficient. xidel can do this much more efficiently.
xidel's equivalent for FOR %A IN (*.json) DO #ECHO %A is xidel -se "file:list(.,false(),'*.json')"
Then you can use the following query to process all your JSON-files at once:
xidel -se "for $x in file:list(.,false(),'*.json') return json-doc($x)/concat('ffmpeg -i \"',replace($x,'json','m4a'),'\" -metadata:s:a:0 year=',year,' --metadata:s:a:0 track=',track,' -acodec libmp3lame \"',replace($x,'json','mp3'),'\"')"
Prettified command/query:
xidel -se ^"^
for $x in file:list(.,false(),'*.json') return^
json-doc($x)/concat(^
'ffmpeg -i \^"',^
replace($x,'json','m4a'),^
'\^" -metadata:s:a:0 year=',^
year,^
' --metadata:s:a:0 track=',^
track,^
' -acodec libmp3lame \^"',^
replace($x,'json','mp3'),^
'\^"'^
)^
"

Getting variable from jq.exe output in Windows batch reading JSON

I managed having a JSON file created by ffprobe which contains basic info about a video stream in a MKV container. By jq-win64.exe "[.format.duration]" %%~ni.mkv.json the duration of the movie is read correctly from the file and jq echos ["1:36:55.184000"]. Now I want to store this value in a global variable of my script for further processing. I tried several approaches but each of them led to errors and/or left %duration% empty. I tried e.g.
for %%i in (*.mkv) do (
SETLOCAL ENABLEDELAYEDEXPANSION
for /F "tokens=* USEBACKQ" %%F IN ('_tools\jq\jq-win64.exe "[.format.duration]" %%~ni.mkv.json') DO (SET duration=%%F)
echo Duration is: %duration%
ENDLOCAL
)
but could not manage to echo %duration%. I think it can't be that hard, most likely I don't do the syntax right on Windows batch. Any ideas? Here is the JSON file as well:
{
"format": {
"filename": "TestFile_1080p_26Mbs_8bit_BT709.mkv",
"nb_streams": 1,
"nb_programs": 0,
"format_name": "matroska,webm",
"format_long_name": "Matroska / WebM",
"start_time": "0:00:00.000000",
"duration": "1:36:55.184000",
"size": "17.586597 Gibyte",
"bit_rate": "25.978148 Mbit/s",
"probe_score": 100,
"tags": {
"title": "TestFile",
"encoder": "libmakemkv v1.14.4 (1.3.5/1.4.7) win(x64-release)",
"creation_time": "2019-08-17T21:01:18.000000Z"
}
}
}
Here's a batch-file solution based upon my understanding after the comments:
For /F Tokens^=2Delims^=^" %%F In (
'_tools\jq\jq-win64.exe "[.format.duration]" "%%~ni.mkv.json" 2^>NUL')Do (
Set "duration=%%F"
SetLocal EnableDelayedExpansion
Echo( !duration!
EndLocal
)
If all you want is the duration, then there's no need for intermediate JSONs, because FFprobe can also tell you that:
ffprobe.exe -v 0 -i <input> -show_entries format=duration -of compact=p=0:nk=1
1:36:55.184000
Create a variable:
FOR /F "delims=" %%A IN (
'ffprobe.exe -v 0 -i <input> -show_entries format=duration -of compact=p=0:nk=1'
) DO SET duration=%%A
SET duration=1:36:55.184000
If you still want to parse FFprobe's JSON, then there's no need to create json-files either, as you can simply pipe it to JQ instead:
ffprobe.exe -v 0 -i <input> -show_format -of json | jq.exe -r .format.duration
1:36:55.184000
Create a variable:
FOR /F "delims=" %%A IN (
'ffprobe.exe -v 0 -i <input> -show_format -of json ^| jq.exe -r .format.duration'
) DO SET duration=%%A
SET duration=1:36:55.184000

How to read JSON data in batch script

I have a JSON file, named "Config.json", that looks like this:
{ "RunEnvironment": "DEV"}
In a batch file under the same directory, I want to read the value of the "RunEnvironment" element.
My batch script would look like:
if [jsonElement] == 'DEV' (
:: do something
)
Can anyone show me how to do this?
In PowerShell for example, you could do:
> If((Get-Content '.\Config.json'|ConvertFrom-Json).RunEnvironment -eq 'DEV'){"is DEV:Whatever"}
is DEV:Whatever
To be on topic on cmd line
> for /f "tokens=1,2 delims=:{} " %A in (Config.json) do #If "%~B"=="Dev" #Echo (%~A = %~B)
(RunEnvironment=DEV)
In a batch file
#Echo off
for /f "tokens=1,2 delims=:{} " %%A in (Config.json) do (
If "%%~B"=="Dev" Echo (%%~A=%%~B^)
)
The Echo (%%~A=%%~B^) could be replaced with whatever you plan to do.
If all you need to do is perform a specific command only if the value of RunEnvironment is DEV then this should be all you require, (from a batch-file or the command-line):
"%__APPDIR__%FindStr.exe" /IRC:"\"RunEnvironment\":\ \ *\"DEV\"" "Config.json">NUL 2>&1&&Echo Your command here
You'd simply replace Echo Your command here with your intended command.
Yo can do in this way
#echo off
set string={ "RunEnvironment": "DEV" }
for /f "tokens=3,5" %%a in ('echo %string%') do set d=%%~a
if "%d%" == "DEV" echo %d%
pause & goto :EOF
In a batch file:
#ECHO off
SET "FilenameForJsonFile=Config.json"
SET "FilenameForRunEnvironment=RunEnvironment.txt"
powershell -Command "Select-String -Pattern 'RunEnvironment\"\: \".*?\"' .\%FilenameForJsonFile% ^| ForEach-Object { $_.Matches.Value.substring(18,$_.Matches.Value.Length-19) }>%FilenameForRunEnvironment%
FOR /f "delims=" %%x IN (%FilenameForRunEnvironment%) DO SET JsonElement=%%x
IF %JsonElement%==DEV (ECHO development environment) ELSE (ECHO abc123 environment)

How delete last comma in json file using Bash?

I wrote some script that takes all user data of aws ec2 instance, and echo to local.json. All this happens when I install my node.js modules.
I don't know how to delete last comma in the json file. Here is the bash script:
#!/bin/bash
export DATA_DIR=/data
export PATH=$PATH:/usr/local/bin
#install package from git repository
sudo -- sh -c "export PATH=$PATH:/usr/local/bin; export DATA_DIR=/data; npm install git+https://reader:secret#bitbucket.org/somebranch/$1.git#$2"
#update config files from instance user-data
InstanceConfig=`cat /instance-config`
echo '{' >> node_modules/$1/config/local.json
while read line
do
if [ ! -z "$line" -a "$line" != " " ]; then
Key=`echo $line | cut -f1 -d=`
Value=`echo $line | cut -f2 -d=`
if [ "$Key" = "Env" ]; then
Env="$Value"
fi
printf '"%s" : "%s",\n' "$Key" "$Value" >> node_modules/*/config/local.json
fi
done <<< "$InstanceConfig"
sed -i '$ s/.$//' node_modules/$1/config/local.json
echo '}' >> node_modules/$1/config/local.json
To run him im doing that way: ./script
I get json(OUTPUT), but with comma in all lines. Here is local.json that I get:
{
"Env" : "dev",
"EngineUrl" : "engine.url.net",
}
All I trying to do, is delete in last line of the json file - comma(",").
I try many ways, that I found in internet. I know that it should be after last "fi"(end of the loop). And I know that it should be something like this line:
sed -i "s?${Key} : ${Value},?${Key} : ${Value}?g" node_modules/$1/config/local.json
Or this:
sed '$ s/,$//' node_modules/$1/config/local.json
But they not work for me.
Can someone help me with that? Who knows Bash scripting well?
Thanks!
If you know that it is the last comma that needs to be replaced, a reasonably robust way is to use GNU sed in "slurp" mode like this:
sed -zr 's/,([^,]*$)/\1/' local.json
Output:
{
"Env" : "dev",
"EngineUrl" : "engine.url.net"
}
If you'd just post some sample input/output it'd remove the guess-work but IF this is your input file:
$ cat file
Env=dev
EngineUrl=engine.url.net
Then IF you're trying to do what I think you are then all you need is:
$ cat tst.awk
BEGIN { FS="="; sep="{\n" }
{
printf "%s \"%s\" : \"%s\"", sep, $1, $2
sep = ",\n"
}
END { print "\n}" }
which you'd execute as:
$ awk -f tst.awk file
{
"Env" : "dev",
"EngineUrl" : "engine.url.net"
}
Or you can execute the awk script inline within a shell script if you prefer:
awk '
BEGIN { FS="="; sep="{\n" }
{
printf "%s \"%s\" : \"%s\"", sep, $1, $2
sep = ",\n"
}
END { print "\n}" }
' file
{
"Env" : "dev",
"EngineUrl" : "engine.url.net"
}
The above is far more robust, portable, efficient and better in every other way than the shell script you posted because it's using the right tool for the job. A UNIX shell is an environment from which to call tools with a language to sequence those calls. It is NOT a language to process text which is why it's so difficult to get it right. The UNIX tool for general text processing is awk so when you need to process text in UNIX, you just have shell call awk, that's all.
Here a jq version if it's available:
jq --raw-input 'split("=") | {(.[0]):.[1]}' /instance-config | jq --slurp 'add'
There might be a way to do it with one jqpass, but I couldn't see it.
You an remove all trailing commas from invalid json with:
sed -i.bak ':begin;$!N;s/,\n}/\n}/g;tbegin;P;D' FILE
sed -i.bak = creates a backup of the original file, then applies changes to the file
':begin;$!N;s/,\n}/\n}/g;tbegin;P;D' = anything ending with , followed by
"new line and }". Remove the , on the previous line
FILE = the file you want to make the change to
If you're willing to use it, xidel is rather forgiving for trailing commas:
xidel -s local.json -e '$json'
{
"Env": "dev",
"EngineUrl": "engine.url.net"
}
xidel - -se '$json' <<< '{"Env":"dev","EngineUrl":"engine.url.net",}'
#or
xidel - -se 'parse-json($raw,{"liberal":true()})' <<< '{"Env":"dev","EngineUrl":"engine.url.net",}'
{
"Env": "dev",
"EngineUrl": "engine.url.net"
}