I am trying to form where clause from the JSON file. I want to fetch the key:value pair from the where part of below input and convert it into key = value
Input:
{
"Table":{
"TableName":"Employee",
"Columns":[
"ID",
"Name"
],
"Where":{
"ID":"A-0001",
"Name":"xyz"
}
}
}
Expected Output:
I want to get where ID = 'A-0001' and Name = 'xyz'
I am using mule 4. Please help.
Thanks in advance
Another take at a similar approach.
%dw 2.0
output application/json
---
"where " ++ ((payload.Table.Where mapObject {
($$): (($$) ++ "=" ++ "'" ++ ($) ++ "'")
}) pluck ($) joinBy " and ")
Building dynamic SQL? I would suggest avoiding that and sticking to parameterized queries.
I would strongly urge you to use this path: https://docs.mulesoft.com/connectors/db/database-connector-examples#use-input-parameters-to-protect-database-queries
It would be really easy to make this parameterized because all you'd have to do is passing in payload.Table.Where as your parameterized object.
If you're pulling this from some kind of trusted source and have to do things this way I suppose you could ignore the columns array and do it this way pretty easily:
%dw 2.0
output application/json
---
if (payload.Table.Where?)
"where " ++ ((payload.Table.Where pluck "$($$) = '$($)'") reduce ($$ ++ " and " ++ $))
else
""
I'm sure there is a better way to do this, but if you wanted to still use the columns array you could also do this.
%dw 2.0
output application/json
---
"where " ++ (
payload.Table.Columns reduce ((col, wClause="") ->
if (not payload.Table.Where[col]?) wClause
else wClause ++ (if (wClause != "") " and " else "") ++ ("$(col) = '$(payload.Table.Where[col])'")
)
)
I have the below requirement.
Input is
{ "packageConfiguration": [
{
"packageId": [
"AIM_PACKAGE"
],
"component": [
"Handbook"
],
"fieldName": [
"Upload Handbook Document"
],
"assetUrl": [
"sflydamlocation.handbookfilename.pdf"
]
}
]}
I need to convert above json array into this output format:
{
"pakage": ""packageId":"AIM_PACKAGE", "component":"Handbook", "fieldName":"Upload Handbook Document","assetUrl":"sflydamlocation.handbookfilename.pdf""
}
You can do that treating all fields as strings, however note that:
The inner quotes must be escaped. Otherwise the output is not valid JSON.
Take in account that the value of "package" is not really valid JSON either, in case you want to parse it. It should an object (eg " { \"package\":... }")
This script expects all the arrays to have exactly 1 element. More elements are ignored and less could give an error. This is not a very robust design.
Script (not recommended):
%dw 2.0
output application/json
---
package: using (pc = payload.packageConfiguration[0]) (
" \"packageId\": \"$(pc.packageId[0])\", " ++
" \"component\": \"$(pc.component[0])\" " ++
" \"fieldName\": \"$(pc.fieldName[0])\" " ++
" \"assetUrl\": \"$(pc.assetUrl[0])\" "
)
Output:
{
"package": " \"packageId\": \"AIM_PACKAGE\", \"component\": \"Handbook\" \"fieldName\": \"Upload Handbook Document\" \"assetUrl\": \"sflydamlocation.handbookfilename.pdf\" "
}
This is an ugly string concatenation. Instead I would suggest to just write the desired output as a JSON object.
Script (recommended):
%dw 2.0
output application/dw
var pc = payload.packageConfiguration[0]
---
package:
write({
packageId: pc.packageId[0],
component: pc.component[0],
fieldName: pc.fieldName[0],
assetUrl: pc.assetUrl[0]
}, "application/json") replace /\n/ with ""
Output
{
"package": "{ \"packageId\": \"AIM_PACKAGE\", \"component\": \"Handbook\", \"fieldName\": \"Upload Handbook Document\", \"assetUrl\": \"sflydamlocation.handbookfilename.pdf\"}"
}
The second script is much cleaner, less error prone and returns an escaped JSON object that you could unescape to use as JSON.
Something like this should work, unless you require something more flexible. I'm assuming you're working w/ Mule3/DW1:
%dw 1.0
%output application/json
%var packageConfig = payload.packageConfiguration[0]
---
{
package: packageConfig mapObject ((value, key) -> {
(key): value[0]
})
}
I have a problem string which has multiple lines:
line1
Link1: //website/go/<example>
line2
I am trying to make trim to get just web page link part (just address which can be various - in this case "//website/go/") but there are some extra signs before and after.
My try:
set temp [string map { " " "" "line1" "" "Link1: " "" "line2" "" } $output]
puts "found link : $temp
And the output of it is:
found link :<empty line>
//website/go/<example>
<empty line>`
How can I remove all white spaces, newlines, etc. and trim it in way to get just the part of the string which I am looking for. In this case to get just: //website/go/<example>?
Given this input data:
line1
Link1: //website/go/<example>
line2
You can use string map as you do and then post-process the result with string trim (assuming you only expect one thing left at the end) or remove the newlines with another mapping element. On two lines for clarity:
set temp [string map { " " "" "line1" "" "Link1: " "" "line2" "" } $output]
set temp [string trim $temp]
puts "found link : $temp
However, in this case I'd actually use regexp to pick the data I want:
regexp -line {Link1:\s+(.*\S)} $output -> temp
puts "found link : $temp
Regular expressions tend to be more suitable for parsing part-formatted data, provided you remember to keep them short. The longer a RE is, the harder it is to understand.
I am searching for an HTML node that contains this text:
(~! # # $ % ^ & * ( ) _ + { } | : " < > ? ` - = [ ] \ ; ‘ , . / ).
But xpath is obviously having problems because it's not recognizing the above as an actual text to search for but rather as part of the xpath expression, which is why I am getting the error:
is not a valid XPath expression.
So how would I convert the above to a text or string to avoid this issue?
Thanks
I want to convert my sqlite data from my database to JSON format.
I would like to use this syntax:
sqlite3 -line members.db "SELECT * FROM members LIMIT 3" > members.txt
OUTPUT:
id = 1
fname = Leif
gname = Håkansson
genderid = 1
id = 2
fname = Yvonne
gname = Bergman
genderid = 2
id = 3
fname = Roger
gname = Sjöberg
genderid = 1
How to do this with nice and structur code in a for loop?
(Only in Bash)
I have tried some awk and grep but not with a great succes yet.
Would be nice with some tips.
I want a result similar to this:
[
{
"id":1,
"fname":"Leif",
"gname":"Hakansson",
"genderid":1
},
{
"id":2,
"fname":"Yvonne",
"gname":"Bergman",
"genderid":2
},
{
"id":3,
"fname":"Roger",
"gname":"Sjberg",
"genderid":1
}
}
If your sqlite3 is compiled with the json1 extension (or if you can obtain a version of sqlite3 with the json1 extension), then you can use it to generate JSON objects (one JSON object per row). For example:
select json_object('id', id, 'fname', fname, 'gname', gname, 'genderid', genderid) ...
You can then use a tool such as jq to convert the stream of objects into an array of objects, e.g. pipe the output of the sqlite3 to jq -s ..
(A less tiresome alternative might be to use the sqlite3 function json_array(), which produces an array, which you can reassemble into an object using jq.)
If the json1 extension is unavailable, then you could use the following as a starting point:
awk 'BEGIN { print "["; }
function out() {if (n++) {print ","}; if (line) {print "{" line "}"}; line="";}
function trim(x) { sub(/^ */, "", x); sub(/ *$/, "", x); return x; }
NF==0 { out(); next};
{if (line) {line = line ", " }
i=index($0,"=");
line = line "\"" trim(substr($0,1,i-1)) ": \"" substr($0, i+2) "\""}
END {out(); print "]"} '
Alternatively, you could use the following jq script, which converts numeric strings that occur on the RHS of "=" to numbers:
def trim: sub("^ *"; "") | sub(" *$"; "");
def keyvalue: index("=") as $i
| {(.[0:$i] | trim): (.[$i+2:] | (tonumber? // .))};
[foreach (inputs, "") as $line ({object: false, seed: {} };
if ($line|trim) == "" then { object: .seed, seed : {} }
else {object: false,
seed: (.seed + ($line | keyvalue)) }
end;
.object | if . and (. != {}) then . else empty end ) ]
Just type -json argument with SQLite 3.33.0 or higher and get json output:
$ sqlite3 -json database.db "select * from TABLE_NAME"
from SQLite Release 3.33.0 note:
...
CLI enhancements:
Added four new output modes: "box", "json", "markdown", and "table".
The "column" output mode automatically expands columns to contain the longest output row and automatically turns ".header" on if it has
not been previously set.
The "quote" output mode honors ".separator"
The decimal extension and the ieee754 extension are built-in to the CLI
...
I think I would prefer to parse sqlite output with a single line per record rather than the very wordy output format you suggested with sqlite3 -line. So, I would go with this:
sqlite3 members.db "SELECT * FROM members LIMIT 3"
which gives me this to parse:
1|Leif|Hakansson|1
2|Yvonne|Bergman|2
3|Roger|Sjoberg|1
I can now parse that with awk if I set the input separator to | with
awk -F '|'
and pick up the 4 fields on each line with the following and save them in an array like this:
{ id[++i]=$1; fname[i]=$2; gname[i]=$3; genderid[i]=$4 }
Then all I need to do is print the output format you need at the end. However, you have double quotes in your output and they are a pain to quote in awk, so I temporarily use another pipe symbol (|) as a double quote and then, at the very end, I get tr to replace all the pipe symbols with double quotes - just to make the code easier on the eye. So the total solution looks like this:
sqlite3 members.db "SELECT * FROM members LIMIT 3" | awk -F'|' '
# sqlite output line - pick up fields and store in arrays
{ id[++i]=$1; fname[i]=$2; gname[i]=$3; genderid[i]=$4 }
END {
printf "[\n";
for(j=1;j<=i;j++){
printf " {\n"
printf " |id|:%d,\n",id[j]
printf " |fname|:|%s|,\n",fname[j]
printf " |gname|:|%s|,\n",gname[j]
printf " |genderid|:%d\n",genderid[j]
closing=" },\n"
if(j==i){closing=" }\n"}
printf closing;
}
printf "]\n";
}' | tr '|' '"'
Sqlite-utils does exactly what you're looking for. By default, the output will be JSON.
Better late than never to plug jo.
Save sqlite3 to a text file.
Get jo (jo's also available in distro repos)
and use this bash script.
while read line
do
id=`echo $line | cut -d"|" -f1`
fname=`echo $line | cut -d"|" -f2`
gname=`echo $line | cut -d"|" -f3`
genderid=`echo $line | cut -d"|" -f4`
jsonline=`jo id="$id" fname="$fname" gname="$gname" genderid="$genderid"`
json="$json $jsonline"
done < "$1"
jo -a $json
Please don't create (or parse) json with awk. There are dedicated tools for this. Tools like xidel.
While first and foremost a html, xml and json parser, xidel can also parse plain text.
I'd like to offer a very elegant solution using this tool (with much less code than jq).
I'll assume your 'members.txt'.
First to create a sequence of each json object to-be:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n")'
Or...
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! (position(),.)'
1
id = 1
fname = Leif
gname = Håkansson
genderid = 1
2
id = 2
fname = Yvonne
gname = Bergman
genderid = 2
3
id = 3
fname = Roger
gname = Sjöberg
genderid = 1
...to better show you the individual items in the sequence.
Now you have 3 multi-line strings. To turn each item/string into another sequence where each item is a new line:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! x:lines(.)'
(x:lines(.) is a shorthand for tokenize(.,'\r\n?|\n'))
Now for each line tokenize on the " = " (which creates yet another sequence) and save it to a variable. For the first line for example this sequence is ("id","1"), for the second line ("fname","Leif"), etc.:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! (for $x in x:lines(.) let $a:=tokenize($x," = ") return ($a[1],$a[2]))'
Finally remove leading whitespace (normalize-space()), create a json object ({| {key-value-pair} |}) and put all json objects in an array ([ ... ]):
xidel -s members.txt --xquery '[tokenize($raw,"\n\n") ! {|for $x in x:lines(.) let $a:=tokenize($x," = ") return {normalize-space($a[1]):$a[2]}|}]'
Prettified + output:
xidel -s members.txt --xquery '
[
tokenize($raw,"\n\n") ! {|
for $x in x:lines(.)
let $a:=tokenize($x," = ")
return {
normalize-space($a[1]):$a[2]
}
|}
]
'
[
{
"id": "1",
"fname": "Leif",
"gname": "Håkansson",
"genderid": "1"
},
{
"id": "2",
"fname": "Yvonne",
"gname": "Bergman",
"genderid": "2"
},
{
"id": "3",
"fname": "Roger",
"gname": "Sjöberg",
"genderid": "1"
}
]
Note: For xidel-0.9.9.7173 and newer --json-mode=deprecated is needed to create a json array with [ ]. The new (XQuery 3.1) way to create a json array is to use array{ }.