I was looking for some built-in functions inside awk to easily generate JSON objects. I came across several answers and decided to create my own.
I'd like to generate JSON from multidimensional arrays, where I store table style data, and to use separate and dynamic definition of JSON schema to be generated from that data.
Desired output:
{
"Name": JanA
"Surname": NowakA
"ID": 1234A
"Role": PrezesA
}
{
"Name": JanD
"Surname": NowakD
"ID": 12341D
"Role": PrezesD
}
{
"Name": JanC
"Surname": NowakC
"ID": 12342C
"Role": PrezesC
}
Input file:
pierwsza linia
druga linia
trzecia linia
dane wspólników
imie JanA
nazwisko NowakA
pesel 11111111111A
funkcja PrezesA
imie Ja"nD
nazwisko NowakD
pesel 11111111111
funkcja PrezesD
imie JanC
nazwisko NowakC
pesel 12342C
funkcja PrezesC
czwarta linia
reprezentanci
imie Tomek
Based on input file i created a multidimensional array:
JanA NowaA 1234A PrezesA
JanD NowakD 12341D PrezesD
JanC NowakC 12342C PrezesC
I'll take a stab at a gawk solution. The indenting isn't perfect and the results aren't ordered (see "Sorting" note below), but it's at least able to walk a true multidimensional array recursively and should produce valid, parsable JSON from any array. Bonus: the data array is the schema. Array keys become JSON keys. There's no need to create a separate schema array in addition to the data array.
Just be sure to use the true multidimensional array[d1][d2][d3]... convention of constructing your data array, rather than the concatenated index array[d1,d2,d3...] convention.
Update:
I've got an updated JSON gawk script posted as a GitHub Gist. Although the script below is tested as working with OP's data, I might've made improvements since this post was last edited. Please see the Gist for the most thoroughly tested, bug-squashed version.
#!/usr/bin/gawk -f
BEGIN { IGNORECASE = 1 }
$1 ~ "imie" { record[++idx]["name"] = $2 }
$1 ~ "nazwisko" { record[idx]["surname"] = $2 }
$1 ~ "pesel" { record[idx]["ID"] = $2 }
$1 ~ "funkcja" { record[idx]["role"] = $2 }
END { print serialize(record, "\t") }
# ==== FUNCTIONS ====
function join(arr, sep, _p, i) {
# syntax: join(array, string separator)
# returns a string
for (i in arr) {
_p["result"] = _p["result"] ~ "[[:print:]]" ? _p["result"] sep arr[i] : arr[i]
}
return _p["result"]
}
function quote(str) {
gsub(/\\/, "\\\\", str)
gsub(/\r/, "\\r", str)
gsub(/\n/, "\\n", str)
gsub(/\t/, "\\t", str)
return "\"" str "\""
}
function serialize(arr, indent_with, depth, _p, i, idx) {
# syntax: serialize(array of arrays, indent string)
# returns a JSON formatted string
# sort arrays on key, ensures [...] values remain properly ordered
if (!PROCINFO["sorted_in"]) PROCINFO["sorted_in"] = "#ind_num_asc"
# determine whether array is indexed or associative
for (i in arr) {
_p["assoc"] = or(_p["assoc"], !(++_p["idx"] in arr))
}
# if associative, indent
if (_p["assoc"]) {
for (i = ++depth; i--;) {
_p["end"] = _p["indent"]; _p["indent"] = _p["indent"] indent_with
}
}
for (i in arr) {
# If key length is 0, assume its an empty object
if (!length(i)) return "{}"
# quote key if not already quoted
_p["key"] = i !~ /^".*"$/ ? quote(i) : i
if (isarray(arr[i])) {
if (_p["assoc"]) {
_p["json"][++idx] = _p["indent"] _p["key"] ": " \
serialize(arr[i], indent_with, depth)
} else {
# if indexed array, dont print keys
_p["json"][++idx] = serialize(arr[i], indent_with, depth)
}
} else {
# quote if not numeric, boolean, null, already quoted, or too big for match()
if (!((arr[i] ~ /^[0-9]+([\.e][0-9]+)?$/ && arr[i] !~ /^0[0-9]/) ||
arr[i] ~ /^true|false|null|".*"$/) || length(arr[i]) > 1000)
arr[i] = quote(arr[i])
_p["json"][++idx] = _p["assoc"] ? _p["indent"] _p["key"] ": " arr[i] : arr[i]
}
}
# I trial and errored the hell out of this. Problem is, gawk cant distinguish between
# a value of null and no value. I think this hack is as close as I can get, although
# [""] will become [].
if (!_p["assoc"] && join(_p["json"]) == "\"\"") return "[]"
# surround with curly braces if object, square brackets if array
return _p["assoc"] ? "{\n" join(_p["json"], ",\n") "\n" _p["end"] "}" \
: "[" join(_p["json"], ", ") "]"
}
Output resulting from OP's example data:
[{
"ID": "1234A",
"name": "JanA",
"role": "PrezesA",
"surname": "NowakA"
}, {
"ID": "12341D",
"name": "JanD",
"role": "PrezesD",
"surname": "NowakD"
}, {
"ID": "12342C",
"name": "JanC",
"role": "PrezesC",
"surname": "NowakC"
}, {
"name": "Tomek"
}]
Sorting
Although the results by default are ordered in a manner only gawk understands, it is possible for gawk to sort the results on a field. If you'd like to sort on the ID field for example, add this function:
function cmp_ID(i1, v1, i2, v2) {
if (!isarray(v1) && v1 ~ /"ID"/ ) {
return v1 < v2 ? -1 : (v1 != v2)
}
}
Then insert this line within your END section above print serialize(record):
PROCINFO["sorted_in"] = "cmp_ID"
See Controlling Array Traversal for more information.
My updated awk implementation of simple array printer with regex based validation for each column(running using gawk):
function ltrim(s) { sub(/^[ \t]+/, "", s); return s }
function rtrim(s) { sub(/[ \t]+$/, "", s); return s }
function sTrim(s){
return rtrim(ltrim(s));
}
function jsonEscape(jsValue) {
gsub(/\\/, "\\\\", jsValue)
gsub(/"/, "\\\"", jsValue)
gsub(/\b/, "\\b", jsValue)
gsub(/\f/, "\\f", jsValue)
gsub(/\n/, "\\n", jsValue)
gsub(/\r/, "\\r", jsValue)
gsub(/\t/, "\\t", jsValue)
return jsValue
}
function jsonStringEscapeAndWrap(jsValue) {
return "\42" jsonEscape(jsValue) "\42"
}
function jsonPrint(contentArray, contentRowsCount, schemaArray){
result = ""
schemaLength = length(schemaArray)
for (x = 1; x <= contentRowsCount; x++) {
result = result "{"
for(y = 1; y <= schemaLength; y++){
result = result "\42" sTrim(schemaArray[y]) "\42:" sTrim(contentArray[x, y])
if(y < schemaLength){
result = result ","
}
}
result = result "}"
if(x < contentRowsCount){
result = result ",\n"
}
}
return result
}
function jsonValidateAndPrint(contentArray, contentRowsCount, schemaArray, schemaColumnsCount, errorArray){
result = ""
errorsCount = 1
for (x = 1; x <= contentRowsCount; x++) {
jsonRow = "{"
for(y = 1; y <= schemaColumnsCount; y++){
regexValue = schemaArray[y, 2]
jsonValue = sTrim(contentArray[x, y])
isValid = jsonValue ~ regexValue
if(isValid == 0){
errorArray[errorsCount, 1] = "\42" sTrim(schemaArray[y, 1]) "\42"
errorArray[errorsCount, 2] = "\42Value " jsonValue " not match format: " regexValue " \42"
errorArray[errorsCount, 3] = x
errorsCount++
jsonValue = "null"
}
jsonRow = jsonRow "\42" sTrim(schemaArray[y, 1]) "\42:" jsonValue
if(y < schemaColumnsCount){
jsonRow = jsonRow ","
}
}
jsonRow = jsonRow "}"
result = result jsonRow
if(x < contentRowsCount){
result = result ",\n"
}
}
return result
}
BEGIN{
rowsCount =1
matchCount = 0
errorsCount = 0
shareholdersJsonSchema[1, 1] = "Imie"
shareholdersJsonSchema[2, 1] = "Nazwisko"
shareholdersJsonSchema[3, 1] = "PESEL"
shareholdersJsonSchema[4, 1] = "Funkcja"
shareholdersJsonSchema[1, 2] = "\\.*"
shareholdersJsonSchema[2, 2] = "\\.*"
shareholdersJsonSchema[3, 2] = "^[0-9]{11}$"
shareholdersJsonSchema[4, 2] = "\\.*"
errorsSchema[1] = "PropertyName"
errorsSchema[2] = "Message"
errorsSchema[3] = "PositionIndex"
resultSchema[1]= "ShareHolders"
resultSchema[2]= "Errors"
}
/dane wspólników/,/czwarta linia/{
if(/imie/ || /nazwisko/ || /pesel/ || /funkcja/){
if(/imie/){
shareholdersArray[rowsCount, 1] = jsonStringEscapeAndWrap($2)
matchCount++
}
if(/nazwisko/){
shareholdersArray[rowsCount, 2] = jsonStringEscapeAndWrap($2)
matchCount ++
}
if(/pesel/){
shareholdersArray[rowsCount, 3] = $2
matchCount ++
}
if(/funkcja/){
shareholdersArray[rowsCount, 4] = jsonStringEscapeAndWrap($2)
matchCount ++
}
if(matchCount==4){
rowsCount++
matchCount = 0;
}
}
}
END{
shareHolders = jsonValidateAndPrint(shareholdersArray, rowsCount - 1, shareholdersJsonSchema, 4, errorArray)
shareHoldersErrors = jsonPrint(errorArray, length(errorArray) / length(errorsSchema), errorsSchema)
resultArray[1,1] = "\n[\n" shareHolders "\n]\n"
resultArray[1,2] = "\n[\n" shareHoldersErrors "\n]\n"
resultJson = jsonPrint(resultArray, 1, resultSchema)
print resultJson
}
Produces output:
{"ShareHolders":
[
{"Imie":"JanA","Nazwisko":"NowakA","PESEL":null,"Funkcja":"PrezesA"},
{"Imie":"Ja\"nD","Nazwisko":"NowakD","PESEL":11111111111,"Funkcja":"PrezesD"},
{"Imie":"JanC","Nazwisko":"NowakC","PESEL":null,"Funkcja":"PrezesC"}
]
,"Errors":
[
{"PropertyName":"PESEL","Message":"Value 11111111111A not match format: ^[0-9]{11}$ ","PositionIndex":1},
{"PropertyName":"PESEL","Message":"Value 12342C not match format: ^[0-9]{11}$ ","PositionIndex":3}
]
}
I tried to convert the JSON data in my Cloudant db to csv format, using the List function. It works perfectly for all values except JSON array values, i.e. the nested values. For these, I am getting [object object] as the output in my csv document.
Please find the sample JSON document which I am using, below:
{
"NAME": "Aparna",
"EMAIL": "something#domain.com",
"PUBLIC_OFFICIALS_CONTACTED": [
{ "NAME_PUBLIC_OFFICIAL": [ "ab"],
"TITLE_PUBLIC_OFFICIAL": ["cd"]}
],
"COMMUNICATION_TYPE": [
"Meeting",
"Phone",
"Handout",
"Conference"
],
"NAMES_OF_OTHERS_FROM_IBM": [
{ "NAME_OF_OTHERS": ["ef"],
"TITLE_OF_OTHERS": [ "gh"]}
],
"COMMUNICATION_BENEFIT": "Yes",
"LAST_UPDATE_BY" : "ap"
}
Please find the map and list functions used below :
"map" : "function(doc){
if((\"SAVE_TYPE_SUBMIT\" in doc) && (doc.SAVE_TYPE_SUBMIT== \"Submit\")) {
emit (doc. LAST_UPDATE_BY,[doc.NAME,doc.EMAIL,doc.PUBLIC_OFFICIALS_CONTACTED[0].NAME_PUBLIC_OFFICIAL,\n doc.PUBLIC_OFFICIALS_CONTACTED[0].TITLE_PUBLIC_OFFICIAL,doc.COMMUNICATION_TYPE,doc.NAMES_OF_OTHERS_FROM_IBM[0].NAME_OF_OTHERS, doc.NAMES_OF_OTHERS_FROM_IBM[0].TITLE_OF_OTHERS, doc.COMMUNICATION_BENEFIT,doc. LAST_UPDATE_BY,doc.LAST_UPDATE_DATE]) ;
}
}
"list" : "function (head, req) {
var row;
start({\n headers: {'Content-Type': 'text/csv' },
});
var first = true;
while(row = getRow()) {
var doc = row.doc;
if (first) {
send(Object.keys(doc).join(',') + '\\n');
first = false;\n }
var line = '';
for(var i in doc) {
// comma separator
if (line.length > 0) {
line += ',';\n }
// output the value, ensuring values that themselves
// contain commas are enclosed in double quotes
var val = doc[i];
if (typeof val == 'string' && val.indexOf(',') > -1) {
line += '\"' + val.replace(/\"/g,'\"\"') + '\"';
}
else {
line += val;
}
}
line += '\\n';
send(line);
}}"
Note : In the map, only the first values have been fetched from the JSON arrays for now, on purpose, to simplify the function.
Please help understand how to fetched the nested JSON values or arrays and download the same in csv format. Any guidance would be much appreciated!
You can try to stringify the object you are trying to export and you will get some clue
if (typeof val == 'string' && val.indexOf(',') > -1) {
line += '\"' + val.replace(/\"/g,'\"\"') + '\"';
}
else {
line += JSON.stringify(val);
}
Or even better
if (typeof val == 'string' && val.indexOf(',') > -1) {
line += '\"' + val.replace(/\"/g,'\"\"') + '\"';
}
else if(val instanceof Array){
line += val.join(',');
}
else {
line += JSON.stringify(val);
}
There are a couple of things to change here that might help. The first thing is that you don't need to emit all the values you want to use, because you can access the document itself from the list when dealing with a view.
With this in mind, the map could have an emit like
emit (doc.LAST_UPDATE_BY, null);
With this in place, if you request the list/view with include_docs=true then you can refer to the fields in your document inside the while(row = getRow()) section like this:
send(row.doc.NAME + ',' + row.doc.EMAIL + '\\n');
And for the nested documents, try something like:
row.doc.PUBLIC_OFFICIALS_CONTACTED.0.NAME_PUBLIC_OFFICIAL
You already referred in another question to the article I'd recommend for a full working example https://developer.ibm.com/clouddataservices/2015/09/22/export-cloudant-json-as-csv-rss-or-ical/ - hopefully this explanation helps also.
I want to get the id value from the following json object
answerTag:
[ '[{"id":64,"name":"Coronary Artery Disease"}]',
'[{"id":64,"name":"Coronary Artery Disease"}]' ],
risk: '1' }
I had the same issue. I found that I had to remove the leading and trailing brackets before calling JSON.parse for it to work. The JSON.parse inside a try-catch didn't fail but just the same I couldn't access the values in the resultant object.
Here's the excerpt. "rows[0]" is the the first row of my MySQL result array
result = JSON.stringify(rows[0])
result = result.replace(/(^\[)/, '');
result = result.replace(/(\]$)/, '');
try {
var resultObj = JSON.parse(result);
} catch (e) {
console.log("Error, not a valid JSON string");
}
var my_value = resultObj["my_key"];
Hope this helps!
You dont have a valid JSON
as a valid json should have a key-value pair
var answerTag=
{
"key1":{"id":64,"name":"Coronary Artery Disease"},
"key2":{"id":64,"name":"Coronary Artery Disease"},
"key3":{risk: '1' }
}
So if u want to traverse that ,put it in for loop like this
getKeyValueFromJSON(answerTag);
function getKeyValueFromJSON(
for(var key in obj) {
alert("Key: " + key + " value: " + obj[key]);
}
}
Or if i am able to get it
The formed JSON is like this
var obj={
answerTag: [
'[{"id":64,"name":"Coronary Artery Disease"}]',
'[{"id":64,"name":"Coronary Artery Disease"}]' ],
risk: '1'
}
So You can use it like this
for(var key in obj) {
for(var innerKey in obj[key]) {
console.log("Key: " + innerKey + " value: " + obj[key][innerKey]);
}
}
Note: Below there is part of code that I think will be more then enough for my question. But I also put the zip archive of whole script at the end of this post.
What is need to do: I need to replace line " argc ; number " with the value that is returned by string map [ list // $function_parameter_comment_sign ] "\" argc // number \"" and the line " argv ; arguments " with the value that is returned by string map [ list // $function_parameter_comment_sign ] "\" argv // arguments \""
I tried different ways to do that: to include them to [] or {}, to assign value to variable and place it in that line and many other ways but did not succeed. How can I do it?
if { [ llength $argv ] == 1 } {
set test [ lindex $argv 0 ]
testone $test
} else {
testmain
}
proc testmain { } {
global ut_all_tests ut_current_test
foreach test $ut_all_tests {
set ut_current_test $test
puts $test
$test
}
}
proc testone { torun } {
global ut_all_tests ut_current_test
foreach test $ut_all_tests {
if { $torun == $test } {
set ut_current_test $test
puts $test
$test
}
}
}
proc tproc { name args body } {
global ut_all_tests
lappend ut_all_tests $name
proc $name $args $body
}
tproc extract_tcl_signature_test { } {
proc load_generators {} {
set script_path [ file dirname [ file normalize [ info script ] ] ]
set drakon_editor_path [string trimright $script_path "unittest" ]
set scripts [ glob -- "$drakon_editor_path/generators/*.tcl" ]
foreach script $scripts {
source $script
}
}
namespace eval gen {
array set generators {}
# This procedure is called by language generator files. In the beginning of every language generator there is calling code.
proc add_generator { language generator } {
variable generators
if { [ info exists generator($language) ] } {
error "Generator for language $language already registered."
}
set gen::generators($language) $generator
}
}
load_generators
puts "=================================="
puts "Started: extract_tcl_signature_test"
foreach { language generator } [ array get gen::generators ] {
puts "----------------------------------"
puts $language
puts $generator
namespace eval current_file_generation_info {}
set current_file_generation_info::language $language
set current_file_generation_info::generator $generator
set find [string first :: $generator]
set generator_namespace [ string range $generator 0 $find-1 ]
# These 3 lines is to check is current generator have commentator procedure.
# If not commentator_status_var is set to "" .
set commentator_for_namespace_text "::commentator"
set commentator_call_text "$generator_namespace$commentator_for_namespace_text"
set commentator_status_var [ namespace which $commentator_call_text ]
# If current language does not have commentator procedure or current languages is in if conditions, then // sign for function parameter commenting will be used.
# It is done so for compability with diagrams which are made with previous versions of DRAKON Editor.
# If you are adding new language generator to DRAKON Editor and want to use line comment sign as
# commenting sign for function parameters, just make commentator procedure in your language generator
# as it is for example in AutoHotkey code generator.
if { $commentator_status_var == "" ||
$language == "C" ||
$language == "C#" ||
$language == "C++" ||
$language == "D" ||
$language == "Erlang" ||
$language == "Java" ||
$language == "Javascript" ||
$language == "Lua" ||
$language == "Processing.org" ||
$language == "Python 2.x" ||
$language == "Python 3.x" ||
$language == "Tcl" ||
$language == "Verilog" } {
good_signature_tcl { " " " #comment " "" "what?" } comment "" {} ""
good_signature_tcl { ""
""
" argc // number "
" argv // arguments "
" "
} procedure public { "argc" "argv" } ""
good_signature_tcl { "one"
"two" } procedure public { "one" "two" } ""
} else {
# Get current generator line comment simbol and calculate its length without space sign.
set function_parameter_comment_sign [ $commentator_call_text "" ]
set function_parameter_comment_sign [string trim $function_parameter_comment_sign " " ]
if { $function_parameter_comment_sign == "#" } {
#good_signature_tcl { " " " #comment " "" "what?" } comment "" {} ""
good_signature_tcl { ""
""
" argc # number "
" argv # arguments "
" "
} procedure public { "argc" "argv" } ""
good_signature_tcl { "one"
"two" } procedure public { "one" "two" } ""
} else {
good_signature_tcl { " " " #comment " "" "what?" } comment "" {} ""
good_signature_tcl { ""
""
" argc ; number "
" argv ; arguments "
" "
} procedure public { "argc" "argv" } ""
good_signature_tcl { "one"
"two" } procedure public { "one" "two" } ""
}
}
#puts $function_parameter_comment_sign
}
puts "----------------------------------"
puts "Successfully ended: extract_tcl_signature_test"
puts "=================================="
}
proc good_signature_tcl { lines type access parameters returns } {
set text [ join $lines "\n" ]
unpack [ gen_tcl::extract_signature $text foo ] message signature
equal $message ""
unpack $signature atype aaccess aparameters areturns
equal $atype $type
equal $aaccess $access
set par_part0 {}
foreach par $aparameters {
lappend par_part0 [ lindex $par 0 ]
}
list_equal $par_part0 $parameters
equal [ lindex $areturns 0 ] $returns
}
Above code parts are from files: unittest.tcl , utest_utils.tcl and gen_test.tcl
Download code link for whole code: https://mega.co.nz/#!QFhlnSIS!8lxgCFbXAweqrj72Gj8KRbc6o9GVlX-V9T1Fw9jwhN0
I'm not entirely sure what you're looking for, but if I got it right, you could try using something like this:
good_signature_tcl [list "" \
"" \
[string map [list // $function_parameter_comment_sign] " argc // number "] \
[string map [list // $function_parameter_comment_sign] " argv // arguments "] \
" " \
] procedure public { "argc" "argv" } ""
Since your {} are being used to make a list, then using [list] should yield the same thing with the bonus of having functions and substitutions work.
Assuming that you always want a constant replacement (i.e., that you're not varying $function_parameter_comment_sign at every site!) then it's not too hard. Just build the map to apply with string map in several stages. (I'm not 100% sure that I've got what you want to replace right, but it should be easy for you to fix it from here.)
# Shared value to keep line length down
set replacer [list "//" $function_parameter_comment_sign]
# Build the map (no need to do it in one monster [list]!)
set map {}
lappend map {" argc ; number "} [string map $replacer "\" argc // number \""]
lappend map {" argv ; arguments "} [string map $replacer "\" argv // arguments \""]
# Apply the map
set theMappedText [string map $map $theInputText]
If you want to do something other than replace each instance of the unwanted string with a constant, things will quite a bit more complex as you would need to use subst (and hence you would also need an extra cleaning step). Fortunately, I don't think you need that; your case appears to be simple enough.
I was looking at this answer (copied below)
What I didn't understand is why is / escaped?
There is also the solution from mustache.js
https://github.com/janl/mustache.js/blob/master/mustache.js#L82
var entityMap = {
"&": "&",
"<": "<",
">": ">",
'"': '"',
"'": ''',
"/": '/'
};
function escapeHtml(string) {
return String(string).replace(/[&<>"'\/]/g, function (s) {
return entityMap[s];
});
}
There is no need at all to escape the / in HTML.