How do I get yason:encode-alist to return the encoded string instead of sending it to a stream? - json

I'm trying to encode a JSON string from an alist using YASON. The problem is, the return value I'm getting is the original alist I fed it. It's printing the JSON string, and according to the documentation, it goes to *STANDARD-OUTPUT*.
Simple example session:
(ql:quickload :yason)
To load "yason":
Load 1 ASDF system:
yason
; Loading "yason"
(:YASON)
* (defparameter starving-json-eater (yason:encode-alist '(("foo" . "bar") ("baz" . "qux"))))
{"foo":"bar","baz":"qux"}
STARVING-JSON-EATER
* starving-json-eater
(("foo" . "bar") ("baz" . "qux"))
I've tried passing 'starving-json-eater into the stream parameter, but I get an error:
* (setf starving-json-eater (yason:encode-alist '(("foo" . "bar") ("baz" . "qux")) 'starving-json-eater))
debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {1001E06783}>:
There is no applicable method for the generic function
#<STANDARD-GENERIC-FUNCTION SB-GRAY:STREAM-WRITE-CHAR (1)>
when called with arguments
(STARVING-JSON-EATER #\{).
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [RETRY] Retry calling the generic function.
1: [ABORT] Exit debugger, returning to top level.
((:METHOD NO-APPLICABLE-METHOD (T)) #<STANDARD-GENERIC-FUNCTION SB-GRAY:STREAM-WRITE-CHAR (1)> STARVING-JSON-EATER #\{) [fast-method]
How can I get {"foo":"bar","baz":"qux"} into starving-json-eater?

You can use WITH-OUTPUT-TO-STRING to temporarily bind a variable to an open stream which writes into a string. You may even bind the special variable *standard-output* so that you only change the dynamic context of your code without providing explicitly a different stream argument (like when you redirect streams with processes).
(with-output-to-string (*standard-output*)
(yason:encode-alist '(("a" . "b"))))
Note that binding *standard-output* means that anything that writes to *standard-output* will end up being written in the string during the extent of with-output-to-string. In the above case, the scope is sufficiently limited to avoid unexpectedly capturing output from nested code. You could also use a lexical variable to control precisely who gets to write to the string:
(with-output-to-string (json)
(yason:encode-alist '(("a" . "b")) json))

The trick is to create a throwaway string output stream to catch the value, and then grab it from later:
* (ql:quickload :yason)
To load "yason":
Load 1 ASDF system:
yason
; Loading "yason"
(:YASON)
* (defparameter sated-json-eater (make-string-output-stream))
SATED-JSON-EATER
* (yason:encode-alist '(("foo" . "bar") ("baz" . "qux")) sated-json-eater)
(("foo" . "bar") ("baz" . "qux"))
* (defparameter json-string (get-output-stream-string sated-json-eater))
JSON-STRING
* json-string
"{\"foo\":\"bar\",\"baz\":\"qux\"}"
This can be hidden away in a function:
(defun json-string-encode-alist (alist-to-encode)
(let ((stream (make-string-output-stream)))
(yason:encode-alist alist-to-encode stream)
(get-output-stream-string stream)))

Related

Cannot index into null array from function returned variable, or issues accessing regex data returned

I'm not sure if I'm returning the value from the function incorrectly, but when I try to access it's info, it has the above error,
Cannot index into a null array
I've tried a couple different ways, and I'm not sure if I'm not returning this correctly from the function, or if I'm just accessing the info returned incorrectly. Looking at Cannot index into null array, it looks like for him, some of his array had null values. But when I print my info to screen before I exit the function, it has info. How do I return the value found in the function such that I can loop through the contents in my main code and use one of the strings in the object? This is a continuation of parsing repeated pattern.
#parse data out of cpp code and loop through to further process
#function
Function Get-CaseContents{
[cmdletbinding()]
Param ( [string]$parsedCaseMethod, [string]$parseLinesGroupIndicator)
Process
{
# construct regex
$fullregex = [regex]"_stprintf[\s\S]*?_T\D*", # Start of error message, capture until digits
"(?<sdkErr>\d+)", # Error number, digits only
"\D[\s\S]*?", # match anything, non-greedy
"(?<sdkDesc>\((.+?)\))", # Error description, anything within parentheses, non-greedy
"([\s\S]*?outError\s*=(?<sdkOutErr>\s[a-zA-Z_]*))", # Capture OutErr string and parse out part after underscore later
"[\s\S]*?", # match anything, non-greedy
"(?<sdkSeverity>outSeverity\s*=\s[a-zA-Z_]*)", # Capture severity string and parse out part after underscore later
'' -join ''
# run the regex
$Values = $parsedCaseMethod | Select-String -Pattern $fullregex -AllMatches
# Convert Name-Value pairs to object properties
$result = foreach ($match in $Values.Matches){
[PSCustomObject][ordered]#{
sdkErr = $match.Groups['sdkErr']
sdkDesc = $match.Groups['sdkDesc']
sdkOutErr = $match.Groups['sdkOutErr']
sdkSeverity = ($match.Groups['sdkSeverity'] -split '_')[-1] #take part after _
}
}
#Write-Host "result:" $result -ForegroundColor Green
$result
return $Values
...
#main code
...
#call method to get case info (sdkErr, sdkDesc, sdkOutErr, sdkSeverity)
$ValuesCase = Get-CaseContents -parsedCaseMethod $matchFound -parseLinesGroupIndicator "_stprintf" #need to get returned info back
$result = foreach ($match in $ValuesCase.Matches){
[PSCustomObject][ordered]#{
sdkErr = $match.Groups['sdkErr']
sdkDesc = $match.Groups['sdkDesc']
sdkOutErr = $match.Groups['sdkOutErr']
sdkSeverity = ($match.Groups['sdkSeverity'] -split '_')[-1] #take part after _
} #result
} #foreach ValuesCase
The example of string sent to the function to parse is:
...
case kRESULT_STATUS_Undefined_Opcode:
_stprintf( outDevStr, _T("8004 - (Comm. Err 04) - %s(Undefined Opcode)"), errorStr);
outError = INVALID_PARAM;
outSeverity = CCA_WARNING;
break;
case kRESULT_STATUS_Comm_Timeout:
_stprintf( outDevStr, _T("8005 - (Comm. Err 05) - %s(Timeout sending command)"), errorStr);
outError = INVALID_PARAM;
outSeverity = CCA_WARNING;
break;
case kRESULT_STATUS_TXD_Failed:
_stprintf( outDevStr, _T("8006 - (Comm. Err 06) - %s(TXD Failed--Send buffer overflow.)"), errorStr);
outError = INVALID_PARAM;
outSeverity = CCA_WARNING;
break;
...
Another thing I tried is (but it also had the index into null array issue):
foreach($matchRegex in $ValuesCase.Matches)
{
$sdkOutErr = $matchRegex.Groups['sdkOutErr']
Write-Host sdkOutErr -ForegroundColor DarkMagenta
}
Ultimately, I need to grab $sdkOutErr to further process. I'll need to use the other variables too in the returned object, but this is the first one I need. I love the way the output is formatted in the function, but probably don't know how to return the info and use what is returned. I'm not sure what to search for to figure out the issue other than the error message, which leads me to believe I'm returning the info wrong. I don't think I need to return $result, because I think that's just a string with the values in the $values.Matches in the function. I need to access the values returned as I mentioned.
I checked, and the contents sent to the function is not blank.
I tried returning $results, and it looks like this when I write-Host, which would be difficult to access each sdkOutErr:
#{sdkErr=1000; sdkDesc=(Out of Memory); sdkOutErr= NO_MEMORY; sdkSeverity=FATAL} #{sdkErr=1002; sdkDesc=(Failed to load DLL); sdkOutErr= OTHER_ERROR; sdkSeverity=FATAL} #{sdkErr=1003; sdkDesc=(Failed to load DLL); sdk
OutErr= OTHER_ERROR; sdkSeverity=FATAL} #{sdkErr=1004; sdkDesc=(Failed to open); sdkOutErr= OTHER_ERROR; sdkSeverity=FATAL} #{sdkErr=1005; sdkDesc=(Unable to access the specified profile); sdkOutErr= OTHER_ERROR; sdkSeverity=
FATAL} #{sdkErr=100 ...
How can I return this from the function so that it's not a null array/index, and the data is accessible if I use a foreach loop (or two) in the main code to get the sdkOutErr (to start).
I'm fairly new to (complicated)powershell and I have a feeling I need a map inside the array in my function, but I'm not sure.
Before I returned the function Values or results, it was printing something like this out. Once I added in main $ValuesCase=Get-CaseContents... (returning $values from function), or $parsedCase = Get-CaseContents... (returning $results from function), it stopped showing this on the screen:
sdkErr sdkDesc sdkOutErr sdkSeverity
------ ------- --------- -----------
1000 (Out of Memory) NO_MEMORY FATAL
1002 (Failed to load DLL) OTHER_ERROR FATAL
1003 (Failed to load DLL) OTHER_ERROR FATAL
1004 (Failed to open) OTHER_ERROR FATAL
I tried returning $results, and it looks like this when I write-Host, which would be difficult to access each sdkOutErr:
Getting all the sdkOutErr values is not as difficult as you might imagine:
$results.sdkOutErr # this will output the `sdkOutErr` value from each object in the array
Or, outside the function:
(Get-CaseContents -parsedCaseMethod $matchFound -parseLinesGroupIndicator "_stprintf").sdkOutErr
Another option, which might perform better if the result set is large, is to use ForEach-Object to grab just the sdkOutErr values:
$fullResults = Get-CaseContents -parsedCaseMethod $matchFound -parseLinesGroupIndicator "_stprintf"
$sdkOutErrValuesOnly = $fullResults |ForEach-Object -MemberName sdkOutErr

How to stream a large CSV response from a compojure API so that the whole response is not held in memory at once?

I'm new to using compojure, but have been enjoying using it so far. I'm
currently encountering a problem in one of my API endpoints that is generating
a large CSV file from the database and then passing this as the response body.
The problem I seem to be encountering is that the whole CSV file is being kept
in memory which is then causing an out of memory error in the API. What is the
best way to handle and generate this, ideally as a gzipped file? Is it possible
to stream the response so that a few thousand rows are returned at a time? When
I return a JSON response body for the same data, there is no problem returning
this.
Here is the current code I'm using to return this:
(defn complete
"Returns metrics for each completed benchmark instance"
[db-client response-format]
(let [benchmarks (completed-benchmark-metrics {} db-client)]
(case response-format
:json (json-grouped-output field-mappings benchmarks)
:csv (csv-output benchmarks))))
(defn csv-output [data-seq]
(let [header (map name (keys (first data-seq)))
out (java.io.StringWriter.)
write #(csv/write-csv out (list %))]
(write header)
(dorun (map (comp write vals) data-seq))
(.toString out)))
The data-seq is the results returned from the database, which I think is a
lazy sequence. I'm using yesql to perform the database call.
Here is my compojure resource for this API endpoint:
(defresource results-complete [db]
:available-media-types ["application/json" "text/csv"]
:allowed-methods [:get]
:handle-ok (fn [request]
(let [response-format (keyword (get-in request [:request :params :format] :json))
disposition (str "attachment; filename=\"nucleotides_benchmark_metrics." (name response-format) "\"")
response {:headers {"Content-Type" (content-types response-format)
"Content-Disposition" disposition}
:body (results/complete db response-format)}]
(ring-response response))))
Thanks to all the suggestion that were provided in this thread, I was able to create a solution using piped-input-stream:
(defn csv-output [data-seq]
(let [headers (map name (keys (first data-seq)))
rows (map vals data-seq)
stream-csv (fn [out] (csv/write-csv out (cons headers rows))
(.flush out))]
(piped-input-stream #(stream-csv (io/make-writer % {})))))
This differs from my solution because it does not realise the sequence using dorun and does not create a large String object either. This instead writes to a PipedInputStream connection asynchronously as described by the documentation:
Create an input stream from a function that takes an output stream as its
argument. The function will be executed in a separate thread. The stream
will be automatically closed after the function finishes.
Your csv-output function completely realises the dataset and turns it into a string. To lazily stream the data, you'll need to return something other than a concrete data type like a String. This suggests ring supports returning a stream, that can be lazily realised by Jetty. The answer to this question might prove useful.
I was also struggling with the streaming of large csv file. My solution was to use httpkit-channel to stream every single line of the data-seq to the client and then close the channel. My solution looks like that:
[org.httpkit.server :refer :all]
(fn handler [req]
(with-channel req channel (let [header "your$header"
data-seq ["your$seq-data"]]
(doseq [line (cons header data-seq)]
(send! channel
{:status 200
:headers {"Content-Type" "text/csv"}
:body (str line "\n")}
false))
(close channel))))

vimscript: commands that work in mappings, but not in functions

How can I rewrite these 2 commands, which work fine in a mapping, so that they will work in a function?
:if has_key(glos,#g)==1<cr>:let #j=eval('glos.'.#g)<cr>
The function concerned is executed by vim without comment, but #j remains unchanged, as if they had failed, but no message/error is generated.
Here is the complete code involved, the command that loads the dictionary, the function that does not work, and the mapping from that function that does.
" read the glossary into the dictionary, glos
let glos=eval(join(readfile("glossary.dict")))
" 2click item of interest and this will
" send image filepath to xv
" if item all-caps find same at start of its line
" If capitalized at eol find same at start of its line
" if all lower-case at eol find next occurrence of same
" look lower-case or capitalized word up in glossary.txt
" find _\d\+ (page no.) alone on its line in text
com! F call F()
function! F()
normal "ayiw"cyE"by$
let #c=substitute(#c,"[,.?':;!]\+","","g")
if #c=~'images\/ss\d\d\d*'
let #i='!display -geometry +0+0 '.#c.' &'
pkill display
#i
elseif #c==toupper(#c)
let #n=search('^'.#c,'sw')
elseif #c!=#b
let #f=3
let #g=tolower(#c)
while #f>0
try
let #j=eval('glos.'.#g)
catch
let #f=#f-1
let #g=strpart(#g,0,strlen(#g)-1)
continue
endtry
break
endwh
if #f>0
let #h=substitute(#j," glosimgs.*",'','')
if #h!=#j
let #i='!xv -geometry +0+380 '.substitute(#j,'^.\{-}\( glosimgs.*\)$','\1','').' &'
!pkill xv
#i
endif
echo #h
else
echo 'No matching entry for '.#c
endif
elseif #c=~'\u\l\+$'
let #n=search('^'.#c,'sw')
elseif #c=~'\l\+$'
norm *
elseif #c=~'^_\w\+$'
let #/='^'.#c.'$'
norm nzz
endif
endfunction
map <silent> <2-LeftMouse> "ayiw"cyE"by$:let #c=substitute(#c,"[,.?':;!]\+","","g")<cr>:if #c=~'images\/ss\d\d\d*'<cr>:let #i='!display -geometry +0+0 '.#c.' &'<cr>:pkill display<cr>:#i<cr>:elseif #c==toupper(#c)<cr>:let #n=search('^'.#c,'sw')<cr>:elseif #c!=#b<cr>:let #f=3<cr>:let #g=tolower(#c)<cr>:while #f>0<cr>:try<cr>:let #j=eval('glos["'.#g.'"]')<cr>:catch<cr>:let #f=#f-1<cr>:let #g=strpart(#g,0,strlen(#g)-1)<cr>:continue<cr>:endtry<cr>:break<cr>:endwh<cr>:if #f>0<cr>:let #h=substitute(#j," glosimgs.*",'','')<cr>:if #h!=#j<cr>:let #i='!xv -geometry +0+380 '.substitute(#j,'^.\{-}\( glosimgs.*\)$','\1','').' &'<cr>:!pkill xv<cr>:#i<cr>:endif<cr><cr<cr>>:echo #h<cr>:else<cr>:echo 'No matching entry for '.#c<cr>:endif<cr>:elseif #c=~'\u\l\+$'<cr>:let #n=search('^'.#c,'sw')<cr>:elseif #c=~'\l\+$'<cr>:norm *<cr>:elseif #c=~'^_\w\+$'<cr>:let #/='^'.#c.'$'<cr>:norm nzz<cr>:endif<cr>
Specifically, I should have written:
:if has_key(**g:**glos,#g)==1:let #j=eval('**g:**glos.'.#g)
:h g: goes straight to the heart of the matter; in a function all references are local to that function, so references to any variable outside the function must be global, by prepending 'g:' to the variable name. As I created the dictionary independent of the function, the function must reference it as a global item.
Of course, if you are not aware of 'g:', it is rather difficult to find that help reference, but that's a frequent problem using help.
And, of course, the ** surrounding g: aren't required, that's what this site gives you in lieu of bolded text, apparently.

Lua-cjson -> require("cjson") successful, then errors when calling cjson.encode

I'm trying to encode/decode JSON in Lua using CJSON. I downloaded lua-cjson using Luarocks (http://www.kyne.com.au/~mark/software/lua-cjson-manual.html).
In the Lua interpreter, I'm using an example from the cjson manual:
> local cjson = require "cjson"
> value = { true, { foo = "bar" } }
> json_text = cjson.encode(value)
stdin:1: attempt to index a nil value (global 'cjson')
stack traceback:
stdin:1: in main chunk
[C]: in ?
I know that cjson is being found, because if I were to do ' require "foobar" ', Lua would error. It's just not able to use the module. Any help would be appreciated.
Each line in an interactive session is a separate chunk. So, the local variable created in line 1 no longer exists in the next lines. Note how the error message mentions a global variable. Try removing local.

How do I get an unhandled exception to be reported in SML/NJ?

I have the following SML program in a file named testexc.sml:
structure TestExc : sig
val main : (string * string list -> int)
end =
struct
exception OhNoes;
fun main(prog_name, args) = (
raise OhNoes
)
end
I build it with smlnj-110.74 like this:
ml-build sources.cm TestExc.main testimg
Where sources.cm contains:
Group is
csx.sml
I invoke the program like so (on Mac OS 10.8):
sml #SMLload testimg.x86-darwin
I expect to see something when I invoke the program, but the only thing I get is a return code of 1:
$ sml #SMLload testimg.x86-darwin
$ echo $?
1
What gives? Why would SML fail silently on this unhandled exception? Is this behavior normal? Is there some generic handler I can put on main that will print the error that occurred? I realize I can match exception OhNoes, but what about larger programs with exceptions I might not know about?
The answer is to handle the exception, call it e, and print the data using a couple functions available in the system:
$ sml
Standard ML of New Jersey v110.74 [built: Tue Jan 31 16:23:10 2012]
- exnName;
val it = fn : exn -> string
- exnMessage;
val it = fn : exn -> string
-
Now, we have our modified program, were we have the generic handler tacked on to main():
structure TestExc : sig
val main : (string * string list -> int)
end =
struct
exception OhNoes;
open List;
fun exnToString(e) =
List.foldr (op ^) "" ["[",
exnName e,
" ",
exnMessage e,
"]"]
fun main(prog_name, args) = (
raise OhNoes
)
handle e => (
print("Grasshopper disassemble: " ^ exnToString(e));
42)
end
I used lists for generating the message, so to make this program build, you'll need a reference to the basis library in sources.cm:
Group is
$/basis.cm
sources.cm
And here's what it looks like when we run it:
$ sml #SMLload testimg.x86-darwin
Grasshopper disassemble: [OhNoes OhNoes (more info unavailable: ExnInfoHook not initialized)]
$ echo $?
42
I don't know what ExnInfoHook is, but I see OhNoes, at least. It's too bad the SML compiler didn't add a basic handler for us, so as to print something when there was an unhandled exception in the compiled program. I suspect ml-build would be responsible for that task.