all. I'm trying to parse a CSV file using CsvParser, but I get an IOException: Stream closed error after reading in line 57 (of ~6500). Does anyone know what could be causing this? Here is a code snippet and the error:
#!/usr/bin/ groovy
package csvTest
#Grab ('com.xlson.groovycsv:groovycsv:1.0')
import com.xlson.groovycsv.CsvParser
def csvFile = new File("file.csv").withReader {
CsvParser.parseCsv(it)
}
csvFile.each {
println it
}
Caught: java.io.IOException: Stream closed
java.io.IOException: Stream closed
at au.com.bytecode.opencsv.CSVReader.getNextLine(CSVReader.java:245)
at au.com.bytecode.opencsv.CSVReader.readNext(CSVReader.java:212)
at au.com.bytecode.opencsv.CSVReader$readNext.call(Unknown Source)
at com.xlson.groovycsv.CsvIterator.hasNext(CsvIterator.groovy:72)
at csvTest.CsvTest.run(CsvTest.groovy:12)
The CsvParser is lazy, so reads rows as they are requested (rather than loading them all into memory.
The withReader call closes the Reader once the Closure is finished.
So when you try to do csvFile.each, the stream is closed.
This should work:
new File("file.csv").withReader {
def csvFile = CsvParser.parseCsv( it )
csvFile.each {
println it
}
}
Related
Working on a Jenkins pipeline, I observed what looks like infinite recursion causing a stack overflow when I use JsonOutput.toJson() on a net.sf.json.JSONObject that slurped a JSON string containing null.
The following minimal code demonstrates the problem:
// Jenkinsfile
#Library('libs#dev') libs
import groovy.json.JsonOutput
pipeline {
agent any
stages {
stage( "json" ) {
steps {
script {
my_lib.to_json_handbuilt_linkedhashmap()
my_lib.to_json_readjson()
my_lib.to_json_readjson_as_linkedhashmap()
}
}
}
}
}
// vars/my_lib.groovy
import groovy.json.JsonOutput
def asMap(j) {
return j as LinkedHashMap
}
// This function is successful.
def to_json_handbuilt_linkedhashmap() {
def d = [:]
d.issues = null
echo "---- handmade LinkedHashMap ----"
echo "d ${d}"
echo "d.getClass() ${d.getClass()}"
echo "JsonOutput.toJson(d) ${JsonOutput.toJson(d)}"
}
// This function fails from infinite recursion causing a stack overflow.
def to_json_readjson() {
def d = readJSON(text: '{ "issues" : null }')
echo "---- readJSON ----"
echo "d ${d}"
echo "d.getClass() ${d.getClass()}"
echo "JsonOutput.toJson(d) ${JsonOutput.toJson(d)}"
}
// This function also fails from infinite recursion causing a stack overflow.
def to_json_readjson_as_linkedhashmap() {
def d = asMap(readJSON(text: '{ "issues" : null }'))
echo "---- readJSON -> asMap ----"
echo "d ${d}"
echo "d.getClass() ${d.getClass()}"
echo "JsonOutput.toJson(d) ${JsonOutput.toJson(d)}"
}
In the code above, to_json_readjson() fails with a stack overflow when JsonOutput.toJson() is called with the net.sf.json.JSONObject returned by readJSON(text: '{ "issues" : null }').
The Jenkins console output is at the end of this post.
In to_json_handbuilt_linkedhashmap() JsonOutput.toJson() is successful when called with a handcrafted LinkedHashMap equivalent to { "issues" : null }.
Lastly, in to_json_readjson_as_linkedhashmap(), JsonOutput.toJson() again fails with a stack overflow when called with a LinkedHashMap created from a net.sf.json.JSONObject.
Question:
Can someone please explain what's causing the stack overflow when readJSON() and/or JsonOutput.toJson() are used with a JSON string that has null?
Because my handcrafted LinkedHashMap was successful with JsonOutput.toJson(), I thought the problem was passing JsonOutput.toJson() a net.sf.json.JSONObject.
But I think that theory is ruled out because in to_json_readjson_as_linkedhashmap(), I give JsonOutput.toJson() a LinkedHashMap, albeit created from a net.sf.json.JSONObject.
The problem would appear to be some combination of readJSON() and/or JsonOutput.toJson() that I'm failing to grasp.
I tried, but have given up trying to use a JsonSlurper, because I'm unable to even create an instance of one.
The (truncated) stack overflow error likely showing infinite recursion:
Posting build status of FAILED to bitbucket.company.comjava.lang.StackOverflowError
at java.io.PrintStream.flush(PrintStream.java:338)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.util.logging.StreamHandler.flush(StreamHandler.java:259)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:117)
at java.util.logging.Logger.log(Logger.java:738)
at java.util.logging.Logger.doLog(Logger.java:765)
at java.util.logging.Logger.throwing(Logger.java:1447)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.getProperties(DefaultGroovyMethods.java:391)
at groovy.json.JsonOutput.getObjectProperties(JsonOutput.java:327)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:320)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
Can you sidestep this immediate problem by using readJSON's returnPojo: true parameter, thereby solving your overall task sooner?
Getting plain old nulls rather than net.sf.json.JSONNull objects really helped me today, though my problem involved producing CSV rather than using JsonOutput.
I am trying to parse a json payload sent via a POST request to a NGINX/Openresty location. To do so, I combined Openresty's content_by_lua_block with its cjson module like this:
# other locations above
location /test {
content_by_lua_block {
ngx.req.read_body()
local data_string = ngx.req.get_body_data()
local cjson = require "cjson.safe"
local json = cjson.decode(data_string)
local endpoint_name = json['endpoint']['name']
local payload = json['payload']
local source_address = json['source_address']
local submit_date = json['submit_date']
ngx.say('Parsed')
}
}
Parsing sample data containing all required fields works as expected. A correct JSON object could look like this:
{
"payload": "the payload here",
"submit_date": "2018-08-17 16:31:51",
},
"endpoint": {
"name": "name of the endpoint here"
},
"source_address": "source address here",
}
However, a user might POST a differently formatted JSON object to the location. Assume a simple JSON document like
{
"username": "JohnDoe",
"password": "password123"
}
not containing the desired fields/keys.
According to the cjson module docs, using cjson (without its safe mode) will raise an error if invalid data is encountered. To prevent any errors being raised, I decided to use its safe mode by importing cjson.safe. This should return nil for invalid data and provide the error message instead of raising the error:
The cjson module will throw an error during JSON conversion if any invalid data is encountered. [...]
The cjson.safe module behaves identically to the cjson module, except when errors are encountered during JSON conversion. On error, the cjson_safe.encode and cjson_safe.decode functions will return nil followed by the error message.
However, I do not encounter any different error handling behavior in my case and the following traceback is shown in Openresty's error.log file:
2021/04/30 20:33:16 [error] 6176#6176: *176 lua entry thread aborted: runtime error: content_by_lua(samplesite:50):16: attempt to index field 'endpoint' (a nil value)
Which in turn results in an Internal Server Error:
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
I think a workaround might be writing a dedicated function for parsing the JSON data and calling it with pcall() to catch any errors. However, this would make the safe mode kind of useless. What am I missing here?
Your “simple JSON document” is a valid JSON document. The error you are facing is not related to cjson, it's a standard Lua error:
resty -e 'local t = {foo = 1}; print(t["foo"]); print(t["foo"]["bar"])'
1
ERROR: (command line -e):1: attempt to index field 'foo' (a number value)
stack traceback:
...
“Safeness” of cjson.safe is about parsing of malformed documents:
cjson module raises an error:
resty -e 'print(require("cjson").decode("[1, 2, 3"))'
ERROR: (command line -e):1: Expected comma or array end but found T_END at character 9
stack traceback:
...
cjson.safe returns nil and an error message:
resty -e 'print(require("cjson.safe").decode("[1, 2, 3"))'
nilExpected comma or array end but found T_END at character 9
I get this error when I try to retrieve items from Parse.com
In 10 attempts it failed around 6 times.
PFQuery *query = [PFQuery queryWithClassName:#"ServiceCatalogue"];
items = [[NSMutableArray alloc] init];
[query findObjectsInBackgroundWithBlock:^(NSArray *objects, NSError *error) {
if (!error) {
NSLog(#"ServiceCatalogue retrieved");
} else {
// Log details of the failure
NSLog(#"Error: %# %#", error, [error userInfo]);
}
}];
And getting error:
"Error: Error Domain=NSCocoaErrorDomain Code=3840 "The operation
couldn’t be completed. (Cocoa error 3840.)" (JSON text did not start
with array or object and option to allow fragments not set.)
UserInfo=0x7fcfbaadc480 {NSDebugDescription=JSON text did not start
with array or object and option to allow fragments not set.} {
NSDebugDescription = "JSON text did not start with array or object and option to allow fragments not set.";
}"
As indicated in this status report from Parse, it looks like their servers were creating an elevated number of errors.
2nd October 2015 - Elevated Error Rates - Incident Report for Parse
I'm trying to use Apache Bench to load test a group of large (4MB each) JSON requests. When running with a large file and many concurrent requests I get the following error:
Exception caught in RequestBodyHandler java.nio.channels.ClosedChannelException: null
Here is my ab command:
ab -p large.json -n 1000 -c 10 http://127.0.0.1:9000/json-tests
If I run this with no concurrency and only 10 requests it works fine. Increasing the number of requests or the concurrency causes this error to occur over and over.
My controller currently has no logic in it:
def addJsonTest = Action {
Ok("OK")
}
Here is the full error:
[error] play - Exception caught in RequestBodyHandler
java.nio.channels.ClosedChannelException: null
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.setInterestOps(AbstractNioWorker.java:506) [netty-3.9.3.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker$1.run(AbstractNioWorker.java:455) [netty-3.9.3.Final.jar:na]
at org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40) [netty-3.9.3.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) [netty-3.9.3.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) [netty-3.9.3.Final.jar:na]
This is just using Play in development mode, is there any setup or configuration for having Play handle multiple large requests?
Thanks!
You need to do it reactively, with Iterators
val iteratee = Iteratee.foldM[Array[Byte], Either[Result, String]](Right("start")) { case (str, bytes) =>
Future.successful(Left(Ok))
}
val parser = BodyParser(rh => iteratee)
def eatDust = Action(parser) { req =>
Ok
}
See these links.
https://www.playframework.com/documentation/2.2.x/Iteratees
Play 2.x : Reactive file upload with Iteratees
I'm a Groovy novice and trying to connect to the GitHub API from a groovy script however I am getting a strange error when HTTPBuilder tries to parse the JSON response.
My simple script is as follows
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7.2' )
import groovyx.net.http.*
import static groovyx.net.http.ContentType.*
import static groovyx.net.http.Method.*
def apiToken = '[api_token]'
def http = new HTTPBuilder( 'https://api.github.com/' )
// perform a GET request, expecting JSON response data
http.request( GET, JSON ) {
uri.path = '/orgs/octokit/repos'
headers.'Accept' = 'application/json'
headers.'Authorization' = 'Basic ' + (apiToken + ':').bytes.encodeBase64().toString()
headers.'User-Agent' = '[username]'
// response handler for a success response code:
response.success = { resp, json ->
println resp.statusLine
}
// handler for any failure status code:
response.failure = { resp ->
println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
}
}
But when the JSON is parsed I get the following Error
Sep 06, 2014 10:21:54 PM groovyx.net.http.HTTPBuilder$1 handleResponse
WARNING: Error parsing 'application/json; charset=utf-8' response
groovy.json.JsonException: expecting '}' or ',' but got current char '"' with an int value of 34
The current character read is '"' with an int value of 34
expecting '}' or ',' but got current char '"' with an int value of 34
line number 1
index number 256
[{"id":417862,"name":"octokit.rb","full_name":"octokit/octokit.rb","owner":{"login":"octokit","id":3430433,"avatar_url":"https://avatars.githubusercontent.com/u/3430433?v=2","gravatar_id":"43f38795089d56a2a7092b7d0c71fa76","url":"h
................................................................................................................................................................................................................................................................^
at groovy.json.internal.JsonParserCharArray.complain(JsonParserCharArray.java:162)
at groovy.json.internal.JsonParserCharArray.decodeJsonObject(JsonParserCharArray.java:152)
at groovy.json.internal.JsonParserCharArray.decodeValueInternal(JsonParserCharArray.java:196)
at groovy.json.internal.JsonParserCharArray.decodeJsonObject(JsonParserCharArray.java:140)
at groovy.json.internal.JsonParserCharArray.decodeValueInternal(JsonParserCharArray.java:196)
at groovy.json.internal.JsonParserCharArray.decodeJsonArray(JsonParserCharArray.java:346)
at groovy.json.internal.JsonParserCharArray.decodeValueInternal(JsonParserCharArray.java:192)
at groovy.json.internal.JsonParserCharArray.decodeValue(JsonParserCharArray.java:166)
at groovy.json.internal.JsonParserCharArray.decodeFromChars(JsonParserCharArray.java:45)
at groovy.json.internal.JsonParserCharArray.parse(JsonParserCharArray.java:409)
at groovy.json.internal.BaseJsonParser.parse(BaseJsonParser.java:121)
at groovy.json.JsonSlurper.parse(JsonSlurper.java:224)
at groovyx.net.http.ParserRegistry.parseJSON(ParserRegistry.java:280)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1074)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
at groovy.lang.Closure.call(Closure.java:423)
at groovy.lang.Closure.call(Closure.java:439)
at groovyx.net.http.HTTPBuilder.parseResponse(HTTPBuilder.java:560)
at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:489)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1070)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1044)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:515)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:434)
at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:383)
at groovyx.net.http.HTTPBuilder$request.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at sprint-snapshot.run(sprint-snapshot.groovy:11)
at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:258)
at groovy.lang.GroovyShell.run(GroovyShell.java:502)
at groovy.lang.GroovyShell.run(GroovyShell.java:491)
at groovy.ui.GroovyMain.processOnce(GroovyMain.java:650)
at groovy.ui.GroovyMain.run(GroovyMain.java:381)
at groovy.ui.GroovyMain.process(GroovyMain.java:367)
at groovy.ui.GroovyMain.processArgs(GroovyMain.java:126)
at groovy.ui.GroovyMain.main(GroovyMain.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:106)
at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:128)
Caught: groovyx.net.http.ResponseParseException: OK
groovyx.net.http.ResponseParseException: OK
at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:495)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1070)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1044)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:515)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:434)
at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:383)
at groovyx.net.http.HTTPBuilder$request.call(Unknown Source)
at sprint-snapshot.run(sprint-snapshot.groovy:11)
Caused by: groovy.json.JsonException: expecting '}' or ',' but got current char '"' with an int value of 34
The current character read is '"' with an int value of 34
expecting '}' or ',' but got current char '"' with an int value of 34
line number 1
index number 256
[{"id":417862,"name":"octokit.rb","full_name":"octokit/octokit.rb","owner":{"login":"octokit","id":3430433,"avatar_url":"https://avatars.githubusercontent.com/u/3430433?v=2","gravatar_id":"43f38795089d56a2a7092b7d0c71fa76","url":"h
................................................................................................................................................................................................................................................................^
at groovyx.net.http.ParserRegistry.parseJSON(ParserRegistry.java:280)
at groovyx.net.http.HTTPBuilder.parseResponse(HTTPBuilder.java:560)
at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:489)
... 7 more
The error seems to suggest invalid JSON however I have confirmed the response is indeed valid, as you would expect from the Github API.
What is even stranger, is if I remove the 'Authorization' header completely and send the request unauthenticated, it works fine :S
It's an issue with the latest release of Groovy. I had the same issue. Your code should work with version of Groovy before 2.3.0. Optionally you can try a different default JSONParserType for HTTPBuilder.
See
http://beta.groovy-lang.org/docs/groovy-2.3.0/html/gapi/groovy/json/JsonSlurper.html
http://alfonsorv.com/instruct-groovy-httpbuilder-to-handle-a-json-response-with-wrong-content-type/
Here is my code work-around
/*
* Normally we would use the method "request(GET, JSON)" to return a json result, but with the release
* of groovy 2.3.0, which included extensive changes to JSON parsing, the result returned from the
* Artifactory REST API call 'File List' could not be parsed by HTTPBuilder's default JSON parser.
*
* Specifying the content type as TEXT removes the parsing from the call. Code was added to parse the returned
* text into JSON using the more lenient 'LAX' parser.
*/
http.request( GET, TEXT ) {
//Since content type is text, we need to specify that a json response is acceptable.
headers.Accept = 'application/json'
headers.'User-Agent' = USER_AGENT
response.success = { respnse, reader ->
def jsonText = reader.text
def parser = new JsonSlurper().setType(JsonParserType.LAX)
def jsonResp = parser.parseText(jsonText)
//Add up the size of all the files in this directory
//
jsonResp.files.each {
directorySize = directorySize + Integer.parseInt("${it.size}".toString())
}
}
}
def bufferedText = response.entity.content.getText( ParserRegistry.getCharset( response ) ).trim()
def result = new JsonSlurper().parseText( bufferedText )
def 'test hello resource'(){
when:
def info =""
// http.auth.basic('sysadmin', 'yax64smi')
// http.ignoreSSLIssues()//add for this issue Groovy:SSLPeerUnverifiedException: peer not authenticated new version (0.7.1) of HttpBuilder introduces method:ignoreSSLIssues()
http.request( GET,JSON ) {
uri.path = '/hello/'
//headers.'Authorization' = 'Basic c3lzYWRtaW46eWF4NjRzbWk='
//headers.'Authorization' = "Basic ${"username:password".bytes.encodeBase64().toString()}"
headers.'User-Agent' = 'Mozilla/5.0 Ubuntu/8.10 Firefox/3.0.4'
headers.'Content-Type' = 'application/json'
headers.Accept = 'application/json'
response.success = { resp ->
def parser = new JsonSlurper().setType(JsonParserType.LAX)
def jsonResptext = parser.parseText( resp.entity.content.getText())
println "PUT response status: ${resp.statusLine}"
assert resp.statusLine.statusCode == 200 ||resp.statusLine.statusCode ==201 ||resp.statusLine.statusCode ==202
info = resp.statusLine.statusCode
// def bufferedText = resp.entity.content.getText( ParserRegistry.getCharset( resp ) ).trim()
println 'Headers: -----------Response data: -----'
println "Content-Type: ${resp.headers.'Content-Type'}"
println jsonResptext
}
response.failure = { resp -> println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}" }
}
println "Result:\n"+info
then:
assert info ==200 || info ==201 || info ==202
}