I use http4s and scalaz.steam._
def usersService = HttpService{
case req # POST -> Root / "check" / IntVar(userId) =>{
//val jsonobj=parse(req.as[String])
val taskAsJson:String = write[req.body]
Ok(read[Task](taskAsJson))
}
}
For http4s, the request can get body as the type Process[Task, ByteVector]
The Process is scalaz.stream.process class. You can find details on scalaz.
I want to write a Task here can deal with JSON ByteVector and translate into a Map (like HashMap), layer by layer.
I mean, for example:
{
"id":"12589",
"customers":[{
"id": "898970",
"name":"Frank"
...
},
]
}
I do not know how to write the function via scalaz.stream.Process to change the JSON to mapobject.
I want response and to parse the JSON object, returning another refactored JSON object; how can I do it?
Related
I'm trying to parse / process a JSON response while receiving (streaming / chunked). But I cannot get this to work. When I receive JSON I can only process it after the whole response has been received.
If I test the same code with a static file download, this is working correctly.
Retrofit: 2.6.1
OkHTTP 3.12.0
When I download the 5MB file the notice 'Call OK' will be shown immediately and the 'download complete' later on. For the JSON files the 'Call OK' will take a while.
Logger.info("Start download")
val response = fileApi.download5MBFile() // Streaming
val response = dataHubApi.download3MBJson() //Does not stream
val response = articleApi.download10MBJson() // Does not stream
if (response.isSuccessful) {
Logger.info("Call ok")
val input = response.body()?.byteStream()
val buffer = ByteArray(8192)
var size = 0
while (true) {
val read = input!!.read(buffer)
if (read == -1) {
break
}
size += read
//Logger.info("Progress: ${size/1024/1024}mb")
}
Logger.info("Download complete")
} else {
Logger.info("Call not ok")
}
The factory methods for creating the APIs are all the same like this:
fun create(): FileApi {
val logLevel = Level.HEADERS
val retrofit = Retrofit.Builder()
.baseUrl(URL)
.client(getOkHttpClient(logLevel))
.build()
return retrofit.create(FileApi::class.java)
}
private fun getOkHttpClient(logLevel: Level): OkHttpClient {
return OkHttpClient.Builder()
.readTimeout(60L, TimeUnit.SECONDS) // default retrofit value is 10sec
.build()
}
And all the interface are also in the same format:
interface FileApi {
#GET("/5MB.zip")
#Streaming
suspend fun download5MB(): Response<ResponseBody>
}
What can I do to also stream the JSON?
If it is that large, probably it is because it contains a list of several items instead of one or a small chunk of elements.
[
{ item1 },
{ item2 },
...
]
Even if it is something as a firebase json representation you'll have something as:
{
"key1": {element },
"key2": {element },
"key3": { element },
...
}
If you have trouble parsing the complete json, there is a good chance that you'll also face some trouble saving that into a kotlin list, so the best thing would be to directly save it into a DB where you can manage individual objects or pages of objects later on.
So, the trick is to download the JSON as a streamed text that you parse as you stream, and determine when each element is split, so you will only have to process the element as JSON on the text chunk instead of all the text.
so for example with a stack you may start adding braces, but once you reach 0 level stack, you surely has the stream of one element and parse the text in it.
I need to find json response length. Sample response looks like below:
{
"resource": {
"name":"aaaaaaaaaaa",
"emailid":"bbbbbbbbb"
}
}
As two parameters are present in resource. So, i should have got response as 2.
Please let me know hoe i can find json length as 2
This is the working solution, try this
import groovy.json.JsonSlurper // import this class
def jsonText = '''{
"resource": {
"name":"aaaaaaaaaaa",
"emailid":"bbbbbbbbb"
}
}'''
def json = new JsonSlurper().parseText(jsonText)
println "Json length---------->"+json.resource.size()
If you have the JSON object, you don't need to parse JSON string to json, yo can directly do the following,
println jsonObject.resource.size() // Here resource is the key(sub node) inside your json
If you want to get the length of parent JSON key, just do as follows,
println jsonObject.size()
Based on your question, it appears that you would like to know the count of properties within a JSON object. So we can do that by following these steps:
STEP 1 : Parse the response string into JSON object
STEP 2 : Convert JSON object into groovy Map object
Step 3 : Call size() method on Map object to get the elements count within the map object
So your code would like this :
import groovy.json.JsonSlurper
def jsonSlurper = new JsonSlurper()
def response = jsonSlurper.parseText('{ "resource": {"name":"aaaaaaaaaaa","emailid":"bbbbbbbbb"}}')
def object = (Map)response.resource
log.info object.size()
So your output will be 2. You can try adding more elements to JSON object check if it works.
Hope this helps :)
I come from a Python Background and recently started programming using TypeScript and Angular2. I want to know how to obtain keys from a JSON object using TypeScript.
I have a response like so:
response.text()
I pass this object to a function
removeMetaData (my_data: string){
//(Various manipulation)...etc
}
i have read that I can call a json() method instead of text(). If I do that, what is the type I should use for my_data?
Then,
If my JSON looks like this:
{
"count": 100,
"next_page": "http://www.test.com/users/?page=2",
"prev_page": "http://www.test.com/users/?page=3",
"results":[
{
"username": "johnny"
},
Etc....
]
How do I parse that?
I've read I might have to use an interface but I don't understand how.
In python it's just response["next_page"] to get the key of a dictionary, then I can assign that value to a variable within the class. That is exactly what I'm trying to achieve within a component.
Thank you.
ADDITION
list() {
this.requestService.get(this.api_path)
.subscribe(
response => this.populate(response.json()),
error => this.response = error.text()
)
}
populate(payload: object) {
this.count = payload.count;
this.next = payload.next;
this.previous = payload.previous;
*payload.results => this.users = payload.results;******
}
Declare an interface which will be used as value object.
export interface IPage
{
count:number;
next_page:string;
prev_page:string;
results:Array<any>;
...
...
}
var my_data:IPage;
Now assign parsed json value to my_data and access all the properties with '.' operator i.e. my_data.count, my_data.results.
Feel free to throw any question.
If I do that, what is the type I should use for my_data?
Its just a javascript object.
As an example if you json looks like:
{
"foo": {
"bar": "bas"
}
}
Then in the parsed json (in variable someObj) the value someObj.foo.bar would be bas 🌹
Please pardon me if this is a repeat question. I have been through some of the questions/answers with a similar requirement but somehow got a bit overwhelmed and confused at the same time. My requirement is:
I get a JSON string/object as a request parameter. ( eg: params.timesheetJSON )
I then have to parse/iterate through it.
Here is the JSON that my grails controller will be receiving:
{
"loginName":"user1",
"timesheetList":
[
{
"periodBegin":"2014/10/12",
"periodEnd":"2014/10/18",
"timesheetRows":[
{
"task":"Cleaning",
"description":"cleaning description",
"paycode":"payCode1"
},
{
"task":"painting",
"activityDescription":"painting description",
"paycode":"payCode2"
}
]
}
],
"overallStatus":"SUCCESS"
}
Questions:
How can I retrieve the whole JSON string from the request? Does request.JSON be fine here? If so, will request.JSON.timesheetJSON yield me the actual JSON that I want as a JSONObject?
What is the best way to parse through the JSON object that I got from the request? Is it grails.converters.JSON? Or is there any other easy way of parsing through? Like some API which will return the JSON as a collection of objects by automatically taking care of parsing. Or is programatically parsing through the JSON object the only way?
Like I said, please pardon me if the question is sounding vague. Any good references JSON parsing with grails might also be helpful here.
Edit: There's a change in the way I get the JSON string now. I get the JSON string as a request paramter.
String saveJSON // This holds the above JSON string.
def jsonObject = grails.converters.JSON.parse(saveJSON) // No problem here. Returns a JSONObject. I checked the class type.
def jsonArray = jsonArray.timesheetList // No problem here. Returns a JSONArray. I checked the class type.
println "*** Size of jsonArray1: " + jsonArray1.size() // Returns size 1. It seemed fine as the above JSON string had only one timesheet in timesheetList
def object1 = jsonArray[1] // This throws the JSONException, JSONArray[1] not found. I tried jsonArray.getJSONObject(1) and that throws the same exception.
Basically, I am looking to seamlessly iterate through the JSON string now.
I have wrote some code that explains how this can be done, that you can see below, but to be clear, first the answers to your questions:
Your JSON String as you wrote above will be the contents of your POST payload to the rest controller. Grails will use its data binding mechanism to bind the incomming data to a Command object that your should prepare. It has to have fields corresponding to the parameters in your JSON String (see below). After you bind your command object to your actual domain object, you can get all the data you want, by simply operating on fields and lists
The way to parse thru the JSON object is shown in my example below. The incomming request is esentially a nested map, with can be simply accessed with a dot
Now some code that illustrates how to do it.
In your controller create a method that accepts "YourCommand" object as input parameter:
def yourRestServiceMethod (YourCommand comm){
YourClass yourClass = new YourClass()
comm.bindTo(yourClass)
// do something with yourClass
// println yourClass.timeSheetList
}
The command looks like this:
class YourCommand {
String loginName
List<Map> timesheetList = []
String overallStatus
void bindTo(YourClass yourClass){
yourClass.loginName=loginName
yourClass.overallStatus=overallStatus
timesheetList.each { sheet ->
TimeSheet timeSheet = new TimeSheet()
timeSheet.periodBegin = sheet.periodBegin
timeSheet.periodEnd = sheet.periodEnd
sheet.timesheetRows.each { row ->
TimeSheetRow timeSheetRow = new TimeSheetRow()
timeSheetRow.task = row.task
timeSheetRow.description = row.description
timeSheetRow.paycode = row.paycode
timeSheet.timesheetRows.add(timeSheetRow)
}
yourClass.timeSheetList.add(timeSheet)
}
}
}
Its "bindTo" method is the key piece of logic that understands how to get parameters from the incomming request and map it to a regular object. That object is of type "YourClass" and it looks like this:
class YourClass {
String loginName
Collection<TimeSheet> timeSheetList = []
String overallStatus
}
all other classes that are part of that class:
class TimeSheet {
String periodBegin
String periodEnd
Collection<TimeSheetRow> timesheetRows = []
}
and the last one:
class TimeSheetRow {
String task
String description
String paycode
}
Hope this example is clear enough for you and answers your question
Edit: Extending the answer according to the new requirements
Looking at your new code, I see that you probably did some typos when writting that post
def jsonArray = jsonArray.timesheetList
should be:
def jsonArray = jsonObject.timesheetList
but you obviously have it properly in your code since otherwise it would not work, then the same with that line with "println":
jsonArray1.size()
shuold be:
jsonArray.size()
and the essential fix:
def object1 = jsonArray[1]
shuold be
def object1 = jsonArray[0]
your array is of size==1, the indexing starts with 0. // Can it be that easy? ;)
Then "object1" is again a JSONObject, so you can access the fields with a "." or as a map, for example like this:
object1.get('periodEnd')
I see your example contains errors, which lead you to implement more complex JSON parsing solutions.
I rewrite your sample to the working version. (At least now for Grails 3.x)
String saveJSON // This holds the above JSON string.
def jsonObject = grails.converters.JSON.parse(saveJSON)
println jsonObject.timesheetList // output timesheetList structure
println jsonObject.timesheetList[0].timesheetRows[1] // output second element of timesheetRows array: [paycode:payCode2, task:painting, activityDescription:painting description]
i asked myself if there is a generic approach where i can parse requests in a generic way or add fields to a JSON response with every response sent.
I would like to receive and parse something like:
{
transactionId:456, // every response contains this
statuscode:1,
content:{ // only content is changing
{
class:"org.something.test",
id:123,
name:"test",
referenceIdToOtherClass:345
}
}
}
The contents of "content" should be converted e.g. to an instance of the given class.
The response should work also in such generic way (changing content, some fields always).
Would you reccomend a JSON marshaller, a Filter or something different?
Thank you
You can convert the json to a map with JSON.parse
import grails.converters.*
import org.codehaus.groovy.grails.web.json.*; // package containing JSONObject, JSONArray,...
def o = JSON.parse("{ foo: 'bar' }"); // Parse a JSON String
assert o instanceof JSONObject // In this case, JSON.parse returns a JSONObject instance
assert o instanceof Map // which implements the Map interface
assert o.foo == 'bar' // access a property
and then you can pass the map to your constructor to create a new instance
from http://grails.org/Converters+Reference