Below is my JSON format:
{"copybook": {
"item": {
"storage-length": 1652,
"item": [
{
"storage-length": 40,
"level": "05",
"name": "OBJECT-NAME",
"display-length": 40,
"position": 1,
"picture": "X(40)"
},
{
"storage-length": 8,
"occurs-min": 0,
"level": "05",
"name": "C-TCRMANKEYBOBJ-OFFSET",
"numeric": true,
"display-length": 8,
"position": 861,
"occurs": 99,
"depending-on": "C-TCRMANKEYBOBJ-COUNT",
"picture": "9(8)"
}
],
"level": "01",
"name": "TCRMCONTRACTBOBJ",
"display-length": 1652,
"position": 1
},
"filename": "test.cbl"
}}
How can I parse this json and convert it to CSV format? I am using Scala default JSON parser. The main problem I am facing is that I can not use case class to extract the data as all the item names are not same in item array.
This format is ok for me, please follow this link and paste the JSON - https://konklone.io/json/. Any scala code is appreciated. I am getting the below data:
implicit val formats = DefaultFormats
val json2 = parse(jsonString, false) \\ "item"
val list = json2.values.asInstanceOf[List[Map[String, String]]]
for (obj <- list) {
//println(obj.keys)
//obj.values
println (obj.toList.mkString(","))
}
(name,OBJECT-NAME),(storage-length,40),(picture,X(40)),(position,1),(display-length,40),(level,05)
(name,C-TCRMANKEYBOBJ-OFFSET),(storage-length,8),(occurs-min,0),(occurs,99),(picture,9(8)),(position,861),(numeric,true),(depending-on,C-TCRMANKEYBOBJ-COUNT),(display-length,8),(level,05)
https://circe.github.io/circe/ can work better for you in terms of traversing. Just try and read.
Related
I have a socket response that is this:
{"op":0,"d":{"author":{"id":"6699457769390473216","name":"Test","verified":false},"unixTime":1597277057132,"id":"6699465549836976128","group":"64632423765273287342","content":"Yo","_id":"5f34838198980c0023fa49e3"},"t":"MESSAGE"}
and I need to access the "d" object, I've tried doing
print(JSON(data)["d"])
and it just Returns null every time.
If data is of type String, you are probably using the wrong init method to initialize the JSON object. Try using init(parseJSON:) like this:
let jsonString = """
{
"op": 0,
"d": {
"author": {
"id": "6699457769390473216",
"name": "Test",
"verified": false
},
"unixTime": 1597277057132,
"id": "6699465549836976128",
"group": "64632423765273287342",
"content": "Yo",
"_id": "5f34838198980c0023fa49e3"
},
"t": "MESSAGE"
}
"""
let json = JSON(parseJSON: jsonString)
print(json["d"])
I want to parse json file in spark 2.0(scala). Next i want to save the data.. in Hive table.
How can i parse json file by using scala?
json file example) metadata.json:
{
"syslog": {
"month": "Sep",
"day": "26",
"time": "23:03:44",
"host": "cdpcapital.onmicrosoft.com"
},
"prefix": {
"cef_version": "CEF:0",
"device_vendor": "Microsoft",
"device_product": "SharePoint Online",
},
"extensions": {
"eventId": "7808891",
"msg": "ManagedSyncClientAllowed",
"art": "1506467022378",
"cat": "SharePoint",
"act": "ManagedSyncClientAllowed",
"rt": "1506466717000",
"requestClientApplication": "Microsoft SkyDriveSync",
"cs1": "0bdbe027-8f50-4ec3-843f-e27c41a63957",
"cs1Label": "Organization ID",
"cs2Label": "Modified Properties",
"ahost": "cdpdiclog101.cgimss.com",
"agentZoneURI": "/All Zones",
"amac": "F0-1F-AF-DA-8F-1B",
"av": "7.6.0.8009.0",
}
},
Thanks
You can use something like:
val jsonDf = sparkSession
.read
//.option("wholeFile", true) if its not a Single Line JSON
.json("resources/json/metadata.json")
jsonDf.printSchema()
jsonDf.registerTempTable("metadata")
More details about this https://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables
I'm building a gatling 2.1.3 scenario and I need to extract data from a json body.
Example of the body:
[
{
"objectId": "FirstFoo",
"pvrId": "413"
"type": "foo",
"name": "the first name",
"fooabilities": {
"foo1": true,
"foo2": true
},
"versions": [23, 23, 23, 23, 23, 23, 24, 23, 23],
"logo": [
{
"type": "firstlogo",
"width": 780,
"height": 490,
"url": "firstlogos/HD/{resolution}.png"
}
]
},
{
"objectId": "SecondFoo",
"pvrId": "414"
"type": "foo",
"name": "the second name",
"fooabilities": {
"foo1": true,
"foo2": false
},
"versions": [23, 23, 23, 23, 23, 23, 24, 23, 23],
"logo": [
{
"type": "secondlogo",
"width": 780,
"height": 490,
"url": "secondlogos/HD/{resolution}.png"
}
]
}
]
and I have this code trying to extract de data:
exec(
http("get object")
.get(commons.base_url_ws + "/my-resource/2.0/object/")
.headers(commons.headers_ws_session).asJSON
.check(jsonPath("$..*").findAll.saveAs("MY_RESULT"))) (1)
.exec(session => {
foreach("${MY_RESULT}", "result") { (2)
exec(session => {
val result= session("result").as[Map[String, Any]]
val objectId = result("objectId")
val version = result("version")
session.set("MY_RESULT_INFO", session("MY_RESULT_INFO").as[List[(String,Int)]] :+ Tuple2(objectId, version))
})
}
session
})
My goal is:
To extract the objectId and the 9th value from the version array.
I want it to look as Vector -> [(id1, version1),(id2, version2)] in the session to reuse later in another call to the API.
My concerns are:
(1) Is this going to create entries in the session with the complete sub objects? Because in other answers I was that is was always a map that was saved ("id" = [{...}]) and here I do not have ids.
(2) In the logs, I see that the session is loaded with a lot of data, but this foreach is never called. What could cause this ?
My experience in Scala is of a beginner - there may be issues I did not see.
I have looked into this issue: Gatling - Looping through JSON array and it is not exactly answering my case.
I found a way to do it with a regex.
.check(regex("""(?:"objectId"|"version"):"(.*?)",.*?(?:"objectId"|"version"):\[(?:.*?,){9}([0-9]*?),.*?\]""").ofType[(String, String)].findAll saveAs ("OBJECTS")))
I can then use this
foreach("${OBJECTS}", "object") {
exec(
http("Next API call")
.get(commons.base_url_ws + "/my-resource/2.0/foo/${object._1}/${object._2}")
[...]
}
I'm using the next Json
{
"ID": 8,
"MenuItems": [
{
"ID": 38,
"Name": "Home",
"URL": "{\"PageLayout\":\"Home\",\"Icon\":\"home\"}",
"Culture": "en",
"Children": null
},
{
"ID": 534,
"Name": "GUIDE ",
"URL": "{''PageLayout'':''Page A'', ''Icon'':''A''}",
"MenuType": 1,
"PageID": 0,
"Culture": "en",
"Children": [
{
"ID": 6,
"Name": "Form A",
"URL": "[''Type'':''Form A'',''Icon'':''Form'',''ItemID'':\"358\"]",
"Culture": "he",
"RuleID": 0
},
{
"ID": 60,
"Name": "Drama",
"URL": "[''Type'':''Form B'',''Icon'':''Form'',''ItemID'':\"3759\"]",
"Culture": "en",
"RuleID": 0
}
]
}
]
}
i'm using Groovy script in soapUI and i need to:
Assert the exitance of node that has the name GUIDE
Extract a list of all Itemsid
You can parse the JSON content using JsonSlurper and then work with the results like so:
import groovy.json.JsonSlurper
// Assuming your JSON is stored in "jsonString"
def jsonContent = new JsonSlurper().parseText(jsonString)
// Assert node exists with name GUIDE
assert(jsonContent.MenuItems.Name.contains("GUIDE"))
// Get list of ItemIDs
def itemList = jsonContent.MenuItems.Children.URL.ItemID[0].toList()
// List the items
itemList.each {log.info it}
Note that the above will fail given your current example, because of a few issues. Firstly, Name contains "GUIDE " (trailing space) rather than "GUIDE" (so you will need to adapt accordingly). Secondly, it is invalid JSON; the URL nodes have various erroneous characters.
On a side note, if you first need to retrieve your JSON from the response associated with a previous TestStep (say one called "SendMessage") within the existing testCase, this can be done as follows:
def jsonString = context.testCase.getTestStepByName("SendMessage").testRequest.response.getContentAsString()
Given the following Json array:
{
"success": true,
"data": [
{
"id": 594,
"stage_id": 15,
"title": "test deal",
"value": 0,
"currency": "EUR",
"add_time": "2014-03-18 17:45:51",
"update_time": "2014-03-24 13:30:27",
"stage_change_time": "2014-03-24 13:30:27",
"active": true,
"deleted": false,
"status": "open",
"expected_close_date": null,
"stage_order_nr": 1,
"person_name": "test"
},
{
"id": 601,
"stage_id": 15,
"title": "test deal2 deal",
"value": 0,
"currency": "EUR",
"add_time": "2014-03-24 14:11:00",
"update_time": "2014-03-24 14:11:00",
"stage_change_time": "2014-03-24 14:11:00",
"active": true,
"deleted": false,
"status": "open",
"expected_close_date": null,
"stage_order_nr": 1,
"person_name": "test deal2"
}
],
"additional_data": {
"pagination": {
"start": 0,
"limit": 100,
"more_items_in_collection": false
}
}
}
I want to get a List of deals out of it and I am trying it like so
case class Deal(id: Long, stage_id: Long)
def getAllDeals(): List [Deal] = {
var holder : WSRequestHolder = WS.url(PipeDriveApiBaseUrl + "/deals")
val complexHolder: WSRequestHolder = holder.withQueryString("filter_id" -> "9", "api_token" -> SalesManagerApiKey)
val futureResponse: Future[Response] = complexHolder.get()
implicit val dealReader = Json.reads[List[Deal]]
val futureJson: Future[List[Deal]] = futureResponse.map(
response => (response.json \ "data").validate[List[Deal]].get
)
I get the exception No unapply function found which is related to the implicit reads value. But commenting it out, I will get No Json deserializer found for type List[models.Deal]. Try to implement an implicit Reads or Format for this type.
I couldn't solve the problem with these answers here and here.
What do I miss or misunderstand?
Instead of defining an implicit Json.reads for List[Deal], create one for Deal:
implicit val dealReader = Json.reads[Deal]
Play already has a built-in implicit JSON reads converter for Lists. Actually, it has one for all Traversables. You can look at the code, but it's a bit hard to read. The thing is: Play can convert JSON lists to List objects. What it doesn't know is how to read/convert a Deal, and that's why you need the implicit definition mentioned above.