Extract individual values from JArray of Lists of JInts - json

I am practising Scala and Akka Streams on Stripe-payment API. I am looking to extract the transaction date: created from the Json response.
This is the response (showing 2 transactions):
{
"object": "list",
"data": [
{
"id": "txn_1Fqdyl2eZvKYlo2CXfUAnz1z",
"object": "balance_transaction",
"amount": 10000,
"available_on": 1577145600,
"created": 1576581159,
"currency": "usd",
"description": null,
"exchange_rate": null,
"fee": 320,
"fee_details": [
{
"amount": 320,
"application": null,
"currency": "usd",
"description": "Stripe processing fees",
"type": "stripe_fee"
}
],
"net": 9680,
"source": "ch_1Fqdyk2eZvKYlo2CwjSNI1vO",
"status": "available",
"type": "charge"
},
{
"id": "txn_1Fqdyk2eZvKYlo2C7MuBhLpe",
"object": "balance_transaction",
"amount": 2000,
"available_on": 1577145600,
"created": 1576581158,
"currency": "usd",
"description": "テスト支払い",
"exchange_rate": null,
"fee": 88,
"fee_details": [
{
"amount": 88,
"application": null,
"currency": "usd",
"description": "Stripe processing fees",
"type": "stripe_fee"
}
],
"net": 1912,
"source": "ch_1Fqdyk2eZvKYlo2Ccg96i1QQ",
"status": "available",
"type": "charge"
}
],
"has_more": true,
"url": "/v1/balance_transactions"
}
It can vary from 1 to 100 transactions. So far I have been able to extract the value I need when there is only 1 transaction. I need to be able to extract values when there are multiple results. At the moment the value is hardcoded as: case Some(JArray(List(JInt(int)))).
My code:
def epochToDate(epochMillis: BigInt): String = {
val convertedToLong = epochMillis * 1000L
val df:SimpleDateFormat = new SimpleDateFormat("dd-MM-yyyy HH:mm:ss")
df.format(convertedToLong)
}
val stripe = new Stream("txn_1Fqdyl2eZvKYlo2Cn0OSmwaY", "starting_after", hasMoreFromJson, 1, idFromJson)
stripe.asSource().runForeach(id => {
val rawValue = (parse(id) \ "data" \ "created").toOption
rawValue match {
case Some(JArray(List(JInt(int)))) => println("Date: " + epochToDate(int))
case _ => println("Value not present in JSON")
}
})

Try
val rawValue = (parse(id) \ "data" \\ "created").toOption
rawValue match {
case createdObjOpt: Option[JObject] =>
createdObjOpt match {
case Some(createdObj) =>
createdObj.obj.foreach { created =>
println(created)
}
case None => println("oh no!")
}
case _ => println("Value not present in JSON")
}

Related

How to get required json output from complex nested json format.?

My original file is in CSV format which I have converted to python JSON array to JSON Sring.
jsonfile
<class 'list'>
<class 'dict'>
[
{
"key": "timestamp",
"source": "eia007",
"turnover": "65million",
"url": "abc.com",
"record": "",
"loc.reg": "nord000",
"loc.count": "abs39i5",
"loc.town": "cold54",
"co.gdp": "nscrt77",
"co.pop.min": "min50",
"co.pop.max": "max75",
"co.rev": "",
"chain.system": "5t5t5",
"chain.type": "765ef",
"chain.strat": "",
}
]
I would like to get the output as below:
{
"timestamp001": {
"key": "timestamp001",
"phNo": "ner007",
"turnover": "65million",
"url": "abc.com",
"record": "",
"loc": {
"reg": "nord000",
"count": "abs39i5",
"town": "cold54"
},
"co": {
"form": "nscrt77",
"pop": {
"min": "min50",
"max": "max75"
},
"rev: ""
},
"chain":{
"system": "5t5t5",
"type": "765ef",
"strat": ""
}
...
}
...
}
]
I have tried different options; tried to enumerate, but cannot get the required output. Please help me with this. Thanks in advance.
You can use something like this to create the nested dict:
import json
def unflatten(somedict):
unflattened = {}
for key, value in somedict.items():
splitkey = key.split(".")
print(f"doing {key} {value} {splitkey}")
# subdict is the dict that goes deeper in the nested structure
subdict = unflattened
for subkey in splitkey[:-1]:
# if this is the first time we see this key, add it
if subkey not in subdict:
subdict[subkey] = {}
# shift the subdict a level deeper
subdict = subdict[subkey]
# add the value
subdict[splitkey[-1]] = value
return unflattened
data = {
"key": "timestamp",
"source": "eia007",
"turnover": "65million",
"url": "abc.com",
"record": "",
"loc.reg": "nord000",
"loc.count": "abs39i5",
"loc.town": "cold54",
"co.gdp": "nscrt77",
"co.pop.min": "min50",
"co.pop.max": "max75",
"co.rev": "",
"chain.system": "5t5t5",
"chain.type": "765ef",
"chain.strat": "",
}
unflattened = unflatten(data)
print(json.dumps(unflattened, indent=4))
Which produces:
{
"key": "timestamp",
"source": "eia007",
"turnover": "65million",
"url": "abc.com",
"record": "",
"loc": {
"reg": "nord000",
"count": "abs39i5",
"town": "cold54"
},
"co": {
"gdp": "nscrt77",
"pop": {
"min": "min50",
"max": "max75"
},
"rev": ""
},
"chain": {
"system": "5t5t5",
"type": "765ef",
"strat": ""
}
}
Cheers!

Ruby API call to get data from complex json

I'm making an API GET call using Ruby - the call is made to a Learning Management System and returns the following JSON:
{
"id": 12345,
"body": null,
"url": null,
"grade": "75",
"score": 75,
"submitted_at": "2020-05-02T11:30:53Z",
"assignment_id": 9876,
"user_id": 1111,
"submission_type": "online_upload",
"workflow_state": "graded",
"grade_matches_current_submission": true,
"graded_at": "2017-06-05T08:47:49Z",
"grader_id": 2222,
"attempt": 1,
"cached_due_date": "2020-05-03T15:00:00Z",
"excused": false,
"late_policy_status": null,
"points_deducted": null,
"grading_period_id": null,
"late": false,
"missing": false,
"seconds_late": 0,
"entered_grade": "75",
"entered_score": 75,
"preview_url": "https://etcetc",
"turnitin_data": {
"attachment_33333": {
"status": "scored",
"object_id": "44444444",
"similarity_score": 0,
"web_overlap": 0,
"publication_overlap": 0,
"student_overlap": 0,
"state": "none"
}
},
"attachments": [
{
"id": 33333,
"uuid": "kjsdkjhsdfkhsfd",
"folder_id": 55555,
"display_name": "Submission.pdf",
"filename": "Submission.pdf",
"content-type": "application/pdf",
"url": "https://etcetc",
"size": 2668226,
"created_at": "2020-05-02T11:30:51Z",
"updated_at": "2020-06-06T15:01:46Z",
"unlock_at": null,
"locked": false,
"hidden": false,
"lock_at": null,
"hidden_for_user": false,
"thumbnail_url": null,
"modified_at": "2020-05-02T11:30:51Z",
"mime_class": "pdf",
"media_entry_id": null,
"locked_for_user": false,
"preview_url": "api/etcetc"
}
],
"submission_comments": [
{
"id": 99999,
"comment": "here’s a comment",
"author_id": 1,
"author_name": "Mickey Mouse",
"created_at": "2020-05-15T12:54:08Z",
"edited_at": null,
"avatar_path": "/images/users/1",
"author": {
"id": 1,
"display_name": " Mickey Mouse ",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
},
{
"id": 223344,
"comment": "another comment",
"author_id": 2,
"author_name": "Donald Duck",
"created_at": "2020-06-05T10:48:51Z",
"edited_at": null,
"avatar_path": "/images/users/2",
"author": {
"id": 2,
"display_name": "Donald Duck",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
}
]
}
I need to be able to retrieve specific values from "submission_comments", namely the values for "comment", "author_id" and "author_name". At the moment the best I can do is retrieve "submission_comments" as one big entity. Here's how I'm getting that far:
require 'typhoeus'
require 'link_header'
require 'json'
require 'csv'
the_url = 'https://etctetc'
token = 'mytoken'
api_endpoint = '/api/etc'
output_csv = 'C:\Users\me\Desktop\Ruby Canvas course\assignment_comments.csv'
CSV.open(output_csv, 'wb') do |csv|
csv << ["user_id", "TII", "marker"]
end
request_url = "#{the_url}#{api_endpoint}"
count = 0
more_data = true
while more_data
get_comments = Typhoeus::Request.new(
request_url,
method: :get,
headers: { authorization: "Bearer #{token}" }
)
get_comments.on_complete do |response|
#get next link
links = LinkHeader.parse(response.headers['link']).links
next_link = links.find { |link| link['rel'] == 'next' }
request_url = next_link.href if next_link
if next_link && "#{response.body}" != "[]"
more_data = true
else
more_data = false
end
if response.code == 200
data = JSON.parse(response.body)
data.each do |comments|
CSV.open(output_csv, 'a') do |csv|
csv << [comments['id'], comments['turnitin_data'], comments['submission_comments']]
end
end
else
puts "Something went wrong! Response code was #{response.code}"
end
end
get_comments.run
end
puts "Script done running"
I'm new to this (the ruby code is based on an exercise so I may not fully understand it)- any help/advice would be really appreciated!
EDIT: I should also note that this isn't the total JSON response I'm dealing with - this is just one of ten items that are returned
"submission_comments": [
{
"id": 99999,
}
]
the [] means it is array. {} means it is an object.
So you probably need to do something like this:
json["submission_comments"].first["id"]
or better iterate through it:
ids = json["submission_comments"].map{|comment| comment["id"]}
I'm able to get the variables you need if you can read the JSON file in as text, then use Ruby's JSON.parse(...) method on it. I think the main problem is that JSON uses null but Ruby hashes use nil. You could do a string replace or try something like this (I did not modify your JSON, only put it into a single quoted string):
json_text = '{
"id": 12345,
"body": null,
"url": null,
"grade": "75",
"score": 75,
"submitted_at": "2020-05-02T11:30:53Z",
"assignment_id": 9876,
"user_id": 1111,
"submission_type": "online_upload",
"workflow_state": "graded",
"grade_matches_current_submission": true,
"graded_at": "2017-06-05T08:47:49Z",
"grader_id": 2222,
"attempt": 1,
"cached_due_date": "2020-05-03T15:00:00Z",
"excused": false,
"late_policy_status": null,
"points_deducted": null,
"grading_period_id": null,
"late": false,
"missing": false,
"seconds_late": 0,
"entered_grade": "75",
"entered_score": 75,
"preview_url": "https://etcetc",
"turnitin_data": {
"attachment_33333": {
"status": "scored",
"object_id": "44444444",
"similarity_score": 0,
"web_overlap": 0,
"publication_overlap": 0,
"student_overlap": 0,
"state": "none"
}
},
"attachments": [
{
"id": 33333,
"uuid": "kjsdkjhsdfkhsfd",
"folder_id": 55555,
"display_name": "Submission.pdf",
"filename": "Submission.pdf",
"content-type": "application/pdf",
"url": "https://etcetc",
"size": 2668226,
"created_at": "2020-05-02T11:30:51Z",
"updated_at": "2020-06-06T15:01:46Z",
"unlock_at": null,
"locked": false,
"hidden": false,
"lock_at": null,
"hidden_for_user": false,
"thumbnail_url": null,
"modified_at": "2020-05-02T11:30:51Z",
"mime_class": "pdf",
"media_entry_id": null,
"locked_for_user": false,
"preview_url": "api/etcetc"
}
],
"submission_comments": [
{
"id": 99999,
"comment": "here’s a comment",
"author_id": 1,
"author_name": "Mickey Mouse",
"created_at": "2020-05-15T12:54:08Z",
"edited_at": null,
"avatar_path": "/images/users/1",
"author": {
"id": 1,
"display_name": " Mickey Mouse ",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
},
{
"id": 223344,
"comment": "another comment",
"author_id": 2,
"author_name": "Donald Duck",
"created_at": "2020-06-05T10:48:51Z",
"edited_at": null,
"avatar_path": "/images/users/2",
"author": {
"id": 2,
"display_name": "Donald Duck",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
}
]
}'
Part I added:
ruby_hash = JSON.parse(json_text)
submission_comments = ruby_hash["submission_comments"]
submission_comments.each do |submission_comment|
comment = submission_comment["comment"]
author_id = submission_comment["author_id"]
author_name = submission_comment["author_name"]
puts "Comment: #{comment}, Author ID: #{author_id}, Author Name: #{author_name}\n\n"
end
Terminal Result:
=> Comment: here’s a comment, Author ID: 1, Author Name: Mickey Mouse
=> Comment: another comment, Author ID: 2, Author Name: Donald Duck
Edit: I added a jenky af one-liner version just for fun (presuming the json_text variable above is already initialized)
JSON.parse(json_text)["submission_comments"]
.map{|txt| puts(["comment","author_id","author_name"]
.map{|k| k.instance_eval{"#{upcase}: #{txt[to_s]}"}}.join(', '))}
COMMENT: here’s a comment, AUTHOR_ID: 1, AUTHOR_NAME: Mickey Mouse
COMMENT: another comment, AUTHOR_ID: 2, AUTHOR_NAME: Donald Duck

Mapping Seq to JSON in Play 2

I am working with play 2 with scala. I have Seq named customers like
customers: scala.Seq[(Option[Int], String, String)]
How can I convert it to a json object?
I am using this code
Json.obj(
"status" -> true,
"message" -> Json.toJson("Successful"),
"customers" -> Json.toJson(customers.toString())
)
This give output like
{
"status": true,
"message": "Successful",
"customers": "Vector((Some(1),ACTIVE,Md. Khairul Anam (Some(2),ACTIVE,Johirul Islam))"
}
I want to see output like
{
"status": true,
"message": "Successful",
"customers": [
{
"id": 1,
"name": "Md. Khairul Anam "
"status": "ACTIVE"
},
{
"id": 2,
"name": "Johirul Islam"
"status": "ACTIVE"
}
]
}

Remove proxies from Json Response Symfony2

I'm working in Symfony2 application and what I'm trying to do is to remove unwanted fields from response and show only fields that I want.
My JSON looks like this:
[
{
"id": 1,
"title": "Granit",
"typeId": {
"id": 1,
"name": "X or Y",
"acroname": "xory",
"__initializer__": null,
"__cloner__": null,
"__isInitialized__": true
},
"pushDate": {
"timezone": {
"name": "Europe/Berlin",
"location": {
"country_code": "DE",
"latitude": 52.5,
"longitude": 13.36666,
"comments": "most locations"
}
},
"offset": 7200,
"timestamp": 1460584800
},
"addedAt": {
"timezone": {
"name": "Europe/Berlin",
"location": {
"country_code": "DE",
"latitude": 52.5,
"longitude": 13.36666,
"comments": "most locations"
}
},
"offset": 7200,
"timestamp": 1460548644
},
"deviceToShow": {
"id": 2,
"name": "Mobile",
"__initializer__": null,
"__cloner__": null,
"__isInitialized__": true
},
"statusSurvey": false,
"slides": [
{
"id": 1,
"title": "First Question",
"picture1": "160413015724bazinga2.jpg",
"picture2": "160413015724th.jpg",
"idSurvey": 1,
"absolutePathpic1": "C:\\xampp\\htdocs\\stu-wrapper\\src\\AppBundle\\Entity/../../../web/uploads/slideSurvey/160413015724bazinga2.jpg",
"webPathpic1": "uploads/slideSurvey/160413015724bazinga2.jpg",
"absolutePathpic2": "C:\\xampp\\htdocs\\stu-wrapper\\src\\AppBundle\\Entity/../../../web/uploads/slideSurvey/160413015724th.jpg",
"webPathpic2": "uploads/slideSurvey/160413015724th.jpg",
"file": null,
"file1": null
}
],
"categories": []
}
]
I want to remove fields like "initializer": null,"cloner": null, "isInitialized": true and hide timezone object and show only "timestamp".
Here is my controller where I'm doing serializing and creating Json Response.
public function getAction()
{
$em = $this->getDoctrine ()->getManager ();
$survey = $em->getRepository ( 'AppBundle:Survey' )->findAll ();
if ( !$survey ) {
throw $this->createNotFoundException ( 'Data not found.' );
}
$encoder = new JsonEncoder();
$normalizer = new ObjectNormalizer();
$normalizer->setCircularReferenceHandler ( function ( $survey ) {
return $survey->getid ();
} );
$serializer = new Serializer( array ( $normalizer ), array ( $encoder ) );
$jsonContent = $serializer->serialize ( $survey, 'json' );
return new Response( $jsonContent );
}
Thank you.
Try to exclude this fields:
$normalilzer->setIgnoredAttributes(
[
"__initializer__",
"__cloner__",
"__isInitialized__"
]);
Detach entity to remove doctrine2 links to the object
$em->detach($survey);

Search JSON for multiple values, not using a library

I'd like to be able to search the following JSON object for objects containing the key 'location' then get in return an array or json object with the 'name' of the person plus the value of location for that person.
Sample return:
var matchesFound = [{Tom Brady, New York}, {Donald Steven,Los Angeles}];
var fbData0 = {
"data": [
{
"id": "X999_Y999",
"location": "New York",
"from": {
"name": "Tom Brady", "id": "X12"
},
"message": "Looking forward to 2010!",
"actions": [
{
"name": "Comment",
"link": "http://www.facebook.com/X999/posts/Y999"
},
{
"name": "Like",
"link": "http://www.facebook.com/X999/posts/Y999"
}
],
"type": "status",
"created_time": "2010-08-02T21:27:44+0000",
"updated_time": "2010-08-02T21:27:44+0000"
},
{
"id": "X998_Y998",
"location": "Los Angeles",
"from": {
"name": "Donald Steven", "id": "X18"
},
"message": "Where's my contract?",
"actions": [
{
"name": "Comment",
"link": "http://www.facebook.com/X998/posts/Y998"
},
{
"name": "Like",
"link": "http://www.facebook.com/X998/posts/Y998"
}
],
"type": "status",
"created_time": "2010-08-02T21:27:44+0000",
"updated_time": "2010-08-02T21:27:44+0000"
}
]
};
#vsiege - you can use this javascript lib (http://www.defiantjs.com/) to search your JSON structure.
var fbData0 = {
...
},
res = JSON.search( fbData0, '//*[./location and ./from/name]' ),
str = '';
for (var i=0; i<res.length; i++) {
str += res[i].location +': '+ res[i].from.name +'<br/>';
}
document.getElementById('output').innerHTML = str;
Here is a working fiddle;
http://jsfiddle.net/hbi99/XhRLP/
DefiantJS extends the global object JSON with the method "search" and makes it possible to query JSON with XPath expressions (XPath is standardised query language). The method returns an array with the matches (empty array if none were found).
You can test XPath expressions by pasting your JSON here:
http://www.defiantjs.com/#xpath_evaluator