I follow Object Detection Demo in https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb:
# Actual detection.
output_dict = run_inference_for_single_image(image_np, detection_graph)
But I want to convert output_dict (output from function run_inference_for_single_image(image_np, detection_graph)) to COCO annotation JSON type so I can input it to make benchmark between different Object Detection models.
Here is code to benchmark model: https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb
#initialize COCO detections api
resFile='%s/results/%s_%s_fake%s100_results.json'
resFile = resFile%(dataDir, prefix, dataType, annType)
cocoDt=cocoGt.loadRes(resFile)
But you need to input a COCO Json type.
Are there anyone can tell me how to Convert from output_dict to COCO Json?
This is what you are looking for. The last code snippet:
https://lijiancheng0614.github.io/2017/08/22/2017_08_22_TensorFlow-Object-Detection-API/
Related
I am using Kotlin and I have a service that is getting an object of type Any (this cannot be changed)
The problem with Any is that is an object of 20+ fields and I just need one of them to use it as a filter... therefore I cannot do a simple cast.
So, my object is like: (when I print it)
{messageId=123, userId=32323, address=Some city, phone=111605,type=TYPE1.....
I want to convert it using Kotlinx or Jackson but I cannot convert it first to the expected String format, doing a parseFromString(myObject) will result in an exception as well of a wrong Json format.
I want to convert it to a class like this
#Serializable
private data class UserType(val type: String)
type is the only field I care about.
My convertion is via kotlinx
val format = Json { ignoreUnknownKeys = true }
format.decodeFromString<UserType>(myObject)
I even tried this to see if I can make it in the proper Json format
format.encodeToString(original)
Any idea what I could do here that would be a lightweight solution?
This is my Any type https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-any/
After training an xgboost model, I would like to save it and some other custom fields as a json object as below. The purpose being so I can load the json object later, use the model object to make predictions as well as inspecting the other custom fields.
model = xgb.train(params=tree_params, dtrain=data)
my_model_dict = {
"model": ...<json serializable model object>..., # need help here
"features": model.feature_names,
"tree_params": tree_params,
...
}
json.dumps(my_model_dict, file_path)
my_model_dict = json.load(file_path)
model = my_model_dict["model"]
predictions = model.predict(new_data)
Is it possible to convert an xgboost model object into an object that is json serializable and that can then be loaded to make standard xgboost predictions?
I appreciate I can save the raw model seperately as a json using
model.save_model("my_model.json")
model = xgb.Booster("my_model.json")
model.predict(new_data)
but really what I would like to do is create a dictionary containing the model along with other custom fields that can be saved as a json, then loaded to make predictions.
I have seen the terms "deserialize" and "serialize" with JSON. What do they mean?
JSON is a format that encodes objects in a string. Serialization means to convert an object into that string, and deserialization is its inverse operation (convert string -> object).
When transmitting data or storing them in a file, the data are required to be byte strings, but complex objects are seldom in this format. Serialization can convert these complex objects into byte strings for such use. After the byte strings are transmitted, the receiver will have to recover the original object from the byte string. This is known as deserialization.
Say, you have an object:
{foo: [1, 4, 7, 10], bar: "baz"}
serializing into JSON will convert it into a string:
'{"foo":[1,4,7,10],"bar":"baz"}'
which can be stored or sent through wire to anywhere. The receiver can then deserialize this string to get back the original object. {foo: [1, 4, 7, 10], bar: "baz"}.
Serialize and Deserialize
In the context of data storage, serialization (or serialisation) is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer) or transmitted (for example, across a network connection link) and reconstructed later. [...]
The opposite operation, extracting a data structure from a series of bytes, is deserialization.
– wikipedia.org
JSON
JSON (JavaScript Object Notation) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other serializable values). It is a common data format with diverse uses in electronic data interchange, including that of web applications with servers.
JSON is a language-independent data format. It was derived from JavaScript, but many modern programming languages include code to generate and parse JSON-format data. JSON filenames use the extension .json.
– wikipedia.org
Explained using Python
In Python serialization does nothing else than just converting the given data structure into its valid JSON pendant (e.g., Python's True will be converted to JSON's true and the dictionary itself will be converted to a string) and vice versa for deserialization.
Python vs. JSON
You can easily spot the difference between Python and JSON representations in a side-by-side comparison. For example, by examining their Boolean values. Have a look at the following table for the basic types used in both contexts:
Python
JSON
True
true
False
false
None
null
int, float
number
str (with single ', double " and tripple """ quotes)
string (only double " quotes)
dict
object
list, tuple
array
Code Example
Python builtin module json is the standard way to do serialization and deserialization:
import json
data = {
'president': {
"name": """Mr. Presidente""",
"male": True,
'age': 60,
'wife': None,
'cars': ('BMW', "Audi")
}
}
# serialize
json_data = json.dumps(data, indent=2)
print(json_data)
# {
# "president": {
# "name": "Mr. Presidente",
# "male": true,
# "age": 60,
# "wife": null,
# "cars": [
# "BMW",
# "Audi"
# ]
# }
# }
# deserialize
restored_data = json.loads(json_data) # deserialize
Sources: realpython.com, geeksforgeeks.org
Explanation of Serialize and Deserialize using Python
In python, pickle module is used for serialization. So, the serialization process is called pickling in Python. This module is available in Python standard library.
Serialization using pickle
import pickle
#the object to serialize
example_dic={1:"6",2:"2",3:"f"}
#where the bytes after serializing end up at, wb stands for write byte
pickle_out=open("dict.pickle","wb")
#Time to dump
pickle.dump(example_dic,pickle_out)
#whatever you open, you must close
pickle_out.close()
The PICKLE file (can be opened by a text editor like notepad) contains this (serialized data):
€}q (KX 6qKX 2qKX fqu.
Deserialization using pickle
import pickle
pickle_in=open("dict.pickle","rb")
get_deserialized_data_back=pickle.load(pickle_in)
print(get_deserialized_data_back)
Output:
{1: '6', 2: '2', 3: 'f'}
Share what I learned about this topic.
What is Serialization
Serialization is the process of converting a data object into a byte stream.
What is byte stream
Byte stream is just a stream of binary data. Because only binary data can be stored or transported.
What is byte string vs byte stream
Sometime you see people use the word byte string as well. String encodings of bytes are called byte strings. Then it can explain what is JSON as below.
What’s the relationship between JSON and serialization
JSON is a string format representational of byte data. JSON is encoded in UTF-8. So while we see human readable strings, behind the scenes strings are encoded as bytes in UTF-8.
I am using Play Framework and I am trying to convert a Scala object to a JSON string.
Here is my code where I get my object:
val profile: Future[List[Profile]] = profiledao.getprofile(profileId);
The object is now in the profile value.
Now I want to convert that profile object which is a Future[List[Profile]] to JSON data and then convert that data into a JSON string then write into a file.
Here is the code that I wrote so far:
val jsondata = Json.toJson(profile)
Jackson.toJsonString(jsondata)
This is how I am trying to convert into JSON data but it is giving me the following output:
{"empty":false,"traversableAgain":true}
I am using the Jackson library to do the conversion.
Can someone help me with this ?
Why bother with Jackson? If you're using Play, you have play-json available to you, which uses Jackson under the hood FWIW:
First, you need an implicit Reads to let play-json know how to serialize Profile. If Profile is a case class, you can do this:
import play.api.libs.json._
implicit val profileFormat = Json.format[Profile]
If not, define your own Reads like this.
Then since getprofile (which should follow convention and be getProfile) returns Future[List[Profile]], you can do this to get a JsValue:
val profilesJson = profiledao.getprofile(profileId).map(toJson)
(profiledao should also be profileDao.)
In the end, you can wrap this in a Result like Ok and return that from your controller.
What is the fastest way to convert this
{"a":"ab","b":"cd","c":"cd","d":"de","e":"ef","f":"fg"}
into mutable map in scala ? I read this input string from ~500MB file. That is the reason I'm concerned about speed.
If your JSON is as simple as in your example, i.e. a sequence of key/value pairs, where each value is a string. You can do in plain Scala :
myString.substring(1, myString.length - 1)
.split(",")
.map(_.split(":"))
.map { case Array(k, v) => (k.substring(1, k.length-1), v.substring(1, v.length-1))}
.toMap
That looks like a JSON file, as Andrey says. You should consider this answer. It gives some example Scala code. Also, this answer gives some different JSON libraries and their relative merits.
The fastest way to read tree data structures in XML or JSON is by applying streaming API: Jackson Streaming API To Read And Write JSON.
Streaming would split your input into tokens like 'beginning of an object' or 'beginning of an array' and you would need to build a parser for these token, which in some cases is not a trivial task.
Keeping it simple. If reading a json string from file and converting to scala map
import spray.json._
import DefaultJsonProtocol._
val jsonStr = Source.fromFile(jsonFilePath).mkString
val jsonDoc=jsonStr.parseJson
val map_doc=jsonDoc.convertTo[Map[String, JsValue]]
// Get a Map key value
val key_value=map_doc.get("key").get.convertTo[String]
// If nested json, re-map it.
val key_map=map_doc.get("nested_key").get.convertTo[Map[String, JsValue]]
println("Nested Value " + key_map.get("key").get)