I need my events to be stored as json in cassandra (So I can read them with some gui client directly from db).
I've followed lagom's guide https://www.lagomframework.com/documentation/1.4.x/scala/Serialization.html (Enabling JSON Serialization), but events are still stored in something like binary or other format.
Here is what I've done:
Created Serializer Registry
object ProjectSerializerRegistry extends JsonSerializerRegistry {
override def serializers: Seq[JsonSerializer[_]] = Seq(
JsonSerializer[ProjectCreated],
)
}
Registered it:
abstract class ProjectsApplication(context: LagomApplicationContext)
extends LagomApplication(context)
with CassandraPersistenceComponents
with LagomKafkaComponents
with AhcWSComponents {
...
// Register the JSON serializer registry
override lazy val jsonSerializerRegistry = ProjectSerializerRegistry
}
Here is the event itself:
case class ProjectCreated(id: String, name: String, createdAt: DateTime) extends ProjectEvent
object ProjectCreated {
implicit val format: OFormat[ProjectCreated] = Json.format[ProjectCreated]
}
After sending command to entity which causes ProjectCreated event and executing query select event from projects.messages I expected to see something like this in cassandra:
{
"id": "prj-1",
"name": "Project 1",
"createdAt": "2018-05-04 01:16:00"
}
But instead, I see something like this in event column:
0x7b226d657373616765223a224869227d
Did I miss something? Or may be it is some compressed or encoded json value?
This is in hexadecimal format. You can convert it into string using online hex-to-string decoder like https://codebeautify.org/hex-string-converter or can create one on your own.
This is correct. The events are stored in binary blobs. if you want to decode these values, you can use the cassandra function blobAsText:
select blobastext(event) from table.messages
This shows the decoded event. If it results in a json object, your serialization works correctly.
There is also a textAsBlob function.
Related
hope I can get some valued assistance with a little issue I'm trying to work.
I've got an API endpoint which is sending data with the following request schema:
{
"type": String,
"coordinates": [
0.949492,
48.77163
]
}
As can be seen; the coordinates from the search are provided as two INT values, without parameters.
I'm trying to create an automated test for this, and I've put the above in a data class so it can be used all over the suite as-needed.
My data class is currently looking like the below example, but I don't know how to properly define a list for coordinates without a val or var parameter. I've defined it as a var called "list" for now so it stops throwing compilation errors. How should I be representing this list of coordinates?
data class SearchRequest(
val type: String,
val coordinates: List<Coordinates>
)
data class Coordinates(
var list: Int
)
The second parameter is a list of Float values, there is no need to create a separate class for that, Float can be used:
data class SearchRequest(
val type: String,
val coordinates: List<Float>
)
In addition to the answers above, you can also download a plugin from android studio that does this for you by just pasting the API's JSON format. The plugin name is JSON to Kotlin class converter, I think.👍🏻
I'm modelling some JSON - and using the following lines
data class Metadata(
val id: String,
val creators: Array<CreatorsModel>
)
along with:
data class CreatorsModel (
val role: String,
val name: String
)
However keep seeing the error: Array property in data class error.
Any ideas why this is?
FYI, the JSON looks like:
{
"id": "123",
"creators": [{
"role": "Author",
"name": "Marie"
}
]
}
In Kotlin you should aim to use List instead of Array where possible. Array has some JVM implications, and although the compiler will let you, the IDE may prompt you to override equals and hashcode manually. Using List will make things much simpler.
You can find out more about the difference here: Difference between List and Array types in Kotlin
During my fresh adventures with kotlin-react I hit a hard stop when trying to parse some data from my backend which contains enum values.
Spring-Boot sends the object in JSON form like this:
{
"id": 1,
"username": "Johnny",
"role": "CLIENT"
}
role in this case is the enum value and can have the two values CLIENT and LECTURER. If I were to parse this with a java library or let this be handled by Spring-Boot, role would be parsed to the corresponding enum value.
With kotlin-js' JSON.parse, that wouldn't work and I would have a simple string value in there.
After some testing, I came up with this snippet
val json = """{
"id": 1,
"username": "Johnny",
"role": "CLIENT",
}"""
val member: Member = JSON.parse(json) { key: String, value: Any? ->
if (key == "role") Member.Role.valueOf(value.toString())
else value
}
in which I manually have to define the conversion from the string value to the enum.
Is there something I am missing that would simplify this behaviour?
(I am not referring to using ids for the JSON and the looking those up, etc. I am curious about some method in Kotlin-JS)
I have the assumption there is not because the "original" JSON.parse in JS doesn't do this and Kotlin does not add any additional stuff in there but I still have hope!
As far as I know, no.
The problem
Kotlin.JS produces an incredibly weird type situation when deserializing using the embedded JSON class, which actually is a mirror for JavaScript's JSON class. While I haven't done much JavaScript, its type handling is near non-existent. Only manual throws can enforce it, so JSON.parse doesn't care if it returns a SomeCustomObject or a newly created object with the exact same fields.
As an example of that, if you have two different classes with the same field names (no inheritance), and have a function that accepts a variable, it doesn't care which of those (or a third for that matter) it receives as long as the variables it tries accessing on the class exists.
The type issues manifest themselves into Kotlin. Now wrapping it back to Kotlin, consider this code:
val json = """{
"x": 1, "y": "yes", "z": {
"x": 42, "y": 314159, "z": 444
}
}""".trimIndent()
data class SomeClass(val x: Int, val y: String, val z: Struct)
data class Struct(val x: Int, val y: Int, val z: Int)
fun main(args: Array<String>) {
val someInstance = JSON.parse<SomeClass>(json)
if(someInstance.z::class != Struct::class) {
println("Incompatible types: Required ${Struct::class}, found ${someInstance.z::class}");
}
}
What would you expect this to print? The natural would be to expect a Struct. The type is also explicitly declared
Unfortunately, that is not the case. Instead, it prints:
Incompatible types: Required class Struct, found class Any
The point
The embedded JSON de/serializer isn't good with types. You might be able to fix this by using a different serializing library, but I'll avoid turning this into a "use [this] library".
Essentially, JSON.parse fails to parse objects as expected. If you entirely remove the arguments and try a raw JSON.parse(json); on the JSON in your question, you'll get a role that is a String and not a Role, which you might expect. And with JSON.parse doing no type conversion what so ever, that means you have two options: using a library, or using your approach.
Your approach will unfortunately get complicated if you have nested objects, but with the types being changed, the only option you appear to have left is explicitly parsing the objects manually.
TL;DR: your approach is fine.
I store a blob of Json in the datastore using JsonProperty.
I don't know the structure of the json data.
I am using endpoints proto datastore in order to retrieve my data.
The probleme is the json property is encoded in base64 and I want a plain json object.
For the example, the json data will be:
{
first: 1,
second: 2
}
My code looks something like:
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Model(EndpointsModel):
data = ndb.JsonProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class DataEndpoint(remote.Service):
#Model.method(path='mymodel2', http_method='POST',
name='mymodel.insert')
def MyModelInsert(self, my_model):
my_model.data = {"first": 1, "second": 2}
my_model.put()
return my_model
#Model.method(path='mymodel/{entityKey}',
http_method='GET',
name='mymodel.get')
def getMyModel(self, model):
print(model.data)
return model
API = endpoints.api_server([DataEndpoint])
When I call the api for getting a model, I get:
POST /_ah/api/myapi/v1/mymodel2
{
"data": "eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ=="
}
where eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ== is the base64 encoded of {"second": 2, "first": 1}
And the print statement give me: {u'second': 2, u'first': 1}
So, in the method, I can explore the json blob data as a python dict.
But, in the api call, the data is encoded in base64.
I expeted the api call to give me:
{
'data': {
'second': 2,
'first': 1
}
}
How can I get this result?
After the discussion in the comments of your question, let me share with you a sample code that you can use in order to store a JSON object in Datastore (it will be stored as a string), and later retrieve it in such a way that:
It will show as plain JSON after the API call.
You will be able to parse it again to a Python dict using eval.
I hope I understood correctly your issue, and this helps you with it.
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class MyApi(remote.Service):
# URL: .../_ah/api/myapi/v1/mymodel - POSTS A NEW ENTITY
#Sample.method(path='mymodel', http_method='GET', name='Sample.insert')
def MyModelInsert(self, my_model):
dict={'first':1, 'second':2}
dict_str=str(dict)
my_model.column1="Year"
my_model.column2=2018
my_model.column3=dict_str
my_model.put()
return my_model
# URL: .../_ah/api/myapi/v1/mymodel/{ID} - RETRIEVES AN ENTITY BY ITS ID
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
dict=eval(my_model.column3)
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
application = endpoints.api_server([MyApi], restricted=False)
I have tested this code using the development server, but it should work the same in production using App Engine with Endpoints and Datastore.
After querying the first endpoint, it will create a new Entity which you will be able to find in Datastore, and which contains a property column3 with your JSON data in string format:
Then, if you use the ID of that entity to retrieve it, in your browser it will show the string without any strange encoding, just plain JSON:
And in the console, you will be able to see that this string can be converted to a Python dict (or also a JSON, using the json module if you prefer):
I hope I have not missed any point of what you want to achieve, but I think all the most important points are covered with this code: a property being a JSON object, store it in Datastore, retrieve it in a readable format, and being able to use it again as JSON/dict.
Update:
I think you should have a look at the list of available Property Types yourself, in order to find which one fits your requirements better. However, as an additional note, I have done a quick test working with a StructuredProperty (a property inside another property), by adding these modifications to the code:
#Define the nested model (your JSON object)
class Structured(EndpointsModel):
first = ndb.IntegerProperty()
second = ndb.IntegerProperty()
#Here I added a new property for simplicity; remember, StackOverflow does not write code for you :)
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
column4 = ndb.StructuredProperty(Structured)
#Modify this endpoint definition to add a new property
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
#Add the new nested property here
dict=eval(my_model.column3)
my_model.column4=dict
print(json.dumps(my_model.column3))
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
With these changes, the response of the call to the endpoint looks like:
Now column4 is a JSON object itself (although it is not printed in-line, I do not think that should be a problem.
I hope this helps too. If this is not the exact behavior you want, maybe should play around with the Property Types available, but I do not think there is one type to which you can print a Python dict (or JSON object) without previously converting it to a String.
I want to convert a scala list of strings, List[String], to an Json object.
For each string in my list I want to add it to my Json object.
So that it would look like something like this:
{
"names":[
{
"Bob",
"Andrea",
"Mike",
"Lisa"
}
]
}
How do I create an json object looking like this, from my list of strings?
To directly answer your question, a very simplistic and hacky way to do it:
val start = """"{"names":[{"""
val end = """}]}"""
val json = mylist.mkString(start, ",", end)
However, what you almost certainly want to do is pick one of the many JSON libraries out there: play-json gets some good comments, as does lift-json. At the worst, you could just grab a simple JSON library for Java and use that.
Since I'm familiar with lift-json, I'll show you how to do it with that library.
import net.liftweb.json.JsonDSL._
import net.liftweb.json.JsonAST._
import net.liftweb.json.Printer._
import net.liftweb.json.JObject
val json: JObject = "names" -> List("Bob", "Andrea", "Mike", "Lisa")
println(json)
println(pretty(render(json)))
The names -> List(...) expression is implicitly converted by the JsonDSL, since I specified that I wanted it to result in a JObject, so now json is the in-memory model of the json data you wanted.
pretty comes from the Printer object, and render comes from the JsonAST object. Combined, they create a String representation of your data, which looks like
{
"names":["Bob","Andrea","Mike","Lisa"]
}
Be sure to check out the lift documentation, where you'll likely find answers to any further questions about lift's json support.