I have a tuple consisting of an String and a Uuid that I serialize using serde_json:
let log_and_id = (String::from("Test string"), test_id);
let log_and_id_serialized = serde_json::to_string(&log_and_id)
.expect("Serialization failed");
//After serialization (debug print): "[\"Test string\",\"32a8e12d-69d2-421d-a52e-1ee76cc03ed5\"]"
Then I transfer this serialized value over the network and receive a BytesMut (serialized_tuple) on the other end, which I try to deserialize:
//Bytesmut value (debug print): b"\"[\\\"Test string\\\",\\\"32a8e12d-69d2-421d-a52e-1ee76cc03ed5\\\"]\""
let (log, operation_state_id) = serde_json::from_slice::<(String, Uuid)>(&serialized_tuple)?;
But I get the following error:
ERROR actix_http::response] Internal Server Error: SerdeError(Error("invalid type: string \"[\\\"Test string\\\",\\\"32a8e12d-69d2-421d-a52e-1ee76cc03ed5\\\"]\", expected a tuple of size 2", line: 1, column: 68))
(De)serializing single objects this way used to work in other parts of this code, so what could cause it to fail when used with tuples?
You don't have a serialized tuple, but a serialized serialized tuple.
I mean the serialization of the tuple, which was a JSON string, was again serialized.
You can check this with this code (playground):
let serialized_tuple = b"\"[\\\"Test string\\\",\\\"32a8e12d-69d2-421d-a52e-1ee76cc03ed5\\\"]\"";
let serialized_tuple: String = serde_json::from_slice(serialized_tuple).unwrap();
let (log, operation_state_id) = serde_json::from_slice::<(String, String)>(serialized_tuple.as_bytes()).unwrap();
which produces the desired tuple.
Of course, rather than deserializing twice, you should remove the unnecessary serialization from your application (it's not in the code you've shown).
Related
I'm carrying this on from this question, since the focus has changed.
I am trying to send string data from a vapor server over a websocket. The client side is where the main question is. This code successfully receives the string, which is expected to be JSON (but not absolutely guaranteed -- out of scope).
switch message {
case .data(let data):
print("data: \(data)")
case .string(let str):
// let data = str.message(using: .utf8)
let jsonData = Data(str.utf8)
print("string: \(jsonData)")
do {
struct Person : Codable {
var name: String
}
let decoder = JSONDecoder()
let people = try decoder.decode([Person].self, from: jsonData)
print("result: \(people)")
} catch {
print(error.localizedDescription)
}
}
After some very helpful guidance, sending a string such as "{\"name\": \"Bobberoo\"}" will print out
string: 20 bytes
The data couldn’t be read because it isn’t in the correct format.
If I wrap it in braces "[{\"name\": \"Bobberoo\"}]" produces the more helpful but still mystifing (to me) output:
result: [wb2_socket_client.WebSocketController.(unknown context at $101a35028).(unknown context at $101a350c0).(unknown context at $101a35158).Person(name: "Bobberoo")]
Clearly, the decoding is happening, but it's wrapped in these contexts. What are they? I can see that the first is the instance of the WebSocketController. How do I access this data.
And as a non-inflammatory aside: managing JSON is a trivial operation in any number of contexts. Python/Flask, Node, Ruby/Rails and on and on; I've used all these and implementing this kind of interaction is trivial. In Swift, it's a horrible, underdocumented nightmare. At least, that's my experience. Why? I know the language is type safe, but this is ridiculous.
error.localizedDescription won't give you an error message that is useful message for debugging. On the other hand, if you print error directly:
print(error)
You'd get something along the lines of "expected to decode array but found dictionary instead", which is exactly what is happening in the case of
{
"name": "Bobberoo"
}
You are decoding a [Person].self, i.e. an array of Person, but your JSON root is not a JSON array. The above JSON can be decoded if you did:
let people = try decoder.decode(Person.self, from: jsonData)
Clearly, the decoding is happening, but it's wrapped in these contexts. What are they?
This is the default string representation of a type. Your Person struct does not conform to CustomStringConvertible or CustomDebugStringConvertible or TextOutputStreamable, so "an unspecified result is supplied automatically by the Swift standard library" (the link points to String.init(reflecting:), which presumably gets called somewhere along the way when you print the array of Person) and used as the string representation.
From what I can see, its current implementation is the fully qualified name of the struct - starting with the module, then the top-level class, then each enclosing scope, ending with the struct name, followed by the struct's members in brackets. It turns out that the enclosing scopes has no "names", and so are just called (unknown context at xxxxx). This is all very much implementation details, and things that you shouldn't care about.
What you should do, is provide an implementation of CustomStringConvertible:
struct Person: CustomStringConvertible {
...
var description: String { "name: \(name)" }
}
Now printing people gives:
[name: Bobberoo]
I can see that the first is the instance of the WebSocketController.
No. The WebSocketController is part of the fully qualified name of your Person struct. There is exactly one instance in your decoded array, and it's an instance of Person, as you would expect!
How do I access this data?
To access its name:
if let firstPerson = people.first {
let firstPersonsName = firstPerson.name
}
I'm using Flink to process the data coming from some data source (such as Kafka, Pravega etc).
In my case, the data source is Pravega, which provided me a flink connector.
My data source is sending me some JSON data as below:
{"key": "value"}
{"key": "value2"}
{"key": "value3"}
...
...
Here is my piece of code:
PravegaDeserializationSchema<ObjectNode> adapter = new PravegaDeserializationSchema<>(ObjectNode.class, new JavaSerializer<>());
FlinkPravegaReader<ObjectNode> source = FlinkPravegaReader.<ObjectNode>builder()
.withPravegaConfig(pravegaConfig)
.forStream(stream)
.withDeserializationSchema(adapter)
.build();
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<ObjectNode> dataStream = env.addSource(source).name("Pravega Stream");
dataStream.map(new MapFunction<ObjectNode, String>() {
#Override
public String map(ObjectNode node) throws Exception {
return node.toString();
}
})
.keyBy("word") // ERROR
.timeWindow(Time.seconds(10))
.sum("count");
As you see, I used the FlinkPravegaReader and a proper deserializer to get the JSON stream coming from Pravega.
Then I try to transform the JSON data into a String, KeyBy them and count them.
However, I get an error:
The program finished with the following exception:
Field expression must be equal to '*' or '_' for non-composite types.
org.apache.flink.api.common.operators.Keys$ExpressionKeys.<init>(Keys.java:342)
org.apache.flink.streaming.api.datastream.DataStream.keyBy(DataStream.java:340)
myflink.StreamingJob.main(StreamingJob.java:114)
It seems that KeyBy threw this exception.
Well, I'm not a Flink expert so I don't know why. I've read the source code of the official example WordCount. In that example, there is a custtom splitter, which is used to split the String data into words.
So I'm thinking if I need to use some kind of splitter in this case too? If so, what kind of splitter should I use? Can you show me an example? If not, why did I get such an error and how to solve it?
I guess you have read the document about how to specify keys
Specify keys
The example codes use keyby("word") because word is a field of POJO type WC.
// some ordinary POJO (Plain old Java Object)
public class WC {
public String word;
public int count;
}
DataStream<WC> words = // [...]
DataStream<WC> wordCounts = words.keyBy("word").window(/*window specification*/);
In your case, you put a map operator before keyBy, and the output of this map operator is a string. So there is obviously no word field in your case. If you actually want to group this string stream, you need to write it like this .keyBy(String::toString)
Or you can even implement a customized keySelector to generate your own key.
Customized Key Selector
Is there a way to marshall a struct to JSON that skips any fields that can not be serialised?
E.G. if I marshal
type aStruct struct {
request *http.Request
name string
}
the JSON representation of
type aStruct struct {
name string
}
would result?
By 'fields that can not be serialised' I mean any field that would cause json.Marshal(..) to return an 'error serializing json: unsupported type' error. Having a look through the json package now with a mind to create another Marshal function that will skip a field it fails to serialise rather than abort and return the error.
==== UPDATE ====
I have created this https://github.com/myles-mcdonnell/jsonx which satisfies my use case. At first I thought this is a terrible way to extend the base json package (it's a copy from 1.6.2 with new behaviour added) but the consensus among my colleagues is that this is the idiomatic way to do this.
I am trying to get CouchDB-like response from Redis - by using ServiceStack WebServices to access data stored via ServiceStack .Net RedisTypedClient onto Redis.
Now the web services are described as providing a CouchDB layer onto Redis. However, unlike equivalent CouchDB calls to a CouchDB, I never get back json but only strings - to be clear I do get back json but with the payload is in string format.
This applies to getting items from Redis List, Set and HashSet collections. All the items via either the xml,json or csv web services always deliver the payload as a string. Now I can see the serialized form of the type I stored in the string - such as a json array of strings or whatever, but the data is not delivered as json (or csv or xml) but as a string. I cannot find a query flag (.e.g. 'format=json' say) in any of the autogenerated documentation for these web services -which does say it delivers the payload as a string which is what I see.
Further apart from using the default jsv serializer through RedisTypedCLient I also tried directly calling the ServiceStack json serializer to serialize as json not jsv and also the Newtonsoft json serializer. None of this makes any difference. I did not expect it too as I imagine the default services would likely only manage the expected jsv version and would deliver anything else as strings. However I did not expect this of the internal serialization format.
So is it possible to get CouchDB like json responses from ServiceStack/Redis/Builtin-WebServices?
Update
Here is a typical query via the ServiceStack json web service:
http://myserver.com/redis/json/syncreply/GetAllItemsFromList?id=test
This is a Redis List collection containing strongly typed items:
type TestItem( gr,eType,pillar,offset) =
let mutable _gr = gr
let mutable _eType = eType
let mutable _pillar = pillar
let mutable _offset = offset
member x.Gr with get()= _gr and set(v) = _gr <- v
member x.EType with get()= _eType and set(v) = _eType <- v
member x.Pillar with get()= _pillar and set(v) = _pillar <- v
member x.Offset with get()= _offset and set(v) = _offset <- v
override x.ToString() = sprintf "%s %s %s %M" x.Gr x.EType x.Pillar x.Offset
The list collection was added using IRedisTypedClient Interface/API and I am expecting back a json list of json objects - a set of key/value pairs each pair corresponding to one of the four public properties in the type above. Instead I get
{"Items":[" {\"Gr\":\"BON13\",\"EType\":\"MARKET\",\"Pillar\":\"BON3.R0\",\"Offset\":0.0}","{\"Gr\":\"BOQ13\",\"EType\":\"MARKET\",\"Pillar\":\"BOQ3.R0\",\"Offset\":0.0}","{\"Gr\":\"BOU13\",\"EType\":\"MARKET\",\"Pillar\":\"BOU3.R0\",\"Offset\":0.0}","{\"Gr\":\"BOV13\",\"EType\":\"SETTLEPILLAR\",\"Pillar\":\"BOU3.R0\",\"Offset\":0.0}","{\"Gr\":\"BOZ16\",\"EType\":\"SETTLEPILLAR\",\"Pillar\":\"BOU3.R0\",\"Offset\":0.0}"],"ResponseStatus":{}}
In other words a string representation of the json object not the json object itself.
So again, how can I get back, in this case, a json list of of json objects rather than a json list of strings. (And the same goes for Sets, Dictionaries and more basic keyed json documents a la other NoSql dbs)?
I have the same issues getting back csv - it comes back as a string rather than either a csv of key/value pairs or a csv of keys and values and in XML where this,again, comes back as a string not not an XML format of key/value pairs.
Update 1
It does not need to be a strongly typed as above. It could be a list of a list of strings. In which case I get back a json list of strings rather than a json list containing items comprising json list of string.
Update 2
Whilst the problem clearly seems to be in the ServiceStack webservice implementation not being like CouchDB although it claims it is, here is some sample code to put the data into Redis via ServiceStack.
open System
open System.Collections.Generic
open ServiceStack.Redis
open System.Linq
type Repository() =
static let mutable __port = 6379
static let mutable __host = "myserver.com"
static let mutable __client = new RedisClient(__host,__port)
static member Client = __client :> IRedisClient
type Repository<'T>() =
let _client = Repository.Client
member x.GetList key =
use client = _client.As<'T>()
match _client.GetEntryType key with
| RedisKeyType.List ->
client.Lists.Item key |> client.GetAllItemsFromList
| _ -> new List<'T>()
member x.SetList (key, values: List<'T>) =
if (values.Count <> 0) then
use client = _client.As<'T>()
let list = client.Lists.Item key
values |> Seq.iter (fun x -> client.AddItemToList(list, x))
Usage
let repo = new Repository<List<string>>
let items = [["key0";"data0"];["key1";"data1"]]
|> Seq.map (fun kd -> List.init kd ))
|> List.init
repo.SetList("test",items)
The is just a cut and paste of longer code. I have tried this in c#, f# and with non default serialization as already stated. That is I have tried six different methods to date and none delivered the data payload as json objects via ServiceStack WebServices only as strings.
JSON data, when serialized and sent over the wire, is just a string. Your client needs to be JSON aware and de-serialize it into your object. The JsonClient for C# within ServiceStack is more than capable of handling this, and so are the many JavaScript frameworks that assist with AJAX calls (jQuery, AngularJS, etc).
what is the difference between json string and parsed json string?
for eg in javascript suppose i have a string in the json format say [{},{}]
parsing this string will also produce the same thing.
So why do we need to parse?
It's just serialization/deserialization.
In Javscript code you normally work with the object, as that lets you easily get its properties, etc, while a JSON string doesn't do you much good.
var jsonobj = { "arr": [ 5, 2 ], "str": "foo" };
console.log(jsonobj.arr[1] + jsonobj.str);
// 2foo
var jsonstr = JSON.stringify(jsonobj);
// cannot do much with this
To send it to the server via an Ajax call, though, you need to serialize (stringify) it first. Likewise, you need to deserialize (parse) from a string into an object when receiving JSON back from the server.
Great question. The difference is transfer format.
JSON is only the 'Notation' of a JavaScript Object, it is not actually the JavaScript 'object-literal' itself. So as the data is received in JSON, it is just a string to be interpreted, evaluated, parsed, in order to become an actual JavaScript 'Object-Literal.
There is one physical difference between the two, and that is quotation marks. It makes sense, that JSON needs to be a string to be transferred. Here is how:
//A JavaScript Object-Literal
var anObj = { member: 'value'}
//A JSON representation of that object
var aJSON = { "member":"value" }
Hope that helps. All the best! Nash
I think a parsed json string should be the string data into the actual javascript objects and data arrays (or whichever language the json string contains)
The JSON object contains methods for parsing JSON and converting values to JSON.
It can't be called or constructed, and aside from its two method properties it has no interesting functionality of its own.
JSONParser parser = new JSONParser();
Object object = parser.parse(Message.toString());
JSONObject arObj = (JSONObject) object;