Handling different ways to represent null in serde - json

I'm writing a client library around this REST server. Octoprint (A server to manage a 3d printer), to be precise.
Here's one of the types i'm working with:
#[derive(Serialize, Deserialize, Debug)]
pub struct JobInfo {
/// The file that is the target of the current print job
pub file: FileInfo,
/// The estimated print time for the file, in seconds
#[serde(rename = "estimatedPrintTime")]
pub estimated_print_time: Option<f64>,
/// The print time of the last print of the file, in seconds
#[serde(rename = "lastPrintTime")]
pub last_print_time: Option<f64>,
/// Information regarding the estimated filament usage of the print job
pub filament: Option<Filament>,
}
Pretty straightforward, Using the multiplicity property defined in the specification of the API, I determined which properties should be considered optional, hence why some of these props are wrapped in options.
Unfortunately the documentation lies a little bit in the way multiplicity works here; here's an example on what a response looks like when the printer is in an offline state. For the sake of brevity, I will omit most of the body of this JSON message and keep just enough to get the point across
{
"job": {
"file": null,
"filepos": null,
"printTime": null,
... etc
},
...
"state": "Offline"
}
Here's the type that I'm expecting for this response:
#[derive(Serialize, Deserialize, Debug)]
pub struct JobInformationResponse {
/// Information regarding the target of the current print job
job: JobInfo,
/// Information regarding the progress of the current print job
progress: ProgressInfo,
/// A textual representation of the current state of the job
/// or connection. e.g. "Operational", "Printing", "Pausing",
/// "Paused", "Cancelling", "Error", "Offline", "Offline after error",
/// "Opening serial connection" ... - please note that this list is not exhaustive!
state: String,
/// Any error message for the job or connection. Only set if there has been an error
error: Option<String>,
}
Now I could just wrap all of these types in Options, but the previous example json wouldn't parse, since technically since job is an object, it's not going to deserialize as None despite the fact that each of it's keys are null. I was wondering if there were some sort of attribute in serde that would be able to handle this weird kind of serialization issue. I'd like to avoid just wrapping every single property in Options just to handle the edge case where the printer is offline
Edit: I guess what I'm trying to say is that I would expect that if all props on a struct in the json representation were null, that the object itself would serialize as None

If you're willing to redesign a little bit, you might be able to do something like this:
#[serde(tag = "state")]
enum JobInformationResponse {
Offline {}
// If a field only appears on one type of response, use a struct variant
Error { error: String },
// If multiple response types share fields, use a newtype variant and a substruct
Printing(JobInformationResponseOnline),
Paused(JobInformationResponseOnline),
// ...
}
struct JobInformationResponseOnline {
job: JobInfo,
progress: ProgressInfo,
}
This works in the Offline case because by default, serde ignores properties that don't fit into any field of the struct/enum variant. So it won't check whether all entries of job are null.
If you have fields that appear in every message, you can further wrap JobInformationResponse (you should probably rename it):
struct JobInformationResponseAll {
field_appears_in_all_responses: FooBar,
#[serde(flatten)]
state: JobInformationResponse // Field name doesn't matter to serde
}
But I'm not sure whether that works for you, since I certainly haven't seen enough of the spec or any real example messages.
To answer your question directly: No, there is no attribute in serde which would allow an all-null map to be de/serialized as None. You'd need two versions of the struct, one without options (to be used in your rust code) and one with (to be used in a custom deserialization function where you first deserialize to the with-options struct and then convert). Might not be worth the trouble.
And a side note: You might be happy to find #[serde(rename_all = "camelCase")] exists.

Related

Parsing Vector to Array String

I'm new to Rust.
I try to write a websocket client.
This is my format message:
"[integer, "string", object]"
Is there away to store all value in Vector something like:
let msg: Vec<interface> = vec![123, "event", AnyObject{}];
How to convert it into string and vice versa.
Thanks in advance
Conceptually speaking you want to use an enum. The type of enums used in rust are called tagged unions. Essentially you can think of them as enums which can hold data.
enum Interface {
Int(i32),
String(&'static str),
Object(AnyObject),
// etc
}
// You can then create a vec of different enum variants
let msg: Vec<Interface> = vec![Interface::Int(123), Interface::String("event"), Interface::Object(AnyObject{})];
Assuming you are referring to JSON, then the recommended solution is to use serde with serde_json. serde_json provides a Value enum you can use to represent JSON data in an unknown layout.
// Copied from serde_json docs
enum Value {
Null,
Bool(bool),
Number(Number),
String(String),
Array(Vec<Value>),
Object(Map<String, Value>),
}
However, you don't need to use Value directly (unless you want to) since serde_json provides a macro to make this easier by letting you format your code like JSON.
use serde_json::{json, Value};
let msg: Value = json!([123, "event", {}]);
// You can then serialize a Value into JSON format
println!("{:?}", msg.to_string());
// Output: "[123,\"event\",{}]"
I recommend reading through the overview since they have a bunch of convenient ways for working with JSON.

Rust: Serializing a nested json stirng to a BTree<string,string>

I am trying to serialise a nested json to a BTree<string,string>.
I will be using specific elements of this collection to bind to different structs as required.
JSON
{
"data": "some_data",
"key": "some_key",
"nestedValue": {
"timestamp": "0",
"d1": "d1",
"d2": "d2",
"time": 0,
"secindaryNestedValue": [{
"d3": "test1",
"d4": "test2"
},
{
"d3": "test3",
"d4": "test4"
}
]
},
"timestamp": 0
}
I am attempting to serialise this as follows:
let input: BTreeMap<String, String> = serde_json::from_str(INPUT).unwrap();
println!("input -> {:?}",input);
I want to get an output as following:
BTree items
Key Value
data some_data
key some_key
nested_value "{\"d1\":\"d1\",\"d2\":\"d2\",\"time\":0,\"secindaryNestedValue\":[{\"d3\":\"test1\",\"d4\":\"test2\"},{\"d3\":\"test3\",\"d4\":\"test4\"}]}"
timestamp 0
I am doing this so that my nested jsons can be as generic as possible.
In subsequent operations I will be binding the nested json to a struct using serde as follows using the struct :
use serde_derive::Deserialize;
use serde_derive::Serialize;
#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Root {
pub data: String,
pub key: String,
pub nested_value: NestedValue,
pub timestamp: i64,
}
#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct NestedValue {
pub timestamp: String,
pub d1: String,
pub d2: String,
pub time: i64,
pub secindary_nested_value: Vec<SecindaryNestedValue>,
}
#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct SecindaryNestedValue {
pub d3: String,
pub d4: String,
}
I would want to use the nested value later,
Convert the json string to json and bind it to a similar struct.
Open to suggestions on not using a BTree and something better, but my usecase requires me to have the inner nested jsons as a string which I can bind later.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=4afbad3da5584bc67c9e70ae08f41cee
(Probably not useful to OP after 2 months but giving some input for other readers.)
I strongly suspect you don't want a BTreeMap. These are primarily useful for being sorted, but these JSON key-value pairs are not something which you want to be sorted by the key. You could try a HashMap<String, String>, but again I suspect this isn't really what you want. The closest reasonable answer to your question is probably to use serde_json::Value, which is a generic type that can match any valid JSON. It is implemented as
pub enum Value {
Null,
Bool(bool),
Number(Number),
String(String),
Array(Vec<Value>),
Object(Map<String, Value>),
}
So this JSON would deserialise to an Object, and you can work with the contained map to do whatever you need. If you really want to put the nested value back to a string, use serde_json::to_string to convert it back at whatever point that is necessary. You can subsequently convert a Value to a more specific type like Root using serde_json::from_value.
However I will note that, in my experience, using Value is rarely the best approach. I have only used it when an API I was integrating with sent some arbitrary JSON as a part of some data, and subsequently required that same JSON to be sent back as part of another request. Your exact requirements aren't entirely clear but I will do my best to guess at what you really want:
To serialise (Rust to JSON, which was specified in the question but maybe not really what you meant), your structs shown should convert nicely to the JSON format specified with serde_json::to_string. Based on the rest of the question, it seems like you have some more complex requirements which may mean those need to be changed, but the principle is correct.
To deserialise (JSON to Rust), this depends on the data that you are getting. If you know that the data comes in the format given, you can deserialise it directly to the Root type you have defined, and the nested values will be build correctly as the appropriate types. If you need to modify the data in some way, I would strongly encourage you to do so on the deserialised types rather than on strings. Even if you want to replace some parts of it with other data which you get as a JSON string, it's better to deserialise that data (perhaps to NestedValue, or some other type) and work with those Rust types. This is far more reliable than trying to edit JSON strings. If you need to present some part of the data to an external user or service as JSON, you can serialise it again.
If you aren't sure that the data is going to come in the format you have written (part of your question suggests you need more generic types, perhaps because of this?), you have a couple of options based on what it may be. If some values may be missing, you can use an Option for the deserialisation type: eg. if nestedValue may be missing, use pub nested_value: Option<NestedValue> in the Root struct. If there are some other known patterns it could follow, you can use an enum to match them: eg.
#[derive(Debug, Deserialize)]
#[serde(untagged)] // This tells serde to figure out from the type structure rather than the name.
pub enum RootEnum {
Root {
data: String,
key: String,
nested_value: NestedValue, // This can be used as it was.
timestamp: i64,
}
SomeOtherType {
foo: Bar
}
}
If you do this, you may need a match block to determine how to use the data. If you really don't know what this data may look like, you would need to use Value to deserialise it. However at that point I suspect that you wouldn't be able to use the data anyway, so you should probably attempt to deserialise to a type you know, and do something appropriate with the error you get if it doesn't match. You will need some error handling anyway since the JSON may be invalid.
In any case, this has become a long answer, hopefully it will be useful to someone.

How to get this response JSON structure with Rocket (Rust)

I have a simple web service that I'm writing with Rocket, whenever data comes as a 200 response, it contains a vector of Strings. When an error comes, I want to have custom errors. And the structure I want to impose on those responses should be like in here https://stackoverflow.com/a/23708903/4402306 , specifically:
for successful responses:
{
"data": {
"topics": ["topic1", "topic2", ...]
}
}
and for errors:
{
"error": {
"status_code": 404,
"message": "not found (or any other custom error)"
}
}
For the errors I have the confusion of using a catcher in rocket vs implementing my own error structures like an enum (with rocket derivations):
#[derive(Serialize, Deserialize, Responder)]
#[response(content_type = "json")]
pub enum APIError{
CustomError1(String, #[response(ignore)] String),
CustomError2(String, #[response(ignore)] String),
...
}
So what should I use to achieve my goal?
I also not sure how to represent the successful case (because with rocket, the first field should implement the Responder trait, but I have a Vec<String> there)
#[derive(Serialize, Deserialize, Responder)]
#[response(content_type = "json")]
pub struct Data {
topics: Vec<String>,
}
And finally, I think I also need to combine Data and APIError in a single Response struct - but then it's not clear to me how to control when to return what.
Assistance will be highly appreciated.
You can register a custom Catcher that returns a custom message.
See this.

Receiving Websocket data in Swift

I'm carrying this on from this question, since the focus has changed.
I am trying to send string data from a vapor server over a websocket. The client side is where the main question is. This code successfully receives the string, which is expected to be JSON (but not absolutely guaranteed -- out of scope).
switch message {
case .data(let data):
print("data: \(data)")
case .string(let str):
// let data = str.message(using: .utf8)
let jsonData = Data(str.utf8)
print("string: \(jsonData)")
do {
struct Person : Codable {
var name: String
}
let decoder = JSONDecoder()
let people = try decoder.decode([Person].self, from: jsonData)
print("result: \(people)")
} catch {
print(error.localizedDescription)
}
}
After some very helpful guidance, sending a string such as "{\"name\": \"Bobberoo\"}" will print out
string: 20 bytes
The data couldn’t be read because it isn’t in the correct format.
If I wrap it in braces "[{\"name\": \"Bobberoo\"}]" produces the more helpful but still mystifing (to me) output:
result: [wb2_socket_client.WebSocketController.(unknown context at $101a35028).(unknown context at $101a350c0).(unknown context at $101a35158).Person(name: "Bobberoo")]
Clearly, the decoding is happening, but it's wrapped in these contexts. What are they? I can see that the first is the instance of the WebSocketController. How do I access this data.
And as a non-inflammatory aside: managing JSON is a trivial operation in any number of contexts. Python/Flask, Node, Ruby/Rails and on and on; I've used all these and implementing this kind of interaction is trivial. In Swift, it's a horrible, underdocumented nightmare. At least, that's my experience. Why? I know the language is type safe, but this is ridiculous.
error.localizedDescription won't give you an error message that is useful message for debugging. On the other hand, if you print error directly:
print(error)
You'd get something along the lines of "expected to decode array but found dictionary instead", which is exactly what is happening in the case of
{
"name": "Bobberoo"
}
You are decoding a [Person].self, i.e. an array of Person, but your JSON root is not a JSON array. The above JSON can be decoded if you did:
let people = try decoder.decode(Person.self, from: jsonData)
Clearly, the decoding is happening, but it's wrapped in these contexts. What are they?
This is the default string representation of a type. Your Person struct does not conform to CustomStringConvertible or CustomDebugStringConvertible or TextOutputStreamable, so "an unspecified result is supplied automatically by the Swift standard library" (the link points to String.init(reflecting:), which presumably gets called somewhere along the way when you print the array of Person) and used as the string representation.
From what I can see, its current implementation is the fully qualified name of the struct - starting with the module, then the top-level class, then each enclosing scope, ending with the struct name, followed by the struct's members in brackets. It turns out that the enclosing scopes has no "names", and so are just called (unknown context at xxxxx). This is all very much implementation details, and things that you shouldn't care about.
What you should do, is provide an implementation of CustomStringConvertible:
struct Person: CustomStringConvertible {
...
var description: String { "name: \(name)" }
}
Now printing people gives:
[name: Bobberoo]
I can see that the first is the instance of the WebSocketController.
No. The WebSocketController is part of the fully qualified name of your Person struct. There is exactly one instance in your decoded array, and it's an instance of Person, as you would expect!
How do I access this data?
To access its name:
if let firstPerson = people.first {
let firstPersonsName = firstPerson.name
}

'Safe' struct to JSON marshalling in Go

Is there a way to marshall a struct to JSON that skips any fields that can not be serialised?
E.G. if I marshal
type aStruct struct {
request *http.Request
name string
}
the JSON representation of
type aStruct struct {
name string
}
would result?
By 'fields that can not be serialised' I mean any field that would cause json.Marshal(..) to return an 'error serializing json: unsupported type' error. Having a look through the json package now with a mind to create another Marshal function that will skip a field it fails to serialise rather than abort and return the error.
==== UPDATE ====
I have created this https://github.com/myles-mcdonnell/jsonx which satisfies my use case. At first I thought this is a terrible way to extend the base json package (it's a copy from 1.6.2 with new behaviour added) but the consensus among my colleagues is that this is the idiomatic way to do this.