How can I deserialize an optional field with custom functions using Serde? - json

I want to serialize and deserialize a chrono::NaiveDate with custom functions, but the Serde book does not cover this functionality and the code docs also do not help.
#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate serde_json;
extern crate chrono;
use chrono::NaiveDate;
mod date_serde {
use chrono::NaiveDate;
use serde::{self, Deserialize, Serializer, Deserializer};
pub fn serialize<S>(date: &Option<NaiveDate>, s: S) -> Result<S::Ok, S::Error>
where S: Serializer {
if let Some(ref d) = *date {
return s.serialize_str(&d.format("%Y-%m-%d").to_string())
}
s.serialize_none()
}
pub fn deserialize<'de, D>(deserializer: D)
-> Result<Option<NaiveDate>, D::Error>
where D: Deserializer<'de> {
let s: Option<String> = Option::deserialize(deserializer)?;
if let Some(s) = s {
return Ok(Some(NaiveDate::parse_from_str(&s, "%Y-%m-%d").map_err(serde::de::Error::custom)?))
}
Ok(None)
}
}
#[derive(Debug, Serialize, Deserialize)]
struct Test {
pub i: u64,
#[serde(with = "date_serde")]
pub date: Option<NaiveDate>,
}
fn main() {
let mut test: Test = serde_json::from_str(r#"{"i": 3, "date": "2015-02-03"}"#).unwrap();
assert_eq!(test.i, 3);
assert_eq!(test.date, Some(NaiveDate::from_ymd(2015, 02, 03)));
test = serde_json::from_str(r#"{"i": 5}"#).unwrap();
assert_eq!(test.i, 5);
assert_eq!(test.date, None);
}
I know that Option<chrono::NaiveDate> can be easily deserialized by Serde because Chrono has Serde support, but I'm trying to learn Serde so I want to implement it myself. When I run this code I have a error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ErrorImpl { code: Message("missing field `date`"), line: 1, column: 8 }', /checkout/src/libcore/result.rs:859
note: Run with `RUST_BACKTRACE=1` for a backtrace.

The default behaviour of struct deserialization is to assign fields with their respective default value when they are not present in their serialized form. Note that this is different from the container #[serde(default)] attribute, which fills in the fields with the struct's default value.
#[derive(Debug, PartialEq, Deserialize)]
pub struct Foo<'a> {
x: Option<&'a str>,
}
let foo: Foo = serde_json::from_str("{}")?;
assert_eq!(foo, Foo { x: None });
However, this rule changes when we use another deserializer function (#[serde(deserialize_with = "path")]). A field of type Option here no longer tells the deserializer that the field may not exist. Rather, it suggests that there is a field with possible empty or null content (none in Serde terms). In serde_json for instance, Option<String> is the JavaScript equivalent to "either null or string" (null | string in TypeScript / Flow notation). This code below works fine with the given definition and date deserializer:
let test: Test = serde_json::from_str(r#"{"i": 5, "date": null}"#)?;
assert_eq!(test.i, 5);
assert_eq!(test.date, None);
Luckily, the deserialization process can become more permissive just by adding the serde(default) attribute (Option::default yields None):
#[derive(Debug, Serialize, Deserialize)]
struct Test {
pub i: u64,
#[serde(default)]
#[serde(with = "date_serde")]
pub date: Option<NaiveDate>,
}
Playground
See also:
How to deserialize a JSON file which contains null values using Serde?
How can I distinguish between a deserialized field that is missing and one that is null?

Related

Serde JSON deserializing enums

I have an enum:
#[derive(Serialize, Deserialize)]
enum Action {
Join,
Leave,
}
and a struct:
#[derive(Serialize, Deserialize)]
struct Message {
action: Action,
}
and I pass a JSON string:
"{\"action\":0}" // `json_string` var
but when I try deserialzing this like this:
let msg: Message = serde_json::from_str(json_string)?;
I get the error expected value at line 1 column 11.
In the JSON if I were to replace the number 0 with the string "Join" it works, but I want the number to correspond to the Action enum's values (0 is Action::Join, 1 is Action::Leave) since its coming from a TypeScript request. Is there a simple way to achieve this?
You want serde_repr!
Here's example code from the library's README:
use serde_repr::{Serialize_repr, Deserialize_repr};
#[derive(Serialize_repr, Deserialize_repr, PartialEq, Debug)]
#[repr(u8)]
enum SmallPrime {
Two = 2,
Three = 3,
Five = 5,
Seven = 7,
}
fn main() -> serde_json::Result<()> {
let j = serde_json::to_string(&SmallPrime::Seven)?;
assert_eq!(j, "7");
let p: SmallPrime = serde_json::from_str("2")?;
assert_eq!(p, SmallPrime::Two);
Ok(())
}
For your case:
use serde_repr::{Serialize_repr, Deserialize_repr};
#[derive(Serialize_repr, Deserialize_repr)]
#[repr(u8)]
enum Action {
Join = 0,
Leave = 1,
}
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize)]
struct Message {
action: Action,
}
Without adding any extra dependencies, the least verbose way is possibly to use "my favourite serde trick", the try_from and into container attributes. But in this case I feel that custom implementations of Deserialize and Serialize are more appropriate:
impl<'de> Deserialize<'de> for Action {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
match i8::deserialize(deserializer)? {
0 => Ok(Action::Join),
1 => Ok(Action::Leave),
_ => Err(serde::de::Error::custom("Expected 0 or 1 for action")),
}
}
}
impl Serialize for Action {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
serializer.serialize_i8(match self {
Action::Join => 0,
Action::Leave => 1,
})
}
}
The custom implementations only redirect to serializing/deserializing i8. Playground

deserializing serde_json from API in rust

I am attempting to scrape the JSON from this endpoint https://prices.runescape.wiki/api/v1/osrs/latest .
#[derive(Serialize, Deserialize, Debug)]
struct Latest {
high: u32,
highTime: String,
low: String,
lowTime: String,
}
#[derive(Serialize, Deserialize, Debug)]
struct Data {
#[serde(with = "serde_with::json::nested")]
data: Vec<Latest>,
}
#[derive(Serialize, Deserialize, Debug)]
struct Datav2 {
#[serde(with = "serde_with::json::nested")]
data: HashMap<u32, Vec<Latest>>,
}
#[cfg(not(target_arch = "wasm32"))]
#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
let res = reqwest::get(url).await?;
let response = &res.json::<Datav2>().await?;
}
I've tried two versions of the datastructure. Data is using a vector of latest, but i noticed each object has a unique ID, so in DataV2 i tried using a hashmap but i get the same error. I've also tried unnested versions without using Serde_with.
I get the error Error: reqwest::Error { kind: Decode, source: Error("invalid type: map, expected valid json object", line: 1, column: 8)
It seems my datastructure is messed up, but have been trying for hours to figure out the correct data structure to use.
There are multiple issues with your current code.
Datav2 is closer, but still not correct. It is not a HashMap<u32, Vec<Latest>>but a HashMap<u32, Latest>. There is no need to have another Vec, since each number is assigned a value in the JSON.
highTime, low, lowTime are not of type String (since they have no quotes in the JSON), but are unsigned numbers (to be on the safe side I just assumed them to be u64).
Apparently the fields of Latest can be null, so they need to be Options.
I would still use snake_case for the field names in the structs and then rename them to camelCase using the serde macro
I modified your code like I would do this, in order to give you an example of how it could be done:
use std::collections::HashMap;
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
#[serde(rename_all = "camelCase")]
struct Latest {
high: Option<u64>,
high_time: Option<u64>,
low: Option<u64>,
low_time: Option<u64>,
}
#[derive(Serialize, Deserialize, Debug)]
struct Data {
data: HashMap<u64, Latest>,
}
#[cfg(not(target_arch = "wasm32"))]
#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
let url = "https://prices.runescape.wiki/api/v1/osrs/latest";
let res = reqwest::get(url).await?;
let response = &res.json::<Data>().await?;
println!("{:#?}", response);
Ok(())
}

Rust | Crate: Serde Json | How to make custom parser for specific field? [duplicate]

I'm using Serde to deserialize an XML file which has the hex value 0x400 as a string and I need to convert it to the value 1024 as a u32.
Do I need to implement the Visitor trait so that I separate 0x and then decode 400 from base 16 to base 10? If so, how do I do that so that deserialization for base 10 integers remains intact?
The deserialize_with attribute
The easiest solution is to use the Serde field attribute deserialize_with to set a custom serialization function for your field. You then can get the raw string and convert it as appropriate:
use serde::{de::Error, Deserialize, Deserializer}; // 1.0.94
use serde_json; // 1.0.40
#[derive(Debug, Deserialize)]
struct EtheriumTransaction {
#[serde(deserialize_with = "from_hex")]
account: u64, // hex
amount: u64, // decimal
}
fn from_hex<'de, D>(deserializer: D) -> Result<u64, D::Error>
where
D: Deserializer<'de>,
{
let s: &str = Deserialize::deserialize(deserializer)?;
// do better hex decoding than this
u64::from_str_radix(&s[2..], 16).map_err(D::Error::custom)
}
fn main() {
let raw = r#"{"account": "0xDEADBEEF", "amount": 100}"#;
let transaction: EtheriumTransaction =
serde_json::from_str(raw).expect("Couldn't derserialize");
assert_eq!(transaction.amount, 100);
assert_eq!(transaction.account, 0xDEAD_BEEF);
}
playground
Note how this can use any other existing Serde implementation to decode. Here, we decode to a string slice (let s: &str = Deserialize::deserialize(deserializer)?). You can also create intermediate structs that map directly to your raw data, derive Deserialize on them, then deserialize to them inside your implementation of Deserialize.
Implement serde::Deserialize
From here, it's a tiny step to promoting it to your own type to allow reusing it:
#[derive(Debug, Deserialize)]
struct EtheriumTransaction {
account: Account, // hex
amount: u64, // decimal
}
#[derive(Debug, PartialEq)]
struct Account(u64);
impl<'de> Deserialize<'de> for Account {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
let s: &str = Deserialize::deserialize(deserializer)?;
// do better hex decoding than this
u64::from_str_radix(&s[2..], 16)
.map(Account)
.map_err(D::Error::custom)
}
}
playground
This method allows you to also add or remove fields as the "inner" deserialized type can do basically whatever it wants.
The from and try_from attributes
You can also place the custom conversion logic from above into a From or TryFrom implementation, then instruct Serde to make use of that via the from or try_from attributes:
#[derive(Debug, Deserialize)]
struct EtheriumTransaction {
account: Account, // hex
amount: u64, // decimal
}
#[derive(Debug, PartialEq, Deserialize)]
#[serde(try_from = "IntermediateAccount")]
struct Account(u64);
#[derive(Deserialize)]
struct IntermediateAccount<'a>(&'a str);
impl<'a> TryFrom<IntermediateAccount<'a>> for Account {
type Error = std::num::ParseIntError;
fn try_from(other: IntermediateAccount<'a>) -> Result<Self, Self::Error> {
// do better hex decoding than this
u64::from_str_radix(&other.0[2..], 16).map(Self)
}
}
See also:
How to transform fields during serialization using Serde?

How can I do key-agnostic deserialization of JSON objects?

I am working with a less than ideal API that doesn't follow any rigid standard for sending data. Each payload comes with some payload info before the JSON, followed by the actual data inside which can be a single string or several fields.
As it stands right now, if I were to map every different payload to a struct I would end up with roughly 50 structs. I feel like this is not ideal, because a ton of these structs overlap in all but key. For instance, there are I believe 6 different versions of payloads that could be mapped to something like the following, but they all have different keys.
I have these two JSON examples:
{"key": "string"}
{"key2": "string"}
And I want to serialize both into this struct:
#[derive(Debug, Deserialize)]
struct SimpleString {
key: String,
}
The same can be said for two strings, and even a couple cases for three. The payloads are frustratingly unique in small ways, so my current solution is to just define the structs locally inside the function that deserializes them and then pass that data off wherever it needs to go (in my case a cache and an event handler)
Is there a better way to represent this that doesn't have so much duplication? I've tried looking for things like key-agnostic deserializing but I haven't found anything yet.
You can implement Deserialize for your type to decode a "map" and ignore the key name:
extern crate serde;
extern crate serde_json;
use std::fmt;
use serde::de::{Deserialize, Deserializer, Error, MapAccess, Visitor};
fn main() {
let a = r#"{"key": "string"}"#;
let b = r#"{"key2": "string"}"#;
let a: SimpleString = serde_json::from_str(a).unwrap();
let b: SimpleString = serde_json::from_str(b).unwrap();
assert_eq!(a, b);
}
#[derive(Debug, PartialEq)]
struct SimpleString {
key: String,
}
struct SimpleStringVisitor;
impl<'de> Visitor<'de> for SimpleStringVisitor {
type Value = SimpleString;
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
formatter.write_str("an object with a single string value of any key name")
}
fn visit_map<M>(self, mut access: M) -> Result<Self::Value, M::Error>
where
M: MapAccess<'de>,
{
if let Some((_, key)) = access.next_entry::<String, _>()? {
if access.next_entry::<String, String>()?.is_some() {
Err(M::Error::custom("too many values"))
} else {
Ok(SimpleString { key })
}
} else {
Err(M::Error::custom("not enough values"))
}
}
}
impl<'de> Deserialize<'de> for SimpleString {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
deserializer.deserialize_map(SimpleStringVisitor)
}
}

rustc-serialize custom enum decoding

I have a JSON structure where one of the fields of a struct could be either an object, or that object's ID in the database. Let's say the document looks like this with both possible formats of the struct:
[
{
"name":"pebbles",
"car":1
},
{
"name":"pebbles",
"car":{
"id":1,
"color":"green"
}
}
]
I'm trying to figure out the best way to implement a custom decoder for this. So far, I've tried a few different ways, and I'm currently stuck here:
extern crate rustc_serialize;
use rustc_serialize::{Decodable, Decoder, json};
#[derive(RustcDecodable, Debug)]
struct Car {
id: u64,
color: String
}
#[derive(Debug)]
enum OCar {
Id(u64),
Car(Car)
}
#[derive(Debug)]
struct Person {
name: String,
car: OCar
}
impl Decodable for Person {
fn decode<D: Decoder>(d: &mut D) -> Result<Person, D::Error> {
d.read_struct("root", 2, |d| {
let mut car: OCar;
// What magic must be done here to get the right OCar?
/* I tried something akin to this:
let car = try!(d.read_struct_field("car", 0, |r| {
let r1 = Car::decode(r);
let r2 = u64::decode(r);
// Compare both R1 and R2, but return code for Err() was tricky
}));
*/
/* And this got me furthest */
match d.read_struct_field("car", 0, u64::decode) {
Ok(x) => {
car = OCar::Id(x);
},
Err(_) => {
car = OCar::Car(try!(d.read_struct_field("car", 0, Car::decode)));
}
}
Ok(Person {
name: try!(d.read_struct_field("name", 0, Decodable::decode)),
car: car
})
})
}
}
fn main() {
// Vector of both forms
let input = "[{\"name\":\"pebbles\",\"car\":1},{\"name\":\"pebbles\",\"car\":{\"id\":1,\"color\":\"green\"}}]";
let output: Vec<Person> = json::decode(&input).unwrap();
println!("Debug: {:?}", output);
}
The above panics with an EOL which is a sentinel value rustc-serialize uses on a few of its error enums. Full line is
thread '<main>' panicked at 'called `Result::unwrap()` on an `Err` value: EOF', src/libcore/result.rs:785
What's the right way to do this?
rustc-serialize, or at least its JSON decoder, doesn't support that use case. If you look at the implementation of read_struct_field (or any other method), you can see why: it uses a stack, but when it encounters an error, it doesn't bother to restore the stack to its original state, so when you try to decode the same thing differently, the decoder is operating on an inconsistent stack, eventually leading to an unexpected EOF value.
I would recommend you look into Serde instead. Deserializing in Serde is different: instead of telling the decoder what type you're expecting, and having no clear way to recover if a value is of the wrong type, Serde calls into a visitor that can handle any of the types that Serde supports in the way it wants. This means that Serde will call different methods on the visitor depending on the actual type of the value it parsed. For example, we can handle integers to return an OCar::Id and objects to return an OCar::Car.
Here's a full example:
#![feature(custom_derive, plugin)]
#![plugin(serde_macros)]
extern crate serde;
extern crate serde_json;
use serde::de::{Deserialize, Deserializer, Error, MapVisitor, Visitor};
use serde::de::value::MapVisitorDeserializer;
#[derive(Deserialize, Debug)]
struct Car {
id: u64,
color: String
}
#[derive(Debug)]
enum OCar {
Id(u64),
Car(Car),
}
struct OCarVisitor;
#[derive(Deserialize, Debug)]
struct Person {
name: String,
car: OCar,
}
impl Deserialize for OCar {
fn deserialize<D>(deserializer: &mut D) -> Result<Self, D::Error> where D: Deserializer {
deserializer.deserialize(OCarVisitor)
}
}
impl Visitor for OCarVisitor {
type Value = OCar;
fn visit_u64<E>(&mut self, v: u64) -> Result<Self::Value, E> where E: Error {
Ok(OCar::Id(v))
}
fn visit_map<V>(&mut self, visitor: V) -> Result<Self::Value, V::Error> where V: MapVisitor {
Ok(OCar::Car(try!(Car::deserialize(&mut MapVisitorDeserializer::new(visitor)))))
}
}
fn main() {
// Vector of both forms
let input = "[{\"name\":\"pebbles\",\"car\":1},{\"name\":\"pebbles\",\"car\":{\"id\":1,\"color\":\"green\"}}]";
let output: Vec<Person> = serde_json::from_str(input).unwrap();
println!("Debug: {:?}", output);
}
Output:
Debug: [Person { name: "pebbles", car: Id(1) }, Person { name: "pebbles", car: Car(Car { id: 1, color: "green" }) }]
Cargo.toml:
[dependencies]
serde = "0.7"
serde_json = "0.7"
serde_macros = "0.7"