I am using serde_json in rust, and I am calling an api and get a very large json in return.
My question is this, is this possible to de-serialize this JSON partially. By partially, I mean to some, but not all properties of the JSON response.
for Example, I have this JSON:
Object {
"age_group": String(""),
"amazon_product_url": String("https://www.amazon.com/dp/0063221489?tag=NYTBSREV-20"),
"article_chapter_link": String(""),
"asterisk": Number(0),
"author": String("Jared Kushner"),
"book_image": String("https://storage.googleapis.com/du-prd/books/images/9780063221482.jpg"),
"book_image_height": Number(500),
"book_image_width": Number(331),
"book_review_link": String(""),
"book_uri": String("nyt://book/e5ec4777-5f2f-5622-9288-9b1d96e8fe1d"),
"buy_links": Array [
Object {
"name": String("Amazon"),
"url": String("https://www.amazon.com/dp/0063221489?tag=NYTBSREV-20"),
},
Object {
"name": String("Apple Books"),
"url": String("https://goto.applebooks.apple/9780063221482?at=10lIEQ"),
},
Object {
"name": String("Barnes and Noble"),
"url": String("https://www.anrdoezrs.net/click-7990613-11819508?url=https%3A%2F%2Fwww.barnesandnoble.com%2Fw%2F%3Fean%3D9780063221482"),
}
}
Then in this case, is it possible to just catch buy_links and amazon_product_url properties and never mind others?
If you only declare the fields you need, yes it is possible to only deserialize a subset of the data:
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
struct Point {
x: i32,
y: i32,
}
fn test(s: &str)
{
let p: Point = serde_json::from_str(s).unwrap();
println!("{:?}", p);
}
fn main()
{
test("{\"x\":0,\"y\":3}");
test("{\"x\":0,\"y\":2, \"z\": 4}");
}
Output:
Point { x: 0, y: 3 }
Point { x: 0, y: 2 }
As you can see, the "z" field that is present in the second test is ignored.
See play.rust-lang.org example.
Related
Here is my struct
#[derive(Serialize, Deserialize, Clone, Copy, Debug, JsonSchema)]
pub struct Position {
x: SignedDecimal,
y: SignedDecimal,
}
#[derive(Serialize, Deserialize, Clone, Copy, Debug, JsonSchema)]
pub struct SignedDecimal {
value: Decimal256,
sign: bool,
}
An example output of the default schema looks like this:
{
"x": { "value": "1.5", "sign": true },
"y": { "value": "1.5", "sign": true }
}
What I want it to look like is this:
{
"x": "1.5",
"y": "1.5"
}
But I can't seem to get the Schemars macros to do what I want.
I've tried many things, like this for example:
#[derive(Serialize, Deserialize, Clone, Copy, Debug, JsonSchema)]
#[schemars(schema_with = "String")]
pub struct SignedDecimal {
value: Decimal256,
sign: bool,
}
Ideally I am not using any additional macros on the Position struct, and instead just modifying the SignedDecimal struct. All of the relevant String impls have been defined, I just can't make the macro do what I want.
Not a perfect solution but I'm going to manually implement JsonSchema since I can't figure out how to make it work with the provided macros.
impl JsonSchema for SignedDecimal {
fn schema_name() -> String {
"signed_decimal".to_string()
}
fn json_schema(gen: &mut schemars::gen::SchemaGenerator) -> schemars::schema::Schema {
String::json_schema(gen)
}
fn is_referenceable() -> bool {
true
}
}
You could overwrite the schema for fields with #[schemars(schema_with = "…")]. For example:
#[derive(Serialize, Deserialize, Clone, Copy, Debug, JsonSchema)]
pub struct Position {
#[schemars(schema_with = "pretend_to_be_number_string")]
x: SignedDecimal,
#[schemars(schema_with = "pretend_to_be_number_string")]
y: SignedDecimal,
}
fn pretend_to_be_number_string(
gen: &mut schemars::gen::SchemaGenerator,
) -> schemars::schema::Schema {
use schemars::schema::*;
let mut obj = SchemaObject::default();
obj.instance_type = Some(SingleOrVec::Single(InstanceType::String.into()));
obj.string().pattern = Some("[+-]?[0-9]*\\.[0-9]*".into()); // TODO: rudimentary
let obj = Schema::Object(obj);
gen.definitions_mut()
.insert("SignedDecimal".into(), obj.clone());
let mut or = SchemaObject::default();
or.reference = Some("#/definitions/SignedDecimal".into());
Schema::Object(or)
}
will result in the schema
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Position",
"type": "object",
"required": [
"x",
"y"
],
"properties": {
"x": {
"$ref": "#/definitions/SignedDecimal"
},
"y": {
"$ref": "#/definitions/SignedDecimal"
}
},
"definitions": {
"SignedDecimal": {
"type": "string",
"pattern": "[+-]?[0-9]*\\.[0-9]*"
}
}
}
There is probably a way of not having to specify this on every field, but I can't find it.
I'm writing for a program that hooks into a web service which sends back JSON.
When a certain property isn't there it provides a empty object, with all its fields as empty strings instead of excluding the value. When the property exists, some of the properties are u64. How can I have it so Serde handles this case?
Rust Structs
#[derive(Clone, Debug, Deserialize)]
struct WebResponse {
foo: Vec<Foo>,
}
#[derive(Clone, Debug, Deserialize)]
struct Foo {
points: Points,
}
#[derive(Clone, Debug, Deserialize)]
struct Points {
x: u64,
y: u64,
name: String,
}
Example JSON
{
"foo":[
{
"points":{
"x":"",
"y":"",
"name":""
}
},
{
"points":{
"x":78,
"y":92,
"name":"bar"
}
}
]
}
Serde supports an interesting selection of attributes that can be used to customize the serialization or deserialization for a type while still using the derived implementation for the most part.
In your case, you need to be able to decode a field that can be specified as one of multiple types, and you don't need information from other fields to decide how to decode the problematic fields. The #[serde(deserialize_with="$path")] annotation is well suited to solve your problem.
We need to define a function that will decode either an empty string or an integer value into an u64. We can use the same function for both fields, since we need the same behavior. This function will use a custom Visitor to be able to handle both strings and integers. It's a bit long, but it makes you appreciate all the work that Serde is doing for you!
extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate serde_json;
use serde::Deserializer;
use serde::de::{self, Unexpected};
use std::fmt;
#[derive(Clone, Debug, Deserialize)]
struct WebResponse {
foo: Vec<Foo>,
}
#[derive(Clone, Debug, Deserialize)]
struct Foo {
points: Points,
}
#[derive(Clone, Debug, Deserialize)]
struct Points {
#[serde(deserialize_with = "deserialize_u64_or_empty_string")]
x: u64,
#[serde(deserialize_with = "deserialize_u64_or_empty_string")]
y: u64,
name: String,
}
struct DeserializeU64OrEmptyStringVisitor;
impl<'de> de::Visitor<'de> for DeserializeU64OrEmptyStringVisitor {
type Value = u64;
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
formatter.write_str("an integer or a string")
}
fn visit_u64<E>(self, v: u64) -> Result<Self::Value, E>
where
E: de::Error,
{
Ok(v)
}
fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
where
E: de::Error,
{
if v == "" {
Ok(0)
} else {
Err(E::invalid_value(Unexpected::Str(v), &self))
}
}
}
fn deserialize_u64_or_empty_string<'de, D>(deserializer: D) -> Result<u64, D::Error>
where
D: Deserializer<'de>,
{
deserializer.deserialize_any(DeserializeU64OrEmptyStringVisitor)
}
fn main() {
let value = serde_json::from_str::<WebResponse>(
r#"{
"foo": [
{
"points": {
"x": "",
"y": "",
"name": ""
}
},
{
"points": {
"x": 78,
"y": 92,
"name": "bar"
}
}
]
}"#,
);
println!("{:?}", value);
}
Cargo.toml:
[dependencies]
serde = "1.0.15"
serde_json = "1.0.4"
serde_derive = "1.0.15"
In str_or_u64, we use an untagged enum to represent either a string or a number. We can then deserialize the field into that enum and convert it to a number.
We annotate the two fields in Points using deserialize_with to tell it to use our special conversion:
use serde::{Deserialize, Deserializer}; // 1.0.124
use serde_json; // 1.0.64
#[derive(Debug, Deserialize)]
struct WebResponse {
foo: Vec<Foo>,
}
#[derive(Debug, Deserialize)]
struct Foo {
points: Points,
}
#[derive(Debug, Deserialize)]
struct Points {
#[serde(deserialize_with = "str_or_u64")]
x: u64,
#[serde(deserialize_with = "str_or_u64")]
y: u64,
name: String,
}
fn str_or_u64<'de, D>(deserializer: D) -> Result<u64, D::Error>
where
D: Deserializer<'de>,
{
#[derive(Deserialize)]
#[serde(untagged)]
enum StrOrU64<'a> {
Str(&'a str),
U64(u64),
}
Ok(match StrOrU64::deserialize(deserializer)? {
StrOrU64::Str(v) => v.parse().unwrap_or(0), // Ignoring parsing errors
StrOrU64::U64(v) => v,
})
}
fn main() {
let input = r#"{
"foo":[
{
"points":{
"x":"",
"y":"",
"name":""
}
},
{
"points":{
"x":78,
"y":92,
"name":"bar"
}
}
]
}"#;
dbg!(serde_json::from_str::<WebResponse>(input));
}
See also:
How to transform fields during deserialization using Serde?
Just adding a note for future viewers: in case it is helpful, I have implemented the solution from the accepted answer and published it as a crate serde-this-or-that.
I've added a section on Performance to explain that an approach with a custom Visitor as suggested, should perform overall much better than a version with an untagged enum, which does also work.
Here is a shortened implementation of the accepted solution above (should have the same result):
use serde::Deserialize;
use serde_json::from_str;
use serde_this_or_that::as_u64;
#[derive(Clone, Debug, Deserialize)]
struct WebResponse {
foo: Vec<Foo>,
}
#[derive(Clone, Debug, Deserialize)]
struct Foo {
points: Points,
}
#[derive(Clone, Debug, Deserialize)]
struct Points {
#[serde(deserialize_with = "as_u64")]
x: u64,
#[serde(deserialize_with = "as_u64")]
y: u64,
name: String,
}
fn main() {
let value = from_str::<WebResponse>(
r#"{
"foo": [
{
"points": {
"x": "",
"y": "",
"name": ""
}
},
{
"points": {
"x": 78,
"y": 92,
"name": "bar"
}
}
]
}"#,
);
println!("{:?}", value);
}
The Cargo.toml would look like:
[dependencies]
serde = { version = "1.0.136", features = ["derive"] }
serde_json = "1.0.79"
serde-this-or-that = "0.4"
Hello I have a &str type variable. How can I convert it to json? for example I have &str
{"index":0,"name":"AB/CDE/FG/402/test_int4","sts":"on","time":"2021-06-05 03:28:24.044284300 UTC","value":8}
How can I convert this to json
{
"index": 0,
"name": "AB/CDE/FG/402/test_int4",
"sts": "on",
"time": "2021-06-05 03:28:24.044284300 UTC",
"value": 8
}
is there anything like python json.loads() method available or something equivalent can be done in rust?
There's a misconception here. JSON is a serialization format, its always a string. Your two blocks are both JSON and functionally identical. What json.loads() is deserialize it into a value that can be accessed and manipulated natively.
You probably want to use a serde_json::Value if you want to this JSON to be used generically.
use serde_json::Value;
fn main() {
let input = r#"{"index":0,"name":"AB/CDE/FG/402/test_int4","sts":"on","time":"2021-06-05 03:28:24.044284300 UTC","value":8}"#;
let mut object: Value = serde_json::from_str(input).unwrap();
if let Some(name) = object.get_mut("name") {
*name = "new name".into();
}
println!("{}", object);
}
{"index":0,"name":"new name","sts":"on","time":"2021-06-05 03:28:24.044284300 UTC","value":8}
Playground
Or, its typical to create a structure that reflects what the JSON is expected to be and deserialize into that type via serde:
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize, Debug)]
struct Data {
index: i32,
name: String,
sts: String,
time: String,
value: i32
}
fn main() {
let input = r#"{"index":0,"name":"AB/CDE/FG/402/test_int4","sts":"on","time":"2021-06-05 03:28:24.044284300 UTC","value":8}"#;
let mut object: Data = serde_json::from_str(input).unwrap();
object.name = "new name".to_string();
println!("{:#?}", object);
}
Data {
index: 0,
name: "new name",
sts: "on",
time: "2021-06-05 03:28:24.044284300 UTC",
value: 8,
}
Playground
I don't know anything built-in, but this package can do it:
use tinyjson::{JsonParseError, JsonValue};
fn main() -> Result<(), JsonParseError> {
let s = r#"
{
"index":0,
"name":"AB/CDE/FG/402/test_int4",
"sts":"on",
"time":"2021-06-05 03:28:24.044284300 UTC",
"value":8
}
"#;
let m: JsonValue = s.parse()?;
println!("{:?}", m);
Ok(())
}
Result:
Object({
"value": Number(8.0),
"index": Number(0.0),
"name": String("AB/CDE/FG/402/test_int4"),
"time": String("2021-06-05 03:28:24.044284300 UTC"),
"sts": String("on")
})
https://docs.rs/tinyjson
I have a some JSON data that is returned from a web service. The JSON is a top-level array:
[
{
"data": "value1"
},
{
"data": "value2"
},
{
"data": "value3"
}
]
Using serde_derive to make structs I can can deserialize the data contained within the array, however, I am unable to get Serde to deserialize the top-level array.
Am I missing something, or can Serde not deserialize top level-arrays?
You can simply use a Vec:
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
struct Foo {
data: String,
}
fn main() -> Result<(), serde_json::Error> {
let data = r#"[
{
"data": "value1"
},
{
"data": "value2"
},
{
"data": "value3"
}
]"#;
let datas: Vec<Foo> = serde_json::from_str(data)?;
for data in datas.iter() {
println!("{:#?}", data);
}
Ok(())
}
If you wish, you could also use transparent:
#[derive(Serialize, Deserialize, Debug)]
#[serde(transparent)]
struct Foos {
foos: Vec<Foo>,
}
let foos: Foos = serde_json::from_str(data)?;
This allows to encapsulate your data with your type.
I have the example below, where 2 "unusual" things can happen:
if the status is NOK, the data element will not be included at all
some attributes of the elements inside list might be missing (in the example below, key2 is missing in the second element of list.
Is there a way to de-serialise an irregular JSON string using automatic deriving?
Which is the easiest/better way to handle this kind of irregular JSON in Rust? Can I avoid writing a very complex match based code to check for every possible combination?
extern crate serialize;
static JSON: &'static str = r#"
{
"status": {
"status": "OK"
},
"data": {
"container": {
"key": "value",
"list": [
{
"key1": "value1",
"key2": "value2"
},
{
"key1": "value1"
}
]
}
}
}"#;
#[deriving(Decodable, Show)]
struct Top {
status: Status,
data: Data,
}
#[deriving(Decodable, Show)]
struct Data {
container: Container,
}
#[deriving(Decodable, Show)]
struct Status {
status: String,
}
#[deriving(Decodable, Show)]
struct Container {
key: String,
list: Vec<KeyVals>,
}
#[deriving(Decodable, Show)]
struct KeyVals {
key1: String,
key2: String,
}
fn main() {
let result: Top = match serialize::json::decode(JSON) {
Ok(x) => x,
Err(why) => fail!("Failed decoding the JSON! Reason: {}", why),
};
println!("{}", result);
}
When running the code, it fails because the second element of the list is missing the key2 attribute.
task '<main>' failed at 'Failed decoding the JSON! Reason: MissingFieldError(key2)', hello.rs:56
Thank you
Potentially existing data can be represented via an enum. In the simplest case, with an Option.
I believe using an enum will also solve your problem.
#[deriving(Encodable, Decodable)]
enum Status {
Good(Container),
Bad,
VeryBad
}
If the Container also contains potentially existing data, then you can again use an enum to represent that.