deserializing serde_json from API in rust - json

I am attempting to scrape the JSON from this endpoint https://prices.runescape.wiki/api/v1/osrs/latest .
#[derive(Serialize, Deserialize, Debug)]
struct Latest {
high: u32,
highTime: String,
low: String,
lowTime: String,
}
#[derive(Serialize, Deserialize, Debug)]
struct Data {
#[serde(with = "serde_with::json::nested")]
data: Vec<Latest>,
}
#[derive(Serialize, Deserialize, Debug)]
struct Datav2 {
#[serde(with = "serde_with::json::nested")]
data: HashMap<u32, Vec<Latest>>,
}
#[cfg(not(target_arch = "wasm32"))]
#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
let res = reqwest::get(url).await?;
let response = &res.json::<Datav2>().await?;
}
I've tried two versions of the datastructure. Data is using a vector of latest, but i noticed each object has a unique ID, so in DataV2 i tried using a hashmap but i get the same error. I've also tried unnested versions without using Serde_with.
I get the error Error: reqwest::Error { kind: Decode, source: Error("invalid type: map, expected valid json object", line: 1, column: 8)
It seems my datastructure is messed up, but have been trying for hours to figure out the correct data structure to use.

There are multiple issues with your current code.
Datav2 is closer, but still not correct. It is not a HashMap<u32, Vec<Latest>>but a HashMap<u32, Latest>. There is no need to have another Vec, since each number is assigned a value in the JSON.
highTime, low, lowTime are not of type String (since they have no quotes in the JSON), but are unsigned numbers (to be on the safe side I just assumed them to be u64).
Apparently the fields of Latest can be null, so they need to be Options.
I would still use snake_case for the field names in the structs and then rename them to camelCase using the serde macro
I modified your code like I would do this, in order to give you an example of how it could be done:
use std::collections::HashMap;
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
#[serde(rename_all = "camelCase")]
struct Latest {
high: Option<u64>,
high_time: Option<u64>,
low: Option<u64>,
low_time: Option<u64>,
}
#[derive(Serialize, Deserialize, Debug)]
struct Data {
data: HashMap<u64, Latest>,
}
#[cfg(not(target_arch = "wasm32"))]
#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
let url = "https://prices.runescape.wiki/api/v1/osrs/latest";
let res = reqwest::get(url).await?;
let response = &res.json::<Data>().await?;
println!("{:#?}", response);
Ok(())
}

Related

Cannot retrieve struct from JSON in Rust

I'm having problems obtaining a struct from reading a JSON file.
This is how the config file looks like:
config.json
{
"config_group_1":{
"variable_1": 10
}
}
Its struct is define as:
pub struct ConfigGroup1 {
variable_1: usize
}
pub struct MainConfig {
config_group_1: ConfigGroup1
}
I'm reading the file with a function that returns String in config_content.
let config: Result = serde_json::from_str(&config_content);
But the compiler is pointing to the following error:
missing generics for type alias `serde_json::Result`
expected 1 type argument
help: use angle brackets to add missing type argument: `<T>`rustc(E0107)
How can I solve this? Where to define that MainConfig is the expected type?
It seems that a lot is missing. I suggest you read the following documentation https://docs.serde.rs/serde_json/
Below is a version that might help you
use serde::{Deserialize, Serialize};
use serde_json::Result;
#[derive(Serialize, Deserialize)]
struct MainConfig {
config_group_1: ConfigGroup1
}
#[derive(Serialize, Deserialize)]
pub struct ConfigGroup1 {
variable_1: usize
}
fn main() -> Result<()> {
let data = r#"{"config_group_1": {"variable_1": 10 }}"#;
let p: MainConfig = serde_json::from_str(data).expect("JSON was not well-formatted");
println!("Value {} ", p.config_group_1.variable_1);
Ok(())
}

rust cvs: invalid enum value

I am trying to use rust csv crate to deserialize simple csv string into the rust struct:
use serde::Deserialize;
use stringreader::StringReader;
#[derive(Debug, Deserialize)]
enum MatchOutcomeCSV {
win,
loss,
draw,
}
#[derive(Debug, Deserialize)]
struct MatchResultCSV {
team1: String,
team2: String,
status: MatchOutcomeCSV,
}
However when I try to deserialize simple input string "Allegoric Alaskans;Blithering Badgers;win" I get the following error from deserialization:
(invalid enum value) 0
For deserialization I use the following code:
ReaderBuilder::new()
.delimiter(b';')
.has_headers(false)
.from_reader(StringReader::new(match_results))
.deserialize()
.map(|x| x.unwrap())
.into_iter()
.collect()
I wonder what is the issue and how can one properly parse the struct I have above?
Edit:
Included my Cargo.toml for clarification around stringreader package:
[dependencies]
serde = { version = "1", features = ["derive"] }
csv = "1.1"
stringreader = "0.1.1"

How can I validate that the headers of a CSV file match my struct?

I need to parse a CSV file, but before actually parsing, I need to check if the file header can be assigned to my needs.
The problem is that some fields may be missing or the order of the fields may be different for different files.
I have a struct for dish
struct Dish {
title: String,
ingredients: Vec<String>,
spicy: Option<bool>,
vegetarian: Option<bool>,
}
I need to generate an error for any CSV file with a header that has missing fields from the structure (not Option) or has extra fields:
title;spicy;vegeterian
title;ingredients;poisoned
The csv crate has support for serde. The following example, adapted from the docs should do what you want:
use std::error::Error;
use std::io;
use std::process;
use serde::Deserialize;
#[derive(Debug, Deserialize)]
struct Dish {
title: String,
ingredients: Vec<String>,
spicy: Option<bool>,
vegetarian: Option<bool>,
}
fn example() -> Result<(), Box<dyn Error>> {
let mut rdr = csv::Reader::from_reader(io::stdin());
for result in rdr.deserialize() {
// Notice that we need to provide a type hint for automatic
// deserialization.
let dish: Dish = result?;
println!("{:?}", dish);
}
Ok(())
}
fn main() {
if let Err(err) = example() {
println!("error running example: {}", err);
process::exit(1);
}
}

How to convert Vec to JsonValue in Rust

I am querying my database and getting an Vec<Bookable> struct, using diesel library.
#[derive(QueryableByName)]
pub struct Bookable {
#[sql_type = "BigInt"]
pub id: i64,
#[sql_type = "Text"]
pub title: String
}
When I query the elements, I can access the result, but it's not possible to convert the Vec<Bookable> to json! macro:
pub fn get_terms(conn: &MysqlConnection) -> Vec<Bookable> {
diesel::sql_query(r#"SELECT title, LAST_INSERT_ID() 'id' from bookable_term;"#)
.load::<Bookable>(conn).expect("Query failed")
}
And later I call it this way:
let conn = connect();
let terms = bookable::get_terms(&conn);
json!({ "data": {
"items": terms }
})
The question is how to put the terms into this object and send the whole array to the API? I can stringify a json like this:
"items:" &vec!["to", "be", "or", "not", "to", "be"]
But when it comes to an existing Vec I get compiler error. I am using Rocket so it provides a rocket_contrib::json::JsonValue which holds json! macro
First, you derive your struct like -
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use rocket_contrib::{json::{Json}};
#[derive(QueryableByName, Deserialize, Serialize)]
pub struct Bookable {
#[sql_type = "BigInt"]
pub id: i64,
#[sql_type = "Text"]
pub title: String
}
Then you can do something like -
let conn = connect();
let terms = bookable::get_terms(&conn);
let mut data: Vec<Bookable> = HashMap::new();
data.insert("data", terms);
return Json(Vec<Bookable>);
Json(Vec<Bookable>) will also set the content-type as json for the response.

How can I deserialize an optional field with custom functions using Serde?

I want to serialize and deserialize a chrono::NaiveDate with custom functions, but the Serde book does not cover this functionality and the code docs also do not help.
#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate serde_json;
extern crate chrono;
use chrono::NaiveDate;
mod date_serde {
use chrono::NaiveDate;
use serde::{self, Deserialize, Serializer, Deserializer};
pub fn serialize<S>(date: &Option<NaiveDate>, s: S) -> Result<S::Ok, S::Error>
where S: Serializer {
if let Some(ref d) = *date {
return s.serialize_str(&d.format("%Y-%m-%d").to_string())
}
s.serialize_none()
}
pub fn deserialize<'de, D>(deserializer: D)
-> Result<Option<NaiveDate>, D::Error>
where D: Deserializer<'de> {
let s: Option<String> = Option::deserialize(deserializer)?;
if let Some(s) = s {
return Ok(Some(NaiveDate::parse_from_str(&s, "%Y-%m-%d").map_err(serde::de::Error::custom)?))
}
Ok(None)
}
}
#[derive(Debug, Serialize, Deserialize)]
struct Test {
pub i: u64,
#[serde(with = "date_serde")]
pub date: Option<NaiveDate>,
}
fn main() {
let mut test: Test = serde_json::from_str(r#"{"i": 3, "date": "2015-02-03"}"#).unwrap();
assert_eq!(test.i, 3);
assert_eq!(test.date, Some(NaiveDate::from_ymd(2015, 02, 03)));
test = serde_json::from_str(r#"{"i": 5}"#).unwrap();
assert_eq!(test.i, 5);
assert_eq!(test.date, None);
}
I know that Option<chrono::NaiveDate> can be easily deserialized by Serde because Chrono has Serde support, but I'm trying to learn Serde so I want to implement it myself. When I run this code I have a error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ErrorImpl { code: Message("missing field `date`"), line: 1, column: 8 }', /checkout/src/libcore/result.rs:859
note: Run with `RUST_BACKTRACE=1` for a backtrace.
The default behaviour of struct deserialization is to assign fields with their respective default value when they are not present in their serialized form. Note that this is different from the container #[serde(default)] attribute, which fills in the fields with the struct's default value.
#[derive(Debug, PartialEq, Deserialize)]
pub struct Foo<'a> {
x: Option<&'a str>,
}
let foo: Foo = serde_json::from_str("{}")?;
assert_eq!(foo, Foo { x: None });
However, this rule changes when we use another deserializer function (#[serde(deserialize_with = "path")]). A field of type Option here no longer tells the deserializer that the field may not exist. Rather, it suggests that there is a field with possible empty or null content (none in Serde terms). In serde_json for instance, Option<String> is the JavaScript equivalent to "either null or string" (null | string in TypeScript / Flow notation). This code below works fine with the given definition and date deserializer:
let test: Test = serde_json::from_str(r#"{"i": 5, "date": null}"#)?;
assert_eq!(test.i, 5);
assert_eq!(test.date, None);
Luckily, the deserialization process can become more permissive just by adding the serde(default) attribute (Option::default yields None):
#[derive(Debug, Serialize, Deserialize)]
struct Test {
pub i: u64,
#[serde(default)]
#[serde(with = "date_serde")]
pub date: Option<NaiveDate>,
}
Playground
See also:
How to deserialize a JSON file which contains null values using Serde?
How can I distinguish between a deserialized field that is missing and one that is null?