Deserialize JSON when structure known but not the keys - json

I am trying to deserialize some crypto exchange JSON. The structure of the JSON is predictable but the keys are not. Each time the server adds a new currency it becomes a new key. For example, it looks something like this:
{
"timestamp":"1562405",
"username":"1234",
"BTC":
{"available":"0.00","orders":"0.00000000"},
"BCH":
{"available":"0.0000000","orders":"0.00000000"},
..
..
"GUSD":
{"available":"0.00","orders":"0.00"}
}
I tried the usual approach of defining a struct to deserialize into but every time there is a new currency my program will have an error.
I thought I would just read it all into a Value and then just iterate over it manually to put it all into a struct.
let balance_data: Value = serde_json::from_str(&String::from_utf8_lossy(&body)).unwrap();
println!("balance_data: {:?}", balance__data);
for element in balance_data.iter() {
//push into a proper map...
}
But I can't:
no method named `iter` found for type `serde_json::Value`.
I thought I would put it into a Vec<Value> but this causes a panic:
thread 'main' panicked at 'called ``Result::unwrap()` on an `Err` value: Error("invalid type: map, expected a sequence", line: 1, column: 0)'`
I also experimented with reading it into a HashMap but couldn't quite seem to crack it.
It would be amazing if we could do something like:
let balance_data: RecvBalance = serde_json::from_str(&String::from_utf8_lossy(&body)).unwrap();
#[derive(Debug, Deserialize)]
struct RecvBalance {
timestamp: String,
username: String,
HashMap<String, RecvBalanceData>,
}
#[derive(Debug, Deserialize)]
struct RecvBalanceData {
available: String,
orders: String,
}
Has anyone dealt with this situation? I need a struct with the balance data in it that I can lookup later in my program.

The doc has a related example. You need to put the HashMap and use flatten attribute:
#[derive(Debug, Deserialize)]
struct RecvBalance {
timestamp: String,
username: String,
#[serde(flatten)]
moneys: HashMap<String, RecvBalanceData>,
}

Related

Deserializing JSON field with possible choices using serde

I'm trying to use serde json deserialization to support "choices" using enum, but it doesn't seep to be working (I have python enum background)
let's say I have this json :
{"name": "content", "state": "open"}
and state can be open or closed
in python I would just create an enum and the state type would be that enum eg:
#[derive(Deserialize)]
enum State {
Open(String),
Closed(String),
}
#[derive(Deserialize)]
struct MyStruct {
name: String,
state: State,
}
and the problem is that I don't know how to derserialize open to State::Open and closed to State::Closed
I have looked into implementing my own deserializer, but it seems very complicated and very advanced for me.
is there any straightforward way ?
You should remove the String. Then you'll get another error:
unknown variant `open`, expected `Open` or `Closed`
Because your enum variants are in PascalCase while your JSON is in camelCase (or snake_case, I don't know). To fix that, add #[serde(rename_all = "camelCase")]:
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
enum State {
Open,
Closed,
}

Rust/Serde: serialize external struct to json camelcase

I am working on some code that takes a struct returned by an external library, serializes it to json, and serializes the json to protobuf using pbjson. The external library uses serde and implements Serialize, but the json that is returned is snake case. The problem is that pbjson is expecting the json to be camelcase.
How can I get a camelcase version of the serde json object? (ie configure the external library to use something like #[serde(rename_all = "camelCase")] or to convert the json keys to camelcase?)
Note: I am working with many remote structs that in total add up to almost 2k lines of code. I would like to avoid recreating these types locally if possible.
If I understand correctly, you want something like this? Basically you just need to turn the foreign items into items of your own type and then serialize those.
// foreign struct
#[derive(Serialize, Deserialize)]
struct Foreign {
the_first_field: u32,
the_second_field: String,
}
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
struct Mine {
the_first_field: u32,
the_second_field: String,
}
impl From<Foreign> for Mine {
fn from(
Foreign {
the_first_field,
the_second_field,
}: Foreign,
) -> Self {
Self {
the_first_field,
the_second_field,
}
}
}
fn main() {
// only used to construct the foreign items
let in_json = r#"[{"the_first_field": 1, "the_second_field": "second"}]"#;
let foreign_items = serde_json::from_str::<Vec<Foreign>>(in_json).unwrap();
let mine_items = foreign_items.into_iter().map(Mine::from).collect::<Vec<_>>();
let out_json = serde_json::to_string(&mine_items).unwrap();
println!("{}", out_json); // [{"theFirstField":1,"theSecondField":"second"}]
}

Rust: Serializing a nested json stirng to a BTree<string,string>

I am trying to serialise a nested json to a BTree<string,string>.
I will be using specific elements of this collection to bind to different structs as required.
JSON
{
"data": "some_data",
"key": "some_key",
"nestedValue": {
"timestamp": "0",
"d1": "d1",
"d2": "d2",
"time": 0,
"secindaryNestedValue": [{
"d3": "test1",
"d4": "test2"
},
{
"d3": "test3",
"d4": "test4"
}
]
},
"timestamp": 0
}
I am attempting to serialise this as follows:
let input: BTreeMap<String, String> = serde_json::from_str(INPUT).unwrap();
println!("input -> {:?}",input);
I want to get an output as following:
BTree items
Key Value
data some_data
key some_key
nested_value "{\"d1\":\"d1\",\"d2\":\"d2\",\"time\":0,\"secindaryNestedValue\":[{\"d3\":\"test1\",\"d4\":\"test2\"},{\"d3\":\"test3\",\"d4\":\"test4\"}]}"
timestamp 0
I am doing this so that my nested jsons can be as generic as possible.
In subsequent operations I will be binding the nested json to a struct using serde as follows using the struct :
use serde_derive::Deserialize;
use serde_derive::Serialize;
#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Root {
pub data: String,
pub key: String,
pub nested_value: NestedValue,
pub timestamp: i64,
}
#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct NestedValue {
pub timestamp: String,
pub d1: String,
pub d2: String,
pub time: i64,
pub secindary_nested_value: Vec<SecindaryNestedValue>,
}
#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct SecindaryNestedValue {
pub d3: String,
pub d4: String,
}
I would want to use the nested value later,
Convert the json string to json and bind it to a similar struct.
Open to suggestions on not using a BTree and something better, but my usecase requires me to have the inner nested jsons as a string which I can bind later.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=4afbad3da5584bc67c9e70ae08f41cee
(Probably not useful to OP after 2 months but giving some input for other readers.)
I strongly suspect you don't want a BTreeMap. These are primarily useful for being sorted, but these JSON key-value pairs are not something which you want to be sorted by the key. You could try a HashMap<String, String>, but again I suspect this isn't really what you want. The closest reasonable answer to your question is probably to use serde_json::Value, which is a generic type that can match any valid JSON. It is implemented as
pub enum Value {
Null,
Bool(bool),
Number(Number),
String(String),
Array(Vec<Value>),
Object(Map<String, Value>),
}
So this JSON would deserialise to an Object, and you can work with the contained map to do whatever you need. If you really want to put the nested value back to a string, use serde_json::to_string to convert it back at whatever point that is necessary. You can subsequently convert a Value to a more specific type like Root using serde_json::from_value.
However I will note that, in my experience, using Value is rarely the best approach. I have only used it when an API I was integrating with sent some arbitrary JSON as a part of some data, and subsequently required that same JSON to be sent back as part of another request. Your exact requirements aren't entirely clear but I will do my best to guess at what you really want:
To serialise (Rust to JSON, which was specified in the question but maybe not really what you meant), your structs shown should convert nicely to the JSON format specified with serde_json::to_string. Based on the rest of the question, it seems like you have some more complex requirements which may mean those need to be changed, but the principle is correct.
To deserialise (JSON to Rust), this depends on the data that you are getting. If you know that the data comes in the format given, you can deserialise it directly to the Root type you have defined, and the nested values will be build correctly as the appropriate types. If you need to modify the data in some way, I would strongly encourage you to do so on the deserialised types rather than on strings. Even if you want to replace some parts of it with other data which you get as a JSON string, it's better to deserialise that data (perhaps to NestedValue, or some other type) and work with those Rust types. This is far more reliable than trying to edit JSON strings. If you need to present some part of the data to an external user or service as JSON, you can serialise it again.
If you aren't sure that the data is going to come in the format you have written (part of your question suggests you need more generic types, perhaps because of this?), you have a couple of options based on what it may be. If some values may be missing, you can use an Option for the deserialisation type: eg. if nestedValue may be missing, use pub nested_value: Option<NestedValue> in the Root struct. If there are some other known patterns it could follow, you can use an enum to match them: eg.
#[derive(Debug, Deserialize)]
#[serde(untagged)] // This tells serde to figure out from the type structure rather than the name.
pub enum RootEnum {
Root {
data: String,
key: String,
nested_value: NestedValue, // This can be used as it was.
timestamp: i64,
}
SomeOtherType {
foo: Bar
}
}
If you do this, you may need a match block to determine how to use the data. If you really don't know what this data may look like, you would need to use Value to deserialise it. However at that point I suspect that you wouldn't be able to use the data anyway, so you should probably attempt to deserialise to a type you know, and do something appropriate with the error you get if it doesn't match. You will need some error handling anyway since the JSON may be invalid.
In any case, this has become a long answer, hopefully it will be useful to someone.

How to (de)serialize a strongly typed JSON dictionary in Serde?

I am writing a Rust application that handles JSON messages from a TypeScript client with a public interface. I have written some code using serde_derive and it works well, but I can't figure out how to implement dictionaries; e.g.:
{
"foo" : { "data" : 42 },
"bar" : { "data" : 1337 }
}
Here the keys are the strings "foo" and "bar" and the dictionary's values follow this schema:
use serde_derive;
use serde_json::Number;
#[derive(Serialize, Deserialize)]
struct DictionaryValue {
data: Number,
}
I am looking to access the JSON data in this manner:
#[derive(Serialize, Deserialize)]
struct Dictionary {
key: String,
value: DictionaryValue,
}
How can I (de)serialize my JSON data into/from Dictionary using Serde?
You have a logic error in your code. The structure in your JSON file describes an associative array but your Dictionary does not support multiple key-value-pairs. As Stargateur stated in the comments, you may use HashMap as Serde has Serialize and Deserialize implementations for HashMap.
Instead of using a single key-value-pair, you can rewrite your Dictionary as
type Dictionary = HashMap<String, DictionaryValue>;
and you can retrieve the data for example by
let dict: Dictionary = serde_json::from_str(json_string).unwrap();
If you now want to wrap everything in a Dictionary-struct it will look like this:
#[derive(Serialize, Deserialize)]
struct Dictionary {
inner: HashMap<String, DictionaryValue>,
}
The problem is, that serde_json now expects
{
"inner": {
"foo" : { "data" : 42 },
"bar" : { "data" : 1337 }
}
}
To get rid of this, you can add the serde(flatten) attribute to Dictionary:
#[derive(Serialize, Deserialize, Debug)]
struct Dictionary {
#[serde(flatten)]
inner: HashMap<String, DictionaryValue>,
}
If HashMap or any BTreeMap from std does not fit your needs, you can also implement your Dictionary on your own. See the docs here and here for more details.

How can I decode a generic JSON response in Swift 4?

I'm building an app with Swift 4 that consumes a JSON-RPC API. The responses all have the same general format:
{
"jsonrpc": "2.0",
"result" : { "data_type" : [ ...a bunch of instances of data_type... ] }
"id": 1
}
Where data_type would be payments, channels, peers, and so on depending on the query.
I have Decodable struct definitions for each of the data types, but I don't know how to handle the main response.
I really don't care about the jsonrpc or id fields, I'm just interested in the contents of result.
I tried:
struct LightningRPCResponse: Decodable {
let id: Int
let result: String
let json_rpc: String
}
But I got the error:
Expected to decode String but found a dictionary instead
So I tried:
struct LightningRPCResponse: Decodable {
let id: Int
let result: Dictionary
let json_rpc: String
}
But I got the error:
Reference to generic type 'Dictionary' requires arguments in <...>
Is what I'm trying to do possible or do I need to create separate response decoders to correspond to every single RPC request?
Or...should I just use string manipulation to lop off the superfluous data?
You could make two structs:
struct generalStruct:Codable {
let jsonrpc:String
let id:Int
let result:[resultsStruct]
}
struct resultsStruct{
//assuming that you have strings in here, cause you didn't specify that. And it's considered as a Dictionary like: "data_tupe":"string_value" or if you have an array also here than just make another struct or just make data_type:[String]
let data_type:String
}
With that structs you can decode now. Example:
let json = try decoder.decode(generalStruct.self, from: response.data!)
//here you can get access to each element of your 'data_type'
for obj in json.result{
for data in obj.data_type {
//you have every element from dict access here if its more objects inside every 'data_type'
}
}