Convert serde_json Value keys to camelCase - json

I'm writing a CLI tool that reads JSON files and is supposed to convert the JSON object keys into camelCase.
Because this should work with any JSON file, I obviously can't just use strong typing and then #[serde(rename_all = "camelCase")].
I can't seem to find an obvious way in serde_json to make it use the already existing renaming code that serde clearly has and apply it to a serde_json::value::Value.
Am I missing something obvious?

You'll have to write a function that recurses through the serde_json::Value structure and replaces the keys of serde_json::Map whenever it encounters one. That's a bit awkward to implement, as there is no Map::drain.
fn rename_keys(json: &mut serde_json::Value) {
match json {
serde_json::Value::Array(a) => a.iter_mut().for_each(rename_keys),
serde_json::Value::Object(o) => {
let mut replace = serde_json::Map::with_capacity(o.len());
o.retain(|k, v| {
rename_keys(v);
replace.insert(
heck::ToLowerCamelCase::to_lower_camel_case(k.as_str()),
std::mem::replace(v, serde_json::Value::Null),
);
true
});
*o = replace;
}
_ => (),
}
}
use std::io::Read;
fn main() {
let mut stdin = vec![];
std::io::stdin()
.read_to_end(&mut stdin)
.expect("Read stdin");
let mut json = serde_json::from_slice::<serde_json::Value>(&stdin).expect("Parse Json");
rename_keys(&mut json);
println!("{}", serde_json::to_string_pretty(&json).unwrap());
}
(Note that rename_keys will produce a stack overflow on deep JSON structures, but serde_json only parses to a limited depth by default, so no need to worry. If you do need support for deeply nested structures, have a look at serde_stacker.)
If you're not interested in the serde_json::Value itself and just want to transform a JSON string, there's two more ways to go on about this:
You could do the renaming on serialization, by writing a custom serializer for a wrapper struct around serde_json::Value. An example of such a serializer is here, but you'd have to adopt it to be recursive. (Possibly, doing it at deserialization might be easier than at serialization)
Write a JSON tokenizer (or grab a crate that contains one) to skip creating the actual serde_json::Value structure and to the renaming on the token stream (no need to worry when working with GBs of JSON)

Related

Parsing Vector to Array String

I'm new to Rust.
I try to write a websocket client.
This is my format message:
"[integer, "string", object]"
Is there away to store all value in Vector something like:
let msg: Vec<interface> = vec![123, "event", AnyObject{}];
How to convert it into string and vice versa.
Thanks in advance
Conceptually speaking you want to use an enum. The type of enums used in rust are called tagged unions. Essentially you can think of them as enums which can hold data.
enum Interface {
Int(i32),
String(&'static str),
Object(AnyObject),
// etc
}
// You can then create a vec of different enum variants
let msg: Vec<Interface> = vec![Interface::Int(123), Interface::String("event"), Interface::Object(AnyObject{})];
Assuming you are referring to JSON, then the recommended solution is to use serde with serde_json. serde_json provides a Value enum you can use to represent JSON data in an unknown layout.
// Copied from serde_json docs
enum Value {
Null,
Bool(bool),
Number(Number),
String(String),
Array(Vec<Value>),
Object(Map<String, Value>),
}
However, you don't need to use Value directly (unless you want to) since serde_json provides a macro to make this easier by letting you format your code like JSON.
use serde_json::{json, Value};
let msg: Value = json!([123, "event", {}]);
// You can then serialize a Value into JSON format
println!("{:?}", msg.to_string());
// Output: "[123,\"event\",{}]"
I recommend reading through the overview since they have a bunch of convenient ways for working with JSON.

How do I generate a serde_json object from a "." separated text format?

The Problem
I am trying to generate a json object (with serde) by parsing a custom macro format that looks like this:
Plot.Polar.max: 20
Plot.Polar.min: 0
Plot.Polar.numberlabel: 0101
Plot.Polar.chartname: small-chart
Plot.Polar.Var.1:
Plot.Polar.Var.2: A label: with T+ES[T] #Data
What I get stuck on is how to set the keys for the object. In my old JavaScript code I split on \n, ., and :, had a couple of nested loops, and a reduceRight in the end to create the object like this:
// rowObject equals one row in the old macro format
let rowObject = keys.reduceRight(
(allKeys, item) => ({ [item]: allKeys }),
val,
);
My Goal
My goal is to use that json object to generate a highcharts config (json) depending on the keys and values from the custom macro. I want to be able to print just the macro in json format as well hence why I want to convert the macro to json first and not use a separate data structure (though that might be a good idea?). The json I want to produce from the macro is this:
{
"Plot": {
"Polar": {
"max": 20,
"min": 0
}
}
}
What I Have Tried
Map::insert though I am not sure how to structure the key string. How do I manage the Map objects in this case?
Another solution I see is creating the object from a raw string and merging each rowObject with the main object though this approach feels a bit hacky.
The current loop I have:
// pseudo
// let mut json_macro = new Map();
for row in macro_rows.iter() {
let row_key_value: Vec<&str> = row.split(':').collect();
let keys = row_key_value[0];
let value = row_key_value[1];
let keys_split: Vec<&str> = keys.split('.').collect();
for key in keys_split.iter() {
// TODO: accumulate a objects to row_object
}
// TODO: insert row_object to json_macro
}
The Question
Is it possible to do something like reduceRight in JavaScript or something similar in rust?
Update
I realized that I will have to treat all values as strings because it is impossible to know if a number is a string or not. What worked in the end was the solution #gizmo provided.
To insert your row into json_macro you can fold keys_split from the left and insert every key into the top-level object:
let row_key_value: Vec<&str> = row.split(':').collect();
let keys = row_key_value[0];
let value: Value = serde_json::from_str(row_key_value[1]).unwrap();
let keys_split: Vec<&str> = keys.split('.').collect();
keys_split[..keys_split.len() - 1]
.iter()
.fold(&mut json_macro, |object, &key| {
object
.entry(key)
.or_insert(Map::new().into())
.as_object_mut()
.unwrap()
})
.insert(keys_split.last().unwrap().to_string(), value);
A couple things to note here about unwrap()s:
from_str(...).unwrap(): I parse val as a JSON object here. This might not be what you want. Maybe instead you want str::parse::<i32> or something else. In any case, this parsing might fail.
.as_object_mut().unwrap(): This will explode if the input redefines a key like
Plot.Polar: 0
Plot.Polar.max: 20
The other way around, you probably want to handle the case where the key is already defined as an object.
keys_split.last().unwrap() won't fail but you might want to check if it's the empty string

How I can I lazily read multiple JSON values from a file/stream in Rust?

I'd like to read multiple JSON objects from a file/reader in Rust, one at a time. Unfortunately serde_json::from_reader(...) just reads until end-of-file; there doesn't seem to be any way to use it to read a single object or to lazily iterate over the objects.
Is there any way to do this? Using serde_json would be ideal, but if there's a different library I'd be willing use that instead.
At the moment I'm putting each object on a separate line and parsing them individually, but I would really prefer not to need to do this.
Example Use
main.rs
use serde_json;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let stdin = std::io::stdin();
let stdin = stdin.lock();
for item in serde_json::iter_from_reader(stdin) {
println!("Got {:?}", item);
}
Ok(())
}
in.txt
{"foo": ["bar", "baz"]} 1 2 [] 4 5 6
example session
Got Object({"foo": Array([String("bar"), String("baz")])})
Got Number(1)
Got Number(2)
Got Array([])
Got Number(4)
Got Number(5)
Got Number(6)
This was a pain when I wanted to do it in Python, but fortunately in Rust this is a directly-supported feature of the de-facto-standard serde_json crate! It isn't exposed as a single convenience function, but we just need to create a serde_json::Deserializer reading from our file/reader, then use its .into_iter() method to get a StreamDeserializer iterator yielding Results containing serde_json::Value JSON values.
use serde_json; // 1.0.39
fn main() -> Result<(), Box<dyn std::error::Error>> {
let stdin = std::io::stdin();
let stdin = stdin.lock();
let deserializer = serde_json::Deserializer::from_reader(stdin);
let iterator = deserializer.into_iter::<serde_json::Value>();
for item in iterator {
println!("Got {:?}", item?);
}
Ok(())
}
One thing to be aware of: if a syntax error is encountered, the iterator will start to produce an infinite sequence of error results and never move on. You need to make sure you handle the errors inside of the loop, or the loop will never end. In the snippet above, we do this by using the ? question mark operator to break the loop and return the first serde_json::Result::Err from our function.

Is there a better way to directly convert a Rust BSON document to JSON?

The idea is getting a cursor from Mongo and serializing the result set to JSON in a string. I have working code:
extern crate bson;
extern crate mongodb;
use mongodb::db::ThreadedDatabase;
use mongodb::{Client, ThreadedClient};
extern crate serde;
extern crate serde_json;
fn main() {
let client =
Client::connect("localhost", 27017).expect("Failed to initialize standalone client.");
let coll = client.db("foo").collection("bar");
let cursor = coll.find(None, None).ok().expect("Failed to execute find.");
let docs: Vec<_> = cursor.map(|doc| doc.unwrap()).collect();
let serialized = serde_json::to_string(&docs).unwrap();
println!("{}", serialized);
}
Is there a better way to do this? If not I will close this thread.
This is the sort of situation that serde-transcode was made for. What it does is it converts directly between serde formats. How it works is it takes in a Deserializer and a Serializer, then directly calls the corresponding serialize function for each deserialized item. Conceptually this is a bit similar to using serde_json::Value as an intermediate format, but it may include some extra type information if available in the input format.
Unfortunatly, the bson crate does not expose bson::de::raw::Deserializer or bson::ser::raw::Serializer so this is not currently possible. If you look in the documentation, the Deserializer and Serializer actually refer to different structs which handle the conversion to and from the Bson enum.
If bson::de::raw::Deserializer was public, then this code would have the desired effect. Hopefully this will be helpful to anyone who has a similar problem (or anyone who wants this enough to raise an issue on their repository).
let mut buffer = Vec::new();
// Manually add array separators because the proper way requires going through
// DeserializeSeed and that is a whole other topic.
buffer.push(b'[');
while cursor.advance().await? {
let bytes = cursor.current().as_bytes();
// Create deserializer and serializer
let deserializer = bson::de::raw::Deserializer::new(bytes, false);
let serializer = serde_json::Serializer::new(&mut buffer);
// Transcode between formats
serde_transcode::transcode(deserializer, serializer).unwrap();
// Manually add array separator
buffer.push(b',');
}
// Remove trailing comma and add closing bracket
if buffer.len() > 1 {
buffer.pop();
}
buffer.push(']');
// Do something with the result
println!("{}", String::from_utf8(buffer).unwrap())

Swift 3 parsing values from a json file

Swift 3 , Xcode8.2.1,
I'm trying to extract specific values from a json file in the project. The name of the file is city.list.json, and the syntax of the json file is as follows:
{"_id":707860,"name":"Hurzuf","country":"UA","coord":{"lon":34.283333,"lat":44.549999}}
{"_id":519188,"name":"Novinki","country":"RU","coord":{"lon":37.666668,"lat":55.683334}}
The input I have is the country name and i need the id value or the country code relevant returned as a string.
I get an error:
"Type 'Any?' has no subscript members",
The method I wrote:
private func findCountryCodeBy(location: String)->String{
var result:String="";
let bundle = Bundle(for: type(of: self));
if let theURL = bundle.url(forResource: "city.list", withExtension: "json") {
do {
let data = try Data(contentsOf: theURL);
if let parsedData = try? JSONSerialization.jsonObject(with: data, options:[]) as! [String:Any] {
result = parsedData["_id"][location][0] as! String;
}
} catch {
print(error);
result = "error";
}
}
return result;
}
That is not valid JSON. I think the nearest valid JSON equivalent would be EITHER a JSON list like:
[
{"_id":707860,"name":"Hurzuf","country":"UA","coord":{"lon":34.283333,"lat":44.549999}},
{"_id":519188,"name":"Novinki","country":"RU","coord":{"lon":37.666668,"lat":55.683334}}
]
Ie, a list enclosed within square brackets with each item separated by a comma.
OR a JSON dictionary:
{
"707860": {"name":"Hurzuf","country":"UA","coord":{"lon":34.283333,"lat":44.549999}},
"519188": {"name":"Novinki","country":"RU","coord":{"lon":37.666668,"lat":55.683334}}
}
Ie, a dictionary enclosed within curly brackets with the key (in this case I've used your _id as the key) before the : and the value (a dictionary of all the other items" after the :.
(Newlines, tabs, whitespace are ignored, I've just included them to make it obvious what I've done).
I think that the dictionary version may suit your code better, but it depends on what else you want to do with the data. A list may suit some situations better.
I wrote a quick Python script to simply read JSON from a file (and not do anything else with it), and it produced a parsing error for the not-quite-JSON that you had, but it worked fine on both of my JSON examples, above.
NB: If you do NOT have control over the format of the file you are reading (ie, if you are receiving it from some other source which cannot produce it in any other format) then you will have to either modify the format of the file after you receiv it to make it valid JSON, OR you will have to use something other than JSONSerialization to read it. You could modify it by replacing all occurrences of }{ or }\n{ with },{ and then put [ at the beginning and ] at the end. That should do the job for converting this particular file to valid JSON for a list. Converting to a dictionary would be a little more involved.
Ideally though, you may have control over the file format yourself, in which case, just change whatever generates the file to produce correct JSON in the first place.
Once you have your valid JSON and parsed it into your parsedData variable, you'll then need to fix this line:
result = parsedData["_id"][location][0] as! String;
Assuming that location is the the equivalent of the _id string in the JSON, then you may be able to use the dictionary version of the JSON above and replace that line with something like:
result = parsedData[location]["country"];
However, if location is not the _id string in the JSON, then you'd be better off using the list version of the JSON above, and use a for loop to compare the values of each list item (or use a dictionary version of the JSON keyed on whatever location actually relates to in the JSON).