How to write a vector to a JSON file in rust?
Code:
use std::fs::File;
use std::io::prelude::*;
let vec1 = vec![1.0,2.0,2.1,0.6];
let mut file = File::create("results.json").unwrap();
let data = serde_json::to_string(&vec1).unwrap();
file.write(&data);
Error:
mismatched types
expected reference `&[u8]`
found reference `&std::string::String`rustc(E0308)
Instead of writing the data to an in-memory string first, you can also write it directly to the file:
use std::fs::File;
use std::io::{BufWriter, Write};
fn main() -> std::io::Result<()> {
let vec = vec![1, 2, 3];
let file = File::create("a")?;
let mut writer = BufWriter::new(file);
serde_json::to_writer(&mut writer, &vec)?;
writer.flush()?;
Ok(())
}
This approach has a lower memory footprint and is generally preferred. Note that you should use buffered writes to the file, since serialization could otherwise result in many small writes to the file, which would severely reduce performance.
If you need to write the data to memory first for some reason, I suggest using serde_json::to_vec() instead of serde_json::to_string(), since that function will give you a Vec<u8> immediately.
The idea is getting a cursor from Mongo and serializing the result set to JSON in a string. I have working code:
extern crate bson;
extern crate mongodb;
use mongodb::db::ThreadedDatabase;
use mongodb::{Client, ThreadedClient};
extern crate serde;
extern crate serde_json;
fn main() {
let client =
Client::connect("localhost", 27017).expect("Failed to initialize standalone client.");
let coll = client.db("foo").collection("bar");
let cursor = coll.find(None, None).ok().expect("Failed to execute find.");
let docs: Vec<_> = cursor.map(|doc| doc.unwrap()).collect();
let serialized = serde_json::to_string(&docs).unwrap();
println!("{}", serialized);
}
Is there a better way to do this? If not I will close this thread.
This is the sort of situation that serde-transcode was made for. What it does is it converts directly between serde formats. How it works is it takes in a Deserializer and a Serializer, then directly calls the corresponding serialize function for each deserialized item. Conceptually this is a bit similar to using serde_json::Value as an intermediate format, but it may include some extra type information if available in the input format.
Unfortunatly, the bson crate does not expose bson::de::raw::Deserializer or bson::ser::raw::Serializer so this is not currently possible. If you look in the documentation, the Deserializer and Serializer actually refer to different structs which handle the conversion to and from the Bson enum.
If bson::de::raw::Deserializer was public, then this code would have the desired effect. Hopefully this will be helpful to anyone who has a similar problem (or anyone who wants this enough to raise an issue on their repository).
let mut buffer = Vec::new();
// Manually add array separators because the proper way requires going through
// DeserializeSeed and that is a whole other topic.
buffer.push(b'[');
while cursor.advance().await? {
let bytes = cursor.current().as_bytes();
// Create deserializer and serializer
let deserializer = bson::de::raw::Deserializer::new(bytes, false);
let serializer = serde_json::Serializer::new(&mut buffer);
// Transcode between formats
serde_transcode::transcode(deserializer, serializer).unwrap();
// Manually add array separator
buffer.push(b',');
}
// Remove trailing comma and add closing bracket
if buffer.len() > 1 {
buffer.pop();
}
buffer.push(']');
// Do something with the result
println!("{}", String::from_utf8(buffer).unwrap())
How to iterate over a gziped file which contains a single text file (csv)?
Searching crates.io I found flate2 which has the following code example for decompression:
extern crate flate2;
use std::io::prelude::*;
use flate2::read::GzDecoder;
fn main() {
let mut d = GzDecoder::new("...".as_bytes()).unwrap();
let mut s = String::new();
d.read_to_string(&mut s).unwrap();
println!("{}", s);
}
How to stream a gzip csv file?
For stream io operations rust has the Read and Write traits. To iterate over input by lines you usually want the BufRead trait, which you can always get by wrapping a Read implementation in BufReader::new.
flate2 already operates with these traits; GzDecoder implements Read, and GzDecoder::new takes anything that implements Read.
Example decoding stdin (doesn't work well on playground of course):
extern crate flate2;
use std::io;
use std::io::prelude::*;
use flate2::read::GzDecoder;
fn main() {
let stdin = io::stdin();
let stdin = stdin.lock(); // or just open any normal file
let d = GzDecoder::new(stdin).expect("couldn't decode gzip stream");
for line in io::BufReader::new(d).lines() {
println!("{}", line.unwrap());
}
}
You can then decode your lines with your usual ("without gzip") logic; perhaps make it generic by taking any input implementing BufRead.
I need to store QML source code in a JSON file, in such a way that the formatting (newlines and spacing and whatnot) are preserved. I thought about programmatically inserting special unicode characters that I would never use in my source code as markers into the JSON (when saving it) to represent new lines and spaces. When reading the source code from JSON, I would replace these markers with either a newline or a space. However, this doesn't feel like a very robust solution.
Is there a better way to do this?
You can use QByteArray::toBase64() to convert the QML source to a string that can be saved to JSON:
void SourceCodeSerialiser::read(const QJsonObject &json)
{
mQml = QByteArray::fromBase64(json["qml"].toString().toUtf8());
}
And QByteArray::toBase64() to read the saved Base64 string back to a string of QML:
void SourceCodeSerialiser::write(QJsonObject &json) const
{
json["qml"] = QString(mQml.toUtf8().toBase64());
}
(mQml is a QString)
This turns the following QML:
import QtQuick 2.0
Item {
id: item
}
into this Base64 string:
aW1wb3J0IFF0UXVpY2sgMi4wCgpJdGVtIHsKICAgIGlkOiBpdGVtCn0=
As mentioned by #dtech, it's also possible to compress the byte array using qCompress() and qUncompress() to save some memory:
void SourceCodeSerialiser::read(const QJsonObject &json)
{
mQml = qUncompress(QByteArray::fromBase64(json["qml"].toString().toUtf8()));
}
void SourceCodeSerialiser::write(QJsonObject &json) const
{
json["qml"] = QString(qCompress(mQml.toUtf8(), 9).toBase64());
}
This results in the following Base64 string:
AAAAKXjay8wtyC8qUQgsCSzNTM5WMNIz4OLyLEnNVajmUgCCzBQrhUwgl6sWABKDDFM=
This is larger than the uncompressed version because the QML snippet was so small. Larger QML files will see a benefit from compression.
I'm trying to serialize using boost property tree write_json, it saves everything as strings, it's not that data are wrong, but I need to cast them explicitly every time and I want to use them somewhere else. (like in python or other C++ json (non boost) library)
here is some sample code and what I get depending on locale:
boost::property_tree::ptree root, arr, elem1, elem2;
elem1.put<int>("key0", 0);
elem1.put<bool>("key1", true);
elem2.put<float>("key2", 2.2f);
elem2.put<double>("key3", 3.3);
arr.push_back( std::make_pair("", elem1) );
arr.push_back( std::make_pair("", elem2) );
root.put_child("path1.path2", arr);
std::stringstream ss;
write_json(ss, root);
std::string my_string_to_send_somewhare_else = ss.str();
and my_string_to_send_somewhere_else is sth. like this:
{
"path1" :
{
"path2" :
[
{
"key0" : "0",
"key1" : "true"
},
{
"key2" : "2.2",
"key3" : "3.3"
}
]
}
}
Is there anyway to save them as the values, like:
"key1" : true or "key2" : 2.2?
Ok, I've solved it like this, (of course it won't suite for everybody, as it is a bit of a hack, that need further work).
I've wrote my own write_json function (simply copied the files, json_parser.hpp and json_parser_write.hpp to my project) and modified the following lines in json_parser_write.hpp:
commented line 37 - escaping the quote '"'
changed line 76 - so that it doesn't add quotes anymore:
stream << Ch('"') << data << Ch('"'); ==> stream << data;
Then values will be saved properly except for strings, so I wrote custom translator for it:
template <typename T>
struct my_id_translator
{
typedef T internal_type;
typedef T external_type;
boost::optional<T> get_value(const T &v) { return v.substr(1, v.size() - 2) ; }
boost::optional<T> put_value(const T &v) { return '"' + v +'"'; }
};
and simply saved string using:
elem2.put<std::string>("key2", "asdf", my_id_translator<std::string>());
complete program:
#include <iostream>
#include <string>
#include <sstream>
#include <boost/property_tree/ptree.hpp>
#include "property_tree/json_parser.hpp" // copied the headers
template <typename T>
struct my_id_translator
{
typedef T internal_type;
typedef T external_type;
boost::optional<T> get_value(const T &v) { return v.substr(1, v.size() - 2) ; }
boost::optional<T> put_value(const T &v) { return '"' + v +'"'; }
};
int main(int, char *[])
{
using namespace std;
using boost::property_tree::ptree;
using boost::property_tree::basic_ptree;
try
{
ptree root, arr,elem2;
basic_ptree<std::string, std::string> elem1;
elem1.put<int>("int", 10 );
elem1.put<bool>("bool", true);
elem2.put<double>("double", 2.2);
elem2.put<std::string>("string", "some string", my_id_translator<std::string>());
arr.push_back( std::make_pair("", elem1) );
arr.push_back( std::make_pair("", elem2) );
root.put_child("path1.path2", arr);
std::stringstream ss;
write_json(ss, root);
std::string my_string_to_send_somewhere_else = ss.str();
cout << my_string_to_send_somewhere_else << endl;
}
catch (std::exception & e)
{
cout << e.what();
}
return 0;
}
result :)
{
"path1":
{
"path2":
[
{
"int": 10,
"bool": true
},
{
"double": 2.2,
"string": "some string"
}
]
}
}
Boost confirms its implementation has no 100% conformance to JSON standard. Check the following link to see their explanation:
Making a ptree variant that preserves JSON types is a future plan, but far off.!
I ended up adding another function to my utils to solve this:
#include <string>
#include <regex>
#include <boost/property_tree/json_parser.hpp>
namespace bpt = boost::property_tree;
typedef bpt::ptree JSON;
namespace boost { namespace property_tree {
inline void write_jsonEx(const std::string & path, const JSON & ptree)
{
std::ostringstream oss;
bpt::write_json(oss, ptree);
std::regex reg("\\\"([0-9]+\\.{0,1}[0-9]*)\\\"");
std::string result = std::regex_replace(oss.str(), reg, "$1");
std::ofstream file;
file.open(path);
file << result;
file.close();
}
} }
Hope that helps.
The simplest and cleanest solution that i could come up with was generating the JSON with placeholders and in the end string replacing with the actual value ditching the extra quotes.
static string buildGetOrdersCommand() {
ptree root;
ptree element;
element.put<string>("pendingOnly", ":pendingOnly");
element.put<string>("someIntValue", ":someIntValue");
root.put("command", "getOrders");
root.put_child("arguments", element);
std::ostringstream buf;
write_json(buf, root, false);
buf << std::endl;
string json = buf.str();
replace(json, ":pendingOnly", "true");
replace(json, ":someIntValue", std::to_string(15));
return json;
}
static void replace(string& json, const string& placeholder, const string& value) {
boost::replace_all<string>(json, "\"" + placeholder + "\"", value);
}
And the result is
{"command":"getOrders","arguments":{"pendingOnly":true,"someIntValue":15}}
As we have typedef basic_ptree<std::string, std::string> ptree; in the boost libraries, boost will always serialize each value as string and parse all values to a string equivalent.
From the outputted JSON it is clear that the serializer serializes everything to strings using some sort of .toString() method - that is, its unaware of the type of each member and so encloses everything in " ".
See Creating JSON arrays in Boost using Property Trees for more about this problem .
All solutions, that require custom translators for strings explicitly, seem to be quite error prone for me since it's likely to forget it sometimes. It would be nice to have some kind of overload way via inheritance for the property tree's put method to handle this implicitly but that's not possible in a robust way since it's a template and you would have to ensure full covariance for all methods of the tree. Also changing boost library stuff as a workaround should be avoided in general if possible.
The most robust way without hacks I found so far is (since C++11):
use a boost-property tree with <KeyType, std::variant<yourTypes>>
Provide a translator for your variant (details see link below!) but do not "hackify" the content in terms of JSON-specifics! These are almost totally orthogonal aspects!
Write an own reader and writer for JSON, should be easily adapted from the Boost ones
Pros:
no hacks required to affect the property tree in terms of JSON-specific details
no pollution of the Boost libraries and their namespaces except the specializations of your new own types (the variant translator)
type safer than the custom string based property tree approach
should be faster for many runtime scenarios with non-frequent serialization of the tree
Cons:
requires some efforts for a quite small detail of the behavior
might be a bit slower in terms of compilation
might be a bit slower for runtime scenarios with frequent serialization and minor changes of the tree (can be optimized for sure)
reading the json back into the tree is some kind of philosophic work in doubt to ensure as much symmetry as possible between the used types (for many purposes rather an academic issue)
For more details, for instance see
http://marko-editor.com/articles/property_tree_store_anything/
You can easily adapt this for variant usage.