I am trying to retrieve and parse a JSON file using reqwest.
I used this question as a starting point but it doesn't work with my API.
The error:
Error: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let resp = reqwest::get("https://tse.ir/json/MarketWatch/data_7.json")
.await?
.json::<serde_json::Value>()
.await?;
println!("{:#?}", resp);
Ok(())
}
The API works fine with other languages. thank for your help.
Cargo.toml:
[package]
name = "rust_workspace"
version = "0.1.0"
edition = "2021"
[dependencies]
serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] }
reqwest = { version = "0.11", features = ["json", "blocking"] }
tokio = { version = "1", features = ["full"] }
bytes = "1"
The error happens usually when the response doesn't contain
"content-type":"application/json"
Even though the content is a valid json you will get that error.
To solve the issue you need to use
let text_response = reqwest.get("...").await?.text().await?;
let resp: serde_json::Value = serde_json::from_str(&text_response)?;
Related
In my new project that act like a RPC, in some moment i receive a JSON with function name and a list of parameters. Example:
{
"f": "sample.login",
"p": [
{"n": "param1", "v": "value1"},
{"n": "param2", "v": true},
{"n": "param3", "v": {"id": 1, "title": "title xyz"}}
[...] any amount of params [...]
]
}
In other moments, i need create the same structure and encode as JSON. Example:
public class Param: Codable {
public var n: String
public var v: Any?
init(n: String, v: Any?) {
self.n = n
self.v = v
}
}
struct Todo: Codable {
var id: Int64
var title: String
var data: [String: String]
var done: Bool
}
public class JsonSerializer: Serializer {
private var decoder = JSONDecoder()
private var encoder = JSONEncoder()
public func encodeRequest(functionName: String, params: [Param]) -> String {
do {
let request = JsonRequestData(f: functionName, p: params)
let data = try encoder.encode(request)
if let result = String(data: data, encoding: .utf8) {
return result
} else {
print("[JsonSerializer : encodeRequest] Error when try to encode data")
}
} catch let e {
print("[JsonSerializer : encodeRequest] Error when try to encode data: \(e.localizedDescription)")
}
return ""
}
struct JsonRequestData: Codable {
let f: String
var p: [Param]
init(f: String, p: [Param]) {
self.f = f
self.p = p
}
}
}
let todo = Todo(id: 1, title: "Title 1", data: [:], done: true)
let name = "sample.todo.single"
var params: [Param] = []
params.append(Param(n: "suffix", v: "%"))
params.append(Param(n: "p2", v: todo))
let s = JsonSerializer()
let json = s.encodeRequest(functionName: name, params: params)
print(json)
I made it work in C++ (nlohmann json) and Kotlin (with gson). Only left make it work in Swift.
I know of course Swift doesn't support encoding ANY type. And I'm aware of some limitations on this in Swift.
But I would like to find a plausible solution to my problem.
Even if the user has to implement a protocol on his side for his types, or enter his type in a list of known types or something.
The project is at this URL, if you want to see the codes in more depth:
https://github.com/xplpc/xplpc
Removing this lock, the code is practically ready.
I tried on Apple forums, search on Google and on iOS group inside Slack.
Thanks for answers.
But after try a lot, i decide to use AnyCodable project (https://github.com/Flight-School/AnyCodable) with my modifications (https://github.com/xplpc/xplpc/tree/main/swift/lib/Sources).
AnyCodable let me use all swift types and if i use these types on my class/struct it works without problems.
To use any custom type, only need add more lines on AnyEncodable and AnyDecodable class.
Thanks.
I would like to load the following csv file, which has a difference of notation rules between before 2nd line and after 3rd line.
test.csv
[YEAR],2022,[Q],1,
[TEST],mid-term,[GRADE],3,
FirstName,LastName,Score,
AA,aaa,97,
BB,bbbb,15,
CC,cccc,66,
DD,ddd,73,
EE,eeeee,42,
FF,fffff,52,
GG,ggg,64,
HH,h,86,
II,iii,88,
JJ,jjjj,72,
However, I have the following error. I think this error is caused by the difference of notation rules. How do I correct this error and load the csv file as I want.
error message
StringRecord(["[YEAR]", "2022", "[Q]", "1", ""])
StringRecord(["[TEST]", "mid-term", "[GRADE]", "3", ""])
Error: Error(UnequalLengths { pos: Some(Position { byte: 47, line: 2, record: 2 }), expected_len: 5, len: 4 })
error: process didn't exit successfully: `target\debug\read_csv.exe` (exit code: 1)
main.rs
use csv::Error;
use csv::ReaderBuilder;
use encoding_rs;
use std::fs;
fn main() -> Result<(), Error> {
let path = "./test.csv";
let file = fs::read(path).unwrap();
let (res, _, _) = encoding_rs::SHIFT_JIS.decode(&file);
let mut reader = ReaderBuilder::new()
.has_headers(false)
.from_reader(res.as_bytes());
for result in reader.records() {
let record = result?;
println!("{:?}", record)
}
Ok(())
}
Version
cargo = "1.62.0"
rustc = "1.62.0"
csv = "1.1.6"
encoding_rs = "0.8.31"
I can correct this error by using "flexible" method.
use csv::Error;
use csv::ReaderBuilder;
use encoding_rs;
use std::fs;
fn main() -> Result<(), Error> {
let path = "./test.csv";
let file = fs::read(path).unwrap();
let (res, _, _) = encoding_rs::SHIFT_JIS.decode(&file);
let mut reader = ReaderBuilder::new()
+ .flexible(true)
.has_headers(false)
.from_reader(res.as_bytes());
for result in reader.records() {
let record = result?;
println!("{:?}", record)
}
Ok(())
}
I am trying to output the Hyper HTTP Request Object as JSON. I am using Serde to Serialize the object to JSON.
Since I am not defining the HTTP Request Struct, I can't add the attribute #[derive(Serialize, Deserialize)] to it. any ideas on how to get around this?
my code:
use hyper::body::HttpBody as _;
use hyper::{Body, Client, Method, Request, Uri};
use hyper_tls::HttpsConnector;
#[cfg(feature = "std")]
use serde::{Deserialize, Deserializer, Serialize, Serializer};
use serde_json::{json, Result as re_, Value};
use tokio::io::{stdout, AsyncWriteExt as _};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let https = HttpsConnector::new();
// Client Object
let client = Client::builder().build::<_, hyper::Body>(https);
let uri: Uri = "https://api.api.com".parse()?;
let http_body = String::from(" body");
let content_length = http_body.chars().count();
//Build Request
let req = Request::builder()
.method(Method::POST)
.uri(uri)
.header("accept", "application/xml")
.header("content-type", "application/xml")
.header("Content-Length", content_length)
.body(Body::from(http_body))?;
// Serialize it to a JSON string.
let j = serde_json::to_string(&req)?;
// Print HTTP Request
println!("{}", j);
// Await the response...
let mut resp = client.request(req).await?;
println!("Response: {}", resp.status());
Ok(())
}
stack trace shows:
| let j = serde_json::to_string(&req)?;
| --------------------- ^^^^ the trait `serde::ser::Serialize` is not implemented for `Request<Body>`
| |
| required by a bound introduced by this call
|
note: required by a bound in `serde_json::to_string`
--> /home/joel/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.79/src/ser.rs:2225:17
|
2225 | T: ?Sized + Serialize,
| ^^^^^^^^^ required by this bound in `serde_json::to_string`
Cargo.toml
[dependencies]
serde = { version = "1.0.136", features = ["derive"] }
serde_json = "1.0.79"
serde_derive = "1.0.136"
hyper = { version = "0.14.18", features = ["full"] }
tokio = { version = "1.17.0", features = ["full"] }
hyper-tls = "0.5.0"
rustls = "0.20.4"
If you want to derive de/serialize for an external crate you can follow the oficial serde guide for doing so.
Also, if you are trying to serialize it just to print it, maybe you could consider writing or using a default Debug or Display implementation over a wrapper. The default Debug one may do:
// Print HTTP Request
println!("{req:?}");
I'm trying to hack together a simple WebSocket client using tungstenite that parses JSON. Most of the code comes from this example.
If I do println!("{}", msg) It works fine, so I'm not sure why the type is mismatched (msg is supposedly a string right?). https://docs.rs/tungstenite/0.6.1/tungstenite/protocol/enum.Message.html
use tungstenite::{connect};
use url::Url;
use serde_json;
fn main() {
let (mut socket, _response) =
connect(Url::parse("wss://www.bitmex.com/realtime?subscribe=trade:XBTUSD").unwrap()).expect("Can't connect");
loop {
let msg = socket.read_message().expect("Error reading message");
let parsed: serde_json::Value = serde_json::from_str(&msg).expect("Can't parse to JSON");
println!("{:?}", parsed["data"][0]["price"]);
}
}
error[E0308]: mismatched types
--> src/main.rs:12:62
|
12 | let parsed: serde_json::Value = serde_json::from_str(&msg).expect("Can't parse to JSON");
| ^^^^ expected `str`, found enum `tungstenite::protocol::message::Message`
|
= note: expected reference `&str`
found reference `&tungstenite::protocol::message::Message`
[dependencies]
tungstenite = "0.10.1"
url = "2.1.1"
serde_json = "1.0.53"
Something like this will work:
use tungstenite::{connect};
use url::Url;
use serde_json;
fn main() {
let (mut socket, _response) =
connect(Url::parse("wss://www.bitmex.com/realtime?subscribe=trade:XBTUSD").unwrap()).expect("Can't connect");
loop {
let msg = socket.read_message().expect("Error reading message");
let msg = match msg {
tungstenite::Message::Text(s) => { s }
_ => { panic!() }
};
let parsed: serde_json::Value = serde_json::from_str(&msg).expect("Can't parse to JSON");
println!("{:?}", parsed["data"][0]["price"]);
}
}
We often want to use JSON for human readability. As such, it is common to ask to sort the JSON keys alphabetically (or alphanumerically) in Go, in .NET, in Python, in Java, ...
But how to output a JSON with JSON keys sorted alphabetically in Swift?
PrettyPrinted output is easy:
JSONSerialization.writeJSONObject(jsonObject, to: outputStream, options: [.prettyPrinted], error: nil)
Yet the keys are not alphabetically sorted for human readability. They are likely in the order given by NSDictionary.keyEnumerator(). But sadly, we can't subclass Dictionary, NSDictionary or CFDictionary in Swift, so we can't override the behavior of keys order.
[edit: actually, we can subclass NSDictionary, see one of my answers below]
For iOS 11+ and macOS High Sierra (10.13+), a new option .sortedKeys solves the problem easily:
JSONSerialization.writeJSONObject(jsonObject, to: outputStream, options: [.sortedKeys, .prettyPrinted], error: nil)
Thank you Hamish for the hint.
Pure Swift solution for iOS 7+, macOS 10.9+ (OS X Mavericks and up).
A solution is to subclass NSDictionary (but not overriding the default init method as it wouldn't compile with Swift).
class MutableOrderedDictionary: NSDictionary {
let _values: NSMutableArray = []
let _keys: NSMutableOrderedSet = []
override var count: Int {
return _keys.count
}
override func keyEnumerator() -> NSEnumerator {
return _keys.objectEnumerator()
}
override func object(forKey aKey: Any) -> Any? {
let index = _keys.index(of: aKey)
if index != NSNotFound {
return _values[index]
}
return nil
}
func setObject(_ anObject: Any, forKey aKey: String) {
let index = _keys.index(of: aKey)
if index != NSNotFound {
_values[index] = anObject
} else {
_keys.add(aKey)
_values.add(anObject)
}
}
}
With it, we can order the keys of our object with .forcedOrdering before writing it with .prettyPrinted:
// force ordering
let orderedJson = MutableOrderedDictionary()
jsonObject.sorted { $0.0.compare($1.0, options: [.forcedOrdering, .caseInsensitive]) == .orderedAscending }
.forEach { orderedJson.setObject($0.value, forKey: $0.key) }
// write pretty printed
_ = JSONSerialization.writeJSONObject(orderedJson, to: outputJSON, options: [.prettyPrinted], error: nil)
But be careful: you will need to subclass and sort all subdictionaries of your JSON object if you have any. Here is an extension for doing that recursion, inspired by Evgen Bodunov's gist (thank you).
extension MutableOrderedDictionary {
private static let defaultOrder: ((String, Any), (String, Any)) -> Bool = {
$0.0.compare($1.0, options: [.forcedOrdering, .caseInsensitive]) == .orderedAscending
}
static func sorted(object: Any, by areInIncreasingOrder: ((key: String, value: Value), (key: String, value: Value)) -> Bool = defaultOrder) -> Any {
if let dict = object as? [String: Any] {
return MutableOrderedDictionary(dict, by: areInIncreasingOrder)
} else if let array = object as? [Any] {
return array.map { sorted(object: $0, by: areInIncreasingOrder) }
} else {
return object
}
}
convenience init(_ dict: [String: Any], by areInIncreasingOrder: ((key: String, value: Value), (key: String, value: Value)) -> Bool = defaultOrder) {
self.init()
dict.sorted(by: areInIncreasingOrder)
.forEach { setObject(MutableOrderedDictionary.sorted(object: $0.value, by: areInIncreasingOrder), forKey: $0.key) }
}
}
Usage:
// force ordering
let orderedJson = MutableOrderedDictionary(jsonObject)
// write pretty printed
_ = JSONSerialization.writeJSONObject(orderedJson, to: outputJSON, options: [.prettyPrinted], error: nil)
Solution for iOS 7+, macOS 10.9+ (OS X Mavericks and up).
A solution is to subclass NSDictionary in Objective-C, then use the subclass from a framework (for Application) or static library (for Command Line Tool).
For this demonstration, I will use nicklockwood/OrderedDictionary (700+ lines of code) instead of doing it from scratch, but there may be untested alternatives like quinntaylor/CHOrderedDictionary or rhodgkins/RDHOrderedDictionary. To integrate it as a Framework, add this dependency in your PodFile:
pod 'OrderedDictionary', '~> 1.4'
Then we will order the keys of our object:
import OrderedDictionary
let orderedJson = MutableOrderedDictionary()
jsonObject.sorted { $0.0.compare($1.0, options: [.forcedOrdering, .caseInsensitive]) == .orderedAscending }
.forEach { orderedJson.setObject($0.value, forKey: $0.key) }
(note: setObject(_,forKey:) is specific to MutableOrderedDictionary)
And finally we can write it prettyPrinted:
_ = JSONSerialization.writeJSONObject(orderedJson, to: outputJSON, options: [.prettyPrinted], error: nil)
But be careful: you need to subclass and sort all subdictionaries of your JSON object.
Here is one possible workaround that works only for macOS Swift scripts. It is not for iOS.
We workaround Swift Foundation limitations with a different programming language (Python 2.7 for example).
import Cocoa
// sample jsonObject
let jsonObject = ["hello": "world", "foo": "bar"]
// writing JSON to file
let jsonPath = "myJson.json"
let outputJSON = OutputStream(toFileAtPath: jsonPath, append: false)!
outputJSON.open()
_ = JSONSerialization.writeJSONObject(jsonObject, to: outputJSON, options: [], error: nil)
outputJSON.close()
// sortedKeys equivalent using `sort_keys=True`
// prettyPrinted equivalent using `indent=2, separators=(',', ' : ')`
// unicode using `io.open` and `ensure_ascii=False`
func shell(_ args: String...) {
let task = Process()
task.launchPath = "/usr/bin/env"
task.arguments = args
task.launch()
task.waitUntilExit()
}
shell("python", "-c", ""
+ "import io\n"
+ "import json\n"
+ "jsonPath = 'myJson.json'\n"
+ "with io.open(jsonPath, mode='r', encoding='utf8') as json_file:\n"
+ " all_data = json.load(json_file)\n"
+ "with io.open(jsonPath, mode='w', encoding='utf8') as json_file:\n"
+ " json_file.write(unicode(json.dumps(all_data, ensure_ascii=False, sort_keys=True, indent=2, separators=(',', ' : '))))\n"
)