I have a really simple example:
import org.json4s._
import org.json4s.native.JsonMethods._
import org.json4s.JsonDSL._
val json = ("english" -> JString("serialization")) ~ ("japanese" -> JString("シリアライゼーション"))
println(pretty(render(json)))
What I get out of that is:
{
"english":"serialization",
"japanese":"\u30b7\u30ea\u30a2\u30e9\u30a4\u30bc\u30fc\u30b7\u30e7\u30f3"
}
What I want is this (perfectly valid AFAIK) JSON:
{
"english":"serialization",
"japanese":"シリアライゼーション"
}
I can't find it now, but I think I've read somewhere that JSON only requires two special UTF-8 characters to be escaped.
Looking at the code for render, it appears that Strings always get this extra double-escaping for non-ASCII characters.
Anyone know how I can get valid JSON without double-escaping all the UTF-8 extended characters? This seems like a very similar issue to: Why does the PHP json_encode function convert UTF-8 strings to hexadecimal entities?
Update: It turns out this is an open issue in json4s with a pending PR #327 which was closed in favor of PR #339 which in turn merged into the 3.4 release branch in a commit on Feb 13, 2016.
#0__, it is not clear what answer you want to get with your bounty. The bug mentioned in the original question has already been fixed, so you can customize whether you want Unicode characters to be encoded or not. You just need to build with a current version, e.g. with a build.sbt like this:
name := "SO_ScalaJson4sUnicodeChars"
version := "1.0"
scalaVersion := "2.12.1"
libraryDependencies += "org.json4s" %% "json4s-native" % "3.5.1"
As #kriegaex mentioned in his comment, UTF-8 is the default encoding for JSON according to RFC 7159, so encoding is not strictly necessary. This is why by default json4s does not encode, just as the OP requested:
package so
import org.json4s.JsonDSL._
import org.json4s._
import org.json4s.native.JsonMethods._
object SOTest extends App {
val json = ("english" -> JString("serialization")) ~ ("japanese" -> JString("シリアライゼーション"))
println(pretty(render(json)))
}
Console log:
{
"english":"serialization",
"japanese":"シリアライゼーション"
}
However if for some compatibility reason you need the output to be encdeded, json4s supports that as well. If you add your own customJsonFormats like this, you get encoded output:
package so
import org.json4s.JsonDSL._
import org.json4s._
import org.json4s.native.JsonMethods._
object SOTest extends App {
val json = ("english" -> JString("serialization")) ~ ("japanese" -> JString("シリアライゼーション"))
implicit val customJsonFormats = new DefaultFormats {
override def alwaysEscapeUnicode: Boolean = true
}
println(pretty(render(json)))
}
Console log:
{
"english":"serialization",
"japanese":"\u30b7\u30ea\u30a2\u30e9\u30a4\u30bc\u30fc\u30b7\u30e7\u30f3"
}
Related
Getting my feet wet with building stuff, and not being able to get Aeson to work properly I decided my new project is building a JSON parser. Very abstract since it is one way or another, so it wouldn't make sense to put all the code here.
The ByteString library lets me do what I need. Remove characters, replace stuff, but: I have a very hard time reconstructing it the exact way I took it apart. Data.Text however seems more appropriate for the job but when generated a lot of noise with /"/, \n etc.
What would be the best and fastest way to clear a file from all rubbish and restore the remaining parts to useful text? Very small part below. Remarks on the code are welcome. Learning here.
import Network.HTTP.Simple
import GHC.Generics
import qualified Data.ByteString as B
import qualified Data.ByteString.Char8 as C
import Data.Text as T
import Data.Char
import Data.Text.Encoding as DTE
word8QuoteMark = fromIntegral (ord '"')
word8Newline = fromIntegral (ord '\n')
word8Backslash = fromIntegral (ord ':')
filterJson jsonData = B.filter (/= word8Backslash)
(B.filter (/= word8Newline)
(B.filter (/= word8QuoteMark) jsonData))
importJson :: IO ()
importJson = do
jsonData <- B.readFile "local.json"
output <- return (filterJson jsonData)
print $ (output)
Now the downside is, that if someone is called eg. François, it is now returned as Fran\195\167ois. I think I would need a lot more steps to do this in Data.Text, but correct me if I am wrong...
Note: i saw in a post that Daniel Wagner strongly advises against ByteString for text, but just for the sake of argument.
JSON is, by definition, a Unicode string that represents a data structure. What you get from B.readFile, though, is a raw byte string that you must first decode to get a Unicode string. To do that, you need to know what encoding was used to create the file. Assuming the file uses UTF-8 encoding, you can do something like
import Data.Text
importJson :: String -> IO Text
importJson name = do
jsonData <- B.readFile name
return (Data.Text.Encoding.decodeUtf8 jsonData)
Once you have a Text value, you can parse that into some data structure according to the JSON grammar.
I'm using JSON feeder to compare JSON output by web services as follows,
val jsonFileFeeder = jsonFile("test_data.json")
val strategy = (value: Option[String], session: Session) => value.map { jsonFileFeeder =>
val result = JSONCompare.compareJSON("expectedStr", "actualStr", JSONCompareMode.STRICT)
if (result.failed) Failure(result.getMessage)
else Success(value)
}.getOrElse(Failure("Missing body"))
val login = exec(http("Login")
.get("/login"))
.pause(1)
.feed(feeder)
.exec(http("authorization")
.post("/auth")
.headers(headers_10)
.queryParam("""email""", "${email}")
.queryParam("""password""", "${password}")
.check(status.is(200))
.check(bodyString.matchWith(strategy)))
.pause(1)
But it throws error
value matchWith is not a member of io.gatling.core.check.DefaultFindChe
ckBuilder[io.gatling.http.check.HttpCheck,io.gatling.http.response.Response,String,String]
15:10:01.963 [ERROR] i.g.a.ZincCompiler$ - .check(bodyString.matchWith(jsonFileFeeder)))
s\lib\Login.scala:18: not found: value JSONCompare
15:10:05.224 [ERROR] i.g.a.ZincCompiler$ - val result = JSONCompare.compareJSON(jsonFileFeeder, j
sonFileFeeder, JSONCompareMode.STRICT)
^
15:10:05.631 [ERROR] i.g.a.ZincCompiler$ - two errors found
Compilation failed
Here's a sample script that semantically compares a JSON response with expected output:
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.core.json.Jackson
import java.nio.charset.StandardCharsets.UTF_8
import scala.concurrent.duration._
class BasicSimulation extends Simulation {
lazy val expectedJson = Jackson.parse(
getClass.getResourceAsStream("/output.json"),
UTF_8
)
val scn = scenario("Scenario Name")
.exec(http("request_1")
.get("http://localhost:8000/output.json")
.check(bodyString.transform(Jackson.parse).is(expectedJson))
)
setUp(scn.inject(atOnceUsers(1)))
}
It assumes there is a file output.json in the resources directory (the directory that also contains your data and request-bodies).
However, I think you should carefully consider whether this solution is right for your needs. It won't scale as well as JSONPath or regex checks (especially for large JSON files), it's inflexible, and it seems more like a functional testing task than a performance task. I suspect that if you're trying to compare JSON files in this way, then you're probably trying to solve the wrong problem.
Note that it doesn't use jsonFile, as jsonFile is designed for use as a feeder, whereas I suspect you want to compare a single request with a hard-coded response. However, jsonFile may prove useful if you will be making a number of different requests with different parameters and expect different (known) responses. Here's an example script that takes this approach:
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.core.json.Jackson
import scala.concurrent.duration._
class BasicSimulation extends Simulation {
val myFeed = jsonFile("json_data.json").circular
val scn = scenario("Scenario Name")
.feed(myFeed)
.exec(http("request_1")
.get("${targetUrl}")
.check(bodyString.transform(Jackson.parse).is("${expectedResponse}"))
)
setUp(scn.inject(atOnceUsers(2)))
}
It assumes there is a json resource in data/json_data.json, that looks something like the following:
[
{
"targetUrl":"http://localhost:8000/failure.json",
"expectedResponse":
{
"success": false,
"message": "Request Failed"
}
},
{
"targetUrl":"http://localhost:8000/success.json",
"expectedResponse":
{
"success": true,
"message": "Request Succeeded"
}
}
]
The expectedResponse should be the exact JSON you expect to get back from the server. And of course you don't just have to parameterise targetUrl, you can parameterise whatever you want in this way.
As an aside, you may also be interested to know that Gatling 2.1 is expected to allow comparing an response with a file without using hacks like these (although the current development version only supports comparing byte-for-byte, not comparing-as-json).
I am attempting to produce JSON in a Scala app using json4s. Fairly straight forward, Here's some sample value I put together to test it in my Scalatra app:
import org.json4s._
import org.json4s.JsonDSL._
object JsonStub {
val getPeople =
("people" ->
("person_id" -> 5) ~
("test_count" -> 5))
}
In my controller, I simply have:
import org.json4s._
import org.json4s.JsonDSL._
import org.json4s.{DefaultFormats, Formats}
class FooController(mongoDb: MongoClient)(implicit val swagger: Swagger) extends ApiStack with NativeJsonSupport with SwaggerSupport {
get ("/people", operation(getPeople)) {
JsonStub.getPeople
}
}
The output I'm seeing in the browser however, is the following:
{
"_1": "people",
"_2": {
"person_id": 5,
"test_count": 5
}
}
Any clue where the _1 and _2 keys are coming from? I was expecting this output instead:
{
"people":{
"person_id": 5,
"test_count": 5
}
}
What you're seeing in the output is a reflectively serialized tuple, which has fields _1 and _2. This is because the return type that the compiler has inferred for JsonStub.getPeople is Tuple2[String, JObject].
The json4s DSL uses implicit conversions to turn values like the tuple into a JValue. But, if you don't tell the compiler you wanted a JValue, it won't apply the conversion.
Ideally, this would result in a compile error, because you tried to produce JSON from something that isn't the right type. Unfortunately, because your web framework assumes you want to fall back to reflection-based serialization, it means there is another way to turn the tuple into JSON, which isn't what you wanted.
If you explicitly tell the compiler that you want a JValue and not a Tuple2, the DSL's implicit conversion will be applied in the correct place.
val getPeople: JValue =
("people" ->
("person_id" -> 5) ~
("test_count" -> 5))
I'm working on simple Haskell programme that fetches a JSON string from a server, parses it, and does something with the data. The specifics are not really pertinent for the moment, the trouble I'm having is with parsing the JSON that is returned.
I get the JSON string back from the server as an IO String type and can't seem to figure out how to parse that to a JSON object.
Any help would be much appreciated :)
Here is my code thus far.
import Data.Aeson
import Network.HTTP
main = do
src <- openURL "http://www.reddit.com/user/chrissalij/about.json"
-- Json parsing code goes here
openURL url = getResponseBody =<< simpleHTTP (getRequest url)
Note: I'm using Data.Aeson in the example as that is what seems to be recommended, however I'd be more than willing to use another library.
Also any and all of this code can be changed. If getting the
Data.Aeson is designed to be used with Attoparsec, so it only gives you a Parser that you must then use with Attoparsec. Also, Attoparsec prefers to work on ByteString, so you have to alter the way the request is made slightly to get a ByteString result instead of a String.
This seems to work:
import Data.Aeson
import Data.Attoparsec
import Data.ByteString
import Data.Maybe
import Network.HTTP
import Network.URI
main = do
src <- openURL "http://www.reddit.com/user/chrissalij/about.json"
print $ parse json src
openURL :: String -> IO ByteString
openURL url = getResponseBody =<< simpleHTTP (mkRequest GET (fromJust $ parseURI url))
Here I've just parsed the JSON as a plain Value, but you'll probably want to create your own data type and write a FromJSON instance for it to handle the conversion neatly.
Using Lift's json parser, how can I output parsed json objects into a template?
The datatypes that net.liftweb.json.JsonParser provides are not
standard lists.
package rem.lift_client
package snippet
import net.liftweb._
import util._
import Helpers._
import net.liftweb.json.JsonParser._
class SearchResults {
def render() = {
val json_raw = "[ {\"userName\":\"John\"}, {\"userName\":\"Michael\"} ]"
val json_parsed = parse(input)
"li *" #> json_parsed.toString <---- NOT CORRECT
}
}
In the above example, I wanted to output a list of users as:
John
Michael
How do I interpret the parsed object? Any ideas are welcome, thanks.
NOTE: In addition to the accepted answer, lift-json has an excellent documentation on this subject.
One way is to extract the data with case classes.
implicit val formats = DefaultFormats
case class User(userName: String)
json_parsed.extract[List[User]]