Background
I am using the FSharp.Data JSON Type Provider with a sample that has an array of objects that may have different properties. Here is an illustrative example:
[<Literal>]
let sample = """
{ "input": [
{ "name": "Mickey" },
{ "year": 1928 }
]
}
"""
type InputTypes = JsonProvider< sample >
The JSON Type Provider creates an Input type which has both an Optional Name and an Optional Year property. That works well.
Problem
When I try to pass an instance of this to the web service, I do something like this:
InputTypes.Root(
[|
InputTypes.Input(Some("Mouse"), None)
InputTypes.Input(None, Some(2028))
|]
)
The web service is receiving the following and choking on the nulls.
{
"input": [
{
"name": "Mouse",
"year": null
},
{
"name": null,
"year": 2028
}
]
}
What I Tried
I find that this works:
InputTypes.Root(
[|
InputTypes.Input(JsonValue.Parse("""{ "name": "Mouse" }"""))
InputTypes.Input(JsonValue.Parse("""{ "year": 2028 }"""))
|]
)
It sends this:
{
"input": [
{
"name": "Mouse"
},
{
"year": 2028
}
]
}
However, on my real project, the structures are larger and would require a lot more conditional JSON string building. It kind of defeats the purpose.
Questions
Is there a way to cause the JSON Type Provider to not serialize null properties?
Is there a way to cause the JSON Type Provider to not serialize empty arrays?
As a point of comparison, the Newtonsoft.JSON library has a NullValueHandling attribute.
I don't think there is an easy way to get the JSON formatting in F# Data to drop the null fields - I think the type does not clearly distinguish between what is null and what is missing.
You can fix that by writing a helper function to drop all null fields:
let rec dropNullFields = function
| JsonValue.Record flds ->
flds
|> Array.choose (fun (k, v) ->
if v = JsonValue.Null then None else
Some(k, dropNullFields v) )
|> JsonValue.Record
| JsonValue.Array arr ->
arr |> Array.map dropNullFields |> JsonValue.Array
| json -> json
Now you can do the following and get the desired result:
let json =
InputTypes.Root(
[|
InputTypes.Input(Some("Mouse"), None)
InputTypes.Input(None, Some(2028))
|]
)
json.JsonValue |> dropNullFields |> sprintf "%O"
I'm trying to traverse some JSON response I'm getting from the OpenWeatherMap API but I'm getting some issues to retrieve some values. Here is my code:
{-# LANGUAGE OverloadedStrings #-}
import Control.Lens
import Data.Aeson.Lens (_String, key)
import Network.Wreq
myAPIKey :: String
myAPIKey = "my_api_key_here"
conditionsQuery :: String -> String -> String -> String
conditionsQuery city country key =
"https://api.openweathermap.org/data/2.5/forecast?q=" ++ city ++ "," ++ country ++ "&appid=" ++ key
main = do
print "What's the city?"
city <- getLine
print "And the country?"
country <- getLine
r <- get (conditionsQuery city country myAPIKey)
print $ r ^. responseBody . key "name" . _String
print $ r ^. responseBody . key "cod" . _String
print $ r ^. responseBody . key "id" . _String
The issue is that only the value of "cod" is returned ("200" in that case). The values for "name" and "id" appear as "", if we try with London,GB, Chicago, US (for instance). Yet the response body looks like:
{
...
"id": 2643743,
"name": "London",
"cod": 200
}
I first thought it was a type mismatch, but 200 is an Int there (unless I'm mistaken?) so I am not sure where the issue lies? "" seems to indicate that those 2 keys (id and name) do not exist, but they do.
Any ideas? Thanks in advance.
The response body does not look like that.
According to https://openweathermap.org/forecast5, the key "cod" appears at the outermost level of the JSON object, but "id" and "name" do not.
{
"city":{
"id":1851632,
"name":"Shuzenji",
...
}
"cod":"200",
...
}
I want to parse XML strings to erlang list and then to JSON.
Example Input :
<?xml version="1.0" encoding="UTF-8"?>
<!--some message here-->
<start>
<data>
<number id="333">test message</number>
<data>current date</data>
</data>
<mass>
<client>35</client>
<address>lattitude</address>
<code>3454343</code>
<foo tipo="casa">Some text message 2</foo>
<product>TEST</product>
</mass>
</start>
Output should be:
{
"start": {
"data": {
"number": {
"#id": "333",
"#text": "test message"
},
"data": "current date"
},
"mass": {
"client": "35",
"address": "lattitude",
"code": "3454343",
"foo": {
"#tipo": "casa",
"#text": "Some text message 2"
},
"product": "TEST"
}
}
}
I am trying to use erlsom:simple_form(Xml).
and getting :
{ok,{"start",[],
[{"data",[],
[{"number",[{"id","333"}],["test message"]},
{"data",[],["current date"]}]},
{"mass",[],
[{"client",[],["35"]},
{"address",[],["lattitude"]},
{"code",[],["3454343"]},
{"foo",[{"tipo","casa"}],["Some text message 2"]},
{"product",[],["TEST"]}]}]},
[]}
Now I want to delete these empty attrs. Is there any simple way to do this?
thanks in advance.
UPDATE: Make it work w/ solution from
Erlang xml to tuples and lists
BUT Getting
{"start",
[{"data",
[{"number","test message"},{"data","current date"}]},
{"mass",
[{"client","35"},
{"address","lattitude"},
{"code","3454343"},
{"foo","Some text message 2"},
{"product","TEST"}]}]}
there is no [{"id","333"}] & [{"tipo","casa"}] lists :(
The output of your simple parsing is in a set format: {Node, Attributes, Children}, so you can write a simple parser that turns that structure you have into a nested proplist. With that, you can either use mochijson or jsx to turn that proplist into a JSON string.
-module(transform).
-export([test/0]).
test() -> parse(data()).
parse({Node, [], [Value]}) when is_list(Value) ->
[{Node, Value}];
parse({Node, [], Children}) ->
V = children_to_struct(Children, []),
[{Node, V}];
parse({Node, Attributes, Children}) ->
V = attributes_to_struct(Attributes, []) ++ children_to_struct(Children, []),
[{Node, V}].
children_to_struct([], Acc) -> Acc;
children_to_struct([Value], Acc) when is_list(Value) ->
Acc ++ [{"#text", Value}];
children_to_struct([Value | T], Acc) when is_tuple(Value) ->
children_to_struct(T, Acc ++ parse(Value)).
attributes_to_struct([], Acc) -> Acc;
attributes_to_struct([{K, V}|T], Acc) ->
attributes_to_struct(T, Acc ++ [{"#" ++ K, V}]).
data() ->
{"start",[],
[{"data",[],
[{"number",[{"id","333"}],["test message"]},
{"data",[],["current date"]}]},
{"mass",[],
[{"client",[],["35"]},
{"address",[],["lattitude"]},
{"code",[],["3454343"]},
{"foo",[{"tipo","casa"}],["Some text message 2"]},
{"product",[],["TEST"]}]}]}.
Running it in the shell with mochijson:
Eshell V7.3 (abort with ^G)
1> c(transform).
{ok,transform}
2> T = transform:test().
[{"start",
[{"data",
[{"number",[{"#id","333"},{"#text","test message"}]},
{"data","current date"}]},
{"mass",
[{"client","35"},
{"address","lattitude"},
{"code","3454343"},
{"foo",[{"#tipo","casa"},{"#text","Some text message 2"}]},
{"product","TEST"}]}]}]
3>
4> iolist_to_binary(mochijson2:encode(T)).
<<"{\"start\":{\"data\":{\"number\":{\"#id\":[51,51,51],\"#text\":[116,101,115,116,32,109,101,115,115,97,103,101]},\"data\":{\"#text"...>>
i suggest to use jiffy for JSON and exml for XML.
jiffy and exml have native code which means they are so fast.
Clone and compile them.
before compiling them, you should install g++ and libexpat-dev
Example:
-module(test).
-export([convert/1]).
-include("exml/include/exml.hrl"). %% In my test
convert(XML) when erlang:is_binary(XML) ->
{ok, XMLEl} = exml:parse(XML),
jiffy:encode({[convert2(XMLEl)]}).
convert2(#xmlel{name = Name
,attrs = []
,children = [{xmlcdata, Data}]}) ->
{Name, Data};
convert2(#xmlel{name = Name
,attrs = Attrs
,children = Children}) ->
{Name, {convert_attrs(Attrs) ++ convert_children(Children)}}.
convert_attrs(Attrs) ->
convert_attrs(Attrs,[]).
convert_attrs([Attr|Attrs1], Attrs2) ->
convert_attrs(Attrs1, [convert_attr(Attr)|Attrs2]);
convert_attrs([], Attrs2) ->
lists:reverse(Attrs2).
convert_attr({Attr, Value}) ->
{<<$#, Attr/binary>>, Value}.
convert_children(Children) ->
convert_children(Children, []).
convert_children([Child|Children1], Children2) ->
convert_children(Children1, [convert_child(Child)|Children2]);
convert_children([], Children2) ->
lists:reverse(Children2).
convert_child({xmlcdata, Data}) ->
{<<"#text">>, Data};
convert_child(#xmlel{}=XMLEl) ->
convert2(XMLEl).
In the shell:
p#jahanbakhsh ~/Projects/test $ ls
exml jiffy test.erl
p#jahanbakhsh ~/Projects/test $ erl -pa jiffy/ebin exml/ebin
Erlang/OTP 19 [erts-8.2.2] [source-1ca84a4] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
Eshell V8.2.2 (abort with ^G)
1> c(test).
{ok,test}
2> XML = <<"<start><data><number id=\"333\">test message</number><data>current date</data></data><mass><client>35</client><address>lattitude</address><code>3454343</code><foo tipo=\"casa\">Some text message 2</foo><product>TEST</product></mass></start>">>.
<<"<start><data><number id=\"333\">test message</number><data>current date</data></data><mass><client>35</client><address"...>>
3> test:convert(XML).
<<"{\"start\":{\"data\":{\"number\":{\"#id\":\"333\",\"#text\":\"test message\"},\"data\":\"current date\"},\"mass\":{\"client\":\"35\",\"addres"...>>
4> io:format("~s~n", [test:convert(XML)]).
{"start":{"data":{"number":{"#id":"333","#text":"test message"},"data":"current date"},"mass":{"client":"35","address":"lattitude","code":"3454343","foo":{"#tipo":"casa","#text":"Some text message 2"},"product":"TEST"}}}
ok
5>
I'm expecting following json format:
'{
"teamId" : 9,
"teamMembers" : [ {
"userId" : 1000
}, {
"userId" : 2000
}]
}'
If I test my code with following format:-
'{
"teaXmId" : 9,
"teamMembers" : [ {
"usXerId" : 1000
}, {
"userXId" : 2000
}]
}'
I'm parsing json value as follows:-
val userId = (request.body \\ "userId" )
val teamId = (request.body \ "teamId")
val list = userId.toList
list.foreach( x => Logger.info("x val: "+x)
It doesn't throw any error to handle. Code execution goes one. Later if I try to use teamId or userId, of course it doesn't work then.
So how to check whether parsing was done correctly or stop execution right away and notify user to provide correct json format
If a value is not found when using \, then the result will be of type JsUndefined(msg). You can throw an error immediately by making sure you have the type you expect:
val teamId = (request.body \ "teamId").as[Int]
or:
val JsNumber(teamId) = request.body \ "teamId" //teamId will be BigDecimal
When using \\, if nothing is found, then an empty List is returned, which makes sense. If you want to throw an error when a certain key is not found on any object of an array, you might get the object that contains the list and proceed from there:
val teamMembers = (request.body \"teamMembers").as[Seq[JsValue]]
or:
val JsObject(teamMembers) = request.body \ "teamMembers"
And then:
val userIds = teamMembers.map(v => (v \ "userId").as[Int])
I am writing some helper functions to convert my R variables to JSON. I've come across this problem: I would like my values to be represented as JSON arrays, this can be done using the AsIs class according to the RJSONIO documentation.
x = "HELLO"
toJSON(list(x = I(x)), collapse="")
"{ \"x\": [ \"HELLO\" ] }"
But say we have a list
y = list(a = "HELLO", b = "WORLD")
toJSON(list(y = I(y)), collapse="")
"{ \"y\": {\n \"a\": \"HELLO\",\n\"b\": \"WORLD\" \n} }"
The value found in y -> a is NOT represented as an array. Ideally I would have
"{ \"y\": [{\n \"a\": \"HELLO\",\n\"b\": \"WORLD\" \n}] }"
Note the square brackets. Also I would like to get rid of all "\n"s, but collapse does not eliminate the line breaks in nested JSON. Any ideas?
try writing as
y = list(list(a = "HELLO", b = "WORLD"))
test<-toJSON(list(y = I(y)), collapse="")
when you write to file it appears as:
{ "y": [
{
"a": "HELLO",
"b": "WORLD"
}
] }
I guess you could remove the \n as
test<-gsub("\n","",test)
or use RJSON package
> rjson::toJSON(list(y = I(y)))
[1] "{\"y\":[{\"a\":\"HELLO\",\"b\":\"WORLD\"}]}"
The reason
> names(list(a = "HELLO", b = "WORLD"))
[1] "a" "b"
> names(list(list(a = "HELLO", b = "WORLD")))
NULL
examining the rjson::toJSON you will find this snippet of code
if (!is.null(names(x)))
return(toJSON(as.list(x)))
str = "["
so it would appear to need an unnamed list to treat it as a JSON array. Maybe RJSONIO is similar.