I am a total beginner with Erlang and functional programming in general. For fun, to get me started, I am converting an existing Ruby Sinatra REST(ish) API that queries PostgreSQL and returns JSON.
On the Erlang side I am using Cowboy, Epgsql and Jiffy as the JSON library.
Epgsql returns results in the following format:
{ok, [{column,<<"column_name">>,int4,4,-1,0}], [{<<"value">>}]}
But Jiffy expects the following format when encoding to JSON:
{[{<<"column_name">>,<<"value">>}]}
The following code works to convert epgsql output into suitable input for jiffy:
Assuming Data is the Epgsql output and Key is the name of the JSON object being created:
{_, C, R} = Data,
Columns = [X || {_, X, _, _, _, _} <- C,
Rows = tuple_to_list(hd(R)),
Result = {[{atom_to_binary(Key, utf8), {lists:zip(Columns, Rows)}}]}.
However, I am wondering if this is efficient Erlang?
I've looked into the documentation for Epgsql and Jiffy and can't see any more obvious ways to perform the conversion.
Thank you.
Yes, need parse it.
For example function parse result
parse_result({error, #error{ code = <<"23505">>, extra = Extra }}) ->
{match, [Column]} =
re:run(proplists:get_value(detail, Extra),
"Key \\(([^\\)]+)\\)", [{capture, all_but_first, binary}]),
throw({error, {non_unique, Column}});
parse_result({error, #error{ message = Msg }}) ->
throw({error, Msg});
parse_result({ok, Cols, Rows}) ->
to_map(Cols, Rows);
parse_result({ok, Counts, Cols, Rows}) ->
{ok, Counts, to_map(Cols, Rows)};
parse_result(Result) ->
Result.
And function convert result to map
to_map(Cols, Rows) ->
[ maps:from_list(lists:zipwith(fun(#column{name = N}, V) -> {N, V} end,
Cols, tuple_to_list(Row))) || Row <- Rows ].
And encode it to json. You can change my code and make output as proplist.
Related
I have a set of Strings and using it as key values to get JValues from a Map:
val keys: Set[String] = Set("Metric_1", "Metric_2", "Metric_3", "Metric_4")
val logData: Map[String, JValue] = Map("Metric_1" -> JInt(0), "Metric_2" -> JInt(1), "Metric_3" -> null)
In the below method I'm parsing values for each metric. First getting all values, then filtering to get rid of null values and then transforming existing values to booleans.
val metricsMap: Map[String, Boolean] = keys
.map(k => k -> logData(k).extractOpt[Int]).toMap
.filter(_._2.isDefined)
.collect {
case (str, Some(0)) => str -> false
case (str, Some(1)) => str -> true
}
I've faced a problem when one of the keys is not found in the logData Map. So I'm geting a java.util.NoSuchElementException: key not found: Metric_4.
Here I'm using extractOpt to extract a value from a JSON and don't need default values. So probably extractOrElse will not be helpful since I only need to get values for existing keys and skip non-existing keys.
What could be a correct approach to handle a case when a key is not present in the logData Map?
UPD: I've achieved the desired result by .map(k => k -> apiData.getOrElse(k, null).extractOpt[Int]).toMap. However still not sure that it's the best approach.
That the values are JSON is a red herring--it's the missing key that's throwing the exception. There's a method called get which retrieves a value from a map wrapped in an Option. If we use Ints as the values we have:
val logData = Map("Metric_1" -> 1, "Metric_2" -> 0, "Metric_3" -> null)
keys.flatMap(k => logData.get(k).map(k -> _)).toMap
> Map(Metric_1 -> 1, Metric_2 -> 0, Metric_3 -> null)
Using flatMap instead of map means unwrap the Some results and drop the Nones. Now, if we go back to your actual example, we have another layer and that flatMap will eliminate the Metric_3 -> null item:
keys.flatMap(k => logData.get(k).flatMap(_.extractOpt[Int]).map(k -> _)).toMap
You can also rewrite this using a for comprehension:
(for {
k <- keys
jv <- logData.get(k)
v <- jv.extractOpt[Int]
} yield k -> v).toMap
I used Success and Failure in place of the JSON values to avoid having to set up a shell with json4s to make an example:
val logData = Map("Metric_1" -> Success(1), "Metric_2" -> Success(0), "Metric_3" -> Failure(new RuntimeException()))
scala> for {
| k <- keys
| v <- logData.get(k)
| r <- v.toOption
| } yield k -> r
res2: scala.collection.immutable.Set[(String, Int)] = Set((Metric_1,1), (Metric_2,0))
I'm trying to send a dict to javascript via port for storing the value in localStorage, and retrieve it next time the Elm app starts via flag.
Below code snippets show the dict sent as well as the raw json value received through flag. The Json decoding fails showing the error message at the bottom.
The issue seems to be the extra backslashes (as in \"{\\"Left\\") contained in the raw flag value. Interestingly, console.log shows that the flag value passed by javascript is "dict1:{"Left":"fullHeightVerticalCenter","Right":"fullHeightVerticalCenter","_default":"fullHeightVerticalBottom"}"as intended, so the extra backslashes seem to be added by Elm, but I can't figure out why. Also, I'd be interested to find out a better way to achieve passing a dict to and from javascript.
import Json.Decode as JD
import Json.Encode as JE
dict1 = Dict.fromList[("_default", "fullHeightVerticalBottom")
, ("Left", "fullHeightVerticalCenter")
, ("Right", "fullHeightVerticalCenter")]
type alias FlagsJEValue =
{dict1: String}
port setStorage : FlagsJEValue -> Cmd msg
-- inside Update function Cmd
setStorage {dict1 = JE.encode 0 (dictEncoder JE.string model.dict1)}
dictEncoder enc dict =
Dict.toList dict
|> List.map (\(k,v) -> (k, enc v))
|> JE.object
--
type alias Flags =
{dict1: Dict String String}
flagsDecoder : Decoder Flags
flagsDecoder =
JD.succeed Flags
|> required "dict1" (JD.dict JD.string)
-- inside `init`
case JD.decodeValue MyDecoders.flagsDecoder raw_flags of
Err e ->
_ = Debug.log "raw flag value" (Debug.toString (JE.encode 2 raw_flags) )
_ = Debug.log "flags error msg" (Debug.toString e)
... omitted ...
Ok flags ->
... omitted ...
-- raw flag value
"{\n \"dict1\": \"{\\\"Left\\\":\\\"fullHeightVerticalCenter\\\",\\\"Right\\\":\\\"fullHeightVerticalCenter\\\",\\\"_default\\\":\\\"fullHeightVerticalBottom\\\"}\"\n}"
--flags error msg
"Failure \"Json.Decode.oneOf failed in the following 2 ways:\\n\\n\\n\\n
(1) Problem with the given value:\\n \\n \\\"{\\\\\\\"Left\\\\\\\":\\\\\\\"fullHeightVerticalCenter\\\\\\\",\\\\\\\"Right\\\\\\\":\\\\\\\"fullHeightVerticalCenter\\\\\\\",\\\\\\\"_default\\\\\\\":\\\\\\\"fullHeightVerticalBottom\\\\\\\"}\\\"\\n \\n Expecting an OBJECT\\n\\n\\n\\n
(2) Problem with the given value:\\n \\n \\\"{\\\\\\\"Left\\\\\\\":\\\\\\\"fullHeightVerticalCenter\\\\\\\",\\\\\\\"Right\\\\\\\":\\\\\\\"fullHeightVerticalCenter\\\\\\\",\\\\\\\"_default\\\\\\\":\\\\\\\"fullHeightVerticalBottom\\\\\\\"}\\\"\\n \\n Expecting null\" <internals>”
You don't need to use JE.encode there.
You can just use your dictEncoder to produce a Json.Encode.Value and pass that directly to setStorage.
The problem you're encountering it that you've encoded the dict to a json string (using JE.encode) and then sent that string over a port and the port has encoded that string as json again. You see extra slashes because the json string is double encoded.
I've been using the Newtonsoft.Json and Newtonsoft.Json.Fsharp libraries to create a new JSON serializer and stream to a file. I like the ability to stream to a file because I'm handling large files and, prior to streaming, often ran into memory issues.
I stream with a simple fx:
open Newtonsoft.Json
open Newtonsoft.Json.FSharp
open System.IO
let writeToJson (path: string) (obj: 'a) : unit =
let serialized = JsonConvert.SerializeObject(obj)
let fileStream = new StreamWriter(path)
let serializer = new JsonSerializer()
serializer.Serialize(fileStream, obj)
fileStream.Close()
This works great. My problem is that the JSON string is then absolutely cluttered with stuff I don't need. For example,
let m =
[
(1.0M, None)
(2.0M, Some 3.0M)
(4.0M, None)
]
let makeType (tup: decimal * decimal option) = {FieldA = fst tup; FieldB = snd tup}
let y = List.map makeType m
Default.serialize y
val it : string =
"[{"FieldA": 1.0},
{"FieldA": 2.0,
"FieldB": {
"Case": "Some",
"Fields": [3.0]
}},
{"FieldA": 4.0}]"
If this is written to a JSON and read into R, there are nested dataframes and any of the Fields associated with a Case end up being a list:
library(jsonlite)
library(dplyr)
q <- fromJSON("default.json")
x <-
q %>%
flatten()
x
> x
FieldA FieldB.Case FieldB.Fields
1 1 <NA> NULL
2 2 Some 3
3 4 <NA> NULL
> sapply(x, class)
FieldA FieldB.Case FieldB.Fields
"numeric" "character" "list"
I don't want to have to handle these things in R. I can do it but it's annoying and, if there are files with many, many columns, it's silly.
This morning, I started looking at the Microsoft.FSharpLu.Json documentation. This library has a Compact.serialize function. Quick tests suggest that this library will eliminate the need for nested dataframes and the lists associated with any Case and Field columns. For example:
Compact.serialize y
val it : string =
"[{
"FieldA": 1.0
},
{
"FieldA": 2.0,
"FieldB": 3.0
},
{
"FieldA": 4.0
}
]"
When this string is read into R,
q <- fromJSON("compact.json")
x <- q
x
> x
FieldA FieldB
1 1 NA
2 2 3
3 4 NA
> sapply(x, class)
FieldA FieldB
"numeric" "numeric
This is much simpler to handle in R. and I'd like to start using this library.
However, I don't know if I can get the Compact serializer to serialize to a stream. I see .serializeToFile, .desrializeStream, and .tryDeserializeStream, but nothing that can serialize to a stream. Does anyone know if Compact can handle writing to a stream? How can I make that work?
The helper to serialize to stream is missing from the Compact module in FSharpLu.Json, but you should be able to do it by following the C# example from
http://www.newtonsoft.com/json/help/html/SerializingJSON.htm. Something along the lines:
let writeToJson (path: string) (obj: 'a) : unit =
let serializer = new JsonSerializer()
serializer.Converters.Add(new Microsoft.FSharpLu.Json.CompactUnionJsonConverter())
use sw = new StreamWriter(path)
use writer = new JsonTextWriter(sw)
serializer.Serialize(writer, obj)
I am trying to get erlang-mysql-driver working, I managed to set it up and make queries but there are two things I cannot do.(https://code.google.com/archive/p/erlang-mysql-driver/issues)
(BTW, I am new to Erlang)
So Here is my code to connect MySQL.
<erl>
out(Arg) ->
mysql:start_link(p1, "127.0.0.1", "root", "azzkikr", "MyDB"),
{data, Result} = mysql:fetch(p1, "SELECT * FROM messages").
</erl>
1. I cannot get data from table.
mysql.erl doesn't contain any specific information on how to get table datas but this is the farthest I could go.
{A,B} = mysql:get_result_rows(Result),
B.
And the result was this:
ERROR erlang code threw an uncaught exception:
File: /Users/{username}/Sites/Yaws/index.yaws:1
Class: error
Exception: {badmatch,[[4,0,<<"This is done baby!">>,19238],
[5,0,<<"Success">>,19238],
[6,0,<<"Hello">>,19238]]}
Req: {http_request,'GET',{abs_path,"/"},{1,1}}
Stack: [{m181,out,1,
[{file,"/Users/{username}/.yaws/yaws/default/m181.erl"},
{line,18}]},
{yaws_server,deliver_dyn_part,8,
[{file,"yaws_server.erl"},{line,2818}]},
{yaws_server,aloop,4,[{file,"yaws_server.erl"},{line,1232}]},
{yaws_server,acceptor0,2,[{file,"yaws_server.erl"},{line,1068}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]
I understand that somehow I need to get second element and use foreach to get each data but strings are returned in different format like queried string is Success but returned string is <<"Success">>.
{badmatch,[[4,0,<<"This is done baby!">>,19238],
[5,0,<<"Success">>,19238],
[6,0,<<"Hello">>,19238]]}
First Question is: How do I get datas from table?
2. How to insert values into table using variables?
I can insert data into table using this method:
Msg = "Hello World",
mysql:prepare(add_message,<<"INSERT INTO messages (`message`) VALUES (?)">>),
mysql:execute(p1, add_message, [Msg]).
But there are two things I am having trouble,
1. I am inserting data without << and >> operators, because When I do Msg = << ++ "Hello World" >>, erlang throws out an exception (I think I am doing something wrong), i don't know wether they are required but without them I am able to insert data into table except this error bothers me after execution:
yaws code at /Users/{username}/Yaws/index.yaws:1 crashed or ret bad val:{updated,
{mysql_result,
[],
[],
1,
[]}}
Req: {http_request,'GET',{abs_path,"/"},{1,1}}
returned atom is updated while I commanded to insert data.
Question 2 is: How do I insert data into table in a proper way?
Error:
{badmatch,[[4,0,<<"This is done baby!">>,19238],
[5,0,<<"Success">>,19238],
[6,0,<<"Hello">>,19238]]}
Tells you that returned values is:
[[4,0,<<"This is done baby!">>,19238],
[5,0,<<"Success">>,19238],
[6,0,<<"Hello">>,19238]]
Which obviously can't match with either {data, Data} nor {A, B}. You can obtain your data as:
<erl>
out(Arg) ->
mysql:start_link(p1, "127.0.0.1", "root", "azzkikr", "MyDB"),
{ehtml,
[{table, [{border, "1"}],
[{tr, [],
[{td, [],
case Val of
_ when is_binary(Val) -> yaws_api:htmlize(Val);
_ when is_integer(val) -> integer_to_binary(Val)
end}
|| Val <- Row
]}
|| Row <- mysql:fetch(p1, "SELECT * FROM messages")
]}
]
}.
</erl>
I'm trying to find out how to convert an Erlang bitstring to a tuple, but so far without any luck.
What I want is to get from for example <<"{1,2}">> the tuple {1,2}.
You can use the modules erl_scan and erl_parse, as in this answer. Since erl_scan:string requires a string, not a binary, you have to convert the value with binary_to_list first:
> {ok, Scanned, _} = erl_scan:string(binary_to_list(<<"{1,2}">>)).
{ok,[{'{',1},{integer,1,1},{',',1},{integer,1,2},{'}',1}],1}
Then, you'd use erl_parse:parse_term to get the actual term. However, this function expects the term to end with a dot, so we have to add it explicitly:
> {ok, Parsed} = erl_parse:parse_term(Scanned ++ [{dot,0}]).
{ok,{1,2}}
Now the variable Parsed contains the result:
> Parsed.
{1,2}
You can use binary functions and erlang:list_to_tuple/1
1> B = <<"{1,2}">>.
<<"{1,2}">>
2> list_to_tuple([list_to_integer(binary_to_list(X)) || X <- binary:split(binary:part(B, 1, byte_size(B)-2), <<",">>, [global])]).
{1,2}