Chain http request and merge json response in ELM - json

I've succeeded in triggering a simple http request in ELM and decoding the JSON response into an ELM value - [https://stackoverflow.com/questions/43139316/decode-nested-variable-length-json-in-elm]
The problem I'm facing now-
How to chain (concurrency preferred) two http requests and merge the json into my new (updated) model. Note - please see the updated Commands.elm
Package used to access remote data - krisajenkins/remotedata http://package.elm-lang.org/packages/krisajenkins/remotedata/4.3.0/RemoteData
Github repo of my code - https://github.com/areai51/my-india-elm
Previous Working Code -
Models.elm
type alias Model =
{ leaders : WebData (List Leader)
}
initialModel : Model
initialModel =
{ leaders = RemoteData.Loading
}
Main.elm
init : ( Model, Cmd Msg )
init =
( initialModel, fetchLeaders )
Commands.elm
fetchLeaders : Cmd Msg
fetchLeaders =
Http.get fetchLeadersUrl leadersDecoder
|> RemoteData.sendRequest
|> Cmd.map Msgs.OnFetchLeaders
fetchLeadersUrl : String
fetchLeadersUrl =
"https://data.gov.in/node/85987/datastore/export/json"
Msgs.elm
type Msg
= OnFetchLeaders (WebData (List Leader))
Update.elm
update msg model =
case msg of
Msgs.OnFetchLeaders response ->
( { model | leaders = response }, Cmd.none )
Updated Code - (need help with Commands.elm)
Models.elm
type alias Model =
{ lsLeaders : WebData (List Leader)
, rsLeaders : WebData (List Leader) <------------- Updated Model
}
initialModel : Model
initialModel =
{ lsLeaders = RemoteData.Loading
, rsLeaders = RemoteData.Loading
}
Main.elm
init : ( Model, Cmd Msg )
init =
( initialModel, fetchLeaders )
Commands.elm
fetchLeaders : Cmd Msg
fetchLeaders = <-------- How do I call both requests here ? And fire separate msgs
Http.get fetchLSLeadersUrl lsLeadersDecoder <----- There will be a different decoder named rsLeadersDecoder
|> RemoteData.sendRequest
|> Cmd.map Msgs.OnFetchLSLeaders
fetchLSLeadersUrl : String
fetchLSLeadersUrl =
"https://data.gov.in/node/85987/datastore/export/json"
fetchRSLeadersUrl : String <------------------ New data source
fetchRSLeadersUrl =
"https://data.gov.in/node/982241/datastore/export/json"
Msgs.elm
type Msg
= OnFetchLSLeaders (WebData (List Leader))
| OnFetchRSLeaders (WebData (List Leader)) <-------- New message
Update.elm
update msg model =
case msg of
Msgs.OnFetchLSLeaders response ->
( { model | lsLeaders = response }, Cmd.none )
Msgs.OnFetchRSLeaders response -> <--------- New handler
( { model | rsLeaders = response }, Cmd.none )

The way to fire off two concurrent requests is to use Cmd.batch:
init : ( Model, Cmd Msg )
init =
( initialModel, Cmd.batch [ fetchLSLeaders, fetchRSLeaders ] )
There is no guarantee on which request will return first and there is no guarantee that they will both be successful. One could fail while the other succeeds, for example.
You mention that you want to merge the results, but you didn't say how the merge would work, so I'll just assume you want to append the lists of leaders together in one list, and it will be useful to your application if you had only to deal with a single RemoteData value rather than multiple.
You can merge multiple RemoteData values together with a custom function using map and andMap.
mergeLeaders : WebData (List Leader) -> WebData (List Leader) -> WebData (List Leader)
mergeLeaders a b =
RemoteData.map List.append a
|> RemoteData.andMap b
Notice that I'm using List.append there. That can really be any function that takes two lists and merges them however you please.
If you prefer an applicative style of programming, the above could be translated to the following infix version:
import RemoteData.Infix exposing (..)
mergeLeaders2 : WebData (List Leader) -> WebData (List Leader) -> WebData (List Leader)
mergeLeaders2 a b =
List.append <$> a <*> b
According to the documentation on andMap (which uses a result tuple rather than an appended list in its example):
The final tuple succeeds only if all its children succeeded. It is still Loading if any of its children are still Loading. And if any child fails, the error is the leftmost Failure value.

Related

Cannot update or delete many in ReactiveMongo 0.16

I'm having some difficulty with delete.many and update.many using the new builders whilst trying to convert my previous version's (working) code into reactivemongo 0.16.5 ("org.reactivemongo" %% "play2-reactivemongo" % "0.16.5-play26", "org.reactivemongo" %% "reactivemongo-akkastream" % "0.16.5". As you'll see; I'm using this within the Play plugin so dealing with JSON (rather than BSON)
I'm going from the official documentation here. My errors are similar for both update & delete so I'll just post for update here to keep it trim.
Update command
def updateMany(collName: String)(quJsa: JsArray)(orderedBool: Boolean = false): Future[MultiBulkWriteResult] = {
lazy val updateBuilder = getCollection(collName).map(_.update(orderedBool))
quJsa.asOpt[Seq[JsObject]].map(
_.map(
x => x.as[MongoUpdateBuilder]
)
).map(
_.map(
x => updateBuilder.flatMap(
_.element(q = x.q, u = x.getSetUpdate, upsert = x.upsertBool, multi = x.multiBool)
)
)
).map(
x => Future.sequence(x)
).map(
_.flatMap(
x => updateBuilder.flatMap(
_.many(x)
)
)
).getOrElse(getMultiBulkWriteResultErrorF("input not recognised as jsarr"))
}
Custom update builder model
case class MongoUpdateBuilder(q: JsObject, u: JsObject, upsertBool: Boolean, multiBool: Boolean) {
def getSetUpdate = Json.obj("$set" -> u)
}
object MongoUpdateBuilder {
implicit val mongoUpdateBuilderFormat = Json.format[MongoUpdateBuilder]
}
Error container
def getMultiBulkWriteResultErrorF(errStr: String): Future[MultiBulkWriteResult] = {
val mbwr = MultiBulkWriteResult(
ok = false,
n = 0,
nModified = 0,
upserted = Seq(),
writeErrors = Seq(WriteError(index = 0, code = 404, errmsg = errStr)),
writeConcernError = None,
code = Some(404),
errmsg = Some(errStr),
totalN = 0
)
Future.successful(mbwr)
}
And the main issue:
no type parameters for method flatMap: (f: reactivemongo.play.json.collection
.JSONCollection#UpdateBuilder => scala.concurrent.Future[S])(implicit executor: scala.concurrent.ExecutionContext)scala.concurrent.Future[S] exist so that it can be applied to arguments (reactivemongo.play.json.collection.JSONCollection#UpdateBuilder => scala.concurrent.Future[_1.UpdateCommand.UpdateElement] forSome { val _1: reactivemongo.play.json.collection.J
SONCollection }) [error] --- because ---
[error] argument expression's type is not compatible with formal parameter type;
[error] found : reactivemongo.play.json.collection.JSONCollection#UpdateBuilder => scala.concurrent.Future[_1.UpdateCommand.UpdateElement] forSome { val _1: reactivemongo.play.jso
n.collection.JSONCollection }
[error] required: reactivemongo.play.json.collection.JSONCollection#UpdateBuilder => scala.concurrent.Future[?S]
[error] x => updateBuilder.flatMap(
So the issue seems to be this line - updateBuilder.flatMap. The Future cannot be flattened with these types (JSONCollection#UpdateBuilder & JSONCollection#UpdateCommand.UpdateElement). So I'm struggling with this one. Please reach out if you can see the issue here. Many thanks!

Scala - How to handle key not found in a Map when need to skip non-existing keys without defaults?

I have a set of Strings and using it as key values to get JValues from a Map:
val keys: Set[String] = Set("Metric_1", "Metric_2", "Metric_3", "Metric_4")
val logData: Map[String, JValue] = Map("Metric_1" -> JInt(0), "Metric_2" -> JInt(1), "Metric_3" -> null)
In the below method I'm parsing values for each metric. First getting all values, then filtering to get rid of null values and then transforming existing values to booleans.
val metricsMap: Map[String, Boolean] = keys
.map(k => k -> logData(k).extractOpt[Int]).toMap
.filter(_._2.isDefined)
.collect {
case (str, Some(0)) => str -> false
case (str, Some(1)) => str -> true
}
I've faced a problem when one of the keys is not found in the logData Map. So I'm geting a java.util.NoSuchElementException: key not found: Metric_4.
Here I'm using extractOpt to extract a value from a JSON and don't need default values. So probably extractOrElse will not be helpful since I only need to get values for existing keys and skip non-existing keys.
What could be a correct approach to handle a case when a key is not present in the logData Map?
UPD: I've achieved the desired result by .map(k => k -> apiData.getOrElse(k, null).extractOpt[Int]).toMap. However still not sure that it's the best approach.
That the values are JSON is a red herring--it's the missing key that's throwing the exception. There's a method called get which retrieves a value from a map wrapped in an Option. If we use Ints as the values we have:
val logData = Map("Metric_1" -> 1, "Metric_2" -> 0, "Metric_3" -> null)
keys.flatMap(k => logData.get(k).map(k -> _)).toMap
> Map(Metric_1 -> 1, Metric_2 -> 0, Metric_3 -> null)
Using flatMap instead of map means unwrap the Some results and drop the Nones. Now, if we go back to your actual example, we have another layer and that flatMap will eliminate the Metric_3 -> null item:
keys.flatMap(k => logData.get(k).flatMap(_.extractOpt[Int]).map(k -> _)).toMap
You can also rewrite this using a for comprehension:
(for {
k <- keys
jv <- logData.get(k)
v <- jv.extractOpt[Int]
} yield k -> v).toMap
I used Success and Failure in place of the JSON values to avoid having to set up a shell with json4s to make an example:
val logData = Map("Metric_1" -> Success(1), "Metric_2" -> Success(0), "Metric_3" -> Failure(new RuntimeException()))
scala> for {
| k <- keys
| v <- logData.get(k)
| r <- v.toOption
| } yield k -> r
res2: scala.collection.immutable.Set[(String, Int)] = Set((Metric_1,1), (Metric_2,0))

Using Microsoft.FSharpLu to serialize JSON to a stream

I've been using the Newtonsoft.Json and Newtonsoft.Json.Fsharp libraries to create a new JSON serializer and stream to a file. I like the ability to stream to a file because I'm handling large files and, prior to streaming, often ran into memory issues.
I stream with a simple fx:
open Newtonsoft.Json
open Newtonsoft.Json.FSharp
open System.IO
let writeToJson (path: string) (obj: 'a) : unit =
let serialized = JsonConvert.SerializeObject(obj)
let fileStream = new StreamWriter(path)
let serializer = new JsonSerializer()
serializer.Serialize(fileStream, obj)
fileStream.Close()
This works great. My problem is that the JSON string is then absolutely cluttered with stuff I don't need. For example,
let m =
[
(1.0M, None)
(2.0M, Some 3.0M)
(4.0M, None)
]
let makeType (tup: decimal * decimal option) = {FieldA = fst tup; FieldB = snd tup}
let y = List.map makeType m
Default.serialize y
val it : string =
"[{"FieldA": 1.0},
{"FieldA": 2.0,
"FieldB": {
"Case": "Some",
"Fields": [3.0]
}},
{"FieldA": 4.0}]"
If this is written to a JSON and read into R, there are nested dataframes and any of the Fields associated with a Case end up being a list:
library(jsonlite)
library(dplyr)
q <- fromJSON("default.json")
x <-
q %>%
flatten()
x
> x
FieldA FieldB.Case FieldB.Fields
1 1 <NA> NULL
2 2 Some 3
3 4 <NA> NULL
> sapply(x, class)
FieldA FieldB.Case FieldB.Fields
"numeric" "character" "list"
I don't want to have to handle these things in R. I can do it but it's annoying and, if there are files with many, many columns, it's silly.
This morning, I started looking at the Microsoft.FSharpLu.Json documentation. This library has a Compact.serialize function. Quick tests suggest that this library will eliminate the need for nested dataframes and the lists associated with any Case and Field columns. For example:
Compact.serialize y
val it : string =
"[{
"FieldA": 1.0
},
{
"FieldA": 2.0,
"FieldB": 3.0
},
{
"FieldA": 4.0
}
]"
When this string is read into R,
q <- fromJSON("compact.json")
x <- q
x
> x
FieldA FieldB
1 1 NA
2 2 3
3 4 NA
> sapply(x, class)
FieldA FieldB
"numeric" "numeric
This is much simpler to handle in R. and I'd like to start using this library.
However, I don't know if I can get the Compact serializer to serialize to a stream. I see .serializeToFile, .desrializeStream, and .tryDeserializeStream, but nothing that can serialize to a stream. Does anyone know if Compact can handle writing to a stream? How can I make that work?
The helper to serialize to stream is missing from the Compact module in FSharpLu.Json, but you should be able to do it by following the C# example from
http://www.newtonsoft.com/json/help/html/SerializingJSON.htm. Something along the lines:
let writeToJson (path: string) (obj: 'a) : unit =
let serializer = new JsonSerializer()
serializer.Converters.Add(new Microsoft.FSharpLu.Json.CompactUnionJsonConverter())
use sw = new StreamWriter(path)
use writer = new JsonTextWriter(sw)
serializer.Serialize(writer, obj)

Websharper, Sitelets and Forms

I've been trying to create a form using Websharper to collect user input. So far I've identified three actions for my site:
type MyAction =
| [<CompiledName "">] Index
| [<Method "POST">] GetUser of username : string
| Stats of username: string
Using Sitelet.Infer I've managed to implement basic UI, but I have no idea how to refer to the content of my input box (usernameInput):
Sitelet.Infer <| function
| Index ->
Content.PageContent <| fun ctx ->
let usernameInput= Input [Text ""]
{ Page.Default with
Title = Some "Welcome!"
Body =
[
Div [
Form
[
usernameInput-< [Name "username" ]
Input [Value "Request"] -< [Type "submit" ]
] -< [ Attr.Action (ctx.Link (* GetUser usernameInput.Content *) ); Method "POST" ]
]
]
}
| GetUser username ->
Content.Redirect <| Stats username
| Stats username ->
Content.PageContent <| fun ctx ->
{ Page.Default with
Body = [Text ("Stats for " + username)] }
I noticed usernameInput doesn't have any field like "Value" or so and I guess either it needs casting or I'm doing something wrong.
I would prefer not to use JavaScript in my code (Is it possible to mix Html.Server and Html.Client Elements in a Sitelet at all ?).
Form POST data is not passed via the URL, so you cannot pass it with ctx.Link. It is automatically passed via the request body, with a format similar to GET query arguments (for example in your case, username=myusername). This is currently not parsed by Sitelet.Infer, although we will probably add it in the future. For now you can use an action without arguments and then extract the data from the request:
type MyAction =
| [<Method "POST">] GetUser
| // ...
Sitelet.Infer <| function
| GetUser ->
Content.CustomContentAsync <| fun ctx ->
match ctx.Request.Post.["username"] with
| None -> Content.NotFound
| Some username -> Content.Redirect <| Stats username
|> Content.ToResponseAsync ctx
| // ...

error in json in corona lua

hi i have found a tutorial on how to use post json in lua.
here is the code :
http = require("socket.http")
crypto = require("crypto")
ltn12 = require("ltn12")
url = require("socket.url")
local json = require("json")
local commands_json =
{
["message"] = "Hello",
}
print (commands_json)
local json = {}
json.api_key = "6_192116334"
json.ver = 1
json.commands_json = json.encode(commands_json)
json.commands_hash = crypto.digest(crypto.md5, json.commands_json .. 'hkjhkjhkjh')
local post = "api=" .. url.escape(Json.Encode(json))
local response = {}
local r, c, h = http.request {
url = "http://127.0.0.1/?page=api",
method = "POST",
headers = {
["content-length"] = #post,
["Content-Type"] = "application/x-www-form-urlencoded"
},
source = ltn12.source.string(post),
sink = ltn12.sink.table(response)
}
local path = system.pathForFile("r.txt", system.DocumentsDirectory)
local file = io.open (path, "w")
file:write (Json.Encode(json) .. "\n")
file:write (post .. "\n")
file:write (response[1] .. "\n")
io.close (file)
json = Json.Decode(table.concat(response,''))
native.showAlert("hey", json.commands[1].tot_nbr_rows)
now i got these error:
Windows simulator build date: Dec 9 2011 # 14:01:29
Copyright (C) 2009-2011 A n s c a , I n c .
Version: 2.0.0
Build: 2011.704
table: 0346D6D0
Runtime error
...nistrator\my documents\corona projects\json\main.lua:17: attempt to c
all field 'encode' (a nil value)
stack traceback:
[C]: in function 'encode'
...nistrator\my documents\corona projects\json\main.lua:17: in main chun
k
Runtime error: ...nistrator\my documents\corona projects\json\main.lua:17: attem
pt to call field 'encode' (a nil value)
stack traceback:
[C]: in function 'encode'
...nistrator\my documents\corona projects\json\main.lua:17: in main chun
k
i don't know why i got the error from encode.
can anyone can help me about my case?
thanks in advance ...
This includes the Json code provided externally, likely with an encode function:
local json = require("json")
This throws away your old json variable and replaces it with an empty table:
local json = {}
And this tries to call json.encode which is now undefined since you redefined json as an empty table above:
json.commands_json = json.encode(commands_json)
The solution is to pick a different variable name.