Cannot update or delete many in ReactiveMongo 0.16 - json

I'm having some difficulty with delete.many and update.many using the new builders whilst trying to convert my previous version's (working) code into reactivemongo 0.16.5 ("org.reactivemongo" %% "play2-reactivemongo" % "0.16.5-play26", "org.reactivemongo" %% "reactivemongo-akkastream" % "0.16.5". As you'll see; I'm using this within the Play plugin so dealing with JSON (rather than BSON)
I'm going from the official documentation here. My errors are similar for both update & delete so I'll just post for update here to keep it trim.
Update command
def updateMany(collName: String)(quJsa: JsArray)(orderedBool: Boolean = false): Future[MultiBulkWriteResult] = {
lazy val updateBuilder = getCollection(collName).map(_.update(orderedBool))
quJsa.asOpt[Seq[JsObject]].map(
_.map(
x => x.as[MongoUpdateBuilder]
)
).map(
_.map(
x => updateBuilder.flatMap(
_.element(q = x.q, u = x.getSetUpdate, upsert = x.upsertBool, multi = x.multiBool)
)
)
).map(
x => Future.sequence(x)
).map(
_.flatMap(
x => updateBuilder.flatMap(
_.many(x)
)
)
).getOrElse(getMultiBulkWriteResultErrorF("input not recognised as jsarr"))
}
Custom update builder model
case class MongoUpdateBuilder(q: JsObject, u: JsObject, upsertBool: Boolean, multiBool: Boolean) {
def getSetUpdate = Json.obj("$set" -> u)
}
object MongoUpdateBuilder {
implicit val mongoUpdateBuilderFormat = Json.format[MongoUpdateBuilder]
}
Error container
def getMultiBulkWriteResultErrorF(errStr: String): Future[MultiBulkWriteResult] = {
val mbwr = MultiBulkWriteResult(
ok = false,
n = 0,
nModified = 0,
upserted = Seq(),
writeErrors = Seq(WriteError(index = 0, code = 404, errmsg = errStr)),
writeConcernError = None,
code = Some(404),
errmsg = Some(errStr),
totalN = 0
)
Future.successful(mbwr)
}
And the main issue:
no type parameters for method flatMap: (f: reactivemongo.play.json.collection
.JSONCollection#UpdateBuilder => scala.concurrent.Future[S])(implicit executor: scala.concurrent.ExecutionContext)scala.concurrent.Future[S] exist so that it can be applied to arguments (reactivemongo.play.json.collection.JSONCollection#UpdateBuilder => scala.concurrent.Future[_1.UpdateCommand.UpdateElement] forSome { val _1: reactivemongo.play.json.collection.J
SONCollection }) [error] --- because ---
[error] argument expression's type is not compatible with formal parameter type;
[error] found : reactivemongo.play.json.collection.JSONCollection#UpdateBuilder => scala.concurrent.Future[_1.UpdateCommand.UpdateElement] forSome { val _1: reactivemongo.play.jso
n.collection.JSONCollection }
[error] required: reactivemongo.play.json.collection.JSONCollection#UpdateBuilder => scala.concurrent.Future[?S]
[error] x => updateBuilder.flatMap(
So the issue seems to be this line - updateBuilder.flatMap. The Future cannot be flattened with these types (JSONCollection#UpdateBuilder & JSONCollection#UpdateCommand.UpdateElement). So I'm struggling with this one. Please reach out if you can see the issue here. Many thanks!

Related

How do I make an individual Rocket tile asynchronous to the rest of the system

I have a multicore rocket-chip system. However I'd like one of those rocket tiles to be asynchronous from the rest.
We're trying to do that with the following:
class WithTilesCrossing extends Config((site, here, up) => {
case RocketCrossingKey => site(RocketTilesKey).head.hartId match {
case 2 => up(RocketCrossingKey) map { r =>
r.copy(crossingType = AsynchronousCrossing(),
master = TileMasterPortParams())}
case _ => up(RocketCrossingKey) map { r =>
r.copy(crossingType = SynchronousCrossing(BufferParams(1)),
master = TileMasterPortParams())}
}
})
So that is the hart with hartId = 2 should be async, the rest should be synchronous.
The above, when added to our config, doesn't appear to do anything.
However, if I use the WithAsynchronousRocketTiles from src/main/scala/subsystem/Config.scala then I get all of the tiles converted to async.
So, how would I do just a single tile?
Update based on Jack's suggestion:
Trying that code straight up gave:
[error] Config.scala:189:16: not found: value crossingType
[error] r.copy(crossingType = AsynchronousCrossing(),
[error] ^
[error] Config.scala:190:11: not found: value master
[error] master = TileMasterPortParams())
[error] ^
[error] Config.scala:192:16: not found: value crossingType
[error] r.copy(crossingType = SynchronousCrossing(BufferParams(1)),
[error] ^
[error] Config.scala:193:11: not found: value master
[error] master = TileMasterPortParams())
[error] ^
[error] four errors found
Which is surprising. So I thought I might need to do the up() thing and tried this:
case RocketCrossingKey => site(RocketTilesKey).map { r =>
r.hartId match {
case 2 => up(RocketCrossingKey) map { k => k.copy(crossingType = AsynchronousCrossing(), master = TileMasterPortParams()) }
case _ => up(RocketCrossingKey) map { k => k.copy(crossingType = SynchronousCrossing(BufferParams(1)), master = TileMasterPortParams()) }
}
}
However that results in an elab error:
[error] Caused by: java.lang.ClassCastException: scala.collection.immutable.$colon$colon cannot be cast to freechips.rocketchip.subsystem.RocketCrossingParams
[error] at freechips.rocketchip.subsystem.HasRocketTiles.$anonfun$rocketTiles$1(RocketSubsystem.scala:41)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
[error] at scala.collection.immutable.List.foreach(List.scala:389)
[error] at scala.collection.TraversableLike.map(TraversableLike.scala:234)
[error] at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
[error] at scala.collection.immutable.List.map(List.scala:295)
[error] at freechips.rocketchip.subsystem.HasRocketTiles.$init$(RocketSubsystem.scala:41)
[error] at freechips.rocketchip.subsystem.RocketSubsystem.<init>(RocketSubsystem.scala:70)
So still stuck on how to modify this original RocketCrossingParams on a per-tile basis and return it.
Looking at the definition of RocketCrossingKey:
case object RocketCrossingKey extends Field[Seq[RocketCrossingParams]](List(RocketCrossingParams()))
You'll notice that the type is Seq[RocketCrossingParams]. This implies (although I may be wrong), that we have 1 RocketCrossingParams per tile. In your code snippet, you are only looking at the first of the this Seq (via .head) checking if it's hartId is equal to 2, and then if so, iterating on all of the RocketCrossingKeys and setting them to AsynchronousCrossing.
Try something like the following where we iterate on them and only replace the index for the one with hartId == 2:
case RocketCrossingKey => site(RocketTilesKey).map { r =>
if (r.hartId == 2) { // or you can match on it, whatever
r.copy(crossingType = AsynchronousCrossing(),
master = TileMasterPortParams())
} else {
r.copy(crossingType = SynchronousCrossing(BufferParams(1)),
master = TileMasterPortParams())
}
}
EDIT: I missed the fact that there is both RocketCrossingKey and RocketTilesKey
So the issue here is that there are two parallel Seqs of parameters:
RocketTilesKey which gives us RocketTileParams, 1 for each tile
RocketCrossingKey which gives us RocketCrossingParams, 1 for each tile OR if there's only 1, it applies to all
It's also possible that there is no RocketTileParams containing hartId == 2, so let's handle everything appropriately:
case RocketCrossingKey =>
val tileParams = site(RocketTilesKey)
val crossingParams = site(RocketCrossingKey)
// One might assume hartId 2 == index 2 but that may not be the case
// Also there may not even *be* a tile with hartId == 2
val indexOfHartId2: Option[Int] =
tileParams.zipWithIndex.collectFirst { case (r, idx) if r.hartId == 2 => idx }
indexOfHartId2.map { idx =>
// This duplicates logic from HasTiles.perTileOrGlobalSetting
// If there's only 1, it applies to all
val crossings = site(RocketCrossingKey) match {
case Seq(one) => List.fill(tileParams.size)(one)
case many => many
}
// Back to the original answer using the proper index for hartId == 2
crossings.zipWithIndex.map { case (c, i) =>
if (i == idx) { // or you can match on it, whatever
c.copy(crossingType = AsynchronousCrossing(),
master = TileMasterPortParams())
} else {
c.copy(crossingType = SynchronousCrossing(BufferParams(1)),
master = TileMasterPortParams())
}
}
}.getOrElse(crossingParams) // If we don't even have hardId == 2, return original value

Chain http request and merge json response in ELM

I've succeeded in triggering a simple http request in ELM and decoding the JSON response into an ELM value - [https://stackoverflow.com/questions/43139316/decode-nested-variable-length-json-in-elm]
The problem I'm facing now-
How to chain (concurrency preferred) two http requests and merge the json into my new (updated) model. Note - please see the updated Commands.elm
Package used to access remote data - krisajenkins/remotedata http://package.elm-lang.org/packages/krisajenkins/remotedata/4.3.0/RemoteData
Github repo of my code - https://github.com/areai51/my-india-elm
Previous Working Code -
Models.elm
type alias Model =
{ leaders : WebData (List Leader)
}
initialModel : Model
initialModel =
{ leaders = RemoteData.Loading
}
Main.elm
init : ( Model, Cmd Msg )
init =
( initialModel, fetchLeaders )
Commands.elm
fetchLeaders : Cmd Msg
fetchLeaders =
Http.get fetchLeadersUrl leadersDecoder
|> RemoteData.sendRequest
|> Cmd.map Msgs.OnFetchLeaders
fetchLeadersUrl : String
fetchLeadersUrl =
"https://data.gov.in/node/85987/datastore/export/json"
Msgs.elm
type Msg
= OnFetchLeaders (WebData (List Leader))
Update.elm
update msg model =
case msg of
Msgs.OnFetchLeaders response ->
( { model | leaders = response }, Cmd.none )
Updated Code - (need help with Commands.elm)
Models.elm
type alias Model =
{ lsLeaders : WebData (List Leader)
, rsLeaders : WebData (List Leader) <------------- Updated Model
}
initialModel : Model
initialModel =
{ lsLeaders = RemoteData.Loading
, rsLeaders = RemoteData.Loading
}
Main.elm
init : ( Model, Cmd Msg )
init =
( initialModel, fetchLeaders )
Commands.elm
fetchLeaders : Cmd Msg
fetchLeaders = <-------- How do I call both requests here ? And fire separate msgs
Http.get fetchLSLeadersUrl lsLeadersDecoder <----- There will be a different decoder named rsLeadersDecoder
|> RemoteData.sendRequest
|> Cmd.map Msgs.OnFetchLSLeaders
fetchLSLeadersUrl : String
fetchLSLeadersUrl =
"https://data.gov.in/node/85987/datastore/export/json"
fetchRSLeadersUrl : String <------------------ New data source
fetchRSLeadersUrl =
"https://data.gov.in/node/982241/datastore/export/json"
Msgs.elm
type Msg
= OnFetchLSLeaders (WebData (List Leader))
| OnFetchRSLeaders (WebData (List Leader)) <-------- New message
Update.elm
update msg model =
case msg of
Msgs.OnFetchLSLeaders response ->
( { model | lsLeaders = response }, Cmd.none )
Msgs.OnFetchRSLeaders response -> <--------- New handler
( { model | rsLeaders = response }, Cmd.none )
The way to fire off two concurrent requests is to use Cmd.batch:
init : ( Model, Cmd Msg )
init =
( initialModel, Cmd.batch [ fetchLSLeaders, fetchRSLeaders ] )
There is no guarantee on which request will return first and there is no guarantee that they will both be successful. One could fail while the other succeeds, for example.
You mention that you want to merge the results, but you didn't say how the merge would work, so I'll just assume you want to append the lists of leaders together in one list, and it will be useful to your application if you had only to deal with a single RemoteData value rather than multiple.
You can merge multiple RemoteData values together with a custom function using map and andMap.
mergeLeaders : WebData (List Leader) -> WebData (List Leader) -> WebData (List Leader)
mergeLeaders a b =
RemoteData.map List.append a
|> RemoteData.andMap b
Notice that I'm using List.append there. That can really be any function that takes two lists and merges them however you please.
If you prefer an applicative style of programming, the above could be translated to the following infix version:
import RemoteData.Infix exposing (..)
mergeLeaders2 : WebData (List Leader) -> WebData (List Leader) -> WebData (List Leader)
mergeLeaders2 a b =
List.append <$> a <*> b
According to the documentation on andMap (which uses a result tuple rather than an appended list in its example):
The final tuple succeeds only if all its children succeeded. It is still Loading if any of its children are still Loading. And if any child fails, the error is the leftmost Failure value.

Mapping a sequence of results from Slick monadic join to Json

I'm using Play 2.4 with Slick 3.1.x, specifically the Slick-Play plugin v1.1.1. Firstly, some context... I have the following search/filter method in a DAO, which joins together 4 models:
def search(
departureCity: Option[String],
arrivalCity: Option[String],
departureDate: Option[Date]
) = {
val monadicJoin = for {
sf <- slickScheduledFlights.filter(a =>
departureDate.map(d => a.date === d).getOrElse(slick.lifted.LiteralColumn(true))
)
fl <- slickFlights if sf.flightId === fl.id
al <- slickAirlines if fl.airlineId === al.id
da <- slickAirports.filter(a =>
fl.departureAirportId === a.id &&
departureCity.map(c => a.cityCode === c).getOrElse(slick.lifted.LiteralColumn(true))
)
aa <- slickAirports.filter(a =>
fl.arrivalAirportId === a.id &&
arrivalCity.map(c => a.cityCode === c).getOrElse(slick.lifted.LiteralColumn(true))
)
} yield (fl, sf, al, da, aa)
db.run(monadicJoin.result)
}
The output from this is a Vector containing sequences, e.g:
Vector(
(
Flight(Some(1),123,216,2013,3,1455,2540,3,905,500,1150),
ScheduledFlight(Some(1),1,2016-04-13,90,10),
Airline(Some(216),BA,BAW,British Airways,United Kingdom),
Airport(Some(2013),LHR,Heathrow,LON,...),
Airport(Some(2540),JFK,John F Kennedy Intl,NYC...)
),
(
etc ...
)
)
I'm currently rendering the JSON in the controller by calling .toJson on a Map and inserting this Vector (the results param below), like so:
flightService.search(departureCity, arrivalCity, departureDate).map(results => {
Ok(
Map[String, Any](
"status" -> "OK",
"data" -> results
).toJson
).as("application/json")
})
While this sort of works, it produces output in an unusual format; an array of results (the rows) within each result object the joins are nested inside objects with keys: "_1", "_2" and so on.
So the question is: How should I go about restructuring this?
There doesn't appear to be anything which specifically covers this sort of scenario in the Slick docs. Therefore I would be grateful for some input on what might be the best way to refactor this Vector of Seq's, with a view to renaming each of the joins or even flattening it out and only keeping certain fields?
Is this best done in the DAO search method before it's returned (by mapping it somehow?) or in the controller after I get back the Future results Vector from the search method?
Or I'm wondering whether it would be preferable to abstract this sort of mutation out somewhere else entirely, using a transformer perhaps?
You need JSON Reads/Writes/Format Combinators
In the first place you must have Writes[T] for all your classes (Flight, ScheduledFlight, Airline, Airport).
Simple way is using Json macros
implicit val flightWrites: Writes[Flight] = Json.writes[Flight]
implicit val scheduledFlightWrites: Writes[ScheduledFlight] = Json.writes[ScheduledFlight]
implicit val airlineWrites: Writes[Airline] = Json.writes[Airline]
implicit val airportWrites: Writes[Airport] = Json.writes[Airport]
You must implement OWrites[(Flight, ScheduledFlight, Airline, Airport, Airport)] for Vector item also. For example:
val itemWrites: OWrites[(Flight, ScheduledFlight, Airline, Airport, Airport)] = (
(__ \ "flight").write[Flight] and
(__ \ "scheduledFlight").write[ScheduledFlight] and
(__ \ "airline").write[Airline] and
(__ \ "airport1").write[Airport] and
(__ \ "airport2").write[Airport]
).tupled
for writing whole Vector as JsAray use Writes.seq[T]
val resultWrites: Writes[Seq[(Flight, ScheduledFlight, Airline, Airport, Airport)]] = Writes.seq(itemWrites)
We have all to response your data
flightService.search(departureCity, arrivalCity, departureDate).map(results =>
Ok(
Json.obj(
"status" -> "Ok",
"data" -> resultWrites.writes(results)
)
)

How to send Json from client with missing fields for its corresponding Case Class after using Json.format function

I have a case Class and its companion object like below. Now, when I send JSON without id, createdAt and deletedAt fields, because I set them elsewhere, I get [NoSuchElementException: JsError.get] error. It's because I do not set above properties.
How could I achieve this and avoid getting the error?
case class Plan(id: String,
companyId: String,
name: String,
status: Boolean = true,
#EnumAs planType: PlanType.Value,
brochureId: Option[UUID],
lifePolicy: Seq[LifePolicy] = Nil,
createdAt: DateTime,
updatedAt: DateTime,
deletedAt: Option[DateTime]
)
object Plan {
implicit val planFormat = Json.format[Plan]
def fromJson(str: JsValue): Plan = Json.fromJson[Plan](str).get
def toJson(plan: Plan): JsValue = Json.toJson(plan)
def toJsonSeq(plan: Seq[Plan]): JsValue = Json.toJson(plan)
}
JSON I send from client
{
"companyId": "e8c67345-7f59-466d-a958-7c722ad0dcb7",
"name": "Creating First Plan with enum Content",
"status": true,
"planType": "Health",
"lifePolicy": []
}
You can introduce another case class just to handle serialization from request:
like this
case class NewPlan(name: String,
status: Boolean = true,
#EnumAs planType: PlanType.Value,
brochureId: Option[UUID],
lifePolicy: Seq[LifePolicy] = Nil
)
and then use this class to populate your Plan class.
The fundamental issue is that by the time a case class is instantiated to represent your data, it must be well-typed. To shoe horn your example data into your example class, the types don't match because some fields are missing. It's literally trying to call the constructor without enough arguments.
You've got a couple options:
You can make a model that represents the incomplete data (as grotrianster suggested).
You can make the possible missing fields Option types.
You can custom-write the Reads part of your Format to introduce intelligent values or dummy values for the missing ones.
Option 3 might look something like:
// Untested for compilation, might need some corrections
val now: DateTime = ...
val autoId = Reads[JsObject] {
case obj: JsObject => JsSuccess(obj \ 'id match {
case JsString(_) => obj
case _ => obj.transform(
__.update((__ \ 'id).json.put("")) andThen
__.update((__ \ 'createdTime).json.put(now)) andThen
__.update((__ \ 'updatedTime).json.put(now))
)
})
case _ => JsError("JsObject expected")
}
implicit val planFormat = Format[Plan](
autoId andThen Json.reads[Plan],
Json.writes[Plan])
Once you do this once, if the issue is the same for all your other models, you can probably abstract it into some Format factory utility function.
This may be slightly cleaner for autoId:
val autoId = Reads[JsObject] {
// Leave it alone if we have an ID already
case obj: JsObject if (obj \ 'id).asOpt[String].isSome => JsSuccess(obj)
// Insert dummy values if we don't have an `id`
case obj: JsObject => JsSuccess(obj.transform(
__.update((__ \ 'id).json.put("")) andThen
__.update((__ \ 'createdTime).json.put(now)) andThen
__.update((__ \ 'updatedTime).json.put(now))
))
case _ => JsError("JsObject expected")
}

MongoDB: How to deal with invalid ObjectIDs

Here below is my code to find a document by ObjectID:
def find(selector: JsValue, projection: Option[JsValue], sort: Option[JsValue],
page: Int, perPage: Int): Future[Seq[JsValue]] = {
var query = collection.genericQueryBuilder.query(selector).options(
QueryOpts(skipN = page * perPage)
)
projection.map(value => query = query.projection(value))
sort.map(value => query = query.sort(value.as[JsObject]))
// this is the line where the call crashes
query.cursor[JsValue].collect[Vector](perPage).transform(
success => success,
failure => failure match {
case e: LastError => DaoException(e.message, Some(DATABASE_ERROR))
}
)
}
Now let's suppose we invoke this method with an invalid ObjectID:
// ObjectId 53125e9c2004006d04b605abK is invalid (ends with a K)
find(Json.obj("_id" -> Json.obj("$oid" -> "53125e9c2004006d04b605abK")), None, None, 0, 25)
The call above causes the following exception when executing query.cursor[JsValue].collect[Vector](perPage) in the find method:
Caused by: java.util.NoSuchElementException: JsError.get
at play.api.libs.json.JsError.get(JsResult.scala:11) ~[play-json_2.10.jar:2.2.1]
at play.api.libs.json.JsError.get(JsResult.scala:10) ~[play-json_2.10.jar:2.2.1]
at play.modules.reactivemongo.json.collection.JSONGenericHandlers$StructureBufferWriter$.write(jsoncollection.scala:44) ~[play2-reactivemongo_2.10-0.10.2.jar:0.10.2]
at play.modules.reactivemongo.json.collection.JSONGenericHandlers$StructureBufferWriter$.write(jsoncollection.scala:42) ~[play2-reactivemongo_2.10-0.10.2.jar:0.10.2]
at reactivemongo.api.collections.GenericQueryBuilder$class.reactivemongo$api$collections$GenericQueryBuilder$$write(genericcollection.scala:323) ~[reactivemongo_2.10-0.10.0.jar:0.10.0]
at reactivemongo.api.collections.GenericQueryBuilder$class.cursor(genericcollection.scala:342) ~[reactivemongo_2.10-0.10.0.jar:0.10.0]
at play.modules.reactivemongo.json.collection.JSONQueryBuilder.cursor(jsoncollection.scala:110) ~[play2-reactivemongo_2.10-0.10.2.jar:0.10.2]
at reactivemongo.api.collections.GenericQueryBuilder$class.cursor(genericcollection.scala:331) ~[reactivemongo_2.10-0.10.0.jar:0.10.0]
at play.modules.reactivemongo.json.collection.JSONQueryBuilder.cursor(jsoncollection.scala:110) ~[play2-reactivemongo_2.10-0.10.2.jar:0.10.2]
at services.common.mongo.MongoDaoComponent$MongoDao$$anon$1.find(MongoDaoComponent.scala:249) ~[classes/:na]
... 25 common frames omitted
Any idea? Thanks.