Related
JSON string input: https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=MSFT&apikey=demo
I am trying to return just the first key (current day) in the hash but have been unable to do so. My code looks like the following
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
use Data::Dumper;
use JSON;
my $html = get("https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=AMD&apikey=CMDPTEHVYH7W5VSZ");
my $decoded = decode_json($html);
my ($open) = $decoded->{'Time Series (Daily)'}->[0]->{'1. open'};
I keep getting "Not an ARRAY reference" which I researched and got more confused.
I can access what I want directly with the below code but I want to access just the first result or the current day:
my ($open) = $decoded->{'Time Series (Daily)'}{'2017-12-20'}{'1. open'};
Also if I do something like this:
my ($open) = $decoded->{'Time Series (Daily)'};
print Dumper($open);
The output is as follows:
$VAR1 = {
'2017-09-07' => {
'1. open' => '12.8400',
'5. volume' => '35467788',
'2. high' => '12.9400',
'4. close' => '12.6300',
'3. low' => '12.6000'
},
'2017-11-15' => {
'3. low' => '10.7700',
'4. close' => '11.0700',
'2. high' => '11.1300',
'5. volume' => '33326871',
'1. open' => '11.0100'
},
'2017-11-30' => {
'1. open' => '10.8700',
'2. high' => '11.0300',
'5. volume' => '43101899',
'3. low' => '10.7600',
'4. close' => '10.8900'
},
Thank you in advance for any help you can provide a noob.
Problem 1: { denotes the start of a JSON object, which gets decoded into a hash. Trying to derefence an array is going to fail.
Problem 2: Like Perl hashes, JSON objects are unordered, so talking about the
"first key" makes no sense. Perhaps you want the most recent date?
use List::Util qw( maxstr );
my $time_series_daily = $decoded->{'Time Series (Daily)'};
my $latest_date = maxstr #$time_series_daily;
my $open = $time_series_daily->{$latest_date}{'1. open'};
You are picking among hashref keys, not array (sequential container) elements. Since hashes are inherently unordered you can't index into that list but need to sort keys as needed.
With the exact format you show this works
my $top = (sort { $b cmp $a } keys %{ $decoded->{'Time Series (Daily)'} } )[0];
say $decoded->{'Time Series (Daily)'}{$top}{'1. open'};
It gets the list of keys, inverse-sorts them (alphabetically), and takes the first element of that list.
If your date-time format may vary then you'll need to parse it for sorting.
If you will really ever only want the most-recent one this is inefficient since it sorts the whole list. Then use a more specific tool to extract only the "largest" element, like
use List::Util qw(reduce);
my $top = reduce { $a gt $b ? $a : $b }
keys %{ $decoded->{'Time Series (Daily)'} };
But then in your case this can be done simply by maxstr from the same List::Util module, as shown in ikegami's answer. On the other hand, if the datetime format doesn't lend itself to a direct lexicographical comparison used by strmax then the reduce allows use of custom comparisons.
I want to validate a password field as follows:
First check that the password has only lower case
Then check that its length is min 8 characters.
I have written the two validation codes as follows:
/*check lowercase*/
def checkPasswordCase[T]: Reads[String] = {
Reads.StringReads.filter(ValidationError("Password not in lowercase"))(str => {
(str.matches("""[a-z]+"""))
})
}
/*check length*/
def checkPasswordLength[T](min:Int): Reads[String] = {
Reads.StringReads.filter(ValidationError("Invalid password length"))(str => {
(str.length >= min)
})
}
I called the code as follows but got compilation error Cannot prove that String <:< String => C.
(JsPath \ "user" \ "password").read[String](checkPasswordCase)(checkPasswordLength(8)) and ...
I tried using Reads.minLength(8) but got a different error.
Couldn't I use two validation codes back to back?
You can use either the keepAnd or andKeep validators to do this. These combinators will run both Reads but only keep the successful result of either the left or right operands:
val r = (JsPath \ "user" \ "password").read[String](
checkPasswordCase keepAnd checkPasswordLength(8))
val js = Json.obj("user" -> Json.obj("password" -> "FOO"))
println(js.validate[String](r))
// JsError(List((/user/password,List(
// ValidationError(List(Password not in lowercase),WrappedArray()),
// ValidationError(List(Invalid password length),WrappedArray())))))
I have the following reads function for parsing JSON files.
case class tables(col1 : Option[List[another case class]], col2 : Option[List[another case class]], col3 : Option[List[another case class]], col4 : Option[List[another case class]])
implicit val tablesRead: Reads[tables] = (
(JsPath \ "col1").read(Reads.optionWithNull[List[data1]]).filterNot(_.get.isEmpty) and
(JsPath \ "col2").read(Reads.optionWithNull[List[data2]]).filterNot(_.get.isEmpty) and
(JsPath \ "col3").read(Reads.optionWithNull[List[data3]]).filterNot(_.get.isEmpty) and
(JsPath \ "col4").read(Reads.optionWithNull[List[data4]]).filterNot(_.get.isEmpty)
) (tables.apply _)
I want to then insert the JSON into a database after having validated it. I have therefore declared the following function.
def createFromJson = Action.async(parse.json) { request =>
request.body.validate[jsonWrapper] match {
case JsSuccess(data, _) =>
for {
dbFuture <- dataFuture(data.userID)
lastError <- dbFuture.insert(data.tables)
} yield {
Ok("Success\n")
}
case JsError(errors) => Future.successful(BadRequest("Failed :" + Error.show(errors)))
}
}
This works and correctly rejects JSONs looking like this:
{"tables":{"col1":[],"col2":[],"col3":[],"col4":[]}, "userID":"irrelavent"}
and accepts JSONs with actual data in, like so:
{"tables":{"col1":[{data1}],"col2":[{data2}],"col3":[{data3}],"col4":[{data4}]}, "userID":"irrelavent"}
But want i need is something that does this but also accepts a JSON with missing fields
{"tables":{"col1":[{data1}],"col2":[],"col3":[{data3}],"col4":[{data4}]}, "userID":"irrelavent"}
And preferable ignore them (i.e. return something like :
{"tables":{"col1":[{data1}],"col3":[{data3}],"col4":[{data4}]}, "userID":"irrelavent"})
Is this possible to do?
Many thanks,
Peter M.
You can automatically generate a Reads[tables] using Json.reads macro with the behavior you want:
implicit val tablesRead: Reads[tables] = Json.reads[tables]
If the fields is missing from the JSON the column will be None.
On a minor note, the common form in scala is to start a class name with a capital letter so you should rename tables to Tables.
I'm reading a nested JSON document using several Reads[T] implementations, however, I'm stuck with the following sub-object:
{
...,
"attributes": {
"keyA": [1.68, 5.47, 3.57],
"KeyB": [true],
"keyC": ["Lorem", "Ipsum"]
},
...
}
The keys ("keyA", "keyB"...) as well as the amount of keys are not known at compile time and can vary. The values of the keys are always JsArray instances, but of different size and type (however, all elements of a particular array must have the same JsValue type).
The Scala representation of one single attribute:
case class Attribute[A](name: String, values: Seq[A])
// 'A' can only be String, Boolean or Double
The goal is to create a Reads[Seq[Attribute]] that can be used for the "attributes"-field when transforming the whole document (remember, "attributes" is just a sub-document).
Then there is a simple map that contains allowed combinations of keys and array types that should be used to validate attributes. Edit: This map is specific for each request (or rather specific for every type of json document). But you can assume that it is always available in the scope.
val required = Map(
"KeyA" -> "Double",
"KeyB" -> "String",
"KeyD" -> "String",
)
So in the case of the JSON shown above, the Reads should create two errors:
"keyB" does exist, but has the wrong type (expected String, was boolean).
"keyD" is missing (whereas keyC is not needed and can be ignored).
I'm having trouble creating the necessary Reads. The first thing I tried as a first step, from the perspective of the outer Reads:
...
(__ \ "attributes").reads[Map[String, JsArray]]...
...
I thought this is a nice first step because if the JSON structure is not an object containing Strings and JsArrays as key-value pairs, then the Reads fails with proper error messages. It works, but: I don't know how to go on from there. Of course I just could create a method that transforms the Map into a Seq[Attribute], but this method somehow should return a JsResult, since there are further validations to do.
The second thing I tried:
val attributeSeqReads = new Reads[Seq[Attribute]] {
def reads(json: JsValue) = json match {
case JsObject(fields) => processAttributes(fields)
case _ => JsError("attributes not an object")
}
def processAttributes(fields: Map[String, JsValue]): JsResult[Seq[Attribute]] = {
// ...
}
}
The idea was to validate each element of the map manually within processAttributes. But I think this is too complicated. Any help is appreciated.
edit for clarification:
At the beginning of the post I said that the keys (keyA, keyB...) are unknown at compile time. Later on I said that those keys are part of the map required which is used for validation. This sounds like a contradiction, but the thing is: required is specific for each document/request and is also not known at compile time. But you don't need to worry about that, just assume that for every request the correct required is already available in the scope.
You are too confused by the task
The keys ("keyA", "keyB"...) as well as the amount of keys are not known at compile time and can vary
So the number of keys and their types are known in advance and the final?
So in the case of the JSON shown above, the Reads should create two
errors:
"keyB" does exist, but has the wrong type (expected String, was
boolean).
"keyD" is missing (whereas keyC is not needed and can be ignored).
Your main task is just to check the availability and compliance?
You may implement Reads[Attribute] for every your key with Reads.list(Reads.of[A]) (this Reads will check type and required) and skip omitted (if not required) with Reads.pure(Attribute[A]). Then tuple convert to list (_.productIterator.toList) and you will get Seq[Attribute]
val r = (
(__ \ "attributes" \ "keyA").read[Attribute[Double]](list(of[Double]).map(Attribute("keyA", _))) and
(__ \ "attributes" \ "keyB").read[Attribute[Boolean]](list(of[Boolean]).map(Attribute("keyB", _))) and
((__ \ "attributes" \ "keyC").read[Attribute[String]](list(of[String]).map(Attribute("keyC", _))) or Reads.pure(Attribute[String]("keyC", List()))) and
(__ \ "attributes" \ "keyD").read[Attribute[String]](list(of[String]).map(Attribute("keyD", _)))
).tupled.map(_.productIterator.toList)
scala>json1: play.api.libs.json.JsValue = {"attributes":{"keyA":[1.68,5.47,3.57],"keyB":[true],"keyD":["Lorem","Ipsum"]}}
scala>res37: play.api.libs.json.JsResult[List[Any]] = JsSuccess(List(Attribute(keyA,List(1.68, 5.47, 3.57)), Attribute(KeyB,List(true)), Attribute(keyC,List()), Attribute(KeyD,List(Lorem, Ipsum))),)
scala>json2: play.api.libs.json.JsValue = {"attributes":{"keyA":[1.68,5.47,3.57],"keyB":[true],"keyC":["Lorem","Ipsum"]}}
scala>res38: play.api.libs.json.JsResult[List[Any]] = JsError(List((/attributes/keyD,List(ValidationError(List(error.path.missing),WrappedArray())))))
scala>json3: play.api.libs.json.JsValue = {"attributes":{"keyA":[1.68,5.47,3.57],"keyB":["Lorem"],"keyC":["Lorem","Ipsum"]}}
scala>res42: play.api.libs.json.JsResult[List[Any]] = JsError(List((/attributes/keyD,List(ValidationError(List(error.path.missing),WrappedArray()))), (/attributes/keyB(0),List(ValidationError(List(error.expected.jsboolean),WrappedArray())))))
If you will have more than 22 attributes, you will have another problem: Tuple with more than 22 properties.
for dynamic properties in runtime
inspired by 'Reads.traversableReads[F[_], A]'
def attributesReads(required: Map[String, String]) = Reads {json =>
type Errors = Seq[(JsPath, Seq[ValidationError])]
def locate(e: Errors, idx: Int) = e.map { case (p, valerr) => (JsPath(idx)) ++ p -> valerr }
required.map{
case (key, "Double") => (__ \ key).read[Attribute[Double]](list(of[Double]).map(Attribute(key, _))).reads(json)
case (key, "String") => (__ \ key).read[Attribute[String]](list(of[String]).map(Attribute(key, _))).reads(json)
case (key, "Boolean") => (__ \ key).read[Attribute[Boolean]](list(of[Boolean]).map(Attribute(key, _))).reads(json)
case _ => JsError("")
}.iterator.zipWithIndex.foldLeft(Right(Vector.empty): Either[Errors, Vector[Attribute[_ >: Double with String with Boolean]]]) {
case (Right(vs), (JsSuccess(v, _), _)) => Right(vs :+ v)
case (Right(_), (JsError(e), idx)) => Left(locate(e, idx))
case (Left(e), (_: JsSuccess[_], _)) => Left(e)
case (Left(e1), (JsError(e2), idx)) => Left(e1 ++ locate(e2, idx))
}
.fold(JsError.apply, { res =>
JsSuccess(res.toList)
})
}
(__ \ "attributes").read(attributesReads(Map("keyA" -> "Double"))).reads(json)
scala> json: play.api.libs.json.JsValue = {"attributes":{"keyA":[1.68,5.47,3.57],"keyB":[true],"keyD":["Lorem","Ipsum"]}}
scala> res0: play.api.libs.json.JsResult[List[Attribute[_ >: Double with String with Boolean]]] = JsSuccess(List(Attribute(keyA,List(1.68, 5.47, 3.57))),/attributes)
I'm using Play 2.4 with Slick 3.1.x, specifically the Slick-Play plugin v1.1.1. Firstly, some context... I have the following search/filter method in a DAO, which joins together 4 models:
def search(
departureCity: Option[String],
arrivalCity: Option[String],
departureDate: Option[Date]
) = {
val monadicJoin = for {
sf <- slickScheduledFlights.filter(a =>
departureDate.map(d => a.date === d).getOrElse(slick.lifted.LiteralColumn(true))
)
fl <- slickFlights if sf.flightId === fl.id
al <- slickAirlines if fl.airlineId === al.id
da <- slickAirports.filter(a =>
fl.departureAirportId === a.id &&
departureCity.map(c => a.cityCode === c).getOrElse(slick.lifted.LiteralColumn(true))
)
aa <- slickAirports.filter(a =>
fl.arrivalAirportId === a.id &&
arrivalCity.map(c => a.cityCode === c).getOrElse(slick.lifted.LiteralColumn(true))
)
} yield (fl, sf, al, da, aa)
db.run(monadicJoin.result)
}
The output from this is a Vector containing sequences, e.g:
Vector(
(
Flight(Some(1),123,216,2013,3,1455,2540,3,905,500,1150),
ScheduledFlight(Some(1),1,2016-04-13,90,10),
Airline(Some(216),BA,BAW,British Airways,United Kingdom),
Airport(Some(2013),LHR,Heathrow,LON,...),
Airport(Some(2540),JFK,John F Kennedy Intl,NYC...)
),
(
etc ...
)
)
I'm currently rendering the JSON in the controller by calling .toJson on a Map and inserting this Vector (the results param below), like so:
flightService.search(departureCity, arrivalCity, departureDate).map(results => {
Ok(
Map[String, Any](
"status" -> "OK",
"data" -> results
).toJson
).as("application/json")
})
While this sort of works, it produces output in an unusual format; an array of results (the rows) within each result object the joins are nested inside objects with keys: "_1", "_2" and so on.
So the question is: How should I go about restructuring this?
There doesn't appear to be anything which specifically covers this sort of scenario in the Slick docs. Therefore I would be grateful for some input on what might be the best way to refactor this Vector of Seq's, with a view to renaming each of the joins or even flattening it out and only keeping certain fields?
Is this best done in the DAO search method before it's returned (by mapping it somehow?) or in the controller after I get back the Future results Vector from the search method?
Or I'm wondering whether it would be preferable to abstract this sort of mutation out somewhere else entirely, using a transformer perhaps?
You need JSON Reads/Writes/Format Combinators
In the first place you must have Writes[T] for all your classes (Flight, ScheduledFlight, Airline, Airport).
Simple way is using Json macros
implicit val flightWrites: Writes[Flight] = Json.writes[Flight]
implicit val scheduledFlightWrites: Writes[ScheduledFlight] = Json.writes[ScheduledFlight]
implicit val airlineWrites: Writes[Airline] = Json.writes[Airline]
implicit val airportWrites: Writes[Airport] = Json.writes[Airport]
You must implement OWrites[(Flight, ScheduledFlight, Airline, Airport, Airport)] for Vector item also. For example:
val itemWrites: OWrites[(Flight, ScheduledFlight, Airline, Airport, Airport)] = (
(__ \ "flight").write[Flight] and
(__ \ "scheduledFlight").write[ScheduledFlight] and
(__ \ "airline").write[Airline] and
(__ \ "airport1").write[Airport] and
(__ \ "airport2").write[Airport]
).tupled
for writing whole Vector as JsAray use Writes.seq[T]
val resultWrites: Writes[Seq[(Flight, ScheduledFlight, Airline, Airport, Airport)]] = Writes.seq(itemWrites)
We have all to response your data
flightService.search(departureCity, arrivalCity, departureDate).map(results =>
Ok(
Json.obj(
"status" -> "Ok",
"data" -> resultWrites.writes(results)
)
)