I've recently started using Scala and probably missing something about the map function. I understand that it returns a new value resulting from applying the given function.
For example I have an array of JValues and want to concatenate each value in the array with another JValue or just transform it to a String as in an example below.
val salesArray = salesJValue.asInstanceOf[JArray]
val storesWithSales = salesArray.map(sale => compact(render(sale)) //Type mismatch here
val storesWithSales = salesArray.map(sale => compact(render(sale) + compact(render(anotherJvalue))) //Type mismatch here
As I can see there is a Type mismatch because the actual value is a String and expected is JValue. Even if I do compact(render(sale).asInstanceOf[JValue] it's not allowed to cast string to JValue. Is it possible to return a different type from a map function? And how can I process the array values to transform each of them to another type?
Take a look at the type signature of the map method:
def map(f: JValue => JValue): JValue
So it's a bit different than other map methods in that you must specify a function whose return type is JValue. This is because a JArray represents specifically a deserialized JSON tree, and cannot hold arbitrary objects or data, only JValues.
If you want to process each of the values of a JArray, call .children on it first. That gives you a List[JValue] which then has a more general map method since Lists can hold any type. Its type signature is:
def map[B](f: A => B): List[B]
So you can do:
val storesWithSales = salesArray.children.map(sale => compact(render(sale)))
Related
Using F# I am trying to scan through a JSON file and compare its arrays against a single array of (randomly generated) numbers. The formatting for the json is:
{"1":[#,#,#,#,#],"2":[#,#,#,#,#],...}
etc for 121 entries. I'm currently trying Json.NET. My problems are:
How can I import a local file with Json.NET?
How would I set about making a simple call of the json key that'd return it's array value that's fit to run it through a for loop?
Here is my code of how far I've gotten:
open System
open System.IO
open Newtonsoft.Json
(*open FSharp.Data
type Provider = JsonProvider<"powernum.json">
let numbers = Provider.Load("powernum.json")
//numbers.``1`` gets me the array but can't scan through all the IDs with an iterating for loop
(*
if numbers.``3`` = [|29;30;41;48;64|] then
printfn "True"
else printfn "False"
*)
(*numbers.JsonValue.Item "1"
let test (a: int) = string a
let testPile =
for i = 1 to 10 do
let goNum = numbers.JsonValue.Item (test i)
Console.Write goNum
Console.WriteLine ""
testPile // This works but is not usable for data comparison with an F# Array
*)
*)
let r = new StreamReader("\PATH\powernum.json")
let (json: string) = r.ReadToEnd();
let conv = JsonConvert.DeserializeObject<> (json)
Console.WriteLine("{0}",conv)//where I'm stuck with Json.NET
[<EntryPoint>]
let main argv =
let rnd = Random()
let numberGen = Set.empty.Add(rnd.Next(1,69)).Add(rnd.Next(1,69)).Add(rnd.Next(1,69)).Add(rnd.Next(1,69)).Add(rnd.Next(1,69)) |>Set.toArray |>Array.sort
Console.ReadKey() |> ignore
0// return an integer exit code
Jsontocsharp.com renders invalid.
I've tried using F# Data but from what I've found it's impossible to make an iterating loop because I have to pull up the "key" with accents encapsulating the number to read it as an int like this numbers.``1``.It doesn't take variables. Tried another method while still using F# Data but it only works as a string that errors when I try to convert it.
For comparison this is the version I prototyped in python:
import random
import json
with open('/PATH/powernum.json') as pnumbers:
data = json.load(pnumbers)
#makes an array with the range of numbers
Valid_numbers =[]
for x in range(69):
Valid_numbers.append(x+1)
generated = []
def pulledNumber():
generated[:]=[]
#adds numbers to the array from 0-4
while len(generated) !=5:
#takes a random number from the range of numbers
generate_number = random.choice(Valid_numbers)
#check if the two arrays have the same values
if generate_number not in generated:
#add to the array if values don't match
generated.append(generate_number)
generated.sort()
return generated
pulledNumber()
for x, y in data.items():
if generated not in y:
print("Id: %s passed" % x)
else:
print("Id: %s failed" % x)
pulledNumber()
break
print (pulledNumber())
f# is a statically typed language - we simply often don't notice because of its excellent type inferencing. But when deserializing from a JSON file, before writing any code, it is useful to determine whether the JSON has a fixed schema, and if so, create or choose an appropriate data model to which the JSON can be mapped automatically.
In your case, your JSON looks like:
{
"1": [29,30,41,48,64],
"2": [29,320,441,548,11]
// Additional items omitted
}
When you have here is a root object with variable property names whose values are always an array of integers. As explained in the Newtonsoft documentation Deserialize a Dictionary, such a JSON object can be deserialized to some IDictionary<string, T> for appropriate value type T. And as explained in Deserialize a Collection a JSON array can be deserialized to a collection of an appropriate item type.
In f# we use Map<'Key,'Value> to represent a dictionary, and lists to represent a materialized list of statically typed values. Thus your JSON corresponds to a
Map<string, int list>
Having determined an appropriate data model, introduce the following function to deserialize JSON from a file:
//https://www.newtonsoft.com/json/help/html/DeserializeWithJsonSerializerFromFile.htm
let jsonFromFile<'T>(fileName : string, settings : JsonSerializerSettings) =
use streamReader = new StreamReader(fileName)
use jsonReader = new JsonTextReader(streamReader)
let serializer = JsonSerializer.CreateDefault(settings)
serializer.Deserialize<'T>(jsonReader)
Now you can deserialize your JSON and filter the values for lists matching some
let fileName = "\PATH\powernum.json"
let requiredValues = [29;30;41;48;64] // Or whatever list of values you are looking for
let map = jsonFromFile<Map<string, int list>>(fileName, null)
let filteredMap =
map |> Map.toSeq
|> Seq.filter (fun (key, value) -> requiredValues = value)
|> Map.ofSeq
// Print results
filteredMap |> Map.iter (fun key value ->
printf "Key = %A has matching list of values = %A\n" key value)
Which prints out
Key = "1" has matching list of values = [29; 30; 41; 48; 64]
Notes:
Always be sure to dispose of disposable resources such as StreamReader after you are done with them. The use keyword ensures this happens automatically when the resource goes out of scope.
If you would prefer to search for an unordered set of values, you can use set instead of list:
let requiredValues = set [64;29;30;41;48] // Or whatever set of values you are looking for
let map = jsonFromFile<Map<string, Set<int>>>(fileName, null)
let filteredMap =
map |> Map.toSeq
|> Seq.filter (fun (key, value) -> requiredValues = value)
|> Map.ofSeq
As explained in Equality and Comparison Constraints in F# by Don Syme, both list and set support structural equality comparisons, which is why requiredValues = value checks that the collections have identical contents.
Demo fiddle #1 here for list and #2 here for set.
Using jackson library I read json data from a file (each row of file is a JSON object) an parse it to a map object of String and Any. My goal is to save specified keys (id and text) to a collection.
val input = scala.io.Source.fromFile("data.json").getLines()
val mapper = new ObjectMapper() with DefaultScalaModule
val data_collection = mutable.HashMap.empty[Int, String]
for (i <- input){
val parsedJson = mapper.readValue[Map[String, Any]](i)
data_collection.put(
parsedJson.get("id"),
parsedJson.get("text")
)
But as the values in the parsedJson map have the Any type, getting some keys like id and text, it returns Some(value) not just the value with the appropriate type. I expect the values for the id key to be Integer and values for the text to be String.
Running the code I got the error:
Error:(31, 23) type mismatch;
found : Option[Any]
required: Int
parsedJson.get("id"),
Here is a sample of JSON data in the file:
{"text": "Hello How are you", "id": 1}
Is it possible in Scala to parse id values to Int and text values to String, or at least convert Some(value) to value with type Int or String?
If you want to get a plain value from a Map instead of a Option you can use the () (apply) method - However it will throw an exception if the key is not found.
Second, Scala type system is static not dynamic, if you have an Any that's it, it won't change to Int or String at runtime, and the compiler will fail - Nevertheless, you can cast them using the asInstanceOf[T] method, but again if type can't be casted to the target type it will throw an exception.
Please note that even if you can make your code work with the above tricks, that code wouldn't be what you would expect in Scala. There are ways to make the code more typesafe (like pattern matching), but parsing a Json to a typesafe object is an old problem, I'm sure jackson provides a way to parse a json into case class that represent your data. If not take a look to circe it does.
Try the below code :
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import
com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
val input = scala.io.Source.fromFile("data.json").getLines()
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
val obj = mapper.readValue[Map[String, Any]](input)
val data_collection = mutable.HashMap.empty[Int, String]
for (i <- c) {
data_collection.put(
obj.get("id").fold(0)(_.toString.toInt),
obj.get("text").fold("")(_.toString)
)
}
println(data_collection) // Map(1 -> Hello How are you)
I am trying to query an API which returns a JSON array (e.g. [{"name":"obj1", "value":5}, {"name":"obj2", "value":2}]) and process the result, which gets parsed as an Option[List[Map[String,Any]]]. However, I am not sure how to properly extract each Map, since the types are erased at runtime.
import scala.util.parsing.json._
import scalaj.http._
val url = "https://api.github.com/users/torvalds/repos"
val req = Http(url).asString
val parsed = JSON.parseFull(req.body) match {
case Some(data) => data match {
case list: List[_] => list
case _ => sys.error("Result is not a list.")
}
case None => sys.error("Invalid JSON received.")
}
parsed.foreach{
case x: Map[_,_] => x.get("full_name") // error here
}
The error occurs because I cannot apply the function with a String key type. However, because of type erasure, the key and value type are unknown, and specifying that it's a String map throws compiler warnings.
Am I going about things the wrong way? Or maybe I'd have better luck with a different HTTP/JSON library?
You can replace your last line with:
parsed.collect{ case x: Map[_,_] => x.asInstanceOf[Map[String,Any]].get("full_name") }
We sort of "cheat" here since we know the keys in a JSON are always Strings.
As for your last question, if you need something lightweight, I think what you have here is as simple as it gets.
Take a look at this SO post if you want to do something more powerful with your pattern matching.
I have an RDD which contains a String and JSON object (as String). I extracted required values from the JSON object. How can I use the values to create a new RDD which stores each value in each column?
RDD
(1234,{"id"->1,"name"->"abc","age"->21,"class"->5})
From which a map was generated as shown below.
"id"->1,
"name"->"abc",
"age"->21
"id"->2,
"name"->"def",
"age"->31
How to convert this to RDD[(String, String, String)], which stores data like:
1 abc 21
2 def 31
Not in front of a compiler right now, but something like this should work:
def parse(val row: (String, JValue)) : Seq((String, String, String)) = {
// Here goes your code to parse a Json into a sequence of tuples, seems like you have this already well in hand.
}
val rdd1 = ??? // Initialize your RDD[(String, JValue)]
val rdd2: RDD[(String, String, String)] = rdd1.flatMap(parse)
flatMap does the trick, as your extraction function can extract multiple rows on each Json input (or none) and they will be seamlessly be integrated into the final RDD.
I have a string that is a Json array of two objects.
> val ss = """[ {"key1" :"value1"}, {"key2":"value2"}]"""
I want to use the Play Json libraries to deserialize it and create a map from the key values to the objects.
def deserializeJsonArray(ss:String):Map[String, JsValue] = ???
// Returns Map("value1" -> {"key1" :"value1"}, "value2" -> {"key2" :"value2"})
How do I write the deserializeJsonArray function? This seems like it should be easy, but I can't figure it out from either the Play documentation or the REPL.
I'm a bit rusty, so please forgive the mess. Perhaps another overflower can come in here and clean it up for me.
This solution assumes that the JSON is an array of objects, and each of the objects contains exactly one key-value pair. I would highly recommend spicing it up with some error handling and/or pattern matching to validate the parsed JSON string.
def deserializeJsonArray(ss: String): Map[String, JsValue] = {
val jsObjectSeq: Seq[JsObject] = Json.parse(ss).as[Seq[JsObject]]
val jsValueSeq: Seq[JsValue] = Json.parse(ss).as[Seq[JsValue]]
val keys: Seq[String] = jsObjectSeq.map(json => json.keys.head)
(keys zip jsValueSeq).toMap
}