Crystal handle json file of known format but dynamic keys - json

So I have a JSON file of a somewhat known format { String => JSON::Type, ... }. So it is basically of type Hash(String, JSON::Type). But when I try and read it from file to memory like so: JSON.parse(File.read(#cache_file)).as(Hash(String, JSON::Type)) I always get an exception: can't cast JSON::Any to Hash(String, JSON::Type)
I'm not sure how I am supposed to handle the data if I can't cast it.
What I basically want to do is the following:
save JSON::Type data under a String key
replace JSON::Type data with other JSON::Type data under a String key
And of course read from / write to file...
Here's the whole thing I've got so far:
class Cache
def initialize(#cache_file = "/tmp/cache_file.tmp")
end
def cache(cache_key : (String | Symbol))
mutable_cache_data = data
value = mutable_cache_data[cache_key.to_s] ||= yield.as(JSON::Type)
File.write #cache_file, mutable_cache_data
value
end
def clear
File.delete #cache_file
end
def data
unless File.exists? #cache_file
File.write #cache_file, {} of String => JSON::Type
end
JSON.parse(File.read(#cache_file)).as(Hash(String, JSON::Type))
end
end
puts Cache.new.cache(:something) { 10 } # => 10
puts Cache.new.cache(:something) { 'a' } # => 10
TL;DR I want to read a JSON file into a Hash(String => i_dont_care), replace a value under a given key name and serialize it to file again. How do I do that?

JSON.parse returns an JSON::Any, not a Hash so you can't cast it. You can however access the underlying raw value as JSON.parse(file).raw and cast this as hash.
Then your code is basically working (I've fixed a few error): https://carc.in/#/r/28c1

You can use use Hash(String, JSON::Type).from_json(File.read(#cache_file)). Hopefully you can restrict the type of JSON::Type down to something more sensible too. JSON::Any and JSON.parse_raw are very much a last resort compared to simply representing your schema using Hash, Array and custom types using JSON.mapping.

Related

How to serialize/deserialize ruby hashes/structs with objects as keys to json

I would like to dump a nested datastructure in ruby to json (I am aware of the Marshal module but I need a standard format) and be able to load/parse the datastructure again. Catch: I use structs (or easier for the example: hashes) as keys of hashes. Example:
require 'json'
h = {{hello: 123} => 123}
JSON.parse(JSON.generate(h)) #=> {"{:hello=>123}"=>123}
So the problem is, that JSON.generate(h) serialises the key {:hello=>123} as a string and when I parse the result again, it remains a string.
How can I solve this and regain the original structure after generate/parse?
JSON only allows strings as object keys. For this reason to_s is called for all keys.
You'll have the following options to solve your issue:
The best option is changing the data structure so it can properly be serialized to JSON.
You'll have to handle the stringified key yourself. An Hash produces a perfectly valid Ruby syntax when converted to a string that can be converted using Kernel#eval like Andrey Deineko suggested in the comments.
result = json.transform_keys { |key| eval(key) }
# json.transform_keys(&method(:eval)) is the same as the above.
The Hash#transform_keys method is relatively new (available since Ruby 2.5.0) and might currently not be in you development environment. You can replace this with a simple Enumerable#map if needed.
result = json.map { |k, v| [eval(k), v] }.to_h
Note: If the incoming JSON contains any user generated content I highly sugest you stay away from using eval since you might allow the user to execute code on your server.
I need a standard format
YAML is a standard format that would suffice here:
▶ h = {{hello: 123} => 123}
#⇒ {{:hello=>123}=>123}
▶ YAML.dump h
#⇒ "---\n? :hello: 123\n: 123\n"
▶ YAML.load _
#⇒ {{:hello=>123}=>123}
As already pointed by mudasobwa, YAML is a good tool: allows you to store also custom class objects:
require 'yaml'
class MyCaptain
attr_accessor :name, :ship
def initialize(name, ship)
#name = name
#ship = ship
end
end
kirk = MyCaptain.new('James T. Kirk', 'USS Enterprise NCC-1701')
picard = MyCaptain.new('Jean-Luc Picard', 'Enterprise NCC-1701D')
captains = [kirk, picard]
File.open("my_captains.yml","w") do |file|
file.write captains.to_yaml
end
p YAML.load_file('my_captains.yml')
#=> [#<MyCaptain:0x007f889d0973b0 #name="James T. Kirk", #ship="USS Enterprise NCC-1701">, #<MyCaptain:0x007f889d096b40 #name="Jean-Luc Picard", #ship="Enterprise NCC-1701D">]

Pass data from JSON to variable for comparison

I have a request that I make in an API using GET
LWP::UserAgent,
the data is returned as JSON, with up to two results at most as follows:
{
"status":1,
"time":1507891855,
"response":{
"prices":{
"nome1\u2122":{
"preco1":1111,
"preco2":1585,
"preco3":1099
},
"nome2":{
"preco1":519,
"preco2":731,
"preco3":491
}
}
}
}
Dump:
$VAR1 = {
'status' => 1,
'time' => 1507891855,
'response' => {
'prices' => {
'nome1' => {
'preco1' => 1111,
'preco3' => 1099,
'preco2' => 1585
},
'nome2' => {
'preco3' => 491,
'preco1' => 519,
'preco2' => 731
}
}
}
};
What I would like to do is:
Take this data and save it in a variable to make a comparison using if with another variable that already has the name stored. The comparison would be with name1 / name2 and if it is true with the other variable it would get preco2 and preco3 to print everything
My biggest problem in the case is that some of these names in JSON contain characters like (TradeMark) that comes as \u2122 (some cases are other characters), so I can not make the comparison with the name of the other variable that is already with the correct name
nome1™
If I could only save the JSON already "converted" the characters would help me with the rest.
Basically after doing the request for the API I want to save the contents in a variable already converting all \u2122 to their respective character (this is the part that I do not know how to do in Perl) and then using another variable to compare them names are equal to show the price
Thanks for the help and any questions please tell me that I try to explain again in another way.
If I understand correctly, you need to get the JSON that you receive in UTF8 format to an internal variable that you can process. For that, you may use JSON::XS:
use utf8;
use JSON::XS;
my $name = "nome1™";
my $var1 = decode_json $utf8_encoded_json_text;
# Compare with name in $name
if( defined $var1->{'response'}->{'prices'}->{$name} ) {
# Do something with the name that matches
my $match = $var1->{'response'}->{'prices'}->{$name};
print $match->{'preco1'}, "\n";
}
Make sure you tell the Perl interpreter that your source is in UTF8 by specifying use utf8; at the beginning of the script. Then make sure you are editing the script with an editor that supports that format.
The function decode_json will return a ref to the converted value. In this case a hash ref. From there you work your way into the JSON.
If you know $name is going to be in the JSON you may omit the defined part. Otherwise, the defined clause will tell you whether the hash value is there. One you know, you may do something with it. If the hash values are a single word with no special characters, you may use $var1->{response}->{prices}->{$name}, but it is always safer to use $var1->{'response'}->{'prices'}->{$name}. Perl gets a bit ugly handling hash refs...
By the way, in JSON::XS you will also find the encode_json function to do the opposite and also an object oriented interface.

Convert json to array using Perl

I have a chunk of json that has the following format:
{"page":{"size":7,"number":1,"totalPages":1,"totalElements":7,"resultSetId":null,"duration":0},"content":[{"id":"787edc99-e94f-4132-b596-d04fc56596f9","name":"Verification","attributes":{"ruleExecutionClass":"VerificationRule"},"userTags":[],"links":[{"rel":"self","href":"/endpoint/787edc99-e94f-4132-b596-d04fc56596f9","id":"787edc99-e94f-...
Basically the size attribute (in this case) tells me that there are 7 parts to the content section. How do I convert this chunk of json to an array in Perl, and can I do it using the size attribute? Or is there a simpler way like just using decode_json()?
Here is what I have so far:
my $resources = get_that_json_chunk(); # function returns exactly the json you see, except all 7 resources in the content section
my #decoded_json = #$resources;
foreach my $resource (#decoded_json) {
I've also tried something like this:
my $deserialize = from_json( $resources );
my #decoded_json = (#{$deserialize});
I want to iterate over the array and handle the data. I've tried a few different ways because I read a little about array refs, but I keep getting "Not an ARRAY reference" errors and "Can't use string ("{"page":{"size":7,"number":1,"to"...) as an ARRAY ref while "strict refs" in use"
Thank you to Matt Jacob:
my $deserialized = decode_json($resources);
print "$_->{id}\n" for #{$deserialized->{content}};

Parse complex Json string contained in Hadoop

I want to parse a string of complex JSON in Pig. Specifically, I want Pig to understand my JSON array as a bag instead of as a single chararray. I found that complex JSON can be parsed by using Twitter's Elephant Bird or Mozilla's Akela library. (I found some additional libraries, but I cannot use 'Loader' based approach since I use HCatalog Loader to load data from Hive.)
But, the problem is the structure of my data; each value of Map structure contains value part of complex JSON. For example,
1. My table looks like (WARNING: type of 'complex_data' is not STRING, a MAP of <STRING, STRING>!)
TABLE temp_table
(
user_id BIGINT COMMENT 'user ID.',
complex_data MAP <STRING, STRING> COMMENT 'complex json data'
)
COMMENT 'temp data.'
PARTITIONED BY(created_date STRING)
STORED AS RCFILE;
2. And 'complex_data' contains (a value that I want to get is marked with two *s, so basically #'d'#'f' from each PARSED_STRING(complex_data#'c') )
{ "a": "[]",
"b": "\"sdf\"",
"**c**":"[{\"**d**\":{\"e\":\"sdfsdf\"
,\"**f**\":\"sdfs\"
,\"g\":\"qweqweqwe\"},
\"c\":[{\"d\":21321,\"e\":\"ewrwer\"},
{\"d\":21321,\"e\":\"ewrwer\"},
{\"d\":21321,\"e\":\"ewrwer\"}]
},
{\"**d**\":{\"e\":\"sdfsdf\"
,\"**f**\":\"sdfs\"
,\"g\":\"qweqweqwe\"},
\"c\":[{\"d\":21321,\"e\":\"ewrwer\"},
{\"d\":21321,\"e\":\"ewrwer\"},
{\"d\":21321,\"e\":\"ewrwer\"}]
},]"
}
3. So, I tried... (same approach for Elephant Bird)
REGISTER '/path/to/akela-0.6-SNAPSHOT.jar';
DEFINE JsonTupleMap com.mozilla.pig.eval.json.JsonTupleMap();
data = LOAD temp_table USING org.apache.hive.hcatalog.pig.HCatLoader();
values_of_map = FOREACH data GENERATE complex_data#'c' AS attr:chararray; -- IT WORKS
-- dump values_of_map shows correct chararray data per each row
-- eg) ([{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... }])
([{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... }]) ...
attempt1 = FOREACH data GENERATE JsonTupleMap(complex_data#'c'); -- THIS LINE CAUSE AN ERROR
attempt2 = FOREACH data GENERATE JsonTupleMap(CONCAT(CONCAT('{\\"key\\":', complex_data#'c'), '}'); -- IT ALSO DOSE NOT WORK
I guessed that "attempt1" was failed because the value doesn't contain full JSON. However, when I CONCAT like "attempt2", I generate additional \ mark with. (so each line starts with {\"key\": ) I'm not sure that this additional marks breaks the parsing rule or not. In any case, I want to parse the given JSON string so that Pig can understand. If you have any method or solution, please Feel free to let me know.
I finally solved my problem by using jyson library with jython UDF.
I know that I can solve it by using JAVA or other languages.
But, I think that jython with jyson is the most simplist answer to this issue.

How can I get ruby's JSON to follow object references like Pry/PP?

I've stared at this so long I'm going in circles...
I'm using the rbvmomi gem, and in Pry, when I display an object, it recurses down thru the structure showing me the nested objects - but to_json seems to "dig down" into some objects, but just dump the reference for others> Here's an example:
[24] pry(main)> g
=> [GuestNicInfo(
connected: true,
deviceConfigId: 4000,
dynamicProperty: [],
ipAddress: ["10.102.155.146"],
ipConfig: NetIpConfigInfo(
dynamicProperty: [],
ipAddress: [NetIpConfigInfoIpAddress(
dynamicProperty: [],
ipAddress: "10.102.155.146",
prefixLength: 20,
state: "preferred"
)]
),
macAddress: "00:50:56:a0:56:9d",
network: "F5_Real_VM_IPs"
)]
[25] pry(main)> g.to_json
=> "[\"#<RbVmomi::VIM::GuestNicInfo:0x000000085ecc68>\"]"
Pry apparently just uses a souped-up pp, and while "pp g" gives me close to what I want, I'm kinda steering as hard as I can toward json so that I don't need a custom parser to load up and manipulate the results.
The question is - how can I get the json module to dig down like pp does? And if the answer is "you can't" - any other suggestions for achieving the goal? I'm not married to json - if I can get the data serialized and read it back later (without writing something to parse pp output... which may already exist and I should look for it), then it's all win.
My "real" goal here is to slurp up a bunch of info from our vsphere stuff via rbvmomi so that I can do some network/vm analysis on it, which is why I'd like to get it in a nice machine-parsed format. If I'm doing something stupid here and there's an easier way to go about this - lay it on me, I'm not proud. Thank you all for your time and attention.
Update: Based on Arnie's response, I added this monkeypatch to my script:
class RbVmomi::BasicTypes::DataObject
def to_json(*args)
h = self.props
m = h.merge({ JSON.create_id => self.class.name })
m.to_json(*args)
end
end
and now my to_json recurses down nicely. I'll see about submitting this (or the def, really) to the project.
The .to_json works in a recursive manner, the default behavior is defined as:
Converts this object to a string (calling to_s), converts it to a JSON string, and returns the result. This is a fallback, if no special method to_json was defined for some object.
json library has added some implementation for some common classes (check the left hand side of this documentation), such as Array, Range, DateTime.
For an array, to_json first convert all the elements to json object, concat then together, and then add the array mark [/].
For your case, you need to define your customized to_json method for GuestNicInfo, NetIpConfigInfo and NetIpConfigInfoIpAddress. I don't know your implementation about these three classes, so I wrote a example to demonstrate how to achieve this:
require 'json'
class MyClass
attr_accessor :a, :b
def initialize(a, b)
#a = a
#b = b
end
end
data = [MyClass.new(1, "foobar")]
puts data.to_json
#=> ["#<MyClass:0x007fb6626c7260>"]
class MyClass
def to_json(*args)
{
JSON.create_id => self.class.name,
:a => a,
:b => b
}.to_json(*args)
end
end
puts data.to_json
#=> [{"json_class":"MyClass","a":1,"b":"foobar"}]