The end goal for this is to be part of a chatbot that returns an airport's weather.
Using import.io, I built an endpoint to query the weather service I'd which provides this response:
{"extractorData"=>
{"url"=>
"https://www.aviationweather.gov/metar/data?ids=kokb&format=decoded&hours=0&taf=off&layout=on&date=0",
"resourceId"=>"66ca907842aabb6b08b8bc12049ad533",
"data"=>
[{"group"=>
[{"Timestamp"=>[{"text"=>"Data at: 2135 UTC 12 Dec 2016"}],
"Airport"=>[{"text"=>"KOKB (Oceanside Muni, CA, US)"}],
"FullText"=>
[{"text"=>
"KOKB 122052Z AUTO 24008KT 10SM CLR 18/13 A3006 RMK AO2 SLP179 T01780133 58021"}],
"Temperature"=>[{"text"=>"17.8°C ( 64°F)"}],
"Dewpoint"=>[{"text"=>"13.3°C ( 56°F) [RH = 75%]"}],
"Pressure"=>
[{"text"=>
"30.06 inches Hg (1018.0 mb) [Sea level pressure: 1017.9 mb]"}],
"Winds"=>
[{"text"=>"from the WSW (240 degrees) at 9 MPH (8 knots; 4.1 m/s)"}],
"Visibility"=>[{"text"=>"10 or more sm (16+ km)"}],
"Ceiling"=>[{"text"=>"at least 12,000 feet AGL"}],
"Clouds"=>[{"text"=>"sky clear below 12,000 feet AGL"}]}]}]},
"pageData"=>
{"resourceId"=>"66ca907842aabb6b08b8bc12049ad533",
"statusCode"=>200,
"timestamp"=>1481578559306},
"url"=>
"https://www.aviationweather.gov/metar/data?ids=kokb&format=decoded&hours=0&taf=off&layout=on&date=0",
"runtimeConfigId"=>"2ddb288f-9e57-4b58-a690-1cd409f9edd3",
"timestamp"=>1481579246454,
"sequenceNumber"=>-1}
I seem to be running into two issues. How do I:
pull each field and write it into its own variable
ignore the "text" modifier in the response.
If you're getting a response object, you might want to do something like
parsed_json = JSON.parse(response.body)
Then you can do things like parsed_json[:some_field]
The simple answer is:
require 'json'
foo = JSON['{"a":1}']
foo # => {"a"=>1}
JSON is smart enough to look at the parameter and, based on whether it's a string or an Array or Hash, parse it or serialize it. In the above case it parsed it back into a Hash.
From that point it takes normal Ruby to dive into the hash you got back and access particular values:
foo = JSON['{"a":1, "b":[{"c":3}]}']
foo # => {"a"=>1, "b"=>[{"c"=>3}]}
foo['b'][0]['c'] # => 3
How to walk through a hash is covered extensively on the internet and here on Stack Overflow, so search around and see what you can find.
Related
I try to read out OBD-2 data from Hyundai Ioniq Electro (Version 28kWh), using a Raspberry PI and a Bluetooth ELM327 interface. Connection and data transfer works fine.
For example: sending 2105<cr><lf> gives a response (<cr> is value 0x0d = 13):
7F2112<cr>7F2112<cr>7F2112<cr>02D<cr>0:6105FFFFFFFF<cr>7F2112<cr>1:00000000001616<cr>2:161616161621FA<cr>3:26480001501616<cr>4:03E82403E80FC0<cr>5:003A0000000000<cr>6:00000000000000<cr><cr>>
The value C0 in 4:03E82403E80FC0 seems to be the State of charge (SOC) display value:
C0 -> 192 -> 192/2 % = 96%
There are some tables for decoding available (see https://github.com/JejuSoul/OBD-PIDs-for-HKMC-EVs/tree/master/Ioniq%20EV%20-%2028kWh), but how to use these tables?
For example sending 2101<cr><lf> gives the response:
02C<cr>
0:6101FFFFF800<cr>
01E<cr>
0:6101000003FF<cr>
03D<cr>
0:6101FFFFFFFF<cr>
016<cr>
0:6101FFE00000<cr>
1:0002D402CD03F0<cr>
1:0838010A015C2F<cr>
7F2112<cr>
1:B4256026480000<cr>
1:0921921A061B03<cr>
2:000582003401BD<cr>
2:0000000A002702<cr>
2:000F4816161616<cr>
2:00000000276234<cr>
3:04B84100000000<cr>
3:5B04692F180018<cr>
3:01200000000000<cr>
3:1616160016CB3F<cr>
4:00220000600000<cr>
4:00D0FF00000000<cr>
4:CB0100007A0002<cr>
5:000001F3026A02<cr>
5:5D4000025D4600<cr>
6:D2000000000000<cr>
6:00DECA0000D8E6<cr>
7:008A2FEB090002<cr>
8:0000000003E800<cr>
<cr>
>
Please note, that the line feed was added behind every carriage return (<cr>) for better readability and is not part of the original data response.
How can I decode temperature, currents, ... from these data?
I have found the mistake by myself. The ELM327 description (http://elmelectronics.com/DSheets/ELM327DS.pdf) explains the AT commands in detail.
The problem on this issue was the mixing of CAN responses from multiple ECU's caused by the AT H0 command (headers off) in the initialization phase (not described in question). See also EM327DS.pdf page 44 (Multiple Responses).
When using AT H1 on startup, the responses can be decoded without problem.
Initialization (with AT H1 = headers on)
AT D\r\n
AT Z\r\n
AT L0\r\n
AT E0\r\n
AT S0\r\n
AT H1\r\n
AT SP 0\r\n
Afterwards communication with ECU's:
Response on first command 0100\r\n:
SEARCHING...\r7EB06410080000001\r7EC06410080000001\r\r>
Response on second command 2101\r\n:
7EE037F2112\r7ED102C6101FFFFF800\r7EA10166101FFE00000\r7EC103D6101FFFFFFFF\r7EB101E6101000003FF\r7EA2109211024062703\r7EC214626482648A3FF\r7ED2100907D87E15592\r7EB210838011D88B132\r7ED2202A1A7024C0134\r7EA2200000000546900\r7EC22C00D9E1C1B1B1B\r7EB220000000A000802\r7EA2307200000000000\r7ED23050343102000C8\r7EC231B1B1C001BB50F\r7EB233C04B8320000D0\r7EC24B5010000810002\r7ED24047400C8760017\r7EB24FF300000000000\r7ED25001401F387F46A\r7EC256AC100026CB100\r7EC2600E3C50000DE69\r7ED263F001300000000\r7EC27008CC38209015C\r7EC280000000003E800\r\r>
Response on third command 2105\r\n:
7EE037F2112\r7ED037F2112\r7EA037F2112\r7EC102D6105FFFFFFFF\r7EB037F2112\r7EC2100000000001B1C\r7EC221C1B1B1B1B2648\r7EC2326480001641A1B\r7EC2403E80803E80147\r7EC25003A0000000000\r7EC2600000000000000\r\r>
Now every response starts with the id of the ECU. Take attention only to responses starting with 7EC.
Example:
Looking for battery current in amps. In the document Spreadsheet_IoniqEV_BMS_2101_2105.xls you find the battery current on:
response 21 for 2101: last byte = High Byte of battery current
response 22 for 2101: first byte = Low Byte of battery current
So look to the response of 2101\r\n and search for 7EC21 and 7EC22: You will find:
7EC214626482648A3FF: take last byte for battery high value -> FF
7EC22C00D9E1C1B1B1B: take first byte after 7EC22 for battery low value -> C0
The battery current value is: FFC0
This value is two complements encoded:
0xffc0 = 65472 -> 65472 - 65536 = -64 -> -6.4A
Result: the battery is charged with 6.4A
For a coding example see:
https://github.com/greenenergyprojects/obd2-gateway, file src/obd2/obd2.ts
I would like to dump a nested datastructure in ruby to json (I am aware of the Marshal module but I need a standard format) and be able to load/parse the datastructure again. Catch: I use structs (or easier for the example: hashes) as keys of hashes. Example:
require 'json'
h = {{hello: 123} => 123}
JSON.parse(JSON.generate(h)) #=> {"{:hello=>123}"=>123}
So the problem is, that JSON.generate(h) serialises the key {:hello=>123} as a string and when I parse the result again, it remains a string.
How can I solve this and regain the original structure after generate/parse?
JSON only allows strings as object keys. For this reason to_s is called for all keys.
You'll have the following options to solve your issue:
The best option is changing the data structure so it can properly be serialized to JSON.
You'll have to handle the stringified key yourself. An Hash produces a perfectly valid Ruby syntax when converted to a string that can be converted using Kernel#eval like Andrey Deineko suggested in the comments.
result = json.transform_keys { |key| eval(key) }
# json.transform_keys(&method(:eval)) is the same as the above.
The Hash#transform_keys method is relatively new (available since Ruby 2.5.0) and might currently not be in you development environment. You can replace this with a simple Enumerable#map if needed.
result = json.map { |k, v| [eval(k), v] }.to_h
Note: If the incoming JSON contains any user generated content I highly sugest you stay away from using eval since you might allow the user to execute code on your server.
I need a standard format
YAML is a standard format that would suffice here:
▶ h = {{hello: 123} => 123}
#⇒ {{:hello=>123}=>123}
▶ YAML.dump h
#⇒ "---\n? :hello: 123\n: 123\n"
▶ YAML.load _
#⇒ {{:hello=>123}=>123}
As already pointed by mudasobwa, YAML is a good tool: allows you to store also custom class objects:
require 'yaml'
class MyCaptain
attr_accessor :name, :ship
def initialize(name, ship)
#name = name
#ship = ship
end
end
kirk = MyCaptain.new('James T. Kirk', 'USS Enterprise NCC-1701')
picard = MyCaptain.new('Jean-Luc Picard', 'Enterprise NCC-1701D')
captains = [kirk, picard]
File.open("my_captains.yml","w") do |file|
file.write captains.to_yaml
end
p YAML.load_file('my_captains.yml')
#=> [#<MyCaptain:0x007f889d0973b0 #name="James T. Kirk", #ship="USS Enterprise NCC-1701">, #<MyCaptain:0x007f889d096b40 #name="Jean-Luc Picard", #ship="Enterprise NCC-1701D">]
I am trying to show the "sgv" value on a Dashing / Smashing dashboard widget. Ultimately I would also like to show the "direction" value as well. I am running into problems pulling that precise value down which changes every 3 to 5 minutes. I have already been able to mirror the exact string using the following:
require 'net/http'
require 'rest-client'
require 'json'
url = "https://dnarnianbg.herokuapp.com/api/v1/entries/current.json"
response = RestClient.get(url)
JSON.parse(response)
# :first_in sets how long it takes before the job is first run. In this case, it is run immediately
current_nightscout = 0
SCHEDULER.every '5m' do
last_nightscout = current_nightscout
current_nightscout = response
send_event('nightscout', { current: current_nightscout, last: last_nightscout })
end
I have also searched the archives several times. I don't wish to write this to a file like this one shows and the duplicate question has been deleted or moved.
I realize that the JSON.parse(response) is just going to parse out whatever I tell it the response equals, but I don't know how to get that response to equal SGV. Maybe the solution isn't in the RestClient, but that is where I am lost.
Here is the JSON URL: http://dnarnianbg.herokuapp.com/api/v1/entries/current.json
EDIT: The output of that link is something like this:
[{"_id":"5ba295ddb8a1ee0aede71822","sgv":87,"date":1537381813000,"dateString":"2018-09-19T18:30:13.000Z","trend":4,"direction":"Flat","device":"share2","type":"sgv"}]
You need something like response[0]["sgv"] which should return 52 if you end up with many items in the list you will need to iterate over them.
The best thing you can do is to break your problem down into easier parts to debug. As you are having problems accessing some JSON via an API you should make a simple script which only does the function you want in order to test it and see where the problem is.
Here is a short example you can put into a .rb file and run;
#!/usr/bin/ruby
require 'open-uri'
require 'json'
test = JSON.parse(open("https://dnarnianbg.herokuapp.com/api/v1/entries/current.json", :read_timeout => 4).read)
puts test[0]["sgv"]
That should return the value from sgv
I realise that short sweet example may be little use as a learner so here is a more verbose version with some comments;
#!/usr/bin/ruby
require 'open-uri'
require 'json'
# Open the URL and read the result. Time out if this takes longer then 4 sec.
get_data = open("https://dnarnianbg.herokuapp.com/api/v1/entries/current.json", :read_timeout => 4).read
# Parse the response (get_data) to JSON and put in variable output
output = JSON.parse(get_data)
# Put the output to get the 'sgv figure'
p output[0]["sgv"]
It always pays to manually examine the data you get back, in your case the data looks like this (when make pretty)
[
{
"_id": "5ba41a0fb8a1ee0aedf6eb2c",
"sgv": 144,
"date": 1537481109000,
"dateString": "2018-09-20T22:05:09.000Z",
"trend": 4,
"direction": "Flat",
"device": "share2",
"type": "sgv"
}
]
What you actually have is an Array. Your server returns only 1 result, numbered '0' hence you need [0] in your p statement. Once you have accessed the array id then you can simply use the object you need as [sgv]
If your app ever returns more than one record then you will need to change your code to read all of the results and iterate over them in order to get all the values you need.
Here is the final code that made it work
require 'net/http'
require 'json'
require 'rest-client'
# :first_in sets how long it takes before the job is first run. In this case, it is run immediately
current_nightscout = 0
SCHEDULER.every '1m' do
test = JSON.parse(open("https://dnarnianbg.herokuapp.com/api/v1/entries/current.json", :read_timeout => 4).read)
last_nightscout = current_nightscout
current_nightscout = p test[0]["sgv"]
send_event('nightscout', { current: current_nightscout, last: last_nightscout })
end
I can probably eliminate require 'rest-client' since that is no longer being used, but it works right now and that is all that matters.
I want to parse a string of complex JSON in Pig. Specifically, I want Pig to understand my JSON array as a bag instead of as a single chararray. I found that complex JSON can be parsed by using Twitter's Elephant Bird or Mozilla's Akela library. (I found some additional libraries, but I cannot use 'Loader' based approach since I use HCatalog Loader to load data from Hive.)
But, the problem is the structure of my data; each value of Map structure contains value part of complex JSON. For example,
1. My table looks like (WARNING: type of 'complex_data' is not STRING, a MAP of <STRING, STRING>!)
TABLE temp_table
(
user_id BIGINT COMMENT 'user ID.',
complex_data MAP <STRING, STRING> COMMENT 'complex json data'
)
COMMENT 'temp data.'
PARTITIONED BY(created_date STRING)
STORED AS RCFILE;
2. And 'complex_data' contains (a value that I want to get is marked with two *s, so basically #'d'#'f' from each PARSED_STRING(complex_data#'c') )
{ "a": "[]",
"b": "\"sdf\"",
"**c**":"[{\"**d**\":{\"e\":\"sdfsdf\"
,\"**f**\":\"sdfs\"
,\"g\":\"qweqweqwe\"},
\"c\":[{\"d\":21321,\"e\":\"ewrwer\"},
{\"d\":21321,\"e\":\"ewrwer\"},
{\"d\":21321,\"e\":\"ewrwer\"}]
},
{\"**d**\":{\"e\":\"sdfsdf\"
,\"**f**\":\"sdfs\"
,\"g\":\"qweqweqwe\"},
\"c\":[{\"d\":21321,\"e\":\"ewrwer\"},
{\"d\":21321,\"e\":\"ewrwer\"},
{\"d\":21321,\"e\":\"ewrwer\"}]
},]"
}
3. So, I tried... (same approach for Elephant Bird)
REGISTER '/path/to/akela-0.6-SNAPSHOT.jar';
DEFINE JsonTupleMap com.mozilla.pig.eval.json.JsonTupleMap();
data = LOAD temp_table USING org.apache.hive.hcatalog.pig.HCatLoader();
values_of_map = FOREACH data GENERATE complex_data#'c' AS attr:chararray; -- IT WORKS
-- dump values_of_map shows correct chararray data per each row
-- eg) ([{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... }])
([{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... }]) ...
attempt1 = FOREACH data GENERATE JsonTupleMap(complex_data#'c'); -- THIS LINE CAUSE AN ERROR
attempt2 = FOREACH data GENERATE JsonTupleMap(CONCAT(CONCAT('{\\"key\\":', complex_data#'c'), '}'); -- IT ALSO DOSE NOT WORK
I guessed that "attempt1" was failed because the value doesn't contain full JSON. However, when I CONCAT like "attempt2", I generate additional \ mark with. (so each line starts with {\"key\": ) I'm not sure that this additional marks breaks the parsing rule or not. In any case, I want to parse the given JSON string so that Pig can understand. If you have any method or solution, please Feel free to let me know.
I finally solved my problem by using jyson library with jython UDF.
I know that I can solve it by using JAVA or other languages.
But, I think that jython with jyson is the most simplist answer to this issue.
I've stared at this so long I'm going in circles...
I'm using the rbvmomi gem, and in Pry, when I display an object, it recurses down thru the structure showing me the nested objects - but to_json seems to "dig down" into some objects, but just dump the reference for others> Here's an example:
[24] pry(main)> g
=> [GuestNicInfo(
connected: true,
deviceConfigId: 4000,
dynamicProperty: [],
ipAddress: ["10.102.155.146"],
ipConfig: NetIpConfigInfo(
dynamicProperty: [],
ipAddress: [NetIpConfigInfoIpAddress(
dynamicProperty: [],
ipAddress: "10.102.155.146",
prefixLength: 20,
state: "preferred"
)]
),
macAddress: "00:50:56:a0:56:9d",
network: "F5_Real_VM_IPs"
)]
[25] pry(main)> g.to_json
=> "[\"#<RbVmomi::VIM::GuestNicInfo:0x000000085ecc68>\"]"
Pry apparently just uses a souped-up pp, and while "pp g" gives me close to what I want, I'm kinda steering as hard as I can toward json so that I don't need a custom parser to load up and manipulate the results.
The question is - how can I get the json module to dig down like pp does? And if the answer is "you can't" - any other suggestions for achieving the goal? I'm not married to json - if I can get the data serialized and read it back later (without writing something to parse pp output... which may already exist and I should look for it), then it's all win.
My "real" goal here is to slurp up a bunch of info from our vsphere stuff via rbvmomi so that I can do some network/vm analysis on it, which is why I'd like to get it in a nice machine-parsed format. If I'm doing something stupid here and there's an easier way to go about this - lay it on me, I'm not proud. Thank you all for your time and attention.
Update: Based on Arnie's response, I added this monkeypatch to my script:
class RbVmomi::BasicTypes::DataObject
def to_json(*args)
h = self.props
m = h.merge({ JSON.create_id => self.class.name })
m.to_json(*args)
end
end
and now my to_json recurses down nicely. I'll see about submitting this (or the def, really) to the project.
The .to_json works in a recursive manner, the default behavior is defined as:
Converts this object to a string (calling to_s), converts it to a JSON string, and returns the result. This is a fallback, if no special method to_json was defined for some object.
json library has added some implementation for some common classes (check the left hand side of this documentation), such as Array, Range, DateTime.
For an array, to_json first convert all the elements to json object, concat then together, and then add the array mark [/].
For your case, you need to define your customized to_json method for GuestNicInfo, NetIpConfigInfo and NetIpConfigInfoIpAddress. I don't know your implementation about these three classes, so I wrote a example to demonstrate how to achieve this:
require 'json'
class MyClass
attr_accessor :a, :b
def initialize(a, b)
#a = a
#b = b
end
end
data = [MyClass.new(1, "foobar")]
puts data.to_json
#=> ["#<MyClass:0x007fb6626c7260>"]
class MyClass
def to_json(*args)
{
JSON.create_id => self.class.name,
:a => a,
:b => b
}.to_json(*args)
end
end
puts data.to_json
#=> [{"json_class":"MyClass","a":1,"b":"foobar"}]