Looking for smarter way to build directory tree - json

I have large Json object. The object describes among other things a tree type relation of how its components objects are connected hierarchally. The object knows who its children are, but does not know (directly) who its parent is. "my_hash" below exemplifies the structure. Every object has an id 101, 102, etc., a name "one", "two" etc and it can have 0, 1 or more children. I am trying to build the "path" to the every object. E.g. object name "five" should a result have a path of "/one/two/four" as a result of the code. Basically I am trying to build a sort of directory structure of the hierarchy of the objects.
The code below works, but it looks quite long'ish, not very elegant, not very Ruby'ish.
I would be grate for suggestion on how to to do this more efficiently and elegantly. And I got hunch that my code may not be very robust, ie deal well with exceptions.
Any thoughts or help ar appreciated.
On a side note, I am just learning Ruby and so far have mainly programmed in Perl.
class Tree
def initialize
#my_hash = {
101 => ["one", [102, 107]],
102 => ["two", [103, 104]],
103 => ["three", []],
104 => ["four", [105, 106]],
105 => ["five", []],
106 => ["six", []],
107 => ["seven", [108]],
108 => ["eight", []],
}
#child_to_parent_node = {}
#id_to_name = {}
#my_path_hash = {}
#my_hash.keys.each do |key|
#my_path_hash[key] = ""
end
#parent_path_id = []
end
def map_child_to_parent
#my_hash.each do |key, value|
#id_to_name.store(key, value[0])
node_name, children = value[0], value[1]
children.each do |child_id|
#child_to_parent_node.store(child_id, node_name)
end
end
end
def build_path(id)
parent = #child_to_parent_node[id]
parent.nil? ? return : #parent_path_id << parent
id = #id_to_name.key(parent)
build_path(id)
#parent_path_id
end
def update_tree
#id_to_name.keys.each do |id|
tmp_array = self.build_path(id)
path = ""
if (tmp_array.nil?)
path = "/"
else
tmp_array.reverse.each do
path = path + "/" + tmp_array.pop
end
end
puts "id: #{id} path: #{path}"
end
end
end
my_tree = Tree.new
my_tree.map_child_to_parent
my_tree.update_tree

In fact, to solve this task you have to traverse the tree from the leaf to root, right? So, your representation of the tree is very inconvenient for this particular task. If you are ok to trade off some memory for a more clean solution, I'd create an auxiliary structure that contains parents for each node. Let's say
#parents = #my_hash.each_with_object({}) do |(pid, props), acc|
_, children = props
children.each { |cid| acc[cid] = pid }
end
#=> {102=>101, 107=>101, 103=>102, 104=>102, 105=>104, 106=>104, 108=>107}
Now, the task can be solved in a quite concise way with a couple of auxiliary functions. For example:
def id_by_name(name)
id, _ = #my_hash.find { |k, v| v.first == name }
id
end
def name_by_id(id)
#my_hash[id].first
end
def path_to(node)
path = [node]
id = id_by_name(node)
path.unshift(name_by_id(id)) while id = #parents[id]
path.join("/")
end
path_to "five" #=> "one/two/four/five"
Please, note: the solution is still very inefficient - mostly because to fetch a node's id by its name we have to iterate over the whole initial hash in the worst case. This is the price we pay for the data structure that doesn't fit the task well.

Related

Stuck trying to display info on a CLI from an API that uses a hash and a nested hash

Trying to display second level information about characters from this Futurama API.
Currently using this code to get information:
def self.character
uri = URI.parse(URL)
response = Net::HTTP.get_response(uri)
data = JSON.parse(response.body)
data.each do |c|
Character.new(c["name"], c["gender"], c["species"], c["homePlanet"], c["occupation"], c["info"], c["sayings"])
end
end
I'm then stuck either returning (gender and species) from the nested hash (if character id > 8) or the original hash (character id < 8) when using this code:
def character_details(character)
puts "Name: #{character.name["first"]} #{character.name["middle"]} #{character.name["last"]}"
puts "Species: #{character.info["species"]}"
puts "Occupation: #{character.homePlanet}"
puts "Gender: #{character.info["gender"]}"
puts "Quotes:"
character.sayings.each_with_index do |s, i|
iplusone = i + 1
puts "#{iplusone}. #{s} "
end
end
Not sure where or what logic to use to get the correct information to display.
Maybe you have a problem when save c['info] in Character.new(c["name"], c["gender"], c["species"], c["homePlanet"], c["occupation"], c["info"], c["sayings"])
I'm running your code and I see info does not exist in the response of API, the gender should be accessed in character.gender
irb(main):037:0> character.gender
=> "Male"
irb(main):039:0> character.species
=> "Human"
I don't understand this comment: (if character id > 8) or the original hash (character id < 8) Can you explain us what u need do?

In Elixir, How can I extract a lambda to a named function when the lambda is in a closure?

I have the following closure:
def get!(Item, id) do
Enum.find(
#items,
fn(item) -> item.id == id end
)
end
As I believe this looks ugly and difficult to read, I'd like to give this a name, like:
def get!(Item, id) do
defp has_target_id?(item), do: item.id = id
Enum.find(#items, has_target_id?/1)
end
Unfortunately, this results in:
== Compilation error in file lib/auction/fake_repo.ex ==
** (ArgumentError) cannot invoke defp/2 inside function/macro
(elixir) lib/kernel.ex:5238: Kernel.assert_no_function_scope/3
(elixir) lib/kernel.ex:4155: Kernel.define/4
(elixir) expanding macro: Kernel.defp/2
lib/auction/fake_repo.ex:28: Auction.FakeRepo.get!/2
Assuming it is possible, what is the correct way to do this?
The code you posted has an enormous amount of syntax errors/glitches. I would suggest you start with getting accustomed to the syntax, rather than trying to make Elixir better by inventing the things that nobody uses.
Here is the correct version that does what you wanted. The task might be accomplished with an anonymous function, although I hardly see a reason to make a perfectly looking idiomatic Elixir look ugly.
defmodule Foo do
#items [%{id: 1}, %{id: 2}, %{id: 3}]
def get!(id) do
has_target_id? = fn item -> item.id == id end
Enum.find(#items, has_target_id?)
end
end
Foo.get! 1
#⇒ %{id: 1}
Foo.get! 4
#⇒ nil
You can do this:
def get!(Item, id) do
Enum.find(
#items,
&compare_ids(&1, id)
)
end
defp compare_ids(%Item{}=item, id) do
item.id == id
end
But, that's equivalent to:
Enum.find(
#items,
fn item -> compare_ids(item, id) end
)
and may not pass your looks ugly and difficult to read test.
I was somehow under the impression Elixir supports nested functions?
Easy enough to test:
defmodule A do
def go do
def greet do
IO.puts "hello"
end
greet()
end
end
Same error:
$ iex a.ex
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
** (ArgumentError) cannot invoke def/2 inside function/macro
(elixir) lib/kernel.ex:5150: Kernel.assert_no_function_scope/3
(elixir) lib/kernel.ex:3906: Kernel.define/4
(elixir) expanding macro: Kernel.def/2
a.ex:3: A.go/0
wouldn't:
defp compare_ids(item, id), do: item.id == id
be enough? Is there any advantage to including %Item{} or making
separate functions for returning both true and false conditions?
What you gain by specifying the first parameter as:
func(%Item{} = item, target_id)
is that only an Item struct will match the first parameter. Here is an example:
defmodule Item do
defstruct [:id, :name, :description]
end
defmodule Dog do
defstruct [:id, :name, :owner]
end
defmodule A do
def go(%Item{} = item), do: IO.inspect(item.id, label: "id: ")
end
In iex:
iex(1)> item = %Item{id: 1, name: "book", description: "old"}
%Item{description: "old", id: 1, name: "book"}
iex(2)> dog = %Dog{id: 1, name: "fido", owner: "joe"}
%Dog{id: 1, name: "fido", owner: "joe"}
iex(3)> A.go item
id: : 1
1
iex(4)> A.go dog
** (FunctionClauseError) no function clause matching in A.go/1
The following arguments were given to A.go/1:
# 1
%Dog{id: 1, name: "fido", owner: "joe"}
a.ex:10: A.go/1
iex(4)>
You get a function clause error if you call the function with a non-Item, and the earlier an error occurs, the better, because it makes debugging easier.
Of course, by preventing the function from accepting other structs, you make the function less general--but because it's a private function, you can't call it from outside the module anyway. On the other hand, if you wanted to call the function on both Dog and Item structs, then you could simply specify the first parameter as:
|
V
func(%{}=thing, target_id)
then both an Item and a Dog would match--but not non-maps.
What you gain by specifying the first parameter as:
|
V
func(%Item{id: id}, target_id)
is that you let erlang's pattern matching engine extract the data you need, rather than calling item.id as you would need to do with this definition:
func(%Item{}=item, target_id)
In erlang, pattern matching in a parameter list is the most efficient/convenient/stylish way to write functions. You use pattern matching to extract the data that you want to use in the function body.
Going even further, if you write the function definition like this:
same variable name
| |
V V
func(%Item{id: target_id}, target_id)
then erlang's pattern matching engine not only extracts the value for the id field from the Item struct, but also checks that the value is equal to the value of the target_id variable in the 2nd argument.
Defining multiple function clauses is a common idiom in erlang, and it is considered good style because it takes advantage of pattern matching rather than logic inside the function body. Here's an erlang example:
get_evens(List) ->
get_evens(List, []).
get_evens([Head|Tail], Results) when Head rem 2 == 0 ->
get_evens(Tail, [Head|Results]);
get_evens([Head|Tail], Results) when Head rem 2 =/= 0 ->
get_evens(Tail, Results);
get_evens([], Results) ->
lists:reverse(Results).

ruby sketchup scene serialization

I am very new on Sketchup and ruby , I have worked with java and c# but this is the first time with ruby.
Now I have one problem, I need to serialize all scene in one json (scene hierarchy, object name, object material and position this for single object) how can I do this?
I have already done this for unity3D (c#) without a problem.
I tried this:
def main
avr_entities = Sketchup.active_model.entities # all objects
ambiens_dictionary = {}
ambiens_list = []
avr_entities.each do |root|
if root.is_a?(Sketchup::Group) || root.is_a?(Sketchup::ComponentInstance)
if root.name == ""
UI.messagebox("this is a group #{root.definition.name}")
if root.entities.count > 0
root.entities.each do |leaf|
if leaf.is_a?(Sketchup::Group) || leaf.is_a?(Sketchup::ComponentInstance)
UI.messagebox("this is a leaf #{leaf.definition.name}")
end
end
end
else
# UI.messagebox("this is a leaf #{root.name}")
end
end
end
end
Have you tried the JSON library
require 'json'
source = { a: [ { b: "hello" }, 1, "world" ], c: 'hi' }.to_json
source.to_json # => "{\"a\":[{\"b\":\"hello\"},1,\"world\"],\"c\":\"hi\"}"
Used the code below to answer a question Here, but it might also work here.
The code can run outside of SketchUp for testing in the terminal. Just make sure to follow these steps...
Copy the code below and paste it on a ruby file (example: file.rb)
Run the script in terminal ruby file.rb.
The script will write data to JSON file and also read the content of JSON file.
The path to the JSON file is relative to the ruby file created in step one. If the script can't find the path it will create the JSON file for you.
module DeveloperName
module PluginName
require 'json'
require 'fileutils'
class Main
def initialize
path = File.dirname(__FILE__)
#json = File.join(path, 'file.json')
#content = { 'hello' => 'hello world' }.to_json
json_create(#content)
json_read(#json)
end
def json_create(content)
File.open(#json, 'w') { |f| f.write(content) }
end
def json_read(json)
if File.exist?(json)
file = File.read(json)
data_hash = JSON.parse(file)
puts "Json content: #{data_hash}"
else
msg = 'JSON file not found'
UI.messagebox(msg, MB_OK)
end
end
# # #
end
DeveloperName::PluginName::Main.new
end
end

Logstash indexing JSON arrays

Logstash is awesome. I can send it JSON like this (multi-lined for readability):
{
"a": "one"
"b": {
"alpha":"awesome"
}
}
And then query for that line in kibana using the search term b.alpha:awesome. Nice.
However I now have a JSON log line like this:
{
"different":[
{
"this": "one",
"that": "uno"
},
{
"this": "two"
}
]
}
And I'd like to be able to find this line with a search like different.this:two (or different.this:one, or different.that:uno)
If I was using Lucene directly I'd iterate through the different array, and generate a new search index for each hash within it, but Logstash currently seems to ingest that line like this:
different: {this: one, that: uno}, {this: two}
Which isn't going to help me searching for log lines using different.this or different.that.
Any got any thoughts as to a codec, filter or code change I can make to enable this?
You can write your own filter (copy & paste, rename the class name, the config_name and rewrite the filter(event) method) or modify the current JSON filter (source on Github)
You can find the JSON filter (Ruby class) source code in the following path logstash-1.x.x\lib\logstash\filters named as json.rb. The JSON filter parse the content as JSON as follows
begin
# TODO(sissel): Note, this will not successfully handle json lists
# like your text is '[ 1,2,3 ]' JSON.parse gives you an array (correctly)
# which won't merge into a hash. If someone needs this, we can fix it
# later.
dest.merge!(JSON.parse(source))
# If no target, we target the root of the event object. This can allow
# you to overwrite #timestamp. If so, let's parse it as a timestamp!
if !#target && event[TIMESTAMP].is_a?(String)
# This is a hack to help folks who are mucking with #timestamp during
# their json filter. You aren't supposed to do anything with
# "#timestamp" outside of the date filter, but nobody listens... ;)
event[TIMESTAMP] = Time.parse(event[TIMESTAMP]).utc
end
filter_matched(event)
rescue => e
event.tag("_jsonparsefailure")
#logger.warn("Trouble parsing json", :source => #source,
:raw => event[#source], :exception => e)
return
end
You can modify the parsing procedure to modify the original JSON
json = JSON.parse(source)
if json.is_a?(Hash)
json.each do |key, value|
if value.is_a?(Array)
value.each_with_index do |object, index|
#modify as you need
object["index"]=index
end
end
end
end
#save modified json
......
dest.merge!(json)
then you can modify your config file to use the/your new/modified JSON filter and place in \logstash-1.x.x\lib\logstash\config
This is mine elastic_with_json.conf with a modified json.rb filter
input{
stdin{
}
}filter{
json{
source => "message"
}
}output{
elasticsearch{
host=>localhost
}stdout{
}
}
if you want to use your new filter you can configure it with the config_name
class LogStash::Filters::Json_index < LogStash::Filters::Base
config_name "json_index"
milestone 2
....
end
and configure it
input{
stdin{
}
}filter{
json_index{
source => "message"
}
}output{
elasticsearch{
host=>localhost
}stdout{
}
}
Hope this helps.
For a quick and dirty hack, I used the Ruby filter and below code , no need to use the out of box 'json' filter anymore
input {
stdin{}
}
filter {
grok {
match => ["message","(?<json_raw>.*)"]
}
ruby {
init => "
def parse_json obj, pname=nil, event
obj = JSON.parse(obj) unless obj.is_a? Hash
obj = obj.to_hash unless obj.is_a? Hash
obj.each {|k,v|
p = pname.nil?? k : pname
if v.is_a? Array
v.each_with_index {|oo,ii|
parse_json_array(oo,ii,p,event)
}
elsif v.is_a? Hash
parse_json(v,p,event)
else
p = pname.nil?? k : [pname,k].join('.')
event[p] = v
end
}
end
def parse_json_array obj, i,pname, event
obj = JSON.parse(obj) unless obj.is_a? Hash
pname_ = pname
if obj.is_a? Hash
obj.each {|k,v|
p=[pname_,i,k].join('.')
if v.is_a? Array
v.each_with_index {|oo,ii|
parse_json_array(oo,ii,p,event)
}
elsif v.is_a? Hash
parse_json(v,p, event)
else
event[p] = v
end
}
else
n = [pname_, i].join('.')
event[n] = obj
end
end
"
code => "parse_json(event['json_raw'].to_s,nil,event) if event['json_raw'].to_s.include? ':'"
}
}
output {
stdout{codec => rubydebug}
}
Test json structure
{"id":123, "members":[{"i":1, "arr":[{"ii":11},{"ii":22}]},{"i":2}], "im_json":{"id":234, "members":[{"i":3},{"i":4}]}}
and this is whats output
{
"message" => "{\"id\":123, \"members\":[{\"i\":1, \"arr\":[{\"ii\":11},{\"ii\":22}]},{\"i\":2}], \"im_json\":{\"id\":234, \"members\":[{\"i\":3},{\"i\":4}]}}",
"#version" => "1",
"#timestamp" => "2014-07-25T00:06:00.814Z",
"host" => "Leis-MacBook-Pro.local",
"json_raw" => "{\"id\":123, \"members\":[{\"i\":1, \"arr\":[{\"ii\":11},{\"ii\":22}]},{\"i\":2}], \"im_json\":{\"id\":234, \"members\":[{\"i\":3},{\"i\":4}]}}",
"id" => 123,
"members.0.i" => 1,
"members.0.arr.0.ii" => 11,
"members.0.arr.1.ii" => 22,
"members.1.i" => 2,
"im_json" => 234,
"im_json.0.i" => 3,
"im_json.1.i" => 4
}
The solution I liked is the ruby filter because that requires us to not write another filter. However, that solution creates fields that are on the "root" of JSON and it's hard to keep track of how the original document looked.
I came up with something similar that's easier to follow and is a recursive solution so it's cleaner.
ruby {
init => "
def arrays_to_hash(h)
h.each do |k,v|
# If v is nil, an array is being iterated and the value is k.
# If v is not nil, a hash is being iterated and the value is v.
value = v || k
if value.is_a?(Array)
# "value" is replaced with "value_hash" later.
value_hash = {}
value.each_with_index do |v, i|
value_hash[i.to_s] = v
end
h[k] = value_hash
end
if value.is_a?(Hash) || value.is_a?(Array)
arrays_to_hash(value)
end
end
end
"
code => "arrays_to_hash(event.to_hash)"
}
It converts arrays to has with each key as the index number. More details:- http://blog.abhijeetr.com/2016/11/logstashelasticsearch-best-way-to.html

java.lang.Object.equals and java.lang.Object.hashCode have trouble with jruby object after serialization

I have some trouble using jruby objects into java
java side
package com.pp;
public interface ZeroI {
boolean equals(Object o);
int hashCode();
int hash();
}
package com.pp;
public class Tester {
public Object[] compare(ZeroI one, ZeroI two) {
return new Object[] {one.hashCode(), two.hashCode(), one.equals(two), one == two};
}
}
jruby side
include Java
import com.pp.Tester
import com.pp.ZeroI
module MMM
module Zero
def hash= value
#hash = value
end
def hash
#hash
end
def hashCode
#hash
end
def equals other
false
end
def == other
true
end
end
class OneClass
include ZeroI
include Zero
end
class TwoClass
include ZeroI
include Zero
end
def self.create clazz
begin
dump = IO.readlines("C:/#{clazz.to_s.rpartition('::')[2]}.txt", '').to_s
instance = Marshal.load dump
rescue => err
puts err.message
instance = clazz.new
dump = Marshal.dump instance
File.open("C:/#{clazz.to_s.rpartition('::')[2]}.txt", 'w') { |f| f.write dump }
end
instance
end
tester = Tester.new
one = create OneClass
two = create TwoClass
puts one
puts two
one.hash = 22
two.hash = 22
puts one.hashCode
puts two.hashCode
puts one.equals two
puts one == two
tester.compare(one, two).each { |value| puts value }
end
First pass result:
No such file or directory - C:/OneClass.txt
No such file or directory - C:/TwoClass.txt
#<MMM::OneClass:0x1971eb3>
#<MMM::TwoClass:0x1408a75>
22
22
false
true
22
22
true
false
true # it's OK because JAVA.equals works with JRUBY.==
false # it's OK because org.pp.ZeroI can't declare == method and JAVA.== is used
Second pass result (with deserialized objects)
#<MMM::OneClass:0xd510e8>
#<MMM::TwoClass:0x490342>
22
22
false
true
13046738 # but what is it?
31877484 # but what is it?
false # but what is it?
false
Can anybody explain it?
I don't know all the details about why this happens like it happens but I have a solution/workaround for you. (I've seen similar behaviour when passing objects that are created on the ruby side to the Java side.)
As far as I can tell JRuby needs to have already "seen" an instance of the class it is trying to unmarshal before it can get the Java-inheritance side of things right. It's almost as if creating an object within JRuby has an undocumented side-effect that registers the required inheritance hierarchy. If that isn't well worded it's because I don't understand it myself!
So the workaround is to simply create an instance of OneClass and TwoClass before doing the unmarshal. If I change the self.create method to the following:
def self.create clazz
begin
clazz.new # <<< just create an instance and throw it away!
dump = IO.readlines("C:/#{clazz.to_s.rpartition('::')[2]}.txt", '').to_s
instance = Marshal.load dump
rescue => err
puts err.message
instance = clazz.new
dump = Marshal.dump instance
File.open("C:/#{clazz.to_s.rpartition('::')[2]}.txt", 'w') { |f| f.write dump }
end
instance
end
Then the output of the two passes are as follows:
First pass
No such file or directory - C:/OneClass.txt
No such file or directory - C:/TwoClass.txt
#<MMM::OneClass:0x4de6f0ef>
#<MMM::TwoClass:0x4526ba64>
22
22
false
true
22
22
true
false
Second pass
#<MMM::OneClass:0x4858cca9>
#<MMM::TwoClass:0x3de4905a>
22
22
false
true
22
22
true
false
According to this bug report this is scheduled to be fixed in JRuby 1.7. It's worth noting that while the comments in the report say that the workaround is to call a method, passing an object instance in, it seems to be that the prior creation of the object is enough.