Related
First, I have a json:
json = "{\"string_1\": \"{{string_1_value}}\", \"number_1\": \"{{number_1_value}}\"}"
And this hash:
hash = {
"{{string_1_value}}" => "test" //string
"{{number_1_value}}" => 2 //integer
}
What I'd like to do is to replace json with this hash and generate below json.
"{\"string_1\": \"test\", \"number_1\": 2}"
When I do this by String#gsub, I got an Error.
hash.map {|k, v| json.gsub!(k, v)}
=> TypeError (no implicit conversion of Integer into String)
I don't want 2 to be string, i.e.)
"{"string_1": "test", "number_1": "2"}"
Do you have any idea?
Thank you in advance.
First, in ruby comments are marked by # not //. And remember about the comma in hash.
gsub is not the fastest way to replace things, it's better to convert json to regular hash and then convert it again to json.
require 'json'
json = "{\"string_1\": \"{{string_1_value}}\", \"number_1\": \"{{number_1_value}}\"}"
hash = {
"{{string_1_value}}" => "test", #string
"{{number_1_value}}" => 2 #integer
}
# First you should parse your json and change it to hash:
parsed_json = JSON.parse(json)
# Then create keys array
keys = parsed_json.keys
# Create new empty hash
new_hash = {}
# And now fill new hash with keys and values
# (take a look at to_s, it converts values to a String)
hash.each.with_index do |(_k, v), i|
new_hash[keys[i]] = v.to_s
end
# Convert to json at the end
new_hash.to_json
# => "{\"string_1\":\"test\",\"number_1\":\"2\"}"
You can use the Regexp,Hash version of String#gsub to just substitute the patterns with the desired values as follows:
require 'json'
json_string = "{\"string_1\": \"{{string_1_value}}\", \"number_1\": \"{{number_1_value}}\"}"
original_hash= {
"{{string_1_value}}" => "test", #string
"{{number_1_value}}" => 2 #integer
}
#Convert JSON to hash and invert the key value pairs
parsed_json = JSON.parse(json_string).invert
#=>{"{{string_1_value}}"=>"string_1", "{{number_1_value}}"=>"number_1"}
# Convert the hash to JSON and substitute the patterns
original_hash.to_json.gsub(/\{\{.+?\}\}/, parsed_json)
#=> "{\"string_1\":\"test\",\"number_1\":2}"
I am using a subroutine (stats) to calculate statistics for a list of numbers.
These numbers may be big enough to lose precision if stored as normal perl numbers.
I recieve such numbers as JSON formatted strings.
To decode these strings without losing precision,
I use a JSON::PP object with allow_nonref and allow_bignum activated.
I send the list of such decoded numbers to stats subroutine
(see in code shown below).
This routine calculates some statistics.
These statistics are then encoded to JSON and saved to file.
Most of the time the process seems to work correctly, but
for some inputs (see code for examples) the calculated value of mean and variance statistics
are either clearly wrong, or are encoded as JSON strings by the encoder, or both.
I suspect this is due to interaction of Math::BigInt and Math::BigFloat objects created by JSON decode, and List::Util::sum0.
I am trying to figure out what causes this and a way to avoid/fix this,
preferably without resorting to big non core modules.
I am willing to accept imprecise calculation of mean and variance,
but not entirely inaccurate results
or numerical results encoded as string in JSON.
A script (stats.pl) to demonstrate the problem:
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Varname = "DUMPED_RAWDATA";
use JSON::PP;
use List::Util;
my $JSON = JSON::PP->new->allow_bignum->utf8->pretty->canonical;
sub stats {
#TODO fix bug about negative variance. AVOID OVERFLOW
#TODO use GMP, XS?
# #_ has decoded numbers (called RAWDATA here)
my $n = scalar #_;
my $sum = List::Util::sum0(#_);
my $mean = $sum / $n;
my $var = List::Util::sum0( map { $_**2 } #_ ) / $n - $mean**2;
my $s = {
n => $n,
sum => $sum,
max => List::Util::max(#_),
min => List::Util::min(#_),
mean => $mean,
variance => $var
};
# DUMP STATE IF SOME ERROR OCCURS
print Dumper( \#_ ),
$JSON->encode( { json_encoded_stats => $s, json_encoded_rawdata => \#_ } )
if ( '"' eq substr( $JSON->encode($var), 0, 1 ) #MEAN ENCODED AS STRING
or '"' eq substr( $JSON->encode($mean), 0, 1 ) #VARIANCE ENCODED AS STRING
or $var < 0 ); #VARIANCE IS NEGATIVE!
$s;
}
my #test = (
[
qw( 919300112739897344 919305709216464896 919305709216464896 985592115567603712 959299136196456448)
],
[qw(479655558 429035600 3281034608 3281034608 2606592908 3490045576)],
[ qw(914426431563644928) x 3142 ]
);
for (#test) {
print "---\n";
stats( map { $JSON->decode($_) } #$_ );
}
Below is the curtailed output of perl stats.pl with problems indicated as <---.
---
$DUMPED_RAWDATA1 = [
'919300112739897344',
'919305709216464896',
'919305709216464896',
'985592115567603712',
'959299136196456448'
];
{
"json_encoded_rawdata" : [
919300112739897344,
919305709216464896,
919305709216464896,
985592115567603712,
959299136196456448
],
"json_encoded_stats" : {
"max" : 985592115567603712,
"mean" : "9.40560556587377e+17", <--- ENCODED AS STRING
"min" : 919300112739897344,
"n" : 5,
"sum" : 4702802782936887296,
"variance" : 7.46903843214008e+32
}
}
---
$DUMPED_RAWDATA1 = [
479655558,
429035600,
3281034608,
3281034608,
2606592908,
3490045576
];
{
"json_encoded_rawdata" : [
479655558,
429035600,
3281034608,
3281034608,
2606592908,
3490045576
],
"json_encoded_stats" : {
"max" : 3490045576,
"mean" : 2261233143,
"min" : 429035600,
"n" : 6,
"sum" : 13567398858,
"variance" : "-1.36775568782523e+18" <--- NEGATIVE VARIANCE, STRING ENCODED
}
}
---
$DUMPED_RAWDATA1 = [
'914426431563644928',
.
.
.
<snip 3140 identical lines>
'914426431563644928'
];
{
"json_encoded_rawdata" : [
914426431563644928,
.
.
.
<snip 3140 identical lines>
914426431563644928
],
"json_encoded_stats" : {
"max" : 914426431563644928,
"mean" : "9.14426431563676e+17", <--- STRING ENCODED
"min" : 914426431563644928,
"n" : 3142,
"sum" : 2.87312784797307e+21,
"variance" : -9.75463826617761e+22 <--- NEGATIVE VARIANCE
}
}
None of your inputs are big enough to require JSON::PP to create Math::BigInt objects on a system with 64-bit ints, so it doesn't.
You could do something like the following at the start of your sub.
#_ = map { Math::BigInt->new($_) } #_; # Or ::BigFloat?
Alternatively,
my $zero_B = Math::BigInt->new(0);
sub stats {
my $n = #_;
my $sum_B = sum($zero_B, #_);
my $mean_B = $sum_B / $n;
my $var_B = sum( map { Math::BigInt->new($_) ** 2 } #_ ) / $n - $mean_B ** 2;
my ($min, $max) = minmax(#_);
return {
n => $n,
sum => $sum_B,
max => $max,
min => $min,
mean => $mean_B,
variance => $var_B,
};
}
All together:
use strict;
use warnings;
use Data::Dumper qw( Dumper );
use JSON::PP qw( );
use List::MoreUtils qw( minmax );
use List::Util qw( sum );
use Math::BigInt qw( );
my $zero_B = Math::BigInt->new(0);
my $JSON = JSON::PP->new->allow_bignum->utf8->pretty->canonical;
sub stats {
my $n = #_;
my $sum_B = sum($zero_B, #_);
my $mean_B = $sum_B / $n;
my $var_B = sum( map { Math::BigInt->new($_) ** 2 } #_ ) / $n - $mean_B ** 2;
my ($min, $max) = minmax(#_);
return {
n => $n,
sum => $sum_B,
max => $max,
min => $min,
mean => $mean_B,
variance => $var_B,
};
}
my #test = (
[qw( 919300112739897344 919305709216464896 919305709216464896 985592115567603712 959299136196456448 )],
[qw( 479655558 429035600 3281034608 3281034608 2606592908 3490045576 )],
[ qw( 914426431563644928 ) x 3142 ]
);
for (#test) {
print "---\n";
my $s = stats( map { $JSON->decode($_) } #$_ );
if (
$JSON->encode($s->{variance}) =~ /"/ # MEAN ENCODED AS STRING
|| $JSON->encode($s->{mean}) =~ /"/ # VARIANCE ENCODED AS STRING
|| $s->{variance} < 0 # VARIANCE IS NEGATIVE!
) {
local $Data::Dumper::Varname = "DUMPED_RAWDATA";
print Dumper($_);
print $JSON->encode({
json_encoded_rawdata => $_,
json_encoded_stats => $s,
});
} else {
print "ok\n";
}
}
Notes:
Both approaches will work even if the objects are already Math::* objects.
I identified the vars are guaranteed to contain a Math:Big* object using _B for clarity.
I moved the testing code to the test harness.
I used minmax because it's more efficient than calling min and max separately.
I imported the subs from the modules to avoid having to use use their full name.
No need to force something in scalar context into scalar context.
#ikegami's answer works correctly
but it is too slow for me as this subroutine
is called a lot of times in my program's inner loop.
I think that is the cost of ensuring that
all numbers are converted to arbitrary precision ones.
I ended up using the following implementation
which avoids converting all numbers to arbitrary
precision type.
sub stats {
my $n = scalar #_;
my $sum = List::Util::sum0(#_);
my $mean = $sum / $n;
my $var = List::Util::sum0( map { ( $_ - $mean )**2 } #_ ) / $n;
$mean += 0;
$var += 0; # TO ENSURE THAT THEY ARE ENCODED AS NUMBERS IN JSON
{
n => $n,
sum => $sum,
max => List::Util::max(#_),
min => List::Util::min(#_),
mean => $mean,
variance => $var,
};
}
I changed the method of calculating variance
to ensure that negative results are avoided
(as suggested by #Robert).
It may sacrifice precision in $sum
(and everything that depends on $sum)
due to floating point addition of large integers.
It completes the job in an acceptable execution time though.
The unintended JSON encoding of numbers as strings
is explained in https://metacpan.org/pod/JSON::PP#simple-scalars.
This problem is solved by using the method
suggested there to force encoding as numbers.
JSON::PP will encode undefined scalars as JSON null values, scalars
that have last been used in a string context before encoding as JSON
strings, and anything else as number value
You can force the type to be a JSON number by numifying it:
my $x = "3"; # some variable containing a string
$x += 0; # numify it, ensuring it will be dumped as a number
$x *= 1; # same thing, the choice is yours. in to force
Logstash is awesome. I can send it JSON like this (multi-lined for readability):
{
"a": "one"
"b": {
"alpha":"awesome"
}
}
And then query for that line in kibana using the search term b.alpha:awesome. Nice.
However I now have a JSON log line like this:
{
"different":[
{
"this": "one",
"that": "uno"
},
{
"this": "two"
}
]
}
And I'd like to be able to find this line with a search like different.this:two (or different.this:one, or different.that:uno)
If I was using Lucene directly I'd iterate through the different array, and generate a new search index for each hash within it, but Logstash currently seems to ingest that line like this:
different: {this: one, that: uno}, {this: two}
Which isn't going to help me searching for log lines using different.this or different.that.
Any got any thoughts as to a codec, filter or code change I can make to enable this?
You can write your own filter (copy & paste, rename the class name, the config_name and rewrite the filter(event) method) or modify the current JSON filter (source on Github)
You can find the JSON filter (Ruby class) source code in the following path logstash-1.x.x\lib\logstash\filters named as json.rb. The JSON filter parse the content as JSON as follows
begin
# TODO(sissel): Note, this will not successfully handle json lists
# like your text is '[ 1,2,3 ]' JSON.parse gives you an array (correctly)
# which won't merge into a hash. If someone needs this, we can fix it
# later.
dest.merge!(JSON.parse(source))
# If no target, we target the root of the event object. This can allow
# you to overwrite #timestamp. If so, let's parse it as a timestamp!
if !#target && event[TIMESTAMP].is_a?(String)
# This is a hack to help folks who are mucking with #timestamp during
# their json filter. You aren't supposed to do anything with
# "#timestamp" outside of the date filter, but nobody listens... ;)
event[TIMESTAMP] = Time.parse(event[TIMESTAMP]).utc
end
filter_matched(event)
rescue => e
event.tag("_jsonparsefailure")
#logger.warn("Trouble parsing json", :source => #source,
:raw => event[#source], :exception => e)
return
end
You can modify the parsing procedure to modify the original JSON
json = JSON.parse(source)
if json.is_a?(Hash)
json.each do |key, value|
if value.is_a?(Array)
value.each_with_index do |object, index|
#modify as you need
object["index"]=index
end
end
end
end
#save modified json
......
dest.merge!(json)
then you can modify your config file to use the/your new/modified JSON filter and place in \logstash-1.x.x\lib\logstash\config
This is mine elastic_with_json.conf with a modified json.rb filter
input{
stdin{
}
}filter{
json{
source => "message"
}
}output{
elasticsearch{
host=>localhost
}stdout{
}
}
if you want to use your new filter you can configure it with the config_name
class LogStash::Filters::Json_index < LogStash::Filters::Base
config_name "json_index"
milestone 2
....
end
and configure it
input{
stdin{
}
}filter{
json_index{
source => "message"
}
}output{
elasticsearch{
host=>localhost
}stdout{
}
}
Hope this helps.
For a quick and dirty hack, I used the Ruby filter and below code , no need to use the out of box 'json' filter anymore
input {
stdin{}
}
filter {
grok {
match => ["message","(?<json_raw>.*)"]
}
ruby {
init => "
def parse_json obj, pname=nil, event
obj = JSON.parse(obj) unless obj.is_a? Hash
obj = obj.to_hash unless obj.is_a? Hash
obj.each {|k,v|
p = pname.nil?? k : pname
if v.is_a? Array
v.each_with_index {|oo,ii|
parse_json_array(oo,ii,p,event)
}
elsif v.is_a? Hash
parse_json(v,p,event)
else
p = pname.nil?? k : [pname,k].join('.')
event[p] = v
end
}
end
def parse_json_array obj, i,pname, event
obj = JSON.parse(obj) unless obj.is_a? Hash
pname_ = pname
if obj.is_a? Hash
obj.each {|k,v|
p=[pname_,i,k].join('.')
if v.is_a? Array
v.each_with_index {|oo,ii|
parse_json_array(oo,ii,p,event)
}
elsif v.is_a? Hash
parse_json(v,p, event)
else
event[p] = v
end
}
else
n = [pname_, i].join('.')
event[n] = obj
end
end
"
code => "parse_json(event['json_raw'].to_s,nil,event) if event['json_raw'].to_s.include? ':'"
}
}
output {
stdout{codec => rubydebug}
}
Test json structure
{"id":123, "members":[{"i":1, "arr":[{"ii":11},{"ii":22}]},{"i":2}], "im_json":{"id":234, "members":[{"i":3},{"i":4}]}}
and this is whats output
{
"message" => "{\"id\":123, \"members\":[{\"i\":1, \"arr\":[{\"ii\":11},{\"ii\":22}]},{\"i\":2}], \"im_json\":{\"id\":234, \"members\":[{\"i\":3},{\"i\":4}]}}",
"#version" => "1",
"#timestamp" => "2014-07-25T00:06:00.814Z",
"host" => "Leis-MacBook-Pro.local",
"json_raw" => "{\"id\":123, \"members\":[{\"i\":1, \"arr\":[{\"ii\":11},{\"ii\":22}]},{\"i\":2}], \"im_json\":{\"id\":234, \"members\":[{\"i\":3},{\"i\":4}]}}",
"id" => 123,
"members.0.i" => 1,
"members.0.arr.0.ii" => 11,
"members.0.arr.1.ii" => 22,
"members.1.i" => 2,
"im_json" => 234,
"im_json.0.i" => 3,
"im_json.1.i" => 4
}
The solution I liked is the ruby filter because that requires us to not write another filter. However, that solution creates fields that are on the "root" of JSON and it's hard to keep track of how the original document looked.
I came up with something similar that's easier to follow and is a recursive solution so it's cleaner.
ruby {
init => "
def arrays_to_hash(h)
h.each do |k,v|
# If v is nil, an array is being iterated and the value is k.
# If v is not nil, a hash is being iterated and the value is v.
value = v || k
if value.is_a?(Array)
# "value" is replaced with "value_hash" later.
value_hash = {}
value.each_with_index do |v, i|
value_hash[i.to_s] = v
end
h[k] = value_hash
end
if value.is_a?(Hash) || value.is_a?(Array)
arrays_to_hash(value)
end
end
end
"
code => "arrays_to_hash(event.to_hash)"
}
It converts arrays to has with each key as the index number. More details:- http://blog.abhijeetr.com/2016/11/logstashelasticsearch-best-way-to.html
I want to write a CSV with ruby's default CSV library in order to use MySQL's fast import LOAD DATA INFILE.
Currently, when I enter nil for a field, it is written as ...;;..., instead I want it to be ...;\N;... (capital N for NULL, not to be confused with \n newline).
CSV.open(product_updates_file_name, "wb", {col_sep: ";", headers: false, force_quotes: false}) do |product_csv|
product_csv << ["foo", nil, "bar"]
end
It currently leads to the decimal field being loaded as 0.00 in the database instead of NULL.
I know, I could set it NULL afterwards, but there are millions of rows and several columns affected, so I strongly prefere to write the CSV as MySQL expects me to:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
Try overriding nil's to_s method like so:
class << nil
def to_s
"my nil placeholder text"
end
end
All code using to_s will use this implementation for the value nil.
You could modify the CSV-methods:
require 'csv'
class Array
alias :old_to_csv :to_csv
#Extend to_csv for usage like ["foo", nil, "bar"].to_csv( :col_sep => ";")
def to_csv(options)
self.map{|s| s.nil? ? '\N' : s }.old_to_csv
end
end
class CSV
alias :old_push :<<
def <<(data)
case data
when Array
old_push data.map{|s| s.nil? ? '\N' : s }
else
old_push data
end
end
end
#Testcode:
puts ["foo", nil, "bar"].to_csv( :col_sep => ";") #--> [["foo", "\\N", "bar"]]
CSV.open('test.csv', "wb",
{col_sep: ";", headers: false, force_quotes: false }
) do |product_csv|
product_csv << ["foo", nil, "bar"]
end
#-> Creates test.csv with 'foo;\N;bar'
This works only, if you insert Arrays. If you insert other stuff, you must modify the logic.
Remark:
My first idea was to use a converter. But it worked only for parsing a csv, not for writing.
CSV::Converters[:nil_N] = lambda{|s|
s.nil? ? '\N' : s
}
p CSV.parse('foo;;bar', :col_sep => ";", :converters => :nil_N)
#-> [["foo", "\\N", "bar"]]
Perhaps somebody else knows a way to use converters to build csv-files.
I am taking information from my MongoDB database (#bs). #bs has tons of information that I'm not interested, so what I need is to cycle trough all the information and create a new object with the information I need.
For that, I created a new array (#final) and I'm getting information and adding it to #final. The information seems to be getting there, however, when I convert it to JSON it's not a valid JSON object. What I intend to create in #final.json is this:
{ Something: [ {Email: "xxx#xxx.com", At: "date", ....}, {...}, ....] }
But when I do to_json I get [["At: date","Email: mail_test#tidgdfp.org","Message-id: .....
#bs = coll.find("headers.from" => email, "date" => {"$gte" => initial_date, "$lte" => Time.now.utc})
#bs = #bs.to_a.map { |obj| obj.delete("completo"); obj.delete("_id"); obj.delete("date"); obj.delete("headers" => "content_type"); obj }
#final = Array.new
#bs.each do |a|
elem = Array.new
elem << "At: #{a["date"]}"
elem << "Email: #{a["headers"]["to"]}"
elem << "Message: #{a["headers"]["message_id"]}"
elem << "Type: #{a["headers"]["status"]}"
#final << elem
end
puts #final
#final = #final.to_json
puts #final["Email"]
Please help.
Thanks
In your loop, create a hash rather than an array. to_json should make this a JSON Object.
#bs.each do |a|
#final << { :At => a['date'], :Email => a['headers']['to'], :Message => a['headers']['message_id'], :Type => a['headers']['status'] }
end