Altering and updating a JSON file - json

I wrote a small script which loops my current Jsonfile but it's not possible to append a key and value.
This is my file
[
{
"URL": "https://p59-caldav.icloud.com",
}
]
I would like to append a key and value like this
[
{
"URL": "https://p59-caldav.icloud.com",
"test": "test"
}
]
My current script setup
#!/usr/bin/env ruby
require 'rubygems'
require 'json'
require 'domainatrix'
jsonHash = File.read("/Users/ReffasCode/Desktop/sampleData.json")
array = JSON.parse(jsonHash)
File.open("/Users/BilalReffas/Desktop/sampleData.json","w") do |f|
array.each do |child|
url = Domainatrix.parse(child["URL"])
json = {
"url" => child["URL"],
"provider" => url.domain,
"service" => url.subdomain,
"context" => url.path,
"suffix" => url.public_suffix
}
f.write(JSON.pretty_generate(json))
end
end
The script overwrite my whole jsonfile...this is not what I want :/

This is untested but looks about right:
ORIGINAL_JSON = 'sampleData.json'
NEW_JSON = ORIGINAL_JSON + '.new'
OLD_JSON = ORIGINAL_JSON + '.old'
json = JSON.parse(File.read(ORIGINAL_JSON))
ary = json.map do |child|
url = Domainatrix.parse(child['URL'])
{
'url' => child['URL'],
'provider' => url.domain,
'service' => url.subdomain,
'context' => url.path,
'suffix' => url.public_suffix
}
end
File.write(NEW_JSON, ary.to_json)
File.rename(ORIGINAL_JSON, OLD_JSON)
File.rename(NEW_JSON, ORIGINAL_JSON)
File.delete(OLD_JSON)
It's important to not overwrite the original file until all processing has occurred, hence writing to a new file, closing the new and old, then renaming the old one to something safe, renaming the new to the original name, and then being able to remove the original. If you don't follow a process like that you run a risk of corrupting or losing your original data if the code or machine crashes mid-stream.
See "How to search file text for a pattern and replace it with a given value" for more information.

Probably the simplest way to use f.write would be to use it to replace the entire contents of the file. With that in mind, let's see if we can compose what we want the entire contents of the file to be, in memory, and then write it.
#!/usr/bin/env ruby
require 'rubygems'
require 'json'
require 'domainatrix'
write_array = []
jsonHash = File.read("/Users/ReffasCode/Desktop/sampleData.json")
read_array = JSON.parse(jsonHash)
read_array.each do |child|
url = Domainatrix.parse(child["URL"])
write_array << {
"url" => child["URL"],
"provider" => url.domain,
"service" => url.subdomain,
"context" => url.path,
"suffix" => url.public_suffix
}
end
File.open("/Users/BilalReffas/Desktop/sampleData.json","w") do |f|
f.write(JSON.pretty_generate(write_array))
end
Note some changes:
Fixed indentation :)
Removed some nesting
Write once with the entire contents of the file. Unless you have a really big file or a really important need for pipelined I/O this is probably the simplest thing to do.

Related

How to set custom metadata for Google Cloud Storage using JSON?

I am trying to add custom metadata to all uploads from my site to GCS.
I found something but I can't get it to work, in the Service Account JSON I added:
"metadata": {
"customMeta": "text here"
}
But it is not working, am I missing something here?
Thannks.
UPDATE
I am using Wordpress and a plugin called WP-Stateless, I asked the plugin author and he directed me to [link] (https://github.com/wpCloud/wp-stateless/blob/v2.2/lib/classes/class-utility.php) I tried adding a couple of lines, but after I saved it and tried uploading something, I checked the GCS console and there is no new metadata.
/* Add Google Storage metadata to our attachment */
$fileLink = $bucketLink . '/' . ( !empty($media['name']) ? $media['name'] : $file );
$cloud_meta = array(
'id' => $media[ 'id' ],
'name' => !empty($media['name']) ? $media['name'] : $file,
'fileLink' => $fileLink,
'storageClass' => $media[ 'storageClass' ],
'mediaLink' => $media[ 'mediaLink' ],
'selfLink' => $media[ 'selfLink' ],
'bucket' => ud_get_stateless_media()->get( 'sm.bucket' ),
'object' => $media,
'sizes' => array(),
'newDATA' => $_newData[ 'some text' ],
);
But I uploaded something and check GCS but there was now newDATA some text in the metadata for the file I uploaded.
As the documentation says, request body can have metadata field with JSON object having key/value pairs for metadata. Or metadata.key fields with specific values:
metadata - object - User-provided metadata, in key/value pairs.
metadata.(key) - string - An individual metadata entry. writable
Here is an example that worked for me (notice the metadata within metadata)
const file = bucket.file(filename);
await file.save(JSON.stringify(data), { metadata: { metadata: { user: 'user1' }}});

how to convert DBIx::Class::ResultSet into JSON

I am trying to obtain JSON from a DBIx::Class::ResultSet and I get the exception
encountered object 'Sql2Json::Model::DB::Book=HASH(0x6014f88)',
but neither allow_blessed, convert_blessed nor allow_tags settings are
enabled (or TO_JSON/FREEZE method missing)
The controller class is Books.pm:
package Sql2Json::Controller::Books;
use Moose;
use namespace::autoclean;
use JSON::XS;
BEGIN { extends 'Catalyst::Controller'; }
my $json = JSON::XS->new;
sub list : Local {
my($self, $c) = #_;
$c->stash(books_rs => $c->model('DB::Book'));
$c->stash(books => [$c->stash->{books_rs}->search({}, {order_by => 'name ASC'})]);
$c->stash(json_data => $json->convert_blessed->encode($c->stash->{books}));
$c->forward('View::JSON');
}
__PACKAGE__->meta->make_immutable;
1;
According to this article is enough the encoding of blessed objects:
$json->convert_blessed->encode($c->stash->{books})
Do I missing somethig here?
Almost all the time, the best way to do this is using the get_inflated_column method of the rows returned from a query.
$books = $c->model('DB::Book');
$c->stash(json_data => [map {$_->get_inflated_columns} $books->all]);
$c->forward('View::JSON');

Printing an HTML page header is different when saved in a var?

this works fine:
print $cgi->header({-type => "text/html", -expires => "now"}),
$cgi->start_html({-title => "TEST PAGE",
-style => [{-type => "text/css", -src =>"one.css"},
{-type => "text/css", -src =>"calendar.css"}],
-script => [
{-type => "text/javascript", -src => "main.js"},
]}),
$cgi->start_form({-name => "test", -autocomplete => "off"});
but if instead I try:
my $html= $cgi->header({-type => "text/html", -expires => "now"}),
$cgi->start_html({-title => "TEST PAGE",
-style => [{-type => "text/css", -src =>"one.css"},
{-type => "text/css", -src =>"calendar.css"}],
-script => [
{-type => "text/javascript", -src => "main.js"},
]}),
$cgi->start_form({-name => "test", -autocomplete => "off"});
print $html;
The HTML is then displayed on the page instead of rendered? Why would saving all the HTML in a var then printing that var produce a different result than printing directly? It's difficult to capture what's different because of the subtle differences like capturing a \n\n ..
Do I have to sprint that header into the var or something?
Thanks!
Why would saving all the HTML in a var then printing that var produce a different result than printing directly?
For the same reason that
print 'foo', 'bar'; # outputs foobar
is different than
my $foo = 'foo', 'bar';
print $foo; # outputs foo
print takes a list and outputs all the elements separated by $, (undef by default).
The comma operator in scalar context evaluates its left operand, throws away the result, evaluates its right operand, and returns that value. So my $foo = 'foo', 'bar' assigns the string foo to $foo and returns the string bar.
Also note that the HTML generation functions in CGI.pm are deprecated:
HTML Generation functions should no longer be used
All HTML generation functions within CGI.pm are no longer being maintained. Any issues, bugs, or patches will be rejected unless they relate to fundamentally broken page rendering.
The rationale for this is that the HTML generation functions of CGI.pm are an obfuscation at best and a maintenance nightmare at worst. You should be using a template engine for better separation of concerns. See CGI::Alternatives for an example of using CGI.pm with the Template::Toolkit module.
These functions, and perldoc for them, are considered deprecated, they are no longer being maintained and no fixes or features for them will be accepted.

Logstash dynamically split events

is there a way to split a logstash (1.4.2) event into multiple other events?
My input looks like this:
{ "parts" => ["one", "two"],
"timestamp" => "2014-09-27T12:29:17.601Z"
"one.key=> "1", "one.value"=>"foo",
"two.key" => "2", "two.value"=>"bar"
}
And I'd like to create two events with the following content:
{ "key" => "1", "value" => "foo", "timestamp" => "2014-09-27T12:29:17.601Z" }
{ "key" => "2", "value" => "bar", "timestamp" => "2014-09-27T12:29:17.601Z" }
Problem is that I can't know the actual "parts"...
Thanks for your help :)
Updating a very old answer because there is a better way to do this in newer versions of logstash without resorting to a custom filter.
You can do this using a ruby filter and a split filter:
filter {
ruby {
code => '
arrayOfEvents = Array.new()
parts = event.get("parts")
timestamp = event.get("timestamp")
parts.each { |part|
arrayOfEvents.push({
"key" => event.get("#{part}.key"),
"value" => event.get("#{part}.value"),
"timestamp" => timestamp
})
event.remove("#{part}.key")
event.remove("#{part}.value")
}
puts arrayOfEvents
event.remove("parts")
event.set("event",arrayOfEvents)
'
}
split {
field => 'event'
}
mutate {
rename => {
"[event][key]" => "key"
"[event][value]" => "value"
"[event][timestamp]" => "timestamp"
}
remove_field => ["event"]
}
}
My original answer was:
You need to resort to a custom filter for this (you can't call yield from a ruby code filter which is what's needed to generate new events).
Something like this (dropped into lib/logstash/filters/custom_split.rb):
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
# custom code to break up an event into multiple
class LogStash::Filters::CustomSplit < LogStash::Filters::Base
config_name "custom_split"
milestone 1
public
def register
# Nothing
end # def register
public
def filter(event)
return unless filter?(event)
if event["parts"].is_a?(Array)
event["parts"].each do |key|
e = LogStash::Event.new("timestamp" => event["timestamp"],
"key" => event["#{key}.key"],
"value" => event["#{key}.value"])
yield e
end
event.cancel
end
end
end
And then just put filter { custom_split {} } into your config file.
For future reference and based on #alcanzar answer, it is now possible to do things like this:
ruby {
code => "
# somefield is an array
array = event.get('somefield')
# drop the current event (this was my use case, I didn't need the feeding event)
event.cancel
# iterate over to construct new events
array.each { |a|
# creates a new logstash event
generated = LogStash::Event.new({ 'foo' => 'something' })
# puts the event in the pipeline queue
new_event_block.call(generated)
}
"
}

What is the best means to get the unique hash data into a JSON object without using an array when cycling through a DB Dataset?

I need to get the data into a JSON object but because I'm using the %data hash and it has the same address I'm getting the same data repeatedly in my JSON object.
This is the code that produces the JSON.
while (my ($orderID, $possessorName, $itemDescription, $...) = $sth->fetchrow_array)
{
%data = (orderID => $orderID, possessorName => $possessorName, itemDescription => $itemDescription,...);
$query_results{"job$index"} = {"data" => \%data};
$index++;
}
return $json_obj->pretty->encode(\%query_results, {ascii => 1, pretty => 1});
The problem is that the last item in my data set is masking all the previous items so I end up with one large JSON of the same exact data. I could use an array of hashes I suppose but this seems really messy and sloppy. How do I write the cleanest code to get my data? If an array of hashes is the best way to go please let me know and I'll do it. I all ready know how or can figure it out on my own.
What happens when you try:
my $index = 0;
my %query_results;
while (my ($orderID, $possessorName, $itemDescription, $...) = $sth->fetchrow_array) {
my %data = (orderID => $orderID, possessorName => $possessorName, itemDescription => $itemDescription,...);
$query_results{"job$index"}{'data'} = \%data;
$index++;
}
Previously, you used a %data hash declared in an outside scope; or worse, you didn't use strict; use warnings so %data was in fact an implicit global. Now, we declare the %data inside the loop which makes all the hashes distinct.
You could also copy the hash into a new hashref by {%data}.
That said, you don't even need that variable:
$query_results{"job$index"}{data} = {
# anonymous hashref here
orderID => $orderId,
possessorName => $possessorName,
itemDescription => ...
};