JSON Summarize for Dashing List - json

I am creating a Dashing job that pulls down some data from an API (this runs Ruby)
SCHEDULER.every '15m', :first_in => 0 do |job|
url = "https://blah.com
response = RestClient.get(url, {:Authorization => 'blahblah'})
current_timelist = JSON.parse(response)
acctitems = current_timelist.map do |row|
row = {
:label => row['member']['name'],
:value => row['actualHours']
}
end
# Update the List widget
send_event('timelist', { items: acctitems } )
end
I want to summarize based on the member name, but it lists every entry.
The JSON that is received from the API looks as follows (I have shortened this, and changed the names only), note the actualHours can be 0:
[
{
"member": {
"name": "User 1"
},
"actualHours": 0.2
},
{
"member": {
"name": "User 2"
},
"actualHours": 1.5
},
{
"member": {
"name": "User 2"
},
"actualHours": 0.17
}
]
I would also like to sort this so that I can have the top member at the top ect. I would also like to send a second event with the top person in the list (so they can get a gold star).

As I understood from your question, you need something like this aggregation
out = h.reduce({}) do |result, item|
key = item[:member][:name]
if result[key].nil?
result[key] = [item[:actualHours]]
else
result[key].push(item[:actualHours])
end
result
end
it will return the following output
{"User 1"=>[0.2], "User 2"=>[1.5, 0.17]}
You can iterate over it and filter what you need.

Thanks to #Stanislav for his code as that formed the basis of the fix I managed to get working.
SCHEDULER.every '15m', :first_in => 0 do |job|
url = "website.com"
response = RestClient.get(url, {:Authorization => 'BlahBlah'})
current_timelist = JSON.parse(response)
time = {}
acctitems = current_timelist.map do |row|
key = row['member']['name']
value = row['actualHours']
if time[key].nil?
time[key] = [value]
else
time[key].push(value)
end
end
resulttime = time.map do |result|
result = {
:label => result[0],
:value => result[1].inject(:+)
}
end
# Update the List widget
send_event('timelist', { items: resulttime })
end

Related

Perl, checking various types of hash for empty key values, json

I'm implementing right now some kind of mock for my function. And i have a little problem here. Depending on situation, i could get different kind of json as well as a different kind of hash of it. It can be a simple hash with empty values of keys or hash of array of hashes with empty or not empty values.
my %ch1 = (
"a" => "",
"b" => "",
"c" => ""
);
my %ch2 = (
"tab" => [
{
"a" => 11,
"b" => 22,
"c" => 33
},
{
"a" => 44,
"b" => 55,
"c" => 66
}
]
);
I need to make a function that checks both types of hash, counts empty values and compares with ammount of keys of hash.
This one kinda works for the first hash, but i don't know how to make it work for both hashes without hardcoding.
my $tck = 0;
for (keys %ch1){
if ($ch1{$_} eq ""){
print "'$ch1{$_}'\n";
$tck++;
}
}
if ($tck == scalar keys %ch1){
# do something
}
Any suggestions?
You could use Data::Visitor::Callback to do that. It's a pretty straight-forward implementation, as long as there are no other things that contain empty strings in your data structure.
The module visits each item in a data structure and calls user-defined callbacks on those items. It will do that for every ref and every value in those refs.
use strict;
use warnings;
use Data::Visitor::Callback;
my %ch1 = (
"a" => "",
"b" => "",
"c" => ""
);
my $empty_strings;
my $v = Data::Visitor::Callback->new(
value => sub {
++$empty_strings if $_ eq q{}; # q{} is like '' but easier to read
},
);
$v->visit( \%ch1 );
print $empty_strings;
This will output 3, as there are three empty strings in the input hash. Note that it wants a hash reference, not the hash itself.
You can just as well pass in a more complex data structure. The layout does not really matter. I've added an empty string to your second example to show that it works.
use strict;
use warnings;
use Data::Visitor::Callback;
my $empty_strings;
my $v = Data::Visitor::Callback->new(
value => sub {
++$empty_strings if $_ eq q{};
},
);
my %ch2 = (
"tab" => [
{
"a" => 11,
"b" => 22,
"c" => 33
},
{
"a" => '',
"b" => 55,
"c" => 66
}
]
);
$v->visit( \%ch2 );
print $empty_strings;
In this case, the output is 1.
Because there is no easy way to distinguish if a value it looks at is a key or a value, the following things would also be counted with this implementation. So it's not perfect, but should work for the type of data you showed.
my %fail = (
"" => "foo", # one
"b" => [ "", "" ], # two, three
);
This data structure would yield a $empty_strings count of 3.
I am not confident that I understand the problem correctly, but assuming those are the only two types of data structures your program needs to be able to handle, here is one way of doing something with them:
#!/usr/bin/env perl
use strict;
use warnings;
use List::MoreUtils qw( all none );
my %ch1 = (
"a" => "",
"b" => "",
"c" => ""
);
my %ch2 = (
"tab" => [
{
"a" => 11,
"b" => 22,
"c" => 33
},
{
"a" => 44,
"b" => 55,
"c" => 66
}
]
);
use YAML::XS;
for my $h ( \(%ch1, %ch2) ) {
print Dump n_keys_empty_values( $h );
}
sub n_keys_empty_values {
my $h = shift;
if ( all { ref } values %$h ) {
return [ map { my $v = $_; map count_empty_values( $_ ), #$v } values %$h ]
}
elsif ( none { ref } values %$h ){
return [ count_empty_values( $h ) ];
}
else {
die "Unexpected data structure\n";
}
}
sub count_empty_values {
my $h = shift;
[ scalar keys %$h, scalar grep $_ eq '', values %$h ];
}
Output:
---
- - 3
- 3
---
- - 3
- 0
- - 3
- 0
The return value of n_keys_empty_values is a reference to an array of array references. The size of the outer array corresponds to the number of inner hashes passed. count_empty_values takes a reference to a hash and counts the number of keys and number of values which are empty strings.
Write your function in that way, that it iterates over the argument list. Then you can pass either a single hash \%ch1 or a list of hashes #{$ch2{tab}} to the function.
#! /usr/bin/perl
use strict;
use warnings;
my %ch1 = (
"a" => "",
"b" => "",
"c" => ""
);
my %ch2 = (
"tab" => [
{
"a" => 11,
"b" => 22,
"c" => 33
},
{
"a" => 44,
"b" => 55,
"c" => 66
}
]
);
my %ch3 = (
"tab" => [
{
"a" => '',
"b" => '',
"c" => ''
},
{
"a" => '',
"b" => '',
"c" => ''
}
]
);
sub fun
{
for (#_) {
my %ch = %{$_};
my $tck = 0;
for (keys %ch){
if ($ch{$_} eq ""){
print "'$ch{$_}'\n";
$tck++;
}
}
if ($tck == scalar keys %ch){
print "do something\n";
}
}
}
fun (\%ch1);
fun (#{$ch2{tab}});
fun (#{$ch3{tab}});

Recursive construstruction of JSON from text file

I have some difficulties to create a structured JSON from a text file. Here is an example of text file
BlocA
value 1 param X
value 2 param Y
value 3 param Z
BlocB
this is a line
this is another line
BlocC foo
1
2
3
BlocC bar
something else
And the JSON I want is this :
{
"BlocA" : [
"value 1 param X",
"value 2 param Y",
"value 3 param Z"
]
},
"BlocB" : [
"this is a line",
"this is another line"
]
},
"BlocC" : [
{"foo" : [1,2,3]},
{"bar" : ["something else"]}
]
}
In my text file, blocs always begin at the first character of a line and properties of a bloc are bellow and always begin with at least a space character.
Sometimes, bloc names are followed by a string (like "BlocC foo" and "BlocC bar") which became a sub document of BlocC array.
I'am using nodeJS to do that. I though doing this using recursion but it's a little bit too tricky for me.
Is there another way to accomplish this (with a parser or whatever)?
Thank you for your help.
You can try this:
const fs = require('fs')
let prop, subProp, json = {}
fs.readFile('data.txt', 'utf8', (err, data) => {
data.split(/\r?\n/)
.filter(line => line.search(/\S/) > -1)
.reduce((obj, line) => {
// array members
if(line.search(/^\s+/) > -1) {
subProp ?
obj[prop][obj[prop].length - 1][subProp].push(line.trim()) :
obj[prop].push(line.trim())
} else {
// sub property
if(line.search(/\s+/) > -1) {
var props = line.split(/\s+/)
prop = props[0].trim()
subProp = props[1].trim()
Array.isArray(obj[prop]) || (obj[prop] = [])
obj[prop].push({
[subProp]: []
})
} else {
prop = line.trim()
subProp = null
obj[prop] = []
}
}
return obj
}, {})
console.log(json)
})

JSON.net parsing of dynamic JSON

I have JSON that looks like this:
{
"status": {
"code": 0,
"message": "OK"
},
"data": {
"_idtype": "cusip",
"_id": "00768Y883",
"api": {
"_name": "PortfolioBreakdownsRaw",
"PortfolioDate": "2015-10-12",
"GlobalBondSuperSectorLongSalePositionBreakdown": [
{
"Name": "Municipal",
"Value": "0.57842"
},
{
"Name": "Corporate",
"Value": "1.79649"
},
{
"Name": "Securitized",
"Value": "5.29493"
},
{
"Name": "Cash & Equivalents",
"Value": "166.20776"
}
],
"GlobalBondSuperSectorShortSalePositionBreakdown": [
{
"Name": "Government",
"Value": "0.90557"
}
]
}
}
}
I am able to get the api portion of the response easily:
var jObject = JObject.Parse(json);
var api = jObject["data"]["api"];
From here, I don't what if any arrays will be included in the response. The ultimate goal will be to create a parser that will be able to get the array names (GlobalBondSuperSectorShortSalePositionBreakdown) and as many rows of key-value pairs that it may contain, without first knowing the names such as (GlobalBondSuperSectorShortSalePositionBreakdown) beforehand.
I can't seem to find a good way to loop through the object, determine there are arrays at the api level and then iterate through those to get the values.
Any help would be appreciated.
Here's an example. In this code, the api variable holds a JObject, so we can iterate over its properties. From there, we look at the Type of each property value to see if it is an array or not. If it is, then we can iterate over that array to get the JObjects within it, and extract the Name and Value values that we expect to find there. Does this help?
var jObject = JObject.Parse(json);
var api = jObject["data"]["api"];
foreach (JProperty prop in api.Children<JProperty>())
{
JToken value = prop.Value;
if (value.Type == JTokenType.Array)
{
Console.WriteLine(prop.Name + ": ");
foreach (JObject jo in value.Children<JObject>())
{
Console.WriteLine(" " + jo["Name"] + ": " + jo["Value"]);
}
}
else
{
Console.WriteLine(prop.Name + ": " + value);
}
}
Output:
_name: PortfolioBreakdownsRaw
PortfolioDate: 2015-10-12
GlobalBondSuperSectorLongSalePositionBreakdown:
Municipal: 0.57842
Corporate: 1.79649
Securitized: 5.29493
Cash & Equivalents: 166.20776
GlobalBondSuperSectorShortSalePositionBreakdown:
Government: 0.90557
Fiddle: https://dotnetfiddle.net/XyoXQy
With Linq you can play pretty nice with Json.net:
Here is an easily readable version of the chunk of code that will create two dictionaries out of the JArray properties under the api element:
var api = jObject["data"]["api"];
var arrays = api.Cast<JProperty>().Where(o => o.Value.Type == JTokenType.Array).Select(token => token.Value).ToArray();
var dictionaries = new List<Dictionary<string, string>>();
foreach (var array in arrays)
{
var dictionary = array.ToDictionary(token => token["Name"].Value<string>(), token => token["Value"].Value<string>());
dictionaries.Add(dictionary);
}
alternative:
The same thing, but a shorter, more compact version :
var api = jObject["data"]["api"];
var dictionaries = api
.Cast<JProperty>()
.Where(o => o.Value.Type == JTokenType.Array)
.Select(token => token.Value)
.Select(array => array.ToDictionary(token => token["Name"].Value<string>(), token => token["Value"].Value<string>()));

Conditional Anonymous type

I am working on Web API and using Anonymous type to make JSON as output. I am stuck in the following scenario:
If there is no record(VALUE) available then i don't want to show that KEY. Meaning, Key should only appear when and only when there is value.
Below is the JSON object i am creating -
"TU": [
{
"BLOCK": [
[
"00:00",
"00:59"
]
]
}
],
"WE": [],// empty
"TH": [],// empty
"FR": [],// empty
"SA": [] // empty
Here for Tuesday we do have records and hence its showing but later for WE,TH,FR,SA there are not records and hence i don't want to show them so my result will be MO/TU only.
I am using below code:
var result = new
{
CustomerID = custId,
DeviceID = dId,
Kind = kind,
WebList = filter.Select(filt => new
{
URL = filt.FilterName,
TimeBlockFlag = new ChicoHelper().GetFlag(browserlimit, filt.ID, filt.FilterOptionID, KindId),
DAILY = browserlimit.Where(xx => xx.FilterID == filt.ID && xx.OptionTypeID == daily).Select(xx => xx.BlockTimeLimit).SingleOrDefault(),
WEEKLY = browserlimit.Where(xx => xx.FilterID == filt.ID && xx.OptionTypeID == weekly).Select(xx => xx.BlockTimeLimit).SingleOrDefault(),
MONTHLY = browserlimit.Where(xx => xx.FilterID == filt.ID && xx.OptionTypeID == monthly).Select(xx => xx.BlockTimeLimit).SingleOrDefault(),
HASVALUES = browserlimit.Where(xx => xx.FilterID == filt.ID).Count() > 0 ? 1 : 0,
BLOCKTYPE = new ChicoHelper().GetBlockType(browserlimit,filt.ID,filt.FilterOptionID,KindId),
SU = blockedlimit.Where(x => x.OptionID == sunday && x.FilterID == filt.ID).GroupBy(x => new { x.BlockDay })
.Select(x => new
{
BLOCK = x.Select(y =>
new[] { y.BlockStartTime.MakeFormatedTime(), y.BlockEndTime.MakeFormatedTime() }
)
}),
MO = blockedlimit.Where(x => x.OptionID == monday && x.FilterID == filt.ID).GroupBy(x => new { x.BlockDay })
.Select(x => new
{
BLOCK = x.Select(y =>
new[] { y.BlockStartTime.MakeFormatedTime(), y.BlockEndTime.MakeFormatedTime() }
)
}),
TU = blockedlimit.Where(x => x.OptionID == tuesday && x.FilterID == filt.ID).GroupBy(x => new { x.BlockDay })
.Select(x => new
{
BLOCK = x.Select(y =>
new[] { y.BlockStartTime.MakeFormatedTime(), y.BlockEndTime.MakeFormatedTime() }
)
}),
// if i can put some condition like if there is not record for WE then don't show it.
WE = blockedlimit.Where(x => x.OptionID == wednesday && x.FilterID == filt.ID).GroupBy(x => new { x.BlockDay })
.Select(x => new
{
BLOCK = x.Select(y =>
new[] { y.BlockStartTime.MakeFormatedTime(), y.BlockEndTime.MakeFormatedTime() }
)
}),
The main reason for doing this is to reduce the JSON size which will be consumed by Mobile Devices.
Please help me with this.
The properties of an anonymous type are fixed at compile-time - you can't make them conditional. However, some other approaches you might want to think about:
You could investigate whether a property is still included in the JSON representation if its value is null. If it's not, you could add an extension method NullIfEmpty() which returns null if its input is empty.
You could try performing the JSON conversion from the anonymous type in code first, then delete any properties with an empty set of results, then just return that JSON object from the API. (I don't know Web API myself, but there must be a way of saying "Here's a JSON object - ask it for its string representation" rather than using an anonymous type.)
You could ditch the anonymous type entirely, and build up the JSON representation programmatically, setting just the properties you want.
In any approach, I would strongly advise you to extract a common method to come up with the property value based on a day of the week, so you can have:
...
SU = blockedLimit.GetDayBlocks(sunday),
MO = blockedLimit.GetDayBlocks(monday),
TU = blockedLimit.GetDayBlocks(tuesday),
...
There's no reason to have all that code repeated 7 times. In fact, I'd probably refactor that part before doing anything else - it'll make it easier to experiment.

JSON formatting in Perl

I am trying to create a JSON object that lists maps associated to a particular user, but haven't ever worked with nested JSON objects. This is what I want:
{
"success":"list of users maps",
"maps":[
{
"id":"1",
"name":"Home to LE",
"date_created":"1366559121"
},
{
"id":"2",
"name":"Test 1",
"date_created":"1366735066"
}
]
}
with this perl code:
my $maps = [];
for (my $x = 0; $x < $sth->rows; $x++) {
my ($id, $name, $date) = $sth->fetchrow_array();
my $map = qq{{"id":"$id","name":"$name","date_created":"$date"}};
push $maps, $map;
}
my $j = JSON::XS->new->utf8;
my $output = $j->encode({
"success"=>"list of users maps",
"maps"=>$maps
});
But the output I am getting is:
{
"success":"list of users maps",
"maps":[
"{\"id\":\"1\",\"name\":\"Home to LE\",\"date_created\":\"1366559121\"}",
"{\"id\":\"2\",\"name\":\"Test 1\",\"date_created\":\"1366735066\"}"
]
}
So when I process it in my Javascript, the data.maps[x].id is undefined. I am pretty sure that the JSON being output is incorrectly formatted.
Can anyone help me fix it?
It's undefined because what you have at data.maps[x] is not an object, but a string. Since a string has no property called id, you're getting undefined. I'd probably do something like this (if I couldn't change the perl script):
var mapData = JSON.parse(data.maps[x]);
//do stuff with mapData.id
But the better thing to do, is to make sure that it doesn't encode it as a string, but as proper JSON.
This part in your perl script:
my $map = qq{{"id":"$id","name":"$name","date_created":"$date"}};
Is simply making a quoted string out of all that data. Instead, what you want is an actual perl hash that can be translated into a JSON map/associative-array. So try this:
my $map = {
"id" => "$id",
"name" => "$name",
"date_created" => "$date"
};
push $maps, $map;
This way you actually have a perl hash (instead of just a string) that will get translated into proper JSON.
As an example, I wrote some test code:
use strict;
use JSON::XS;
my $maps = [];
push $maps, { id => 1, blah => 2 };
push $maps, { id => 3, blah => 2 };
my $j = JSON::XS->new->utf8->pretty(1);
my $output = $j->encode({
success => "list of blah",
maps => $maps
});
print $output;
When you run this, you get:
{
"success" : "list of blah",
"maps" : [
{
"blah" : 2,
"id" : 1
},
{
"blah" : 2,
"id" : 3
}
]
}