i am currently trying to find whether the high price(column 'high') is the highest among 1 and -1 periods away, if so then it would return True in my new added column called 'high_peak'. This is what i've managed but the output returns everything False. Can someone help me out please? im a new learner which just started to learn python, so if there could be some elaboration along the lines would be much appreciated :D
Here's what i did:
SPX = pd.read_csv(r'D:\Users\jimmyli\Downloads\SP_SPX, 1D.csv', index_col = 'time')
def peak(x):
for i in range(len(x)):
if x[i] > x[i+1] and x[i] > x[i-1]:
return True
else:
return False
SPX['high_peak'] = peak(SPX['high'])
print(SPX)
And here's what came out:
open high low high_peak
time
2009/9/14 1040.15 1049.74 1035.00 False
2009/9/15 1049.03 1056.04 1043.42 False
2009/9/16 1053.99 1068.76 1052.87 False
2009/9/17 1067.87 1074.77 1061.20 False
2009/9/18 1066.60 1071.52 1064.27 False
... ... ... ...
2022/2/14 4412.61 4426.22 4364.84 False
2022/2/15 4429.28 4472.77 4429.28 False
2022/2/16 4455.75 4489.55 4429.68 False
2022/2/17 4456.06 4456.06 4373.81 False
2022/2/18 4384.57 4394.60 4327.22 False
[3132 rows x 4 columns]
As you can see, there's an error at the operation because the high price #2009/9/17 should have returned True at 'high_peak' column
When using dbWriteTable() in the RMySQL package, logical values are written as 0 regardless of value. I would expect that TRUE values would return a 1:
# Setup
# con is a valid MySQLConnection object
> df <- data.frame(string = 'Testing Logical Values',
t_lgl = TRUE,
f_lgl = FALSE,
stringsAsFactors = FALSE)
> df
string t_lgl f_lgl
1 Testing Logical Values TRUE FALSE
> class(df[,2])
[1] "logical"
# Test
# This schema has no tables until dbWriteTable() is called
> dbWriteTable(con,'test_table',df)
[1] TRUE
# Result
> dbReadTable(con,'test_table')
string t_lgl f_lgl
1 Testing Logical Values 0 0
> class(dbReadTable(con,'test_table')[,2])
[1] "integer"
Shouldn't the t_lgl value return 1 since it was TRUE and passed to dbWriteTable() as a logical?
The RMySQL package is being phased out in favour of RMariaDB.
Using RMariaDB I am able to successfully write logical values to a MySQL database like this:
con <- dbConnect(RMariaDB::MariaDB(), group = "my-db")
dbWriteTable(con, "test", data.frame(a=T, b=F), overwrite = T)
I have a perl hash that is obtained from parsing JSON. The JSON could be anything a user defined API could generated. The goal is to obtain a date/time string and determine if that date/time is out of bounds according to a user defined threshold. The only issue I have is that perl seems a bit cumbersome when dealing with hash key/subkey iteration. How can I look through all the keys and determine if a key or subkey exists throughout the hash? I have read many threads throughout stackoverflow, but nothing that exactly meets my needs. I only started perl last week so I may be missing something... Let me know if that's the case.
Below is the "relevant" code/subs. For all code see: https://gitlab.com/Jedimaster0/check_http_freshness
use warnings;
use strict;
use LWP::UserAgent;
use Getopt::Std;
use JSON::Parse 'parse_json';
use JSON::Parse 'assert_valid_json';
use DateTime;
use DateTime::Format::Strptime;
# Verify the content-type of the response is JSON
eval {
assert_valid_json ($response->content);
};
if ( $# ){
print "[ERROR] Response isn't valid JSON. Please verify source data. \n$#";
exit EXIT_UNKNOWN;
} else {
# Convert the JSON data into a perl hashrefs
$jsonDecoded = parse_json($response->content);
if ($verbose){print "[SUCCESS] JSON FOUND -> ", $response->content , "\n";}
if (defined $jsonDecoded->{$opts{K}}){
if ($verbose){print "[SUCCESS] JSON KEY FOUND -> ", $opts{K}, ": ", $jsonDecoded->{$opts{K}}, "\n";}
NAGIOS_STATUS(DATETIME_DIFFERENCE(DATETIME_LOOKUP($opts{F}, $jsonDecoded->{$opts{K}})));
} else {
print "[ERROR] Retreived JSON does not contain any data for the specified key: $opts{K}\n";
exit EXIT_UNKNOWN;
}
}
sub DATETIME_LOOKUP {
my $dateFormat = $_[0];
my $dateFromJSON = $_[1];
my $strp = DateTime::Format::Strptime->new(
pattern => $dateFormat,
time_zone => $opts{z},
on_error => sub { print "[ERROR] INVALID TIME FORMAT: $dateFormat OR TIME ZONE: $opts{z} \n$_[1] \n" ; HELP_MESSAGE(); exit EXIT_UNKNOWN; },
);
my $dt = $strp->parse_datetime($dateFromJSON);
if (defined $dt){
if ($verbose){print "[SUCCESS] Time formatted using -> $dateFormat\n", "[SUCCESS] JSON date converted -> $dt $opts{z}\n";}
return $dt;
} else {
print "[ERROR] DATE VARIABLE IS NOT DEFINED. Pattern or timezone incorrect."; exit EXIT_UNKNOWN
}
}
# Subtract JSON date/time from now and return delta
sub DATETIME_DIFFERENCE {
my $dateInitial = $_[0];
my $deltaDate;
# Convert to UTC for standardization of computations and it's just easier to read when everything matches.
$dateInitial->set_time_zone('UTC');
$deltaDate = $dateNowUTC->delta_ms($dateInitial);
if ($verbose){print "[SUCCESS] (NOW) $dateNowUTC UTC - (JSON DATE) $dateInitial ", $dateInitial->time_zone->short_name_for_datetime($dateInitial), " = ", $deltaDate->in_units($opts{u}), " $opts{u} \n";}
return $deltaDate->in_units($opts{u});
}
Sample Data
{
"localDate":"Wednesday 23rd November 2016 11:03:37 PM",
"utcDate":"Wednesday 23rd November 2016 11:03:37 PM",
"format":"l jS F Y h:i:s A",
"returnType":"json",
"timestamp":1479942217,
"timezone":"UTC",
"daylightSavingTime":false,
"url":"http:\/\/www.convert-unix-time.com?t=1479942217",
"subkey":{
"altTimestamp":1479942217,
"altSubkey":{
"thirdTimestamp":1479942217
}
}
}
[SOLVED]
I have used the answer that #HåkonHægland provided. Here are the below code changes. Using the flatten module, I can use any input string that matches the JSON keys. I still have some work to do, but you can see the issue is resolved. Thanks #HåkonHægland.
use warnings;
use strict;
use Data::Dumper;
use LWP::UserAgent;
use Getopt::Std;
use JSON::Parse 'parse_json';
use JSON::Parse 'assert_valid_json';
use Hash::Flatten qw(:all);
use DateTime;
use DateTime::Format::Strptime;
# Verify the content-type of the response is JSON
eval {
assert_valid_json ($response->content);
};
if ( $# ){
print "[ERROR] Response isn't valid JSON. Please verify source data. \n$#";
exit EXIT_UNKNOWN;
} else {
# Convert the JSON data into a perl hashrefs
my $jsonDecoded = parse_json($response->content);
my $flatHash = flatten($jsonDecoded);
if ($verbose){print "[SUCCESS] JSON FOUND -> ", Dumper($flatHash), "\n";}
if (defined $flatHash->{$opts{K}}){
if ($verbose){print "[SUCCESS] JSON KEY FOUND -> ", $opts{K}, ": ", $flatHash>{$opts{K}}, "\n";}
NAGIOS_STATUS(DATETIME_DIFFERENCE(DATETIME_LOOKUP($opts{F}, $flatHash->{$opts{K}})));
} else {
print "[ERROR] Retreived JSON does not contain any data for the specified key: $opts{K}\n";
exit EXIT_UNKNOWN;
}
}
Example:
./check_http_freshness.pl -U http://bastion.mimir-tech.org/json.html -K result.creation_date -v
[SUCCESS] JSON FOUND -> $VAR1 = {
'timestamp' => '20161122T200649',
'result.data_version' => 'data_20161122T200649_data_news_topics',
'result.source_version' => 'kg_release_20160509_r33',
'result.seed_version' => 'seed_20161016',
'success' => 1,
'result.creation_date' => '20161122T200649',
'result.data_id' => 'data_news_topics',
'result.data_tgz_name' => 'data_news_topics_20161122T200649.tgz',
'result.source_data_version' => 'seed_vtv: data_20161016T102932_seed_vtv',
'result.data_digest' => '6b5bf1c2202d6f3983d62c275f689d51'
};
Odd number of elements in anonymous hash at ./check_http_freshness.pl line 78, <DATA> line 1.
[SUCCESS] JSON KEY FOUND -> result.creation_date:
[SUCCESS] Time formatted using -> %Y%m%dT%H%M%S
[SUCCESS] JSON date converted -> 2016-11-22T20:06:49 UTC
[SUCCESS] (NOW) 2016-11-26T19:02:15 UTC - (JSON DATE) 2016-11-22T20:06:49 UTC = 94 hours
[CRITICAL] Delta hours (94) is >= (24) hours. Data is stale.
You could try use Hash::Flatten. For example:
use Hash::Flatten qw(flatten);
my $json_decoded = parse_json($json_str);
my $flat = flatten( $json_decoded );
say "found" if grep /(?:^|\.)\Q$key\E(?:\.?|$)/, keys %$flat;
You can use Data::Visitor::Callback to traverse the data structure. It lets you define callbacks for different kinds of data types inside your structure. Since we're only looking at a hash it's relatively simple.
The following program has a predefined list of keys to find (those would be user input in your case). I converted your example JSON to a Perl hashref and included it in the code because the conversion is not relevant. The program visits every hashref in this data structure (including the top level) and runs the callback.
Callbacks in Perl are code references. These can be created in two ways. We're doing the anonymous subroutine (sometimes called lambda function in other languages). The callback gets passed two arguments: the visitor object and the current data substructure.
We'll iterate all the keys we want to find and simply check if they exist in that current data structure. If we see one, we count it's existence in the %seen hash. Using a hash to store things we have seen is a common idiom in Perl.
We're using a postfix if here, which is convenient and easy to read. %seen is a hash, so we access the value behind the $key with $seen{$key}, while $data is a hash reference, so we use the dereferencing operator -> to access the value behind $key with $data->{$key}.
The callback needs us to return the $data again so it continues. The last line is just there, it's not important.
I've used Data::Printer to output the %seen hash because it's convenient. You can also use Data::Dumper if you want. In production, you will not need that.
use strict;
use warnings;
use Data::Printer;
use Data::Visitor::Callback;
my $from_json = {
"localDate" => "Wednesday 23rd November 2016 11:03:37 PM",
"utcDate" => "Wednesday 23rd November 2016 11:03:37 PM",
"format" => "l jS F Y h:i:s A",
"returnType" => "json",
"timestamp" => 1479942217,
"timezone" => "UTC",
"daylightSavingTime" =>
0, # this was false, I used 0 because that's a non-true value
"url" => "http:\/\/www.convert-unix-time.com?t=1479942217",
"subkey" => {
"altTimestamp" => 1479942217,
"altSubkey" => {
"thirdTimestamp" => 1479942217
}
}
};
my #keys_to_find = qw(timestamp altTimestamp thirdTimestamp missingTimestamp);
my %seen;
my $visitor = Data::Visitor::Callback->new(
hash => sub {
my ( $visitor, $data ) = #_;
foreach my $key (#keys_to_find) {
$seen{$key}++ if exists $data->{$key};
}
return $data;
},
);
$visitor->visit($from_json);
p %seen;
The program outputs the following. Note this is not a Perl data structure. Data::Printer is not a serializer, it's a tool to make data human readable in a convenient way.
{
altTimestamp 1,
thirdTimestamp 1,
timestamp 1
}
Since you also wanted to constraint the input, here's an example how to do that. The following program is a modification of the one above. It allows to give a set of different constraints for every required key.
I've done that by using a dispatch table. Essentially, that's a hash that contains code references. Kind of like the callbacks we use for the Visitor.
The constraints I've included are doing some things with dates. An easy way to work with dates in Perl is the core module Time::Piece. There are lots of questions around here about various date things where Time::Piece is the answer.
I've only done one constraint per key, but you could easily include several checks in those code refs, or make a list of code refs and put them in an array ref (keys => [ sub(), sub(), sub() ]) and then iterate that later.
In the visitor callback we are now also keeping track of the keys that have %passed the constraints check. We're calling the coderef with $coderef->($arg). If a constraint check returns a true value, it gets noted in the hash.
use strict;
use warnings;
use Data::Printer;
use Data::Visitor::Callback;
use Time::Piece;
use Time::Seconds; # for ONE_DAY
my $from_json = { ... }; # same as above
# prepare one of the constraints
# where I'm from, Christmas eve is considered Christmas
my $christmas = Time::Piece->strptime('24 Dec 2016', '%d %b %Y');
# set up the constraints per required key
my %constraints = (
timestamp => sub {
my ($epoch) = #_;
# not older than one day
return $epoch < time && $epoch > time - ONE_DAY;
},
altTimestamp => sub {
my ($epoch) = #_;
# epoch value should be an even number
return ! $epoch % 2;
},
thirdTimestamp => sub {
my ($epoch) = #_;
# before Christmas 2016
return $epoch < $christmas;
},
);
my %seen;
my %passed;
my $visitor = Data::Visitor::Callback->new(
hash => sub {
my ( $visitor, $data ) = #_;
foreach my $key (%constraints) {
if ( exists $data->{$key} ) {
$seen{$key}++;
$passed{$key}++ if $constraints{$key}->( $data->{$key} );
}
}
return $data;
},
);
$visitor->visit($from_json);
p %passed;
The output this time is:
{
thirdTimestamp 1,
timestamp 1
}
If you want to learn more about the dispatch tables, take a look at chapter two of the book Higher Order Perl by Mark Jason Dominus which is legally available for free here.
using MongoDB;
my $content = $db->items->find('something');
my $json;
while($content->next){
push the data into a $json
}
where there is a way
my #variables = $content->all;
foreach (#variables){
push the data into $json
}
Is there any way i can convert directly push the data into json string;
and push the data into Mojolicious
$self->render(json => $json);
Wrote a small Mojo test script for it. Use
$collection->find->all
to get a list of all pages, not an iterator. Here's the test:
#!/usr/bin/env perl
use Mojolicious::Lite;
use MongoDB;
# --- MongoDB preparation ---
my $conn = MongoDB::Connection->new; # create a MongoDB connection
my $test_db = $conn->test; # a MongoDB database
my $tests = $test_db->tests; # a test collection
# insert test objects
$tests->remove();
$tests->insert({name => 'foo', age => 42});
$tests->insert({name => 'bar', age => 17});
# --- Mojolicious::Lite web app ---
# access the tests collection in a mojoy way
helper mongo_tests => sub {$tests};
# list all tests as JSON
get '/' => sub {
my $self = shift;
$self->render(json => [$self->mongo_tests->find->all]);
};
# --- web app test ---
use Test::More tests => 6;
use Test::Mojo;
my $tester = Test::Mojo->new;
$tester->get_ok('/')->status_is(200);
my $json = $tester->tx->res->json;
is $json->[0]{name}, 'foo', 'right name';
is $json->[0]{age}, 42, 'right age';
is $json->[1]{name}, 'bar', 'right name';
is $json->[1]{age}, 17, 'right age';
Output:
1..6
ok 1 - get /
ok 2 - 200 OK
ok 3 - right name
ok 4 - right age
ok 5 - right name
ok 6 - right age
Note: I couldn't use is_deeply to test the $json data structure because MongoDB adds OIDs. You'll see them when you dump $json.
I have some trouble using jruby objects into java
java side
package com.pp;
public interface ZeroI {
boolean equals(Object o);
int hashCode();
int hash();
}
package com.pp;
public class Tester {
public Object[] compare(ZeroI one, ZeroI two) {
return new Object[] {one.hashCode(), two.hashCode(), one.equals(two), one == two};
}
}
jruby side
include Java
import com.pp.Tester
import com.pp.ZeroI
module MMM
module Zero
def hash= value
#hash = value
end
def hash
#hash
end
def hashCode
#hash
end
def equals other
false
end
def == other
true
end
end
class OneClass
include ZeroI
include Zero
end
class TwoClass
include ZeroI
include Zero
end
def self.create clazz
begin
dump = IO.readlines("C:/#{clazz.to_s.rpartition('::')[2]}.txt", '').to_s
instance = Marshal.load dump
rescue => err
puts err.message
instance = clazz.new
dump = Marshal.dump instance
File.open("C:/#{clazz.to_s.rpartition('::')[2]}.txt", 'w') { |f| f.write dump }
end
instance
end
tester = Tester.new
one = create OneClass
two = create TwoClass
puts one
puts two
one.hash = 22
two.hash = 22
puts one.hashCode
puts two.hashCode
puts one.equals two
puts one == two
tester.compare(one, two).each { |value| puts value }
end
First pass result:
No such file or directory - C:/OneClass.txt
No such file or directory - C:/TwoClass.txt
#<MMM::OneClass:0x1971eb3>
#<MMM::TwoClass:0x1408a75>
22
22
false
true
22
22
true
false
true # it's OK because JAVA.equals works with JRUBY.==
false # it's OK because org.pp.ZeroI can't declare == method and JAVA.== is used
Second pass result (with deserialized objects)
#<MMM::OneClass:0xd510e8>
#<MMM::TwoClass:0x490342>
22
22
false
true
13046738 # but what is it?
31877484 # but what is it?
false # but what is it?
false
Can anybody explain it?
I don't know all the details about why this happens like it happens but I have a solution/workaround for you. (I've seen similar behaviour when passing objects that are created on the ruby side to the Java side.)
As far as I can tell JRuby needs to have already "seen" an instance of the class it is trying to unmarshal before it can get the Java-inheritance side of things right. It's almost as if creating an object within JRuby has an undocumented side-effect that registers the required inheritance hierarchy. If that isn't well worded it's because I don't understand it myself!
So the workaround is to simply create an instance of OneClass and TwoClass before doing the unmarshal. If I change the self.create method to the following:
def self.create clazz
begin
clazz.new # <<< just create an instance and throw it away!
dump = IO.readlines("C:/#{clazz.to_s.rpartition('::')[2]}.txt", '').to_s
instance = Marshal.load dump
rescue => err
puts err.message
instance = clazz.new
dump = Marshal.dump instance
File.open("C:/#{clazz.to_s.rpartition('::')[2]}.txt", 'w') { |f| f.write dump }
end
instance
end
Then the output of the two passes are as follows:
First pass
No such file or directory - C:/OneClass.txt
No such file or directory - C:/TwoClass.txt
#<MMM::OneClass:0x4de6f0ef>
#<MMM::TwoClass:0x4526ba64>
22
22
false
true
22
22
true
false
Second pass
#<MMM::OneClass:0x4858cca9>
#<MMM::TwoClass:0x3de4905a>
22
22
false
true
22
22
true
false
According to this bug report this is scheduled to be fixed in JRuby 1.7. It's worth noting that while the comments in the report say that the workaround is to call a method, passing an object instance in, it seems to be that the prior creation of the object is enough.