I need to find a way of transferring an RSA public key to a server for my network communication program. I have done some research, and it seems that the easiest way to do this is to convert the public key (which is stored as some kind of hash reference) to a JSON for transmission. However, in my test code I cannot get the key to convert to a JSON. Here is my test program:
use strict;
use warnings;
use Crypt::RSA;
use JSON;
my %hash = ( name => "bob",
age => 123,
hates=> "Perl"
);
my $hash_ref = \%hash;
my $hash_as_json = to_json($hash_ref);
print $hash_as_json, "\n"; # Works fine for a normal hash
my $rsa = new Crypt::RSA;
my ($public, $private) = $rsa->keygen (
Identity => 'client',
Size => 512,
Password => 'password',
Verbosity => 1,
) or die $rsa->errstr();
my $key_hash_as_json = to_json($public, {allow_blessed => 1, convert_blessed => 1});
print $key_hash_as_json, "\n";
Before I found the line {allow_blessed => 1, convert_blessed => 1} I got an error message saying
encountered object 'Crypt::RSA::Key::Public=HASH(0x3117128)', but
neither allow_blessed, convert_blessed nor allow_tags settings are
enabled (or TO_JSON/FREEZE method missing) at
/home/alex/perl5/lib/perl5/JSON.pm line 154.
What does this mean and why did that line fix it?
After adding the code, it just gives null when I try and print the JSON. Why is this happening and how do I fix it?
Alternatively, is there a better way of doing what I am trying here?
The most common way of representing an RSA public key as text is the PEM encoding. Unfortunately, Crypt::RSA does not provide any way to convert to or from this format, or indeed any other standard format. Don't use it!
Instead, I'd recommend that you use Crypt::OpenSSL::RSA. Generating a private key and printing its public form with this module is simple:
use Crypt::OpenSSL::RSA;
my $key = Crypt::OpenSSL::RSA->generate_key(512);
print $key->get_public_key_string;
This will output a PEM encoding like the following:
-----BEGIN RSA PUBLIC KEY-----
MEgCQQDd/5F9Rc5vsNuKBrd4gfI4BDgre/sTBKu3yXpk+8NjByKpClsi3IQEGYeG
wmv/q/1ZjflFby1MPxMhXZo/82CbAgMBAAE=
-----END RSA PUBLIC KEY-----
Apart from already mentioned PEM there exists JWK format (JSON Web Key). Have a look at Crypt::PK::RSA (my module) which supports generating, importing and exporting RSA keys in both PEM and JWK.
Related
I am using python-gnupg package to create GPG public and private key. The generated private key I am storing in AWS secrets manager as follows.
Key: private_key
value: -----BEGIN PGP PRIVATE KEY BLOCK-----
Version: GnuPG v2.0.22 (GNU/Linux)
lQO+BF37qDIBCADXq0iJVRYFb43+YU8Ts63hDgZl49ZNdnDVhd9H0JMXRHqtPqt9
bbFePPN47NRe6z6GsbPaPmDqEE9l3KjFnSZB/yCii+2wHZR0ij2g3ATbiAbOoQQy
I6bbUADmHtcfIJByoXVoDk489nUPt84Xyp1lHiBfCtUmq4w62Okq6InlRhxjxcEx
VvSXaCY8YnEXUAgNGpvcKHDejGS9V4djh7r7lgJ/Y+3Xb2eepOfiaCx2Cn8ZMI0q
7eWH0MmSeR4ueOLeb79ZKjpJraBfV91XplgHHiM18oECWWwsQCFiwi1GVOLpX6Fh
HIoUyaRAW2vZyFcNnO7iLbetie6fE884lfHxABEBAAH+AwMCO+Qoh7o3GWVga9f2
gHEeuGGH4KB3aspQZt/zwKpx7YlDB/uLd4q7JQt35nIH6pfYdMwgQt001CRhsGvX
QVKIkvipQvJZgCO8Nix7xYCukH0cI4TXD7S9BmRNMCPi74+Q1J3cDfKHCseynMNF
GzBeCDx6LW3CVfKKs0Mc2ecSl0c8gxaPDi3AfACRMefEAuVQyo82qJKjpI+O/Yik
z40C5OgK0XfetKstxcH4B0bx0o/PrUpYFM/gHHgFkbpVg5citcvFY4VcEkWryVcg
yF0qBPXP0OKBtCUU1ZGiCwRJy8iGd/dOOICcSCfMNy+jzzM3FSVzei69x7MYt3xu
IzCsmHpDvpdL7tiDDHgwajZeFFPTzf7Ic90K6TapQ3H59xPMxnL9K5o9rP1glRY0
8e4zYjYxg9A6Yl3K5zdqs+M1A3Os70HUlWZXZ4LQNcidPd1rhnPnm9eXkyV2ScXl
dE38aOA5pnrL0WZUM3/OLAToMP6h4rjw9WLqqgWlrl6yz9bhZrfRxlhZaEtNs1bi
pgrmPK/a5fK++BjMSuA94EkXTVNjKWNQBzcmrff27M1TMwN+34NWj3dk/a1gyflP
QZgK3MT+0GaMCcvy1EoZ87ffLQrWwFJOw5nT83yG7VBbuerSEk/tk30bxmYN6HzO
zvQgSjDiiH+ANXVupnzDjjBREmH6V1Hv+7Q0vrjKQHd3eYvKJpAWfFr9kO8DzKck
ZkSMj487SjlHbh33z1yupuwAtjyYQ5tN1adSlDa92t0Q08udnFDQtxXEnL6rw/Du
llEuCEVC9UYcNwwQGMsGXQBFFfj1389WHr0hkSOvyS1nPiIku5kNXDhSWq7/okTS
FwnCt+wbZa6TWbXjwKzHzu4LOarV1s8DnYHKNH6HHIqsVR2oJuIuqhyREAqjeP/T
3bQjQXV0b2dlbmVyYXRlZCBLZXkgPG1laHVsQHBoZWFhLm9yZz6JATkEEwECACMF
Al37qDICGy8HCwkIBwMCAQYVCAIJCgsEFgIDAQIeAQIXgAAKCRDO+i9CZ70SvqMn
CACCmdzqZW68j1E45XTHz3fvqdft6fXOyrlMuDdcH2y7Zrl5JS7PlCeHzIcsSMlH
wDYpCG8km7nwZsnWqKsOXFWq1nq/j7Kv5AzR7UmPzTw/1HFSVhIFA0ZZMHAnwp7Y
bcAT+ssvo4To9CjzRp/ZI1k26RFXPWuXETa41DBIVz13Ss4SIaf7UG9FQ55o+2BA
TP48yCQqktiWOoZ0rV1ALSFlE4Gs3UWHcYxxCABA0JB4+FuCRfB8QMreLwFb47wc
dIitbVl0mQx5IXCkqhJKqR62rRy25Put4xnPhXGtXqfoYDVYvYvlsl/FA35cX+Z1
QODnLq/jQ7ZPdaFC7cFqxztk
=RvGa
-----END PGP PRIVATE KEY BLOCK-----
Key: passphrase
Value: secret123
All I want to do is extract Key and Value pair from AWS Secrets manager and import key and later decrypt file.
As you all know JSON doesn't interpret new line characters in a multi line value so GPG import_keys fails to import private key. If I just read local file having the same private key then no problem. Please let me know if there is any workaround for this issue ?
try:
secretkey = self.get_secret(secretName)
if not secretkey:
self.logger.error("Empty secret key")
exit(0)
newdict = json.loads(secretkey)**
# newdict = ast.literal_eval(secretkey)
private_key = newdict['private_key']
# private_key = open('/home/ec2-user/GPG/test_private_key.asc').read()
passphrase = newdict['passphrase']
gpg = gnupg.GPG(gnupghome=gpgHomeDir)
import_result = gpg.import_keys(private_key)
count = import_result.count
if count == 0:
self.logger.error("Failed to import private key")
sys.exit(1)
dataPath = srcDir + "/" + self.dataSource
for root, folders, files in os.walk(dataPath):
if not files:
self.logger.info("No files found so skipping .....")
continue
for filename in folders + files:
fullpath = os.path.join(root,filename)
self.logger.info("Fullpath = {0}".format(fullpath))
out_file = "/tmp/" + filename
with open(fullpath, "rb") as f:
status = gpg.decrypt_file(f, passphrase=passphrase, output=out_file)
if status.ok:
s3Prefix = root.replace(srcDir + '/', '')
s3ObjKey = s3Prefix + "/" + filename
s3InPath = "s3://" + self.inBucketName + "/" + s3Prefix + "/" + filename
with open(out_file, "rb") as fl:
self.s3Client.upload_fileobj(fl,
self.inBucketName,
s3ObjKey
)
except Exception as e:
print(str(e))
self.logger.error(str(e))
sys.exit(1)
I have to use base64 format to store PGP key as follows.
import base64
import gnupg
try:
gpg = gnupg.PGP(gnupghome="/home/guest/GPG")
input_data = gpg.gen_key_input(key_type='RSA',
key_length=2048,
name_email="guest#xyz.com"
passphrase="pass123")
key = gpg.gen_key(input_data)
ascii_armored_public_key = gpg.export_keys(key.fingerprint, armor=True)
ascii_armored_private_key = gpg.export_keys(key.fingerprint, True, armor=True)
b64_encoded_private_key = base64.b64encode(ascii_armored_private_key.encode())
binaryPrivKeyFile = "/tmp/b64encoded_private_key.asc"
with open(binaryPrivKeyFile, 'wb') as bPrivFile:
bPrivFile.write(b64_encoded_private_key)
except Exception as e:
print(str(e))
sys.exit(1)
Now we have to store b64encoded_private_key.asc to AWS secrets manager as follows.
$ aws secretsmanager create-secret --name private-key --secret-binary fileb://b64encoded_private_key.asc --region us-east-1
We cannot store passphrase in the same secret so we have to create separate secret for passphrase as follows.
$ aws secretsmanager create-secret --name passwd --secret-string '{"passphrase" : "pass123"}' --region us-east-1
NOTE: The secret type for private key is binary whereas for passphrase it is plain text.
After creating secret, we can use AWS secrets manager code to get private key and passphrase. The AWS Secrets Manager code decodes private key using base64.b64decode(..) method.
Secrets Manager does not require you to store the data in JSON format, it can store arbitrary strings or binary data.
You could either chose to break everything up and store it in separate secrets, or use a data format that supports new lines like XML.
The private key which you will store, won't have special characters like '\n', '\r'.
To resolve this issue, copy the output of private_key, which will have special characters.
private_key = open('/home/ec2-user/GPG/test_private_key.asc').read()
private_key
Place this private key into your secret & get it using get_secret()
Note: you will see the additional '' in the private key which will get using load_json, to handle that you need to use private_key.replace('\n','\n')
Your code will look like below.
private_key = newdict['private_key']
private_key = private_key.replace('\n','\n')
Then you will be able to get the keys.
Im new to Cloud KMS, and I started following exactly what's written here
I encrypted my data file which is saved in UTF-8 format by running this command
gcloud kms encrypt --location global --keyring ring --key key --plaintext-file /path_to_file --ciphertext-file /path_to_enc --project myProject
then as a result my encrypted data has been presented in this format in my new created encrypted file
$�]ˋLݿ���yHI�lS�`&�Nt�b{%�U�� �&�A���XaL��d
here is how I read the encrypted file data:
static Properties properties = new Properties();
static {
try {
InputStream in = new Credentials().getClass().getResourceAsStream("path_to_enc_file");
byte[] encryptedData = IOUtils.toByteArray(in);
byte[] decryptedBytes = decrypt(EnvironmentVariable.getProjectId(), "global", "ring", "key", encryptedData);
ByteArrayInputStream bis = new ByteArrayInputStream(decryptedBytes);
properties.load(bis);
in.close();
bis.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
and now whenever I try to decrypt it by this function:
public static byte[] decrypt(
String projectId, String locationId, String keyRingId, String cryptoKeyId, byte[] ciphertext)
throws IOException {
// Create the KeyManagementServiceClient using try-with-resources to manage client cleanup.
try (KeyManagementServiceClient client = KeyManagementServiceClient.create()) {
// The resource name of the cryptoKey
String resourceName = CryptoKeyName.format(projectId, locationId, keyRingId, cryptoKeyId);
// Decrypt the ciphertext with Cloud KMS.
DecryptResponse response = client.decrypt(resourceName, ByteString.copyFrom(ciphertext));
// Extract the plaintext from the response.
return response.getPlaintext().toByteArray();
}
}
it throw this
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Decryption failed: the ciphertext is invalid.",
"reason" : "badRequest"
} ],
"message" : "Decryption failed: the ciphertext is invalid.",
"status" : "INVALID_ARGUMENT"
}
the key type is: Symmetric encrypt/decrypt Default Algorithm: Google symmetric key
the ring location: global
Can you plz help me out and tell me what's missing in google docs?
Update: As bdhess says in the comment, this is probably due to Maven being "helpful" and corrupting the data during the build process. See the Maven docs for how to avoid this.
The solution below also works, but is less straightforward.
Tamer and I chatted for a while and got a workaround:
Encode the output from gcloud in base64 before including it in a file in src/main/resources.
Decode the file after reading it with java.util.Base64.
Pass the decoded bytes to the KMS API.
For some reason the bytes were getting corrupted between creating the file with gcloud and reading the bytes in with getResourceAsStream(). From the code above I can't see where the corruption would be happening, and it seems like reading in binary resources should be totally supported. But something is breaking somewhere in Tamer's case.
I'll try to reproduce it sometime this week.
To decrypt a secret from a file to a plaintext file:
cat secret.enc | gcloud kms decrypt \
--location=global \
--keyring=keyring \
--key=key \
--ciphertext-file=- \
--plaintext-file=decrypted_secret.txt
You'd need to decode the encrypted key first with base64, and then pipe the output to the whole gcloud kms command, e.g:
cat my-token.enc | base64 --decode | gcloud kms decrypt --plaintext-file=plaintextfile --ciphertext-file=- --location=global --keyring=yourkeyringname --key=yourkeyname
I did that modifications then it worked like a charm with a great help from #hjfreyer
1- to encrypt the plain text secret I did that
run this command -->
gcloud kms encrypt --location global --plaintext-file PATH_TO_SECRET_FILE --ciphertext-file PATH_TO_TMP_FILE --project myProject --key key --keyring ring
Encode the result base64 -->
base64 PATH_TO_TMP_FILE > PATH_TO_FINAL_ENC_FILE
remove new line from the FINAL_ENC_FILE file
2- to decrypt the data back first I need to base64 decode it then pass it to the decrypt KMS function
InputStream in = new Credentials().getClass().getResourceAsStream("PATH_TO_FINAL_ENC_FILE");
byte[] encryptedData = IOUtils.toByteArray(in);
byte[] decryptedBytes = decrypt(EnvironmentVariable.getProjectId(), "global", "ring", "key", Base64.getDecoder().decode(encryptedData));
I've created a basic client and server that pass a string, which I've changed to JSON instead. But the JSON string is only parsable before it gets sent through TCP. After it's sent, the string version is identical (after a chomp), but on the server side it no longer processes the JSON correctly. Here is some of my code (with other bits trimmed)
Some of the client code
require 'json'
require 'socket'
foo = {'a' => 1, 'b' => 2, 'c' => 3}
puts foo.to_s + "......."
foo.to_json
puts foo['b'] # => outputs the correct '2' answer
client = TCPSocket.open('localhost', 2000)
client.puts json
client.close;
Some of the server code
require 'socket'
require 'json'
server = TCPServer.open(2000)
while true
client = server.accept # Accept client
response = client.gets
print response
response = response.chomp
response.to_json
puts response['b'] # => outputs 'b'
end
The output 'b' should be '2' instead. How do I fix this?
Thanks
In your server you wrote response.to_json. This turns a string to JSON, then throws it away. And I don't like the .chomp, either.
Try
response = client.gets
hash = JSON.parse(response)
Now hash is a Ruby Hash object with your data in it, and hash['b'] should work correctly.
The problem is that .to_json does not parse JSON inside a string and replace itself with the result. It is used to convert the string into a format that is an acceptable JSON value.
require 'json'
string = "abc"
puts string
puts string.to_json
This will output:
abc
"abc"
The method is added to the String class by the JSON generator and it uses it internally to generate the JSON document.
But why does your response['b'] return "b"?
Because Ruby strings have a method called [] that can be used to:
Return a substring: "abc"[0,2] => "ab"
Return a single character from index: "abc"[1] => "b"
Return a substring if the string contains it: "abc"["bc"] => "bc", "abc"["fg"] => nil
Return a regexp match: "abc"[/^a([a-z])c/, 1] => "b"
and possibly some other ways I can't think of right now.
So this happens because your response is a string that has the character "b" in it:
response = "something with a b"
puts response["b"]
# outputs b
puts response["x"]
# outputs a blank line because response does not contain "x".
Instead of .to_json your code has to call JSON.parse or JSON.load:
data = JSON.parse(response)
puts data['b']
I have a perl hash that is obtained from parsing JSON. The JSON could be anything a user defined API could generated. The goal is to obtain a date/time string and determine if that date/time is out of bounds according to a user defined threshold. The only issue I have is that perl seems a bit cumbersome when dealing with hash key/subkey iteration. How can I look through all the keys and determine if a key or subkey exists throughout the hash? I have read many threads throughout stackoverflow, but nothing that exactly meets my needs. I only started perl last week so I may be missing something... Let me know if that's the case.
Below is the "relevant" code/subs. For all code see: https://gitlab.com/Jedimaster0/check_http_freshness
use warnings;
use strict;
use LWP::UserAgent;
use Getopt::Std;
use JSON::Parse 'parse_json';
use JSON::Parse 'assert_valid_json';
use DateTime;
use DateTime::Format::Strptime;
# Verify the content-type of the response is JSON
eval {
assert_valid_json ($response->content);
};
if ( $# ){
print "[ERROR] Response isn't valid JSON. Please verify source data. \n$#";
exit EXIT_UNKNOWN;
} else {
# Convert the JSON data into a perl hashrefs
$jsonDecoded = parse_json($response->content);
if ($verbose){print "[SUCCESS] JSON FOUND -> ", $response->content , "\n";}
if (defined $jsonDecoded->{$opts{K}}){
if ($verbose){print "[SUCCESS] JSON KEY FOUND -> ", $opts{K}, ": ", $jsonDecoded->{$opts{K}}, "\n";}
NAGIOS_STATUS(DATETIME_DIFFERENCE(DATETIME_LOOKUP($opts{F}, $jsonDecoded->{$opts{K}})));
} else {
print "[ERROR] Retreived JSON does not contain any data for the specified key: $opts{K}\n";
exit EXIT_UNKNOWN;
}
}
sub DATETIME_LOOKUP {
my $dateFormat = $_[0];
my $dateFromJSON = $_[1];
my $strp = DateTime::Format::Strptime->new(
pattern => $dateFormat,
time_zone => $opts{z},
on_error => sub { print "[ERROR] INVALID TIME FORMAT: $dateFormat OR TIME ZONE: $opts{z} \n$_[1] \n" ; HELP_MESSAGE(); exit EXIT_UNKNOWN; },
);
my $dt = $strp->parse_datetime($dateFromJSON);
if (defined $dt){
if ($verbose){print "[SUCCESS] Time formatted using -> $dateFormat\n", "[SUCCESS] JSON date converted -> $dt $opts{z}\n";}
return $dt;
} else {
print "[ERROR] DATE VARIABLE IS NOT DEFINED. Pattern or timezone incorrect."; exit EXIT_UNKNOWN
}
}
# Subtract JSON date/time from now and return delta
sub DATETIME_DIFFERENCE {
my $dateInitial = $_[0];
my $deltaDate;
# Convert to UTC for standardization of computations and it's just easier to read when everything matches.
$dateInitial->set_time_zone('UTC');
$deltaDate = $dateNowUTC->delta_ms($dateInitial);
if ($verbose){print "[SUCCESS] (NOW) $dateNowUTC UTC - (JSON DATE) $dateInitial ", $dateInitial->time_zone->short_name_for_datetime($dateInitial), " = ", $deltaDate->in_units($opts{u}), " $opts{u} \n";}
return $deltaDate->in_units($opts{u});
}
Sample Data
{
"localDate":"Wednesday 23rd November 2016 11:03:37 PM",
"utcDate":"Wednesday 23rd November 2016 11:03:37 PM",
"format":"l jS F Y h:i:s A",
"returnType":"json",
"timestamp":1479942217,
"timezone":"UTC",
"daylightSavingTime":false,
"url":"http:\/\/www.convert-unix-time.com?t=1479942217",
"subkey":{
"altTimestamp":1479942217,
"altSubkey":{
"thirdTimestamp":1479942217
}
}
}
[SOLVED]
I have used the answer that #HåkonHægland provided. Here are the below code changes. Using the flatten module, I can use any input string that matches the JSON keys. I still have some work to do, but you can see the issue is resolved. Thanks #HåkonHægland.
use warnings;
use strict;
use Data::Dumper;
use LWP::UserAgent;
use Getopt::Std;
use JSON::Parse 'parse_json';
use JSON::Parse 'assert_valid_json';
use Hash::Flatten qw(:all);
use DateTime;
use DateTime::Format::Strptime;
# Verify the content-type of the response is JSON
eval {
assert_valid_json ($response->content);
};
if ( $# ){
print "[ERROR] Response isn't valid JSON. Please verify source data. \n$#";
exit EXIT_UNKNOWN;
} else {
# Convert the JSON data into a perl hashrefs
my $jsonDecoded = parse_json($response->content);
my $flatHash = flatten($jsonDecoded);
if ($verbose){print "[SUCCESS] JSON FOUND -> ", Dumper($flatHash), "\n";}
if (defined $flatHash->{$opts{K}}){
if ($verbose){print "[SUCCESS] JSON KEY FOUND -> ", $opts{K}, ": ", $flatHash>{$opts{K}}, "\n";}
NAGIOS_STATUS(DATETIME_DIFFERENCE(DATETIME_LOOKUP($opts{F}, $flatHash->{$opts{K}})));
} else {
print "[ERROR] Retreived JSON does not contain any data for the specified key: $opts{K}\n";
exit EXIT_UNKNOWN;
}
}
Example:
./check_http_freshness.pl -U http://bastion.mimir-tech.org/json.html -K result.creation_date -v
[SUCCESS] JSON FOUND -> $VAR1 = {
'timestamp' => '20161122T200649',
'result.data_version' => 'data_20161122T200649_data_news_topics',
'result.source_version' => 'kg_release_20160509_r33',
'result.seed_version' => 'seed_20161016',
'success' => 1,
'result.creation_date' => '20161122T200649',
'result.data_id' => 'data_news_topics',
'result.data_tgz_name' => 'data_news_topics_20161122T200649.tgz',
'result.source_data_version' => 'seed_vtv: data_20161016T102932_seed_vtv',
'result.data_digest' => '6b5bf1c2202d6f3983d62c275f689d51'
};
Odd number of elements in anonymous hash at ./check_http_freshness.pl line 78, <DATA> line 1.
[SUCCESS] JSON KEY FOUND -> result.creation_date:
[SUCCESS] Time formatted using -> %Y%m%dT%H%M%S
[SUCCESS] JSON date converted -> 2016-11-22T20:06:49 UTC
[SUCCESS] (NOW) 2016-11-26T19:02:15 UTC - (JSON DATE) 2016-11-22T20:06:49 UTC = 94 hours
[CRITICAL] Delta hours (94) is >= (24) hours. Data is stale.
You could try use Hash::Flatten. For example:
use Hash::Flatten qw(flatten);
my $json_decoded = parse_json($json_str);
my $flat = flatten( $json_decoded );
say "found" if grep /(?:^|\.)\Q$key\E(?:\.?|$)/, keys %$flat;
You can use Data::Visitor::Callback to traverse the data structure. It lets you define callbacks for different kinds of data types inside your structure. Since we're only looking at a hash it's relatively simple.
The following program has a predefined list of keys to find (those would be user input in your case). I converted your example JSON to a Perl hashref and included it in the code because the conversion is not relevant. The program visits every hashref in this data structure (including the top level) and runs the callback.
Callbacks in Perl are code references. These can be created in two ways. We're doing the anonymous subroutine (sometimes called lambda function in other languages). The callback gets passed two arguments: the visitor object and the current data substructure.
We'll iterate all the keys we want to find and simply check if they exist in that current data structure. If we see one, we count it's existence in the %seen hash. Using a hash to store things we have seen is a common idiom in Perl.
We're using a postfix if here, which is convenient and easy to read. %seen is a hash, so we access the value behind the $key with $seen{$key}, while $data is a hash reference, so we use the dereferencing operator -> to access the value behind $key with $data->{$key}.
The callback needs us to return the $data again so it continues. The last line is just there, it's not important.
I've used Data::Printer to output the %seen hash because it's convenient. You can also use Data::Dumper if you want. In production, you will not need that.
use strict;
use warnings;
use Data::Printer;
use Data::Visitor::Callback;
my $from_json = {
"localDate" => "Wednesday 23rd November 2016 11:03:37 PM",
"utcDate" => "Wednesday 23rd November 2016 11:03:37 PM",
"format" => "l jS F Y h:i:s A",
"returnType" => "json",
"timestamp" => 1479942217,
"timezone" => "UTC",
"daylightSavingTime" =>
0, # this was false, I used 0 because that's a non-true value
"url" => "http:\/\/www.convert-unix-time.com?t=1479942217",
"subkey" => {
"altTimestamp" => 1479942217,
"altSubkey" => {
"thirdTimestamp" => 1479942217
}
}
};
my #keys_to_find = qw(timestamp altTimestamp thirdTimestamp missingTimestamp);
my %seen;
my $visitor = Data::Visitor::Callback->new(
hash => sub {
my ( $visitor, $data ) = #_;
foreach my $key (#keys_to_find) {
$seen{$key}++ if exists $data->{$key};
}
return $data;
},
);
$visitor->visit($from_json);
p %seen;
The program outputs the following. Note this is not a Perl data structure. Data::Printer is not a serializer, it's a tool to make data human readable in a convenient way.
{
altTimestamp 1,
thirdTimestamp 1,
timestamp 1
}
Since you also wanted to constraint the input, here's an example how to do that. The following program is a modification of the one above. It allows to give a set of different constraints for every required key.
I've done that by using a dispatch table. Essentially, that's a hash that contains code references. Kind of like the callbacks we use for the Visitor.
The constraints I've included are doing some things with dates. An easy way to work with dates in Perl is the core module Time::Piece. There are lots of questions around here about various date things where Time::Piece is the answer.
I've only done one constraint per key, but you could easily include several checks in those code refs, or make a list of code refs and put them in an array ref (keys => [ sub(), sub(), sub() ]) and then iterate that later.
In the visitor callback we are now also keeping track of the keys that have %passed the constraints check. We're calling the coderef with $coderef->($arg). If a constraint check returns a true value, it gets noted in the hash.
use strict;
use warnings;
use Data::Printer;
use Data::Visitor::Callback;
use Time::Piece;
use Time::Seconds; # for ONE_DAY
my $from_json = { ... }; # same as above
# prepare one of the constraints
# where I'm from, Christmas eve is considered Christmas
my $christmas = Time::Piece->strptime('24 Dec 2016', '%d %b %Y');
# set up the constraints per required key
my %constraints = (
timestamp => sub {
my ($epoch) = #_;
# not older than one day
return $epoch < time && $epoch > time - ONE_DAY;
},
altTimestamp => sub {
my ($epoch) = #_;
# epoch value should be an even number
return ! $epoch % 2;
},
thirdTimestamp => sub {
my ($epoch) = #_;
# before Christmas 2016
return $epoch < $christmas;
},
);
my %seen;
my %passed;
my $visitor = Data::Visitor::Callback->new(
hash => sub {
my ( $visitor, $data ) = #_;
foreach my $key (%constraints) {
if ( exists $data->{$key} ) {
$seen{$key}++;
$passed{$key}++ if $constraints{$key}->( $data->{$key} );
}
}
return $data;
},
);
$visitor->visit($from_json);
p %passed;
The output this time is:
{
thirdTimestamp 1,
timestamp 1
}
If you want to learn more about the dispatch tables, take a look at chapter two of the book Higher Order Perl by Mark Jason Dominus which is legally available for free here.
I am trying to save data from a Hash to a file. I convert it to JSON and dump it into the file.
When I try to parse back from file to hash I get JSON::ParserError
Code to convert Hash to JSON file: (works fine)
user = {:email => "cumber#cc.cc", :passwrd => "hardPASSw0r|)"}
student_file = File.open("students.txt", "a+") do |f|
f.write JSON.dump(user)
end
After adding a few values one by one to the file it looks something like this:
{"email":"test1#gmail.com","passwrd":"qwert123"}{"email":"test3#gmail.com","passwrd":"qwert12345"}{"email":"cumber#cc.cc","passwrd":"hardPASSw0r|)"}
I tried the following code to convert back to Hash but it doesn't work:
file = File.read('students.txt')
data_hash = JSON.parse(file)
I get
System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/json/common.rb:155:in `parse': 757: unexpected token at '{"email":"test3#gmail.com","passwrd":"qwert12345"}{"email":"cumber#cc.cc","passwrd":"hardPASSw0r|)"}' (JSON::ParserError)
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/json/common.rb:155:in `parse'
from hash_json.rb:25:in `<main>'
My goal is to be able to add and remove values from the file.
How do I fix this, where was my mistake? Thank you.
This should work:
https://repl.it/EXGl/0
# as adviced by #EricDuminil, on some envs you need to include 'json' too
require 'json'
user = {:email => "cumber#cc.cc", :passwrd => "hardPASSw0r|)"}
student_file = File.open("students.txt", "w") do |f|
f.write(user.to_json)
end
file = File.read('students.txt')
puts "saved content is: #{JSON.parse(file)}"
p.s. hope that this is only an example, never store passwords in plain-text! NEVER ;-)