Process a file containing a collection of JSON strings - json

Question

(Title edited. The original title: "json content: print out parts of it as is, pipe parts of it to get human readable timestamps, output from the same command")
I have a json alike content in a file:
{
"newState": "runnable",
"setAt": 1587421159359
}
{
"newState": "running",
"setAt": 1587421282891
}
{
"newState": "debug_hold",
"setAt": 1587422014895
}
{
"newState": "terminating",
"setAt": 1587424788577
}
{
"newState": "failed",
"setAt": 1587424796544
}
I can extract the 'newState' by cat timestamps.json | jq -r '.newState':
runnable
running
debug_hold
terminating
failed
I can extract the epoch timestamps and format it into a human readable form by cat timestamps.json | jq -r '.setAt' | rev | cut -c 4- | rev | perl -pe 's/(\d+)/localtime($1)/e':
Mon Apr 20 18:19:19 2020
Mon Apr 20 18:21:22 2020
Mon Apr 20 18:33:34 2020
Mon Apr 20 19:19:48 2020
Mon Apr 20 19:19:56 2020
How can I combine the two outputs so the result becomes
runnable Mon Apr 20 18:19:19 2020
running Mon Apr 20 18:21:22 2020
debug_hold Mon Apr 20 18:33:34 2020
terminating Mon Apr 20 19:19:48 2020
failed Mon Apr 20 19:19:56 2020
I think I can do some bash for loop and array input but was wondering if jq has something that can pipe a portion of the content (e.g. epoch time in this case) out, process it, then feed the value back into the jq parse output.

Answer 1

You may be looking something like this.
cat timestamps.json | jq -r '[.newState, .setAt] | join(" ")'

Answer 2

With input being a collection of (unrelated) valid JSON strings you can read in {} chunks.
Set the input record separator ($/) to } and then the <> operator each time reads up to }
use warnings;
use strict;
use feature 'say';
use JSON qw(decode_json);
my $file = shift // die "Usage: $0 file\n"; #/
open my $fh, '<', $file or die "Can't open $file: $!";
local $/ = '}'; # presumably all this code is in some local scope
while (my $record = <$fh>) {
next if not $record =~ /\S/;
my $json = decode_json($record);
say $json->{newState}, ' ', scalar localtime $json->{setAt}/1000;
}
Comments
This relies on the shown format of the input, in particular that it has no nested objects. If there are nested {...} then slurp the whole file and extract JSON strings using Text::Balanced or equivalent (or, of course, use another approach)
I'd actually recommend to use Cpanel::JSON::XS
When global variables like $/ need be changed that is best done in the smallest scope needed and with local. Here it doesn't matter but I presume this to be a part of a larger program
There may be empty strings and, in particular, newlines left over when reading this way thus the check of whether the record contains any non-whitespace
The timestamps in your input are off by a factor of thousand from seconds-since-epoch, I guess because they carry milliseconds as well. I just divide by 1000 for simplicity
Note that the shown desired timestamps may become a problem if daylight saving time gets involved, and if that is the case you want to extract and include the time zone as well
The simplest (and flexible) way to get timezone from the epoch is by using POSIX::strftime. It takes the list from localtime and returns a string generated according to the given format.
The %z specifier produces the timezone as the UTC offset, while %Z produces the (notorious and unportable) short name. See your system's strftime manpage for details. Example
use POSIX qw(strftime);
say strftime "%z %Z", localtime; #--> -0700 PDT
(thanks to ikegami's answer which proded me to add the timezone discussion)

Answer 3

Using the incremental parsing feature of JSON parsers, one can safely parse sequences of JSON documents such as the one you have with very little code. This means there's no point in hacking together a JSON parser using regex matches.
use Cpanel::JSON::XS qw( );
my $decoder = Cpanel::JSON::XS->new();
while (<>) {
$decoder->incr_parse($_);
while ( my $rec = $decoder->incr_parse() ) {
say sprintf "%-11s %s",
$rec->{newState},
format_ts($rec->{setAt});
}
}
Complete program:
#!/usr/bin/perl
use strict;
use warnings;
use feature qw( say );
use utf8;
use open ':std', ':encoding(UTF-8)';
use Cpanel::JSON::XS qw( );
use POSIX qw( strftime );
sub format_ts {
my ($ts) = #_;
my $ms = $ts % 1000;
my $epoch = ( $ts - $ms ) / 1000;
my #lt = localtime($epoch);
return sprintf("%s.%03d %s",
strftime("%a %b %d %H:%M:%S", #lt),
$ms,
strftime("%Y %z", #lt),
);
}
my $decoder = Cpanel::JSON::XS->new();
while (<>) {
$decoder->incr_parse($_);
while ( my $rec = $decoder->incr_parse() ) {
say sprintf "%-11s %s",
$rec->{newState},
format_ts($rec->{setAt});
}
}
Output:
runnable Mon Apr 20 18:19:19.359 2020 -0400
running Mon Apr 20 18:21:22.891 2020 -0400
debug_hold Mon Apr 20 18:33:34.895 2020 -0400
terminating Mon Apr 20 19:19:48.577 2020 -0400
failed Mon Apr 20 19:19:56.544 2020 -0400
Note that I added time zone information because the timestamps would be ambiguous without it (because of overlaps when switching from daylight-saving time to standard time). I also showed how you can keep the milliseconds if you so desire.

Answer 4

A small perl script can process such data with ease
USAGE: script_name.pl timestamps.json
#!/usr/bin/perl
use strict;
use warnings;
my($state,$time);
while(<>) {
chomp;
$state = $1 if /"newState": "(.*)"/;
$time = $1 if /"setAt": (\d+)/;
printf "%-12s %s\n", $state, "".localtime($time/1000) if /}/;
}
Alternative version
use strict;
use warnings;
my $data = do { local $/; <> };
my %state = $data =~ /"newState": "(.*?)".*?"setAt": (\d+)/sg;
while(my($s,$t) = each %state) {
printf "%-12s %s\n", $s, "".localtime($t/1000);
}
Input file timestamps.json
{
"newState": "runnable",
"setAt": 1587421159359
}
{
"newState": "running",
"setAt": 1587421282891
}
{
"newState": "debug_hold",
"setAt": 1587422014895
}
{
"newState": "terminating",
"setAt": 1587424788577
}
{
"newState": "failed",
"setAt": 1587424796544
}
Output
runnable Mon Apr 20 15:19:19 2020
running Mon Apr 20 15:21:22 2020
debug_hold Mon Apr 20 15:33:34 2020
terminating Mon Apr 20 16:19:48 2020
failed Mon Apr 20 16:19:56 2020