I have a series of JSON objects and I need to replace all the commas at the end of each object with a pipe |
Obviously I can't use Find & Replace because that would replace every comma in the JSON, but I only want to replace those at the end of each object.
For example:
{
"id":123,
"name":Joe,
"last":Smith
} , <----- I want to replace this comma only
{"id":454
"name":Bill,
"last":Smith
}
You could parse the JSON by adding '[]' around it and then re-serialize it.
With a PHP script you could do something like this:
$content = file_get_contents('/path/to/yourfile.json');
// Add [] around the JSON to make it valid:
$json = json_decode('[' . $json . ']', true);
$result = '';
foreach ($json as $j) {
if ($result != '') $result .= '|';
$result .= json_encode($j);
}
echo $result;
There is already a PHP solution. Here's a Regex solution in case.
string s1 = "{\"id\":123,\"name\":Joe,\"last\":Smith} , {\"id\":454,\"name\":Bill,\"last\":Smith}";
string pattern = "} , {";
var s2 = Regex.Split(s1, pattern);
string s3 = string.Join(" | ", s2);
Related
I have csv file containing (the SQL query is simplified):
SQL_QRY
"insert into TTL_CCL_DB_INF.D_1244_CRB_PARTY_EXT (CD_REC_PARTY, CD_REC_OBJECT_TYPE, T_FULL_NAME, ..) select * from TTL_CCL_DB_STG.D_1244_CRB_PARTY aa1 inner join TTL_CCL_DB_STG.D_1244_CRB_COMPANYFACT bb1 on aa1.CD_REC_PARTY = bb1.CD_REC_PARTY"
I have a perl script to read the file into hash:
sub somesub {
...
my $fh;
my $key;
eval { open($fh, '<', $tmp_file); };
if ($#) {
$errmsg = $#;
croak {message=>$errmsg};
};
while(my $lines = <$fh>) {
chomp $lines;
my #data = split(/|/, $lines);
$key = shift #data;
$data_rt{$key} = \#data;
};
close $fh;
unlink $tmp_file;
return %data_rt;
}
But the returned hash looks like:
RETURN >>>>$VAR1 = {
' => [], 'SQL_QRY
'"insert into TTL_CCL_DB_INF.D_1244_CRB_PARTY_EXT (CD_REC_PARTY, CD_REC_OBJECT_TYPE, T_FULL_NAME, CD_TAX_IDENTIFIER, CD_VAT_IDENTIFIER,DWH_TRANSFRM_ID, DWH_TRANSFRM_RUN_ID, DWH_BD, DWH_VSN_NO ) select aa1.CD_REC_PARTY, aa1.CD_REC_OBJECT_TYPE, aa1.T_FULL_NAME, bb1.CD_TAX_IDENTIFIER, bb1.CD_VAT_IDENTIFIER, $transfrmId, \'$transfrmRunId\', cast(\'$bnsDt\' as date format \'YYYYMMDD\'), $occNbr from TTL_CCL_DB_STG.D_1244_CRB_PARTY aa1 inner join TTL_CCL_DB_STG.D_1244_CRB_COMPANYFACT bb1 on aa1.C' => []ARTY = bb1.CD_REC_PARTY and cast(aa1.DWH_BD as date format \'YYYYMMDD\') = \'$10000050_dwh_bd\' and aa1.DWH_VSN_NO = $10000050_vsn_no and cast(bb1.DWH_BD as date format \'YYYYMMDD\') = \'$10000034_dwh_bd\' and bb1.DWH_VSN_NO = $10000034_vsn_no"
};
Would anybody help me how to do that correctly? I need to have it as much flexible as possible (csv file can contain more rows and more columns)
The sample data that you show us doesn't seem to match your code. The code wants to split the data on pipe symbols. But the data doesn't contain any pipe symbols. So I'm not sure how that's going to work.
However, I'm pretty confident that the problem is on this line:
my #data = split(/|/, $lines);
The first argument to split() is a regex. And a pipe symbol is a regex metacharacter. In order to use a metacharacter as itself, you need to escape it with a backslash.
my #data = split(/\|/, $lines);
Hope some Perl gurus out there can help me out here. Basically my issue is when a JSON string starts with a "[" instead of a "{", Perl doesn't treat the variable as a hash after I use decode_json.
Here's a sample code.
#!/usr/bin/perl
use JSON;
use Data::Dumper;
$string1 = '{"Peti Bar":{"Literature":88,"Mathematics":82,"Art":99},"Foo Bar":{"Literature":67,"Mathematics":97}}';
$string = '[{"ActionID":5,"ActionName":"TEST- 051017"},{"ActionID":10,"ActionName":"Something here"},{"ActionID":13,"ActionName":"Some action"},{"ActionID":141,"ActionName":"Email Reminder"}]';
print "First string that starts with \"{\" below:\n$string1\n\n";
my $w = decode_json $string1;
my $count = keys %$w;
print "printing \$count's value -> $count\n\n";
print "Second string starts with \"[\" below:\n$string\n\n";
my $x = decode_json $string;
my $count2 = keys %$x;
print "printing \$count2's value -> $count2\n\n";
Below is the script output.
Both $w and $x works though. It's just I have to use keys $x instead of keys %$x on the other json string.
Now the issue with using that is I get a keys on reference is experimental at tests/jsontest.pl error. It won't stop the script but I'm worried about future compatibility issues.
What's the best way to approach this?
Use the ref function to determine what type the reference is. See perldoc -f ref.
my $w = decode_json $string1;
my $count = 1;
if( my $ref = ref( $w ) ){
if( $ref eq 'HASH' ){
$count = keys %$w;
}elsif( $ref eq 'ARRAY' ){
$count = scalar #$w;
}else{
die "invalid reference '$ref'\n";
}
}
I'm having trouble processing the returned results from a DB SQL Mapper into a recognizable json encoded array.
function apiCheckSupplyId() {
/*refer to the model Xrefs*/
$supply_id = $this->f3->get('GET.supply_id');
$xref = new Xrefs($this->tongpodb);
$supply = $xref->getBySupplyId( $supply_id );
if ( count( $supply ) == 0 ) {
$this->logger->write('no xref found for supply_id=' .$supply_id);
$supply = array( array('id'=>0) );
echo json_encode( $supply );
} else {
$json = array();
foreach ($supply as $row){
$item = array();
foreach($row as $key => $value){
$item[$key] = $value;
}
array_push($json, $item);
}
$this->logger->write('xref found for supply_id=' .$supply_id.json_encode( $json ) );
echo json_encode( $json );
}
}
This is the method I am using but it seems very clunky to me. Is there a better way?
Assuming the getBySupplyId returns an array of Xref mappers, you could simplify the whole thing like this:
function apiCheckSupplyId() {
$supply_id = $this->f3->get('GET.supply_id');
$xref = new Xrefs($this->tongpodb);
$xrefs = $xref->getBySupplyId($supply_id);
echo json_encode(array_map([$xref,'cast'],$xrefs));
$this->logger->write(sprintf('%d xrefs found for supply_id=%d',count($xrefs),$supply_id));
}
Explanation:
The $xrefs variable contains an array of mappers. Each mapper being an object, you have to cast it to an array before encoding it to JSON. This can be done in one line by mapping the $xref->cast() method to each record: array_map([$xref,'cast'],$xrefs).
If you're not confident with that syntax, you can loop through each record and cast it:
$cast=[];
foreach ($xrefs as $x)
$cast[]=$x->cast();
echo json_encode($cast);
The result is the same.
The advantage of using cast() other just reading each value (as you're doing in your original script) is that it includes virtual fields as well.
Edit: modified code and output to make it more clear
Edit 2: Added example input for reproduction
I have a JSON file and a CSV file and I am running comparisons on the two. The problem is that $asset_ip is correctly defined in the outer foreach loop, but when in the nested loop $asset_ip becomes undefined.
Why is $asset_ip becoming undefined?
#!/usr/bin/perl
# perl -e'use CPAN; install "Text::CSV"'
use strict;
use warnings;
use JSON::XS;
use File::Slurp;
use Text::CSV;
my $csv = Text::CSV->new( { sep_char => ',' } );
my $csv_source = "servers.csv";
my $json_source = "assets.json";
my $dest = "servers_for_upload.csv";
# defined these here as I need to use them in foreach loop and if statement:
my $csv_ip;
my #fields;
open( my $csv_fh, '<', $csv_source ) or die "$! error trying to read";
open( my $dest_fh, '>', $dest ) or die "$! error trying to read";
my $json = read_file($json_source);
my $json_array = decode_json $json;
foreach my $item (#$json_array) {
my $id = $item->{id};
my $asset_ip = $item->{interfaces}->[0]->{ip_addresses}->[0]->{value};
# test the data is there:
if ( defined $asset_ip ) {
print "id: " . $id . "\nip: " . $asset_ip . "\n";
}
while (my $line = <$csv_fh>) {
chomp $line;
if ( $csv->parse($line) ) {
#fields = $csv->fields();
$csv_ip = $fields[0];
}
else {
warn "Line could not be parsed: $line\n";
}
if ( $csv_ip eq $asset_ip ) {
# preppend id to csv array and write these lines to new file
unshift( #fields, $id );
print $dest_fh join( ", ", #fields );
}
}
}
close $csv_fh;
Output:
Use of uninitialized value $asset_ip in string eq at script.pl line 43, <$csv_fh> line 1.
Use of uninitialized value $asset_ip in string eq at script.pl line 43, <$csv_fh> line 2.
Use of uninitialized value $asset_ip in string eq at script.pl line 43, <$csv_fh> line 3.
id: 1003
ip: 192.168.0.2
id: 1004
ip: 192.168.0.3
id: 1005
ip: 192.168.0.4
assets.json:
[{"id":1001,"interfaces":[]},{"id":1003,"interfaces":[{"ip_addresses":[{"value":"192.168.0.2"}]}]},{"id":1004,"interfaces":[{"ip_addresses":[{"value":"192.168.0.3"}]}]},{"id":1005,"interfaces":[{"ip_addresses":[{"value":"192.168.0.4"}]}]}]
Note, that for the first iteration, $asset_ip will be undefined. I will therefore alter the code to only run the eq comparison if $asset_ip is defined. However, for this example I am not doing the check because all iterations are undefined.
servers.csv:
192.168.0.3,Brian,Germany
192.168.0.4,Billy,UK
192.168.0.5,Ben,UK
I think your problem will be this:
foreach my $line (<$csv_fh>) {
You execute this within our outer loop. But when you do this, your $csv_fh ends up at the end of file.
Once you have done this, subsequent iterations of your outer loop will not execute this inner loop, because there's nothing left for it to read from $csv_fh.
An easy test if this is your problem is to add a seek e.g. seek ( $csv_fh, 0, 0 );.
But this isn't an efficient thing to do, because then you'll be looping through the file multiple times - you should instead read it into a data structure and use that.
Edit: Here is your problem:
[{"id":1001,"interfaces":[]},{"id":1003,"interfaces":[{"ip_addresses":[{"value":"192.168.0.2"}]}]},{"id":1004,"interfaces":[{"ip_addresses":[{"value":"192.168.0.3"}]}]},{"id":1005,"interfaces":[{"ip_addresses":[{"value":"192.168.0.4"}]}]}]
And specifically:
[{"id":1001,"interfaces":[]}
Your first element in that array doesn't have a $asset_ip defined.
This means - on your first pass - $asset_ip is undefined and generates the errors. (no line is printed because of your if defined test).
But then - the code proceeds to traverse $csv_fh - reading to the end of file - looking for matches (and fails 3 times, generating 3 error messages.
Second iteration - for id 1002 - the IP isn't in the file anyway, but $csv_fh has already been read to end-of-file (EOF) - so that foreach loop doesn't execute at all.
This can be made workable by:
adding else next; after that if defined.
adding seek to after the while loop.
But really - a rewrite would be in order so you're not re-reading a file over and over anyway.
Very crudely:
#!/usr/bin/perl
# perl -e'use CPAN; install "Text::CSV"'
use strict;
use warnings;
use JSON::XS;
use File::Slurp;
use Text::CSV;
my $csv = Text::CSV->new( { sep_char => ',' } );
my $csv_source = "servers.csv";
my $json_source = "assets.json";
my $dest = "servers_for_upload.csv";
# defined these here as I need to use them in foreach loop and if statement:
my $csv_ip;
my #fields;
open( my $csv_fh, '<', $csv_source ) or die "$! error trying to read";
open( my $dest_fh, '>', $dest ) or die "$! error trying to read";
my $json = read_file($json_source);
my $json_array = decode_json $json;
foreach my $item (#$json_array) {
my $id = $item->{id};
my $asset_ip = $item->{interfaces}->[0]->{ip_addresses}->[0]->{value};
# test the data is there:
if ( defined $asset_ip ) {
print "id: " . $id . "\nip: " . $asset_ip . "\n";
}
else {
print "asset_ip undefined for id $id\n";
next;
}
while ( my $line = <$csv_fh> ) {
chomp $line;
if ( $csv->parse($line) ) {
#fields = $csv->fields();
$csv_ip = $fields[0];
}
else {
warn "Line could not be parsed: $line\n";
}
if ( $csv_ip eq $asset_ip ) {
# preppend id to csv array and write these lines to new file
unshift( #fields, $id );
print {$dest_fh} join( ", ", #fields ),"\n";
}
}
seek( $csv_fh, 0, 0 );
}
close $csv_fh;
I would suggest this also needs:
change of while so you're not re-reading the file each time
You're using Text::CSV so using a print join ( ","... doesn't seem a consistent choice. If your data warrants Text::CSV it's worth keeping it for output too.
my $url = "\'http://".$server.":4080/cgi-bin/gen_graph.pl?view=5&SUBSYS=\'";
my $html = HTML::TagParser->new( $url );
my #list = $html->getElementsByTagName( "pre" );
print $list[0];
foreach my $elem ( #list ) {
if($elem->innerText =~ /APIs/){
my $text = $elem->innerText;
if ( $text eq "" ) {
} else {
#API_list = split(/\s+/, $text);
print $API_list[1];
}
}
}
return \#API_list;
}
here the line my #list = $html->getElementsByTagName( "pre" ); not working. if i do this as a seperate script it is working well.. but if i include it in another script there is no value in #list. can anyone help me?
Are you getting an error message? If so, what is it?
Have you thought to check the return value of HTML::TagParser->new()? If it's failing, it may be doing so silently, and you only find out later when you try to use your $html object.
I do think the URL you're handing to it looks odd.
"\'http://".$server.":4080/cgi-bin/gen_graph.pl?view=5&SUBSYS=\'"
Why the two layers of quotes? (double quotes, and then escaped single quotes). Wouldn't this work:
my $url = 'http://'
. $server
. ':4080/cgi-bin-gen_graph.pl?view=5&SUBSYS=';
(Extra whitespace added to make it easier to read the concatenation operator.)