I'm doing the following to capture some ADO JSON data:
iteration="$(az boards iteration team list --team Test --project Test --timeframe current)"
Normally, the output of that command contains a JSON key/value pair like the following:
"path": "Test\\Sprint1"
But after capturing the STDOUT into that iteration variable, if I do
echo "$iteration"
That key/value pair becomes
"path": "Test\Sprint1"
And if I attempt to use jq on that output, it breaks because it's not recognized as valid JSON any longer. I'm very unfamiliar with Bash. How can I get that JSON to remain valid all the way through?
As already commented by markp-fuso:
It looks like your echo command is interpreting the backslashes. You can confirm this by running echo 'a\\b' and looking at the output.
The portable way to deal with such problems is to use printf instead of echo:
printf %s\\n "$iteration"
I'm trying to figure out what dictates if a value is returned from a PowerShell function or not, and I've run into some oddities. The about_return docs say:
In PowerShell, the results of each statement are returned as output, even without a statement that contains the Return keyword.
But this seems to glaze over details. If I run this:
function My-Function {
1
[System.Console]::WriteLine("Hello")
$null
$true
$false
0
2
}
Running this returns an array of (along with printing "Hello"):
1
True
False
0
2
Which means that $null isn't auto-returned. Then I tried incrementing since I'm doing using that in a function:
function My-Function {
$n = 1
$n
$n++
($n++)
-join #(1, 2, 3)
(-join #(1, 2, 3))
}
Returns:
1
2
123
123
So, $n and $n++ were returned, but ($n++) wasn't? But then when compared to another operator (-join), it's the same in each case. Why does wrapping parenthesis around $n++ prevent it from being returned, and why don't other operators behave the same? This is even more confusing since the = operator appears to work in the opposite way:
function My-Function {
($n = 1)
$n
$n++
($n++)
}
Returns:
1
1
2
Now wrapping the assignment causes it to be returned, whereas wrapping $n++ causes it to not be returned.
In summary, I'd just like to know a straightforward way of being able to look at a line of code in a function, and determine if it will cause a value to be returned or not.
This section discusses the specific statements from your sample functions.
See the bottom section for background information.
$n = 1 and $n++ are assignments and therefore do not produce output.
$n is an expression whose value is output
$null - ditto, but even though it is output, it doesn't display by default
($n++) - due to enclosure in (...) - turns the assignment into an expression and therefore does output the assigned value (too).
However, because you've used the post-increment form of the assignment, it is the old value that is output, not the now incremented value; to increment first (pre-increment) and then output, use (++$n)
[System.Console]::WriteLine("Hello") prints directly to the console, which bypasses PowerShell's system of output streams.
This means you cannot capture or redirect such output from inside PowerShell (see the next section).
PowerShell's output ("return value") behavior:
Tip of the hat to iRon for his help.
PowerShell, following the model of traditional shells, is organized around streams - see the conceptual about_Redirection help topic for an overview of all 6 streams that PowerShell supports.[1]
That is, any statement - and therefore potentially multiple ones - in a script or function can write to any of the output streams.
The primary output stream, meant to convey data, is the success output stream (whose number is 1), and only it is sent through the pipeline by default, and therefore by default only it is captured in a variable, suppressed, or redirected to a file.
There are two ways to write to the success output stream, i.e. to produce data output:
Explicitly, with a Write-Output call - although that is rarely needed.
Typically implicitly, by neither capturing, suppressing, nor redirecting output produced by a statement.
In other words: Output from any command (e.g., Get-ChildItem *.txt) or expression (e.g, 1 + 2 or (42).ToString('x')) is sent to the success output stream by default.
Unlike in traditional programming languages, return is not needed to produce output - in fact, its primary purpose is to exit the enclosing scope independently of any output the scope produces, though as a syntactic convenience you can combine the two aspects:
return <command-or-expression> is in effect the same as the following two statements, the first of which (potentially) produces output, the second of which exits the scope: <command-or-expression>; return
This implicit output behavior is convenient and and allows for concise, expressive code, but can also be a pitfall: it is easy to accidentally produce output - typically from a .NET method whose return value isn't needed (see this question for an example).
iRon's GitHub feature request #15781 discusses one potential way to remedy this problem: introduction of an opt-in strict mode that only permits using explicit output statements (Write-Output, return) in order to produce output.
This answer shows troubleshooting techniques you can use with the currently available features.
As for assignments - e.g. $n = 1; $n += 1; ++$n; $n--:
By default they do not produce output.
A hybrid case is the chaining form of a multi-assignment, e.g. $a = $b = 1, which assigns 1 to both variables: statement-internally the assignment value is passed through, but the statement as a whole has no output.
However, as an opt-in you can make them pass the value(s) being assigned through via (...), the grouping operator; e.g. ($n = 1) both assigns 1 to variable $n and outputs 1, which allows it to participate in larger expressions, such as ($n = 1) -gt 0
Note that the related $(...) (subexpression operator) and #(...) (array-subexpression operator) do not have that effect - they wrap one or more entire statement(s), without affecting the enclosed statements' intrinsic output behavior; e.g. $($n = 1) does not produce output, because $n = 1 by itself doesn't produce output; however, $(($n = 1)) does, because ($n = 1) by itself does.
As for output enumeration behavior:
By default, PowerShell enumerates collections that are being output, in the spirit of streaming output: That is, it sends a collection's elements to the pipeline, one by one.
In the rare event that you do need to output a collection as a whole - which in general should be avoided, so as not to confound other commands participating in a pipeline, which usually do expect object-by-object input - you have two options:
, $collection (sic; uses an aux. one-element wrapper array)
More explicitly, but less efficiently: Write-Output -NoEnumerate $collection
See this answer for more information.
As for outputting $null:
$null is output to the pipeline, but by default doesn't show.
$null by itself produces no visible output,
but the following returns $true, demonstrating that the value was sent:
$null | ForEach-Object { $null -eq $_ } # -> $true
Note that PowerShell also has an "array-valued $null" value that it uses to represent the lack of output from a command, which is technically represented as the [System.Management.Automation.Internal.AutomationNull]::Value singleton. In expression contexts, this values is treated the same as $null, but in the pipeline it behaves like an enumerable without elements and therefore sends nothing through the pipeline - see this answer for more information.
As for suppressing (discarding) unwanted output / redirecting to a file:
The best general-purpose way to suppress a statement's success output is to assign to $null ($null = ...); e.g.:
# .Add() outputs a value, which is usually unwanted.
$null = ($list = [System.Collections.ArrayList]::new()).Add('hi')
See this answer for a discussion and alternatives.
Note: The following discusses output suppression, via $null as the redirection target, but applies analogously to redirecting output to a file, by specifying a file name or path as the target.[2]
To selectively suppress a different output stream, prefix >$null with its number; e.g. 3>$null suppresses warning stream output.
To suppress output from all streams, which in the case of external programs covers both stdout and stderr, use redirection *>$null.
As for merging output streams:
Only the success output stream (stream number 1) can be merged into.
You can selectively merge one or more output streams into it (e.g. 2>&1 and/or 3>&1), or merge all (others): *>&1
In the resulting merged success output stream you can still identify what (non-success) stream a given object came from, by examining its type; e.g., error stream objects (stream 2) are [System.Management.Automation.ErrorRecord] instances - see this answer for more information.
As for bypassing PowerShell's system of streams:
Out-Host and [Console]::WriteLine() calls bypass PowerShell's output streams and write directly to the host / console (terminal). (A host is any environment that hosts the PowerShell engine, which is typically, but not necessarily a console (terminal); examples of other hosts are the PowerShell SDK and the host used in PowerShell remoting).
As such, their output cannot be captured, suppressed or redirected from inside PowerShell.
Write-Host formerly unconditionally bypassed PowerShell's output streams and still goes to the host by default, but - since PowerShell version 5 - routes its output via the information stream (stream number 6), where it can be captured / redirected on demand - see this answer for more information.
As for how output is formatted:
If output isn't captured, suppressed, or redirected, it is sent to the host (console) by default, where it is rendered based on PowerShell's rich and customizable for-display output-formatting system. See this answer for a concise overview.
Note that the resulting representations are designed for the human observer, not for programmatic processing. While PowerShell maintains a clear separation between the actual data and its representation, the caveat is that you do end up with just the for-display, string representation in the following scenarios:
When you use Out-File or its effective aliases, the redirection operators > and >>
When you (implicitly) send output to the outside world (an outside caller's stdout stream - see below).
In both cases, if later programmatic processing is required, the data must be transformed into a structured text format, such as CSV (Export-Csv or ConvertTo-Csv) or JSON (using ConvertTo-Json).
As for how the outside world sees PowerShell's output streams:
IPC (inter-process communication) at the OS level knows only two output streams: stdout (standard output) and stderr (standard error), which forces PowerShell to map its 6 output streams onto these two in order to return streaming output to an outside caller.
While it would make sense to map PowerShell's success output stream to stdout and all other streams to stderr, unfortunately all streams are reported via stdout by default as of PowerShell 7.2 - although selectively redirecting stderr in the calling process (typically with 2>) does send PowerShell's error stream (only) to that redirection target. See the bottom section of this answer for more information.
Also note that PowerShell as of version 7.2 only ever communicates via text (strings) with outside callers as well as with external programs called from inside a PowerShell sessions, which means that character-encoding issues can arise - see this answer for more information.
[1] Note that PowerShell has no concept of an input stream as such, and therefore also does not support the stdin redirection operator < familiar from other shells. Instead, commands receive streaming input (only) via the pipeline. In order to receive data from the outside world, via the PowerShell CLI's stdin stream, the automatic $input variable must be used - see this answer.
[2] Using > (or >>) to redirect to a file effectively uses the Out-File cmdlet behind the scenes, and therefore its default character encoding, which is "Unicode" (UTF-16LE) in Windows PowerShell, and BOM-less UTF-8 in PowerShell (Core) 7+. However, in PowerShell version 5.1 and above you can control this encoding via the $PSDefaultParameterValues preference variable - see this answer.
I have 5 different variable coming from different if and loop statements, when I use "put" to take output into text file all characters and digits are altogether like this : alphaclass112098voip. where
variables: name = alpha
category = class1
number = 12098
service = voip
I want output in file as like this with spaces on same line.
Alpha class1 12098 voip
Beta class1 12093 DHCP SIP
Also at certain point I want to through delimiters for future purposes.
The easiest way to deal with this is to construct a Tcl list that represents the record. You can do this piecemeal with lappend, or all at once with list. Or mix them.
foreach name $names category $categories number $numbers service $services {
set record [list $name $category]
lappend record $number
lappend record $service
puts $record
}
This shows the record for each line in a format that Tcl finds easy to parse (you'll see what I mean if you have a name with a space in it). To use a delimiter to separate the values instead, the join command is very useful:
puts [join $record $delimiter]
The default delimiter is space, but try a : instead to see how it works.
If you're generating a CSV file, do use the csv package in Tcllib. It handles the tricky edge-cases (e.g., embedded commas) for you.
I'm relatively new to Perl and trying to self teach. However i have read all of the related threads on this page and others and none of them seem to work for me.
Below is my code - trying to get a lot of data from a webpage in Perl format and export it to update values in an SQL table.
Currently i can't even data dumper the results of the url out.
Any help would be great.
#!/usr/bin/perl
#
use LWP::Simple;
use warnings;
use strict;
use JSON qw( decode_json from_json );
use LWP::Simple;
use Data::Dumper;
use utf8;
my $url = "http://.sensitivedata.txt";
my #json= from_json(get ( $url ));
die "Couldn't get $url" if not defined #json;
##my $decoded_json = decode_json( #json);
print Dumper #json;
exit 0;
This is the error message it is giving me:
defined(#array) is deprecated at alarms.pl line 14.
(Maybe you should just omit the defined()?)
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /opt/csw/share/perl/csw/JSON.pm line 168
The error message is pretty clear about a) what the problem is and b) how to get rid of it.
defined(#array) is deprecated at alarms.pl line 14. (Maybe you should
just omit the defined()?)
Calling defined() on #json is pointless. You're really just checking to see if there is any data in the array so replace if not defined #json with if not #json.
That will get rid of the error message. But you'll still have a problem as your program will almost certainly now die on the same line with the error message "Couldn't get http://.sensitivedata.txt". And that's probably not an accurate error message.
The problem is that this error can be caused by two problems. Either you can't get the data or you can't parse the data. Your error message only mentions one of these possibilities. Better to split the error checking into two.
# Step 1: Get the data
my $raw_json = get($url);
die "Can't get data from $url" unless $raw_json;
# Step 2: Parse the data
my #json = from_json($raw_json);
if (!#json) {
warn $raw_json;
die "Can't parse data from $url";
}
With code more like this, you'll be able to see what the problem is.
There's another little problem here, so to pre-empt your next question...
from_json always returns a scalar. It will either be a hash reference or an array reference (depending on the JSON you get). Looks like you're expecting an array. You'll need to store the reference in a scalar and dereference it.
my $json_array_ref = from_json($raw_json);
if (!#$json_array_ref) {
warn $raw_json;
die "Can't parse data from $url";
}
my #json = #$json_array_ref;
How can I get all meaning of a word in json response .Preferably without key/signing up/api key .If I type test then all meaning of that word.
I tried :
but its not working.
<?php
$query = 'oracle';
$file = file_get_contents('http://www.google.com/dictionary/json?callback=a&q='.$query.'&sl=en&tl=en&restrict=pr,de&client=te');
// var_dump($file);
$file = substr($file, 2, -10);
$file = preg_replace("/\\\x[0-9a-f]{2}/", "", $file);
echo $file;
$json = json_decode($file);
var_dump($json);
?>
Even this is returning null.
I have tried Only the php above.I would like to knowif I can make rest call without api key just words which match the query word .Is there any rest call you have in mind.I really appreciate any help .Thanks in Advance.
Assuming server side
Get a copy of a dictionary in a computer friendly format i.e. http://www.ibiblio.org/webster/ (XML).
Store said dictionary in a database or in memory and perform a lookup.
Would then be trivial to provide a restful service returning all definitions for a particular word.
Also see: Google's "define: " through an API?