The code below doesn't do what I need it to. I want to pass $STR to Underscore and pluck out the "name" attribute from the JSON data.
#!/bin/bash
STR='['$#']';
RESULT=`underscore pluck --data '+$STR+' name`;
echo $RESULT;
JSON Data:
{"maxResults":1,"resultList":[{"#class":"com.sohnar.trafficlite.transfer.crm.refactor.ClientCRMEntryTO","id":331458,"version":2,"dateCreated":"2017-05-31T13:20:22.960+0000","dateModified":"2017-06-05T14:23:59.961+0000","lastUpdatedUserId":71954,"name":"ACME_CLIENT","website":null,"description":null,"billingLocation":null,"primaryLocation":null,"crmEntryType":"CLIENT","industryType":null,"accountManagerId":103049,"crmClientClassificationListItemId":{"id":12405},"companyProfile":{"id":486024,"version":1,"dateCreated":"2017-05-31T13:20:22.960+0000","dateModified":"2017-06-05T14:23:59.962+0000","sourceOfBusinessListItemId":null,"creditTermsListItemId":{"id":4215},"relationshipSince":"2017-05-30T23:00:00.000+0000","turnover":0,"employees":0,"taxNumber":null,"companyNumber":null,"nominalCode":null,"accountPackageId":null,"optOutMarketing":false,"optOutEmail":false,"optOutTelephone":false,"notes":null},"colorCode":0,"externalCode":"SAP-01","clientState":"CLIENT","defaultCustomRateSetId":null,"preferredCurrencyId":{"id":48},"freeTags":[]}],"windowSize":5,"currentPage":1}
Several problems here. First you should use double quotes, not single. Double quotes allows the expansion of variable. Second, I am assuming that you don't want the + as part of the data - they are not a string concatenation operator in bash (there is no need for one).
str="[$#]"
result=$(underscore pluck --data "$str" name)
echo $result
I used the $( ) notation rather than back-ticks ( `` ). Back-ticks are considered deprecated and difficult to read.
I have replaced uppercase variable names with lower case. This is because there are many uppercase variable names used by the shell so using uppercase risks a name collision. Best to use lower or mixed-case variable names.
I have also removed the superfluous semi-colons ;. They are not doing any harm, but they are of no use either. In bash a semi-colon is a statement separator, whereas in languages like C and Java they are a statement terminator.
Edit: double-quoted $#
I figured it out...
#!/bin/bash
name=$(underscore select .name --outfmt text < get.crm.client.acme.json)
echo $name
Related
Should or should I not wrap quotes around variables in a shell script?
For example, is the following correct:
xdg-open $URL
[ $? -eq 2 ]
or
xdg-open "$URL"
[ "$?" -eq "2" ]
And if so, why?
General rule: quote it if it can either be empty or contain spaces (or any whitespace really) or special characters (wildcards). Not quoting strings with spaces often leads to the shell breaking apart a single argument into many.
$? doesn't need quotes since it's a numeric value. Whether $URL needs it depends on what you allow in there and whether you still want an argument if it's empty.
I tend to always quote strings just out of habit since it's safer that way.
In short, quote everything where you do not require the shell to perform word splitting and wildcard expansion.
Single quotes protect the text between them verbatim. It is the proper tool when you need to ensure that the shell does not touch the string at all. Typically, it is the quoting mechanism of choice when you do not require variable interpolation.
$ echo 'Nothing \t in here $will change'
Nothing \t in here $will change
$ grep -F '#&$*!!' file /dev/null
file:I can't get this #&$*!! quoting right.
Double quotes are suitable when variable interpolation is required. With suitable adaptations, it is also a good workaround when you need single quotes in the string. (There is no straightforward way to escape a single quote between single quotes, because there is no escape mechanism inside single quotes -- if there was, they would not quote completely verbatim.)
$ echo "There is no place like '$HOME'"
There is no place like '/home/me'
No quotes are suitable when you specifically require the shell to perform word splitting and/or wildcard expansion.
Word splitting (aka token splitting);
$ words="foo bar baz"
$ for word in $words; do
> echo "$word"
> done
foo
bar
baz
By contrast:
$ for word in "$words"; do echo "$word"; done
foo bar baz
(The loop only runs once, over the single, quoted string.)
$ for word in '$words'; do echo "$word"; done
$words
(The loop only runs once, over the literal single-quoted string.)
Wildcard expansion:
$ pattern='file*.txt'
$ ls $pattern
file1.txt file_other.txt
By contrast:
$ ls "$pattern"
ls: cannot access file*.txt: No such file or directory
(There is no file named literally file*.txt.)
$ ls '$pattern'
ls: cannot access $pattern: No such file or directory
(There is no file named $pattern, either!)
In more concrete terms, anything containing a filename should usually be quoted (because filenames can contain whitespace and other shell metacharacters). Anything containing a URL should usually be quoted (because many URLs contain shell metacharacters like ? and &). Anything containing a regex should usually be quoted (ditto ditto). Anything containing significant whitespace other than single spaces between non-whitespace characters needs to be quoted (because otherwise, the shell will munge the whitespace into, effectively, single spaces, and trim any leading or trailing whitespace).
When you know that a variable can only contain a value which contains no shell metacharacters, quoting is optional. Thus, an unquoted $? is basically fine, because this variable can only ever contain a single number. However, "$?" is also correct, and recommended for general consistency and correctness (though this is my personal recommendation, not a widely recognized policy).
Values which are not variables basically follow the same rules, though you could then also escape any metacharacters instead of quoting them. For a common example, a URL with a & in it will be parsed by the shell as a background command unless the metacharacter is escaped or quoted:
$ wget http://example.com/q&uack
[1] wget http://example.com/q
-bash: uack: command not found
(Of course, this also happens if the URL is in an unquoted variable.) For a static string, single quotes make the most sense, although any form of quoting or escaping works here.
wget 'http://example.com/q&uack' # Single quotes preferred for a static string
wget "http://example.com/q&uack" # Double quotes work here, too (no $ or ` in the value)
wget http://example.com/q\&uack # Backslash escape
wget http://example.com/q'&'uack # Only the metacharacter really needs quoting
The last example also suggests another useful concept, which I like to call "seesaw quoting". If you need to mix single and double quotes, you can use them adjacent to each other. For example, the following quoted strings
'$HOME '
"isn't"
' where `<3'
"' is."
can be pasted together back to back, forming a single long string after tokenization and quote removal.
$ echo '$HOME '"isn't"' where `<3'"' is."
$HOME isn't where `<3' is.
This isn't awfully legible, but it's a common technique and thus good to know.
As an aside, scripts should usually not use ls for anything. To expand a wildcard, just ... use it.
$ printf '%s\n' $pattern # not ``ls -1 $pattern''
file1.txt
file_other.txt
$ for file in $pattern; do # definitely, definitely not ``for file in $(ls $pattern)''
> printf 'Found file: %s\n' "$file"
> done
Found file: file1.txt
Found file: file_other.txt
(The loop is completely superfluous in the latter example; printf specifically works fine with multiple arguments. stat too. But looping over a wildcard match is a common problem, and frequently done incorrectly.)
A variable containing a list of tokens to loop over or a wildcard to expand is less frequently seen, so we sometimes abbreviate to "quote everything unless you know precisely what you are doing".
Here is a three-point formula for quotes in general:
Double quotes
In contexts where we want to suppress word splitting and globbing. Also in contexts where we want the literal to be treated as a string, not a regex.
Single quotes
In string literals where we want to suppress interpolation and special treatment of backslashes. In other words, situations where using double quotes would be inappropriate.
No quotes
In contexts where we are absolutely sure that there are no word splitting or globbing issues or we do want word splitting and globbing.
Examples
Double quotes
literal strings with whitespace ("StackOverflow rocks!", "Steve's Apple")
variable expansions ("$var", "${arr[#]}")
command substitutions ("$(ls)", "`ls`")
globs where directory path or file name part includes spaces ("/my dir/"*)
to protect single quotes ("single'quote'delimited'string")
Bash parameter expansion ("${filename##*/}")
Single quotes
command names and arguments that have whitespace in them
literal strings that need interpolation to be suppressed ( 'Really costs $$!', 'just a backslash followed by a t: \t')
to protect double quotes ('The "crux"')
regex literals that need interpolation to be suppressed
use shell quoting for literals involving special characters ($'\n\t')
use shell quoting where we need to protect several single and double quotes ($'{"table": "users", "where": "first_name"=\'Steve\'}')
No quotes
around standard numeric variables ($$, $?, $# etc.)
in arithmetic contexts like ((count++)), "${arr[idx]}", "${string:start:length}"
inside [[ ]] expression which is free from word splitting and globbing issues (this is a matter of style and opinions can vary widely)
where we want word splitting (for word in $words)
where we want globbing (for txtfile in *.txt; do ...)
where we want ~ to be interpreted as $HOME (~/"some dir" but not "~/some dir")
See also:
Difference between single and double quotes in Bash
What are the special dollar sign shell variables?
Quotes and escaping - Bash Hackers' Wiki
When is double quoting necessary?
I generally use quoted like "$var" for safe, unless I am sure that $var does not contain space.
I do use $var as a simple way to join lines:
lines="`cat multi-lines-text-file.txt`"
echo "$lines" ## multiple lines
echo $lines ## all spaces (including newlines) are zapped
Whenever the https://www.shellcheck.net/ plugin for your editor tells you to.
In Bash, what are the differences between single quotes ('') and double quotes ("")?
Single quotes won't interpolate anything, but double quotes will. For example: variables, backticks, certain \ escapes, etc.
Example:
$ echo "$(echo "upg")"
upg
$ echo '$(echo "upg")'
$(echo "upg")
The Bash manual has this to say:
3.1.2.2 Single Quotes
Enclosing characters in single quotes (') preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.
3.1.2.3 Double Quotes
Enclosing characters in double quotes (") preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !. The characters $ and ` retain their special meaning within double quotes (see Shell Expansions). The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or newline. Within double quotes, backslashes that are followed by one of these characters are removed. Backslashes preceding characters without a special meaning are left unmodified. A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.
The special parameters * and # have special meaning when in double quotes (see Shell Parameter Expansion).
The accepted answer is great. I am making a table that helps in quick comprehension of the topic. The explanation involves a simple variable a as well as an indexed array arr.
If we set
a=apple # a simple variable
arr=(apple) # an indexed array with a single element
and then echo the expression in the second column, we would get the result / behavior shown in the third column. The fourth column explains the behavior.
#
Expression
Result
Comments
1
"$a"
apple
variables are expanded inside ""
2
'$a'
$a
variables are not expanded inside ''
3
"'$a'"
'apple'
'' has no special meaning inside ""
4
'"$a"'
"$a"
"" is treated literally inside ''
5
'\''
invalid
can not escape a ' within ''; use "'" or $'\'' (ANSI-C quoting)
6
"red$arocks"
red
$arocks does not expand $a; use ${a}rocks to preserve $a
7
"redapple$"
redapple$
$ followed by no variable name evaluates to $
8
'\"'
\"
\ has no special meaning inside ''
9
"\'"
\'
\' is interpreted inside "" but has no significance for '
10
"\""
"
\" is interpreted inside ""
11
"*"
*
glob does not work inside "" or ''
12
"\t\n"
\t\n
\t and \n have no special meaning inside "" or ''; use ANSI-C quoting
13
"`echo hi`"
hi
`` and $() are evaluated inside "" (backquotes are retained in actual output)
14
'`echo hi`'
`echo hi`
`` and $() are not evaluated inside '' (backquotes are retained in actual output)
15
'${arr[0]}'
${arr[0]}
array access not possible inside ''
16
"${arr[0]}"
apple
array access works inside ""
17
$'$a\''
$a'
single quotes can be escaped inside ANSI-C quoting
18
"$'\t'"
$'\t'
ANSI-C quoting is not interpreted inside ""
19
'!cmd'
!cmd
history expansion character '!' is ignored inside ''
20
"!cmd"
cmd args
expands to the most recent command matching "cmd"
21
$'!cmd'
!cmd
history expansion character '!' is ignored inside ANSI-C quotes
See also:
ANSI-C quoting with $'' - GNU Bash Manual
Locale translation with $"" - GNU Bash Manual
A three-point formula for quotes
If you're referring to what happens when you echo something, the single quotes will literally echo what you have between them, while the double quotes will evaluate variables between them and output the value of the variable.
For example, this
#!/bin/sh
MYVAR=sometext
echo "double quotes gives you $MYVAR"
echo 'single quotes gives you $MYVAR'
will give this:
double quotes gives you sometext
single quotes gives you $MYVAR
Others explained it very well, and I just want to give something with simple examples.
Single quotes can be used around text to prevent the shell from interpreting any special characters. Dollar signs, spaces, ampersands, asterisks and other special characters are all ignored when enclosed within single quotes.
echo 'All sorts of things are ignored in single quotes, like $ & * ; |.'
It will give this:
All sorts of things are ignored in single quotes, like $ & * ; |.
The only thing that cannot be put within single quotes is a single quote.
Double quotes act similarly to single quotes, except double quotes still allow the shell to interpret dollar signs, back quotes and backslashes. It is already known that backslashes prevent a single special character from being interpreted. This can be useful within double quotes if a dollar sign needs to be used as text instead of for a variable. It also allows double quotes to be escaped so they are not interpreted as the end of a quoted string.
echo "Here's how we can use single ' and double \" quotes within double quotes"
It will give this:
Here's how we can use single ' and double " quotes within double quotes
It may also be noticed that the apostrophe, which would otherwise be interpreted as the beginning of a quoted string, is ignored within double quotes. Variables, however, are interpreted and substituted with their values within double quotes.
echo "The current Oracle SID is $ORACLE_SID"
It will give this:
The current Oracle SID is test
Back quotes are wholly unlike single or double quotes. Instead of being used to prevent the interpretation of special characters, back quotes actually force the execution of the commands they enclose. After the enclosed commands are executed, their output is substituted in place of the back quotes in the original line. This will be clearer with an example.
today=`date '+%A, %B %d, %Y'`
echo $today
It will give this:
Monday, September 28, 2015
Since this is the de facto answer when dealing with quotes in Bash, I'll add upon one more point missed in the answers above, when dealing with the arithmetic operators in the shell.
The Bash shell supports two ways to do arithmetic operation, one defined by the built-in let command and the other the $((..)) operator. The former evaluates an arithmetic expression while the latter is more of a compound statement.
It is important to understand that the arithmetic expression used with let undergoes word-splitting, pathname expansion just like any other shell commands. So proper quoting and escaping need to be done.
See this example when using let:
let 'foo = 2 + 1'
echo $foo
3
Using single quotes here is absolutely fine here, as there isn't any need for variable expansions here. Consider a case of
bar=1
let 'foo = $bar + 1'
It would fail miserably, as the $bar under single quotes would not expand and needs to be double-quoted as
let 'foo = '"$bar"' + 1'
This should be one of the reasons, the $((..)) should always be considered over using let. Because inside it, the contents aren't subject to word-splitting. The previous example using let can be simply written as
(( bar=1, foo = bar + 1 ))
Always remember to use $((..)) without single quotes
Though the $((..)) can be used with double quotes, there isn't any purpose to it as the result of it cannot contain content that would need the double quote. Just ensure it is not single quoted.
printf '%d\n' '$((1+1))'
-bash: printf: $((1+1)): invalid number
printf '%d\n' $((1+1))
2
printf '%d\n' "$((1+1))"
2
Maybe in some special cases of using the $((..)) operator inside a single quoted string, you need to interpolate quotes in a way that the operator either is left unquoted or under double quotes. E.g., consider a case, when you are tying to use the operator inside a curl statement to pass a counter every time a request is made, do
curl http://myurl.com --data-binary '{"requestCounter":'"$((reqcnt++))"'}'
Notice the use of nested double quotes inside, without which the literal string $((reqcnt++)) is passed to the requestCounter field.
There is a clear distinction between the usage of ' ' and " ".
When ' ' is used around anything, there is no "transformation or translation" done. It is printed as it is.
With " ", whatever it surrounds, is "translated or transformed" into its value.
By translation/ transformation I mean the following:
Anything within the single quotes will not be "translated" to their values. They will be taken as they are inside quotes. Example: a=23, then echo '$a' will produce $a on standard output. Whereas echo "$a" will produce 23 on standard output.
A minimal answer is needed for people to get going without spending a lot of time as I had to.
The following is, surprisingly (to those looking for an answer), a complete command:
$ echo '\'
whose output is:
\
Backslashes, surprisingly to even long-time users of bash, do not have any meaning inside single quotes. Nor does anything else.
I have this which works to declare a JSON string in a bash script:
local my_var="foobar"
local json=`cat <<EOF
{"quicklock":"${my_var}"}
EOF`
The above heredoc works, but I can't seem to format it any other way, it literally has to look exactly like that lol.
Is there any way to get the command to be on one line, something like this:
local json=`cat <<EOF{"quicklock":"${my_var}"}EOF`
that would be so much nicer, but doesn't seem to take, obviously simply because that's not how EOF works I guess lol.
I am looking for a shorthand way to declare JSON in a file that:
Does not require a ton of escape chars.
That allows for dynamic interpolation of variables.
Note: The actual JSON I want to use has multiple dynamic variables with many key/value pairs. Please extrapolate.
I'm not a JSON guy, don't really understand the "well-formed" arguments in the discussion above, but, you can use a 'here-string' rather than a 'here-document', like this:
my_var="foobar"
json=`cat <<<{\"quicklock\":\"${my_var}\"}`
why not use jq? It's pretty good at managing string interpolation and it lints your structure.
$ echo '{}' >> foo.json
$ declare myvar="assigned-var"
$ jq --arg ql "$myvar" '.quicklock=$ql' foo.json
the text that comes out on the other end of that call to jq can then be cat into a file or whatever you wanna do. text would look something like this:
{"quicklock": "assigned-var"}
You can do this with printf:
local json="$(printf '{"quicklock":"%s"}' "$my_var")"
(Never mind that SO's syntax highlighting looks odd here. Posix shell command substitution allows nesting one level of quotes.)
A note (thanks to Charles Duffy's comment on the question): I'm assuming $my_var is not controlled by user input. If it is, you'll need to be careful to ensure it is legal for a JSON string. I highly recommend barring non-ASCII characters, double quotes, and backslashes. If you have jq available, you can use it as Charles noted in the comments to ensure you have well-formed output.
You can define your own helper function to address the situation with missing bash syntax:
function begin() { eval echo $(sed "${BASH_LINENO[0]}"'!d;s/.*begin \(.*\) end.*/\1/;s/"/\\\"/g' "${BASH_SOURCE[0]}"); }
Then you can use it as follows.
my_var="foobar"
json=$(begin { "quicklock" : "${my_var}" } end)
echo "$json"
This fragment displays the desired output:
{ "quicklock" : "foobar" }
This is just a proof of concept. You can define your syntax in any way you want (such as end of the input by the custom EOF string, correctly escape invalid characters). For example, since Bash allows function identifiers using characters other than alphanumeric characters, it is possible to define such a syntax:
json=$(/ { "quicklock" : "${my_var}" } /)
Moreover, if you relax the first criterion (escape characters), ordinary assignment will nicely solve this problem:
json="{ \"quicklock\" : \"${my_var}\" }"
How about just using the shell's natural concatenation of strings? If you concatenate ${mybar} rather than interpolate it, you can avoid escapes and get everything on one line:
my_var1="foobar"
my_var2="quux"
json='{"quicklock":"'${my_var1}'","slowlock":"'$my_var2'"}'
That said, this is a pretty crude scheme, and as others have pointed out you'll have problems if the variables, say, contain quote characters.
Since no escape chars is strong requirement here is a here-doc based solution:
#!/bin/bash
my_var='foobar'
read -r -d '' json << EOF
{
"quicklock": "$my_var"
}
EOF
echo "$json"
It will give you the same output as the first solution I mentioned.
Just be careful, if you would put first EOF inside double quotes:
read -r -d '' json << "EOF"
$my_var would not be considered as a variable but as a plain text, so you would get this output:
{
"quicklock": "$my_var"
}
Find/replace space with %20
I must replace all spaces in *.html files which are inside href="something something .pdf" with %20.
I found a regular expression for that task:
find : href\s*=\s*['"][^'" ]*\K\h|(?!^)\G[^'" ]*\K\h
replace : %20
That regular expression works in text editors like Notepad++ or Geany.
I want use that regular expression from the Linux command line with sed or perl.
Solution (1):
cat test002.html | perl -ne 's/href\s*=\s*['\''"][^'\''" ]*\K\h|(?!^)\G[^'\''" ]*\K\h/%20/g; print;' > Work_OK01.html
Solution (2):
cat test002.html | perl -ne 's/href\s*=\s*[\x27"][^\x27" ]*\K\h|(?!^)\G[^\x27" ]*\K\h/%20/g; print;' > Work_OK02.html
The problem is that you don't properly escape the single quotes in your program.
If your program is
...[^'"]...
The shell literal might be
'...[^'\''"]...'
'...[^'"'"'"]...'
'...[^\x27"]...' # Avoids using a single quote to avoid escaping it.
So, you were going for
perl -ne 's/href\s*=\s*['\''"][^'\''" ]*\K\h|(?!^)\G[^'\''" ]*\K\h/%20/g; print;'
Don't try do everything at once. Here are some far cleaner (i.e. far more readable) solutions:
perl -pe's{href\s*=\s*['\''"]\K([^'\''"]*)}{ $1 =~ s/ /%20/rg }eg' # 5.14+
perl -pe's{href\s*=\s*['\''"]\K([^'\''"]*)}{ (my $s = $1) =~ s/ /%20/g; $s }eg'
Note that -p is the same as -n, except that it cause a print to be performed for each line.
The above solutions make a large number of assumptions about the files that might be encountered[1]. All of these assumptions would go away if you use a proper parser.
If you have HTML files:
perl -MXML::LibXML -e'
my $doc = XML::LibXML->new->parse_file($ARGV[0]);
$_->setValue( $_->getValue() =~ s/ /%20/gr )
for $doc->findnodes(q{//#href});
binmode(STDOUT);
print($doc->toStringHTML());
' in_file.html >out_file.html
If you have XML (incl XHTML) files:
perl -MXML::LibXML -e'
my $doc = XML::LibXML->new->parse_file($ARGV[0]);
$_->setValue( $_->getValue() =~ s/ /%20/gr )
for $doc->findnodes(q{//#href});
binmode(STDOUT);
$doc->toFH(\*STDOUT);
' in_file.html >out_file.html
Assumptions made by the substitution-based solutions:
File uses an ASCII-based encoding (e.g. UTF-8, iso-latin-1, but not UTF-16le).
No newline between href and =.
No newline between = and the value.
No newline in the value of href attributes.
Nothing matching /href\s*=/ in text (incl CDATA sections).
Nothing matching /href\s*=/ in comments.
No other attributes have a name ending in href.
No single quote (') in href="...".
No double quote (") in href='...'.
No href= with an unquoted value.
Space in href attributes aren't encoded using a character entity (e.g ).
Maybe more?
(SLePort makes similar assumptions even though they didn't document them. They also assume href attributes don't contain >.)
An xml parser would be more suited for that job(eg. XMLStarlet, xmllint,...), but if you don't have newlines in your a tags, the below sed should work.
Using the t command and backreferences, it loops over and replace all spaces up to last " inside the a tags:
$ sed ':a;s/\(<a [^>]*href=[^"]*"[^ ]*\) \([^"]*">\)/\1%20\2/;ta' <<< '<a href="http://url with spaces">'
<a href="http://url%20with%20spaces">
You seem to have neglected to escape the quotes inside the string you pass to Perl. So Bash sees you giving perl the following arguments:
s/href\s*=\s*[][^', which results from the concatenation of the single-quoted string 's/href\s*=\s*[' and the double-quoted string "][^'"
]*Kh, unquoted, because \K and \h are not special characters in the shell so it just treats them as K and h respectively
Then Bash sees a pipe character |, followed by a subshell invocation (?!^), in which !^ gets substituted with the first argument of the last command invoked. (See "History Expansion > Word Designators" in the Bash man page.) For example, if your last command was echo myface then (?!^) would look for the command named ?myface and runs it in a subshell.
And finally, Bash gets to the sequence \G[^'" ]*\K\h/%20/g; print;', which is interpreted as the concatenation of G (from \G), [^, and the single-quoted string " ]*\K\h/%20/g; print;. Bash has no idea what to do with G[^" ]*\K\h/%20/g; print;, since it just finished parsing a subshell invocation and expects to see a semicolon, line break, or logical operator (or so on) before getting another arbitrary string.
Solution: properly quote the expression you give to perl. You'll need to use a combination of single and double quotes to pull it off, e.g.
perl -ne 's/href\s*=\s*['"'\"][^'\" ]*"'\K\h|(?!^)\G[^'"'\" ]*"'\K\h/%20/g; print;'
I'm trying to pull variables from an API in json format and then put them back together with one variable changed and fire them back as a put.
Only issue is that every value has quote marks in it and must go back to the API separated by commas only.
example of what it should see with redacted information, variables inside the **'s:
curl -skv -u redacted:redacted -H Content-Type: application/json -X PUT -d'{properties:{basic:{request_rules:[**"/(req) testrule","/test-body","/(req) test - Admin","test-Caching"**]}}}' https://x.x.x.x:9070/api/tm/1.0/config/active/vservers/xxx-xx
Obviously if I fire them as a plain array I get spaces instead of commas. However I tried outputting it as a plain string
longstr=$(echo ${valuez[#]})
output=$(echo $longstr |sed -e 's/" /",/g')
And due to the way bash is interpreted it seems to either interpret the quotes wrong or something else. I guess it might well be the single ticks encapsulating after the PUT -d as well but I'm not sure how I can throw a variable into something that has single ticks.
If I put the raw data in manually it works so it's either the way the variable is being sent or the single ticks. I don't get an error and when I echo the line out it looks perfect.
Any ideas?
valuez=( "/(req) testrule" "/test-body" "/(req) test - Admin" "test-Caching" )
# Temporarily set IFS to some character which is known not to appear in the array.
oifs=$IFS
IFS=$'\014'
# Flatten the array with the * expansion giving a string containing the array's elements separated by the first character of $IFS.
d_arg="${valuez[*]}"
IFS=$oifs
# If necessary, quote or escape embedded quotation marks. (Implementation-specific, using doubled double quotes as an example.)
d_arg="${d_arg//\"/\"\"}"
# Substitute the known-to-be-absent character for the desired quote+separator+quote.
d_arg="${d_arg//$'\014'/\",\"}"
# Prepend and append quotes.
d_arg="\"$d_arg\""
# insert the prepared arg into the final string.
d_arg="{properties:{basic:{request_rules:[${d_arg}]}}}"
curl ... -d"$d_arg" ...
if you have gnu awk with version 4 and above, which support FPAT
output=$(echo $longstr |awk '$1=$1' FPAT="(\"[^\"]+\")" OFS=",")
Explanation
FPAT #
This is a regular expression (as a string) that tells gawk to create the fields based on text that matches the regular expression. Assigning a value to FPAT overrides the use of FS and FIELDWIDTHS for field splitting. See Splitting By Content, for more information.
If gawk is in compatibility mode (see Options), then FPAT has no special meaning, and field-splitting operations occur based exclusively on the value of FS.
valuez=( "/(req) testrule" "/test-body" "/(req) test - Admin" "test-Caching" )
csv="" sep=""
for v in "${valuez[#]}"; do csv+="$sep\"$v\""; sep=,; done
echo "$csv"
"/(req) testrule","/test-body","/(req) test - Admin","test-Caching"
If it's something you need to do repeatedly, but it into a function:
toCSV () {
local csv sep val
for val; do
csv+="$sep\"$val\""
sep=,
done
echo "$csv"
}
csv=$(toCSV "${valuez[#]}")