I'm using the following :
#!/bin/sh
export IFS=","
cat myfile.csv | while read a b c d e f g h i j
do
#example do
echo "a=$a"
echo "b=$b:"
echo "c=$c:"
echo "d=$d:"
echo "e=$e:"
echo "f=$f:"
echo "g=$g:"
echo "h=$h:"
echo "i=$i:"
echo "j=$j:"
done
to read in variables from a CSV file. It works quite well, only that it seems to miss off the first line of the file, why is this? can I change it to capture line one also?
Related
Recently github has announced change that echo "::set-output name=x::y" command is deprecated and should be replaced by echo "x=y" >> $GITHUB_OUTPUT
The previous command was able to process multilined value of b while the new approach fails with the folllowing errors
Error: Unable to process file command 'output' successfully.
Error: Invalid format
In my script, I populate a variable message with a message text that should be sent to slack. I need output variables to pass that text to the next job step which performs the send operation.
message="Coverage: $(cat coverage.txt). Covered: $(cat covered.txt). Uncovered: $(cat uncovered.txt). Coverage required: $(cat coverageRequires.csv)"
The last part of message includes context of a csv file which has multiple lines
While the set-output command was able to process such multilined parameters
echo "::set-output name=text::$message"
the new version fails
echo "text=$message" >> $GITHUB_OUTPUT
What can be done to fix or avoid this error?
The documentation describes syntax for multiline strings in a different section but it seems to work even for output parameters.
Syntax:
{name}<<{delimiter}
{value}
{delimiter}
This could be interpreted as:
Set output with the defined name, and a delimiter (typically EOF) that would mark the end of data.
Keep reading each line and concatenating it into one input.
Once reaching the line consisting of the defined delimiter, stop processing. This means that another output could start being added.
Therefore, in your case the following should work and step's text output would consist of a multiline string that $message contains:
echo "text<<EOF" >> $GITHUB_OUTPUT
echo "$message" >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
...and unless you need $message for something else, you could actually avoid setting it and get a more readable set of instructions to construct the output:
echo "text<<EOF" >> $GITHUB_OUTPUT
echo "Coverage: $(cat coverage.txt)." >> $GITHUB_OUTPUT
echo "Covered: $(cat covered.txt)." >> $GITHUB_OUTPUT
echo "Uncovered: $(cat uncovered.txt)." >> $GITHUB_OUTPUT
echo "Coverage required: $(cat coverageRequires.csv)" >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
Note: The last example is not 100% same as yours because it would contain new lines between the sections. You could use echo -n to avoid that.
I ended up having replaced all breaklines in the message variables by the command
message=$(echo $message | tr '\n' ' ')
echo "text=$message" >> $GITHUB_OUTPUT
This eliminated the error.
The previous command was able to process multilined value of b while the new approach fails with the folllowing errors
Actually it has not been, but lately they changed the behaviour:
https://github.com/orgs/community/discussions/26288
What can be done to fix or avoid this error?
The same way as was for the GITHUB_ENV variable:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
echo 'var<<EOF' >> $GITHUB_OUTPUT
echo "<multi-line-output>" >> $GITHUB_OUTPUT
echo 'EOF' >> $GITHUB_OUTPUT
Or more fancy way:
https://github.com/orgs/community/discussions/26288#discussioncomment-3876281
delimiter="$(openssl rand -hex 8)"
echo "output-name<<${delimiter}" >> "${GITHUB_OUTPUT}"
echo "Some\nMultiline\nOutput" >> "${GITHUB_OUTPUT}"
echo "${delimiter}" >> "${GITHUB_OUTPUT}"
Another option to set multilines in outputs could be using this implementation (same as for ENV variables in the $GITHUB_ENV):
- name: Setup output var
id: test1
run: |
MESSAGE=$(cat << EOF
first line
second line
third line
...
EOF
)
echo TEST=$MESSAGE >> $GITHUB_OUTPUT
- name: Check output var
run: |
echo ${{steps.test1.outputs.TEST}}
I made a test here with the same behavior that for environment variables (detailed in this other thread)
EDIT 1:
This syntax also works (and looks easier to use):
run: |
echo "TEST1=first line \
second line \
third line" >> $GITHUB_OUTPUT
EDIT 2:
It's also possible to display the output as multilines (and not on a single line as the other examples above). However, the syntax would be different and you would need yo use echo -e together with \n inside the variable.
Example:
- name: Setup output var
id: test
run: echo "TEST=first line\n second line\n third line" >> $GITHUB_OUTPUT
- name: Check output var
run: |
echo ${{steps.test.outputs.TEST}} #Will keep the n from the \n
echo -e "${{steps.test.outputs.TEST}}" #Will break the line from the \n
steps:
- run: |
some_response=$(curl -i -H "Content-Type: application/json" \
-d "${body}" -X POST "${url}")
echo response_output=$some_response >> $GITHUB_OUTPUT
id: some-request
- run: |
echo "Response is: ${{ steps.some-request.outputs.response_output }}"
Worked for me well. Quotes (and curly brackets) are not needed in case of just setting output var
I have written a script myscript which does an action on particular type of files. However there is an option for displaying the filename with -d. If I've not mentioned
$ myscript -d
also it is saying, invalid file path (because that file should be file_evaluated AFTER the action of my script on it) but -d is triggered even if it's not mentioned. How do I solve this? Do help. Thanks.
myscript.sh:
#!/bin/zsh
echo "Enter source directory: "
read directory
echo
echo "Add a file (with path): "
read file
echo
while getopts ":rd" cal; do
case "$cal" in
r) echo "Renaming..."
echo
;;
d) echo "Displaying the Renamed files.."
echo
;;
*) echo "Invalid option: $OPTARG"
;;
esac
done
shift $((OPTIND -1))
echo
find . "*.txt" -print0 | while read -d $'\0' file
do
anothescript -r "$file" -d
done
echo "done..."
I have a variable that am declaring like below in my shell script:
Variable = awk 'Some code' filename.txt << this is assigning one word from the text file to this variable
And then am trying to display this in an HTML heading like below, but its not working.
<h2><font color="navy"> Network-element : ${variable} </font></h2>
any help is appreciated.
I think there could be 2 points here.
1- Cover the variable value like:
Variable=$(awk 'Some code' filename.txt)
2- Then in shell script html code doesn't work like simple print you have to use echo for it (NOTE this is only an example)eg->
echo "<html>" > $OUTPUT_FILE
echo "<title>" >> $OUTPUT_FILE
echo "A Test script." >> $OUTPUT_FILE
echo "</title>" >> $OUTPUT_FILE
echo "</body>" >> $OUTPUT_FILE
echo "</html>" >> $OUTPUT_FILE
This above code is only an example where I am putting echo statements into output file. You could use it as per your use case. Also use "$Variable" in echo command.
I have several CSV files (all have the same number of rows and columns). Each file follows this format:
1 100.23 1 102.03 1 87.65
2 300.56 2 131.43 2 291.32
. . . . . .
. . . . . .
200 213.21 200 121.81 200 500.21
I need to extract columns 2, 4 and 6, and add them to a single CSV file.
I have a loop in my shell script which goes through all the CSV files, extracts the columns, and appends these columns to a single file:
#output header column
awk -F"," 'BEGIN {OFS=","}{ print $1; }' "$input" > $output
for f in "$1"*.csv;
do
if [[ -f "$f" ]] #removes symlinks (only executes on files with .csv extension)
then
fname=$(basename $f)
arr+=("$fname") #array to store filenames
paste -d',' $output <(awk -F',' '{ print $2","$4","$6; }' "$f") > temp.csv
mv temp.csv "$output"
fi
done
Running this produces this output:
1 100.23 102.03 87.65 219.42 451.45 903.1 ... 542.12 321.56 209.2
2 300.56 131.43 291.32 89.57 897.21 234.52 125.21 902.25 254.12
. . . . . . . . . .
. . . . . . . . . .
200 213.23 121.81 500.21 231.56 5023.1 451.09 ... 121.09 234.45 709.1
My desired output is a single CSV file that looks something like this:
1.csv 1.csv 1.csv 2.csv 2.csv 2.csv ... 700.csv 700.csv 700.csv
1 100.23 102.03 87.65 219.42 451.45 903.1 542.12 321.56 209.2
2 300.56 131.43 291.32 89.57 897.21 234.52 125.21 902.25 254.12
. . . . . . . . . .
. . . . . . . . . .
200 213.23 121.81 500.21 231.56 5023.1 451.09 ... 121.09 234.45 709.1
In other words, I need a header row containing the file names in order to identify which files the columns were extracted from. I can't seem to wrap my head around how to do this.
What is the easiest way to achieve this (preferably using awk)?
I was thinking of storing the file names into an array, inserting a header row and then print the array but I can't figure out the syntax.
So, based on a few assumptions:
the inputs are called "*.csv" but they're actually whitespace-separated, as they appear.
the odd-numbered input columns just repeat the row number 3 times, and can be ignored
the column headings are just the filenames, repeated 3 times each
they are input to some other program, and the numbers are left-justified anyway, so you aren't particular about the column formatting (columns lining up, decimals aligned, ...)
Humble apologies because code PRE formatting is not working for me here
f=$(set -- *.csv; echo $*)
(echo $f; paste $f) |
awk 'NR==1 { for (i=1; i<=NF; i++) {x=x" "$i" "$i" "$i} }
NR > 1 { x=$1; for (i=2; i<= NF; i+=2) {x=x" "$i} }
{print x}'
hth
I have the below script which combines many json files into single one. But the first file is printed twice in final output file even though i have removed the first file from the list.Please advice how to print first file only once.
Bash Script is pasted below.
#!/bin/bash
shopt -s nullglob
declare -a jsons
jsons=(o-*.json)
echo '[' > final.json
if [ ${#jsons[#]} -gt 0 ]; then
cat "${jsons[0]}" >> final.json
unset $jsons[0]
for f in "${jsons[#]}"; do # iterate over the rest
echo "," >>final.json
cat "$f" >>final.json
done
fi
echo ']' >>final.json
You can't use unset ${foo[0]} to remove an item from an array variable in bash.
$ foo=(a b c)
$ echo "${foo[#]}"
a b c
$ unset ${foo[0]}
$ echo "${foo[#]}"
a b c
You'll need to reset the array itself using array slicing.
$ foo=("${foo[#]:1}")
$ echo "${foo[#]}"
b c
See: How can I remove an element from an array completely?