I'm an trying to pipe output data directly from sed into mysql by doing something like:
sed "s/url\/uct\/video/url\/uct\/video/g" 2019-11-25T0330-main.sql.gz | mysql opencast
Is this possible? I keep running out of space and therefore piping it straight into mysql would be ideal (instead of running sed independently).
You need
gunzip main.sql.gz | sed 's#url/uct/video#url/uct/video#g' | mysql opencast
Note if you use a different char besides / for seds s/str/rep/ you can avoid having to escape / chars that are part of the sub/repl patterns. For some OS's (besided Linux) users, you may need to escape the first use of the alternate delimit character, like sed '\#sub#repl#
And if you are using the -i option for sed, it doesn't make sense in the context of reading input from a pipe ;-)
IHTH
Related
I am writing a bash script that extracts pdf files from html and downloads it. Here is the line of code that extracts:
curl -s https://info.uqam.ca/\~privat/INF1070/ |
sed 's/.*href="//' |
sed 's/".*//' |
sed '/^[^\.]/d' |
sed '/\.[^p][^d][^f]$/d' |
sed '/^$/d' |
sed '/\/$/d'
Result:
./07b-reseau.pdf
./07a-reseau.pdf
./06b-script.pdf
./06a-script.pdf
./05-processus.pdf
./04b-regex.pdf
./181-quiz1-g1-sujet.pdf
./03b-fichiers-solution.pdf
./04a-regex.pdf
./03d-fichiers.pdf
./03c-fichiers.pdf
./03b-fichiers.pdf
./03a-fichiers.pdf
./02-shell.pdf
./01-intro.pdf
./01-intro.pdf
./02-shell.pdf
./03a-fichiers.pdf
./03b-fichiers.pdf
./03b-fichiers-solution.pdf
./03c-fichiers.pdf
./03d-fichiers.pdf
./04a-regex.pdf
./04b-regex.pdf
./05-processus.pdf
./06a-script.pdf
./06b-script.pdf
./07a-reseau.pdf
./07b-reseau.pdf
./181-quiz1-g1-sujet.pdf
It's working fine but I was wondering if there is a better way (always by using sed) to do this with less sed commands.
Thank you.
You can translate your original question into something like How to output only captured groups with sed?. This one-liner should do the trick for you:
curl -s https://info.uqam.ca/\~privat/INF1070/ | sed -rn 's/.*href="(.*\.pdf)".*$/\1/p'
which produces the desired output.
Where the combination of the -n option (not to print) and the p flag (print what is matched) will print only the lines where substitution take place based on the regex .*href="(.*\.pdf)".*$. The value of the href attribute (the capture group in parenthesis) is back referenced with \1, thus the whole line is replaced with it.
This might work for you (GNU sed):
sed -r '/\n/!s/href="(\.[^"]*\.pdf)"/\n\1\n/g;/\`[^\n]*\.pdf$/MP;D' file
This puts each pdf file into a separate line (multiple lines within a line) and only prints out a line that ends in .pdf.
I am trying to change the values in a text file using sed in a Bash script with the line,
sed 's/draw($prev_number;n_)/draw($number;n_)/g' file.txt > tmp
This will be in a for loop. Why is it not working?
Variables inside ' don't get substituted in Bash. To get string substitution (or interpolation, if you're familiar with Perl) you would need to change it to use double quotes " instead of the single quotes:
# Enclose the entire expression in double quotes
$ sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
# Or, concatenate strings with only variables inside double quotes
# This would restrict expansion to the relevant portion
# and prevent accidental expansion for !, backticks, etc.
$ sed 's/draw('"$prev_number"';n_)/draw('"$number"';n_)/g' file.txt > tmp
# A variable cannot contain arbitrary characters
# See link in the further reading section for details
$ a='foo
bar'
$ echo 'baz' | sed 's/baz/'"$a"'/g'
sed: -e expression #1, char 9: unterminated `s' command
Further Reading:
Difference between single and double quotes in Bash
Is it possible to escape regex metacharacters reliably with sed
Using different delimiters for sed substitute command
Unless you need it in a different file you can use the -i flag to change the file in place
Variables within single quotes are not expanded, but within double quotes they are. Use double quotes in this case.
sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
You could also make it work with eval, but don’t do that!!
This may help:
sed "s/draw($prev_number;n_)/draw($number;n_)/g"
You can use variables like below. Like here, I wanted to replace hostname i.e., a system variable in the file. I am looking for string look.me and replacing that whole line with look.me=<system_name>
sed -i "s/.*look.me.*/look.me=`hostname`/"
You can also store your system value in another variable and can use that variable for substitution.
host_var=`hostname`
sed -i "s/.*look.me.*/look.me=$host_var/"
Input file:
look.me=demonic
Output of file (assuming my system name is prod-cfm-frontend-1-usa-central-1):
look.me=prod-cfm-frontend-1-usa-central-1
I needed to input github tags from my release within github actions. So that on release it will automatically package up and push code to artifactory.
Here is how I did it. :)
- name: Invoke build
run: |
# Gets the Tag number from the release
TAGNUMBER=$(echo $GITHUB_REF | cut -d / -f 3)
# Setups a string to be used by sed
FINDANDREPLACE='s/${GITHUBACTIONSTAG}/'$(echo $TAGNUMBER)/
# Updates the setup.cfg file within version number
sed -i $FINDANDREPLACE setup.cfg
# Installs prerequisites and pushes
pip install -r requirements-dev.txt
invoke build
Retrospectively I wish I did this in python with tests. However it was fun todo some bash.
Another variant, using printf:
SED_EXPR="$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)"
sed "${SED_EXPR}" file.txt
or in one line:
sed "$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)" file.txt
Using printf to build the replacement expression should be safe against all kinds of weird things, which is why I like this variant.
I have bunch of json files that contains web events. Each event contains lots of stuff and I'm trying to do ip address anonymization (replacing last segment of ip addresses with 0) with sed.
In short:
How to find substrings like "ip":"34.542.3.34" from json files and transform them to "ip":"34.542.3.0"?
Attempts:
Resetting starting point with \K
sed -e 's/"ip":"[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.\K[0-9]{1,3}/0/g' file.json
This would work but unfortunately sed doesn't seem support reseting starting point.
Negative lookbehind
sed -e 's/(?<="ip":"[0-9]{3}\.[0-9]{3}\.[0-9]{3}\.)([0-9]{3})/0/g' file.json
This would also work but negative lookbehind doesn't seem to support non-fixed-width assortions. So [0-9]{1,3} is not supported and hence this won't work.
Matching groups
Third idea was to use matching groups and do something like this
sed -e 's/("ip":"[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|([0-9]{1,3})/\1\20/g' file.json
But couldn't figure out how this would actually work with sed.
Writing regex for every possible length option separately
This would probably work but it would make regex too long and hardly readable. I would like to find more convenient and clean solution.
Any suggestions?
With GNU sed:
sed -r 's/("ip":"[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)[0-9]{1,3}(")/\10\2/' file
When there's no -r-option:
sed -e 's/\("ip":"[[:digit:]]\{1,3\}\(\.[[:digit:]]\{1,3\}\)\{2\}\.\)\([[:digit:]]\{1,3\}\)"/\10"/g' tst.json
I am trying to get a JSON key and change its property in the file using -i flag with sed. The issue is, I cannot get this regex right. It works perfectly fine for the simple replace case, but I cannot get it working using this regex. For simplicity, I have just tried a simple echo, rather than saving it to the file. Ideas?
x=0.0.179
echo "version: 0.0.178" | sed 's/^[ ]*\"version\"[ ]*:[ ]*\"\([0-9]+\.[0-9]+\.[0-9]+\)\".*$/\$x/'
0.0.178
Why should you be complicating it with sed, just do,
x="0.0.179"
echo "version: 0.0.178" | sed "s/version: .*/version: $x/"
version: 0.0.179
and BTW if your JSON input can be parsed and modified via jq, go for it. Use this ONLY as a last resort.
I think your sed regexp is looking for the version number within double-quotes. Your input to sed above is not quoted as such, and hence doesn't get replaced (I'd expect your JSON to be double-quoted though, hence my interest above re. your real JSON input).
This works. Mention the 's around $x
echo "version: 0.0.178" | sed 's/^[ ]*\"version\"[ ]*:[ ]*\"\([0-9]+\.[0-9]+\.[0-9]+\)\".*$/\'$x'/'
Bash variables do not get replaced inside strings with '. Use " whenever possible to avoid this, or just concat your variable into the command as shown above.
I'm trying to replace the content of some HTML tags in an HTML page using sed in a bash script. For some reason I'm not getting the proper result as it's not replacing anything. It has to be something very simple/stupid im overlooking, anyone care to help me out?
HTML to search/replace in:
Unlocked <span id="unlockedCount"></span>/<span id="totalCount"></span> achievements for <span id="totalPoints"></span> points.
sed command used:
cat index.html | sed -i -e "s/\<span id\=\"unlockedCount\"\>([0-9]\{0,\})\<\/span\>/${unlockedCount}/g" index.html
The point of this is to parse the HTML page and update the figures according to some external data. For a first run, the contents of the tags will be empty, after that they will be filled.
EDIT:
I ended up using a combination of the answers which resulted in the following code:
sed -i -e 's|<span id="unlockedCount">\([0-9]\{0,\}\)</span>|<span id="unlockedCount">'"${unlockedCount}"'</span>|g' index.html
Many thanks to #Sorpigal, #tripleee, #classic for the help!
Try this:
sed -i -e "s/\(<span id=\"unlockedCount\">\)\(<\/span>\)/\1${unlockedCount}\2/g" index.html
What you say you want to do is not what you're telling sed to do.
You want to insert a number into a tag or replace it if present. What you're trying to tell sed to do is to replace a span tag and its contents, if any or a number, with the value of in a shell variable.
You're also employing a lot of complex, annoying and erorr-prone escape sequences which are just not necessary.
Here's what you want:
sed -r -i -e 's|<span id="unlockedCount">([0-9]{0,})</span>|<span id="unlockedCount">'"${unlockedCount}"'</span>|g' index.html
Note the differences:
Added -r to turn on extended expressions without which your capture pattern would not work.
Used | instead of / as the delimiter for the substitution so that escaping / would not be necessary.
Single-quoted the sed expression so that escaping things inside it from the shell would not be necessary.
Included the matched span tag in the replacement section so that it would not get deleted.
In order to expand the unlockedCount variable, closed the single-quoted expression, then later re-opened it.
Omitted cat | which was useless here.
I also used double quotes around the shell variable expansion, because this is good practice but if it contains no spaces this is not really necessary.
It was not, strictly speaking, necessary for me to add -r. Plain old sed will work if you say \([0-9]\{0,\}\), but the idea here was to simplify.
sed -i -e 's%<span id="unlockedCount">([0-9]*)</span\>/'"${unlockedCount}/g" index.html
I removed the Useless Use of Cat, took out a bunch of unnecessary backslashes, added single quotes around the regex to protect it from shell expansion, and fixed the repetition operator. You might still need to backslash the grouping parentheses; my sed, at least, wants \(...\).
Note the use of single and double quotes next to each other. Single quotes protect against shell expansion, so you can't use them around "${unlockedCount}" where you do want the shell to interpolate the variable.