I'm building a shell script that will run nightly with the help of crontab and my script keeps bombing out and giving me the error "Syntax error: word unexpected (expected "do").
The script itself is creating a new directory for each MySQL file it finds in a directory and then copies the files to a created directory from the respective mysqldump directory. Please let me know if I am on the right track with this code and what could be causing my end of file error. I am new to shell scripting and would accept any advice given. If there is a better way to write my code then feel free to help in that respect as well.
CDPATH="/backups/mysql"
#
now=$(date +"%Y_%m_%d")
#
# Find Directory Name
#
for file in */; do
dir=${file%/}
if [[ -e "$dir"]]
then
echo "Directory Exists!"
else
echo "Directory doesn't exist."
fi
done
#
# Copy MySQL Databases
#
while [ -d $dir ]; do # Check existing Dirs
cp -upf /dbase/files/*.sql /backups/mysql/$dir/$now # If Dir exists, create copy
if [ ! -d $dir ]; then # If Dir nonexistant create
mkdir -p $dir
cp -upf /dbase/files/*.sql /backups/mysql/$dir/$now
else # If all else fails just create a copy in /mysql
cp -upf /dbase/files/*.sql /backups/mysql/$dir/$now
fi
done
Thanks for the help in advance!
1. do not use "then" after "else"
2. The second line may does not what you want. It just set the variable dir to the string "dir (/backups/mysql/*)"
... obsolete since question has changed.
I assume, files in the following form ...
/dbase/files/db1.sql
/dbase/files/db2.sql
...
... should be backed up to the following destination:
/backups/mysql/db1/2014_11_18/db1.sql
#!/bin/bash
BACKUP_DIR="/backups/mysql/"
FILE_DIR=/dbase/files/
now=$(date +"%Y_%m_%d")
# setting the input field seperator to newline
IFS=$'\n'
# find db backups and loop over
for file in $(find ${FILE_DIR} -maxdepth 1 -name "*.sql" -type f -exec basename {} \;); do
# create backup directory:
mkdir -p "${BACKUP_DIR}${file%.sql}/${now}"
# copy file over
cp "${FILE_DIR}${file}" "${BACKUP_DIR}${file%.sql}/${now}/"
done
Convert the script to unix encoding:
dos2unix script.sh
Related
I am dealing with cuckoo sandbox exported data having report.json file under each zip file.
eg > 123456.zip each zip has the file in zipfile/reports/report.json
I have multiple zip files in a folder. I want to have those zip files to be named as zipfilename.json. I have tried many ways but to fail, here's the code I am trying:
#! /bin/bash
mkdir -p "DESTDIR"
for i in *.zip ; do
unzip "$i" $i/reports/report.json -d "DESTDIR"
mv "DESTDIR/reports/report.json" "DESTDIR/$(basename "$i" .zip)_THEFILE"
done
All I get is this output showing that the file does not exist:
(base) s#Sais-MBP Downloads % sh script.sh
Archive: 1959098.zip
caution: filename not matched: 1959098.zip/reports/report.json
mv: rename DESTDIR/THEFILE to DESTDIR/1959098_THEFILE: No such file or directory
Archive: 1959100.zip
caution: filename not matched: 1959100.zip/reports/report.json
mv: rename DESTDIR/THEFILE to DESTDIR/1959100_THEFILE: No such file or directory
Any help is greatly appreciated as I cannot make any progress for the past few days.
okay, I took help from a friend and he gave me the answer to it as I have done the whole script wrong
#! /bin/bash
#
# save this file as test.sh
#
mkdir -p "DESTDIR"
for ZIPFILE in *.zip ; do
NAME="${ZIPFILE%.*}"
mkdir -p "DESTDIR/$NAME"
unzip -j $ZIPFILE reports/report.json -d "DESTDIR/$NAME/"
mv "DESTDIR/$NAME/report.json" "DESTDIR/$NAME.json"
done
There are numerous ways to do this and the one I present here is one of them, using the “mktemp” tool that is already installed by default in many Linux distros.
Temporary Files and Directories
The "mktemp" tool comes by default in the "GNU coreutils" package and its purpose is solely to create temporary files/directories.
Creating tempfiles and tempdir
Its basic use is quite simple. Calling 'mktemp' from the command line without any parameters will create a file on your disk inside /tmp and whose name will be displayed on the screen. Look:
$ mktemp
/tmp/tmp.JVeOizVlqk
Creating directories is as simple as adding the “-d” parameter to the command line.
$ mktemp -d
/tmp/tmp.sBihXbcmKa
Using in practice
In practice there is no use in the temporary file or directory name being displayed on the screen. It must be stored in a variable that can be accessed at all times that can be read or written during processing.
In the example below there is a small script, useless by the way, but which shows us a suitable step-by-step. Look:
#!/bin/sh
#
#Create a temporary file
TEMP = `mktemp`
#Save all the content of the entered path
ls -l "$1"> $TEMP
#Read temporary file and list only directories
grep ^d $TEMP
sudo rm -f $TEMP
Note that the command "mktemp" was invoked in a subshell (between "- crase) and its output stored in the variable "TEMP".
Then, so that the file can be read or written just use the variable, where there would be the file name, as done in the ls, grep and rm commands.
If you needed to create directories, as already said, the process is the same, just adding a -d to the mktemp command line.
#!/bin/sh
#
TEMP = `mktemp -d`
cd $TEMP
.
.
.
sudo rm -rf $TEMP
If you want to create many temporary files, we use the “-p” parameter that specifies the path where the file should be created.
#!/bin/sh
#
TEMP = `mktemp -d`
cd $TEMP
FILE1 = `mktemp -p $TEMP`
.
.
.
sudo rm -f $FILE1
sudo rm -rf $TEMP
Only that. Your scripts will now be able to use temporary files more professionally.
However, I want to do this with tclsh instead of sh [bourn shell .. but I made a few attempts and nothing worked. Here's an example of what I tried:
# Create a temporary file
exec sh -c "TEMP =`mktemp -d`"
set dir [cd $TEMP]
# Save all content from the entered path
exec ls -l "$1"> $TEMP
set entry [glob -type f $env(dir) *.jpg]
# Read temporary file and list only directories
puts $entry
My biggest problem was and is in creating the variable
# Create a temporary file
exec sh -c "TEMP =`mktemp -d`"
This is not working!
Can someone give me a free sample ?!
Creating temporary files can be done using file tempfile. For directories file tempdir will be available in Tcl 8.7.
On Tcl versions before 8.7, you can use file tempfile to obtain a path to a temporary location and then create a directory by that name:
set fd [file tempfile temp]
close $fd
file delete $temp
file mkdir $temp
The file tempfile command also allows you to specify a template, similar to the -p option of mktemp
To answer your updated question, you can do something like:
# Create a temporary file
set temp [exec mktemp]
# Save all content from the entered path
exec ls -l [lindex $argv 0] > $temp
# Read temporary file
set f [open $temp]
set lines [split [read $f] \n]
close $f
# List only directories
puts [join [lsearch -all -inline $lines {d*}] \n]
I ignored your mixing up of directories and regular files and whatever the *.jpg is supposed to be.
Your attempts to create shell variables from inside Tcl and then use those in the next exec command will always fail because those variables are gone when the first subshell terminates. Keep the results in Tcl variables instead, like I did above.
Of course you could more easily find the directories using glob -type d, but I kept the shell command to serve as an example.
With example, the creating of directory temporary It would be like this:
# Create a temporary directory
set dir [exec mktemp -d] ;
# Now, files insert in directory
# (In this example I am decompressing a ZIP file and only JPEG format images)
exec unzip -x $env(HOME)/file.zip *.jpg -d $dir ;
# Look the directory now with this command:
puts [glob -nocomplain -type f -directory $dir -tails *.jpg] ;
I'm trying to write a shell script to recursively find all files under a directory with the extension .md and execute a command with the .md file and generated new file with the same name but a different extension.
below is the command I'm having but its actually appending the .html to the file instead of replacing .md with .html
find . -name '*.md' -exec markdown-html {} -s
resources/styles/common-custom.css -o {}.html \;
the above command generates a new file "home.md.html" from "home.md" but i want the .md removed. tried different solutions but didn't work
Hi you have to write a small script here, I have given the description how it is going to work, please refer to the comments in the below codes:-
First create a shell script file like convertTohtml.sh and add below codes in it
#!/bin/bash
find . -name '*.md' > filelist.dat
# list down all the file in a temp file
while read file
do
html_file=$( echo "$file" | sed -e 's/\.md//g')
# the above command will store 'home.md' as 'home' to variable 'html_file'
#hence '$html_file.html' equal to 'home.html'
markdown-html $file -s resources/styles/common-custom.css -o $html_file.html
done < filelist.dat
# with while loop read each line from the file. Here each line is a locatin of .md file
rm filelist.dat
#delete the temporary file finally
provide execute permission to your script file like below:-
chmod 777 convertTohtml.sh
Now execute the file:-
./convertTohtml.sh
Below script will work to solve the extension problem.
#!/bin/bash
find . -name '*.md' > filelist.dat
# list down all the file in a temp file
while read file
do
file=`echo ${file%.md}`
#get the filename witout extension
markdown-html $file -s resources/styles/common-custom.css -o $file.html
done < filelist.dat
# with while loop read each line from the file. Here each line is a locatin of .md file
rm filelist.dat
#delete the temporary file finally
If you want to use the output of find multiple times you could try something like this:
find . -name "*.md" -exec sh -c "echo {} && touch {}.foo" \;
Notice the:
sh -c "echo {} && touch {}.foo"
The sh -c will run commands read from the string, then the {} will be replaced with the find output, in this example is first doing an echo {} and if that succeeds && it will then touch {}.foo, in your case this could be like:
find . -name "*.md" -exec sh -c "markdown-html {} -s resources/styles/common-custom.css -o {}.html" \;
I have a shell script that I am trying to run every few days that copies .sql database files and moves them into a designated folder appeneded with /$now/. The script executes perfectly, but I am getting a cp: cannot create regular file '/path/to/dir/$now/': No such file or directory.
I know these folders exist because it is showing up when I 'ls -ltr' the directory.
All of my permissions are executable and writable.This has been puzzling me for about a week now and I just cant put a finger on it.
Here is my code:
#!/bin/bash
BACKUP_DIR="/backups/mysql/"
FILE_DIR=/dbase/files/
now=$(date +"%Y_%m_%d")
# setting the input field seperator to newline
IFS=$'\n'
# find db backups and loop over
for file in $(find ${FILE_DIR} -maxdepth 1 -name "*.sql" -type f -exec basename {} \;); do
# create backup directory:
mkdir -p "${BACKUP_DIR}${file%.sql}/${now}"
# copy file over
cp "${FILE_DIR}${file}" "${BACUP_DIR}${file%.sql}/${now}/"
done
Thanks in advance!
Update:
Error I am getting:
+ mkdir -pv /backups/mysql/health/2014_12_04
+ cp /dbase/files/health.sql health/2014_12_04/
cp: cannot create regular file 'health/2014_12_04/': No such file or directory
This is happening for all 9 directories already created
The error was a typo. I was missing the 'K' in $BACKUP_DIR on the cp line. Here is the correct code:
#!/bin/bash
BACKUP_DIR="/backups/mysql/"
FILE_DIR=/dbase/files/
now=$(date +"%Y_%m_%d")
# setting the input field seperator to newline
IFS=$'\n'
# find db backups and loop over
for file in $(find ${FILE_DIR} -maxdepth 1 -name "*.sql" -type f -exec basename {} \;); do
# create backup directory:
mkdir -p "${BACKUP_DIR}${file%.sql}/${now}"
# copy file over
cp "${FILE_DIR}${file}" "${BACKUP_DIR}${file%.sql}/${now}/"
done
I have the following shell script which writes volumes in a find command to a file, and loads that file into a mysql database:
# find all the paths and print them to a file
sudo find $FULFILLMENT/ > $FILE
sudo find $ARCH1/ >> $FILE
sudo find $ARCH2/ >> $FILE
sudo find $MASTERING/ >> $FILE
# load the file into the mysql database, `files`, table `path`
/usr/local/bin/mysql -u root files -e "TRUNCATE path"
/usr/local/bin/mysql -u root files -e "LOAD DATA INFILE '/tmp/files.txt' INTO TABLE path"
TRUNCATE is to be used to delete all the old entries before adding the new. However, if any of the find commands don't work (for example, if the volume isn't accessible), I want it to skip on the two mysql commands. How would I modify the above script to do this?
This will execute your two mysql commands only if all the find commands succeed:
# find all the paths and print them to a file
sudo find $FULFILLMENT/ > $FILE &&
sudo find $ARCH1/ >> $FILE &&
sudo find $ARCH2/ >> $FILE &&
sudo find $MASTERING/ >> $FILE &&
{
# load the file into the mysql database, `files`, table `path`
/usr/local/bin/mysql -u root files -e "TRUNCATE path"
/usr/local/bin/mysql -u root files -e "LOAD DATA INFILE '/tmp/files.txt' INTO TABLE path"
}
The && operator causes the command on the RHS to run only if the command on the LHS succeeds. The { ... } groups the two mysql commands into one compound command, so either both run (if the last find succeeds) or neither does (if the last find does not succeed).
You can use this if there is more to your script that should run whether or not the finds succeed and the mysql commands run.
Something like:
sudo find $FULFILLMENT > $FILE || exit 1
You can use #!/bin/bash -e or add a line with set -e above the find commands. That way, Bash will automatically abort as soon as a command fails. It's good practice to always do this by default, since you rarely want to continue when the previous command fails:
set -e
# find all the paths and print them to a file
sudo find $FULFILLMENT/ > $FILE
sudo find $ARCH1/ >> $FILE
sudo find $ARCH2/ >> $FILE
sudo find $MASTERING/ >> $FILE
# load the file into the mysql database, `files`, table `path`
/usr/local/bin/mysql -u root files -e "TRUNCATE path"
/usr/local/bin/mysql -u root files -e "LOAD DATA INFILE '/tmp/files.txt' INTO TABLE path"