My scheme
/geomesa-accumulo describe-schema -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder
INFO Describing attributes of feature 'SignalBuilder'
geo | Point (Spatio-temporally indexed) (Spatially indexed)
time | Date (Spatio-temporally indexed) (Attribute indexed)
cam | String (Attribute indexed) (Attribute indexed)
imei | String
dir | Double
alt | Double
vlc | Double
sl | Integer
ds | Integer
dir_y | Double
poi_azimuth_x | Double
poi_azimuth_y | Double
User data:
geomesa.attr.splits | 4
geomesa.feature.expiry | time(30 days)
geomesa.index.dtg | time
geomesa.indices | z3:7:3:geo:time,z2:5:3:geo,attr:8:3:time,attr:8:3:cam,attr:8:3:cam:time
geomesa.stats.enable | true
geomesa.table.partition | time
geomesa.z.splits | 4
geomesa.z3.interval | week
When I try to get count by stat methods it retuns 11:
./geomesa-accumulo stats-count -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder -q "cam='9f471340-dd70-4eca-a8dc-14553a4e708a'"
Estimated count: 11
but without cache:
./geomesa-accumulo stats-count -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder -q "cam='9f471340-dd70-4eca-a8dc-14553a4e708a'" --no-cache
INFO Running stat query...
Count: 1436
Why stats methods not worked properly and return only estimated value?
In redis it's all ok. The problem is only in accumulo.
** Question update:
I try to recalculate statistics
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-analyze -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder
INFO Running stat analysis for feature type SignalBuilder...
INFO Stats analyzed:
Total features: 11527
Bounds for geo: [ 37.598007, 55.736623, 38.661036, 56.9189592 ] cardinality: 10634
Bounds for time: [ 2022-01-30T15:13:58.706Z to 2022-02-09T14:16:03.000Z ] cardinality: 3779
Bounds for cam: [ 3fe961e1-91dd-4931-b82e-d04fcaf24c3e to f767f0fa-dac5-4571-aa47-1ea6bf6e2c82 ] cardinality: 6
INFO Use 'stats-histogram', 'stats-top-k' or 'stats-count' commands for more details
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-count -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder -q "cam='9f471340-dd70-4eca-a8dc-14553a4e708a'"
Estimated count: 14
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-count -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder -q "cam='3fe961e1-91dd-4931-b82e-d04fcaf24c3e'"
Estimated count: 0
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-count -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder -q "cam='3fe961e1-91dd-4931-b82e-d04fcaf24c3e'" --no-cache
INFO Running stat query...
Count: 2675
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-analyze -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder
INFO Running stat analysis for feature type SignalBuilder...
INFO Stats analyzed:
Total features: 11767
Bounds for geo: [ 37.598007, 55.736623, 38.661036, 56.9189592 ] cardinality: 10942
Bounds for time: [ 2022-01-30T15:13:58.706Z to 2022-02-09T14:17:41.000Z ] cardinality: 3841
Bounds for cam: [ 3fe961e1-91dd-4931-b82e-d04fcaf24c3e to f767f0fa-dac5-4571-aa47-1ea6bf6e2c82 ] cardinality: 6
INFO Use 'stats-histogram', 'stats-top-k' or 'stats-count' commands for more details
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-count -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder -q "1=1"
Estimated count: Unknown
Re-run with --no-cache to get an exact count
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-count -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder -q "1=1" --no-cache
INFO Running stat query...
Count: 11872
But it does not help (((. The geo-events continue to arrive to geomesa. But stats does not worked.
May by I'm not using stats-count properly. Stats-top-k shows gathered statistics.
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-count -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder -q "cam like '3fe961e1-91dd-4931-b82e-d04fcaf24c3e'"
Estimated count: 0
~/bin/geomesa-accumulo_2.12-3.2.2/bin ./geomesa-accumulo stats-top-k -c myNamespace.geomesa -z 10.200.217.27 -i accumulo -u root -p qweasd123 -f SignalBuilder
Top values for 'geo':
unavailable
Top values for 'time':
unavailable
Top values for 'cam':
7c0cf8bc-e7e3-4023-8a00-a5f17bda3001 (2925)
9f471340-dd70-4eca-a8dc-14553a4e708a (2924)
f767f0fa-dac5-4571-aa47-1ea6bf6e2c82 (2922)
bfe55ad1-5b0a-405d-9ca9-3bed6aca9313 (2921)
3fe961e1-91dd-4931-b82e-d04fcaf24c3e (2920)
5798a065-d51e-47a1-b04b-ab48df9f1324 (2)
Top values for 'imei':
unavailable
Top values for 'dir':
unavailable
Top values for 'alt':
unavailable
Top values for 'vlc':
unavailable
Top values for 'sl':
unavailable
Top values for 'ds':
unavailable
Top values for 'dir_y':
unavailable
Top values for 'poi_azimuth_x':
unavailable
Top values for 'poi_azimuth_y':
unavailable
Or maybe the reason was in accumulo. When I try to get data from accumulo table. It returns
root#accumulo> scan -t myNamespace.geomesa_SignalBuilder_z3_geo_time_v7_02717
2022-02-09 17:55:12,909 [commands.ShellPluginConfigurationCommand] ERROR: Error: Could not determine the type of file "hdfs://10.200.217.27:9000/accumulo/classpath/myNamespace/[^.].*.jar".
2022-02-09 17:55:12,909 [shell.Shell] ERROR: Could not load the specified formatter. Using the DefaultFormatter
2022-02-09 17:55:12,929 [commands.ShellPluginConfigurationCommand] ERROR: Error: Could not determine the type of file "hdfs://10.200.217.27:9000/accumulo/classpath/myNamespace/[^.].*.jar".
\x01\x0A\x9Dt\x19\x84\xEF\xDD\xAF "5798a065-d51e-47a1-b04b-ab48df9f1324-1643555638706 d: [] \x03\x00\x0C\x02\x00\x1E\x000\x008\x00\\\x00g\x00o\x00w\x00\x7F\x00\x83\x00\x87\x00\x87\x00\x87\x00\x87\x00\x00\x0E\x00\x01\x01#CT\x9C\xD3\xE0\xBDE#Lu\xA0t\x7F-\xDE\x00\x00\x01~\xAB\x8C\xCD\xB25798a065-d51e-47a1-b04b-ab48df9f132\xB43333333333\xB1#f#\x00\x00\x00\x00\x00?\xF3\xAE\x14z\xE1G\xAE\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x01
\x01\x0A\x9Dt\x19\x84\xEF\xDD\xBD!\x065798a065-d51e-47a1-b04b-ab48df9f1324-1643555648706 d: [] \x03\x00\x0C\x02\x00\x1E\x000\x008\x00\\\x00g\x00o\x00w\x00\x7F\x00\x83\x00\x87\x00\x87\x00\x87\x00\x87\x00\x00\x0E\x00\x01\x01#CT\x9C\xD3\xE0\xBDE#Lu\xA0t\x7F-\xDE\x00\x00\x01~\xAB\x8C\xF4\xC25798a065-d51e-47a1-b04b-ab48df9f132\xB43333333333\xB1#f#\x00\x00\x00\x00\x00?\xF3\xAE\x14z\xE1G\xAE\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x01
Stats are gathered during ingestion, but are only written on a "best effort" basis (for example, if your ingest dies, stats may not be written). There are also code paths that don't update stats, for example if you disable them via system property or if you ingest through a bulk map/reduce job. In your particular case, it's hard to say why your stats don't match your data without a detailed description of everything you did to ingest it. However, if you want to re-calculate the cached statistics, you can always run the stats-analyze CLI command.
If you can re-create the issue, please feel free to file a ticket in the GeoMesa JIRA with the steps to re-create.
Related
I'm running an MySQL server docker container using a docker-compose YAML file.
Here is how the file looks like:
version: '3.1'
services:
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
volumes:
- ./mysql-dump/samples:/docker-entrypoint-initdb.d
environment:
MYSQL_ROOT_PASSWORD: example
MYSQL_DATABASE: db_example
adminer:
image: adminer
restart: always
ports:
- 8080:8080
In the db service, the volumne is set to ./mysql-dump/samples:/docker-entrypoint-initdb.d this takes .sql files from ./mysql-dump/sample to inject them to the database.
In my case I have two files file2.sql for the sql schema of the database, and file1.sql for the data.
Since the file appear to be injected in order, I get a NO SUCH TABLE ERROR, surely because the schema is injected last (because it's name is file2.sql)
Is there a way to reverse the order of the injection beside changing the names of the files?
If you go through the documentation of mysql Dockerhub it clearly mentioned that it will dump file in alphabetical order.
When a container is started for the first time, a new database with
the specified name will be created and initialized with the provided
configuration variables. Furthermore, it will execute files with
extensions .sh, .sql and .sql.gz that are found in
/docker-entrypoint-initdb.d. Files will be executed in alphabetical
order. You can easily populate your mysql services by mounting a SQL
dump into that directory and provide custom images with contributed
data. SQL files will be imported by default to the database specified
by the MYSQL_DATABASE variable.
You need to replace file name, suppose db.sql and table.sql so it will first dump db.sql then table.sql
Updated:
To reverse the order of MySQL dump, you have to modify the docker file and entry point.
FROM mysql:8
#From mysql
COPY docker-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
EXPOSE 3306 33060
CMD ["mysqld"]
ENTRYPOINT:
#!/bin/bash
set -x
set -eo pipefail
shopt -s nullglob
# if command starts with an option, prepend mysqld
if [ "${1:0:1}" = '-' ]; then
set -- mysqld "$#"
fi
# skip setup if they want an option that stops mysqld
wantHelp=
for arg; do
case "$arg" in
-'?'|--help|--print-defaults|-V|--version)
wantHelp=1
break
;;
esac
done
# usage: file_env VAR [DEFAULT]
# ie: file_env 'XYZ_DB_PASSWORD' 'example'
# (will allow for "$XYZ_DB_PASSWORD_FILE" to fill in the value of
# "$XYZ_DB_PASSWORD" from a file, especially for Docker's secrets feature)
file_env() {
local var="$1"
local fileVar="${var}_FILE"
local def="${2:-}"
if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
exit 1
fi
local val="$def"
if [ "${!var:-}" ]; then
val="${!var}"
elif [ "${!fileVar:-}" ]; then
val="$(< "${!fileVar}")"
fi
export "$var"="$val"
unset "$fileVar"
}
# usage: process_init_file FILENAME MYSQLCOMMAND...
# ie: process_init_file foo.sh mysql -uroot
# (process a single initializer file, based on its extension. we define this
# function here, so that initializer scripts (*.sh) can use the same logic,
ls -r
process_init_file() {
local f="$1"; shift
local mysql=( "$#" )
case "$f" in
*.sh) echo "$0: running $f"; . "$f" ;;
*.sql) echo "$0: running $f"; "${mysql[#]}" < "$f"; echo ;;
*.sql.gz) echo "$0: running $f"; gunzip -c "$f" | "${mysql[#]}"; echo ;;
*) echo "$0: ignoring $f" ;;
esac
echo
}
_check_config() {
toRun=( "$#" --verbose --help )
if ! errors="$("${toRun[#]}" 2>&1 >/dev/null)"; then
cat >&2 <<-EOM
ERROR: mysqld failed while attempting to check config
command was: "${toRun[*]}"
$errors
EOM
exit 1
fi
}
# Fetch value from server config
# We use mysqld --verbose --help instead of my_print_defaults because the
# latter only show values present in config files, and not server defaults
_get_config() {
local conf="$1"; shift
"$#" --verbose --help --log-bin-index="$(mktemp -u)" 2>/dev/null \
| awk '$1 == "'"$conf"'" && /^[^ \t]/ { sub(/^[^ \t]+[ \t]+/, ""); print; exit }'
# match "datadir /some/path with/spaces in/it here" but not "--xyz=abc\n datadir (xyz)"
}
# allow the container to be started with `--user`
if [ "$1" = 'mysqld' -a -z "$wantHelp" -a "$(id -u)" = '0' ]; then
_check_config "$#"
DATADIR="$(_get_config 'datadir' "$#")"
mkdir -p "$DATADIR"
chown -R mysql:mysql "$DATADIR"
exec gosu mysql "$BASH_SOURCE" "$#"
fi
if [ "$1" = 'mysqld' -a -z "$wantHelp" ]; then
# still need to check config, container may have started with --user
_check_config "$#"
# Get config
DATADIR="$(_get_config 'datadir' "$#")"
if [ ! -d "$DATADIR/mysql" ]; then
file_env 'MYSQL_ROOT_PASSWORD'
if [ -z "$MYSQL_ROOT_PASSWORD" -a -z "$MYSQL_ALLOW_EMPTY_PASSWORD" -a -z "$MYSQL_RANDOM_ROOT_PASSWORD" ]; then
echo >&2 'error: database is uninitialized and password option is not specified '
echo >&2 ' You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD'
exit 1
fi
mkdir -p "$DATADIR"
echo 'Initializing database'
"$#" --initialize-insecure
echo 'Database initialized'
if command -v mysql_ssl_rsa_setup > /dev/null && [ ! -e "$DATADIR/server-key.pem" ]; then
# https://github.com/mysql/mysql-server/blob/23032807537d8dd8ee4ec1c4d40f0633cd4e12f9/packaging/deb-in/extra/mysql-systemd-start#L81-L84
echo 'Initializing certificates'
mysql_ssl_rsa_setup --datadir="$DATADIR"
echo 'Certificates initialized'
fi
SOCKET="$(_get_config 'socket' "$#")"
"$#" --skip-networking --socket="${SOCKET}" &
pid="$!"
mysql=( mysql --protocol=socket -uroot -hlocalhost --socket="${SOCKET}" )
for i in {30..0}; do
if echo 'SELECT 1' | "${mysql[#]}" &> /dev/null; then
break
fi
echo 'MySQL init process in progress...'
sleep 1
done
if [ "$i" = 0 ]; then
echo >&2 'MySQL init process failed.'
exit 1
fi
if [ -z "$MYSQL_INITDB_SKIP_TZINFO" ]; then
# sed is for https://bugs.mysql.com/bug.php?id=20545
mysql_tzinfo_to_sql /usr/share/zoneinfo | sed 's/Local time zone must be set--see zic manual page/FCTY/' | "${mysql[#]}" mysql
fi
if [ ! -z "$MYSQL_RANDOM_ROOT_PASSWORD" ]; then
export MYSQL_ROOT_PASSWORD="$(pwgen -1 32)"
echo "GENERATED ROOT PASSWORD: $MYSQL_ROOT_PASSWORD"
fi
rootCreate=
# default root to listen for connections from anywhere
file_env 'MYSQL_ROOT_HOST' '%'
if [ ! -z "$MYSQL_ROOT_HOST" -a "$MYSQL_ROOT_HOST" != 'localhost' ]; then
# no, we don't care if read finds a terminating character in this heredoc
# https://unix.stackexchange.com/questions/265149/why-is-set-o-errexit-breaking-this-read-heredoc-expression/265151#265151
read -r -d '' rootCreate <<-EOSQL || true
CREATE USER 'root'#'${MYSQL_ROOT_HOST}' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}' ;
GRANT ALL ON *.* TO 'root'#'${MYSQL_ROOT_HOST}' WITH GRANT OPTION ;
EOSQL
fi
"${mysql[#]}" <<-EOSQL
-- What's done in this file shouldn't be replicated
-- or products like mysql-fabric won't work
SET ##SESSION.SQL_LOG_BIN=0;
ALTER USER 'root'#'localhost' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}' ;
GRANT ALL ON *.* TO 'root'#'localhost' WITH GRANT OPTION ;
${rootCreate}
DROP DATABASE IF EXISTS test ;
FLUSH PRIVILEGES ;
EOSQL
if [ ! -z "$MYSQL_ROOT_PASSWORD" ]; then
mysql+=( -p"${MYSQL_ROOT_PASSWORD}" )
fi
file_env 'MYSQL_DATABASE'
if [ "$MYSQL_DATABASE" ]; then
echo "CREATE DATABASE IF NOT EXISTS \`$MYSQL_DATABASE\` ;" | "${mysql[#]}"
mysql+=( "$MYSQL_DATABASE" )
fi
file_env 'MYSQL_USER'
file_env 'MYSQL_PASSWORD'
if [ "$MYSQL_USER" -a "$MYSQL_PASSWORD" ]; then
echo "CREATE USER '$MYSQL_USER'#'%' IDENTIFIED BY '$MYSQL_PASSWORD' ;" | "${mysql[#]}"
if [ "$MYSQL_DATABASE" ]; then
echo "GRANT ALL ON \`$MYSQL_DATABASE\`.* TO '$MYSQL_USER'#'%' ;" | "${mysql[#]}"
fi
echo 'FLUSH PRIVILEGES ;' | "${mysql[#]}"
fi
echo
ls -r /docker-entrypoint-initdb.d/ > /dev/null
for f in $(ls -r /docker-entrypoint-initdb.d/*); do
process_init_file "$f" "${mysql[#]}"
done
if [ ! -z "$MYSQL_ONETIME_PASSWORD" ]; then
"${mysql[#]}" <<-EOSQL
ALTER USER 'root'#'%' PASSWORD EXPIRE;
EOSQL
fi
if ! kill -s TERM "$pid" || ! wait "$pid"; then
echo >&2 'MySQL init process failed.'
exit 1
fi
echo
echo 'MySQL init process done. Ready for start up.'
echo
fi
fi
exec "$#"
If you run the container, You will see the file is in processing reverse order
I am using MySQL on centos7. I have 50 databases Like database1, database2...., database50.
How can I set a cronjob for take a dump every day of all database same as database name [ Like database1.sql, database2.sql .... database50.sql ] using single command or script.
Please provide some adequate solution that will be appreciated.
Thanks.
Convert the current date to an integer number of days since some starting date.
Take that modulo 50. This gives you 0 .. 49.
Add 1 and concatenate. Now you have database1 .. database50. Put that in the shell variable db
mysqldump ... $db >$db.sql
I am using this script
#! /bin/bash
# MySQL database backup (databases in separate files) with daily, weekly and monthly rotation
# Sebastian Flippence (http://seb.flippence.net) originally based on code from: Ameir Abdeldayem (http://www.ameir.net)
# You are free to modify and distribute this code,
# so long as you keep the authors name and URL in it.
# Modified by IVO GELOV
# How many backups do you want to keep?
MAX_DAYS=5
# Date format that is appended to filename
DATE=`date +'%Y-%m-%d'`
DATSTR=`date '+%Y%m%d' -d "-$MAX_DAYS days"`
# MySQL server's name
SERVER=""
# Directory to backup to
BACKDIR="/var/db_arhiv/mysql"
#----------------------MySQL Settings--------------------#
# MySQL server's hostname or IP address
HOST="localhost"
# MySQL username
USER="user"
# MySQL password
PASS="password"
# List all of the MySQL databases that you want to backup,
# each separated by a space. Or set the option below to backup all database
DBS="db1 db2"
# Set to 'y' if you want to backup all your databases. This will override
# the database selection above.
DUMPALL="y"
# Custom path to system commands (enable these if you want use a different
# location for PHP and MySQL or if you are having problems running this script)
MYSQL="/usr/local/mysql/bin/mysql"
MYSQLDUMP="/usr/local/mysql/bin/mysqldump"
function checkMysqlUp() {
$MYSQL -N -h $HOST --user=$USER --password=$PASS -e status > /dev/null
}
trap checkMysqlUp 0
function error() {
local PARENT_LINENO="$1"
local MESSAGE="$2"
local CODE="${3:-1}"
if [[ -n "$MESSAGE" ]] ; then
echo "Error on or near line ${PARENT_LINENO}: ${MESSAGE}; exiting with status ${CODE}"
else
echo "Error on or near line ${PARENT_LINENO}; exiting with status ${CODE}"
fi
exit "${CODE}"
}
trap 'error ${LINENO}' ERR
# Check backup directory exists
# if not, create it
if [ ! -e "$BACKDIR/$DATE" ]; then
mkdir -p "$BACKDIR/$DATE"
echo "Created backup directory (${BACKDIR}/${DATE})"
fi
if [ $DUMPALL = "y" ]; then
echo "Creating list of databases on: ${HOST}..."
$MYSQL -N -h $HOST --user=$USER --password=$PASS -e "show databases;" > ${BACKDIR}/dbs_on_${SERVER}.txt
# redefine list of databases to be backed up
DBS=`sed -e ':a;N;$!ba;s/\n/ /g' -e 's/Database //g' ${BACKDIR}/dbs_on_${SERVER}.txt`
fi
echo "Backing up MySQL databases..."
#cd ${LATEST}
for database in $DBS; do
if [ ${database} = "information_schema" ] || [ ${database} = "performance_schema" ] || [ ${database} = "pinba" ]
then
continue
fi
echo "${database}..."
$MYSQLDUMP --host=$HOST --user=$USER --password=$PASS --default-character-set=utf8 --routines --triggers --lock-tables --disable-keys --force --single-transaction --allow-keywords --dump-date $database > ${BACKDIR}/${DATE}/${SERVER}$database.sql
done
if [ $DUMPALL = "y" ]; then
rm -f ${BACKDIR}/dbs_on_${SERVER}.txt
fi
# dump privileges
$MYSQL -N -h $HOST --user=$USER --password=$PASS --skip-column-names -A -e "SELECT CONCAT('SHOW GRANTS FOR ''',user,'''#''',host,''';') FROM mysql.user" | $MYSQL -N -h $HOST --user=$USER --password=$PASS --skip-column-names -A > ${BACKDIR}/${DATE}/${SERVER}_grants.sql
# delete older files
for x in `find ${BACKDIR}/20* -type d`
do
xd=`basename "${x//-/}"`
if [[ $xd < $DATSTR ]]
then
rm -rf "$x"
fi
done
echo "MySQL backup is complete"
From the docker distribution document: https://github.com/docker/distribution
It says to configure the docker to use the mirror, we should:
Configuring the Docker daemon
You will need to pass the --registry-mirror option to your Docker daemon on startup:
docker --registry-mirror=https://<my-docker-mirror-host> daemon
I'm newbie to docker, and I start docker from mac normal by the provided "Docker Quickstart Termial" app, which actaully invokes a start.sh shell:
#!/bin/bash
VM=default
DOCKER_MACHINE=/usr/local/bin/docker-machine
VBOXMANAGE=/Applications/VirtualBox.app/Contents/MacOS/VBoxManage
BLUE='\033[0;34m'
GREEN='\033[0;32m'
NC='\033[0m'
unset DYLD_LIBRARY_PATH
unset LD_LIBRARY_PATH
clear
if [ ! -f $DOCKER_MACHINE ] || [ ! -f $VBOXMANAGE ]; then
echo "Either VirtualBox or Docker Machine are not installed. Please re-run the Toolbox Installer and try again."
exit 1
fi
$VBOXMANAGE showvminfo $VM &> /dev/null
VM_EXISTS_CODE=$?
if [ $VM_EXISTS_CODE -eq 1 ]; then
echo "Creating Machine $VM..."
$DOCKER_MACHINE rm -f $VM &> /dev/null
rm -rf ~/.docker/machine/machines/$VM
$DOCKER_MACHINE create -d virtualbox --virtualbox-memory 2048 --virtualbox-disk-size 204800 $VM
else
echo "Machine $VM already exists in VirtualBox."
fi
VM_STATUS=$($DOCKER_MACHINE status $VM)
if [ "$VM_STATUS" != "Running" ]; then
echo "Starting machine $VM..."
$DOCKER_MACHINE start $VM
yes | $DOCKER_MACHINE regenerate-certs $VM
fi
echo "Setting environment variables for machine $VM..."
clear
cat << EOF
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
EOF
echo -e "${BLUE}docker${NC} is configured to use the ${GREEN}$VM${NC} machine with IP ${GREEN}$($DOCKER_MACHINE ip $VM)${NC}"
echo "For help getting started, check out the docs at https://docs.docker.com"
echo
eval $($DOCKER_MACHINE env $VM --shell=bash)
USER_SHELL=$(dscl /Search -read /Users/$USER UserShell | awk '{print $2}' | head -n 1)
if [[ $USER_SHELL == *"/bash"* ]] || [[ $USER_SHELL == *"/zsh"* ]] || [[ $USER_SHELL == *"/sh"* ]]; then
$USER_SHELL --login
else
$USER_SHELL
fi
Is it the correct file that I can put my '--registry-mirror' config to it? What should I do?
If you do a docker-machine create --help:
docker-machine create --help
Usage: docker-machine create [OPTIONS] [arg...]
Create a machine.
Run 'docker-machine create --driver name' to include the create flags for that driver in the help text.
Options:
...
--engine-insecure-registry [--engine-insecure-registry option --engine-insecure-registry option] Specify insecure registries to allow with the created en
gine
--engine-registry-mirror [--engine-registry-mirror option --engine-registry-mirror option] Specify registry mirrors to use
So you can modify your script to add one more parameter:
--engine-registry-mirror=...
However, since your 'default' docker-machine probably already exists (do a docker-machine ls), you might need to remove it first (docker-machine rm default: make sure you can easily recreate your images from your local Dockerfiles, and/or that you don't have data container that would need to be saved first)
Open C:\Users\<YourName>\.docker\daemon.json, edit the "registry-mirrors" entry in that file.
{"registry-mirrors":["https://registry.docker-cn.com"],"insecure-registries":[], "debug":true, "experimental": true}
I've been trying to install IDAS GE in a CentOS 7 VM on my machine through the UL2.0 RPMs(download link!) available in its catalogue page.
I followed the instructions on github, but I get stuck in starting the IoT as per section 3 of the Deployment section of the instructions. If I execute the init_iotagent.sh, where I inserted the local IP of the VM, I get the error:
log4cplus:ERROR No appenders could be found for logger (main).
log4cplus:ERROR Please initialize the log4cplus system properly.
HTTPFilter DESTRUCTOR 0
HTTPFilter DESTRUCTOR 0
Also, in the instructions for Starting IoTAgent as a Service, it's said that:
After installing iot-agent-base RPM an init.d script can be found in
this folder /usr/local/iot/init.d .
But this file is not there, leading me to believe that the IoTAgent wasn't installed properly from the RPMs provided.
Also, I can't find log files regarding IoTAgent, only the MongoDB has its log file at /usr/local/iot/mongodb-linux-x86_64-2.6.9/log/mongoc.log.
If anyone could help, it would be apreciated. Also, if more info is needed, please let me know.
Thank you
I recommend you to get the GitHub repository and build the RPMs from the source and then install it in your CentOS. Like is explained in the documentation:
NOTE: I changed the BUILD_TYPE to Release, so I created the Release dir.
GIT_VERSION and GIT_COMMIT are not the latest ones.
git clone https://github.com/telefonicaid/fiware-IoTAgent-Cplusplus.git
cd fiware....
mkdir -p build/Release
cd build/Release
cmake -DGIT_VERSION=20527 -DGIT_COMMIT=217023407f25ed258043cfc00a46b6c05fb0b52c -DMQTT=ON -DCMAKE_BUILD_TYPE=Release ../../
make install
make package
The packages will be in pack/Linux/RPM/
rpm -i iot-agent-base-xxxxxxx (xxxxxxx will be the numbers of the build)
rpm -i iot-agent-ul-xxxxxx (xxxxxxx will be the numbers of the build)
Once installed with RPMs the init.d file is in: /usr/local/iot/init.d/iotagent
This is the content of the file:
#!/bin/bash
# Copyright 2015 Telefonica Investigación y Desarrollo, S.A.U
#
# This file is part of fiware-IoTagent-Cplusplus (FI-WARE project).
#
# iotagent Start/Stop iotagent
#
# chkconfig: 2345 99 60
# description: iotagent
. /etc/rc.d/init.d/functions
PARAM=$1
INSTANCE=$2
USERNAME=iotagent
EXECUTABLE=/usr/local/iot/bin/iotagent
CONFIG_PATH=/usr/local/iot/config
iotagent_start()
{
local result=0
local instance=${1}
if [[ ! -x ${EXECUTABLE} ]]; then
printf "%s\n" "Fail - missing ${EXECUTABLE} executable"
exit 1
fi
if [[ -z ${instance} ]]; then
list_instances="${CONFIG_PATH}/iotagent_*.conf"
else
list_instances="${CONFIG_PATH}/iotagent_${instance}.conf"
fi
for instance_config in ${list_instances}
do
local NAME
NAME=${instance_config%.conf}
NAME=${NAME#*iotagent_}
source ${instance_config}
local IOTAGENT_PID_FILE="/var/run/iot/iotagent_${NAME}.pid"
printf "Starting iotagent ${NAME}..."
status -p ${IOTAGENT_PID_FILE} ${EXECUTABLE} &> /dev/null
if [[ ${?} -eq 0 ]]; then
printf "%s\n" " Already running, skipping $(success)"
continue
fi
# Load the environment
set -a
source ${instance_config}
# Mandatory parameters
IOTAGENT_OPTS=" ${IS_MANAGER} \
-n ${IOTAGENT_SERVER_NAME} \
-v ${IOTAGENT_LOG_LEVEL} \
-i ${IOTAGENT_SERVER_ADDRESS} \
-p ${IOTAGENT_SERVER_PORT} \
-d ${IOTAGENT_LIBRARY_DIR} \
-c ${IOTAGENT_CONFIG_FILE}"
su ${USERNAME} -c "LD_LIBRARY_PATH=\"${IOTAGENT_LIBRARY_DIR}\" \
${EXECUTABLE} ${IOTAGENT_OPTS} & echo \$! > ${IOTAGENT_PID_FILE}" &> /dev/null
sleep 2 # wait some time to leave iotagent start
local PID=$(cat ${IOTAGENT_PID_FILE})
local var_pid=$(ps -ef | grep ${PID} | grep -v grep)
if [[ -z "${var_pid}" ]]; then
printf "%s" "pidfile not found"
printf "%s\n" "$(failure)"
exit 1
else
printf "%s\n" "$(success)"
fi
done
return ${result}
}
iotagent_stop()
{
local result=0
local iotagent_instance=${1}
if [[ -z ${iotagent_instance} ]]; then
list_run_instances="/var/run/iot/iotagent_*.pid"
else
list_run_instances="/var/run/iot/iotagent_${iotagent_instance}.pid"
fi
if [[ $(ls -l ${list_run_instances} 2> /dev/null | wc -l) -eq 0 ]]; then
printf "%s\n" "There aren't any instance of IoTAgent ${iotagent_instance} running $(success)"
return 0
fi
for run_instance in ${list_run_instances}
do
local NAME
NAME=${run_instance%.pid}
NAME=${NAME#*iotagent_}
printf "%s" "Stopping IoTAgent ${NAME}..."
local RUN_PID=$(cat ${run_instance})
kill ${RUN_PID} &> /dev/null
local KILLED_PID=$(ps -ef | grep ${RUN_PID} | grep -v grep | awk '{print $2}')
if [[ -z ${KILLED_PID} ]]; then
printf "%s\n" "$(success)"
else
printf "%s\n" "$(failure)"
result=$((${result}+1))
fi
rm -f ${run_instance} &> /dev/null
done
return ${result}
}
iotagent_status()
{
local result=0
local iotagent_instance=${1}
if [[ -z ${iotagent_instance} ]]; then
list_run_instances="/var/run/iot/iotagent_*.pid"
else
list_run_instances="/var/run/iot/iotagent_${iotagent_instance}.pid"
fi
if [[ $(ls -l ${list_run_instances} 2> /dev/null | wc -l) -eq 0 ]]; then
printf "%s\n" "There aren't any instance of IoTAgent ${iotagent_instance} running."
return 1
fi
for run_instance in ${list_run_instances}
do
local NAME
NAME=${run_instance%.pid}
NAME=${NAME#*iotagent_}
printf "%s\n" "IoTAgent ${NAME} status..."
status -p ${run_instance} ${NODE_EXEC}
result=$((${result}+${?}))
done
return ${result}
}
case ${PARAM} in
'start')
iotagent_start ${INSTANCE}
;;
'stop')
iotagent_stop ${INSTANCE}
;;
'restart')
iotagent_stop ${INSTANCE}
iotagent_start ${INSTANCE}
;;
'status')
iotagent_status ${INSTANCE}
;;
esac
And the logs file are in /tmp/ :
IoTAgent-IoTPlatform.log
IoTAgent.log
IoTAgent-Manager.log
Hope this helps you.
I created a little script which must change engine for tables:
#!/bin/bash
echo "use test_1;" > query.sql
get_list () {
mysql -u teamcity -ppassword -B -N -e 'show tables like "%"' test_1 | xargs -I '{}' echo "alter table {} engine=innodb;" >> query.sql
}
get_list ;
mysql -u teamcity -ppassword < query.sql
But - how can I avoid use query.sql file? I make few attempts with "Here document" - but can't solve it...
For example - trying this:
#!/bin/bash
get_list () {
mysql -u teamcity -ppassword -B -N -e 'show tables like "%"' test_1 | xargs -I '{}' echo "alter table {} engine=innodb;"
}
a="$(get_list)"
b="use test_1;"
c="$b $a"
mysql -u teamcity -ppassword <<< $c
But it is not working...
Put everything in a function and pipe that:
#!/bin/bash
get_list () {
echo "use test_1;"
mysql -u teamcity -ppassword -B -N -e 'show tables like "%"' test_1 | xargs -I '{}' echo "alter table {} engine=innodb;"
}
get_list | mysql -u teamcity -ppassword