I am able to SSH into amazon instance using SSH Sampler plugin after SSH'ing into instance I want to run Mysql queries to edit database entries.
I'm able to run commands like cd, ls after SSH'ing but unable to run Mysql queries.
Mysql queries work fine on running it on terminal.
Here is the snapshot of SSH Sampler plugin:
Option 1: use mysql execute like:
mysql -u root -pROOT_PASSWORD -e "YOUR_QUERY_HERE"
Example:
mysql -u root -psecret -e "use mysql; select name, url from help_topic limit 5;"
Option 2: use JDBC Request Sampler.
Given you have:
MySQL Connector/J library in JMeter classpath
Have port 3306 (or whatever) open in firewall and Amazon Security Group
You should be able to run arbitrary queries using JDBC Request Sampler, if you need to load test MySQL instance - it will be the best way to proceed.
See The Real Secret to Building a Database Test Plan With JMeter article for detailed instructions.
Related
I took over a website at a less-than-optimal hoster with no backups yet.
I do have an FTP-access and I know the database access parameters of the installed web-app to the MySQL server, but I don't have access to the MySQL interface or the underlying server.
I would like to do an automated backup to a Linux server under my control.
I can download all data via FTP, zip it and store it on a backed up storage.
How to do this for the database?
As an initial solution I installed phpMyAdmin and did a manual backup, but I would like to automate this process.
You can use mysqldump to back up a remote MySQL database.
Suppose your MySQL database is on a host called "dbhost". You can reach that host over the network from your new Linux host.
Run this command on your new Linux host:
$ mysqldump --single-transaction --all-databases --host dbhost > datadump.sql
(You might also need to add the --user and --password options.)
You can automate any command you can run at the command-line. Put it in a shell script. Then you an invoke the script for example from cron.
I'm trying to configure hive-site.xml to have MySQL outside of the local MySQL on EMR. How can I modify an existing cluster configuration to add hive-site.xml from S3?
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-dev-create-metastore-outside.html
I'm not sure what you mean by "add hive-site.xml from S3". If you're just looking to get the file off of S3 and into your conf directory, you can do that with the aws-cli while logged into your cluster,
aws s3 cp s3://path/to/hive-site.xml ~/conf
More detailed instructions on migrating an existing EMR cluster's Hive MetaStore to an external service like RDS can be found below
--
Setting up an existing EMR cluster to look at an outside MySQL database is very easy. First, you'll need to dump your MySQL database that's running on your Master node to keep your existing schema information. Assuming you've a large amount of ephemeral storage and your database socket is located at /var/lib/mysql/mysql.sock:
mysqldump -S /var/lig/mysql/mysql.sock hive > /media/ephemeral0/backup.sql
Then you'll need to import this into your outside MySQL instance. If this is in RDS, you'll first need to create the hive database and then import your data into it:
mysql -h rds_host -P 3306 -u rds_master_user -prds_password mysql -e "create database hive"
and,
mysql -h rds_host -P 3306 -u rds_master_user -prds_password hive < /media/ephemeral0/backup.sql
Next up, you'll need to create a user for hive to use. Log into your outside MySQL instance and execute the following statement (with a better username and password):
grant all privileges on hive.* to 'some_hive_user'#'%' identified by 'some_password'; flush privileges;
Lastly, create/make the same changes to hive-site.xml as outlined in the documentation you cited (filling in the proper host, user, and password information) and restart your MetaStore. To restart your MetaStore, kill the already running MetaStore process and start a new one.
ps aux | grep MetaStore
kill pid
hive --service metastore&
If you are in EMR 3.x, you can just use the method in the link you provide(using bootstrap action).
If you are in ERM 4.x+, then that bootstrap action is not available. You could
either add the custom properies thru EMR --configuration with a xxx.json file. The benefit is straightforward. The con is all the config properties you added this way will be on the aws web console which is not ideal if you have things like metastore database credentials there since you are using external metastore.
or you add a Step after cluster is up to overwrite your hive.xml from S3, then another Step to execute sudo reload hive-server2 to restart hive server to get the new config.
I have access to a MySQL database through ssh,
Could someone direct me to a MySQL-python code that will let me do this?
I need to save my query results on my local WINDOWS computer,
Thanks,
You can use SSH port forwarding to do this.. in fact first google hit looks to walk you through this exact thing:
http://www.howtogeek.com/howto/ubuntu/access-your-mysql-server-remotely-over-ssh/
And since you're on windows, translate that to using PuTTY:
https://intranet.cs.hku.hk/csintranet/contents/technical/howto/putty-portforward.jsp
You'll then connect to localhost:3306 with your python script, SSH will forward that over to the other machine and you'll end up connecting to the remote mysql instance.
You need to open up an SSH Tunnel to your sql server and then you can run paramiko to connect locally to the port you are using locally. This is done quite easily in *nix systems and I am sure you can download ssh command line too for windows. Try putty or plink, see here. What I do is I run a shell script like so, then I execute my paramiko python script, then I kill the
ssh -N remote_server#54.221.226.240 -i ~/.ssh/my_ssh_key.pem -L 5433:localhost:5432
python paramiko_connect.py
kill pkill -f my_ssh_key.pem # kill using the pattern,
#see ''ps aux | grep my_ssh_key.pem'' to see what it will kill
-N means don't execute any commands, -L is the local port to tunnel from, followed by the remotes server port, assuming you are connected to that server already.
Works like a charm for me for my postgres server & I did try it on mysql too.
How can I import existing MySQL database into Amazon RDS?
I found this page on the AWS docs which explains how to use mysqldump and pipe it into an RDS instance.
Here's their example code (use in command line/shell/ssh):
mysqldump acme | mysql --host=hostname --user=username --password acme
where acme is the database you're migrating over, and hostname/username are those from your RDS instance.
You can connect to RDS as if it were a regular mysql server, just make sure to add your EC2 IPs to your security groups per this forum posting.
I had to include the password for the local mysqldump, so my command ended up looking more like this:
mysqldump --password=local_mysql_pass acme | mysql --host=hostname --user=username --password acme
FWIW, I just completed moving my databases over. I used this reference for mysql commands like creating users and granting permissions.
Hope this helps!
There are two ways to import data :
mysqldump : If you data size is less than 1GB, you can directly make use of mysqldump command and import your data to RDS.
mysqlimport : If your data size is more than 1GB or in any other format, you can compress the data into flat files and upload the data using sqlimport command.
I'm a big fan of the SqlYog tool. It lets you connect to your source and target databases and sync schema and/or data. I've also used SQLWave, but switched to SqlYog. Been so long since I made the switch that I can't remember exactly why I switched. Anyway, that's my two cents. I know some will object to my suggestion of Windows GUI tools for MySQL. I actually like the SqlYog product so much that I run it from Wine (works flawlessly from Wine on Ubuntu for me).
This blog might be helpful.
A quick summary of a GoSquared Engineering post:
Configuration + Booting
Select a maintenance window and backup window when the instance will be at lowest load
Choose Multi-AZ or not (highly recommended for auto-failover and maintenance)
Boot your RDS instance
Configure security groups so your apps etc can access the new instance
Data migration + preparation
Enable binlogging if you haven't already
Run mysqldump --single-transaction --master-data=2 -C -q dbname -u username -p > backup.sql on the old instance to take a dump of the current data
Run mysql -u username -p -h RDS_endpoint DB_name < backup.sql to import the data into your RDS instance (this may take a while depending on your DB size)
In the meantime, your current production instance is still serving queries - this is where the master-data=2 and binlogging comes in
In your backup.sql file, you'll have a line at the top that looks like CHANGE MASTER TO MASTER_LOG_FILE=’mysql-bin.000003′, MASTER_LOG_POS=350789121;
Get the diff since backup.sql as an SQL file mysqlbinlog /var/log/mysql/mysql-bin.000003 --start-position=350789121 --base64-output=NEVER > output.sql
Run those queries on your RDS instance to update it cat output.sql | mysql -h RDS_endpoint -u username -p DB_name
Get the new log position by finding end_log_pos at the end of the latest output.sql file.
Get the diff since the last output.sql (like step 6) and repeat steps 7 + 8.
The actual migration
Have all your apps ready to deploy quickly with the new RDS instance
Get the latest end_log_pos from output.sql
Run FLUSH TABLES WITH READ LOCK; on the old instance to stop all writes
Start deploying your apps with the new RDS instance
Run steps 6-8 from above to update the RDS instance with the last queries to the old server
Conclusion
Using this method, you'll have a small amount of time (depending on how long it takes to deploy your apps + how many writes your MySQL instance serves - probably only a minute or two) with writes being rejected from your old server, but you will have a consistent migration with no read downtime.
A full and detailed post explaining how we (GoSquared) migrated to RDS with minimal downtime (including error debugging) is available here: https://engineering.gosquared.com/migrating-mysql-to-amazon-rds.
I am completely agree with #SanketDangi.
There are two ways of doing this one way is as suggested using either mysqldump or mysqlimport.
I have seen cases where it creates problem while restoring data on cloud gets corrupt.
However importing applications on cloud has became much easier now a days. You try uploading your DB server on to public cloud through ravello.
You can import your database server itself on Amazon using ravello.
Disclosure: I work for ravello.
Simplest example:
# export local db to sql file:
mysqldump -uroot -p —-databases qwe_db > qwe_db.sql
# Now you can edit qwe_db.sql file and change db name at top if you want
# import sql file to AWS RDS:
mysql --host=proddb.cfrnxxxxxxx.eu-central-1.rds.amazonaws.com --port=3306 --user=someuser -p qwe_db < qwe_db.sql
AWS RDS Customer data Import guide for Mysql is available here : http://aws.amazon.com/articles/2933
Create flat files containing the data to be loaded
Stop any applications accessing the target DB Instance
Create a DB Snapshot
Disable Amazon RDS automated backups
Load the data using mysqlimport
Enable automated backups again
If you are using the terminal this is what worked for me:
mysqldump -u local_username -plocal_password local_db_name | mysql -h myRDS-at-amazon.rds.amazonaws.com -u rds-username -prds_password_xxxxx remote_db_name
and then i used MYSQL WorkBench (free download) to check it was working because the command line was static after pressing submit, i could have probably put -v at end to see it's output
Note: there is no space after -p
Here are the steps which i have done and had sucess.
Take the MySQLdump of the needed database.
mysqldump -u username -p databasename --single-transaction --quick --lock-tables=false >databasename-backup-$(date +%F).sql
( Dont forget to replace the username as root – most of the times, and databasename -> Db name of database which you are going to migrate to RDS )
Once prompted, enter your password.
Once done, login to the RDS Instance from your MySQL server ( Make sure the security groups are configured to allow the connection from Ec2 to RDS )
mysql -h hostaddress -P 3306 -u rdsusername -p
( Dont forget to replace hostaddress with the address of your RDS Instance and rdsusernmae with username for your RDS Instance, when prompted give the password too )
You find that hostaddress under – Connectivity & security -> Endpoint & port under RDS Database From AWS Console.
Once logged in, create the database using MySQL commands :
create database databasename;
\q
Once Database is created in RDS, Import the SQL file created in Step 1 :
mysql -h hostaddress -u rdsusername -p databasename < backupfile.sql
This should import the SQL file to RDS and restore the contents into the new database.
Reference from : https://k9webops.com/blog/migrate-an-existing-database-on-mysql-mariadb-to-an-already-running-rds-instance-on-the-aws/
Can I run mysql delete row remotely from other machine?
Something alike mysqldump which run locally dumping data on other remote machine.
try:
mysql -u yourdblogin -pyourdbpassword -h yourdbdomain.yourdomain.com yourdb -e "delete from table where x = y"
not sure if -e is the correct argument, but the though is sound.
Only if you have TCP connections configured on the MySQL server, and you have a username and password configured that will allow you to remotely connect.
If you check the mysqldump documentation, you'll find the -host option allows you to connect to a remote system.
You can try something like this:
mysql -u yourdblogin -pyourdbpassword -h yourdbdomain.yourdomain.com yourdb
Sorry if I misunderstood your question.
The specific way of doing this is to connect to the remote machine using the mysql prompt and then execute queries against it (queries which cause a row to be deleted.)
# mysql -u username -p -h remote.location.com
mysql> USE `database_name`;
mysql> DELETE FROM `table` WHERE id=1234;
The DELETE FROM syntax is SQL syntax, and the WHERE part of it allows you to specific exactly what you want to be deleted.
Also, you most likely won't be able to log in as 'root' if you're used to doing so, as root by default can't connect remotely. (You can enable this, but it's recommended instead to make another admin account and to use that instead).
Note that code may also remotely delete rows, however it depends on the language and implementation as to how it's done.
References:
MySQL DELETE Syntax
MySQL SELECT Syntax (includes details about WHERE)
You can actually use any MySQL client application to connect to remote servers, not only the supplied command-line tool.
If the server allows TCP client connections, there are no differences between running scripts locally or remotely.
If the MySQL server happens to be also a web server, you can use a software like PHPMyAdmin to do it. Once it runs locally (on the same machine as the MySQL server, not your machine), you can work around the problem of not accepting external TCP connections, if the server was configured that way.