How to pull mysql database from heroku to local machine - mysql

Hi I have a ruby on rails app hosted on heroku and it is using mysql as database.
Now I have to take backup of the database to my local machine. But I am getting issues while taking backup.
For this I installed taps gem and I am using following commands for it
heroku pg:pull mysql2://username#hostname.cleardb.com/heroku_database local_database --app my_app
but it is giving error as !Your app has no databases.
Can any one guide me how to pull mysql database from heroku to local machine.
EDIT
I have used following syntax for the command
heroku pg:pull <REMOTE_SOURCE_DATABASE> <LOCAL_TARGET_DATABASE>
and for getting REMOTE_SOURCE_DATABASE I have used following command
heroku config:get DATABASE_URL --app my_app
I refer this link1 and link2 for more detailed heroku documentation.

The pg:pull command only works with Postgres databases in your Heroku app. But, you are using a third-party MySQL provider. Your database is hosted on the ClearDB servers and it's available to anyone with the right credentials, including both your app server on Heroku and your dev machine.
Even though there aren't special commands to pull the database, you don't need any - plain mysqldump should do.
mysqldump -h hostname.cleardb.com -u username heroku_database | mysql local_database

Running $heroku config | grep ^DATABASE will give you something like this:
DATABASE_URL: mysql2://username:password#host/dbname?reconnect=true`
From there you can construct your db dump command:
mysqldump -h host -p -u username dbname | mysql local_database
This will prompt you for the password which you received from the previous command. If you wanted to create a script that would automatically include the password from the heroku command you could do something like this:
mysqldump -u username --password=`heroku config | grep ^DATABASE | sed 's/.*[a-z0-9][a-z0-9]*:\([a-z][a-z0-9]*\).*/\1/'` -h host dbname | mysql cedric
In this way you can have a script that will import the database without requiring any user input but also does not expose the password to your database.

(IMPORTANT DISCLAIMER: You MUST have your database.yml configured correctly in order for this to work. I am not responsible for any data you lose as a result of running the below script.)
For Ruby on Rails users ... you could consider writing a Rake task like these db:clone tasks below.
I find myself using this script constantly to clone down from production to development. It's way easier than remembering the mysqldump syntax, much less all of the usernames and passwords involved ...
To clone from production to development:
rake db:clone:production
To clone from staging to development:
rake db:clone:staging
To clone from production to staging:
rake db:clone:production_to_staging
And here's the code enjoy (and be careful in setting up your database.yml):
namespace :db do
namespace :clone do
class << self
%w(development test staging production).each do |env|
define_method("#{env}_db") do
Rails.configuration.database_configuration[env]
end
end
end
def clone_db(from_db, to_db)
start_time = Time.now
puts "Cloning Remote DB...."
system("mysqldump -h#{from_db['host']} -u#{from_db['username']} -p#{from_db['password']} #{from_db['database']} | mysql #{to_db['database']} -u#{to_db['username']} -p#{to_db['password']}")
puts "Import Successful"
end_time = Time.now
puts "===================="
puts "Job Completed: #{end_time - start_time} Seconds"
end
task :staging => :environment do
clone_db(staging_db, development_db)
end
task :production => :environment do
clone_db(production_db, development_db)
end
task :production_to_staging => :environment do
clone_db(production_db, staging_db) if Rails.env.staging?
end
end
end

Related

How to import mysql data into postgresql in Heroku?(Django project)

Recently I need to import mysql data into postgres database in Heroku. Actcually it includes several steps:
convert mysql data to postgresql
import postgresql data to Heroku
After referring plenty of materials and testing several tools in github, finally I succeed. Here I want to share some of my experience and references.
Firstly, I list some tools for converting mysql database format into postgresql format.
mysql-postgresql-converter
: I finally use this tool and succeed. Dump MySQL database in PostgreSQL-compatible format
mysqldump -u username -p --compatible=postgresql databasename > outputfile.sql
then use the converter to transfer data into *.psql file. then load new dump into a fresh PostgreSQL database.
mysql2postgres : It is a tool which is introduced in Heroku Dev Center. Just refer here Migrating from MySQL to Postgres on Heroku. it is based on Ruby. However, as for me, I found some issues after I finish installation and cannot solve it.
You have already activated test-unit 2.5.5, but your Gemfile requires test-unit 3.2.3. #95
py-mysql2pgsql: Similar process with mysql2postgres above by editing a *.yml file for configuration. There is a nice reference table in README file called Data Type Conversion Legend, which compares different data type between MySQL and PostgreSQL. You can manually modify the data type.
** This website lists some other converting methods.
Some basic operations in PostgreSQL:
$sudo su - postgres
$createtedb testdb
$psql testdb
=# create user username password ' password ';
-- To change a password:
=# alter role username password ' password ';
=# create database databasename with encoding 'utf8';
How to list all database in postgres: PostgreSQL - SELECT Database
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+-------+-----------------------
postgres | postgres | UTF8 | C | C |
template0 | postgres | UTF8 | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
testdb | postgres | UTF8 | C | C |
(4 rows)
postgres-#
Now type the below command to connect/select a desired database, here we will connect to the testdb database:
postgres=# \c testdb;
psql (9.2.4)
Type "help" for help.
You are now connected to database "testdb" as user "postgres".
testdb=#
After you create your database, import the converted tables into psql. Kindly note that a database should be created before importing data.
$psql -h server -d databasename -U username -f data.sql
(sometimes a sudo -u postgres should be added before psql)
How to generate dump of psql using pg_dump: creating dump file
$sudo -u postgres pg_dump -Fc --no-acl --no-owner -h localhost -U postgres databasename > mydb.dump
The next step, how to import data into Heroku Postgres?
After previous steps, you may have import data into your local PostgresSQL or generate a pg_dump file. Here two methods will be introduced to transfer data to remote Heroku Postgres.
use pg_dump file.reference
Use the raw file URL in the pg:backups restore command:
$ heroku pg:backups:restore 'https://s3.amazonaws.com/me/items/3H0q/mydb.dump' DATABASE_URL
In this case, you should firstly upload the dump file to somewhere with an HTTP-accessible URL. The dev center in Heroku recommend using Amazon S3.
The DATABASE_URL represents the HEROKU_POSTGRESQL_COLOR_URL of the database you wish to restore to. For example, my database url is postgresql-globular-XXXXX.
use pg:push
pg:push will push data from your local psql database into remote Heroku Postgres database. The command looks like this:
$heroku pg:push mylocaldb DATABASE_URL --app sushi
This command will take the local database "mylocaldb" and push it to the database at DATABASE_URL on the app "sushi". Kindly note that the remote database must be empty before performing pg:push in order to prevent accidental data overwrites and loss.
Actually I use this pg:push method and succeed finally.
More information about Heroku Postgres can be found in official document of Heroku.
**others:
Viewing logs of your web application in Heroku: heroku logs --tail
How to deploy Python and Django Apps on Heroku?
How to write the Procfile of Django Apps?
A common Procfile of Django projects will look like this:
web: gunicorn yourprojectname.wsgi --log-file -
Here web is a single process type. What we need to modify is yourprojectname.wsgi. Just replace your project name in the prefix.
How to add Gunicorn to your application?
$ pip install gunicorn
$ pip freeze > requirements.txt
How to run command line in remote Heroku server?
You can execute the bash command in Heroku.
$heroku run bash
Running bash attached to terminal... up, run.1
~ $ ls
Then you can do some command such ls, cd just like in your local bash.
Also, you can use commands in this pattern to execute manage.py in remote Heroku:heroku run python manage.py runserver

Ruby: How do I return a MySQL query up an SSH tunnel?

//update: when feeding mysql passwords, -ppassword works. -p password does -not- work. Problem solved.//
We have hundreds of databases on our work servers, and I'd like to write a ruby script that automates the process of creating duplicate versions of them on my local machine for development purposes.
I've been trying to use net-ssh to create the tunnel, but (host, user, pass, etc. are censored for obvious reasons):
require 'net/ssh'
HOST = 'xxx'
USER = 'yyy'
PASS = 'ppp'
Net::SSH.start( HOST, USER, :password => PASS ) do|ssh|
puts "inside ssh tunnel"
puts ssh.exec!('ruby -v')
puts ssh.exec!('mysql -u zzz -p pswrd -h c3 will_il_raw -e "select * from deeds limit 1"')
end
results in the output:
inside ssh tunnel
ruby 1.8.7 (2010-08-16 patchlevel 302) [x86_64-linux]
followed by an indefinite hang. Interestingly, if I ssh into the same computer via the console, 'ruby -v' returns 1.9.3. If I enter that mysql query there, it successfully returns one line of the named table.
I assume the solution has something to do with port forwarding, but here my limited knowledge begins to fail me utterly.
The machine running the mysql server is not the same machine as I am accessing it from, which is not, in turn, the machine I am actually sitting at. I need to connect the dots and apparently have no idea how to go about this properly.
Any protips would be much appreciated.
In MySQL syntax, when feeding a password using -p[password], there is no space between -p and the password.
Because the script's version of the syntax had a space, the result of the query was a prompt requesting the password, which caused the hang on the far end of the SSH tunnel.

Executing MySQL commands in shell script?

I’m looking to create a deploy script that I can run from a terminal and it automatically deploys my site from a repository. The steps I’ve identified are:
Connect to remote server via SSH
Fetch latest version of site from remote repository
Run any SQL patches
Clean up and exit
I’ve placed the SSH connection and git pull commands in my shell file, but what I’m stuck with is MySQL with it being an (interactive?) shell itself. So in my file I have:
#!/bin/bash
# connect to remote server via SSH
ssh $SSH_USER#$SSH_HOST
# update code via Git
git pull origin $GIT_BRANCH
# connect to the database
mysql --user $MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DBNAME
# run any database patches
# disconnect from the database
# TODO
exit 0
As you can see, I’m connecting to the database, but not sure how to then execute any MySQL statements.
At the moment, I have a directory containing SQL patches in numerical order. So 1.sql, 2.sql, and so on. Then in my database, I have a table that simply records the last patch to be run. So I’d need to do a SELECT statement, read the last patch to be ran, and then run any neccesary patches.
How do I issue the SELECT statement to the mysql prompt in my shell script?
Then what would be the normal flow? Close the connection and re-open it, passing a patch file as the input? Or to run all required patches in one connection?
I assume I’ll be checking the last patch file, and doing a do loop for any patches in between?
Help here would be greatly appreciated.
Assuming you want to do all the business on the remote side:
ssh $SSH_USER#$SSH_HOST << END_SSH
git pull origin $GIT_BRANCH
mysql --user $MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DBNAME << END_SQL
<sql statements go here>
END_SQL
END_SSH
You could get the output from mysql using Perl or similar. This could be used to do your control flow.
Put your mysql commands into a file as you would enter them.
Then run as: mysql -u <user> -p -h <host> < file.sqlcommands.
You can also put queries on the mysql command line using '-e'. Put your 'select max(patch) from .' and read the output in your script.
cat *.sql | mysql --user $MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DBNAME

Moving/copying one remote database to another remote database

I'm using heroku to deploy my app. So far, I've been using a development version of my app, and seeding some data into it. I also have a production version set up on heroku.
Both apps are using a mysql database hosted with ClearDB to store the data.
I simply want to move the data from the development version to the production version.
Using MySql Workbench, I exported the dev data to a file and tried to import it to the prod db, but I got an access denied error because it tried to log into the dev db with production credentials.
The databases have identical table/row/column structure. How can I take the data from one and insert it into the other?
add Taps gem to your Gemfile:
gem 'taps'
you should be able to pull your development data to your local development environment with:
heroku db:pull --app your_development_app_name
then push it to production environment:
heroku db:push --app your_production_app_name
this will completely overwrite the db schema and data, so make sure you are careful with it. hope it helps!
_ryan
You can use a combination of the 'mysql' and 'mysqldump' command line
clients to "copy" one database's contents to the other. Here's an example of
how to do this:
mysqldump --single-transaction -u (old_database_username) -p -h (old_database_host) (database_name) | mysql -h (new_host) -u (new_user) -p -D (new_database)
with line breaks:
mysqldump --single-transaction -u (old_database_username)
-p -h (old_database_host) (database_name) | mysql -h (new_host)
-u (new_user) -p -D (new_database)

How to import a big database from Heroku to local mysql or sqlite3?

As per title I need to import, but PG backups is giving me strict Postgres SQL that doesn't work with MySQL, also with a non-specified encoding that I guess is UTF-16. Using db:pull takes ages and errors before finishing.
I'd appreciate any suggestion. Thanks.
Set up PostgreSQL locally, use PG backups to copy the data from Heroku to your local machine, then pg_restore to import it into your new local PostgreSQL. Then you can copy it from PostgreSQL to MySQL or SQLite locally without having to worry about timeouts. Or, since you'd have a functional PostgreSQL installation after that, just start developing on top of PostgreSQL so that your development stack better matches your deployment stack; developing and deploying on the same database is a good idea.
You're probably getting binary dumps (i.e. pg_dump -Fc) from Heroku, that would explain why the dump looks like some sort of UTF-16 nonsense.
You can use the pgbackups addon to export the database dump
$ heroku addons:add pgbackups # To install the addon
$ curl -o latest.dump `heroku pgbackups:url` # To download a dump
Heroku has instructions on how to do this: https://devcenter.heroku.com/articles/heroku-postgres-import-export#restore-to-local-database, which boil down to:
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
where myuser is the current user and mydb is the current database.
If you're using Postgres.app, it's pretty trivial to copy your production database locally. You can leave out -U myuser unless you have configured it otherwise, and create a database by running $ psql -h localhost and then CREATE DATABASE your_database_name; (from the Postgres.app documentation. Then run the above command and you're set.
Follow these 4 simple steps in your terminal(Heroku Dev Center):
Install the Heroku Backup tool:
$ heroku addons:add pgbackups
Start using it:
$ heroku pgbackups:capture
Download the remote db on Heroku (to your local machine) using curl:
$ curl -o latest.dump 'heroku pg:backups public-url'
Load it*:
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U YOUR_USERNAME -d DATABASE_NAME latest.dump
get your username and choose the desired database from your config/database.yml file.
DATABASE_NAME can be your development/test/production db (Ex. mydb_development)
That's it!
UPDATE: It is updated to the new interface