I'm customizing a phpUnderControl build file to use the symfony propel:build-sql functionality to generate .sql files to be loaded for every build (not every test). What I am running into is that the resulting .sql files expect the databases to exist, but as part of the build process, I would like to drop and recreate every database found in the schema.yml file to ensure a clean testing environment.
My question is, is it possible to configure propel:build-sql to behave as though I passed the --add-drop-database mysqldump flag?
No it's not possible. It's your job to drop/create the database before you run symfony/propel commands.
Related
Disclaimer: I'm a total green horn.
I'm working with PhpStorm on Windows, which offers a convenient way of creating and managing databases during development. Unfortunately, now that I want to push to production, uploading the database to Cloadways (Digital Ocean Server) doesn't seem that simple.
Cloudways' Database Manager has an import function, that requires .gz (gzip) files. gzip files can not be created from directories, but on Windows mysql creates directories for each database and fills them with table files (.ibd).
I've read that you can compress directories into .tar files first and then gzip them (database.tar.gz) and that's what I tried. But when I try to import them with the database manager it only shows this:
Is there any way to do this?
Here's what you need to migrate your data to another MySQL server from your development machine: A text file named whatever.sql containing the definitions and contents of your tables (and views and stored functions and all your other database objects). This kind of text file is often called a mysqldump file. You'll find these files contain a whole mess of SQL statements, mostly CREATE TABLE and INSERT. They also contain some lines like
/*!40103 SET #OLD_TIME_ZONE=##TIME_ZONE */;
These look like comments, but when you load the file they're handled as SQL statements, so leave them alone.
You can use gzip(1) on your file to make it smaller, and Cloudways will handle it correctly. Things will work either with or without gzip.
How to export
With PhpStorm, open up the database panel and right-click on your database name (not the server name itself, but your application's database). You'll see a menu item Export with mysqldump. Click it. You can keep the default checkbox settings.
Then give a filename for your output and run the export.
I use PhpStorm on linux, so I'm not totally sure this export works on Windows. If it doesn't, download Ansgar Becker's free and excellent Windows MySQL client program called HeidiSQL. Right click on the database name then choose Export Database as SQL. Check the Create Tables box and choose Insert from the data pulldown.
How to import to production
Log into the production MySQL server.
Create the database and choose it. Cloudways looks like it does that for you. If not, issue these SQL statements.
CREATE DATABASE myDatabaseName;
USE myDatabaseName;
Use an appropriate tool to run all the SQL in your whatever.sql file. Cloudways looks like it does that for you too. If not, this command line, or something similar, might work.
mysql --user=yourMySQLUserName --password=secret\
--database=myDatabaseName\
--host=cloudways.example.com < whatever.sql
Your migration will be complete.
Extra bonus: If your whatever.sql file contains the initial state of your production database, you can put it into git (or other source control) and use it whenever you deploy a new instance of your software package.
Don't try to copy those .ibd files and other files managed by MySQL to another machine. If you do, you'll be sorry.
The problem is, that one MYI and one MYD file from MySQL database has been accidentally deleted. The only file left intact is FRM one. Only one table from the whole database is damaged that way, all other tables are OK and the database works generally fine, except the table with deleted files, which is obviously inaccessible.
There's a full database dump in pure SQL format available.
The question is, how do I re-create these files and table in safe and proper manner?
My first idea was to extract the full create table command from the dump and run it on live database. It's not so easy, as the whole dump file has over 10GB, so any operations within its content are really pain in . Yes, I know about sed and know how to use it - but I consider it the last option to choose.
Second and current idea is to create copy of this database on independent server, make a dump of the table in question and then use resulting SQL file to create the table again on the production server. I'm not quite experienced with MySQL administration tasks (well, just basic ones), but for me this option seems to be safe and reasonable.
Will the second option work as I expect?
Is it the best option, or are there any more recommendable solutions?
Thank you in advance for your help.
The simplest solution is to copy the table you deleted. There's a chance mysqld still has an open file handle to the data files you deleted. On UNIX/Linux/OS X, a file isn't truly deleted while some process still has an open file handle to it.
So you might be able to do this:
mysql> CREATE TABLE mytable_copy LIKE mytable;
mysql> INSERT INTO mytable_copy SELECT * FROM mytable;
If you've restarted MySQL Server since you deleted the files, this won't work. If the server has closed its file handle to the data file, this won't work. If you're on Windows, I have no idea.
The next simplest solution is to restore your existing 10GB dump file to a temporary instance of MySQL Server, as you said. I'd use MySQL Sandbox but some people would use a virtual machine, or if you're using an AWS environment, launch a spot EC2 instance or a small RDS instance.
Then dump just the table you need:
mysqldump -h tempserver mydatabase mytable > mytable.sql
Then restore it to your real server.
mysql -h realserver mydatabase < mytable.sql
(I'm omitting the user & password options, I prefer to put those in .my.cnf anyway)
I have a SQL script which selects data from DB and stores it to files. I am unable to create a directory to store these files.
I have shell script that loads the SQL file. Shell and the SQL are on separate server than MySQL db. I would prefer to create this directory using SQL as I want to avoid ssh.
Any suggestions? Surprisingly I couldn't find anything on Google.
I will assume that you're using mysql, according to your tags. You could do it with a Microsoft SQL Server or Oracle database but unfortunately, at the moment, there is no solution to create a directory from MySQL.
Some will guide you with a workaround based on the creation of a data directory, I wouldn't recommand this, as it could lead to performances issues in the future, or worst.
The best solution would be to use a script (java, vbscript, SSH, batch, ...). Again, you won't be able to start this script within your SQL query easily. I know that's no good news, but it is important not to lead you on the wrong direction.
I would suggest to reverse your thinking, and start your SQL query from a script (again, any language you're used to).
I couldn't find any other way other than opening ssh session to the target box.
Open ssh session
Create directory
close ssh session
Load sql file using shell
The sql adds the generated files to the directory created in step 2.
ssh -t $USER#$HOST <<-SSH-END;
mkdir -p "dir/path";
exit;
SSH-END
Sharing just in case someone else needs to do the same.
I use Java and MySQL with Appserv. I have a database file named "modb.sql" (in a specific directory relative to my program's location), and I frequently need to drop the old database, create a new database (with the same name every time, inside phpMyAdmin) and import the modb.sql file to the new DB.
Is there a way to automate this process and include it with the program or inside a setup file? Instead of doing it manually by me or the user.
I can use another MySQL database manager or C# instead of Java if that would allow the process to be automatic.
You can do all of this from the Windows command line or through a Scheduled Task. I don't have Appserv installed, so I can't give you all of the exact file path locations which may vary based on your installation location. You could technically do it from a Java application as well, but that's a lot of overhead and not really automating it like a schedule task will.
You'll basically be calling the mysql.exe command directly from your schedule task -- or from a batch file.
First, let's create a new SQL file, perhaps called DropAndCreateMoDb.sql. Put your commands to drop (DROP DATABASE ...) and recreate the database here. Of course the exact drop command depends on what you call your database and the create command depends heavily on what structure is created by modb.sql; you'll also create any permissions that you need to here. This automates the part about dropping and recreating.
Write a batch file. You don't really need to do this, you could call MySQL twice from the schedule task, but we're trying to do this the right way. This is untested, and obviously you'll want to substitute the proper paths and MySQL username/password -- I suggest creating a maintenance account for this so your full credentials aren't in the batch file; the usual disclaimers apply about properly securing your account), but perhaps something like:
#ECHO OFF
C:\Program Files\Appserv\MySQL.exe -u foo -p bar < C:\data\DropAndCreateMoDb.sql
C:\Program Files\Appserv\MySQL.exe -u foo -p bar < C:\data\modb.sql
Obviously it doesn't catch any errors and your username and password are in the clear here, but as long as this is on your local development machine for development purposes and you know and understand the risks, it will work.
At this point, you can double-click the .bat file and it should drop and recreate your database. To finish it off and fully automate it, you'll add a scheduled task. Go to the Scheduled Tasks control panel and add a new task. Tell it the path to your new DropAndRecreate.bat or whatever you decide to call it, tell Windows when you want the task to run, and now it's fully automated.
There are lots of variables here that make the specifics of the implementation very dependent on your exact configuration and so on. Make sure you understand what each step actually does instead of just copying and pasting.
Sorry for a noob question regarding MySQL. I downloaded FlightStats to learn about mysql but I can't figure out how to register it with my localhost mysql db. I know in MS SQL you can simply register any sql db using sql studio. I tried to google but come up with no result. Perhaps, my search phrase is wrong. I'm searching with "how to register a mysql database, register a mysql database...etc.". How do you register or setup an database from existing database like FlightStats? I'm using DBVisualizer. Is there a way in dbVis that I'm not aware of to regsiter a database?
Thanks
edit: sorry for the bad wording. I found this. I have the .myd, .myi and .frm and I want to get it to restore(?) with my local mysql instance. I look at all the answers but I'm still confuse as how you restore the database from those 3 files.
A little background first. The FlightStats download page linked to in the original question appears to provide zipped tarballs of the binary table storage files from the MySQL data directory. Given that this is considered a viable means of distribution, and combined with the use of MERGE tables, I would surmise that this tarball contains a bunch of MyISAM data files (.myi, .myd). Jack's edit confirms that this is the situation.
This is an atypical means of distributing a MySQL data set, although not at all uncommon when backing up MyISAM storage, and probably not all that unheard of for moving large data sets around; it likely works out considerably more space-efficient than a corresponding dump file. Of course, in SQL Server land, it's pretty common to attach database files into an instance.
Broadly speaking, you'd recover the database as follows:
Locate the MySQL data directory; typically /var/mysql or similar
Create a new directory with the desired database name e.g. flightdata
Extract the .myi, .myd and other files from the tarball into this directory
Make sure the entire directory is owned by the user MySQL runs as (usually mysql) - use chmod -R to make sure you get everything
Open a MySQL console
USE <database-name>
SHOW TABLES
You should see some tables listed. In addition, the downloads page linked includes a couple of SQL scripts, which contain SQL commands that you need to run against your database once it's in place. These will cause the merge definitions and table indexes to be rebuilt. You can pipe these into the command-line client, e.g. mysql -u<username> -p<password> <database-name> < <sql-file>.
It may be a good idea to shut down the MySQL server while you're doing this; use e.g. /etc/init.d/mysql stop or similar, and restart once the files are extracted in place.
There's generally a way to import sql files using a GUI database tool. I'm not familiar with DBVisualizer, but as long as you have a MySQL command line client installed you can do it there as well. It's pretty easy:
Create a blank schema. You can do this in your GUI tool or on the command line client. Just use CREATE DATABASE flightstats;, or whatever name you want.
Use the following command line syntax to import/run an sql file on the new schema: mysql -u <username> -p flightstats < /path/to/file.sql
The -p option prompts for a password. I generally set up the database using step 1 as the root user, then GRANT some permissions on it to a new user id, then use that user id to run the SQL file.
This process is pretty much what a GUI tool will do in the background.
Registering a database? dont know what that means however mysql gui tools can help you creating a database. Have a look at it or better you download phpmyadmin.
Google WAMP for Windows.
Google MAMP for Mac.
Google LAMP for Linux.
Any questions?