We are testing the Google Cloud SQL service and we have a problem when importing our database.
Imported file include this command:
DROP DATABASE $my_db
But Google Cloud SQL dashboard still shows a storage size as if nothing was deleted.
Is it the right method for deleting a database without removing the supporting VM?
I don't want to delete the whole VM as instance names seem to be reserved for two months.
InnoDB will hold onto the data that it has allocated itself, so when you create a database and delete it, the storage usage remains the same. This means you can't reduce the maximum storage you have used, but InnoDB will recycle the space it has reserved if you import new data.
Related
I have a server with a database and a table. Upon insert, I want to be able copy the data to another table on another database on another server. I have read intensively on the subject, almost all the articles suggest I use tables with federated engines together with triggers, some others suggest I use replication feature on mysql databases, unfortunately the two servers or databases I am working with have some limitations.
There is no replication feature on either of them, it have been disabled and I can't get them to enable it.
Both databases can't create tables with federated engines.
The client does not want a server script file such as php file checking consistently for new data and sending over to populate the remote database.
However, each server have whitelisted each others IP and communication through a php file is possible between the two servers on either sides. I am able to create triggers on both tables as well. I want to write to the the remote database when ever there is a new data to the local database. There are no foreign key constraints on any of the tables. There is no read latency requirements from when a row is being updated to when it appears on the other server. The updated table does not have a updated timestamp/datetime column. PHP version: 7.1.14. Maria DB Version 5.5.62-0ubuntu0.14.04.1
Question:
Is there anyway to copy data before or after insert using triggers when the tables are not Federated engine tables(They use other engines, eg. INNO DB) ie... Can triggers have a connection to another server?
Is there any other way to copy data to another server without using a federated engine table or replication or using a script that checks for new data every minute or at regular intervals?
I want to create connector (something like Debezium in Kafka-Connect) to reflect every change in MySQL source database in BigQuery tables.
There is one problem - source database is dropped every 10mins and re-created - some of rows are the same, some are updated and some are totally new. So I cannot do it via Debezium, because every 10min I would have all records in Kafka.
I want migrate to BQ tables just new or updated values. The mechanism to "copy" the whole source database but to deduplicate records (which not be excatly the same because this will be a new database). So for example create hash from every record and check - if hash is already in BQ - pass and if there is not add it.
I think it should be this:
Best effort de-duplication
but how to create the whole pipeline with MySQL as a source.
Cloud Data Fusion Replication lets you replicate your data continuously and in real time from operational data stores, such as SQL Server and MySQL, into BigQuery.
To use Replication, you create a new instance of Cloud Data Fusion and add the Replication app.
In short you do below
Set up your MySQL database to enable replication.
Create and run a Cloud Data Fusion Replication pipeline.
View the results in BigQuery.
Yu can see more at Replicating data from MySQL to BigQuery
I'm just looking for a clarification as the documentation state that an empty, just created, SQL Instance should only take 250Mb approximately.
Quoting from documentation about anything I could find on storage space:
MySQL Second Generation instances: The most recent 7 automated backups, and all on-demand backups, are retained. They are charged at the backup storage rate. Binary logs use storage space (not backup space), and are charged as storage.
For the purpose of this test, binary logging is disabled.
MySQL Second Generation [...]: Storage is calculated based on the amount of storage you have provisioned for your instance. Storage for backups is charged by how much space your backups are using. Storage is charged whether your instance is on or off.
Again, freshly created instance. It should be 0 storage space.
A newly created database uses about 270MB of space for system tables and InnoDB logs. If you create a per-use instance but do not use it, you are still charged for the storage cost.
This is where I got the idea about "250 MB" as initial storage space.
As you can see however, a newly created database takes around 1.2GB.
I'd like some clarification on it if someone has any.
Sources:
https://cloud.google.com/sql/faq#storage_limits
https://cloud.google.com/sql/docs/mysql/pricing#2nd-gen-storage-networking-prices
I've been looking into this and the thing that you should take into account is that, the information you quoted about Cloud SQL MySQL empty instances occupying around 270MB is for first generation instances, not for second generation ones. I think that answers your question.
At first I interpreted that the same way as you did, but the only points where 270MB empty instances are specified are here and here which are within the "MySQL First Generation Pricing" category on the right.
Hope this helps.
I have a Google Cloud SQL MySQL 2nd Gen 5.7 instance with 70GB of unused storage. I want to reduce the unused storage of my instance as this is one of the major hosting costs.
The only way that I'm seeing as a possibility is dumping all the database to a new Google Cloud SQL instance created with the right storage capacity. However this approach has several pitfalls:
It would require a lot of time to dump and load the database (this would require severe downtime in the websites that are using the db)
I would have to change all the websites that use that database because the database's IP would change
What alternatives do I have?
As it is indicated in the official documentation, it is not possible to reduce the storage capacity of a Cloud SQL instance. You can give more capacity to an instance if needed, but the change is permanent and cannot be rolled back.
The best option for you right now would be to create a SQL dump file and export your current data into it. Then, in a new Cloud SQL instance with the desired capacity, import the SQL dump file so that all your data is stored in the new instance. To reduce time and costs for this process, please follow these best practices, including the use of the appropriate flags, compressing your data and other tips available for faster and less expensive imports and exports.
However, the possibility of reducing the capacity of a Cloud SQL instance has been requested as a new feature and is currently being worked on. Head to the link and click on the star icon on the top left corner to get email notifications on any progress made on this request.
There are no alternatives to reduce Instance size(Storage capacity) except one way which you mentioned in your question.
So, the way is first, export data from your old db instance:
Then, import the data into your new db instance which you newly created with reduced Instance size:
And as you know, it's impossible to reduce Instance size for all PostgreSQL, MySQL and SQL Server on Cloud SQL and only increasing Instance size is possible after creating your db instance.
Given the number of different storage engines MySQL supports, I'm just a bit surprised I can't find one that uses SQLite files.
Does such a thing exist?
The use case is that I have data in multiple MySQL data bases that I want to process and export as a SQLite data base that other tools can then process.
The current proposed solution is to use a scratch MySQL DB server to access the other instance using the FEDERATED Storage Engine and to access and populate a local SQLite db file using another Storage Engine.
The constraint is that the cost/benefit trade off can barely justify the proposed workflow (and can't justify writing any code beyond the SQL that reads and processes the federated tables) so I'm strictly limited to "works robustly out of the box" solutions.