I decided to play around with Google Could SQL and I setup a test sql instance, loaded it with some data and then setup replication on it in the google dev console. I did my testing and found out it all works great, the master/slave setup works as it should and my little POC was a success. So now I want to delete the POC sql instances but that's not going so well.
I deleted the replica instance fine (aka the 'slave') but for some reason the master instance still thinks there is a slave and therefore will not let me delete it. For example I run the following command in the gclound shell:
gcloud sql instances delete MY-INSTANCE-NAME
I get the following message:
ERROR: (gcloud.sql.instances.delete) The requested operation is not valid for a replication master instance.
This screenshot also shows that in the google dev console it clearly thinks there are no replicas attached to this instance (because I deleted them) but when I run:
gcloud sql instances describe MY-INSTANCE-NAME
It shows that there is a replica name still attached to the instance.
Any ideas on how to delete this for good? Kinda lame to keep on paying for this when it was just a POC that I want to delete (glad I didn't pick a high memory machine!)
Issue was on Google's side and they fixed it. Here were the sequence of events that led to the issue happening:
1) Change master's tier
2) Promote replica to master while the master tier change is in progress
Just had the same problem using GCloud. Deleting the failover replica first and then the master instance worked for me.
Related
I have a small instance running in GCE, had some troubles with the MongoDb so after some tries decided to reset the instance. But... it didn't seem to come back online. So i stopped the instance and restarted it.
It is an Bitnami MEAN stack which starts apache and stuff at startup.
But... i can't reach the instance! No SCP, no SSH, no webservice running. When i try to connect via SSH (in GCE) it times out, cant make connection on port 22. In the information it says 'The instance is booting up and sshd is not running yet', which is possible of course.... But i cant reach the instance in no possible manner not even after an hour wait :) Not sure what's happening if i cant connect to it somehow :(
There is some activity in the console... some CPU usage, mostly 0%, some incomming traffic but no outgoing...
I hope someone can give me a hint here!
Update 1
After the helpfull tip form Serhii... if found this in the logs...
Booting from Hard Disk 0...
[ 0.872447] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
/dev/sda1 contains a file system with errors, check forced.
/dev/sda1: Inodes that were part of a corrupted orphan linked list found.
/dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
fsck exited with status code 4
The root filesystem on /dev/sda1 requires a manual fsck
Update 2...
So, i need to fsck the drive...
Created a snapshot, made a new disk from that snapshot, added the new disk as an extra disk to another instance. Now that instance wont boot with the same problem... removing the extra disk fixed it again. So adding the disk makes it crash even though it isn't the boot-disk?
First, have a look at the Compute Engine -> VM instances -> NAME_OF_YOUR_VM -> Logs -> Serial port 1 (console) and try to find errors and warnings that could be connected to lack of free space or SSH. It'll be helpful if you updated your post by providing this information. In case if your instance run out of free space follow this instructions.
You can try to connect to your VM via Serial console by following this guide, but keep in mind that:
The interactive serial console does not support IP-based access
restrictions such as IP whitelists. If you enable the interactive
serial console on an instance, clients can attempt to connect to that
instance from any IP address.
more details you can find in the documentation.
Have a look at the Troubleshooting SSH guide and Known issues for SSH in browser. In addition, Google provides a troubleshooting script for Compute Engine to identify issues with SSH login/accessibility of your Linux based instance.
If you still have a problem try to use your disk on a new instance.
EDIT It looks like your test VM is trying to boot from the disk that you created from the snapshot. Try to follow this guide.
If you still have a problem, you can try to recreate the boot disk from a snapshot to resize it.
I've been working with MySQL and AuroraDB on AWS for a while now for a site I'm making. Up until a few minutes ago, I've been able to run any and all mutator functions on the instance through Lambda and MySQL workbench.
However, when I attempted to run an INSERT a few minutes ago, it gave me an error. I traced it back to Lambda saying that the --read-only flag is on my DB instance.
I was able to run these queries this morning, and now I cannot UPDATE, INSERT, or DELETE. I don't understand why it set itself to read-only randomly, besides maybe some sort of management or cleanup occurring on the AWS server, but under notifications there wasn't anything relating to any kind of maintenance.
Please let me know if there's any more information you need. I assumed that the code I was running is arbitrary, as I've tried running SQL commands directly through the workbench with the same results.
I found out the issue - the role of the slave instances of my Aurora cluster had swapped, probably during maintenance. I've since pointed all my Lambda functions to the cluster itself, which I actually didn't even know was something you could do, let alone the standard practice.
Master instance of google cloud sql keeps failing randomly and when it goes to fail-over, I got "The MySQL server is running with the --read-only". I changed the fail-over(replica) mysql instance to read-only : false. This removed the read only error but now the master and replica are not in sync.
Also, it keeps randomly pointing to the replica, suggesting that the master is down.
How do I get the master and replica in sync again?
Why does the master keep failing?
Thank you team!
I took a snapshot of a 50GB volume (non-boot) which is attached to an instance. The snapshot was successful.
I shut the instance and tried taking another snapshot of the same volume. This time the command hung. gcloud status reflects "CREATING" for this attempt. It is hours since I started the snapshot command. I tried the same using google developers console. The behaviour remains the same.
I restarted the instance and the status of the snapshot changes to "READY" within seconds.
It seems that snapshots should be taken if the volume is attached to a running instance. Otherwise the command is queued and executed when the volume/instance is live. Is this expected behaviour?
I replicated your issue and indeed the snapshot process halts when the instance is shut down. You may have also noticed that now the Shutdown/Start feature has been introduced - it was not available before.
I believe this is due to how snapshots are being handled on the platform. Your first snapshot creates a full copy of the disk, while the second one is differential - the differential one will fail or stay in pending as it cannot query the source disk while the instance is down to check what has been changed . You can check this for further info.
Dettaching the disk from the instance and then creating a snapshot works, so that could be a workaround for you.
Hope this helps
I followed Google's guide Configuring Replication with Google Cloud SQL to have a replication between these 2 elements:
Master. The Google Cloud SQL instance. Which is supposed to be a modified, but still compatible, MySQL version by Google.
Slave. Standard MySQL version 5.5.35-0+wheezy1 running on Debian 7 Wheezy. This is on an instance of Google Compute Engine.
All the setup when smoothly and the slave is working, but all the users on the slave are not working with passwords. Every user is functional but they can be accessed without passwords which of course is an undesired behavior.
Any idea on how to get the slave working with the same user/passwords as the master?
Note: This would be more appropriate for Server Fault, but I've seen this tag way more active on SO, even with platform-only questions, so I hope this can stay here.