When I start infinidb, I am getting the following error message, making it impossible for the service to start. In the log file, it says there is a problem with the rollback. Is there a way to abort the rollback so I can start the service again? I am ok with losing some data.
Starting Calpont InfiniDB Database Platform: ....... DONE
Warning! The DBRM is currently in Read-Only mode!
Updates will not propagate!
Starting Calpont InfiniDB MySQL:
Starting MySQL
. *
Validate InfiniDB System Catalog:
Validation Succesfully Completed
Perform Functionality test:
InfiniDB Logging check: DONE
Platform Process check: ERROR: Problem with InfiniDB process DMLProc, should be a single version running
***Stopping InfiniDB to allow the process problem to be resolved.
Shutting down Calpont InfiniDB MySQL:
Shutting down MySQL
... *
Shutting down Calpont InfiniDB Database Platform:
/var/log/Calpont/crit.log
DMLProc[4292]: 41.458706 |0|0|0| C 20 CAL0002: DMLProc failed to start due to : problem with rollback. Version buffer file does not exists.
I already tried removing the files in /usr/local/Calpont/data1/systemFiles/dataTransaction but it still won't start.
This is how I was able to solve it.
While the service is starting:
# service infinidb start
In another ssh session go to /usr/local/Calpont/bin and look for the transaction ID causing the problem:
# ./rollback -p
Once you find out, immediately run the rollback command with the transaction id that you see:
./rollback -r 64536
It will respond with OK and the service started successfully this time. I had to do this twice because my first attempt was probably not able to rollback the transaction on time.
Related
My application has three different servers which is implemented in Laravel:-
1. Cron server
2. Mysql server
3. User server
Laravel version - 5.6
PHP version - 7.1
Operating system - Centos
After 1-2 days when I am trying to login into website then following error had occoured:-
ERROR: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. {"exception":"[object] (Predis\Response\ServerException(code: 0): MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. at /var/www/html/vendor/predis/predis/src/Client.php:370)
When I restart the service of redis then I am able to login again.
What is its best solution??
I had tried to solved this issue with following steps:-
1. Dir - set the permission 755 of this directory
2. -- redis-cli
-- config set stop-writes-on-bgsave-error no
-- sudo systemctl restart redis
Following errors are occured time to time in redis logs:-
I'm using Liquibase in all my projects, I really love the way it handles db updates, but recently I'm having this issue:
liquibase : Successfully acquired change log lock
liquibase : Successfully released change log lock
liquibase : Could not release lock
liquibase.exception.LockException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
at liquibase.lockservice.StandardLockService.releaseLock(StandardLockService.java:283) ~[liquibase-core-3.5.5.jar!/:na]
at liquibase.Liquibase.update(Liquibase.java:218) [liquibase-core-3.5.5.jar!/:na]
at liquibase.Liquibase.update(Liquibase.java:192) [liquibase-core-3.5.5.jar!/:na]
.
.
.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
.
.
liquibase : Failed to restore the auto commit to true
.
.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 16,913 milliseconds ago. The last packet sent successfully to the server was 89 milliseconds ago.
.
.
Caused by: java.lang.NullPointerException: null
at com.mysql.jdbc.MysqlIO.clearInputStream(MysqlIO.java:899) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2477) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
I have 3 apps running on my server. One of them running for about 2years. The second running for 2 months and the third one is new.
They are all Spring Boot applications.
This started happening when I want to update the jar of the second app, I stopped the old jar, run the new one and I can not get it to start with the error above.
Then I run the old jar of the same app and it started just fine. I try stopping and restarting this old jar and a few times I get the error above but most of the times it started just fine.
I try restarting the server (all three apps start as services with system boot) but none of them successfully start because of the same error. Not even the one running for 2 years. I stop all other apps and try again with this long-lasting app, it kept failing until it started.
Could it be a memory issue? I'm using a DigitalOcean droplet with 1 Core and 2gb RAM.
Also, what I've notice is that when it successfully start the logs looks like:
liquibase : Successfully acquired change log lock
liquibase : Reading from myDataBase.DATABASECHANGELOG
liquibase : Successfully released change log lock
Note the statement between the acquisition and release of the lock. I also suspect about a timing issue, between acquiring the lock and reading the changelog. But don't know if I can increase that time or how can I do it.
UPDATE
I don't known the source of the issue, but after updating the liquibase-core to v3.7.0 all the apps started correctly
UPDATE 2
About the previous update, I was testing as is not that the problem is solved, is just less frequent. But it has happen
IMPORTANT UPDATE 3
After some digging I realize that by the same time this happen, the team was playing around with DB Connection Pool. The application.properties files contain this:
spring.datasource.tomcat.initial-size=5
spring.datasource.tomcat.max-wait=20000
spring.datasource.tomcat.max-active=5
spring.datasource.tomcat.max-idle=5
spring.datasource.tomcat.min-idle=5
spring.datasource.tomcat.default-auto-commit=true
Sadly I don't understand much about DB Pool. But what raise my attention was the last line about the auto-commit. The last time this error pops up. I disable all those lines and the app started fine. I don't know if it was just coincidence or not.
Thank you very much!
UPDATE 4
I'm back to square one... I don't know how to solve this any clue would be appreciatted
UPDATE 5
I updated mysql-connector-java to v8.0.15. My server has MySQL v5.6.33. I'm still having this issue and now I can not restart my app.
After the above change, now the exception ends with:
The last packet successfully received from the server was 10,036 milliseconds ago. The last packet sent successfully to the server was 10,036 milliseconds ago.
.
.
.
Caused by: java.io.IOException: Socket is closed
at com.mysql.cj.protocol.AbstractSocketConnection.getMysqlInput(AbstractSocketConnection.java:72) ~[mysql-connector-java-8.0.15.jar!/:8.0.15]
at com.mysql.cj.protocol.a.NativeProtocol.clearInputStream(NativeProtocol.java:833) ~[mysql-connector-java-8.0.15.jar!/:8.0.15]
... 92 common frames omitted
UPDATE 6 - WORKAROUND SOLUTION
In a moment of clarity I decided to disable liquibase from my project, by setting liquibase.enabled=false in the application.properties. Now I can start/stop/restart the app anytime with no problem. As I only use liquibase to mantain the DataBase, and the changes I made are executed (the problem is when releasing the lock), I will enable Liquibase just to update the schema and then disable it to run the app
UPDATE 7 - I think I've solve it
I kept stoping and starting the apps over and over and every time I did it I get the Socket is closed so I began to make my head around a timing issue... From the exceptions thrown I notice that every time it says that:
The last packet successfully received from the server was xxxx milliseconds ago.
The smallest value I get of those milliseconds ago was 10.085 and by chance I have some logs of the same app from time ago when this issue never pop-up. From those logs statements I meassure the time between acquiring the change-log-lock and releasing it. To my surprise it was around 9 seconds and never above that.
So I login to mysql console and issue:
show variables LIKE '%timeout%';
Which gave me a list of the defined timeouts. And there it was connect_timeout was the smallest and set to 10 seconds. It has to be it.
I googled how to set that value. I've try creating a file /etc/my.cnf and adding to it:
[mysqld]
connect_timeout=20
Then restart mysql with:
sudo /etc/init.d/mysql restart
But when I log back to the mysql console the value was still 10. So in the mysql console I issued this other command:
SET GLOBAL connect_timeout=20;
And the value was updated (I don't know if this will persist after a restart)
I restart mysql again and... voilà! Now the apps start normally with liquibase enabled. I'm happy! :-)
I kept stoping and starting the apps over and over and every time I did it I get the Socket is closed so I began to make my head around a timing issue... From the exceptions thrown I notice that every time it says that:
The last packet successfully received from the server was xxxx milliseconds ago.
The smallest value I get of those milliseconds ago was 10.085 and by chance I have some logs of the same app from time ago when this issue never pop-up. From those logs statements I meassure the time between acquiring the change-log-lock and releasing it. To my surprise it was around 9 seconds and never above that.
So I login to mysql console and issue:
show variables LIKE '%timeout%';
Which gave me a list of the defined timeouts. And there it was connect_timeout was the smallest and set to 10 seconds. It has to be it.
I googled how to set that value. I've try creating a file /etc/my.cnf and adding to it:
[mysqld]
connect_timeout=20
Then restart mysql with:
sudo /etc/init.d/mysql restart
But when I log back to the mysql console the value was still 10. So in the mysql console I issued this other command:
SET GLOBAL connect_timeout=20;
And the value was updated (I don't know if this will persist after a full server restart)
I restart mysql again and... voilà! Now the apps start normally with liquibase enabled. I'm happy! :-)
We were working with MySQL 8.0.19. For me, it were only a query which used to fail with the exception Caused by: java.io.IOException: Socket is closed.
Found that for MySQL 8.0.19, there was a bug reported with a similar stack trace of mine - https://bugs.mysql.com/bug.php?id=99234
After upgrading MySQL server to 8.0.21, it started working ! No other changes.
Hope it helps for folks who have spent quite some time with selected query(ies) not working and having the same error, and tried out various other options.
have you solved it yet?
I had a similar problem and solved it by forcing Liquibase to execute in synchronous mode.
Was your spring boot app generated by jhipster by any chance? If so, take a look at DatabaseConfiguration and change from AsyncSpringLiquibase to SpringLiquibase
In my website I needed a notifications which would be call me if user put like smile or something else. I did it using aftersave() method, which will work after save like or smile. When I pushed on server I got error ""App\Model\Table\LikesTable" is aborting the transaction before the save process is done.". But locally all work. I search some infromation in google. I think it's problems with mysql but I can't resolve this problem. Can you help me with it?
//Server//
Server version: 5.6.33-79.0 Percona Server (GPL), Release 79.0, Revision 2084bdb;
PHP 5.6.25 (cli) (built: Aug 19 2016 11:03:33);
//locally mysql version: 5.6.33
php-version 5.6.23
I want to write a MySQL daemon plugin that monitors queries on other MySQL servers and compares them with queries running on where the daemon is running. Basically this is in a spider engine setting, where a query initiated on the head node get run on the shards. Whenever a query is killed on the head, I want to have a daemon on the shard nodes that will kill the associated query there.
The idea was to initiate a pthread that uses mysql_real_connect, mysql_real_query... to access "show processlist" on the head node and compare them with local thread list. If I issue mysql_real_connect in the thread of the daemon, I get a segmentation fault. I think this is due to a threading issue in mysql_real_connect. I've used #define MYSQL_SERVER 1 and follow the approach taken in ha_federated::real_connect().
Is it actually possible to run mysql_real_connect from within a daemon plugin?
Thanks for any hints.
After meditating over the handlersocker daemon plugin I realised that in order for mysql_real_connect and friends to work in the daemon thread, a mysql internal thread environment needs to be set up and registered with the server.
This basically involves calling the following in the daemon thread:
my_thread_init();
thd = new THD;
mysql_mutex_lock(&LOCK_thread_count);
thd->thread_id = thread_id++;
mysql_mutex_unlock(&LOCK_thread_count);
thd->store_globals();
... setting up various options in THD ...
mysql_mutex_lock(&LOCK_thread_count);
threads.append(thd);
++thread_count;
mysql_mutex_unlock(&LOCK_thread_count);
I got an exception when i want to start oracle which is
Erorr:ORA-01033: ORACLE initialization or shutdown in progress
this is the first time I face this problem, DBMS asks me to wait and then restart the system , since yesterday and i am trying to start it ,but I couldn't make it.
please can someone help!
As the error message indicates, this means that the database is currently either initializing or is shutting down.
ORA-01033: ORACLE initialization or shutdown in progress
Cause: An attempt was made to log on while Oracle is being started up or shutdown.
Action: Wait a few minutes. Then retry the operation.
Sometimes, if one of the files needed for initialization is corrupted or needs recovery, you might see this message for ever. This is what you can do to see what the actual error is and then start debugging from there.
From your command prompt logon as sysdba
sqlplus /nolog
connect / as sysdba
Do a shutdown and a clean startup.
shutdown immediate;
startup;
This should indicate what is causing the issue and you'll be able to fix it.
A couple of useful links.
How to resolve ORA-011033: ORACLE initialization or shutdown in progress
http://forums.oracle.com/forums/thread.jspa?threadID=338394