I have a project based on a massive ingestion of Kafka messages, those messages interact with a MySQL database.
I want to know the best way to update MySQL tables using scripts (.sql). I'm thinking about deploy them during the application startup, then Kafka will save the messages until the application is started and send them to the application with all database modifications finished.
Any idea/example? I suppose kubernetes orchestration can be a problem to achieve this!
One theoretical possibility here sergio
Attach the script to a PVC to be used in MySQL and add the scripts to
run
Use a post start hook and run the script mounted
(Post start hook)
For the kafka container have a
init-container. This checks for existance of a row or some
check if all is well with the MySQL pod
Bring up the kafka pod
(was over limit for a comment , posted as an answer)
Related
I'm facing a problem in running the Nest JS cron job in sync with Mysql database. The problem right now is once the server gets restarted all the cron jobs are lost. What is the best way to get the existing cron jobs which was set before ?
I would not recommend running cronjobs in your primary server. The cronjobs you run in NestJs are run in the same node process. So if you restart the server, they are all lost.
It's better to run cronjobs in another node process. (you start another nestJs app that does only the cronjobs).
The reason is that you want to monitor cronjobs and decouple them from the main application
Clouds like render.com have a cronjob section, for instance.
The only way that I found it's creating a crons table in the db and use the OnModuleInit provided by NestJS and retrieve all the crons from DB and run them inside the OnModuleInit method.
I'm running optaweb-employee-rostering in a dockerized Wildfly server, persisting data with MySql database running in a container too. The .war file is not built in sever's Docker image, it's manually deployed in it via Wildfly's admin interface. Every time container is stopped a restarted, the application rebuild sample data, deleting any data saved during usage, so that the final behavior is the same as ram based storage: the data is lost if the server stops.
Is there a way to avoid this behavior and keep saved data on server restart?
This is caused by the hbm2dll value here and due to the Generator's post construct. In the current openshift image there are environment variables to change that.
We're working on streamlining this "getting started" and "putting it into production" experience, as part of the refactor to react / springboot.
When I run report jobs (get net sales data from mysql) in crontab (bashscript) , I establish a connection to mysql and get the required data.
Does this leave mysql connections open or all the connections are closed on job completion?
UPDATE - I use bash script to connect to mysql.
In basic scenarios the MySQL connections will be closed when the mysql client within the bash script finishes running.
If you are running the mysql command line Linux client within your bash script then bash would normally wait for the mysql process to exit before continuing to the end of the script.
There are ways to persist processes beyond the life of the bash script but you haven't mentioned using those in your question.
If you are using a mysql library that has a close function - most of the MySQL libraries have this in the api then you should use it. Although the default process behaviour will probably clean up open connections for you, it helps to get in the habit of closing resources that you are not going to require again within your code as this makes it more scalable and also informs other developers of your intended behaviour.
This may sound stupid and it is. But i'm wondering if it is possible to create a GCE instance that its sole purpose is to copy data from another GCE's MySQL database and copies all data to a Google Cloud SQL instance every few minutes and essentially updates the GCloud SQL.
Essentially i'm trying to get around how GAE can't connect to a GCE MySQL database but you can connect to a Google cloud SQL database.
I have tried "FEDERATED Tables" however Google Cloud SQL doesn't support that. So this is my last resort.
Thanks
Why do you need the GCE database at all? That is, why can't you just use a Cloud SQL database for all of your database needs?
You could try manually replaying the binary log to your Cloud SQL instance, ie:
Enable binary logging on your GCE MySQL instance.
Use mysqlbinlog to periodically extract the latest contents of the log as SQL statements. Use the positioning functions to make sure each run starts where the last finished.
Send the SQL outputted by mysqlbinlog to your Cloud SQL instance.
I have a .sql file (initial sql scripts). I have recently deployed application in cloudfoundry, So I want to run these scripts to make application work, Scripts will update more than 5 db tables.
Is there any other way to run the mysql scripts from the grails application on start up Or Is there any provision to run the scripts in the cloudfoundry.
you have several options here.
The first one (which I recommend), is to use something like http://liquibase.org/ (there is a Grails plugin for it: http://grails.org/plugin/liquibase). This tool will make sure that any script you give it will run prior to the app starting, without running the same script twice, etc. This is great to keep track of your database changes.
This works independently of CloudFoundry and would help anyone installing your app having an up to date schema
The second option would be to tunnel to the CloudFoundry database and run the script to the db. Have a look at http://docs.cloudfoundry.com/tools/vmc/caldecott.html or even easier with STS : http://blog.cloudfoundry.com/2012/07/31/cloud-foundry-integration-for-eclipse-now-supports-tunneling-to-services/
Yup, what ebottard said! :-) Although, personally I would opt for using the tunnel feature on VMC, but that said, I am a Ruby guy!
Be weary of the fact that there are timeouts against queries in MySQL if are bootstrapping your database with large datasets!