These are the requirements:
Mysql, Jupyter Notebook (both should be on Kubernetes cluster)
I need to run machine learning models using jupyter notebook by fetching data from mysql database and this whole task needs to be scheduled(just like cron scheduling) on kubernetes cluster environment.
I am new at kubernetes but have knowledge of docker containerization and have built containerized applications before. I beg you to please help.
scheduled(just like cron scheduling)
You can use the feature cornjob to schedule the resources on Kubernetes.
Read more about cronjob : https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
If you have docker image of notebook or code that you are planning to run you can simply create the YAML config and run that docker image with cronjob on Kubernetes as schedule task.
i have not used Jupyter notebook so not sure how it's work.
For running MySQL on kubernetes you can follow same way, config the YAML files and apply the changes in K8s cluster and your container will get deployed to K8s.
You can read more here : https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/
Related
I'm building an application in Python and using GitHub Actions to automate the testing 'on push'. However, I now want to connect my app to an existing MySql database.
From searching the Marketplace, Google and YouTube, I can see the following options:
use the MySql supplied with the GitHub Actions' Ubuntu virtual environment
setup a new MySql dB inside the GitHub Actions VM.
Setup MySql inside a Docker container and connect to it from another Docker container containing my app.
What I can't see is how to connect out of the GitHub Actions VM to an existing database on my network. Is it possible and should I expect to see a pre-built action to do this in the Marketplace.
Sorry for such an obtuse question: old, out-of-date programmer new to both CI/CD and containerisation. Thank you.
I used to test installed mongodb-shard on kubernetes by helm,but I found that those helm charts are not really a qualified mongodb shard.These helm charts can correctly create Pods with names like mongos-1 mongod-server-1 mongod-shard-1,this seems to be a correct shard cluster configuration, but the appropriate mongos, mongod server instance is not created on the corresponding Pod.They just create a normal mongod instance on the corresponding pod, and there is no connection between them.Do I need to add scripts to execute commands similar to rs.addShard(config)?Encountered the same problem when installing mysql cluster using helm.
What I want to know is, is it not appropriate to install mysql/mongodb cluster on kubernetes in general scenarios?Is the database installed independently or deployed on Kubernetes?
Yes, you can deploy MongoDB instances on Kubernetes clusters.
Use standalone instance if you want to test and develop and replica set for production like deployments.
Also to make things easier you can use MongoDB Enterprise Kubernetes Operator:
The Operator enables easy deploys of MongoDB into Kubernetes clusters,
using our management, monitoring and backup platforms, Ops Manager and
Cloud Manager. By installing this integration, you will be able to
deploy MongoDB instances with a single simple command.
This guide has references to the official MongoDB documentation with more necessary details regarding:
Install Kubernetes Operator
Deploy Standalone
Deploy Replica Set
Deploy Sharded Cluster
Edit Deployment
Kubernetes Resource Specification
Troubleshooting Kubernetes Operator
Known Issues for Kubernetes Operator
So basically all you need to know in this topic.
Please let me know if that helped.
I have openshift 3.9 installed in one AWS region ohio. I have jenkins installed in it. I have a pipeline code in where it will take Java code from GitHub bind with jboss and deployed it in project test within the same cluster. It works fine and I'm able to access the app as pod is creating and app is also binding with jboss. Now I want to deploy this application across different clusters either within the same region or across different regions. Is there a way to achieve this?
You can use the oc command line tool in your Jenkins pipeline to deploy it to a different cluster. For a related example, check the Gitlab review apps example using an OpenShift cluster. It does something similar, where the CI pipeline deploys the required artifacts to an OpenShift cluster using oc and appropriate credentials.
I have several questions regarding Docker.
First my project:
I have a blog on a shared host and want to move it to the cloud to have all the server sides in my hands and to have the possibility to scale my server on my needs.
My first intend was to setup a nice ubuntu 14 lts as a server with nginx, php 7 and mysql. But I think it's not that easy to transfer such a server to another cloud i.e. from gce to aws. I then thought about using docker, as a friend told me how easy it is to setup containers and how easy it is to move them from one server to another.
I then read a lot about docker but stumbled upon a few things I wondered about.
In my understanding docker runs just services like php, mysql or similar, but doesn't hold data, right?
Where would I store all the data like database, nginx.conf, php.ini and all the Files I want to serve with nginx (ie. /var/www/)?
Are they stored on the host system? If yes, it would not be easier to move a docker setup then move a whole server, no?
Do I really have an advantage of using Docker to serve a Wordpress Blog or another Website using MySQL and so on?
Thanks in advance
Your data is either stored on the host machine or you data is attached to the docker containers remotely (using a network-attached block device).
When you store your data on the host machine, you have a number of options.
The data can be 'inside' one of your containers (e.g. your mysql databases live inside your mysql container).
You can mount one or more directories from your host machine inside your containers. So then the data lives on your host.
You can create Docker volumes or Docker volume containers that are used to store your data. These volumes or volume containers are mounted inside the container with your application. The data then lives in directories managed by Docker.
For details of these options, see dockervolumes
The last option is that you mount remote storage to your docker containers. Flocker is one of the options you have for this.
At my work I've set up a host (i.e. server) that runs a number of services in docker containers. The data for each of these services 'lives' in a Docker data volume container.
This way, the data and the services are completely separated. That allows me to start, stop, upgrade and delete the containers that are running my services without affecting the data.
I have also made separate Docker containers that are started by cron and these back up the data from the data volume containers.
For mysql, the backup container connects to the mysql container and executes mysqldump remotely.
I can also run the (same) containers that are running my services on my development machine, using the data that I backed up from the production server.
This is useful, for instance, to test upgrading mysql from 5.6 to 5.7.
I have install tomcat 6.0 and mysql 5.5 on amazon linux instance.
now i want to deploy war file on that tomcate & .sql file on mysql running on amazon instance .I am new for amazon services.Plz give details about procedure.
Plz help me for that .Thanks in advance.
The simple way is use scp or rsync to upload file and restart Tomcat.
However if you have many servers or WARs, even more complicated situation, consider other ways:
use jenkins to deploy
write your deploy script leveraged by python-fabric
You should design your own deploy process to overcome the difficulties you met.
In my case is every ec2 instances are spot instance, they are created by scripts or autoscaling.
We should keep the every new spot instance update to date, using the latest software and JARs to run web crawler.
Our design is very simple. Just a script to download files from S3 and unzip it:
ec2 spot instance completed booting
run the software-update script
run the software script getting from the updater
In your case, there are some key point you not figure out:
How many ec2 instance should update ?
A ec2 instance how to know it need to update.
(many other points)
What is the best way to deploy your WARs ? It depends on your situation.