I have installed Kafka on a local Minikube by using the Helm charts https://github.com/confluentinc/cp-helm-charts following these instructions https://docs.confluent.io/current/installation/installing_cp/cp-helm-charts/docs/index.html like so:
helm install -f kafka_config.yaml confluentinc/cp-helm-charts --name kafka-home-delivery --namespace cust360
The kafka_config.yaml is almost identical to the default yaml, with the one exception being that I scaled it down to 1 server/broker instead of 3 (just because I'm trying to conserve resources on my local minikube; hopefully that's not relevant to my problem).
Also running on Minikube is a MySQL instance. Here's the output of kubectl get pods --namespace myNamespace:
I want to connect MySQL and Kafka, using one of the connectors (like Debezium MySQL CDC, for instance). In the instructions, it says:
Install your connector
Use the Confluent Hub client to install this
connector with:
confluent-hub install debezium/debezium-connector-mysql:0.9.2
Sounds good, except 1) I don't know which pod to run this command on, 2) None of the pods seem to have a confluent-hub command available.
Questions:
Does confluent-hub not come installed via those Helm charts?
Do I have to install confluent-hub myself?
If so, which pod do I have to install it on?
Ideally this should be configurable as part of the helm script, but unfortunately it is not as of now. One way to work around this is to build a new Docker from Confluent's Kafka Connect Docker image. Download the connector manually and extract the contents into a folder. Copy the contents of this to a path in the container. Something like below.
Contents of Dockerfile
FROM confluentinc/cp-kafka-connect:5.2.1
COPY <connector-directory> /usr/share/java
/usr/share/java is the default location where Kafka Connect looks for plugins. You could also use different location and provide the new location (plugin.path) during your helm installation.
Build this image and host it somewhere accessible. You will also have to provide/override the image and tag details during the helm installation.
Here is the path to the values.yaml file. You can find the image and plugin.path values here.
Just an add-on to Jegan's comment above: https://stackoverflow.com/a/56049585/6002912
You can choose to do the Dockerfile below. Recommended.
FROM confluentinc/cp-server-connect-operator:5.4.0.0
RUN confluent-hub install --no-prompt debezium/debezium-connector-postgresql:1.0.0
Or you can use a Docker's multi-stage build instead.
FROM confluentinc/cp-server-connect-operator:5.4.0.0
COPY --from=debezium/connect:1.0 \
/kafka/connect/debezium-connector-postgres/ \
/usr/share/confluent-hub-components/debezium-connector-postgres/
This will help you to save time on getting the right jar files for your plugins like debezium-connector-postgres.
From Confluent documentation: https://docs.confluent.io/current/connect/managing/extending.html#create-a-docker-image-containing-c-hub-connectors
The Kafka Connect pod should already have the confluent-hub installed. It is that pod you should run the commands on.
The cp kafka connect pod has 2 containers, one of them is a cp-kafka-connect-server container.That container has confluent-hub installed.You can login into that container and run your connector commands there.To login into that container, run the following command:
kubectl exec -it {pod-name} -c cp-kafka-connect-server -- /bin/bash
As of latest version of chart, this can be achieved using customEnv.CUSTOM_SCRIPT_PATH
See README.md
Script can be passed as a secret and mounted as a volume
Related
I'm trying to run a containerized app which is stored in Nexus docker hosted on url 12.23.34.55:8086
I'm trying to run it on my Openshift Cluster, but I'm getting error. Commands I'm using to run
oc create secret docker-registry mysecret --docker-server=http://12.23.34.55/ --docker-username=aditya --docker-password=aditya --docker-email=aditya#example.org
oc secrets link default mysecret --for=pull
My nexus is running on http://12.23.34.55:8081
Now I'm using command to launch in OpenShift using below command.
oc new-app 12.23.34.55:8085/mytestapp:11 --insecure-registry=true
as per $ oc new-app myregistry:5000/example/myimage
https://docs.openshift.com/container-platform/4.1/applications/application_life_cycle_management/creating-new-applications.html
But it does not work, it asks for password and not able to deploy from console too, can anyone help me with exact commmand.
Creating the secret is not enough for OpenShift to be able to pull from the registry. You still need to link that secret as well.
Take a look at the official documentation here:
https://docs.openshift.com/container-platform/4.1/openshift_images/managing_images/using-image-pull-secrets.html#images-allow-pods-to-reference-images-from-secure-registries_using-image-pull-secrets
Okay! I found an answer, so using private registry first we should import image using
oc import name url/imagename:tag
then we can create new app with the same
oc new app name
I'm trying to access the ContextBroker configuration file in the path /etc/sysconfig/contextBroker and it's empty. What is the problem?
https://fiware-orion.readthedocs.io/en/master/admin/running/index.html
I'm using Docker.
Also I am testing the installation by yum centos and tells me that the repository is wrong.
Is it copied from the web?
https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/admin/yum.md
The /etc/sysconfig/contextBroker is used in RPM-base deployment. Docker is based in compiling Context Broker directly from sources, as you can see in the docker file.
So, in this case, you have to use CLI based configuration. Note the docker is built with some of them:
ENTRYPOINT ["/usr/bin/contextBroker","-fg", "-multiservice", "-ngsiv1Autocast" ]
But you can add aditional ones. For example, in the reference docker-compose.yml we set -dbhost, and more ones could be added in the same way.
command: -dbhost mongo
I guess that using docker run you could also add commands in the same way command works in docker-compose.yml, although I don't know the details. Maybe some docker expert could add more info :)
I am trying to deploy a simple node.js application with the new Kubernetes support in Bluemix. When I run the container I made, I get a ImagePullBackOff error, which means it can't pull down the image.
NAME READY STATUS RESTARTS AGE
hello-node-2399519400-6m8dz 0/1 ImagePullBackOff 0 13m
My Docker image uses the node.js base image.
FROM node:6.9.2
EXPOSE 8080
COPY server.js .
CMD node server.js
I deployed using:
docker build -t hello-node:v1 .
kubectl run hello-node --image=hello-node:v1 --port=8080
I am thinking that Bluemix can't pull down the node.js image, but I am not certain.
I see the docker build of the image, and I'm presuming that you're using the kubectl with the exported cluster config (bx cs cluster-config ...), so that it's targetting your cluster.
Did you tag and push that image from your local docker into the bluemix registry, or to another remote registry that would be accessible from the container service? (My apologies if this is obvious - just didn't see the step there to tag and push it to a registry that would be available).
I had to first push the image to Bluemix with:
docker build -t registry.ng.bluemix.net/namespace/hello-node:1
docker push registry.ng.bluemix.net/namespace/hello-node:1
kubectl run hello-node-deployment --image=registry.ng.bluemix.net/namespace/hello-node:1
On my ubuntu laptop I was issuing some kubectl commands including running kubernetes from a local Docker container all was well ... at some point I then issued this command
kubectl config set-cluster test-doc --server=https://104.196.108.118
now my local kubectl fails to execute ... looks like the Server side needs to get reset back to default
kubectl version
Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.2", GitCommit:"528f879e7d3790ea4287687ef0ab3f2a01cc2718", GitTreeState:"clean"}
error: couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I deleted and reinstalled the gcloud SDK binaries and ran
mv ~/.config/gcloud ~/.config/gcloud~ignore
gcloud init
gcloud components update kubectl
How do I delete my local kubectl settings (on ubuntu 16.04) and start fresh ?
It's important to note that you've set a kubeconfig setting for your client. When you run kubectl version, you're getting the version for client and the server which in your case seems to be the issue with the version command.
Updating your config
You need to update the setting to the appropriate information. You can use the same command you used to set the server to change it to the correct server.
If you want to wipe the slate clean in terms of client config, you should remove the kubeconfig file(s). In my experience with the gcloud setup, this is just ~/.kube/config.
If you are running the cluster through google cloud engine, you can use gcloud to get the kubeconfig set for you as per the container engine quick start guide. The following assumes that you have defaults for the project, zone, and cluster set.
gcloud container clusters get-credentials CLUSTER_NAME
Removing kubectl - this isn't necessary
If your goal is to wholesale get rid of kubectl, you should remove the component rather than reseting gcloud.
gcloud components remove kubectl
But that won't solve your problem as it doesn't remove or reset ~/.kube/config when I run it on Mac and if you want to keep working with it, you'll need to reinstall kubectl.
I am migrating from DotCloud to Elastic Beanstalk.
Using DotCloud, they clearly explained how to set up Python Worker, and how to use supervisord.
Moving to Elastic Beanstalk, I am lost on how I could do that.
I have a script myworker.py and want to make sure it is always running. How?
Elastic Beanstalk is just a stack configuration tools over EC2, ELB and autoscaling.
One approach you can use, is create your own AMI, but since October last year, there is another approach that probably will be more suitable for your needs: ebextensions.
.ebextension is just a directory in your application, that get's detected once your application has been loaded by AWS.
Here is the full documentation: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers.html
With Amazon Linux 2 you need to use the .platform folder to supply elastic beanstalk with installation scripts.
We recommend using platform hooks to run custom code on your environment instances. You can still use commands and container commands in .ebextensions configuration files, but they aren't as easy to work with. For example, writing command scripts inside a YAML file can be cumbersome and difficult to test.
So you should add a prebuild hook (example) into a .platform folder to install supervisor and a postdeploy hook (example) to restart supervisor after each deployment.
There is an ini file (example) used in the script; which is made for laravel specific.
Make sure that the .sh files from the .platform folder are executable before deploying your project:
$ chmod +x .platform/hooks/prebuild/*.sh
$ chmod +x .platform/hooks/postdeploy/*.sh