I have Kafka consumer Golang application. I'm trying to deploy it in PKS cluster. Here is the docker file that I have defined,
FROM golang:1.19-alpine as c-bindings
RUN apk update && apk upgrade && apk add pkgconf git bash build-base sudo
FROM c-bindings as app-builder
WORKDIR /go/app
COPY . .
RUN go mod download
RUN go mod verify
RUN apk add librdkafka-dev pkgconf
RUN go build -race -tags dynamic --ldflags "-extldflags -static -s -w" -o main ./main.go
FROM scratch AS app-runner
WORKDIR /go/app/
COPY --from=app-builder /go/app/main ./main
CMD ["/go/app/main"]
I need GSSAPI as the SASL mechanism, hence added this in the docker (above),
RUN apk add librdkafka-dev pkgconf
However, while building the image it ends giving below error,
ERROR [app-builder 6/6] RUN go build -race -tags dynamic --ldflags "-extldflags -static -s -w" -o main ./main.go 9.9s
------ [app-builder 6/6] RUN go build -race -tags dynamic --ldflags "-extldflags -static -s -w" -o main ./main.go:
#13 4.598 # github.com/confluentinc/confluent-kafka-go/kafka
#13 4.598 ../pkg/mod/github.com/confluentinc/confluent-kafka-go#v1.9.2/kafka/00version.go:44:2: error: #error "confluent-kafka-go requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html"
#13 4.598 44 | #error "confluent-kafka-go requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html"
#13 4.598 | ^~~~~
------ executor failed running [/bin/sh -c go build -race -tags dynamic --ldflags "-extldflags -static -s -w" -o main ./main.go]: exit code: 2
Apparently
RUN apk add librdkafka-dev pkgconf
is not able to pull the latest version of librdkafka for the golang:1.19-alpine base. Am I missing something here?
Related
I have a container file that uses the base image of fedora 37. I want to install some python libraries and some of them require gssapi. I know that gssapi requires krb5-devel. I have a step to install that and I can confirm that it works by calling krb5-config. However, while building the container (using Kaniko) it hit the issue of incompleted installation of gssapi because krb5-config was not found.
I could build my container fine locally using podman/buildar but when pushing my code to GitLab and using Kaniko to build the container, it failed. I doubt if it is because of Kaniko.
Here is my container file:
FROM fedora:37
USER root
WORKDIR /opt/workspace
RUN dnf install -y createrepo_c \
krb5-devel \
python3-pip \
python3-devel \
tree \
diffutils \
git \
gcc
RUN krb5-config
COPY *requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt -r test_requirements.txt
I also tried to install heimdal-devel and did the symblink RUN ln -s /usr/bin/heimdal-krb5-config /usr/bin/krb5-config but it doesn't help at all.
I'm trying to add a predeploy hook for AWS Beanstalk.
The file is
+-- .platform
+-- hooks
+-- predeploy
+-- 01_npm_install_and_build.sh
With the following contents:
curl --silent --location https://rpm.nodesource.com/setup_16.x | sudo bash -
sudo yum -y install nodejs
cd /var/app/current/
sudo npm install
sudo npm run build
I've tested the code works by SSHing to the instance and running sh 01_npm_install_and_build.sh
by looking at the log file tail -f /var/log/eb-engine.log
I also tried postdeploy with the same issue, here's that error:
[ERROR] An error occurred during execution of command [app-deploy] -
[RunAppDeployPostDeployHooks]. Stop running the command. Error:
Command .platform/hooks/postdeploy/01_npm_install_and_build.sh failed
with error fork/exec
.platform/hooks/postdeploy/01_npm_install_and_build.sh: exec format
error
The problem was that I was missing a "shebang" at the top of the sh script.
The sh script should start with:
#!/bin/bash... or see What is the preferred Bash shebang ("#!")? to check which shebang you should be using. which sh should give you an idea.
Furthermore. /var/app/current/ isn't available at predeploy, so use /var/app/staging/ instead.
I am trying to run MySQL in an R script running inside a Docker (actually Rocker) container, but am getting an error on the “install_packages(…” step. The only solution I found that was somewhat similar was specific to running RedHat Linux (I am running Mac OS X 10.15) and the solution included pointing to several directories that I do not have. Running under RStudio or from the command line, RMySQL loads without a problem.
Dockerfile:
FROM rocker/r-ver:latest
RUN mkdir /home/analysis
COPY install_packages.R /home/analysis/install_packages.R
COPY script_basic.R /home/analysis/script_basic.R
RUN Rscript /home/analysis/install_packages.R
CMD Rscript /home/analysis/script_basic.R
R files:
script_basic.R
library( RMySQL )
install_packages.R
install_packages(“RMySQL”)
Commands and responses:
$ docker build -t myapp .
…
Successfully tagged myapp:latest
$ docker run -it --rm myapp
Loading required package: DBI
Error: package or namespace load failed for ‘RMySQL’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/usr/local/lib/R/site-library/RMySQL/libs/RMySQL.so':
libmysqlclient.so.21: cannot open shared object file: No such file or directory
Execution halted
$
I believe that rocker/r-dev does not include the libraries you need to get this to run. Untested, but try something like this.
Dockerfile
FROM rocker/r-ver:latest
RUN mkdir /home/analysis
RUN apt-get update \
&& apt-get install -y --no-install-recommends libmariadbclient-dev
COPY install_packages.R /home/analysis/install_packages.R
COPY script_basic.R /home/analysis/script_basic.R
RUN Rscript /home/analysis/install_packages.R
CMD Rscript /home/analysis/script_basic.R
You can find the need for certain libraries on https://cran.r-project.org/web/packages/RMySQL/index.html, as
SystemRequirements: libmariadb-client-dev | libmariadb-client-lgpl-dev | libmysqlclient-dev (deb), mariadb-devel (rpm), mariadb | mysql-connector-c (brew), mysql56_dev (csw)
and a note on how to install them in RMySQL's github README.md's Installation section.
(The CRAN page references libmysqlclient-dev, but the github page suggests "the mariadb implementation is much better". I don't know the premise of the statement, just explaining why I suggested one and CRAN listed another.)
When I try to test out invoking a serverless function handler using the serverless-rust plugin I get the error
/usr/bin/ld: cannot find -lmysqlclient
collect2: error: ld returned 1 exit status
You will need to use the hooks in the lambda-rust docker builder to install the necessary dependencies and include the library files in the packaged build.
First create a directory at the root of the project named .lambda-rust and in it create two files:
install
#!/usr/bin/env bash
echo "installing mysql dependencies"
yum install -y mysql-devel
and
package
#!/usr/bin/env bash
echo "packaging mysql library"
zip -j "$1.zip" /usr/lib64/mysql/libmysqlclient.so.18
Adjust the name of the library file for whichever is installed for you (I added a line with echo $(ls /usr/lib64/mysql) in one of the scripts to see what files existed after the install)
I have a simple startup script which looks like so:
#!/usr/bin/env bash
sudo apt update
sudo apt install -y ruby-full ruby-bundler build-essential
And create VM instance on GCP like so:
$ gcloud compute instances create test-app --boot-disk-size=10GB --image-family ubuntu-1604-lts --image-project=ubuntu-os-cloud --machine-type=g1-small --zone europe-west1-b --tags test-server --restart-on-failure --metadata-from-file startup-script=startup.sh
My startup.sh is executable. I set its rights like so:
$ chmod +x startup.sh
When however I enter the shell of my newly created instance and check bundler:
test-app:~$ bundle -v
I get these messages:
The program 'bundle' is currently not installed...
So, what is wrong with that and how can I fix it? PS. If I run all my commands just from inside the instance shell, it's all ok, so there is some problem with using startup script on GCP.
I tested with your use case, But the bundle package was installed without making any changes.
Output:
bundle -v
Bundler version 1.11.2
You can check VM serial console log output to verify if start-up script ran. Check the VM instance to verify if the package is installed using the commands below:
sudo apt list --installed | grep -i bundle
sudo egrep bundle /var/log/dpkg.log
In addition, check the gem list bundle