I want to import and use dataset package of python at AWS Lambda. The dataset package is about MySQL connection and executing queries. But, when I try to import it, there is an error.
"libmysqlclient.so.18: cannot open shared object file: No such file or directory"
I think that the problem is because MySQL client package is necessary. But, there is no MySQL package in the machine of AWS Lambda.
How to add the third party program and how to link that?
You should install your packages in your lambda folder :
$ pip install YOUR_MODULE -t YOUR_LAMBDA_FOLDER
And then, compress your whole directory in a zip to upload in you lambda.
What you have to do is to include the binaries needed with your lambda package.
You need to utilize pip and create an isolated environment.Your zip uploaded to lambda needs to have the python2.7/site-packages included (the ones installed with pip).
Now there are extreme cases of os-related dependencies.
This has a tricky solution.
In those cases you have to spawn an amazon linux ec2 instance in order to build/get those dependencies and package them with your lambda.
Once your lambda is packaged you can close the ec2 instance.
Check this guide if virtualenv is not enough.
This is an os dependent system file. I'm guessing that you successfully installed the Python mysql client, but you still need the system mysql client, which seems to be a different version on your system than the lambda one. While building your virtual environment on the official lambda image will definitely fix this problem, you might have some luck copying your own copy of this system file into your lambda zip file.
I found mine with
locate libmysqlclient.so.18
Note: depending on your system, the version number at the end might be different. Use the version in the error you receive.
Adding that file on the top level of my zip file with
cd \path\from\locate\to\libmysqlclient
followed by
zip -u \path\to\lambda\zip\file.zip libmysqlclient.so.18
worked for me.
Related
I am trying to use this library here: multi-rake
However, as stated in the docs, we have to run this before installing multi-rake:
CFLAGS="-Wno-narrowing" pip install cld2-cffi
So I cannot simply put cld2-cffi and multi-rake in requirements.txt because cld2-cffi needs to be installed like this beforehand. How could I overcome this problem?
According to the official documentation you have to package as local dependencies.
You can also package and deploy dependencies alongside your function.
This approach is useful if your dependency is not available via the
pip package manager or if your Cloud Functions environment's internet
access is restricted. For example, you might use a directory structure
such as the following:
You can then use code as usual from the included local dependency,
localpackage. You can use this approach to bundle any Python packages
with your deployment.
Note: You can still use a requirements.txt file to specify additional
dependencies you haven't packaged alongside your function.
Specifying dependencies in Python
I have a web (online calculator for an example) which developed by my fellow tem members. Now they want to deploy in PCF using manifests.
Languages used : python, php and javascipt.
I gone through the docs about pcf with manifest.yml
In that I don't have any idea about services and env.
What is that services and how can I find the services for the above project and also how can I find the environment variables?
And tell whether these fields are mandatory to run the project in PCF.
To your original question:
What is that services and how can I find the services for the above project and also how can I find the environment variables? And tell whether these fields are mandatory to run the project in pcf.
Does your app require any services to run? Services would be things like a database or message queue. If it does not, then you do not need to specify any services in your manifest. They are optional.
Similarly, for environment variables, you would only need to set them if they are required to configure your application. Otherwise, just omit that section of your manifest.
At the end of the day, you should talk with whomever developed the application or read the documentation they produce as that's the only way to know what services or environment variables are required.
In regards to your additional questions:
1)And also I have one more query...like in our application we used python ok! In that we use lots of pacakages say pandas,numpy,scipy and so on...how can I import all the libraries into the PCF ??? Buildpacks will contain version only right?
Correct. The buildpack only includes Python itself. Your dependencies either need to be installed or vendored. To do this for Python, you need to include a requirements.txt file. The buildpack will see this and use pip to install your dependencies.
See the docs for the Python buildpack which explains this in more detail: https://docs.cloudfoundry.org/buildpacks/python/index.html#pushing_apps
2)And also tell me what will be the path for my app name if Java I can enclose jar files
For Java apps, you need to push compiled code. That means, you need to run something like mvn package or gradle assemble to build your executable JAR or WAR file. This should be a self contained file that has everything necessary to run your app, compile class files, config, and all dependent JARs.
You then run cf push -p path/to/my-app.jar (or WAR, whatever you build). The cf cli will take everything in the app and push it up to Cloud Foundry where the Java buildpack will install things like the JVM and possibly Tomcat so you app can run.
what should I do for application devloped using pyhton , JavaScript and php....
You can use multiple buildpacks. See the instructions here.
https://docs.cloudfoundry.org/buildpacks/use-multiple-buildpacks.html
In short, you can have as many buildpacks as you want. The last buildpack in the list is special because that is the buildpack which will set the start command for your application (although you can override this with cf push -c if necessary). The non-final buildpacks will run and simply install dependencies.
3) we were using postgresql how can I use this in pcf with my app
Run cf marketplace and see if there are any Postgres providers in your Marketplace. If there is one, you can just do a cf create-service <provider> <plan> <service name> and the foundation will create a database for you to use. You would then run a cf bind-service <app> <service name> to bind the service you create to your app. This will generate credentials and pass them along to your app when it starts. You app can then read the credentials out of VCAP_SERVICES and use them to make connections to the database.
See here for more details:
https://docs.cloudfoundry.org/devguide/services/application-binding.html
https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES
I tried building recutils version 1.7 downloaded from the home page, using the standard configure, make, sudo make install sequence, but when trying to run the resulting binaries. like recinf, I get the error:
recinf: error while loading shared libraries: librec.so.1: cannot open shared object file: No such file or directory
Does this mean I made a mistake during the build or is the package itself in error?
As Etan Reisner said the problem was that the shared object libraries were installed but not loaded into the cache, hence the need to run ldconfig. After running
sudo ldconfig
the binaries ran properly. If I had looked in /usr/local/lib, I would have seen the libs there.
I'm looking for a tcpdump binary that runs on openwrt. The website only shows source code that must be built. Anybody point me to a location that contains a prebuild binary? Thanks.
I suggest to set up a working repository, i.e.:
https://downloads.openwrt.org/chaos_calmer/15.05-rc2/x86/generic/packages/base/
You can put the url on your OpenWRT system under /etc/opkg.conf.
Using opkg you do not have to deal with all the dependencies.
opkg update
opkg install tcpdump
If there is no internet connection on the Openwrt system, and there is no local repository deployed, the solution above works as well, but it is more painful and consumes more time.
You find binaries at
http://downloads.openwrt.org/snapshots/trunk/
but you have to use the right one for your hardware plattform
e.g for x86
http://downloads.openwrt.org/snapshots/trunk/x86/packages/tcpdump_4.2.1-1_x86.ipk
The prefered method is to use opkg to install the package
I'm using NodeJS under Windows (with node.exe downloaded from NodeJS site). I would like to use this db-mysql extension with it ( https://github.com/mariano/node-db-mysql/ )
I have no idea how to use it. Is it necessary to rebuild the whole node.exe or is it possible to build just the extension? I'm not able to run NPM with node.exe (it results in error) as the installation instructions says.
Thanks
Sorry but you will have to have a full tool-chain(Cygwin) that can compile c. That means you have three alternatives:
Use linux (in a VM) or mac
Install node from cygwin ( https://github.com/joyent/node/wiki/Building-node.js-on-Cygwin-(Windows) )
Use another database (eg. node-dirty: https://github.com/felixge/node-dirty) that doesn't need to be compiled.
You can't use NPM (Node Package Manager) with the Windows executable node.exe as for all I know.
Instead you should download the zip/tar module from https://github.com/mariano/node-db-mysql/downloads and place the contents into a *node_modules* folder above your script(s).
Example structure:
/nodestuff
/node_modules
/db-mysql // The module
/script
app.js // Your script
Then just follow the instructions on https://github.com/mariano/node-db-mysql/ in order to create a database connection.
Good luck.