Docker image fails to build on Live but fine on Dev - mysql

I have a strange issue with Docker.
This is the Dockerfile in question.
FROM python:2.7
RUN apt-get update && apt-get install -y \
build-essential \
python-lxml \
python-dev \
python-pip \
python-cffi \
libcairo2 \
libpango1.0-0 \
libpangocairo-1.0.0 \
libxml2-dev \
libxslt1-dev \
zlib1g-dev \
libpq-dev \
libjpeg-dev \
libgdk-pixbuf2.0-0 \
libffi-dev \
mysql-client \
shared-mime-info
# ... further docker file rules, which doesn't get run cause apt-get fails
The problem I'm having is that on my development machine, this Dockerfile builds
perfectly fine, but on our live servers it's suddenly failing (worked in the
past), with E: Package 'mysql-client' has no installation candidate.
I thought the point of Docker is that everything runs using the same image and
that you shouldn't run into issues like this.
Why is this the case and what can I do to fix it from here so that it runs the
same on both dev and live?

You are using image python with tag 2.7, however this tag is a "shared" tag as per Python readme on Docker Hub which is changing other time: right now python:2.7 is shared with Python python:2.7.16 and python:2 but previously it was probably shared with python:2.7.15, python:2.7.14 etc. (in other words, python:2.7 is following python:2.7.x as it upgrades)
Your machine and live server probably pulled the image at a different time and now have a different image tagged 2.7. The "shared" tags seems to be like latest tags and may point to newer images as they are released.
What you can do:
Enforce image pull when building even if image is already present (using docker build with --pull option
Use a documented Simple tag instead, these should be more consistent (such as python:2.7.16-alpine3.9)
Do not re-build images during your release process, only build once and use the same image in your local and live environment (see below)
EDIT: this can be put into evidence with:
docker images --filter "reference=python" --digests --format "{{.Digest}} {{.Repository}}:{{.Tag}}"
sha256:7a61a96567a2b2ba5db636c83ffa18db584da4024fa5839665e330934cb6b2b2 python:2
sha256:7a61a96567a2b2ba5db636c83ffa18db584da4024fa5839665e330934cb6b2b2 python:2.7
sha256:7a61a96567a2b2ba5db636c83ffa18db584da4024fa5839665e330934cb6b2b2 python:2.7.16
sha256:39224960015b9c0fce12e08037692e8a4be2e940e73a36ed0c86332ce5ce325b python:2.7.15
To precise on:
I thought the point of Docker is that everything runs using the same
image and that you shouldn't run into issues like this.
Why is this the case and what can I do to fix it from here so that it runs the same on both dev and live?
Yes, and the recommended pattern is build image once and use that same image trough all your release process - this ensure you have the exact same context (packages, code, etc.) from development to production. You should not re-build your image from scratch on your live server, but ideally build it during your development phase and use that same image for testing and deploying.

Python:2.7 is now based on Debian Buster. There is no mysql-client apt package, considered https://packages.debian.org/search?keywords=mysql-client

Related

Difficulty installing JSON module for Perl on Mac [duplicate]

I get this error:
Can't locate Foo.pm in #INC
Is there an easier way to install it than downloading, untarring, making, etc?
On Unix:
usually you start cpan in your shell:
$ cpan
and type
install Chocolate::Belgian
or in short form:
cpan Chocolate::Belgian
On Windows:
If you're using ActivePerl on Windows, the PPM (Perl Package Manager) has much of the same functionality as CPAN.pm.
Example:
$ ppm
ppm> search net-smtp
ppm> install Net-SMTP-Multipart
see How do I install Perl modules? in the CPAN FAQ
Many distributions ship a lot of perl modules as packages.
Debian/Ubuntu: apt-cache search 'perl$'
Arch Linux: pacman -Ss '^perl-'
Gentoo: category dev-perl
You should always prefer them as you benefit from automatic (security) updates and the ease of removal. This can be pretty tricky with the cpan tool itself.
For Gentoo there's a nice tool called g-cpan which builds/installs the module from CPAN and creates a Gentoo package (ebuild) for you.
Try App::cpanminus:
# cpanm Chocolate::Belgian
It's great for just getting stuff installed. It provides none of the more complex functionality of CPAN or CPANPLUS, so it's easy to use, provided you know which module you want to install. If you haven't already got cpanminus, just type:
# cpan App::cpanminus
to install it.
It is also possible to install it without using cpan at all. The basic bootstrap procedure is,
curl -L http://cpanmin.us | perl - --sudo App::cpanminus
For more information go to the App::cpanminus page and look at the section on installation.
I note some folks suggesting one run cpan under sudo. That used to be necessary to install into the system directory, but modern versions of the CPAN shell allow you to configure it to use sudo just for installing. This is much safer, since it means that tests don't run as root.
If you have an old CPAN shell, simply install the new cpan ("install CPAN") and when you reload the shell, it should prompt you to configure these new directives.
Nowadays, when I'm on a system with an old CPAN, the first thing I do is update the shell and set it up to do this so I can do most of my cpan work as a normal user.
Also, I'd strongly suggest that Windows users investigate strawberry Perl. This is a version of Perl that comes packaged with a pre-configured CPAN shell as well as a compiler. It also includes some hard-to-compile Perl modules with their external C library dependencies, notably XML::Parser. This means that you can do the same thing as every other Perl user when it comes to installing modules, and things tend to "just work" a lot more often.
If you're on Ubuntu and you want to install the pre-packaged perl module (for example, geo::ipfree) try this:
$ apt-cache search perl geo::ipfree
libgeo-ipfree-perl - A look up country of ip address Perl module
$ sudo apt-get install libgeo-ipfree-perl
A couple of people mentioned the cpan utility, but it's more than just starting a shell. Just give it the modules that you want to install and let it do it's work.
$prompt> cpan Foo::Bar
If you don't give it any arguments it starts the CPAN.pm shell. This works on Unix, Mac, and should be just fine on Windows (especially Strawberry Perl).
There are several other things that you can do with the cpan tool as well. Here's a summary of the current features (which might be newer than the one that comes with CPAN.pm and perl):
-a
Creates the CPAN.pm autobundle with CPAN::Shell->autobundle.
-A module [ module ... ]
Shows the primary maintainers for the specified modules
-C module [ module ... ]
Show the Changes files for the specified modules
-D module [ module ... ]
Show the module details. This prints one line for each out-of-date module (meaning,
modules locally installed but have newer versions on CPAN). Each line has three columns:
module name, local version, and CPAN version.
-L author [ author ... ]
List the modules by the specified authors.
-h
Prints a help message.
-O
Show the out-of-date modules.
-r
Recompiles dynamically loaded modules with CPAN::Shell->recompile.
-v
Print the script version and CPAN.pm version.
sudo perl -MCPAN -e 'install Foo'
Also see Yes, even you can use CPAN. It shows how you can use CPAN without having root or sudo access.
Otto made a good suggestion. This works for Debian too, as well as any other Debian derivative. The missing piece is what to do when apt-cache search doesn't find something.
$ sudo apt-get install dh-make-perl build-essential apt-file
$ sudo apt-file update
Then whenever you have a random module you wish to install:
$ cd ~/some/path
$ dh-make-perl --build --cpan Some::Random::Module
$ sudo dpkg -i libsome-random-module-perl-0.01-1_i386.deb
This will give you a deb package that you can install to get Some::Random::Module. One of the big benefits here is man pages and sample scripts in addition to the module itself will be placed in your distro's location of choice. If the distro ever comes out with an official package for a newer version of Some::Random::Module, it will automatically be installed when you apt-get upgrade.
Already answered and accepted answer - but anyway:
IMHO the easiest way installing CPAN modules (on unix like systems, and have no idea about the wondows) is:
curl -L http://cpanmin.us | perl - --sudo App::cpanminus
The above is installing the "zero configuration CPAN modules installer" called cpanm. (Can take several minutes to install - don't break the process)
and after - simply:
cpanm Foo
cpanm Module::One
cpanm Another::Module
Many times it does happen that cpan install command fails with the message like
"make test had returned bad status, won't install without force"
In that case following is the way to install the module:
perl -MCPAN -e "CPAN::Shell->force(qw(install Foo::Bar));"
Lots of recommendation for CPAN.pm, which is great, but if you're using Perl 5.10 then you've also got access to CPANPLUS.pm which is like CPAN.pm but better.
And, of course, it's available on CPAN for people still using older versions of Perl. Why not try:
$ cpan CPANPLUS
Use cpan command as cpan Modulename
$ cpan HTML::Parser
To install dependencies automatically follow the below
$ perl -MCPAN -e shell
cpan[1]> o conf prerequisites_policy follow
cpan[2]> o conf commit
exit
I prefer App::cpanminus, it installs dependencies automatically. Just do
$ cpanm HTML::Parser
On ubuntu most perl modules are already packaged, so installing is much faster than most other systems which have to compile.
To install Foo::Bar at a commmand prompt for example usually you just do:
sudo apt-get install libfoo-bar-perl
Sadly not all modules follow that naming convention.
On Fedora Linux or Enterprise Linux, yum also tracks perl library dependencies. So, if the perl module is available, and some rpm package exports that dependency, it will install the right package for you.
yum install 'perl(Chocolate::Belgian)'
(most likely perl-Chocolate-Belgian package, or even ChocolateFactory package)
Even it should work:
cpan -i module_name
2 ways that I know of :
USING PPM :
With Windows (ActivePerl) I've used ppm
from the command line type ppm. At the ppm prompt ...
ppm> install foo
or
ppm> search foo
to get a list of foo modules available. Type help for all the commands
USING CPAN :
you can also use CPAN like this (*nix systems) :
perl -MCPAN -e 'shell'
gets you a prompt
cpan>
at the prompt ...
cpan> install foo (again to install the foo module)
type h to get a list of commands for cpan
On Fedora you can use
# yum install foo
as long as Fedora has an existing package for the module.
Easiest way for me is this:
PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install DateTime::TimeZone'
a) automatic recursive dependency detection/resolving/installing
b) it's a shell onliner, good for setup-scripts
If you want to put the new module into a custom location that your cpan shell isn't configured to use, then perhaps, the following will be handy.
#wget <URL to the module.tgz>
##unpack
perl Build.PL
./Build destdir=$HOME install_base=$HOME
./Build destdir=$HOME install_base=$HOME install
Sometimes you can use the yum search foo to search the relative perl module, then use yum install xxx to install.
Secure solution
Many answers mention the use of the cpan utility (which uses CPAN.pm) without a word on security. By default, CPAN 2.27 and earlier configures urllist to use a http URL (namely, http://www.cpan.org/), which allows MITM attacks, thus is insecure. This is what is used to download the CHECKSUMS files, so that it needs to be changed to a secure URL (e.g. https://www.cpan.org/).
So, after running cpan and accepting the default configuration, you need to modify the generated MyConfig.pm file (the full path is output) in the following way. Replace
'urllist' => [q[http://www.cpan.org/]],
by
'urllist' => [q[https://www.cpan.org/]],
Note: https is not sufficient; you also need a web site you can trust. So, be careful if you want to choose some arbitrary mirror.
Then you can use cpan in the usual way.
My bug report on rt.cpan.org about the insecure URL.
Simply executing cpan Foo::Bar on shell would serve the purpose.
Seems like you've already got your answer but I figured I'd chime in. This is what I do in some scripts on an Ubuntu (or debian server)
#!/usr/bin/perl
use warnings;
use strict;
#I've gotten into the habit of setting this on all my scripts, prevents weird path issues if the script is not being run by root
$ENV{'PATH'} = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin';
#Fill this with the perl modules required for your project
my #perl = qw(LWP::Simple XML::LibXML MIME::Lite DBI DateTime Config::Tiny Proc::ProcessTable);
chomp(my $curl = `which curl`);
if(!$curl){ system('apt-get install curl -y > /dev/null'); }
chomp(my $cpanm = system('/bin/bash', '-c', 'which cpanm &>/dev/null'));
#installs cpanm if missing
if($cpanm){ system('curl -s -L http://cpanmin.us | perl - --sudo App::cpanminus'); }
#loops through required modules and installs them if missing
foreach my $x (#perl){
eval "use $x";
if($#){
system("cpanm $x");
eval "use $x";
}
}
This works well for me, maybe there is something here you can use.
On Windows with the ActiveState distribution of Perl, use the ppm command.

Remove older Hyperledger-sawtooth or pull latest repo and rerun build_all?

I had previously brought down 0.8 and want to use new version.
Is it ok to update local repo and 'build_all' or must I remove all the older docker images first?
This may be brute force, but this is what I ended up doing.
Caution, the docker command will take out all images so if you want to
preserve some of them you may want a more selective approach.
Sawtooth platform
Remove all docker images using this command docker rmi -f $(docker images -a -q)
Bring down the latest sawtooth compose file sawtooth-default.yaml
Execute compose docker-compose -f sawtooth-default.yaml up
Sawtooth repo development
Clone the latest repository
Go to the root directory of the repo cd ~\sawtooth-core
At a minimum do bin\build_all -l python
I am using java so I do a bin\build_all -l java as well
Access to individual CLI and dev languages tested out 100% as per the Hyperledger Sawtooth documentation

NaCl helper process running without a sandbox! error when running npm tests

I'n having a problem when running some npm test. The error I'm receiving is: "NaCl helper process running without a sandbox!", which is true, as I'm running the browser with the "--no-sandbox" option. I have to run this option due to the fact that the browser runs as root, and I don't have an option to run it a different user at all(it's a docker image).
Can anyone please help me to sort it out?
P.S I'm installing the browser in the following way:
RUN apt-get update
RUN apt-get install -y nodejs npm
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb https://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
RUN apt-get install -y apt-transport-https
RUN apt-get update
RUN apt-get install -y google-chrome-stable
Thanks in advance!
This error message...
NaCl helper process running without a sandbox!
...implies that you have no setuid sandbox in your system, hence the program was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session.
Solution
A quick solution will be, if you want to run Chrome and only use the namespace sandbox, you can set the flag:
--disable-setuid-sandbox
This flag will disable the setuid sandbox (Linux only). But if you do so on a host without appropriate kernel support for the namespace sandbox, Chrome will not spin up. As an alternative you can also use the flag:
--no-sandbox
This flag will disable the sandbox for all process types that are normally sandboxed.
Example:
chromeOptions: {
args: ['--disable-setuid-sandbox', '--no-sandbox']
},
You can find a detailed discussion in Security Considerations - ChromeDriver - Webdriver for Chrome
Deep dive
As per the documentation in Linux SUID Sandbox Development google-chrome needs a SUID helper binary to turn on the sandbox on Linux. In majority of the cases you can install the proper sandbox for you using the command:
build/update-linux-sandbox.sh
This program will install the proper sandbox for you in /usr/local/sbin and tell you to update your .bashrc if required.
However, there can be some exceptions as an example, if your setuid binary is out of date, you will get messages such as:
Running without the SUID sandbox!
Or
The setuid sandbox provides API version X, but you need Y
You are using a wrong version of the setuid binary!
In these cases, you need to:
Build chrome_sandbox whenever you build chrome (ninja -C xxx chrome chrome_sandbox instead of ninja -C xxx chrome)
After building, execute update-linux-sandbox.sh.
# needed if you build on NFS!
sudo cp out/Debug/chrome_sandbox /usr/local/sbin/chrome-devel-sandbox
sudo chown root:root /usr/local/sbin/chrome-devel-sandbox
sudo chmod 4755 /usr/local/sbin/chrome-devel-sandbox
Finally, you have to include the following line in your ~/.bashrc (or .zshenv):
export CHROME_DEVEL_SANDBOX=/usr/local/sbin/chrome-devel-sandbox
If you are using karma to run your tests, make sure you are using ChromeHeadless as the browser on karma.conf.js

how to build grub2 bootloader from it's source and test it with qemu emulator

I want to know how to build grub 2 bootloader from it's source in ubuntu and test it with qemu emulator.
I would also like to change the default background image of grub2 bootloader in the new build?
Is this possible? If yes, how ?
Of course you can.
As shown on the GRUB website, the grub source code is available via git from git.savannah.gnu.org.
Then it is theoretically just a question of
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install
However, depending on your platform, grub's default target platform may or may not be what you want. So you will need to decide which firmware platform you want to use in QEMU, which depending on your architecture can be something like
(pc) BIOS
coreboot
(U)EFI
ieee1275 (open firmware)
u-boot
Your mentioning of Ubuntu matches at least 3 possible options from the above, but I'm going to be boring and assume you mean x86_64/amd64. Since you will be running GRUB under QEMU, it does not really matter which of the two likely platforms ("pc" or "efi") your physical computer is running. So let's live a little and go for the (U)EFI variant.
You will need some prerequisites installed before configuring and building, so
$ sudo apt-get install build-essential autoconf automake
$ sudo apt-get build-dep grub-efi-amd64
So a practical build may look a bit like this:
$ # Next command is optionnal (languages):
$ ./linguas.sh
$ ./autogen.sh
$ # Next parameters are optionnal:
$ ./configure --prefix=$HOME/local --platform=efi
$ make
$ # Next command is optionnal:
$ make check
$ make install
The easiest way to get a functioning GRUB image is probably with the grub-mkstandalone command:
$ $HOME/local/bin/grub-mkstandalone -O x86_64-efi -o mygrub.efi
Note: To install grub on /dev/sda disk (instead of QEMU), use:
$ sudo grub-install /dev/sda
Note: If you don't see GRUB menu when booting, check this question. It involves pressing Shift when booting or editing /etc/default/grub to comment GRUB_HIDDEN_TIMEOUT.
Then you need some kind of UEFI image to run under your QEMU. The default choice for x86 is called OVMF and is part of Tianocore EDK2 - the de facto open source implementation of UEFI. Due to legal technicalities with regards to redistribution of the FAT filesystem driver, many Linux distributions (including Ubuntu) do not include a pre-built one. But have no fear, it is pretty straightforward to build one yourself.
However, I am not going to take this answer further off-topic than I already have, so all I am going to say is have a read through the OVMF README and look through one or two only slightly outdated blog posts about it.

How to install Cudnn from command line

Cudnn: https://developer.nvidia.com/cudnn
I login and go jump through all the hoops that NVIDIA wants you to do; however, when it comes time to download the file I can't seem to figure out how to do it via wget and the command line.
I was hoping someone has done this. I've copy and pasted the link that they want to click and used this in wget copy-and-pasted-url. But I just get back an html file.
The following trick works with Firefox:
Download the file with your regular machine
Go to the downloads list in firefox, right click on the file and click "Copy original download URL"
Go to your pure-terminal machine, and type:
wget PASTE-YOUR-LINK-FROM-FIREFOX
As #deltheil mentionned, by doing this the link contains a temporary download token, letting you download the file from another machine then the one it was requested from
EDIT
The downloaded filename is libcudnn***.deb?<some download token>. You will need to rename it by stripping the ? and everything after it:
mv libcudnn***.deb?xxx libcudnn***.deb
CUDNN_TAR_FILE="cudnn-8.0-linux-x64-v6.0.tgz"
wget http://developer.download.nvidia.com/compute/redist/cudnn/v6.0/${CUDNN_TAR_FILE}
tar -xzvf ${CUDNN_TAR_FILE}
sudo cp -P cuda/include/cudnn.h /usr/local/cuda-8.0/include
sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda-8.0/lib64/
sudo chmod a+r /usr/local/cuda-8.0/lib64/libcudnn*
The download link that you get right after the accept terms section is authenticated (the GET request gives you a HTTP 302 Moved Temporarily).
If you really want to grab the link from the command line: open your browser, use the developers tools and look at the Location field after the redirection: you can use this link directly with wget as it contains a short-lived authorization token.
You may try the following:
curl -O http://developer.download.nvidia.com/compute/redist/cudnn/v2/cudnn-6.5-linux-x64-v2.tgz
This will download CUDNN 6.5
The location for the latest one is in the NVIDIA latest Docker file, currently at:
https://github.com/NVIDIA/nvidia-docker/tree/master/centos-7/cuda/7.5/runtime/cudnn5
I tried all of these answers and none worked unfortunately.
Though a simple workaround is: apt-get update && apt-get install -y --no-install-recommends <cuDNN Package>
For example, in my case it is:
apt-get update && apt-get install -y --no-install-recommends libcudnn8
And it works!