With a Cloudera install of HBase, I saw three places have config information :
/etc/hbase/conf/hbase-site.xml,
/usr/lib/hbase/conf/hbase-site.xml,
and /var/run/cloudera-scm-agent/process/*-hbase-MASTER
Which one exactly is in effect? Or maybe all of them do?
In all cases of hbase the /etc/hbase/conf/hbase-site.xml file is always read. The /usr/lib/hbase/conf/hbase-site.xml is a symlink to /etc/hbase/conf/hbase-site.xml so it is the same file.
Lastly, anything in /var/run/ is a runtime variable and in your case it is the Cloudera Manager Agent. The Manager Agents are responsible for the management console and logging amongst other tasks.
I hope that helps,
Pat
The config file used is /usr/lib/hbase/conf/hbase-site.xml.
Other files aren't symbolic links.
Since the same configuration information needs to be used for other processes like HMaster,RegionServer, /usr/lib/hbase/conf/hbase-site.xml file is synced at different locations while initializing/preprocessing of these daemons. Hence it is advised to make any configuration changes in /usr/lib/hbase/conf/hbase-site.xml file only.
Also you need to make the same changes to these file on all nodes in your cluster and restart the HBase daemons.
I hope these answer your question.
Per my search and learning, HBase actually has two types of hbase-site.xml files, one for HMaster/RegionServer, and the other for client.
In Cloudera's distribution, the hbase-site.xml file in folder /var/run/cloudera-scm-agent/process/*-hbase-MASTER is the config used by the running HMaster process. Similar is for RegionServer.
Yet the site.xml file under /usr/lib/hbase/conf/ and /etc/hbase/conf/, symlinked from one to the other (according to #apesa), is for client usage. If one starts HBase shell on an HMaster host or a RegionServer host, this client config file will be used so the shell application knows how to connect to the ZooKeeper quorum to obtain the running HBase service. If one wants to use the HBase service from a client host, then he needs to copy this client xml file to the client host.
For regular Apache installation of HBase, as was indicated in Sachin's answer, the same hbase-site.xml is used for both purposes, though the HMaster, the RegionServer, and the client processes will use only the options needed and ignore the rest.
From experimenting with the hbase binary version 1.2.0-cdh5.16.1, it appears to use the Java classpath to find the hbase-site.xml file to use, whether running as a server or a client shell. There is a configuration parameter (--config) you can pass to hbase to control the config directory used, which by default is ./conf (run hbase to view the documented help on this).
This observation is supported by other answers on this topic (e.g. Question 14327367).
Therefore, to answer your specific question, to determine which config file is used on your machine, run hbase classpath and find which of the 3 directories appears first in the classpath.
Related
I'm trying to write a monitoring plugin for Artifactory using MySQL. The idea is that the plugin will collect some data using the Artifactory public API and some command line utilities, then write that data to a MySQL database to be analysed later. The problem is that the plugin can't find the jdbc driver. I placed the driver JAR in <artifactory-home>/etc/plugins/lib and I'm registering it like so:
def sql = Sql.newInstance('jdbc:mysql://localhost:3306/artifactoryTest', <user>, <pass>, 'com.mysql.jdbc.Driver')
This results in a ClassNotFoundException on com.mysql.jdbc.Driver. It seems the class loader used by Artifactory can't find the JAR.
Is there a way to make this work or am I barking up the wrong tree? I'm aware there may be some database technologies that would be better suited for this purpose but many seem like overkill for such a basic use case and I'd like to keep it as simple as possible.
Turns out this is indeed possible. Artifactory can be configured to use a number of different databases, including MySQL, so I figured there must be a way to use the same driver that Artifactory uses and indeed there is.
In the documentation for setting up Artifactory with MySQL it says:
Download the MySQL JDBC driver (available from the MySQL website)
and copy the mysql-connector-java-.jar file into the server's
shared lib directory. For example $TOMCAT_HOME/lib when installed as
a service or $ARTIFACTORY_HOME/tomcat/lib in the standalone version.
Simply dropping the JDBC driver JAR in the shared lib directory makes it available for use within plugins. I imagine this approach would work for any other JARs too.
I am trying to monitor a database using this template https://share.zabbix.com/databases/mysql/template-mysql-800-items, the instructions do not indicate where I should place each file and modify each configuration, does anyone have experience with this?
It's a bit of a large scale question, but in general, the XML file is imported in the Zabbix frontend, while the Perl script goes in /opt/mysql_check.pl (or wherever you point at in the userparameter entry in the agent daemon configuration file).
I'm new to Couchbase. Does anyone know how to connect between Couchbase (CB) servers in order to migrate data? I want to migrate data from a production CB server to my local server.
I found here how to migrate between buckets, but within the same server.
Also, I could do the migration between different servers using a backend application (I'm using C#.Net) using N1QL, but I want to learn how to do the server to server migration since it is a pretty standard and common operation .
Many thanks in advance.
Do you need continuous migration? Or backup/restore would be enough?
If latter, you can use cbbackup/cbrestore tools http://developer.couchbase.com/documentation/server/current/backup-restore/backup-restore.html
Either cbbackup and/or cbrestore or just turning on XDCR to get the changes from production to your local server would work too.
Definitely some security implications on doing this, but that's up to you to figure out! :)
Thanks to #Ade Stringer who finally gave me the best solution, which is to use the cbtransfer tool. This tool simply needs the source and target urls of the servers (and the names of the buckets), which is ideal since -- in general-- one doesn't have access to the file system of the servers.
Here is a sample call:
cbtransfer http://10.10.10.68:8091 http://localhost:8091 -b SourceBucketName --bucket-destination TargetBucketName
Note that the first parameter is the source CB server (http://10.10.10.68:8091) and the second one is the target CB server (http://localhost:8091). The value of the -b parameter is the name of the source bucket and the value of the --bucket-destination parameter is the name of the target bucket.
In order to run this command in Windows, you must first go to the following folder:
C:\Program Files\Couchbase\Server\bin\
If one prefers to use the cbbackupmgr and cbbackup tools which were mentioned in the other answers, one needs to have access to the file system, which in my case was not possible. But still both tools are also useful nevertheless and I appreciate their answers.
MySQL provides a script for initialiazing a new data directory for databases storage.
Unfortunately, on the noinstall zip for Windows, it is only a Perl script.
I'd like to initialize a new data directory without using Perl, since I'm building an auto-install to be launched from several machines and I want to reduce the softwares installation duty.
Is there a workaround for this script that does not require to have Perl installed ?
As stated on the MySQL doc, you can just copy the data dir from the MySQL zip wherever you want.
Then you will just need either launch mysqld using the --datadir=new_path option, or specify datadir=new_path in the my.ini/my.cnf you'll be using.
I am having two mysql instances on same machine. The installations are on /usr/loca/mysql1 and /usr/local/mysql2.
I m having separate my.cnf files located in /etc/mysql1 and /etc/mysql2. I installed the first instance of my sql using source distribution and with the --prefix=/usr/local/mysql1 option. The second one i got from copying and pastinf the same directory to /usr/local/mysql2.
When i start the mysql daemon on /usr.local/mysql/libexec it reads the my.cnf file in /etc/mysql1. And if i start the mysql daemon in /usr/local/mysql2 it reads the same my.cnf file. I have separate port numbers and .sock files defined in the .cnf file in those 2 locations.
I can read the my.cnf file in the second location by using --defaults-file=/etc/mysql2/my.cnf option on mysqld startup. I do not need to enter this each and every time i start the daemon.
If i am going to have more instances how can i point the correct my.cnf file to read to each and every mysql daemon. What is the retionale behind mysqld links with the my.cnf file.
how can i predefine the location of my.cnf file for each instance.
As you've already discovered, MySQL has a compiled-in search location for its configuration file. Although you could recompile yourself, changing this, you've also discovered the --defaults-file option to mysqld, which instructs it to use an entirely different configuration path. Coupled with --data-dir, this means you can start multiple instances of MySQL bound to different ports (and addresses, if liked) and working with entirely separate sets of data, while working off the same binaries and libraries.
Traditionally, most operating system distributions will bundle a single init script for starting the "default" instance of MySQL; that is, the one installed in the "usual" location, and with the standard configuration path. Although this is to cater for the commonest case, what you're after is a bit different, so you'll need to create separate scripts to launch the separate instances.
If you're planning on deploying a lot of MySQL instances on the same machine (and I'd have to ask why), then you may want to write a custom init script which has some way of "discovering" each of these (perhaps by inspecting some directory containing a "common" layout), and then loops over them, starting each one up. Of course, the same init script then needs to be capable of locating and properly shutting down each one.
I think the only sure way is to compile from source with the prefix option. Either that, or create a bash script that starts mysql2 with the --defaults-file option and then use that to start it instead.
However, "creating" another installation the way you did isn't good. When you compile a package, the PREFIX is hard-coded into the executable. So whenever it looks for a resource, it starts from that prefix, unless told not to with command line options.
So, if I were you, I'd just recompile from source with a new prefix.