According to Drill's documentation :
Drill uses /tmp/drill/sys.storage_plugins to store storage plugin configurations. The temporary directory clears when you quit the Drill shell. To save your storage plugin configurations from one session to the next, set the following option in the drill-override.conf file if you are running Drill in embedded mode.
drill.exec.sys.store.provider.local.path = "/mypath"
I checked /tmp/drill/sys.storage_plugins, there is some data in this file. Then I modified drill-override.conf :
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "localhost:2181",
drill.exec.sys.store.provider.local.path = "/home/dev/abc"
}
I restarted drill & even restarted my machine. Nothing is created at this location.
Note: I am using ubuntu 14.0.4 & Drill version - 1.3. Drill is running in embedded mode
Any pointer on this.
I raised an issue regarding this. It is resolved.
Use sys.store.provider.local.path instead of drill.exec.sys.store.provider.local.path because it's already in drill.exec
Modified drill-override.conf :
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "localhost:2181",
sys.store.provider.local.path = "/home/dev/abc"
}
Use this..
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "localhost:2181",
sys.store.provider.local.path="",
Related
I am trying to start Apache Drill 1.10 in Embedded Mode on Windows 10 x64 (with Oracle JVM 1.8.0_131). When launching the command
sqlline.bat -u "jdbc:drill:zk=local"
I get the following:
Error during udf area creation [/C:/Users/<user>/drill/udf/registry] on file system [file:///] (state=,code=0)
So, after some googling, I have changed the drill-override.conf file this way:
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "localhost:2181",
udf: {
# number of retry attempts to update remote function registry
# if registry version was changed during update
retry-attempts: 10,
directory: {
# Override this property if custom file system should be used to create remote directories
# instead of default taken from Hadoop configuration
fs: "file:///",
# Set this property if custom absolute root should be used for remote directories
root: "/c:/work"
}
}
}
Then I have checked the following:
proper permission set on the folder
console started as an Administrator
But I still get the same error:
Error during udf area creation [/c:/work/drill/udf/registry] on file system [file:///] (state=,code=0)
I can't disable UDF since I don't have an active connection.
Any suggestions?
Seems to be related to ownership of the folders, as per this link.
Details of the solution from the link are quoted as follows
Run these commands before the first time you are running sqlline.bat.
mkdir %userprofile%\drill
mkdir %userprofile%\drill\udf
mkdir %userprofile%\drill\udf\registry
mkdir %userprofile%\drill\udf\tmp
mkdir %userprofile%\drill\udf\staging
takeown /R /F %userprofile%\drill
I have a scenario where I need to change several parameters of a hadoop cluster managed by Ambari to document performance of a particular application. The change in the configs entails a restart of the affected components.
I am using the Ambari REST API for achieving this. I figured out how to do this for all service components of hadoop. I' am not sure whether the API provides a way to restart the MySQL server that Hive uses.
I have the following questions:-
Is it the case that a mere stop and start of mysqld on the appropriate machine is enough to ensure that the required configuration changes are recognized by Ambari and the application?
I chose the 'New MySQL database' option while installing Hive via Ambari. Does this mean that restarts are reflected in Ambari only when it is carried out from the Ambari UI?
Your inputs would be highly appreciated.
Thanks!
Found a solution to the problem. I used the following commands using the Ambari REST API for changing configurations and restarting services from the backend.
Login to the host on which the ambari server is running and use the already provided config.sh script as described below.
Modifying configuration files
#!/bin/bash
CLUSTER_NAME=$1
CONFIG_FILE=$2
PROPERTY_NAME=$3
PROPERTY_VALUE=$4
/var/lib/ambari-server/resources/scripts/configs.sh -port <ambari-server-port> set localhost $1 $2 "$3" "$4"
where CONFIG_FILE can take values like tez-site, mapred-site, hadoop-site, hive-site etc. PROPERTY_NAME and PROPERTY_VALUE should be set to values relevant to the specified CONFIG_FILE.
Restarting host components
curl -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d '
{
"RequestInfo":{
"command":"RESTART",
"context":"Restart MySQL server used by Hive Metastore on node3.cluster.com and HDFS client on node1.cluster.com",
"operation_level":{
"level":"HOST",
"cluster_name":"c1"
}
},
"Requests/resource_filters":[
{
"service_name":"HIVE",
"component_name":"MYSQL_SERVER",
"hosts":"node3.cluster.com"
},
{
"service_name":"HDFS",
"component_name":"HDFS_CLIENT",
"hosts":"node1.cluster.com"
}
]
}' http://localhost:<ambari-server-port>/api/v1/clusters/c1/requests
Reference Links:
Restarting components
modifying configurations
Hope this helps!
I am trying to convert VMX to OVF format using OVFTool as below, however it gives error:
C:\Program Files\VMware\VMware OVF Tool>ovftool.exe
vi://vcenter.com:port/folder/myfolder/abc.vmx abc.ovf
Error: Failed to open file: https://vcenter.com:port/folder/myfolder/abc.vmx
Completed with errors
Please let me know if you have any solution.
I had a similar situation in vmware fusion trying to use a .vmx that was probably created on windows. I could boot the VM, but any attempt to export the machine with ovftool or use vmware-vdiskmanager bombed out with:
Error: Failed to open disk: source.vmdk
Completed with errors
the diskname was totally valid, path was valid, permissions were valid, and the only clue was running ovftool with:
ovftool --X:logToConsole --X:logLevel=verbose source.vmx dest.ova
Opening VMX source: source.vmx
verbose -[10C2513C0] Opening source
verbose -[10C2513C0] Failed to open disk: ./source.vmdk
verbose -[10C2513C0] Exception: Failed to open disk: source.vmdk. Reason: Disk encoding error
Error: Failed to open disk: source.vmdk
as others suggested, i took a peek in the .vmdk. therein i found 3 other clues:
encoding="windows-1252"
createType="monolithicSparse"
# Extent description
RW 16777216 SPARSE "source.vmdk"
so first i converted the monolithicSparse vmdk to "preallocated virtual disk split in 2GB files":
vmware-vdiskmanager -r source.vmdk -t3 foo.vmdk
then i could edit the "foo.vmdk" to change the encoding, which now looks like:
encoding="utf-8"
createType="twoGbMaxExtentFlat"
# Extent description
RW 8323072 FLAT "foo-f001.vmdk" 0
RW 8323072 FLAT "foo-f002.vmdk" 0
RW 131072 FLAT "foo-f003.vmdk" 0
and finally, after fixing up the source.vmx:
scsi0:0.fileName = "foo.vmdk"
profit:
ovftool source.vmx dest.ova
...
Opening VMX source: source.vmx
Opening OVA target: dest.ova
Writing OVA package: dest.ova
Transfer Completed
Completed successfully
I had a similar problem with OVFTool trying to export to OVF format.
Export failed: Failed to open file: C:\Virtual\test\test.vmx.
First, I opened .VMX file in editor (it's a text file) and made sure that settings like
scsi0:0.fileName = "test.vmdk"
nvram = "test.nvram"
extendedConfigFile = "test.vmxf"
mention proper file names.
Then I noticed this line:
.encoding = "windows-1251"
This is Cyrillic code page, so I modified it to use Western code page
.encoding = "windows-1252"
Then, running OVFTool gave a different error
Export failed: Failed to open disk: test.vmdk.
To fix it I had to open .VMDK file in HEX editor (because it's usually a big binary file), found there the string
encoding = "windows-1251"
(it's somewhere in the beginning of the file), and replaced "1251" with "1252".
And it did the trick!
In my case, was needed repair the disk 'abc.vmdk' before convert the 'abc.vmx' to 'abc.ovf'.
Use this for Linux:
$ /usr/bin/vmware-vdiskmanager -R /home/user/VMware/abc.vmdk
Look this link https://kb.vmware.com/s/article/2019259 for resolved issue in Windows and Linux
Try to run as described below.
C:\Program Files\VMware\VMware OVF Tool>ovftool C:\Win-Test\Win-Test.vmx(location of your vmx file) C:\Win-Test\win-test.ovf (destination)
Maybe ovftool is unable to recognize the path you are giving.
Try with following command:
ovftool --eula#=[path to eula] --X:logToConsole --targetType=OVA --compress=9 vi://[username]:[ESX address] [target address]
Once you provide the ESX address, it will list down the folders you have created in your ESX box. Then you can trigger the command above mentioned again with appending folder name.
If no folder hierarchy present in your box, then it will simply list down vm names.
Retry the same command appending [foldername]/[vmname no vmx file name required]
ovftool --eula#=[path to eula] --X:logToConsole --targetType=OVA --compress=9 vi://[username]:[ESX address]/[foldername if exist]/[vmname no vmx file name required] [target address]
I had this same exact issue. In my case I opened up the VMX file and dropped the IDE and sound controllers from the file and saved. I was then able to convert everything to an OVA using the tool with the standard syntax.
e.g. I dropped:
ide1:0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
and:
sound.present = "TRUE"
sound.fileName = "-1"
sound.autodetect = "TRUE"
This allowed me to convert the file like normal.
For me opening the .vmx and deleting the following line worked:
sata0:1.deviceType = "cdrom-image"
In my case, this works:
ide1:0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
I did change true to false and works fine, as cdrom-image not exist, this change permit the format conversion.
if your goal is to move a windows based vm to virtual box you only need to:
uninstall vmware tools from the guest vm
shut down the machine
copy the hd to a new folder
create a new empty vm in virtualbox
mount the hd (the .vmdk file) in that vm
Easy and rapid to do.
I am looking for someone that might know how to override the default install of MySql 5.5 on the Opsworks MySql layer.
I have tried enabling the ius repo and then a custom recipe to install the mysql56u-server, mysql56u-common packages, however all attempts thus far have failed due to the fact that mysql is being installed way earlier in the setup process.
I have not located the actual recipe that is selecting the packages for mysql55.
Anyone have any insight on this?
Any help much appreciated!
Looking at the mysql Cookbook provided by AWS, the recipe that installs the client (recipes/client_install.rb) includes the following:
case node[:platform]
when "redhat", "centos", "fedora", "amazon"
package mysql_name
else "ubuntu"
package "mysql-client"
end
The mysql_name variable is set earlier in the recipe:
mysql_name = node[:mysql][:name] || "mysql"
Looking at the attributes file (attributes/server.rb), the default values are set according to the Host's OS:
if rhel7?
default[:mysql][:name] = "mysql55-mysql"
else
default[:mysql][:name] = "mysql"
end
You can overwrite the name value to suit your needs:
default[:mysql][:name] = "mysql56u"
This can be achieved by provisioning your own customize attributes file in your Custom Cookbooks, or simply utilize the following Custom JSON in your Stack settings:
{
"mysql": {
"name": "mysql56u"
}
}
I tried to set-up a elasticsearch on my Windows 7 OS PC. Installed elasticsearch and curl and it's working as the loacahost:9200 is working fine.
Now I am strugging to search in a file located at c:\user\rajesh\raj.txt.
My doubt is, Where do mention that I have tos search in this file? elasticsearch.yml? Which parameter I need to set to point this text file?
Indexing is working with curl but mapping gives nullpointer exception? Do I need to install something else?
I tried to install sense plugin for chrome but says moved to marvel, and from there unable to install marvel!
From what I can tell, you've installed Elasticsearch and you're now expecting to be able to search within files on your local file system. This isn't how ES works. You need to create a mapping for an index and then populate that index with the content you want to search in. If you're looking to index files on your local file system rather than data you have pulled from a database you should look in to the File system River Plugin for Elasticsearch, http://www.pilato.fr/fsriver/. This deals with all of the indexing of file system based documents automatically, once you've got it set up correctly.
EDIT:
I also see you're trying to set up Kibana and Marvel/Sense. To set up Kibana just follow the instructions here: http://www.elasticsearch.org/overview/kibana/installation/
To set up Marvel open powershell, CD to C:\elasticsearch\bin then run plugin.bat -i elasticsearch/marvel/latest then you'll need to restart your cluster. Once you've done that if you go to http://localhost:9200/_plugin/marvel/ you'll see your marvel dashboard. You'll also see a tab for "Sense" which is the other plugin you referred to.
If you are using elastic search for retrieving data from any DB like PostgreSQL, then go to folder bin/rivers.bat and edit as
curl -XPUT localhost:9200/_river/actor_jdbc_river/_meta -d "{\"type\":\"jdbc\",\"jdbc\":{\"strategy\":\"simple\",\"poll\":\"1h\",\"driver\":\"org.postgresql.Driver\",\"url\":\"jdbc:postgresql://10.5.2.132:5432/prodDB\",\"user\":\"UserName\",\"password\":\"Password\",\"sql\":\"select t.id as _id,t.name from topic as t \",\"digesting\" : true},\"index\":{\"index\":\"jdbc\",\"type\":\"actor_jdbc_river1\"}}"
Then create a client in Java side to access data in river.
Here cluster name is same as that mention in folder config/elasticsearch.yml (testDBsearch)
private static Client createClient() {
//Create Client
Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "testDBsearch").build();
TransportClient transportClient = new TransportClient(settings);
transportClient = transportClient.addTransportAddress(new InetSocketTransportAddress("10.5.2.132", 9300));
return (Client) transportClient;
}
public static void main(String[] args) {
Client client = createClient();
String queryString = "python";
search(client, 100, queryString);
}
public static void search(Client client,int size, String queryString) {
queryString=queryString +"*";
try{
SearchResponse responseActor;
responseActor = client.prepareSearch("jdbc").setTypes("actor_jdbc_river1").setSearchType(SearchType.DEFAULT)
.setQuery(QueryBuilders.queryString(queryString)
.field("designation",new Float(2.0)).field("name", new Float(5.0)).field("email") .defaultOperator(Operator.OR)).setFrom(0).setSize(size).setExplain(true).execute().actionGet();
for(SearchHit hit:responseActor.getHits()) {
System.out.println(hit.getSourceAsString());
System.out.println(hit.getScore());
System.out.println("---------------------------");
}
}catch(Exception e){
System.out.println("Error in elastic search "+queryString+" Error :"+e);
}
}
clear installation of elasticsearch in windows:
1) check whether your system has latest java version
2) download and extract elasticsearch from "download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.3.3/elasticsearch-2.3.3.zip"
3) set JAVA_HOME environment variable "C:\Program Files (x86)\Java\jdk1.8.0_91"
4) check JAVA_HOME environment variable using command "service" in bin directry of elasticsearch shown in below figure checking whether JAVA_HOME is set properly or not
5) install service.bat using command service.bat install
6) uncomment network.host and give value as localhost in configuration file of elasticsearch
network.host= localhost in elasticsearch.yml (config file)
7)run elasticsearch "C:\elasticsearch-2.3.3\bin\elasticsearch"
if you get error while running elastic search saying update JVM to latest version means java in your system is not containing latest version (install and run latest java version)
8)install elasticsearch-head plugin to visualize things in elasticsearch
run command "plugin install elasticsearch-head"
if its failed to install elasticsearch-head then use command-
plugin install "github.com/mobz/elasticsearch-head/archive/master.zip"
9)open elasticsearch in browser using link "localhost:9200/_plugin/head/"
elasticsearch visual interface