How to query a whole S3 directory in Apache Drill? - apache-drill

I'm trying to query a whole directory in S3 containing parquet files. The query hangs for a while, then returns an error:
0: jdbc:drill:zk=local> select * from s3.`/data/dt=2015-10-15` limit 10;
Error: CONNECTION ERROR: Connection /13.14.15.254:54549 <--> /13.14.15.254:31010 (user client) closed unexpectedly.
I can see my data folder when running these commands:
USE s3.`root`
SHOW FILES
Thanks!

Related

Heroku - Can't access or create database JSON data of ruby app, error HTTP 500

I'm having a problem in Production when accessing the JSON (let's call it mydata.json) where I store the data of my ruby webapp, deployed with Heroku. The way I access the download of this file is by putting into the browser:
my-heroku-page.herokuapp.com/mydata.json
but I get the HTTP 500 error page.
Always using the browser, when I try to create it I get the Ruby page:
We're sorry, but something went wrong.
If you are the application owner check the logs for more information.
First I must say I'm using the same source code I used for another webapp that is actually working, I just modified the database.yml according to my new host, database, username and password. The previous webapp was perfectly working, i.e I could create the table and access the data. Secondly the error doesn't occur with localhost.
I tried to create a Dataclip in Heroku:
Select * from mydata order by created_at desc;
but I get the error:
"Your query couldn't be created."
ERROR: relation "mydata" does not exist
LINE 3: FROM mydata
Plus when I check the heroku pg:info I get 0 Tables
=== DATABASE_URL
Plan: Hobby-dev
Status: Available
Connections: 0/20
PG Version: 9.5.5
Created: 2016-11-15 08:30 UTC
Data Size: 7.4 MB
Tables: 0
Rows: 0/10000 (In compliance)
Fork/Follow: Unsupported
Rollback: Unsupported
Add-on: postgresql-convex-54172
It seems like mydata.json is not created in production, but in localhost is working fine and I can create/download a blank one. I'm sure I'm missing something easy here, maybe in the database.yml.
I will edit the question if additional info are required. Any help is appreciated.
Thanks,
Simone

Local BLAST Swissprot Database error

I am trying to run the standalone ncbi-blast-2.2.28+ on my machine (Mac) but get this error message when running blastp with SwissProt database:
BLAST Database error: Could not find volume or alias file (nr.00) referenced in alias file (/Users/me/bin/db/swissprot.00).
Here what I did:
1) downloaded the "ncbi-blast-2.2.28+-universal-macosx.tar.gz" from ncbi server and decompressed it
2) move the bin content of the folder to my $PATH directory "/Users/me/bin"
3) In "/Users/me/bin" I created a "db" folder, plus the ".ncbirc" file containing the following path:
[BLAST]
BLASTDB=/Users/me/bin/db
4) I downloaded the SwissProt database and got the following files in "/Users/me/bin/db/:
swissprot.00.msk
swissprot.01.msk
swissprot.02.msk
swissprot.03.msk
swissprot.04.msk
swissprot.05.msk
swissprot.06.msk
swissprot.07.msk
swissprot.08.msk
swissprot.09.msk
swissprot.10.msk
swissprot.00.pal
swissprot.01.pal
swissprot.02.pal
swissprot.03.pal
swissprot.04.pal
swissprot.05.pal
swissprot.06.pal
swissprot.07.pal
swissprot.08.pal
swissprot.09.pal
swissprot.10.pal
swissprot.pal
Then when I run blastp from any working directory (where my query file is), using this command:
blastp -query input.fasta -db swissprot
I get the following error message:
BLAST Database error: Could not find volume or alias file (nr.00) referenced in alias file (/Users/me/bin/db/swissprot.00).
As I read on other threads, I also tried to mention in the command line the whole path where the db is located, and to remove the .pal extension from the file names. But still doesn't work.
Can someone sees what I did wrong ?!!!!
you are storing your database files in db folder so you have to give this command instead of the one you have used:
blastp -query input.fasta -db db/swissprot
and I believe you are looking for an output in the console itself as you haven't used the -out option.
Also this will work only if the bin directory in which db folder is present be declared as an environment variable.
Have you checked the paths in .pal file?
Swissprot database that you have downloaded contains only links to entries in nr database: "nr - Non-redundant GenBank CDS translations + PDB + SwissProt + PIR + PRF, excluding those in env_nr". So you should additionally download nr database to run the standalone blast on your machine with SwissProt database. It weighs about 20 (!) Gb, but without it your blast will not work. Here's a link: ftp://ftp.ncbi.nlm.nih.gov/blast/db/
place all files from 00 to 10 folders into db and then check .pal file should contain 00 to 10 parts for example for nr databas its like
"nr.00" "nr.01" "nr.02" "nr.03" "nr.04" "nr.05" "nr.06" "nr.07" "nr.08" "nr.09" "nr.10"

SOURCE error 2?

When i try to source an sql file i get the error:
mysql> source C:/Users/tom/Documents/insert.sql
ERROR:
Failed to open file 'C:/Users/tom/Documents/insert.sql', error: 2
I have checked the file path, which looks fine to me. I have also tried \. C:/Users/etc
I am trying to source the sql file which holds insert statements for particular tables. All the statements in the file work when entered manually. What else could i be doing wrong?
Have tried using both backslash and forward slash when using this command
Probably a problem of access right on the file (the file is being accessed by the mysqld server process, not yourself). Try placing the file into the data folder of MySQL, then import it from this location. The location of data folder depends on your distribution and on your own configuration.
Alternatively, feed the SQL script directly to your mysql client's stdin:
mysql [all relevant options] your_database < C:\path\to\your\script.sql
I am using Ubuntu 14.04 version.
I too faced below error 2.
mysql> SOURCE home/loc/Downloads/AllTables.sql;
Failed to open file 'home/loc/Downloads/AllTables.sql', error: 2
Solution :
mysql> SOURCE /home/loc/Downloads/AllTables.sql;
Just added a '/' in front of home
Hope this helps some one.
Have you checked if the file exits? I have had this problem before.
This:
this:
and this works:

where to specify the root directory of hadoop on slave node?

I need to setup a hadoop/hdfs cluster with one namenode and two datanodes. I am aware of conf/slaves file which lists the machines datanodes are running. But how can I specify where hadoop/hdfs is locally installed on slave node? Also the user account to start hdfs there?
Edit: in log files, I find following error, when I tried to start-dfs.sh
ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: file:///
The user is expected to be the same as on the master node. The location of the actual data can be modified by changing the dfs.data.dir node inhadoop-site.xml.

Importing CSV to MySQL table returns an error #1148

I am trying to import with DirectAdmin, when I selected CSV without using LOAD DATA - I got the error "Invalid field count in CSV input on line 1."
When I tried with LOAD DATA I got the following error: "#1148 - The used command is not allowed with this MySQL version."
The CSV was created in MS Access from MS Access database.
Here are the first 2 rows:
"product_id","vendor_id"," product_parent_id","product_sku","product_s_desc ","product_desc","product_thumb_image ","product_full_image","product_publish","product_weight","product_weight_uom ","product_length ","product_width","product_height ","product_lwh_uom ","disp_order","price","sale","product_url ","product_in_stock","product_available_date","product_availability ","product_special ","product_discount_id ","ship_code_id ","cdate ","mdate ","product_name ","product_sales ","attribute ","custom_attribute ","product_tax_id ","product_unit ","product_packaging ","child_options ","quantity_options ","child_option_ids ","product_order_levels "
41,2,0,1,,,"resized/Krug-Rose-Champagne-lg.jpg","Krug-Rose-Champagne-lg.jpg","Y","750.0000","grams","4.0000","4.0000",14,,14,3516,0,,,1296518400,,"N",0,"NULL ",1296574622,1297953843,"קרוג רוזה",0,,,2,"piece ",65537,"N,N,N,N,N,Y,20%,10%, ","none,0,0,1 ",,"0,0 "
From mysql command line pass the following parameter:
mysql -u username -p dbname --local-infile
Instead of using: load data infile, use: load data local infile and it should perform the import.
By default, mysql does not enable load data local as per the the security concerns defined here:
http://dev.mysql.com/doc/refman/5.0/en/load-data-local.html
If LOAD DATA LOCAL is disabled, either in the server or the client, a
client that attempts to issue such a statement receives the following
error message:
ERROR 1148: The used command is not allowed with this MySQL version