Query tables in Apache Drill without adding in extensions to the table name - apache-drill

Question

I have an Apache Drill setup with the following storage plugin
{
"type": "file",
"connection": "file:///",
"config": null,
"workspaces": {
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
},
"csv": {
"location": "/home/user/data/csv",
"writable": false,
"defaultInputFormat": "csv",
"allowAccessOutsideWorkspace": false
},
"parquet": {
"location": "/home/user/data/parquet",
"writable": false,
"defaultInputFormat": "parquet",
"allowAccessOutsideWorkspace": false
}
},
"formats": {
"csv": {
"type": "text",
"extensions": [
"csv"
],
"skipFirstLine": true,
"extractHeader": true,
"delimiter": ","
},
"parquet": {
"type": "parquet",
"autoCorrectCorruptDates": false
}
},
"enabled": true
}
The name has been configured to foo.
The issue is that I wanted to write a query where the name of the table does not have the extension.
I tried the following,
select * from foo.csv.`agency` limit 100
I get the following response
org.apache.drill.common.exceptions.UserRemoteException: VALIDATION ERROR: From line 1, column 15 to line 1, column 21: Object 'agency' not found within 'foo.csv' [Error Id: 80be4497-b71c-47dd-bc2c-6abfa425d55a on nn-hadoop-1:30112]
But this works
select * from foo.csv.`agency.csv` limit 100
Is there a way for me to not suffix the table name (file name) with the file extension when I create my query? I have included the defaultInputFormat in the workspace

Answer 1

I found a way to resolve this issue. Instead of querying files directly, I put the file I wanted to query into a directory and queried the directory instead.
https://drill.apache.org/docs/querying-directories/
So, now I have created a folder called agency and moved the file agency.csv is inside this folder. Now I can do
select * from foo.csv.`/agency` limit 100
and I am getting the results I was looking for.

Answer 2

You can query the table or file.
For the table please create it:
select * from foo.csv.`agency.csv`
and query:
select * from agency;
For the second case, Drill should have a full path to the file or directory (including workspace, if you use it)

Answer 3

Another option is to use a view; first use a writable workspace (like /tmp above)
use foo.tmp;
Then create the view
create view agency as select * from foo.csv.`agency.csv`;
Unlike creating a table, a view does not take a significant disk space, and would reflect the latest data in the file (in case the file is updated.)
The usage is the same, like
select * from agency limit 100;