How to use ucanaccess (ms access jdbc driver) in azure data bricks? - ms-access

Question

I am using azure data bricks and am trying to read .mdb files in as part of an ETL program. After doing some research, the only jdbc connector that i've found for ms access (.mdb) formats is "ucanaccess". I've followed some tutorials on azure on how to connect to a jdbc data source, and the connection at first appears successful, but there are some strange behaviors that don't make any sense.
For one, I cannot actually query the data frame because there are different data type errors. This happens for every table in the .mdb file.
connectionProperties = {
"driver" : "net.ucanaccess.jdbc.UcanaccessDriver"
}
url = "jdbc:ucanaccess:///dbfs/mnt/pre-processed/aeaton#legacydirectional.com/DD/DAILIES/5-1-19/MD190062.MDB"
df = spark.read.jdbc(url=url, table="tblbhaitems", properties=connectionProperties)
The result here is a data frame being returned
(data frame returned)
Now, trying to actually get data from the data frame, I get the following error:
df.select("*").show()
error: "org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 10.139.64.6, executor 0): net.ucanaccess.jdbc.UcanaccessSQLException: UCAExc:::4.0.4 incompatible data type in conversion: from SQL type CHARACTER to java.lang.Integer, value: Item No"
Looking at this error, I decide to try querying a specific string column to at least test other data types. When I perform
df.select("`Job no ID`").show()
I get the column name repeated for every row value of that table:
+---------+
|Job no ID|
+---------+
|Job no ID|
|Job no ID|
|Job no ID|
|Job no ID|
|Job no ID|
+---------+
I'm totally at a loss at why it's connecting and sees the columns but not actually fetching any data. Unfortunately, .mdb files aren't very common, so I feel my options are likely limited here in what I have available to parse the data.

Answer 1

I was facing similar issue like you mentioned when using spark with ucanaccess jdbc driver.
In spark we can create and register custom jdbc dialect for ucanaccess jdbc driver like following:
import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcDialects}
case object MSAccessJdbcDialect extends JdbcDialect {
override def canHandle(url: String): Boolean = url.startsWith("jdbc:ucanaccess")
override def quoteIdentifier(colName: String): String = s"[$colName]"
}
JdbcDialects.registerDialect(MSAccessJdbcDialect)