Spark ETL job execute mysql only once - mysql

Question

I have an ETL job in Spark that also connects to MySQL in order to grab some data. Historically, I've been doing it as follows:
hiveContext.read().jdbc(
dbProperties.getProperty("myDbInfo"),
"(SELECT id, name FROM users) r",
new Properties()).registerTempTable("tmp_users");
Row[] res = hiveContext.sql("SELECT "
+ " u.name, "
+ " SUM(s.revenue) AS revenue "
+ "FROM "
+ " stats s "
+ " INNER JOIN tmp_users u "
+ " ON u.id = s.user_id
+ "GROUP BY "
+ " u.name "
+ "ORDER BY "
+ " revenue DESC
+ "LIMIT 10").collect();
String ids = "";
// now grab me some info for users that are in tmp_user_stats
for (i = 0; i < res.length; i++) {
s += (!s.equals("") ? "," : "") + res[i](0);
}
hiveContext.jdbc(
dbProperties.getProperty("myDbInfo"),
"(SELECT name, surname, home_address FROM users WHERE id IN ("+ids+")) r",
new Properties()).registerTempTable("tmp_users_prises");
However, when scaling this to multiple worker nodes, whenever I use the tmp_users table, it runs the query and it gets executed (at least) once per node, which boils down to our db admin running around offices with a knife.
What's the best way to handle this? Can I run the job on like 3 machines, limiting it to 3 queries and then write the data to Hadoop for other nodes to use it or what?
Essentially - as suggested in comments - I could run a query outside of the ETL job which can prepare data from MySQL side and import it to Hadoop. However, there could be subsequent queries, which suggest a solution more in line with Spark and JDBC connection setup.
I'll accept the Sqoop solution as it at least give a more streamlined solution, although I'm still not yet sure it will do the job. If I find something, I'll edit the question again.

Answer 1

You can cache data:
val initialDF = hiveContext.read().jdbc(
dbProperties.getProperty("myDbInfo"),
"(SELECT id, name FROM users) r",
new Properties())
initialDF.cache();
initialDF.registerTempTable("tmp_users");
After first read, data will be cached in memory
Alternative (that doesn't hurt DBA ;) ) is to use Sqoop with parameter --num-mappers=3 and then import result file to Spark