Using DataFlow to read data from Compute Engine - google-compute-engine

Question

I want to read data from MariaDB on Google Compute Engine and to write data to BigQuery by DataFlow but I always get the exception as below when I run the DataFlow program on DataFlowRunner.
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not connect to address=(host=xxx.xxx.xxx.xxx)(port=3306)(type=master) : connect timed out)
I can access successfully the MariaDB by DBeaver.
I can run successfully the DataFlow program on DirectRunner.
Can give me some ideals, Thanks.

Answer 1

To restrict it so that only Dataflow jobs can access it you can leverage the fact that Dataflow's harness VMs are created with the dataflow tag. Otherwise, you can allocate the GCE instance and DF workers on a specific network/subnetwork.
For example, create a GCE instance with a network tag such as mariadb so it can be used as the target to apply firewall rules to and/or select a specific VPC network/subnetwork. Install MariaDB (another option is to use an initialization script or a preinstalled solution through Cloud Launcher).
For the firewall rules, you'll need the Database to be reachable on port tcp:3306. For the GCE instance (target tag mariadb) you'll need to allow ingress traffic from either source tags dataflow or coming from within the subnetwork on the aforementioned port. Take into account that, for the latter option, you'll need to also allow internal communication between DF workers inside the subnetwork.
Now, on the Dataflow side, add the JdbcIO and mariadb connector dependencies to the pom.xmlfile:
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-io-jdbc -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-jdbc</artifactId>
<version>2.3.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.mariadb.jdbc/mariadb-java-client -->
<dependency>
<groupId>org.mariadb.jdbc</groupId>
<artifactId>mariadb-java-client</artifactId>
<version>1.1.7</version>
</dependency>
A sample Dataflow snippet to connect (use internal IP in the JDBC connection string if using the subnetwork approach):
public class MariaDB {
public static void main(String[] args) {
PipelineOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(PipelineOptions.class);
Pipeline p = Pipeline.create(options);
PCollection<String> account = p.apply(JdbcIO.<String>read()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create("org.mariadb.jdbc.Driver", "jdbc:mariadb://INTERNAL_IP:3306/database").withUsername("root").withPassword("pwd"))
.withQuery("SELECT … FROM table")
.withCoder(SerializableCoder.of(String.class))
.withRowMapper(new JdbcIO.RowMapper<String>() {
public String mapRow(ResultSet rs) throws Exception {
...
}}
));
p.run();
}
}
And launch the job specifying the subnetwork and matching zone if needed:
mvn compile exec:java \
-Dexec.mainClass=com.example.MariaDB \
-Dexec.args="--project=PROJECT_ID \
--stagingLocation=gs://BUCKET_NAME/mariadb/staging/ \
--output=gs://BUCKET_NAME/mariadb/output \
--network="dataflow-network" \
--subnetwork="regions/europe-west1/subnetworks/subnet-europe-west" \
--zone="europe-west1" \
--runner=DataflowRunner"