I'm using Debezium and MySQL. In the database there is a table managed by the Flyway and I want to exclude it. I used the configurations below:
name=IRS-Connector
connector.class=io.debezium.connector.mysql.MySqlConnector
database.hostname=mysql
database.port=3306
database.user=user
database.password=user
database.allowPublicKeyRetrieval=true
database.server.name=irs-conn-v1
database.include.list=decider
database.exclude.list=register
database.history.kafka.bootstrap.servers=localhost:9092
database.history.kafka.topic=schema-changes.decider
table.exclude.list=flyway_schema_history
But when I'll see the topic irs-conn-v1.decider.flyway_schema_history had been created.
According the doc of the table.exclude.list option:
Each identifier is of the form databaseName.tableName.
So you need to append databaseName:
table.exclude.list=decider.flyway_schema_history
Related
I am trying to test Debezium in a standalone, non-docker environment, to copy changed data from one MySql table to another MySql table. The source and sink tables are the same structure in different databases (source db.table: testiot.INPUT_TEMP, sink db.table: tiotbak:INPUT_TEMP_BK). It looks like the source table is being read but the sink table is not created. Please help.
My source.properties file:
name=connect-mysql
connector.class=io.debezium.connector.mysql.MySqlConnector
tasks.max=1
database.hostname=localhost
database.port=3306
database.user=debezium
database.password=dbz
database.dbname=testiot
database.server.id=184054
topic.prefix=testiot
database.server.name=localhost
database.allowPublicKeyRetrieval=true
schema.history.internal.kafka.bootstrap.servers=localhost:9092
schema.history.internal.kafka.topic=testiot.INPUT_TEMP_BK
database.history.kafka.bootstrap.servers=localhost:9092
database.history.kafka.topic=testiot.INPUT_TEMP_BK
My sink.properties file:
name=connect-mysql
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
connection.url=jdbc:mysql://localhost:3306/tiotbak?user=debezium&password=dbz&allowPublicKeyRetrieval=true&useSSL=false
topic.prefix=testiot
database.hostname=localhost
database.port=3306
database.user=debezium
database.password=Debez#01
database.dbname=tiotbak
database.server.id=184054
database.server.name=localhost
auto.create=true
insert.mode=upsert
pk.fields=DEVICE_ID
pk.mode=record_value
record_value=record_value
The command I am using to execute is:
sudo ./bin/connect-standalone.sh ./config/connect-debezium-mysql-worker.properties ./config/connect-debezium-mysql-source.properties ./config/connect-debezium-mysql-sink.properties
So I have my location column using Point data type, I'm using Apollo Server and Prisma, and when I use "npx prisma db pull" generates this data type because is not currently supported on Prisma (generated script)
so I say "Ok, I'm using string and I manage how to insert this data type" so I changed to this script, surprise! didn't work enter image description here, try to find any approach to handling MySql Point data type in Prisma but no info at soever, I really appreciate any ideas
You cannot convert it to String and use it as it isn't supported yet. You need to leave it as unsupported and you can only add data via raw queries.
For now, only adding data is supported. You cannot query for it using PrismaClient.
We can query data using Prisma Client, via raw queries as SELECT id, ST_AsText(geom) as geom from training_data where geom has dataType geometry for using Unsupported("geometry").
I set up mailman3 using the mailman-suite configuration. I expected that mailman uses the database provided there (like sqlite or mysql), but it seems to have its own database file mailman.db in var/data. I followed the instructions here and added
[database]
class: mailman.database.mysql.MySQLDatabase
url: mysql+pymysql://myuser:mypassword#mymysqlhost/mailman?charset=utf8&use_unicode=1
to my mailman.cfg. Of course I replaced myuser, mypassword, mymysqlhost with the correct data. But still, mailman3 uses the data from mailman.db
What do I have to do to make mailman use the mysql database?
I have the exact same database definition for multiple databases ( and database servers ). How do I tell Jooq to use the same database as the "Connection" I created to connect to the DB?
Example ( for MySQL ):
jdbc:mysql://localhost:3306/tsm - my development DB ( tsm ), used to generate code
jdbc:mysql://RemoteAmazonDBHost:3306/customer1 - one of my customers
jdbc:mysql://RemoteAmazonDBHost:3306/customer2 - Another customer
All 3 databases have the same definition, the same tables, indexes, etc. The TSM one is the standard our application uses.
Maybe I should be using DSL.using( Connection, Setting ) instead of DSL.using(Connection)? Is that what the manual is implying here?
If I only have one "Input" schema, do I have to specify it? In other words, can I do something like this:
Settings settings = new Settings()
.withRenderMapping(new RenderMapping()
.withSchemata(
new MappedSchema().withOutput(
databaseInfo.getProperties().getProperty("database.db"))));
Or do I have to do something like this:
Settings settings = new Settings()
.withRenderMapping(new RenderMapping()
.withSchemata(
new MappedSchema().withInput("TSM")
.withOutput(databaseInfo.getProperties().getProperty("database.db"))));
I'm assuming you're using code generation. In that case, it might be the easiest to not generate the schema at all but use <outputSchemaToDefault> in the code generation configuration, e.g.
<configuration>
<generator>
<database>
<inputSchema>your_codegen_input_schema_here</inputSchema>
<outputSchemaToDefault>true</outputSchemaToDefault>
</database>
</generator>
</configuration>
See the manual for details: https://www.jooq.org/doc/latest/manual/code-generation/codegen-advanced/codegen-config-database/codegen-database-catalog-and-schema-mapping/
If you want to keep your generated code with schema qualification and map things at runtime, then your second attempt seems correct. Pass this to your Configuration (i.e. the DSL.using() call):
Settings settings = new Settings()
.withRenderMapping(new RenderMapping()
.withSchemata(new MappedSchema()
.withInput("TSM")
.withOutput(databaseInfo.getProperties().getProperty("database.db"))));
More details can be found here: https://www.jooq.org/doc/latest/manual/sql-building/dsl-context/custom-settings/settings-render-mapping
I am trying to generate schema from the DataNucleus SchemaTool for a mysql database, that will store countries and states. Here is a sample of that code:
#PersistenceCapable
Public class State{
private String shortCode;
private String fullName;
#Column(allowsNull = "true",name="country_id")
private Country countryId;
}
The following are my schemaGeneration properties:
datanucleus.ConnectionDriverName=com.mysql.jdbc.Driver
datanucleus.ConnectionURL=jdbc:mysql://localhost:3306/geog
datanucleus.ConnectionUserName=geog
datanucleus.ConnectionPassword=geogPass
datanucleus.schema.validateTables=true
datanucleus.mapping.Catalog=geog
datanucleus.mapping.Schema=geog
In my Country class as well, I have a mapping from a Collection, so that the FK reference for States to the Country table is built correctly.
But there is one problem. In the SQL script generated, the Index part has the Schema name as part of the index name itself, which fails the entire script. Here is that piece:
CREATE INDEX `GEOG`.`MST_STATE_N49` ON `GEOG`.`MST_STATE` (`COUNTRY_ID`);
Notice the schema name in the GEOG.MST_STATE_N49 part of the index' name.
I tried setting the schema and catalog name to blank but that yields a ''.MST_STATE_N49 which still fails.
I am using MySQL Server 5.7.17 using the 5.1.42 version of the JDBC driver (yes, not the latest) on Data nucleus JDO 3.1
Any hints on how I can get rid of the schema/catalog name in the generated DDL?
Why are you putting "datanucleus.mapping.Schema" when using MySQL ? MySQL doesnt use schema last I looked. Similarly the "datanucleus.mapping.Catalog" is effectively defined by your URL! MySQL only actually supports JDBC catalog, mapping on to "database", as per this post. Since DataNucleus simply uses the JDBC driver then catalog is the only useful input.
Consequently removal of both schema and catalog properties will DEFAULT to the right place.
After the comment above from Neil Stockton, I commented out both the properties and it worked. Effectively, this is what is needed:
datanucleus.ConnectionDriverName=com.mysql.jdbc.Driver
datanucleus.ConnectionURL=jdbc:mysql://localhost:3306/geog
datanucleus.ConnectionUserName=geog
datanucleus.ConnectionPassword=geogPass
datanucleus.schema.validateTables=true
Hopefully, I can get the answer to the other question (Pt. 2 in my reply-comment above) as well.