I am trying to test Debezium in a standalone, non-docker environment, to copy changed data from one MySql table to another MySql table. The source and sink tables are the same structure in different databases (source db.table: testiot.INPUT_TEMP, sink db.table: tiotbak:INPUT_TEMP_BK). It looks like the source table is being read but the sink table is not created. Please help.
My source.properties file:
name=connect-mysql
connector.class=io.debezium.connector.mysql.MySqlConnector
tasks.max=1
database.hostname=localhost
database.port=3306
database.user=debezium
database.password=dbz
database.dbname=testiot
database.server.id=184054
topic.prefix=testiot
database.server.name=localhost
database.allowPublicKeyRetrieval=true
schema.history.internal.kafka.bootstrap.servers=localhost:9092
schema.history.internal.kafka.topic=testiot.INPUT_TEMP_BK
database.history.kafka.bootstrap.servers=localhost:9092
database.history.kafka.topic=testiot.INPUT_TEMP_BK
My sink.properties file:
name=connect-mysql
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
connection.url=jdbc:mysql://localhost:3306/tiotbak?user=debezium&password=dbz&allowPublicKeyRetrieval=true&useSSL=false
topic.prefix=testiot
database.hostname=localhost
database.port=3306
database.user=debezium
database.password=Debez#01
database.dbname=tiotbak
database.server.id=184054
database.server.name=localhost
auto.create=true
insert.mode=upsert
pk.fields=DEVICE_ID
pk.mode=record_value
record_value=record_value
The command I am using to execute is:
sudo ./bin/connect-standalone.sh ./config/connect-debezium-mysql-worker.properties ./config/connect-debezium-mysql-source.properties ./config/connect-debezium-mysql-sink.properties
Related
I'm using Debezium and MySQL. In the database there is a table managed by the Flyway and I want to exclude it. I used the configurations below:
name=IRS-Connector
connector.class=io.debezium.connector.mysql.MySqlConnector
database.hostname=mysql
database.port=3306
database.user=user
database.password=user
database.allowPublicKeyRetrieval=true
database.server.name=irs-conn-v1
database.include.list=decider
database.exclude.list=register
database.history.kafka.bootstrap.servers=localhost:9092
database.history.kafka.topic=schema-changes.decider
table.exclude.list=flyway_schema_history
But when I'll see the topic irs-conn-v1.decider.flyway_schema_history had been created.
According the doc of the table.exclude.list option:
Each identifier is of the form databaseName.tableName.
So you need to append databaseName:
table.exclude.list=decider.flyway_schema_history
I cannot run source table_name.sql
when I try, I get the following error:
ERROR:
Failed to open file 'cars.sql', error: 2
I have been following zeetcode:
http://zetcode.com/databases/mysqltutorial/introduction/#mysql
http://zetcode.com/databases/mysqltutorial/firststeps/
The first provides a list of commands to create a database called mydb and a set of tables to be used in the tutorial including one named Cars.
The second link shows you how to access the databases (SHOW DATABASES;) which I could do but when I go to the previously created database mydb, I can see the previously created tables including Cars (even though the tutorial says I should not see anything.
when I follow the next command: source cars.sql, I receive the error above
yet, this query works:
mysql> SELECT * FROM Cars;
any ideas as to why the source function would not work?
This is the first time I am working with mysql.
where is the file cars.sql located? you may need to specify the full path to the file in order for it to be located by mysql.
While trying to output unified2 format into mysql database using Barnyard2, the Source or destination IP addresses in the database come in the format "3232240901" instead of 192.168.1.0(just in case)..How do I fix it?
Need to add SNMP information to a SQL database and update it on a regular schedule. SNMP info can be queried from bash commands.
You can use bash commands to write insert statements to a file, then pipe the file into the mysql program.
Say you have a file that looks like this:
key1,1.0
key2,1.4
key3,1.9
key4,2.0
key5,3.5
you can pipe it into a bash script that looks something like:
#!/bin/bash
while read key, value; do
echo "insert into sometable(key, value) values('$key' $value);"
done >/tmp/inserts.sql
mysql </tmp/inserts.sql >/tmp/inserts.out
If your data comes from somewhere else then same principle, just generate SQL commands into a file and pipe them into mysql.
This strategy isn't as kludgy as it might seem at first. MySQL's own mysqldump backup utility dumps the database to a file in the form of SQL statements.
I want to process my mysql table data inside nant script. My table contatins some image file path. using that path I need to do svn check out also. How can I do like that. Does it possible to fetch all mysql table rows using nant script and process it in a loop?
Use the NAnt task from NAntContrib and output the results to a file.
http://nantcontrib.sourceforge.net/release/0.91/help/tasks/sql.html
Hint: Here is a sample MySql OleDb Connection String that can be used:
Provider=MySQLProv;Data Source=mydb;User Id=myUsername;Password=myPassword;
Use a task to loop over each line in the file:
http://nant.sourceforge.net/release/0.91/help/tasks/foreach.html
Use the task from NAntContrib to checkout the file from svn:
http://nantcontrib.sourceforge.net/release/0.91/help/tasks/svn-checkout.html