I see in the documentation for Db2 Event Store that a CSV file can be loaded into the system when the file is within the system in this document https://www.ibm.com/support/knowledgecenter/en/SSGNPV_2.0.0/local/loadcsv.html. I also found that you can connect to a Db2 Event Store database using the standard Db2 client in How do I connect to an IBM Db2 Event Store instance from a remote Db2 instance?. What I am trying to do now is load a CSV file using that connection. Is it possible to load it remotely ?
This should be doable with an extra keyword specified REMOTESOURCE YES, e.g:
db2 "INSERT INTO import_test SELECT * FROM EXTERNAL '/home/db2v111/data.del' USING (DELIMITER ',' REMOTESOURCE YES)"
see an example here:
IMPORT script on IBM DB2 Cloud using RUN SQL Interface
With other answers mentioned the connection and loading using the traditional db2. I have to add some more details that are required specifically for Db2 Event Store.
Assuming we are using a Db2 Client container, which can be found at docker hub with tag ibmcom/db2.
Basically we have to go through following steps:
1/ establish a remote connection from db2 client container to the remote db2 eventstore database
2/ use db2 CLP commands to load the csv file using the db2's external table load feature, which will load csv file from db2 client container to the remote eventstore database.
Step 1:
Run the following commands, or run the it in a script. Note that the commands need to be run as the db2 user in the db2 client container. The db2 user name is typically db2inst1
#!/bin/bash -x
NODE_NAME=eventstore
. /database/config/db2inst1/sqllib/db2profile
### create new keydb used for authentication
# remote old keydb files
rm -rf $HOME/mydbclient.kdb $HOME/mydbclient.sth $HOME/mydbclient.crl $HOME/mydbclient.rdb
$HOME/sqllib/gskit/bin/gsk8capicmd_64 -keydb -create -db $HOME/mydbclient.kdb -pw ${SSL_KEY_DATABASE_PASSWORD} -stash
KEYDB_PATH=/var/lib/eventstore/clientkeystore
# get the target eventstore cluster's SSL public certificate using REST api
bearerToken=`curl --silent -k -X GET "https://$IP/v1/preauth/validateAuth" -u $EVENT_USER:$EVENT_PASSWORD | python -c "import sys, json; print (json.load(sys.stdin)['accessToken']) "`
curl --silent -k -X GET -H "authorization: Bearer $bearerToken" "https://${IP}:443/com/ibm/event/api/v1/oltp/certificate" -o $HOME/server-certificate.cert
# insert eventstore cluster's SSL public cert into new gskit keydb
$HOME/sqllib/gskit/bin/gsk8capicmd_64 -cert -add -db $HOME/mydbclient.kdb -pw ${SSL_KEY_DATABASE_PASSWORD} -label server -file $HOME/server-certificate.cert -format ascii -fips
# let db2 client use the new keydb
$HOME/sqllib/bin/db2 update dbm cfg using SSL_CLNT_KEYDB $HOME/mydbclient.kdb SSL_CLNT_STASH $HOME/mydbclient.sth
# configure connection from db2Client to remote EventStore cluster.
$HOME/sqllib/bin/db2 UNCATALOG NODE ${NODE_NAME}
$HOME/sqllib/bin/db2 CATALOG TCPIP NODE ${NODE_NAME} REMOTE ${IP} SERVER ${DB2_CLIENT_PORT_ON_EVENTSTORE_SERVER} SECURITY SSL
$HOME/sqllib/bin/db2 UNCATALOG DATABASE ${EVENTSTORE_DATABASE}
$HOME/sqllib/bin/db2 CATALOG DATABASE ${EVENTSTORE_DATABASE} AT NODE ${NODE_NAME} AUTHENTICATION GSSPLUGIN
$HOME/sqllib/bin/db2 terminate
# Ensure to use correct database name, eventstore user credential in remote
# eventstore cluster
$HOME/sqllib/bin/db2 CONNECT TO ${EVENTSTORE_DATABASE} USER ${EVENT_USER} USING ${EVENT_PASSWORD}
Some important variables:
EVENTSTORE_DATABASE: database name in the remote eventstore cluster
EVENT_USER: EventStore user name remote eventstore cluster
EVENT_PASSWORD: EventStore user password remote eventstore cluster
IP: Public IP of remote eventstore cluster
DB2_CLIENT_PORT_ON_EVENTSTORE_SERVER: JDBC port of remote eventstore cluster, which is typically 18730
SSL_KEY_DATABASE_PASSWORD: keystore's password of the gskit keydb file in the db2 client container, you can set it as you like
After running the commands above, you should have established the connection between local db2 client container and the remote eventstore cluster
2/ Load csv file using external table feature of db2
After the connection between db2 client and remote eventstore cluster is established, we can issue db2 CLP commands like issuing command to any local db2 database.
For example:
// establish remote connection to eventstore database
// replace the same variables in ${} with what you used above.
CONNECT TO ${EVENTSTORE_DATABASE} USER ${EVENT_USER} USING ${EVENT_PASSWORD}
SET CURRENT ISOLATION UR
// create table in the remote eventstore database
CREATE TABLE db2cli_csvload (DEVICEID INTEGER NOT NULL, SENSORID INTEGER NOT NULL, TS BIGINT NOT NULL, AMBIENT_TEMP DOUBLE NOT NULL, POWER DOUBLE NOT NULL, TEMPERATURE DOUBLE NOT NULL, CONSTRAINT "TEST1INDEX" PRIMARY KEY(DEVICEID, SENSORID, TS) INCLUDE (TEMPERATURE)) DISTRIBUTE BY HASH (DEVICEID, SENSORID) ORGANIZE BY COLUMN STORED AS PARQUET
// external table load to remote eventstore database
INSERT INTO db2cli_csvload SELECT * FROM EXTERNAL '${DB_HOME_IN_CONTAINER}/${CSV_FILE}' LIKE db2cli_csvload USING (delimiter ',' MAXERRORS 10 SOCKETBUFSIZE 30000 REMOTESOURCE 'YES' LOGDIR '/database/logs' )
CONNECT RESET
TERMINATE
For more information, you can check on the Db2 EventStore's public github repo.
https://github.com/IBMProjectEventStore/db2eventstore-IoT-Analytics/tree/master/db2client_remote/utils
How to connect R to mysql? Do we need to install mysql on local host separately? When I try to connect it says
"Error in .local(drv, ...) : Failed to connect to database: Error:
Can't connect to MySQL server on 'localhost' (0)".
What does it mean? I am using R on windows by the way.
You don't show your R code in the post so it is hard to "fix" what might be wrong. Instead I'll just show you some code that works.
To access a mysql database you can use the RMySQL package. For this to work your mysql server must accept remote requests and you must specify the
address (for example the IP address where the server is running)
the username
the password
the database name
Your mysql server should be set up to accept non-local requests unless it's already running locally on the machine where you run R.
library(RMySQL)
mydb = dbConnect(MySQL(), user="username", password="PASSword",
dbname="dbname", host="192.168.13.15")
then you can submit queries in standard SQL language.
indata <- dbGetQuery(mydb, "select * from students")
You do not need a sql client for the R code to work, but if you cannot get it to run within R then you should try to see if you can log onto the server using a manual client. If that is not possible your server is not setup to allow the logins.
I want to connect mysql DB and insert some values into the table. But I get the result: "INSERT command denied to user '' #'localhost' for table 'paramenttable'".
DB connection page
(to call the two function in route.js like this:)
mysqlinsertp.conn();
mysqlinsertp.insert(array_1);
the error page:
The config for mysql could not be found in that variable. The module exports of settings.js provide a mysql item. You should use that instead of the root.
Your client initialization should be following:
var client = mysql.createConnection(settings.mysql);
instead of mysql.createConnection(settings);
I'm trying to connect to a remote MySQL database via an Ubuntu server using a command like:
mysql -u oooo -p -h ooo.ooo.oo.oo
Then it just returned:
mysql: unknown variable 'log=/var/log/mysql/logfile.log'
I've checked that the file 'logfile.log' does not exist and I have no idea what is the next step.
Search your machine for a file named my.cnf. That file will probably contain your offending variable. Comment it out.
How would I connect to my VPS based MySQL database remotely (from a cloud based app) using the Ruby Net::SSH or Net::SSH::Gateway gems and key, not password, authentication?
And then connect to the database with Sequel or DataMapper. I'm assuming that after I manage to get the SSH connection working, I would just setup a Sequel/DM connection to 'sql_user#localhost:3306/database'.
I did locate a couple of similar question here, but they all use password authentication, not keys, and only demonstrate executing raw commands to query the database.
UPDATE: I just cannot seem to get this (Net::SSH with key manager) to work.
UPDATE2: Alright I have managed to get authorization when logging in from a computer that has authorized keys stored in the users local .ssh folder, with the following (port is my custom SQL port on the VPS):
sql_gate = Net::SSH::Gateway.new('192.xxx.xxx.xx','sqluser', port: 26000)
However, I will not be able to create a .ssh folder in the app's VM, so I need to somehow pass the path and filename (I will be creating a public key just for SQL access for specified user) as an option ... but haven't been able to figure out how.
UPDATE: Just need to figure out DataMapper access now. Current code being tested (remote_user_sql is my Ubuntu user, sql_user is the MySQL database user with localhost/127.0.0.1 privileges):
require 'net/ssh/gateway'
require 'data_mapper'
require 'dm-mysql-adapter'
class User
include DataMapp......
.
.
end
ssh_gate = Net::SSH::Gateway.new('192.n.n.n','remote_user_sql', {port: 25000, keys: ["sql_rsa"], keys_only: true})
port = ssh_gate.open('localhost',3306,3307)
child = fork do
DataMapper.setup(:default, {
adapter: 'mysql',
database: 'sql_test',
username: 'sql_user',
password: 'passwd',
host: 'localhost',
port: port})
DataMapper.auto_upgrade!
exit
end
puts "child: #{child}"
Process.wait
ssh_gate.close(port)
My solution, in two parts:
Well I have figured how to make the Net::SSH::Gateway gem using a specified keyfile, and then connect to the VPS through ssh via a port other than 22:
Part 1: Net::SSH::Gateway key authentication
First you must generate the keyfiles you want to use, copy the .pub to the remove server and append it to the ~/.ssh/authorized_keys file (cat sql_rsa.pub >> authorized_keys), and then make sure user_sql (the user I created on the VPS to be used only for this purpose) has been added to AllowUsers list in sshd_config. Make note of port used for ssh (25000 for this example) and use the following code to establish the connection:
ssh_gate = Net::SSH::Gateway.new('192.n.n.n','user_sql', {port: 25000, keys: ["sql_rsa"], keys_only: true})
That will read the keyfile sql_rsa in the same directory as script file, then create a new ssh gateway for 'user_sql'#'192.n.n.n' on port 25000.
I can successfully execute raw shell commands on the remove VPS with:
ssh_gate.exec("ls -la")
To close:
ssh_gate.shutdown!
Unfortunately I am still having problems using DataMapper (do-mysql-adapter) to use the gateway. I will update this answer if I figure that part out, but at least the first half of the problem has been solved.
These are the errors that DataMapper::Logger has reported:
When 127.0.0.1 was used:
Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) (code: 2002, sql state: HY000, query: , uri: )
When localhost was used:
Access denied for user 'user_sql'#'localhost' (using password: YES) (code: 1045, sql state: 28000, query: , uri: )
When the VPS hostname was used:
Unknown MySQL server host 'hostname' (25) (code: 2005, sql state: HY000, query: , uri: )
UPDATE (No success yet): So far the only way I can access the remote MySQL database is by using Net::SSH::Gateway to establish a gateway, and then use the .sshmethod to open a new Net::SSH connection over that gateway, like so:
ssh_gate.ssh('192.n.n.n','user_sql',{port: 25000, keys: ["sql_rsa"], keys_only: true}) do |ssh|
ssh.exec("mysql -u sql_user -p'passwd' -h localhost -P 3306 -e 'SELECT DATABASE();'")
end
In other words, I can only execute SQL commands using the mysql command line. I cannot figure out how to get Sequel or DataMapper to use the gateway to connect.
Part 2: DataMapper/Sequel/mysql2 connection through Net::SSH::Gateway
Make sure your MySQL server is bound to 127.0.0.1 in /etc/mysql/my.cnf, setup your connection - DataMapper example:
DataMapper.setup(:default, {
adapter: 'mysql',
database: 'DATABASE',
username: 'username',
password: 'passwd',
host: '127.0.0.1',
port: 3307}) # local port being forwarded via Net::SSH:Gateway
Followed by any class table definitions and DataMapper.finalize if required. Note that DataMapper doesn't actually connect to the remote MySQL server until either an auto_upgrade!, auto_migrate!, or query is executed, so no need to create the forwarded port yet.
Then create a new Net::SSH::Gateway, and then whenever you need DataMapper/Sequel to access the remote database, just open a port for the process, like so:
port = ssh_gate.open('127.0.0.1',3306,3307)
child = fork do
DataMapper.auto_upgrade! # DM call that accesses MySQL server
exit
end
Process.wait
ssh_gate.close(port)
You may want to put the Net::SSH::Gateway/.open code in a begin..ensure..end block, ensure'ing the port closure and gateway shutdown.
I had to use a fork and Process.wait to establish the connection, without it the method just hangs.