Grails run query postgresql on second database to generate Raw JSON - json

I have one postgreSQL database which I am using inside my Grails application(configured in Datasource.groovy), lets call it DB1. Now I have other postgreSQL database which has lots of data inside it, lets call it DB2.
I am writing an data export procedure which takes in JSON data generated from DB2, make its domain objects and store it inside the DB1. This data is being sent from another software using DB2. The main problem is that both databases have different column names so it cannot be a direct import export.
PostgreSQL provides direct methods to generate JSON via SQL queries. Eg-
SELECT row_to_json(t)
FROM ( select id, descrizione as description from tableXYZ ) t
It returns a JSON output-
{"id":6013,"description":"TestABC"}
This JSON can be consumed by the code that I have made.
So I want to run this query on DB2 from inside grails application which has DB1 configured inside Datasource.groovy.
How to do that?

In your DataSource.groovy file, you need to create another data source to point at DB2. You can probably clone your dataSource definition for this, eg.
http://grails.github.io/grails-doc/2.3.9/guide/conf.html#multipleDatasources
dataSource_db2 {
pooled = true
jmxExport = true
driverClassName = "org.postgresql.Driver"
username = "XXX"
password = "YYY"
//noinspection GrReassignedInClosureLocalVar
dialect = PostgreSQLDialect
autoreconnect = true
useUnicode = true
characterEncoding = "utf-8"
tcpKeepAlive = true
//noinspection GroovyAssignabilityCheck
properties {
// See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 1000 * 60 * 1 // 1 min
minEvictableIdleTimeMillis = 1000 * 60 * 5 // 5 min
numTestsPerEvictionRun = 3
validationQuery = 'SELECT 1'
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = false
testOnReturn = false
defaultTransactionIsolation = Connection.TRANSACTION_READ_COMMITTED
removeAbandoned = true
removeAbandonedTimeout = 20 // 20s is a long query
logAbandoned = true // causes stacktrace recording overhead, use only for debugging
// use JMX console to change this setting at runtime
// the next options are jdbc-pool specific
// http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Common_Attributes
// https://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/tomcat/jdbc/pool/PoolConfiguration.html
jmxEnabled = true
// ResetAbandonedTimer resets the timer upon every operation on the connection or a statement.
// ConnectionState caches the auto commit, read only and catalog settings to avoid round trips to the DB.
jdbcInterceptors = "ConnectionState;ResetAbandonedTimer;SlowQueryReportJmx(threshold=10000)"
abandonWhenPercentageFull = 25 // settings are active only when pool is full
}
}
To use it for database connection access, you can inject the javax.sql.DataSource into your Services, Controllers, Domain classes, or other Grails Artefakts.
eg.
import javax.sql.DataSource
import groovy.sql.GroovyResultSet
import groovy.sql.Sql
MyService {
DataSource dataSource_db2
def doQuery(String query) {
new Sql(dataSource_db2).rows(query)
}
}
To have a domain object use your db2 dataSource for GORM, add to your domain objects 'mapping' block:
static mapping = {
datasource 'db2'
}
If you want JNDI support, you can also add something like this in your resources.groovy:
xmlns jee: "http://www.springframework.org/schema/jee"
jee.'jndi-lookup'(id: "dataSource", 'jndi-name': "java:comp/env/jdbc/db1")
jee.'jndi-lookup'(id: "dataSource_db2", 'jndi-name': "java:comp/env/jdbc/db2")

Related

Recreate replica when master is recreated

I have a QA setup with a master and a replica. Both are AWS RDS MySQL. It's provisioned with Terraform and the gist is like this
data "aws_db_snapshot" "latest_prod_snapshot" {
db_instance_identifier = var.production_instance_id
most_recent = true
}
resource "aws_db_instance" "qa_master" {
apply_immediately = true
snapshot_identifier = data.aws_db_snapshot.latest_prod_snapshot.id
availability_zone = var.qa_master_zone
instance_class = var.master_instance_class
identifier = var.master_name
parameter_group_name = var.parameter_group_name
auto_minor_version_upgrade = false
multi_az = false
performance_insights_enabled = true
performance_insights_retention_period = 7
vpc_security_group_ids = [var.security_group_id]
option_group_name = var.option_group_name
backup_retention_period = 5
skip_final_snapshot = true
enabled_cloudwatch_logs_exports = ["audit", "error", "slowquery"]
}
resource "aws_db_instance" "qa_replica" {
apply_immediately = true
replicate_source_db = aws_db_instance.qa_master.id
availability_zone = var.qa_replica_zone
instance_class = var.replica_instance_class
identifier = var.replica_name
parameter_group_name = var.parameter_group_name
auto_minor_version_upgrade = false
multi_az = false
performance_insights_enabled = true
performance_insights_retention_period = 7
vpc_security_group_ids = [var.security_group_id]
skip_final_snapshot = true
enabled_cloudwatch_logs_exports = ["audit", "error", "slowquery"]
}
When I want to update it from a new snapshot, the master is always marked for recreation and replica for "change in place". But replication stops working after the update. Is there a workaround for that? Am I doing something weird here? Can I somehow force replica to recreate too?
So far I have been doing terraform destroy before doing the terraform apply.
To trigger the replica being recreated you will need to have a ForceNew parameter on the replica resource change when the non replica changes.
With RDS there is a resource_id attribute that isn't normally surfaced in places people care about (you don't specify it, you don't use it to connect to it and it's not shown in the RDS console) but is different for each database instance (compared to the normal identifier which is specified).
This then brings us to the other part which is that that attribute needs to be in a ForceNew parameter on the replica resource. The obvious choices here are either the identifier or identifier_prefix parameters. The biggest impact here is that these are also used to identify the database instance from others but also used as part of the DNS address to connect to the database in the form of ${identifier}.${random_hash_of_account_and_region}.${region}.rds.amazonaws.com. So if you're needing to connect to the instance then you'll either need to have the client discover the replica address as it will now contain the randomly generated resource_id of the non replica instance or you'll need to have Terraform create a DNS record that either CNAMEs or aliases the RDS address.
So in your case you might want something like this:
data "aws_db_snapshot" "latest_prod_snapshot" {
db_instance_identifier = var.production_instance_id
most_recent = true
}
resource "aws_db_instance" "qa_master" {
apply_immediately = true
snapshot_identifier = data.aws_db_snapshot.latest_prod_snapshot.id
availability_zone = var.qa_master_zone
instance_class = var.master_instance_class
identifier = var.master_name
parameter_group_name = var.parameter_group_name
auto_minor_version_upgrade = false
multi_az = false
performance_insights_enabled = true
performance_insights_retention_period = 7
vpc_security_group_ids = [var.security_group_id]
option_group_name = var.option_group_name
backup_retention_period = 5
skip_final_snapshot = true
enabled_cloudwatch_logs_exports = ["audit", "error", "slowquery"]
}
resource "aws_db_instance" "qa_replica" {
apply_immediately = true
replicate_source_db = aws_db_instance.qa_master.id
availability_zone = var.qa_replica_zone
instance_class = var.replica_instance_class
identifier = "${var.replica_name}-${aws_db_instance.qa_master.resource_id}"
parameter_group_name = var.parameter_group_name
auto_minor_version_upgrade = false
multi_az = false
performance_insights_enabled = true
performance_insights_retention_period = 7
vpc_security_group_ids = [var.security_group_id]
skip_final_snapshot = true
enabled_cloudwatch_logs_exports = ["audit", "error", "slowquery"]
}
resource "aws_route53_zone" "example" {
name = "example.com"
}
resource "aws_route53_record" "replica_instance" {
zone_id = data.aws_route53_zone.example.zone_id
name = "qa-replica-database.${data.aws_route53_zone.example.name}"
type = "CNAME"
ttl = 60
records = [aws_db_instance.replica_qa.address]
}
Now, if the production snapshot changes then the qa_master database instance resource will be recreated which will lead to the qa_replica database instance resource also being recreated and then the Route53 record for the qa_replica instance will be updated with the new address, allowing you to always connect to the replica at qa-replica-database.example.com.

Python SQL Alchemy Multiple Databases - Binding Automap_Base

I am working through SQL Alchemy but struggling with how to structure the information from the docs into my project. I have two databases, my first will be used to store all new information from the python application. Where as the second database (DB1 in this case) is an existing database that I need to access information from. What is the right way to create this structure using SQLAlchemy?
I used the suggested BINDS method for multiple databases. This seems to be working.
class BaseConfig(object):
SECRET_KEY = "SO_SECURE"
DEBUG = True
SQLALCHEMY_DATABASE_URI = 'mssql+pyodbc://sa:funpassword#(localdb)\MSSQLLocalDB/Testing?driver=SQL+Server+Native+Client+11.0'
SQLALCHEMY_BINDS = {
'DB1': 'mssql+pyodbc://sa:$funpassword#ProdDB/DB1?driver=SQL+Server+Native+Client+11.0'
}
SQLALCHEMY_TRACK_MODIFICATIONS = True
This configuration seems to work okay because I am able to create new models in both of these databases using the code below. (This was done to just confirm that I was able to connect to both). db is my SqlAlchemy(app) initialization in my index.py file.
from index import app, db
#Test Writing To Default DB
class Test(db.Model):
id = db.Column(db.Integer(), primary_key=True)
employeeNum = db.Column(db.String(255), unique=False)
job = db.Column(db.String(255))
def __init__(self, employeeNum, job):
self.employeeNum = employeeNum
self.job = job
# Test Writing To DB1
class Test(db.Model):
__bind_key__ = 'DB1'
id = db.Column(db.Integer(), primary_key=True)
employeeNum = db.Column(db.String(255), unique=False)
job = db.Column(db.String(255))
def __init__(self, employeeNum, job):
self.employeeNum = employeeNum
self.job = job
I have tried many combinations using the table and automap_base from SQLAlchemy docs but this does not seem to work. I am not sure how I can use the bind_key of DB1 when I am trying to map existing tables.
from index import app, db
def loadSession():
Table('employee', db.metadata, Column('emp_num', Text, primary_key=True))
Base = automap_base(metadata=metadata)
Base.prepare()
employee = Base.classes.employee
emp = db.session.query(employee).all()
for i in emp:
print(i.data)
Thanks for any help you can provide.
for your DB1 with existing schema & data you could use reflection to get the tables mapped into sqlalchemy.
For your example it would probably look something like this:
from sqlalchemy.ext.automap import automap_base
from sqlalchemy import MetaData
from index import app, db
def loadSession():
# get a db engine object for DB1 configuration
engine = db.get_engine(bind='DB1')
# create a empty db Metadata object and bind it to DB1
database = MetaData(bind=engine)
# load the DB1 table/column structure into the Metadata object. This is quite a heavy operation and should optimally be done once on app setup and then the created automap_base can be stored / reused on demand until app shutdown
database.reflect(
bind=engine, views=True, autoload_replace=False
)
# create a python class structure out of the db metadata structure
auto_base = automap_base(metadata=database)
auto_base.prepare()
# create a db session for DB1
db_session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
# get the reflected employees class from DB1. The employee table must exist in the database
employee = auto_base.classes.employee
# query DB1 with the new session object for all employee records
emp = db_session.query(employee).all()
for i in emp:
print(i.data)
# the following could also be substituted for the query without creating a session object:
for entry in employee.query.all():
print(entry)

Join two database table zend framework 1.12

I have used two mysql database in our projects. one database is connected the basic user information and another database used to store the daily activities. Now need to combine two database tables .
fetch user daily activity with user information , then need to join with master databases.
I found the solution in in PHP. But i want the solution on zend framework 1.12 ?
I used multidb functionality used to fetch different action .
resources.multidb.tb.adapter = "pdo_mysql"
resources.multidb.tb.host = "localhost"
resources.multidb.tb.username = "root"
resources.multidb.tb.password = ""
resources.multidb.tb.dbname = "#####"
resources.multidb.tb.default = true
resources.multidb.pl.adapter = "pdo_mysql"
resources.multidb.pl.host = "localhost"
resources.multidb.pl.username = "root"
resources.multidb.pl.password = ""
resources.multidb.pl.dbname = "#######"
But I want to query for join 2 tables in different databases.
example
SELECT db1.table1.somefield, db2.table1.somefield FROM db1.table1
INNER JOIN db2.table1 ON db1.table1.someid = db2.table1.someid WHERE
db1.table1.somefield = 'queryCrit';
Having in mind Zend's Join Inner declaration:
public function joinInner($name, $cond, $cols = self::SQL_WILDCARD, $schema = null)
And being '$this', for example, a Zend_Db_Table_Abstract implementation with adapter set to db1 (with _setAdapter()) and schema to "#####" (this is not really necessary because it'll use it as default):
$select = $this->select(true)->setIntegrityCheck(false)
->from(array('t1'=>'table1'),array('somefield')
->joinInner(array('t1b'=>'table1'),
't1.someid = t1b.someid',
array('t1b.somefield'),
'######')
->where('t1.somefield = ?', $queryCrit);
Please, note the the fourth parameter of the Inner Join method.
Hope this helps.

Can I get steps to connect my grails to MSSQL?

Please help me with steps to connect grails to SQLserver2008.
I have been trying with setting over grails app-config.properties and DataSource.groovy file too.
File 1 : app-config.properties
dataSource.dialect =org.hibernate.dialect.SQLServerDialect
dataSource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
dataSource.url=jdbc:sqlserver:thin:#xxx.xxx.x.:1433;databaseName=xxxx
dataSource.username=sa
dataSource.password=P#ssw0rd1x
File 2 : Config.groovy
activiti {
processEngineName = "activiti-engine-default"
databaseType = "sql"
databaseSchemaUpdate = true
deploymentName = appName
Not sure if it will work with the mssql-2008, but for my application connection with mssql 2012 I have used following changes to connect it with the database:
Step1: On the datasource.groovy changed the datasource to:
dataSource {
pooled = true
driverClassName = "net.sourceforge.jtds.jdbc.Driver"
url = "jdbc:jtds:sqlserver://localhost:1433;DatabaseName=<Db-Name>"
username = "sa"
password = "root"
dbCreate = "update" // one of 'create', 'create-drop','update'
logSql = true
}
Step2. Then I have placed jtds-1.2.5.jar and jtidy-r938.jar files in the lib folder.
And I think it should connect with the mssql db, for me it worked with sql server 2012, hope it should also work with 2008.
I think need not to make any other change anywhere else. :)

How to use MySQL and MSSQL together in the grails datasource?

I have an grails application that uses MySQL for authentication purpose and another application that uses MSSQL for database stuff. I need to combine these together as one application. The datasource for MySQL contains the following
dataSource {
pooled = true
driverClassName = "org.h2.Driver"
username = "sa"
password = ""
}
The datasource for application using MSSQL contains the following
dataSource {
pooled = true
driverClassName = "com.microsoft.sqlserver.jdbc.SQLServerDriver" //jdbc driver downloaded from internet: sqljdbc4.jar and sqljdbc_auth.dll (see DisplayHistorical/grails-app/lib)
dialect = "org.hibernate.dialect.SQLServer2008Dialect"
ClassName = "org.hsqldb.jdbcDriver" //Original Code
// enable loggingSql to see sql statements in stdout
loggingSql = true
}
How would I combine these? I looked at the tutorial mentioned on this site (How do you access two databases in Grails) but it doesnt talk about adding the drivers
If you follow the link provided earlier, then you would end up with a datasource configuration like below:
environments {
production {
dataSource_authentication {
pooled = true
url = "jdbc:mysql://yourServer/yourDB"
driverClassName = "com.mysql.jdbc.Driver"
username = "yourUser"
password = "yourPassword"
........
}
dataSource {
pooled = true
driverClassName = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
dialect = "org.hibernate.dialect.SQLServer2008Dialect"
........
}
}
}
Where ever required you can use the authentication datasource explicitly.