SnappyData v.0.5
I cannot seem to create row tables for a specific schema. This is important in a schema-based multi-tenant application where each tenant has his own schema.
However, when I create my tables using RowStore DDL, they are queryable is all schemas for the DB.
Here were my steps. Did I do something wrong?
ubuntu#ip-172-x-x-x:~$ snappy-shell
SnappyData RowStore 1.5.0 GA
snappy> connect client '172.x.x.x:1527';
Using CONNECTION0
**snappy> set schema A;**
0 rows inserted/updated/deleted
snappy> run '/home/ubuntu/data/ddl/create_row_tables.sql';
snappy> DROP TABLE IF EXISTS road;
0 rows inserted/updated/deleted
snappy>
CREATE TABLE road
(
road_id VARCHAR(64) NOT NULL,
name VARCHAR(64) NOT NULL,
CONSTRAINT road_PK PRIMARY KEY (road_id)
)
PERSISTENT;
0 rows inserted/updated/deleted
In DBVisualizer using JDBC, I have the following schemas: A, APP, NULLID, Q, SQLQ, etc.
When I change DBVisualizer to point to a specific schema, and run:
select * from road;
The query returns zero rows on ALL SCHEMAS. I would expect a 'Table not found:ROAD;' error on all schemas except "A". What do I need to do to create the tables only on a specific schema?
The schema integration of the store with Spark metadata had some issues which have been fixed in recent builds. As of the released version, you will need to use fully qualified names like:
create table a.road ...
select * from a.road
Btw, if you run the cluster as a pure rowstore (using "snappy-start-all.sh rowstore"), then schema should work as expected.
Related
I’m attempting to use a piece of software (Layer2 Cloud Connector) to sync a local SQL table (Sage software) to a remote MySQL database where the data is used reports generated via the company's web app. We are doing this with about 12 tables, and have been doing so for almost two years without any issues.
Background:
I’m using a simple piece of software the uses a SELECT statement to sync records from one table to another using ODBC. In this case from SQL (SQLTable) to MySQL (MySQLTable). To do so, the software requires a SELECT statement for each table, a PK field, and, being ODBC-based, a provider. For SQL I'm using the Actian Zen 4.5, and for MySQL I'm using the MySQL ODBC 5.3.
Here is a screenshot of what the setup screen looks like for each of the tables. I have omitted the other column names that I'm syncing to make the SELECT statement more readable. The other columns are primarily varchar or int types.
Problem
For unrelated reasons, we must now sync a new table. Like most of the other tables, it has a primary key column named rGUID of type binary. When initially setting up the other tables, I tried to sync the primary key as a binary type to a MySQL binary column, but it failed when attempting to verify the SELECT statement on the SQLServer side with the error “Cannot remove this column, because it is a part of the constraint Constraint1 on the table SQLTable”.
Example of what I see for the the GUID/rGUID primary key values stored in the SQLTable via Access, or in MySQL after syncing as string:
¡狻➽䪏蚯㰛蓪
Ҝ諺䖷ᦶ肸邅
ब惈蠷䯧몰吲론�
ॺ䀙㚪䄔麽骧⸍薉
To get around this, I use CAST in the SQLTable SELECT statement to CAST the binary value as a string using: CAST(GUID as nchar(8)) as GUID, and then set up the MySQL column as a VARCHAR(32) using utf8_general_ci collation.
This has worked great for every other table since we originally set this up. But this additional table has considerably more records (about 120,000 versus 5,000-10,000), and though I’m able to sync 10,000 – 15,000 successfully, when I try to sync the entire table I get about 10-12 errors such as:
The metabase record 'd36d2dbe-fa89-4712-be4c-6b212367004b' is marked
to be added. The table 'SQLTable' does not contain a corresponding
row. Changes made to this metabase record will be reset to the
initial state.
I don't understand what is causing the above error or how to work past it.
What I’ve tried so far:
I’ve confirmed the SQLTable has no other unique fields that could be
used as PK in place of the rGUID column
I’ve tried use different type, length and collation settings on the
MySQL table, and have had mixed success, but ultimately still get
errors when attempting to sync the entire table.
I’ve also tried tweaking the CAST settings for the SQL SELECT
statement, but nchar(8) seems to work best for the other tables
I've tried syncing using HASHBYTES('SHA1', GUID) as GUID and syncing
the value of that, but get the below ODBC error
I was thinking perhaps I could convert the SQL GUID to its value, then sync that as a varchar (or a binary), but my attempts at using CONVERT in the SQLTable SELECT statement have failed
Settings I used for all the other tables:
SQL SELECT Statement: SELECT CAST(GUID as nchar(8)) as GUID, OtherColumns FROM SQLTable;
MYSQL SELECT Statement: SELECT GUID, OtherColumns FROM MySQLTable;
Primary Key Field: GUID
Primary Key Field Type: String
MySQL Column Type/Collation: VARCHAR(32), utf8_general_ci
Any help or suggestions at all would be great. I've been troubleshooting this in my spare time for a couple of weeks now, and have no had much success. I'm not particularly familiar with the binary type, and am hoping someone might have an idea on how I might be able to successfully sync this SQL table to MySQL without these errors.
Given the small size of the datasets involved I would select as CHAR(36) from SQL Server and store in a CHAR(36) in MySQL.
If you are able to control the way the data is inserted by Layer2 Cloud Connector then you could set your MySQLTable GUID column as BINARY(16) -
SELECT CAST(GUID AS CHAR(36)) AS GUID, OtherColumns FROM SQLTable;
INSERT INTO MySQLTable (GUID) VALUES (UUID_TO_BIN(GUID)))
SELECT BIN_TO_UUID(GUID) AS GUID, OtherColumns FROM MySQLTable;
I want to be able to update a specific column of a table using data from another table. Here's what the two tables look like, the DB type and SSIS components used to get the tables data (btw, both ID and Code are unique).
Table1(ID, Code, Description) [T-SQL DB accessed using ADO NET Source component]
Table2(..., Code, Description,...) [MySQL DB accessed using ODBC Source component]
I want to update the column Table1.Description using the Table2.Description by matching them with the right Code first (because Table1.Code is the same as Table2.Code).
What i tried:
Doing a Merge Join transformation using the Code column but I couldn't figure out how to reinsert the table because since Table1 has relationships i can't simply drop the table and replace it with the new one
Using a Lookup transformation but since both tables are not the same type it didn't allow me to create the lookup table's connection manager (which would be for in my case MySQL)
I'm still new to SSIS but any ideas or help would be greatly appreciated
My solution is based on #Akina's comments. Although using a linked server would've definitely fit, my requirement is to make an SSIS package to take care of migrating some old data.
The first and last are SQL tasks, while the Migrate ICDDx is the DFT that transfers the data to a staging table created during the first SQL task.
Here's the SQL commands that gets executed during Create Staging Table :
DROP TABLE IF EXISTS [tempdb].[##stagedICDDx];
CREATE TABLE ##stagedICDDx (
ID INT NOT NULL,
Code VARCHAR(15) NOT NULL,
Description NVARCHAR(500) NOT NULL,
........
);
and here's the sql command (based on #Akina's comment) for transferring from staged to final (inside Transfer Staged):
UPDATE [MyDB].[dbo].[ICDDx]
SET [ICDDx].[Description] = [##stagedICDDx].[Description]
FROM [dbo].[##stagedICDDx]
WHERE [ICDDx].[Code]=[##stagedICDDx].[Code]
GO
Here's the DFT used (both TSQL and MySQL sources return sorted output using ORDER BY Code, so i didnt have to insert Sort components before the Merge Join) :
Note: Btw, you have to setup the connection manager to retain/reuse the same connection so that the temporary table doesn't get deleted before we transfer data to it. If all goes well, then after the Transfer Staged SQL Task, the connection would be closed and the global temporary table would be deleted.
I have two databases on two servers.
The first one contains many tables which contains many codes.
An example is "Products" Table which contains the column "ProductCode".
Lets say there are 5 distinct records in that column i.e ProductCode1 -> ProductCode5.
The second database contains all the fields from each table defined in the first database.
I use the second database to provide definitions for each code found in the first database. I have migrated all the data from all the tables in the first db over to the new one manually via an excel file and script.
However, I would like to create an SQL function which scans the first database and when it finds new rows of data, it adds that data to the second database.
This would save me the hassle of querying all the tables individually and then adding them manually, as i originally did.
Please note that both databases are stored on separate servers.
Is this possible to achieve?
Or is there any better options?
While there are recovery backup and replication methods, if both databases maintain different data, there is no single, convenient SQL function to migrate new data in all tables from one database to another. However, you can build an .sql script or stored procedure that runs duplicate-avoid queries for new data.
Consider following steps where 1 and 2 are to be run for each table:
Create Federated Table from remote MySQL database to be locally available for querying but physical storage remains in remote database. See overview of Federated Storage Engine. Note: this step needs to only be run once for each needed table.
CREATE TABLE federated_table (
id INT(20) NOT NULL AUTO_INCREMENT,
name VARCHAR(32) NOT NULL DEFAULT '',
other INT(20) NOT NULL DEFAULT '0',
PRIMARY KEY (id), INDEX name (name),
INDEX other_key (other)
)
ENGINE=FEDERATED
DEFAULT CHARSET=utf8mb4
CONNECTION='mysql://fed_user#remote_host:9306/federated/test_table
Federated table schema must be identical to remote table. Therefore, align data types of CREATE TABLE to output of SHOW CREATE TABLE in remote database.
Run SQL duplicate-avoid queries such as NOT IN vs. NOT EXISTS vs. LEFT JOIN / IS NULL. One other method is EXCEPT (same family as UNION and INTERSECT operators):
INSERT INTO Products (Col1, Col2, Col3, ...)
SELECT Col1, Col2, Col3, ...
FROM my_federated_products_table
EXCEPT
SELECT Col1, Col2, Col3, ...
FROM Products
Automate step 2 for each table into a single stored procedure (or .sql script) to be run multiple times in future.
DELIMITER $
CREATE PROCEDURE migrate_new_data()
BEGIN
-- ALL INSERT INTO STATEMENTS FROM FEDERATED TABLES
END $
DELIMITER ;
Run procedure each time from Excel, workbench, command line, or elsewhere which can serve as your SQL function:
CALL migrate_new_data;
Looks like FEDERATED Storage is what your are looking for.
From the docs:
The FEDERATED storage engine lets you access data from a remote MySQL database without using replication or cluster technology. Querying a local FEDERATED table automatically pulls the data from the remote (federated) tables. No data is stored on the local tables.
Here is and article that shows how to configure it: https://medium.com/#techrandomthoughts/setting-up-federated-tables-in-mysql-8a17520b988c
My project has its own database. Also, I use table of users, which is on other database. Two offices have their data on the same server, but third one has its own user table on other server.
So, in lots of queries I need to join either table some_db.users or other_server.some_db.users
What solution would you advise for this scenario?
I use MySQL.
There is Federated tables in MySQL:
The FEDERATED storage engine lets you access data from a remote MySQL
database without using replication or cluster technology. Querying a
local FEDERATED table automatically pulls the data from the remote
(federated) tables. No data is stored on the local tables.
First, you must have a table on the remote server that you want to access by using a FEDERATED table. Suppose that the remote table is in the sakila database and is defined like this:
CREATE TABLE test_table (
id INT(20) NOT NULL AUTO_INCREMENT,
name VARCHAR(32) NOT NULL
PRIMARY KEY (id)
)
ENGINE=MyISAM
DEFAULT CHARSET=latin1;
Next, create a FEDERATED table on the local server for accessing the remote table:
CREATE TABLE federated_table (
id INT(20) NOT NULL AUTO_INCREMENT,
name VARCHAR(32) NOT NULL
PRIMARY KEY (id)
)
ENGINE=FEDERATED
DEFAULT CHARSET=latin1
CONNECTION='mysql://fed_user:fed_user#197.186.1.199:3306/sakila/test_table';
Sample connection strings:
CONNECTION='mysql://username:password#hostname:port/database/tablename'
CONNECTION='mysql://username#hostname/database/tablename'
CONNECTION='mysql://username:password#hostname/database/tablename'
The basic structure of this table should match that of the remote table, except that the ENGINE table option should be FEDERATED.
Execute:
show variables like '%federated%';
to check if FEDERATED storage engine is available on your local server.
The table federated_table in localhost becomes virtual table of test_table in remote server.
Now you can use the JOIN between the tables in a DB in the localhost server. If there is a table called test in your localhost server, and you want to JOIN with the former sakila.test_table which is in the remote server, write a query like the one shown below:
SELECT * FROM `federated_table` JOIN `test`;
The federated_table in the query will actually refer to test_table in remote server.
On enabling FEDERATED Storage Engine
The FEDERATED storage engine is not enabled by default in the running server; to enable FEDERATED, you must start the MySQL server binary using the --federated option.
NOTE:
Optional storage engines require privileges and will fail to load when --skip-grant-tables is specified.
The result the entire db will fail to load and the following error will appear in the logs:
110318 21:37:23 [ERROR] /usr/local/libexec/mysqld: unknown option '--federated'
This in turn means that an upgrade from 5.x needs to be done in two steps if you have federated tables. Once with --skip-grant-tables and without --federated, the once without --skip-grant-tables and with --federated.
Source: The FEDERATED Storage Engine
Please mention the databse also.
In SQLServer you can use Linked sever. http://msdn.microsoft.com/en-us/library/ms188279.aspx
In MySQL, you can join tables from different databases using fully qualified names like
`database_name1`. `table_name1` JOIN `database_name2`.`table_name2`
But i fear, you cant join tables from different servers because for that you need to have two different connections and as per my knowledge there are no fully qualified connection names to be used in the query.
Alternatively, you can create local temporary table(s) on one of the servers and run the query there on. But in this case you will need to transfer data from one server to another. You can use MySQL GUI tool like SQLyog or MySQL admin, to transfer data from one server to another and to synchronize databases on two servers.
Hope it helps....
Federated tables are your solution for tables on other servers. They are very slow though if you perform joins on them.
If you just want to read data from another database on the same server you can use a view. This way you have all tables virtually in one database and you have to open only one connection in your application.
CREATE
VIEW `my_db`.`table_name`
AS
(SELECT * FROM `other_db`.`table_name`);
Have a Django / MySQL set up. There's a model, Survey, which currently looks like...
class Survey(models.Model):
company = models.ForeignKey('Company')
I want to set up the model, so company can be a null value:
company = models.ForeignKey('Company', blank = True, null = True)
However, I'm not sure what I should do on the MySQL side to ensure all the existing constraints / models. Do I just alter the column through the console to accept null values? It's a live database, so I don't want to experiment too much (my development environment uses SqlLite3).
Update your model so that blank=True, null=True. Then run the sqlall command on your production server (so that it gives the output for MySQL)
./manage.py sqlall myapp
Find the create table statement This will show the new definition for the survey_id field.
CREATE TABLE `myapp_survey` (
...
`survey_id` integer
...
Then, in your database shell, modify the column to accept null values using the ALTER TABLE command.
ALTER TABLE myapp_survey MODIFY company integer;
Be careful, and consider whether you want to run MySQL in your development environment as well. Do you really want to be copying and pasting commands from Stack Overflow into your live DB shell without testing them first?