send data from mysql to MS-SQL database - mysql

I have a table name student with fields, name and roll in my-sql database. Also, I have a same table name student in MS-SQL database with the fields, name and roll.
When a value inserted into my MySQL database student table, I need to auto-insert the same value into the MS-SQL database student table.

Why are you using two separate databases to do the same thing? can't you just choose one?
If you are just trying to export and import data you should be able to export as a csv from one database and import into the other, and then only use the database you need to.
If not i suppose you could write a class that wraps around two database objects and actually executes every query on both databases.
for example:
public class database
{
public DbConnection mySqlConnection;
public DbConnection sqlConnection;
/// <summary>
/// open two db connections
/// </summary>
public database()
{
mySqlConnection =//string that will open the mysql
mySqlconnection = //insert string will open the sql server database
}
public excecuteNonQuery(String SQL)
{
//cmd1 = mysql command
//cmd2 = mssql command
cmd1.excecute(SQL);
cmd2.execute(SQL);
}
}
And just create functions that handle preparing the commands and inserting the data, so that the operations are performed at the same time on both of the databases. I'm not aware of any method of replicating a mysql database to a mssql database.

You could have a cron that runs every so often to insert records it hasn't seen yet.
// Assuming the primary key is an id int auto_increment
startId = getLastIdFromPreviousRun();
lastId = 0;
offset = 0;
while(true) {
mysqlrs = mysqlQuery(
"SELECT * FROM users WHERE id > ? ORDER BY id LIMIT ?, 1000",
startId,
offset
);
if (!mysqlrs || mysqlrs.count() == 0)
break;
foreach(mysqlrs as mysqlrow) {
mssqlInsert('users', mysqlrow);
lastId = mysqlrow['id'];
}
offset += 1000;
}
if (lastId) {
setLastIdForNextRun(lastId);
}

Related

Automatically append database name to database table names

Is it possible to automatically append database name to a table name in laravel?
The issue is that I have to join data from multiple databases in single queries and sometime I am having to manually replace template names, which is a lot of hassle.
The only solution that I found is that I can append database name to the table name within a model, i.e.
class User extends Model
{
protected $table = 'database_name.table_name';
}
But with above we are losing support for table prefixes.
Example when database name is not applied:
$userQuery = User::where('id', 1)
->with('settings')
->select('some data');
DB::connection('x')
->table('table-on-different-connection')
->insertUsing(['some columns'], $userQuery);
$userQuery is on a different connection and database_name was not applied to the tables within that part of the query. Hence why insertUsing is trying to perform joins on connection x.
Laravel is not appending database name when generating SQL statements. To resolve that, you need to create your own MySQL wrapper and append the database name to the table name that way.
This is where the issue takes place:
vendor\laravel\framework\src\Illuminate\Database\Query\Grammar.php
public function wrapTable($table)
{
if (! $this->isExpression($table)) {
return $this->wrap($this->tablePrefix.$table, true);
}
return $this->getValue($table);
}
You need to override wrapTable method and append database name to the table that way.
i.e.
public function wrapTable($table)
{
$databaseName = $this->wrap('my_database'); // dynamically defined name here
if (! $this->isExpression($table)) {
$tableName = $this->wrap($this->tablePrefix.$table, true);
return "{$databaseName}.{$tableName}";
}
return $this->getValue("{$databaseName}.{$table}");
}
How you go about extending Grammar and override this method depends on your application and your needs. This can be done globally (i.e. via AppProvider) or for an individual query.

generic upper case search on postgres and mysql not working

I am trying to do an easy search on a table that can be on any kind of database. The following query is working an the most databases, but I cannot find a solution which works on mysql.
The tables in my database are generated by the active objects framework, so I cannot change the names or config of those instances.
Here is the query that works fine on all databases but MySQL:
select * from "AO_69D057_FILTER" where "SHARED" = true AND "CONTAINS_PROJECT" = true AND UPPER("FILTER_NAME") like UPPER('%pr%').
MySql is not able to use the table name in double quotes for some reason. If I use the unquoted table name it works on MySQL but not on Postgres. Postgres is converting the table name to lowercase because it is unquoted. AO is generating the table names in upper case.
I also tried to use an alias, but that can not work because of the evaluation hierarchy of the statement.
Any suggestions how to get rid of the table name problem?
By default double quotes are used to columns.
You can change it:
SET SQL_MODE=ANSI_QUOTES;
Here is the documentation about it:
http://dev.mysql.com/doc/refman/5.7/en/sql-mode.html
I had the same problem. I select the query according to the exception I get. In the first call of the db search, I try without quotes if it fails then I try with quotes. Then I set useQueryWithQuotes variable accordingly so that in future calls I do not need to check the exception. Below is the code snipped I am using.
private Boolean useQueryWithQuotes=null;
private final String queryWithQuotes = "\"OWNER\"=? or \"PRIVATE\"=?";
private final String queryWithoutQuotes = "OWNER=? or PRIVATE=?";
public Response getReports() {
List<ReportEntity> reports = null;
if(useQueryWithQuotes==null){
synchronized(this){
try {
reports = new ArrayList<ReportEntity>( Arrays.asList(ao.find(ReportEntity.class, Query.select().where(queryWithoutQuotes, getUserKey(), false))) );
useQueryWithQuotes = false;
} catch (net.java.ao.ActiveObjectsException e) {
log("exception:" + e);
log("trying query with quotes");
reports = new ArrayList<ReportEntity>( Arrays.asList(ao.find(ReportEntity.class, queryWithQuotes, getUserKey(), false)));
useQueryWithQuotes = true;
}
}
}else{
String query = useQueryWithQuotes ? queryWithQuotes : queryWithoutQuotes;
reports = new ArrayList<ReportEntity>( Arrays.asList(ao.find(ReportEntity.class, query, getUserKey(), false)));
}
...
}

MySQL to MSSQL keep foreign key relationships

I am currently trying to migrate from an old MySQL (5.0) to MSSQL. Because I must keep the primary key relationships, I am now facing a problem. Some data inside a table begin with the id of 6102 instead of one. I can solve this by increasing the seed, which works. Now, after several thousand of data sets, I have some leaps e.g. from id 22569 to 22597. This occurres multiple times.
What I basically do at the moment is, select all data from the source db (MySQL), map them into a generated model und try to map this model to my target model (MSSQL). (I do this because the target, new structure differs a little from the existing one.) When I ignore those leaps, I am getting later on several other tables a foreign key violation.
So my solution currently would be, to count from the beginning each mapping and when the id of the current model differs from the counter, to reset manually the seed in the database.
DBCC CHECKIDENT (mytable, RESEED, idFromCurrentModel);
Is there a possibility to force entity framework, respectively SQL Server to accept the id from my model instead of ignoring it and use the ident value?
Thanks for reading and best regards
EDIT
Just if anybody is wondering how I solved this, here it is:
var context = new TestEntities();
// map mysql data to mssql model and convert data
// let's assume I did this
var mapped = new List<Test>()
{
new Test() {id= 42, bar = "foo", created = DateTime.Now},
new Test() {id= 1337, bar = "bar", created = DateTime.Now}
};
var transaction = context.Database.BeginTransaction();
context.Database.ExecuteSqlCommand("SET IDENTITY_INSERT [SeedingTest].[dbo].[Test] ON");
context.Test.AddRange(mapped);
context.SaveChanges();
context.Database.ExecuteSqlCommand("SET IDENTITY_INSERT [SeedingTest].[dbo].[Test] OFF");
transaction.Commit();
context.Dispose();
This only works when I do this:
Right click on my *.edmx file, open, and remove the
StoreGeneratedPattern="Identity"
in your identity column. In my case this looked like this:
...
<Property Name="id" Type="int" Nullable="false" StoreGeneratedPattern="Identity" />
...
After removing this, EF was no longer ignoring my set id's.
Source
Additional information:
Adding this attribute
[DatabaseGenerated(DatabaseGeneratedOption.None)]
to my id in my generated model, did not work.
[DatabaseGenerated(DatabaseGeneratedOption.None)]
public int foo { get; set; }
public string bar { get; set; }
...
The easiest way is to set the seed initially to the current max Id value + 1. Then, when inserting the converted rows do the following
set identity_insert on tablename -- stops generation of IDENTITY, requires user to supply it
insert into tablename values (Id....) -- supply value of Id
set identity_insert_off tablename -- turn inedtity generation back on
This does it quite nicely.

LINQ to SQL - Insert New Record

I have a database table with an auto-incrementing ID field. this table is exposed to by code via a Entity Data Model. I am trying to write a new record to this table. I have a method that needs to be responsible for creating these records. This method takes in the property values of the record. It needs to create a record, and write a reference record in another table. Currently, here is what I am trying
public int CreateRecord(string name, string description, List<int> ids)
{
using (DatabaseContext database = new DatabaseContext())
{
Record record = new Record();
record.Name = name;
record.Description = description;
database.Records.InsertOnSubmit(record);
database.SubmitChanges();
List<RecordTrack> tracks = new List<RecordTrack>();
foreach (int id in ids)
{
RecordTrack track = new RecordTrack();
track.RecordID = record.ID;
track.ID = id;
tracks.Add(track);
}
database.Tracks.InsertAllOnSubmit(tracks);
database.SubmitChanges();
}
}
I can't seem to get the record to save in this manner. I was able to do it when I passed a Record in that I wanted to insert. But due to other factors, I need a way to purely create the record here from scratch. What am I doing wrong?
Thank you!
there should be a AddToRecord() function in your database context. Use that function to add your record variable and then call SaveChanges() from your database context.

Linq to SQL concurrency problem

Hallo,
I have web service that has multiple methods that can be called. Each time one of these methods is called I am logging the call to a statistics database so we know how many times each method is called each month and the average process time.
Each time I log statistic data I first check the database to see if that method for the current month already exists, if not the row is created and added. If it already exists I update the needed columns to the database.
My problem is that sometimes when I update a row I get the "Row not found or changed" exception and yes I know it is because the row has been modified since I read it.
To solve this I have tried using the following without success:
Use using around my datacontext.
Use using around a TransactionScope.
Use a mutex, this doesn’t work because the web service is (not sure I am calling it the right think) replicated out on different PC for performance but still using the same database.
Resolve concurrency conflict in the exception, this doesn’t work because I need to get the new database value and add a value to it.
Below I have added the code used to log the statistics data. Any help would be appreciated very much.
public class StatisticsGateway : IStatisticsGateway
{
#region member variables
private StatisticsDataContext db;
#endregion
#region Singleton
[ThreadStatic]
private static IStatisticsGateway instance;
[ThreadStatic]
private static DateTime lastEntryTime = DateTime.MinValue;
public static IStatisticsGateway Instance
{
get
{
if (!lastEntryTime.Equals(OperationState.EntryTime) || instance == null)
{
instance = new StatisticsGateway();
lastEntryTime = OperationState.EntryTime;
}
return instance;
}
}
#endregion
#region constructor / initialize
private StatisticsGateway()
{
var configurationAppSettings = new System.Configuration.AppSettingsReader();
var connectionString = ((string)(configurationAppSettings.GetValue("sqlConnection1.ConnectionString", typeof(string))));
db = new StatisticsDataContext(connectionString);
}
#endregion
#region IStatisticsGateway members
public void AddStatisticRecord(StatisticRecord record)
{
using (db)
{
var existing = db.Statistics.SingleOrDefault(p => p.MethodName == record.MethodName &&
p.CountryID == record.CountryID &&
p.TokenType == record.TokenType &&
p.Year == record.Year &&
p.Month == record.Month);
if (existing == null)
{
//Add new row
this.AddNewRecord(record);
return;
}
//Update
existing.Count += record.Count;
existing.TotalTimeValue += record.TotalTimeValue;
db.SubmitChanges();
}
}
I would suggest letting SQL Server deal with the concurrency.
Here's how:
Create a stored procedure that accepts your log values (method name, month/date, and execution statistics) as arguments.
In the stored procedure, before anything else, get an application lock as described here, and here. Now you can be sure only one instance of the stored procedure will be running at once. (Disclaimer! I have not tried sp_getapplock myself. Just saying. But it seems fairly straightforward, given all the examples out there on the interwebs.)
Next, in the stored procedure, query the log table for a current-month's entry for the method to determine whether to insert or update, and then do the insert or update.
As you may know, in VS you can drag stored procedures from the Server Explorer into the DBML designer for easy access with LINQ to SQL.
If you're trying to avoid stored procedures then this solution obviously won't be for you, but it's how I'd solve it easily and quickly. Hope it helps!
If you don't want to use the stored procedure approach, a crude way of dealing with it would simply be retrying on that specific exception. E.g:
int maxRetryCount = 5;
for (int i = 0; i < maxRetryCount; i++)
{
try
{
QueryAndUpdateDB();
break;
}
catch(RowUpdateException ex)
{
if (i == maxRetryCount) throw;
}
}
I have not used the sp_getapplock, instead I have used HOLDLOCK and ROWLOCK as seen below:
CREATE PROCEDURE [dbo].[UpdateStatistics]
#MethodName as varchar(50) = null,
#CountryID as varchar(2) = null,
#TokenType as varchar(5) = null,
#Year as int,
#Month as int,
#Count bigint,
#TotalTimeValue bigint
AS
BEGIN
SET NOCOUNT ON;
BEGIN TRAN
UPDATE dbo.[Statistics]
WITH (HOLDLOCK, ROWLOCK)
SET Count = Count + #Count
WHERE MethodName=#MethodName and CountryID=#CountryID and TokenType=#TokenType and Year=#Year and Month=#Month
IF ##ROWCOUNT=0
INSERT INTO dbo.[Statistics] (MethodName, CountryID, TokenType, TotalTimeValue, Year, Month, Count) values (#MethodName, #CountryID, #TokenType, #TotalTimeValue, #Year, #Month, #Count)
COMMIT TRAN
END
GO
I have tested it by calling my web service methods by multiple threads simultaneous and each call is logged without any problems.