Operating system on all servers: Windows Server 2008 R2.
Publisher: Sql Server 2008 R2 Standard
Distributor: SQL Server 2008 R2 Standard
Web Synchronization Agent: sqlce35.dll under IIS 7.5
Subscriber: Windows XP SP 3 or Windows 7 SP1
SQL CE Client 3.1
I have an issue where a merge replication stops updating subscriptions without knowing why.
Premises:
1 Publication
15 items filtered by hostname(). All set to download, None in bidirectional mode.
20 or 30 subscribers
You create a merge replication with several articles (15 tables) filtered by a HostName (). This replica push over 20 or 30 subscribers and the synchronization is done correctly. Data collection is done by request from the subscriber, it pull changes two or three times a day. And all the changes are received by the subscriber without any problem.
All this works fine until there comes a time, after a few days without problems the replica will no longer updating a few changes to the subscriber. We search on publicator and have the changes, check the subscriber and not have them. Returning to modify the changes in publicator, sometimes if they are updated in the subscriber and sometimes not.
The problem is that this replica is no longer reliable, we do not know that it is updated and what not.
Focusing on a single subscription-EVDBASD342232 '013243 ... 'and a single article' table1 'is not getting new data made the following verification steps
Run "sp_showpendingchanges NULL, NULL, 'table1', 1" means the
procedure returns me a series of rows that correspond to the data
that should be replicated, I have found that the ID matches the
subscription-EVDBASD342232 '013243 ... 'and the guid corresponds to
the row on the 'table1' that should be replicated.
We request subscription and we observe in the Replication Monitor to
view subscription-EVDBASD342232 '013243 ... 'indicating 0 changes and
everything is correct, shows no errors but will not say anything
pending synchronization.
After analyzing all the data we do not understand is what is going wrong in the process.
Once a replica does not synchronize, the others do not either.
Please if anyone can help, thanks in advance.
If you need more clarification or details
SQL CE has known issues with replication.
Most advice is to not use CE.
Try contacting Hilary Cotter - I think he has a blog and a twitter.
Twitter #SQLHELP is a great place to get an answer as soon as possible. Most of the SQL gurus are there to help you.
Related
I am an ex multi value developer that over the last 6 months have been thrust in to the world of SQL and apologies in advance for the length of the question. So far I have got by with general instinct (maybe some ignorance!) and a lot of help from the good people on this site answering questions previously asked.
First some background …
I have an existing reporting database (SQL Server) and a new application (using MySQL) that I am looking to copy data from at either 30 min, hourly or daily intervals (will be based on reporting needs). I have a linked server created so that I can see the MySQL database from SQL Server and have relevant privileges on both databases to do read/writes/updates etc.
The data that I am looking to move to reporting on the 30 minute or hourly schedule typically are header/transactions by nature have both created and modified date/time stamp columns available for use.
Looking at the reporting DBs other feeds, Merge is the statement used most frequently across linked servers but to other SQL server databases. The merge statements also seem to do a full table to table comparison which in some cases takes a while (>5mins) to complete. Whilst the merge seems to be a safe options I do notice a performance hit on reporting whist the larger tables are being processed.
In looking at delta loads only, using dynamic date ranges (eg between -1 hour:00:00 and -1 hour:59:59) on created and modified time stamps, my concern would be the failure of any one job execution could leave the databases out of sync.
Rather than initially ask for specific sql statements what I am looking for is a general approach/statement design for the more regular (hourly) executed statements with the ideal being just to perform delta loads of the new or modified rows safely with a SQL Server to MySQL connection.
I hope the information given is sufficient and any help/suggestions/pointers to reading material gratefully accepted.
Thanks in advance
Darren
I have done a bit of “playing” over the weekend.
The approach I have working pulls the data (inserts and updates) from MySQL via openquery into a CTE. I then merge the CTE into the SQL Server table.
The openquery seems slow (by comparison to other linked tables) but the merge is much faster due to limiting the amount of source data.
I have attempted to find the answer here and via Google on how to control connections for a linked server ODBC connection.
Overview
I have a linked server from SQL Server 2014 to MySQL for the purposes of extracting data for our data warehouse. I've queried the database quite a few times without issue. Then yesterday, suddenly the query to read from the table is slow, and then I get reports that the application using this MySQL database are getting a "too many connections" error.
Details
The following query selects the data from MySQL and inserts to the SQL Server table.
INSERT INTO tmpCustomers
(fieldlist)
SELECT
myc.contact_id,
myl.franchise_id,
myl.lead_source,
LEFT(RTRIM(myc.first_name) + ' ' + RTRIM(myc.last_name),100) AS Name,
myc.first_name,
myc.last_name,
myc.company,
myc.Email,
myc.primary_phone,
myc.home_phone,
myc.mobile_phone,
myc.work_phone,
myc.fax,
myc.address1,
myc.Address2,
myc.City,
myc.[state],
myc.zip_code,
myc.created_date,
myc.updated_date
FROM [MYSQLDB]...[franchise] myf
INNER JOIN [MYSQLDB]...[leads] myl
ON myl.franchise_id = myf.franchise_id
INNER JOIN [MYSQLDBE]...[contact] myc
ON myc.contact_id = myl.contact_id
This query returns about 200K rows of data, and will grow. The MySQL database is used by our customer base, and this is a back-end process to pull data into our data warehouse.
The query has been working without issue over the past week of testing, until yesterday, where it caused our MySQL support to restart the MySQL server twice.
The ODBC setup was done using the "mysql-connector-odbc-5.3.6-win64.msi" version. I don't find any settings there to limit the number of connections. ODBC does show "Allow multiple statements", which this is not. It also has "Enable automatic reconnect", which I can't imagine why for a single query would be needed.
Summary
I can't afford to stop customers from connecting, and need to disable the process from using too many connections when doing the import.
Any input on this would be greatly appreciated.
Thanks
KDS
Update: 2016-Oct-05
AWS server - M3.xlarge
4 CPU
15 GiB
2 40 GiB SSD drives
It's better to optimize the MySQL server if you can't afford to stop customers from connecting.
With this much information, it hard to optimize or suggest something for MySQL optimization.
https://dev.mysql.com/doc/refman/5.5/en/too-many-connections.html
Better to update your configuration file. Or max_connections limit and InnoDB variable if you are using innoDB. And RAM also.
Can you update the above information in question section.
I'm going to mark this as answered as it's been about a year and no real solution to it. The issue was locks on the MySQL server as the SQL Server linked server was reading the data. SQL Server arguments like NOLOCK had no impact on resolving this.
So, what was done was to take a backup of the MySQL database nightly and restore it to a separate database that we linked to for SQL Server, and process the data from there. The reads are usually done in a matter of a minute or two. SQL Server was still putting a lock on the MySQL table, and users then started to stack multiple connections until all the connections to MySQL were used up.
So, since I only needed the data for reporting purposes daily, this separate database copy worked, but I don't know of any other fix to this.
Thanks
KD
I have one problem with FEDERATED table in MySQL. I have one server (MySQL version 5.0.51a), who serve to store client data and actually nothing more. The logic database are stored in another server (version 5.1.56), sometimes it should handle that data from first server. So the second server have one FEDERATED table, which connect to the first server.
Actually, it has worked without any problems, but recently I got strange errors with this solution. Some kind of queries on second server cannot be performed correctly.
For example SELECT * FROM table - doesn't work. It hangs exactly 3 minutes and then gives:
Error Code: 1159 Got timeout reading communication packets
Ok, I checked table on the first server and it's Ok. Then I tried some another queries to FEDERATED table and they work...
For example, query like SELECT * FROM table WHERE id=x returns the result. Probably it could have problem with size of result, so I tried query with dummy WHERE-clause like SELECT * FROM table WHERE id > 0 - and it also works...
Finally I found a "solution", which helped only for two days - on the first server I made a copy of table, and on second server I re-declared a new FEDERATED table with new connection string to this copy. And it works, but after two days the same problem with new copied table.
I've already talk with both server providers, they see no problems, everything seems to work and other hosting provider is the causer of problems.
I've checked all variables in MySQL and there is no timeout parameter with 3 minutes etc. So how can I deal so kind of problems? It seems to be something automatic on network or database side, but I don't know, how to detect the reason of problems.
Do You have any ideas?
You may try checking MTU size settings for network interfaces on both servers.
This warning is logged when idle threads are killed by wait_timeout.
Normally, the way to avoid threads getting killed by wait_timeout is to call mysql_close() in scripts when the connection is no longer needed. Unfortunately that doesn't work for queries made through federated tables because the query and the connection are not on the same server.
For example, when a query is executed on server A of a federated table (pointing to data on server B), it creates a connection on server B. Then when you run mysql_close() on server A it obviously can not close the connection that was created on server B.
Eventually the connection gets killed by mysql after the number of seconds specified in "wait_timeout" have passed (the default is 8 hours). This generates the warning in your mysqlerror.log "Got timeout reading communication packets"
We are getting exceptions like this
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#5b7a7896 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#55bc5e2a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#41ca435f (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#460d33b7 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
when load testing our application on MSSQL 2008 R2 (jTDS or official MS JDBC doesn't matter). We never get this exception when running the same tests against PostgreSQL or MySQL.
We don't just want to increase the number of helper threads for c3p0 (which solves the problem, but how long?). We want to know what's the problem as it is workings with other DBMS'.
The applications behaves like:
Send X requests
Wait for a while -> DEADLOCK
Send X requests
Wait for a while -> DEADLOCK
Does anyone know or has an idea why we have this behavior with MSSQL?
Thanks, Adrian
(Btw. BoneCP works without any problem too.)
SQL Server has a much more restrictive locking strategy compared to PostgreSQL or InnoDB.
Especially it will block SELECTs on rows (tables?) that are updated from a different connection/transaction (in the default installation).
You should make sure that you are not selecting the same rows in one session that are being updated from another.
If you can't change the sequence of your code, you might get away with using "dirty reads" in SQL Server.
If I remember that correctly, this is accomplished by adding WITH NOLOCK to the SELECT statements (but I'm not entirely sure)
Edit
A different possibility (if you are on SQL Server 2005 or later) would be to use the new "snapshot isolation" to avoid blocking selects.
I have an SSIS package that connects to a mysql server and attempts to pulls data from different tables and inserts the rows into a SQL Server 2005 database.
One issue i notice is that at any given time it runs, regardless of what step it is on, it almost always fails to bring in the total records from mysql into sql server.
there are no errors thrown.
One morning it will have all 11M records and on the next run anywhere between 3K and 17K records.
Anyone notice anything like this?
I import data from two separate MySQL databases -- one over the Internet and one in-house. I've never had this type of issue. Can you describe how you setup your connection to the MySQL database? I used the ODBC driver available on the MySQL website and connect using an ADO.NET data source in my data flow that references the ODBC connection.
One possible way you could at least prevent yourself from loading incomplete data is only load new records. If the source table has an ID and the records never change once they are inserted, then you could feed in the maximum ID by checking your database first.
Another possible way to prevent loading incomplete data is loading the MySQL database into a staging table on your destination server and then only load records you haven't already loaded.
Yet another way to do it is load the data into a staging table, verify the records are greater than some minimum threshold such as the row count of the target table or the expected minimum number of transactions per day and then only commit the changes after this validation. If the rows are insufficent, then raise an error on the package and send a notification email. The advantage of raising an error is you can set your SQL Server Agent job to retry the step for a defined number of attempts to see if this resolves the issue.
Hope these tips help even if they don't directly address the root cause of your problem.
I've only tried MySQL -> SQL Server via SSIS once, but the error I found related to MySQL datetimes not converting to SQL Server datetimes. I would have thought this would break the whole dataflow, but depending on your configuration you could have set it purely to ignore bad rows?