Each time I perform a SQL Server 2008 FTS Catalog Rebuild, my server is left in a state where the SQL Server process is at 2.8+ GB of memory and is causing performance degradation of the machine. When recycling the SQL Server service, its goes back down to around 350 MB and performance is back to normal.
Is it normal to have to restart the service after a Full Text Rebuild? Not sure what is going on truly behind the scenes but seems like it is a significant memory leak if I could call it that.
I'm open for any advice. Thanks in advance!
Sql server does consume a lot of memory when dealing with full text. It is a normal behavior. Populate a full text catalog is hard task and it requires a lot of processing and memory usage. But there are some things you can do to minimize those effects.
1 - Limit sql server memory usage - Sql Server
2 - Make sure your server collation and database collation are the same to avoid the high usage of tempdb (sql tries to convert database collation to match instance collation).
3 - Use incremental population instead of Full rebuild. Check out the link bellow to see how to do that.
4- Make sure you have sql server 2008 SP2 installed.
5 - Use stop lists.
Also check out this link 10 Ways to Optimize SQL Server Full-text Indexing. It gives nice information you can use to maximize your server performance.
Related
I've got a VB.Net program that updates an SQL Server 2008R2 database.
The disk is using 86.2Gb of 126Gb so there's lots of free space.
The autogrowth is set, as you can see, but I get the following error message on various threads:
Could not allocate space for object 'sys.syscolpars'.'nc' in database 'MyDB' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
The MDF file is currently 9Gb in size.
If Autogrowth is set, and there'e sufficient disk space, why is this happening?
EDIT:
OK, further testing has provided the following...
The SQL server in this case is not 2008R2 as suspected, but 2014 SQLExpress... sorry for the bum steer on that; I got my servers mixed up...
When I try this on a 2008R2 set up exactly the same, it seems to work ok without producing the error message.
So, the question seems to be is SQLexpress limited in size?
The maximum size for the MDF file on SQL Express 2014 is currently 10GB so I imagine its trying to grow above that.
If you review the Cross-Box Scale Limits section (Maximum relational database size) with Features Supported by the Editions of SQL Server 2014 you will see why.
I had been using MySQL server for all my database tasks but it's always giving me problems when trying to connect.Is it possible to use some different server like JBOSS, wildfly or Tomcat to host my SQL database?
If you have problems with the connection, I'd rather advocate for fixing those connection problems than jumping ship just because of this, because you'd give up a lot of additional features that a server like mysql provide under the hood (if only the ability to run an application independent backup).
That being said, there's hsqldb which you can run embedded. Note though that this won't be good if you have large amounts of data (as it's generally "in memory") and the file format is not the most robust: It's saving its content in a raw text format that will sometimes be rewritten. If your server just crashes or runs out of memory, this might corrupt your database. Full fledged servers (like mysql or postgresql or others) typically go quite some length to save you from loss of data in these cases.
hsqldb would basically just be a driver that you embed in your web application and configure its storage location through the database URL that you specify.
Perfect for quick demos, proof of concept, development and test. Your mileage may vary for actual production deployment where you can't risk to ever lose your data.
I haven't got a lot of ETL experience but I haven't found the answer to my question either, although I guess it may be a no-brainer if you've worked with it. We're currently looking into creating a simple data warehouse (simple as in "copy most columns from most tables" and not OLAP-style) and it seems we're leaning towards SQL Server (2008) for a few reasons.
SSIS seems to be the tool for this kind of tasks when it comes to SQL Server, but I can't find anything about how it is affecting the source database cache, if at all, when loading data. Some of our installations are very sensitive performance-wise when it comes to having a usage-style-cache.
But if SSIS runs a "select *"-ish query and the cache is altered, then the performance for the users may degrade to unacceptable levels until it is rebuild from those queries again.
So my question is, does SSIS (or is there a way to avoid) affect the database cache when loading data from a SQL Server database?
Part of the problem is also that the source database could be both an Oracle or SQL Server database, so if there is a way to avoid the cache-affecting part for Oracle, that would be good input as well. (I guess the Attunity connector is the way to go?)
(Some additional info: We have considered plain files as well, but then export-import probably takes longer time than SSIS-transfer? I also guess change data capture is something we'll also look into, so if that is relevant to this question, feel free to include possible issues/benefits.)
Any other relevant suggestions are also welcome!
Thanks!
Tackling the SQL Server side:
First off, SSIS doesn't do anything special to avoid the buffer pool, or the plan cache.
Simple test (on a NON-production instance!):
Create a new SSIS package with a single connection manaager, and a single data flow containing one OLE DB Source, pointing to a table, similar to:
Clear the buffer pool, from SSMS: DBCC DROPCLEANBUFFERS
Verify that the cache has been cleared using the glorified dm_os_buffer_descriptors query at the top of this page: I get this:
Run the package
Re-run the query from step (2), and note that the data pages for the table (BOM_PIECE in my example) have been loaded into the cache:
Note that most SSIS components allow you to provide your own query, so if you have a way to avoid the buffer pool (I don't know that this is possible - I'd defer to someone who knows more about it), you could insert that into a query. So in the above example, instead of selecting Table or view in the OLE DB Source, you would select SQL command, or SQL command from variable if your command requires dynamic text.
Finally, I can imagine why you want to eliminate the cache load - but are you sure you want to do this? SQL Server is fairly good at managing memory, and what you're doing is swapping memory load for disk I/O load, which (depending on your use case) may have a negative impact on other users. This question has a discussion on SQL Server caching.
Read this article about Attunity regarding reading data from oracle
What do you mean "affect the database cache when loading data from a SQL Server database". SQL Server does not cache data, it caches execution plans. The fact that you are using SSIS wont affect your Server (other than the overhead of reading the data of course). Just use a propper transaction isolation level.
Also, read about the fast load property on SSIS components
About change data capture, I don't see how it can replace SSIS. You can use CDC to select the rows that will be loaded, but it wont do the loading for you.
I have a large MySQL table, with proper indices etc. I run a simple select * query from a remote machine and I expect a large result set.
My problem is that when I run the query, the result set returns at a maximum data transfer speed of ~300 KBytes/sec.
I created the same table, and run the same query on SQLServer Express 2008 R2, and the results returned at a transfer speed of 2MBytes/second (my line limit).
The server machine is Windows Server 2008 R2 x64, Quad core, 4GB RAM and the the MySQL version is 5.6.2 m5 64-bit. I tried disabling the compession in the communication protocol but the results where the same.
Does anyone have an idea as to why this is happening ?
--theodore
You might be comparing apples to oranges.
I'd run SELECT * on the MySQL server, and see what kind of data rate you get for retrieving data on the server locally -- without the additional constraint of a network.
If that's slow also -- then it isn't the network's fault.
When the MySQL setup program runs, it asks the person setting up MySQL what role MySQL is going to play on the hardware -- i.e., Development Server, Shared Server, Dedicated.
The difference in all of these is how much memory MySQL will seek to consume on the Server.
The slowest setting is Development (use the least memory), and the fastest one is Dedicated (attempt to use a lot of memory). You can tinker with the my.ini file to change how much memory MySQL will allocate for itself, and/or google 'my.ini memory' for more detailed instructions.
The memory that MySQL is using (or isn't, as the case may be), will make a huge difference on performance.
First, check to see what the speed is retrieving data locally on the MySQL server is. If it's slow, the network isn't the problem -- check MySQL's memory usage -- ideally give it as much as possible. And of course, if it's fast, then either the network and/or some piece of database middleware (ODBC?) or tool-used-to-display-the-data -- is slow...
One more thing -- try the SELECT * TWICE... why? The second time some or all of the results (again, depending on memory) should be cached... the second time it should be faster...
Also, don't forget to restart MySQL when changing the my.ini file (and create a backup before you make any changes...)
We have an application which gets the data from a MS SQL database residing on a server in the headquarter over an internet connection.
Now the problem is that the data is pretty large which makes the query to take large time to transfer the data from the head quarter to the client.
Does SQL provide anything method to speed up queries if your application is geographically separated from the database (e.g. zipping up the data before it starts to transfer) ?
Following quotes from a similar question might provide you with some pointers.
There is never a reason to return a
large result to start with. For those
particular scenarios that need large
amounts of data shipped to the client,
there are better solutions than T-SQL
queries (log shipping, service broker,
replication).
and
You have a couple options that I am
aware of:
Use a third party tool like SQLNitro.
Move to Windows Server 2008 and SQL Server 2008 where they have made
various TCP stack improvements, as
outlined here: Appendix A: A Closer
Look - Examining the Enhancements in
Windows Server 2008 and SQL Server
2008 That Enable Faster Performance
Performance is not only determined by the bandwidth, the latency is also a very strong factor here. Try to minimize the number of requests. You could combine some queries, to give two (or more) resultsets.