I am baffled by this problem. I have an Access 2013 database where most of the key tables are on SharePoint 2013 (not ideal but necessary). To allow the database to run faster, when the database opens, VBA code calls queries to download key SharePoint files to local tables. The main catalog table has 7300 records. This table takes over 30 seconds to populate the local table (by stepping through the code). If I run the query directly in the database, it takes 5 seconds at most. What is going on that it takes so long with VBA??
Related
We have a need to do the initial data copy on a table that has 4+ billion records to target SQL Server (2014) from source MySQL (5.5). The table in question is pretty wide with 55 columns, however none of them are LOB. I'm looking for options for copying this data in the most efficient way possible.
We've tried loading via Attunity Replicate (which has worked wonderfully for tables not this large) but if the initial data copy with Attunity Replicate fails then it starts over from scratch ... losing whatever time was spent copying the data. With patching and the possibility of this table taking 3+ months to load Attunity wasn't the solution.
We've also tried smaller batch loads with a linked server. This is working but doesn't seem efficient at all.
Once the data is copied we will be using Attunity Replicate to handle CDC.
For something like this I think SSIS would be the most simple. It's designed for large inserts as big as 1TB. In fact, I'd recommend this MSDN article We loaded 1TB in 30 Minutes and so can you.
Doing simple things like dropping indexes and performing other optimizations like partitioning would make your load faster. While 30 minutes isn't a feasible time to shoot for, it would be a very straightforward task to have an SSIS package run outside of business hours.
My business doesn't have a load on the scale you do, but we do refresh our databases of more than 100M nightly which doesn't take more than 45 minutes, even with it being poorly optimized.
One of the most efficient way to load huge data is to read them by chunks.
I have answered many similar question for SQLite, Oracle, Db2 and MySQL. You can refer to one of them for to get more information on how to do that using SSIS:
Reading Huge volume of data from Sqlite to SQL Server fails at pre-execute (SQLite)
SSIS failing to save packages and reboots Visual Studio (Oracle)
Optimizing SSIS package for millions of rows with Order by / sort in SQL command and Merge Join (MySQL)
Getting top n to n rows from db2 (DB2)
On the other hand there are many other suggestions such as drop indexes in destination table and recreate them after insert, Create needed indexes on source table, use fast-load option to insert data ...
My requirement is to load data on daily basis from source table from one server to destination table in another server. Both servers are different i.e one is sql server and another one is Oracle server.
what i want is to make the source query fast. Suppose today I Execute the package I should get only the new records instead of all records from source. If I read the whole table it takes much time. Even I am using Lookup transforming to check the record exists or not it is taking much time.
Please look into this
I have a SQL view, which takes 4-5 seconds to run, with no filters, if I run it within SSMS. If I try to open the linked "table" in Access 2010, it times out.
In the Options - Client Side Settings, I set the OLE/DDE timeout to 0, and the ODBC timeout to 0. I still get the ODBC--call failed. [Microsoft][ODBC SQL Server Driver]Query timeout expired (#0). Once I click ok, I get another message [current application name] can't open the table in Datasheet view.
I just don't understand how I can't open this linked table in Access, but the underlying view only has 88 records right now. There are a lot of columns, but few results, and it only takes a few seconds to run in SSMS. Why does it timeout and have such a problem as a linked table in Access?
Any help is greatly appreciated.
Thanks!
So I was looking at this puzzle, with a colleague. It would be difficult and still poor performance, to translate this 118 line query, with 30 table joins, into an Access query.
Instead, I am breaking the sections of the giant view, into separate smaller views. Each independent view, will be joined in an access query, so that each section of the query can be filtered independently, and allow for smaller sets of results, thereby improving the overall performance.
I have been trying to run a report for my CEO that shows income. Our agency management software uses FoxPro databases (it originally came out in the early '80s, I think). I have linked the .dbf files to an Access database, and I have been setting up queries based on queries to get the information I need on a live basis without having to export the data. The problem that I have run into is that I cleaned up the selection criteria in the first query, but I did not run that query (it takes about ten minutes to run each of these). When I ran the last query (with data based on the first), I still had bad data in that result.
So here's the dumb question: (a) do I need to create a macro that runs the queries (there are four of them) in sequence so that they are all updated each time, (b) is there some better way to do this, and/or (c) does Access automatically run the prior queries when I run the downstream query?
Every month, I do some analysis on a customer database. My predecessor would create a segment in Eloqua (Our CRM) for each country, and then spend about 10 (tedious, slow) hours refreshing them all. When I took over, I knew that I wouldn't be able to do it in Excel (we had over 10 million customers) so I used Access.
This process has worked pretty well. We're now up to 12 million records, and it's still going strong. However, when importing the master list of customers prior to doing any work on it, the database is inflating. This month it hit 1.3 GB.
Now, I'm not importing ALL of my columns - only 3. And Access freezes if I try to do my manipulations on a linked table. What can I do to reduce the size of my database during import? My source files are linked CSVs with only the bare minimum of columns; after I import the data, my next steps have to be:
Manipulate the data to get counts instead of individual lines
Store the manipulated data (only a few hundred KB)
Empty my imported table
Compress and Repair
This wouldn't be a problem, but i have to do all of this 8 times (8 segments, each showing a different portion of the database), and the 2GB limit is looming over the next horizon.
An alternate question might be: How can I simulate / re-create the "Linked Table" functionality in MySQL/MariaDB/something else free?
For such big number of records MS Access with 2 GB limit is not good solution as data storage. I would use MySQL as backend:
Create table in MySQL and link it to MS Access
Import CSV data directly to MySQL table using native import features of MySQL. Of course Access can be used for data import, but it will work slower.
Use Access for data analyse using this linked MySQL table as regular table.
You could import the CSV to a (new/empty) separate Access database file.
Then, in your current application, link the table from that file. Access will not freeze during your operations as it will when linking text files directly.