Improving an insert method - linq-to-sql

I'm wrote an insert method that uses linq and loops through 2 lists, the first being able to go up to 14k objects and the send about 8k objects.
Whenever I run this method, I always get "Transaction Timeout Exception". Can you help me improve this?
public void InsertNewInventoryGoodsEvents(List<GoodsEvent> eventsList, List<InventoryGoods> goodsList)
{
InventoryGoodsEvents ige = new InventoryGoodsEvents();
TransactionScope scope= new TransactionScope();
int i = 0;
const int batchSize = 50; // or even 50
foreach (InventoryGoods good in goodsList)
{
if (i == 50)
{
if (scope != null)
{
context.SubmitChanges();
}
i = 0;
}
try
{
foreach (GoodsEvent g in eventsList)
{
if (g.Gid == good.Gid)
{
ige = new InventoryGoodsEvents() { EventId = g.Id, InventoryGood = good.Id };
context.InventoryGoodsEvents.InsertOnSubmit(ige);
}
}
}
catch (Exception ex)
{
ex.ToString();
}
++i;
}
if (scope != null)
{
context.SubmitChanges();
scope.Complete();
}
}

the Foreach in a foreach can can be re-written as this-
var itsReallyJustAJoin = (from g in GoodEvent
join i in InventoryEvent on g.Gid equals i.Gid
select new {g.Id, good.Id}).ToList();
Then you can insert these however you'd like. It looks like you're doing batches of 50 right now? You can continue that if you'd like... This will take all of the work outside of the transaction scope though, so you can probably lob the whole bit in there at once, if your results aren't too huge.

Is it important that all of the inserts be done in the same transaction? If not, could you just use a new transaction for each batch of items you're inserting? That should prevent the transaction from being open for too long and timing out.

I have re-written the method after giving it a good thought and it turned out like this:
public void InsertNewInventoryGoodsEvents(List<GoodsEvent> eventsList, List<InventoryGoods> goodsList)
{
List<InventoryGoodsEvents> mylist = new List<InventoryGoodsEvents>();
InventoryGoodsEvents ige = new InventoryGoodsEvents();
foreach (GoodsEvent g in eventsList)
{
foreach (InventoryGoods ig in goodsList)
{
if (ig.Gid == g.Gid)
{
ige = new InventoryGoodsEvents();
ige.EventId = g.Id;
ige.InventoryGood = ig.Id;
mylist.Add(ige);
}
}
}
using (var scope = new TransactionScope())
{
foreach (InventoryGoodsEvents ip in mylist)
{
context.InventoryGoodsEvents.InsertOnSubmit(ip);
}
context.SubmitChanges();
scope.Complete();
}
}
I liked MikeM's idea of simplifying things but we'll see which method I'll keep.

Related

Connect Processing with SQL database and Get name of columns

I have connect Processing and SQL by using database library "de.Bezier.data.sql".
I don't know How can I get the name of columns in a specific Table.
I get the correct name of database, but i got the following as result of name of columns "Tables_in_sql7363100"
import de.bezier.data.sql.*;
MySQL sql;
String[] tableNames;
String[] columnNames;
void setup() {
size(300, 300);
database_connection();
if (connect) {
tableNames = sql.getTableNames();
for (int i=0; i<tableNames.length; i++) {
println(tableNames[i]);
}
columnNames = sql.getColumnNames();
for (int i=0; i<ColumnNames.length; i++) {
println(columnNames[i]);
}
}
}
void draw() {
background(255);
}
void database_connection() {
sql = new MySQL(this, "ServerName", "DataBase", "DUN", "PW");
if (sql.connect()) {
connect = true;
connect_status = "Conected";
} else {
connect = false;
connect_status = "Connection Failed";
}
}
There are 2 problems with what I'm seeing. The first one, which interests you, is that you didn't select any table. That's why you don't get a list of columns. You can fix this by using a simple query:
sql.query("SELECT * FROM myTable");
But that's not the only thing: you're not accounting for lag. It may work for now on a local database because lag is really low, but this won't fly with something which is over the internet. Here's an example where I show columns from a public test database and how long it takes to get the result from my query back:
import de.bezier.data.sql.*;
MySQL sql;
String user = "rfamro";
String pass = "";
String server = "mysql-rfam-public.ebi.ac.uk:4497";
String database = "Rfam";
String[] columnNames;
void setup() {
size(300, 300);
sql = new MySQL(this, server, database, user, pass);
}
void draw() {
if (columnNames != null) {
println("It took " + millis() + "ms to get this data back.");
for (String s : columnNames) {
println(s);
}
noLoop();
} else if (sql.connect()) {
sql.query("SELECT * FROM family");
sql.next(); // only use .next when you know you have data
columnNames = sql.getColumnNames();
}
}
From here, it takes between 2-7 seconds to get the data back. You'll understand that, the setup() method running taking about a couple milliseconds, you won't have any results back by then.
Hope this helps. Have fun!

How the performance of a JavaFx-MySQL application can be enhanced

In my JavaFx application, i'm loading an ObservableList when a button is clicked and then display the list in a table.
the controller code:
#FXML
private void initialize() throws SQLException, ParseException, ClassNotFoundException {
searchChoice.setItems(criteriaList);
searchChoice.getSelectionModel().selectFirst();
productIdColumn.setCellValueFactory(cellData -> cellData.getValue().productIdProperty());
unitColumn.setCellValueFactory(cellData -> cellData.getValue().unitProperty());
productTitleColumn.setCellValueFactory(cellData -> cellData.getValue().titleProperty());
productTypeColumn.setCellValueFactory(cellData -> cellData.getValue().typeProperty());
productUnitPriceColumn.setCellValueFactory(cellData -> Bindings.format("%.2f", cellData.getValue().unitPriceProperty().asObject()));
productQuantityColumn.setCellValueFactory(cellData -> cellData.getValue().quantityProperty().asObject());
productStatusColumn.setCellValueFactory(cellData -> cellData.getValue().productStatusProperty());
descriptionColumn.setCellValueFactory(cellData -> cellData.getValue().descriptionProperty());
reorderPointColumn.setCellValueFactory(cellData -> cellData.getValue().reOrderPointProperty().asObject());
surplusPointColumn.setCellValueFactory(cellData -> cellData.getValue().surplusPointProperty().asObject());
productIdColumn.setSortType(TableColumn.SortType.DESCENDING);
productTable.getSortOrder().add(productIdColumn);
productTable.setRowFactory(tv -> new TableRow<Product>() {
#Override
public void updateItem(Product item, boolean empty) {
super.updateItem(item, empty);
if (item == null) {
setStyle("");
} else if (item.getQuantity() < item.getReOrderPoint()) {
setStyle("-fx-background-color: tomato;");
} else if (item.getQuantity() > item.getSurplusPoint()) {
setStyle("-fx-background-color: darkorange;");
} else {
setStyle("-fx-background-color: skyblue;");
}
}
});
try {
ObservableList<Product> productData = ProductDAO.searchProducts();
populateProducts(productData);
String[] expireDate = new String[productData.size()];
String[] id = new String[productData.size()];
String[] existingStatus = new String[productData.size()];
for (int i = 0; i < productData.size(); i++) {
expireDate[i] = productData.get(i).getExpireDate();
id[i] = productData.get(i).getProductId();
existingStatus[i] = productData.get(i).getProductStatus();
DateFormat format = new SimpleDateFormat(app.values.getProperty("DATE_FORMAT_PATTERN"), Locale.ENGLISH);
Date expireDateString = format.parse(expireDate[i]);
Date in = new Date();
LocalDateTime ldt = LocalDateTime.ofInstant(in.toInstant(), ZoneId.systemDefault());
Date today = Date.from(ldt.atZone(ZoneId.systemDefault()).toInstant());
if (expireDateString.before(today) && !existingStatus[i].equals(app.values.getProperty("STATUS_TYPE2"))) {
ProductDAO.updateProductStatus(id[i], app.values.getProperty("STATUS_TYPE3"));
}
if (expireDateString.after(today) && !existingStatus[i].equals(app.values.getProperty("STATUS_TYPE2"))) {
ProductDAO.updateProductStatus(id[i], app.values.getProperty("STATUS_TYPE1"));
}
}
ObservableList<Product> productDataRefreshed = ProductDAO.searchProducts();
populateProducts(productDataRefreshed);
ObservableList<Product> productCodesData = ProductDAO.getProductCodes();
ObservableList<Product> productTitlesData = ProductDAO.getProductTitles();
ObservableList<Product> productTypesData = ProductDAO.getProductTypes();
ObservableList<Product> productStatusData = ProductDAO.getProductStatus();
String possibleProducts1[] = new String[productCodesData.size()];
for (int k = 0; k < productCodesData.size(); k++) {
possibleProducts1[k] = productCodesData.get(k).getProductId();
}
String possibleProducts2[] = new String[productTitlesData.size()];
for (int k = 0; k < productTitlesData.size(); k++) {
possibleProducts2[k] = productTitlesData.get(k).getTitle();
}
String possibleProducts3[] = new String[productTypesData.size()];
for (int k = 0; k < productTypesData.size(); k++) {
possibleProducts3[k] = productTypesData.get(k).getType();
}
String possibleProducts4[] = new String[productStatusData.size()];
for (int k = 0; k < productStatusData.size(); k++) {
possibleProducts4[k] = productStatusData.get(k).getProductStatus();
}
TextFields.bindAutoCompletion(searchField, possibleProducts1);
TextFields.bindAutoCompletion(searchField, possibleProducts2);
TextFields.bindAutoCompletion(searchField, possibleProducts3);
TextFields.bindAutoCompletion(searchField, possibleProducts4);
} catch (SQLException e) {
Alert alert = new Alert(Alert.AlertType.ERROR);
alert.setTitle(app.values.getProperty("ERROR_TITLE"));
alert.setHeaderText(app.values.getProperty("FAILURE_MESSAGE"));
alert.setHeaderText(app.values.getProperty("ERROR_GETTING_INFORMATION_FROM_DATABASE_MESSAGE"));
alert.showAndWait();
throw e;
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
the service mysql query :
public static ObservableList<Product> searchProducts() throws SQLException, ClassNotFoundException {
String selectStmt = "SELECT * FROM product";
ResultSet rsPrdcts = DbUtil.dbExecuteQuery(selectStmt);
ObservableList<Product> productList = getProductList(rsPrdcts);
return productList;
}
The issue here is, when there are more than 200-300 items in the list the scene gets really slow to load. What countermeasures can I take regarding this matter? Any idea will be very much appreciated.
Thanks in advance.
You need to implement an ObservableList which only retrieves the data which is rqeusted by the TableView. Currently you retrive all elements in the table and cast the retrieved list to an ObservableList.
The TableView uses the .get(int idx) method of the OL to retrieve all items which should be displayed and the .size() method for determining the size of the scrollbar. When you scroll the TableView will discard all items which are not displayed and call the get method again.
To solve your problem need to create a class which implements ObservableList<E>. First you need to implement the .get(int idx) and the .size() method, for all other methods I would throw new UnsupportedOperationException() and later on see which other method is needed. So the .size() method needs to execute the following query
SELECT count(*) FROM product
and the get(int idx) something like this
int numItems = 30;
int offset = idx - (idx % numItems)
SELECT * FROM product LIMIT offset, numItems
you can create an internal list which only holds e.g. 30 items from your db and whenever the requested idx < offset || idx > offset + numItems you issue a new db request.
I used this this approach with database tables with millions of rows and had a very performant GUI. You can also add paging to the TableView because with to many rows the scrollbar gets useless, but this is a different discussion.
edit:
I forgot to mention that this is called Lazy Loading

Loading millions of DB row in wicket table through a list causing outofmemory error

I'm loading million lines of data by sortabledataprovider
.. the query returns a list(Arraylist) as I sent it to a Wicket ajax enabled table and pagination enable table.
So the problem is that - If there are concurrent queries - the application might get crashed.
I'm already getting Java heap space error with just 100,000 rows in DB.
So what I want to achieve is this - when a user click on next page or may be number 10th page - it will load only the number 10th pages data from the DB - not the whole query which might have million lines in it.
here is how I loading the list by query
final SyslogProvider<SyslogParsed> sysLogProviderAjax = new SyslogProvider<SyslogParsed>(new ArrayList<SyslogParsed>());
*
*
daoList = syslogParsedDao.findByDatesAndHostName(utcStartDate, utcEndDate, null);
sysLogProviderAjax.setList(daoList);
**
DB query returning the big list of all rows
public List<logParsed> findByDatesAndHostName() {
return getJpaTemplate().execute(new JpaCallback<List<SyslogParsed>>() {
return query.getResultList();
}
});
}
=========
my data provider
public class logProvider<logParsed> extends SortableDataProvider{
/**
*
*/
private static final long serialVersionUID = 1L;
#SpringBean(name="logParsedDao")
private logParsedDao logParsedDao;
class SortableDataProviderComparator implements Comparator<logParsed>, Serializable {
public int compare(logParsed log1, logParsed log2) {
PropertyModel<Comparable> model1 = new PropertyModel<Comparable>(log1, getSort().getProperty());
PropertyModel<Comparable> model2 = new PropertyModel<Comparable>(log1, getSort().getProperty());
int result = model1.getObject().compareTo(model2.getObject());
if (!getSort().isAscending()) {
result = -result;
}
return result;
}
}
private List<logParsed> list = new ArrayList<logParsed>();
private SortableDataProviderComparator comparator = new SortableDataProviderComparator();
public logProvider(List<logParsed> sentModel){
setSort("numberOfEntries",SortOrder.DESCENDING);
list = sentModel;
}
public Iterator<logParsed> iterator(int first, int count) {
//ArrayList<logParsed> newList = (ArrayList<logParsed>) logParsedDao.findAll();
//Collections.sort(newList, comparator);
Iterator<logParsed> iterator = null;
try {
if(getSort() != null) {
Collections.sort(list, new Comparator<logParsed>() {
private static final long serialVersionUID = 1L;
public int compare(logParsed sl1, logParsed sl2) {
int result=1;
PropertyModel<Comparable> model1= new PropertyModel<Comparable>(sl1, getSort().getProperty());
PropertyModel<Comparable> model2= new PropertyModel<Comparable>(sl2, getSort().getProperty());
if(model1.getObject() == null && model2.getObject() == null)
result = 0;
else if(model1.getObject() == null)
result = 1;
else if(model2.getObject() == null)
result = -1;
else
result = model1.getObject().compareTo(model2.getObject());
result = getSort().isAscending() ? result : -result;
return result;
}
});
}
if (list.size() > (first+count))
iterator = list.subList(first, first+count).iterator();
else
iterator = list.iterator();
}
catch (Exception e) {
e.printStackTrace();
}
return iterator;
}
public int size() {
// TODO Auto-generated method stub
return list.size();
}
public IModel<logParsed> model(final Object object) {
return new AbstractReadOnlyModel<logParsed>() {
#Override
public logParsed getObject() {
return (logParsed) object;
}
};
}
public void setList(List<logParsed> newList){
list = newList;
}
}
The problem with query.list() is it returns all rows at once.
Instead of query.list() you can use either:
query.scroll(), which returns the query results as ScrollableResults
query.iterate(), which returns the query results as an Iterator
Both of these options return one row at a time, which is what you need.
Note that the query remains "executing" for the duration of processing, so you may find that the tables are locked etc depending on the isolation level you've chosen.
You have to use a JPA query that sort and returns only the desired rows each time that iterator(int first, int count) of your IDataProvider is called.
Something similar to this:
public Iterator<logParsed> iterator(int first, int count) {
Query query = entityManager.createQuery("from LogParsed m ORDER BY m.numberOfEntries DESC", LogParsed.class);
List<LogParsed> output = query.setFirstResult(first).setMaxResults(count).getResultList();
return output.iterator();
}
Have a look at this question.

Connection pool connections consumed when in TransactionScope

Can someone explain why we are experiencing a total consumption of the connection pool when performing queries inside TransactionScope resulting in
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
I have reduced our problem to the following:
SomeNonTransactionalCode()
{
// This code will execute with just one connection to the database
// and complete without exception
List<BusinessEntity> beList;
for (int i = 0; i < 101; i++)
{
BusinessEntityRepository beRepo = new BusinessEntityRepository();
beList = beRepo.ReadAll().ToList();
}
}
SomeTransactionalCode()
{
// This code will cause the connections to the database to increment
// with every iteration eventually timing out after 100 connections
using (TransactionScope transactionScope = new TransactionScope())
{
List<BusinessEntity> beList;
for (int i = 0; i < 101; i++)
{
BusinessEntityRepository beRepo = new BusinessEntityRepository();
beList = beRepo.ReadAll().ToList();
}
transactionScope.Complete();
}
}
EDIT
After Omer's answer below I think the problem is explained better like this:
SomeNonTransactionalCode()
{
// This code will execute with just one connection to the database
List<BusinessEntity1> be1List;
BusinessEntity1Repository be1Repo = new BusinessEntity1Repository();
be1List = be1Repo .ReadAll().ToList();
List<BusinessEntity2> be2List;
BusinessEntity2Repository be2Repo = new BusinessEntity2Repository();
be2List = be2Repo .ReadAll().ToList();
List<BusinessEntity3> be3List;
BusinessEntity3Repository be3Repo = new BusinessEntity3Repository();
be3List = be3Repo.ReadAll().ToList();
}
SomeTransactionalCode()
{
// This code will cause three seperate connections to the database
using (TransactionScope transactionScope = new TransactionScope())
{
// note this is simplified - the code below could be in unknown nested
// methods make creating of the repos prior to calling not possible
List<BusinessEntity1> be1List;
BusinessEntity1Repository beRepo1 = new BusinessEntity1Repository();
be1List = be1Repo.ReadAll().ToList();
List<BusinessEntity2> be2List;
BusinessEntity2Repository beRepo2 = new BusinessEntity2Repository();
be2List = be2Repo.ReadAll().ToList();
List<BusinessEntity3> be3List;
BusinessEntity3Repository beRepo3 = new BusinessEntity3Repository();
be3List = be3Repo.ReadAll().ToList();
transactionScope.Complete();
}
}
Surely this is not expected behaviour? I have read nothing that explains why this might be happening. I can only assume it is something to do with how we have implemented our repositories. Hopefully the following will give a good enough description of the implementation.
public class BusinessEntityRepository
{
private BusinessEntityDal Dal { get; set; }
public BusinessEntityRepository()
{
this.Dal = new BusinessEntityDal ();
}
public IQueryable<BusinessEntity> ReadAll()
{
IQueryable<BusinessEntity> query = null;
if (Dal != null)
{
query = Dal.ReadAll();
}
//-Return
return query;
}
}
public class BusinessEntityDal : BaseDal
{
public IQueryable<BusinessEntity> ReadAll()
{
var result = from de in this.DC.BusinessEntityTable
select new BusinessEntity
{
Property1 = Column1,
Property2 = Column2,
// etc...
};
//-Return
return (result);
}
}
public abstract class BaseDal
{
protected OurDataContext DC;
public BaseDal()
{
// create a DataContext
this.DC = new OurDataContext();
}
}
public class OurDataContext : System.Data.Linq.DataContext
{
private static readonly string _cn = // some static connection string taken from web.config
public OurDataContext()
: base(OurDataContext._cn)
{
}
}
Our connection string is fairly conventional and leaves the number of connections in the pool at the default 100 (hence the 101 iterations to test the issue in my code above).
You are creating new DataContext references inside the for loop.
for (int i = 0; i < 101; i++)
{
BusinessEntityRepository beRepo = new BusinessEntityRepository();
beList = beRepo.ReadAll().ToList();
}
It is keeping all those in different transactions. If you just put repo init code outside the for loop and perform all operations in one context, it will be fine.
using (TransactionScope transactionScope = new TransactionScope())
{
List<BusinessEntity> beList;
BusinessEntityRepository beRepo = new BusinessEntityRepository();
for (int i = 0; i < 101; i++)
{
beList = beRepo.ReadAll().ToList();
}
//do some other things with same context
transactionScope.Complete();
}

How to modify linqtosql entityref objects in handcoded MVC model?

I am trying to set up my own mvc model rather than letting the environment create one via the graphic designer tool. I had hoped that this would make it easier to keep separate repositories for parts of the model space but so far it has caused me nothing but grief.
The first problem I ran into was that the entityref classes had to be updated via a selectlist control in the view. I managed to get that to work by adding an interal ID field to every entityref much like designer.cs would do. However, it has made the model class quite a bit more complex as the code below demonstrates.
Unfortunately, I now run into a problem when I want to explicitly update some of the entities in the controller. If I manually set the ID field, the update is just dropped, if I change the entity I get an exception while saving.
My model
[Table(Name = "dbo.Jobs")]
public class Job {
[Column(IsPrimaryKey = true, IsDbGenerated = true, AutoSync = AutoSync.OnInsert)]
public int JobID { get; set; }
internal string _CompanyID; // string for legacy reasons
[Column(Storage = "_CompanyID")]
public string CompanyID{
get { return _CompanyID}
set {
if ((_CompanyID != value)) {
if (_MittlerOrg.HasLoadedOrAssignedValue) {
throw new System.Data.Linq.ForeignKeyReferenceAlreadyHasValueException();
}
_CompanyID = value;
}
}
}
internal EntityRef<Company> _Company;
[Association(Storage = "_Company", ThisKey = "CompanyID", OtherKey = "CompanyID", IsForeignKey = true)]
public Company Company {
get { return _Company.Entity; }
set {
Organization previousValue = this._Company.Entity;
if ((previousValue != value) || (_Company.HasLoadedOrAssignedValue == false)) {
if ((previousValue != null)) {
_Company.Entity = null;
}
_Company.Entity = value;
if (value != null) {
_CompanyID = value.OrganizationID;
} else {
_CompanyID = default(string);
}
}
}
}
// The contact depends on the choice of company and should be set
// inside an action method once the company is determined.
internal string _ContactID;
[Column(Storage = "_ContactID")]
public string ContactID {
get { return _ContactID; }
set {
if ((_ContactID != value)) {
if (_Contact.HasLoadedOrAssignedValue) {
throw new System.Data.Linq.ForeignKeyReferenceAlreadyHasValueException();
}
_ContactID = value;
}
}
}
internal EntityRef<User> _Contact;
[Association(Storage = "_Contact", ThisKey = "ContactID", OtherKey = "UserID", IsForeignKey = true)]
public User Contact {
get { return _Contact.Entity; }
set {
User previousValue = this._Contact.Entity;
if ((previousValue != value) || (_Contact.HasLoadedOrAssignedValue == false)) {
if ((previousValue != null)) {
_Contact.Entity = null;
}
_Contact.Entity = value;
if (value != null) {
_ContactID = value.UserID;
} else {
_ContactID = default(string);
}
}
}
}
}
The edit function that causes problems is here. If I step though it in the debugger I see that fi.ContactID gets updated but not committed to the DB.
[HttpPost]
public ActionResult Edit(int id, FormCollection collection) {
var user = userrep.FetchByLogin(User.Identity.Name);
var job = jobrep.FetchByID(id);
try {
var oldvalue = job.CompanyID;
UpdateModel(job, "job");
if (oldvalue != job.CompanyID) {
if (job.CompanyID != null) {
job.ContactID = orgrep.FetchByID(job.CompanyID).DefaultContactID;
} else {
job.ContactID = default(string);
}
}
firep.Save();
return RedirectToAction("Index");
} catch (Exception e) {
}
}
Any idea how to get those pesky entityrefs to behave? I searched up and down the internet but all model layer examples seem to cover the simplest relationships only. Should I just chuck the model completely in favor of managing my references manually though id fields.
Cheers,
Duffy
Update: I never got this piece of code to work robustly so I ended up switching back to letting visual studio generate the DataContext via the drag and drop graphical designer.
I still struggle a bit with fixing the names on the relationship links after an update of the db schema (I carefully name all relationships in the db but the designer tool seems to ignore those names) but since I discovered that the db.designer.cs file can be opened in an xml editor rather than with the GUI, the job got a lot easier.