Create a cover index with "Include" columns using nhibernate mapping file - sql-server-2008

I need to create a non-clustered index with INCLUDE columns (see the <create> tag below). Here's the mapping file:
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2" namespace="MyApp" assembly="MyApp">
<class name="User" table="user" >
<id name="Id" type="Guid" column="user_id">
<generator class="guid.comb"/>
</id>
<property name="Name" column="name" not-null="true" />
<property name="Phone" column="phone" />
<property name="Zipcode" column="zipcode" />
</class>
<database-object>
<create>
CREATE NONCLUSTERED INDEX [IX_user_zipcode_id]
ON User (Zipcode)
INCLUDE (Name, Phone)
</create>
<drop>
DROP INDEX IX_user_zipcode_id
</drop>
<dialect-scope name="NHibernate.Dialect.MsSql2000Dialect"/>
<dialect-scope name="NHibernate.Dialect.MsSql2005Dialect"/>
<dialect-scope name="NHibernate.Dialect.MsSql2008Dialect"/>
</database-object>
</hibernate-mapping>
The problem I'm having is the index is not created at all. Nothing appears to be happening. This is my first time using <database-object> so I may be doing something wrong here.
I'm guessing INCLUDE is Sql Server specific which is why the dialect-scope is there. I know how to create a single and multi-column index, but this is not what I want. I want a single column index on zipcode and all other columns in the User table part of the INCLUDE clause of the query. Is there any way to create this type of index using the mapping file or some other way?
This is probably a long shot, but it would be nice to not have to specify every column but the indexed one in the INCLUDE part of the query... Instead to just let nhibernate add any new columns to the index that are added as properties to the mapping file.

So part of the problem was indeed my lack of understanding the database-object tag due mostly to poor documentation. From what I've gathered, the <create> and <drop> tags are only used when using SchemaExport like so:
Dim schemaExport As SchemaExport = New SchemaExport(NhibernateConfiguration)
schemaExport.Execute(False, True, False)
My app doesn't create the schema using that class. Instead it uses SchemaUpdate so the schema isn't blown away every time (the database may already exist on the users machine):
Dim schemaUpdate As SchemaUpdate = New SchemaUpdate(NhibernateConfiguration)
schemaUpdate.Execute(False, True)
That was the problem. The next logical question to ask is then how do you execute sql using SchemaUpdate. The answer... you can't. See this post: https://forum.hibernate.org/viewtopic.php?f=6&t=969584&view=next
Alas I am left to use raw sql. Maybe some day they will add an <update> tag.

Related

Liquibase modifyDataType fails on Mysql

I don't seem to be able to execute a type modification on a Mysql 5.7.10 instance (it works with H2 though).
Here are the changeset steps that are involved with the regarded field:
Creation:
<column name="last_modify_time" type="bigint">
<constraints nullable="false" />
</column>
Modification:
<modifyDataType tableName="USER" columnName="last_modify_time" newDataType="timestamp" />
the error msg in Mysql is
Invalid default value for 'last_modify_time' [Failed SQL: ALTER TABLE USER MODIFY last_modify_time timestamp]
Manually modifying the the request to the following works:
ALTER TABLE USER MODIFY last_modify_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
I don't really understand why Mysql needs the default value. Maybe it's an edge case with the version 5.7.10 (using the default configuration options).
In any case Liquibase should be able to handle it.
I've tried to add/remove default value prior to the modifyDataType, without success.
when you are modifying type of a column using liquibase which is non nullable ... you have to first make it nullable and then modify it use safe modify
It turns out that an exactly identical modification on a field named create_time, placed right before, didn't cause any problem. Swapping the two changsets order did solve the problem.
<changeSet id="11" author="author">
<comment>rename last_modify_time to last_modified_date and change type to timestamp</comment>
<modifyDataType tableName="USER" columnName="last_modify_time" newDataType="timestamp" />
<renameColumn tableName="USER" oldColumnName="last_modify_time" newColumnName="last_modified_date" columnDataType="timestamp" />
</changeSet>
<changeSet id="12" author="author">
<comment>rename create_time to created_date and change type to timestamp</comment>
<modifyDataType tableName="USER" columnName="create_time" newDataType="timestamp" />
<renameColumn tableName="USER" oldColumnName="create_time" newColumnName="created_date" columnDataType="timestamp"/>
</changeSet>
I still can't explain what happend, I'm happy enough to get it working. If someone wants to reproduce the error I'd be happy to help.

How to properly set up DataImportHandler for MySQL database with large number or records?

I have set up Solr's data import handler as instructed in manual. Solr reads the records from a MySQL database. The database has large number of records (expected is milliards/billions).
I have read that batch size does not work for MySQL because the JDBC driver does not support it. I have tried setting it up to -1. In this case, Solr performs one select and gets all records from the DB and indexes them.
Now, I have problem, since a timeout occurred while indexing and caused it to stop. I see that Solr hasn't written any id value in the properties file after the exception occurred. I am not sure how to proceed with indexing the rest of the records.
Can anyone suggest to me how to set up Solr with MySQL for a proper data import?
Below is data config I am currently using.
<dataConfig>
<dataSource type="JdbcDataSource" name="ds-2" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/myowndb" batchSize="-1" />
<document name="statuses">
<entity name="status" query="select s.*, ti.id2, ti.value2 from tblTable1 s inner join tblTable2 ti on s.table2Id = ti.id;">
<field column="id" name="id" />
<field column="statusID" name="statusId" />
<field column="type" name="type" />
<field column="date" name="date" />
<field name="id2" column="id2" />
<field name="value2" column="value2" />
</entity>
</document>
</dataConfig>
EDIT:
Based on my tests today, it looks like batchSize is working. If batchSize is set to -1, it will make single request to MySQL retrieving all rows at once. If set to some value greater than 0, it will put every record in memory before processing.
New question is next: how to set up data import handler so it can index in batches? Not only to perform batch select from database, but to index collected set before collecting next one.
EDIT: Specified question
New question that came up from reading is next: is it possible to mark row in database as processed? There are only two events available in DIH, onImportStart and onImportEnd.
Current flow in ideas lead me to implement EntityProcessor. If it would be possible to know when some row is indexed, it would also be easy to mark isIndexed flag in database for indexed row. This is in case I implement custom EntityProcessor.

Liquibase script for MYSQL

I have a liquibase xml script. When I run it on Postgres I don't face any problem but when I run it for MYSQL it gives error when the structure is of the following type:-
<insert tableName="user_table">
<column name="id" valueComputed="(select max(id)+1 from user_table)"/>
<column name="name" value="someName"/>
</insert>
When the above script is executed for MYSQL it gives error:-
You can't specify target table 'user_table' for update in FROM clause.
I found a solution to this by using alias like this :-
<insert tableName="user_table">
<column name="id" valueComputed="(select max(id)+1 from (Select * from user_table) t)" />
<column name="name" value="someName"/>
</insert>
But there are thousands of entries like this. Is there any generic way of doing it so that I don't have to change the script at so many places. Thanks.
The easiest approach would be to just update the XML, either with an simple XML parser program or even a regexp search and replace in your text editor.
Alternately, you can override the standard liquibase logic to look for that particular valueComputed pattern and replace it. There are a couple points you could make the change at:
Override the liquibase.parser.core.xml.XMLCHangeLogSAXParser class, probably the parseToNode() method to search through the generated ParsedNode for valueComputed nodes
Override the liquibase.change.core.InsertDataChange class generateStatements() method or addColumn() method to replace valueComputed fields.
See http://liquibase.org/extensions for more info on writing extensions.

dbdeploy error with --// in deploy-file

I am generating deployment-files for my mysql-database with phing and dbdeploy.
The output of a dbdeploy-file that is generated looks like the following:
-- Fragment begins: 8 --
INSERT INTO changelog
(change_number, delta_set, start_dt, applied_by, description) VALUES (8, 'Main', NOW(), 'dbdeploy', '8-add_tracking_code.sql');
--//
ALTER TABLE `order` ADD `tracking_code` VARCHAR(255) NOT NULL;
UPDATE changelog
SET complete_dt = NOW()
WHERE change_number = 8
AND delta_set = 'Main';
-- Fragment ends: 8 --
The Problem is the --// before the ALTER Statement. The database got an error with it. If I remove the --// the hole file ist correct.
Here is a piece of my phing build-script so that you can see how I am generating the .sql-File with dbdeploy:
<target name="dbdeploy-migrate-all">
<!-- load the dbdeploy task -->
<taskdef name="dbdeploy" classname="phing.tasks.ext.dbdeploy.DbDeployTask"/>
<echo message="Loading deltas from ${build.dbdeploy.alters_dir}" />
<property name="build.dbdeploy.deployfile" value="${build.dbdeploy.deploy_dir}/deploy-${DSTAMP}${TSTAMP}.sql" />
<property name="build.dbdeploy.undofile" value="${build.dbdeploy.undo_dir}/undo-${DSTAMP}${TSTAMP}.sql" />
<!-- generate the deployment scripts -->
<dbdeploy
url="mysql:host=${db.host};dbname=${db.name}"
userid="${db.user}"
password="${db.pass}"
dir="${build.dbdeploy.alters_dir}"
outputfile="${build.dbdeploy.deployfile}"
undooutputfile="${build.dbdeploy.undofile}" />
<!-- execute the SQL - Use mysql command line to avoid trouble with large files or many statements and PDO -->
<property name="mysql.command" value="${progs.mysql} -h${db.host} -u${db.user} -p${db.pass} ${db.name} < ${build.dbdeploy.deployfile}" />
<echo message="Executing command: ${mysql.command}" />
<exec
command="${mysql.command}"
dir="${base.path}"
checkreturn="true" />
</target>
Why does dbdeploy generate a corrupt file?
Thans for your help!
A long time has passed since this question was asked, however I ran into the same problem and have managed to work out where Niels is coming from on this one.
I think we both ran into the problem because we both followed the popular tutorial on phing and dbdeploy by Dave Marshall here: http://davedevelopment.co.uk/2008/04/14/how-to-simple-database-migrations-with-phing-and-dbdeploy.html
In his example sql delat file he includes --// at the top which, if replaced with a comment in /* .... */ format instead, avoids this problem!
So I would say this is a bug in the tutorial, which is 8 years old now. Dave notes at the top of the tutorial that about 4 years ago he moved onto a different method, so it is kind of understandable that there is now a bug in the tutorial! I will submit a comment requesting an update though, because his page is a top ranking search result when searching for the topic so it would be good if we can save people the same problem we've had!
It is a shame phing didn't give a more detailed error report in the form of the SQL exception - there's an idea for a contribution to the dbdeploy script!

How can I set the Nhibernate type generator to increment when working with SQL Server 2008?

Using HBM files to map my types.
One of my classes uses bag of items called PartnerEnv. One of their fields is set to be the id which should be generated using increment. for some reason I am getting the following error:
could not fetch initial value for increment generator[SQL: SQL not available]
Inner details: "{"Invalid object name 'jj.dbo.Partners2Env'."}"
If I change the generation method to assigned everything is ok.
I will appreciate any help given!
Can you set your Id column on the PartnerEnv table (or whatever that table is called) to an Identity column and then use the following in the .hbm file for that class?
<id name="Id" type="Int32">
<column name="Id" />
<generator class="identity" />
</id>