I am doing a spring application and am kind stack. Iam running a query as shown below
#Autowired
EntityManagerFactory entityManagerFactory;
public List countTransactionsGroupByProvider(){
EntityManager em = entityManagerFactory.createEntityManager();
String query = "SELECT t.order_name,count(t.order_name) as number_of_transactions from transactions_view t where "
+ "t.transaction_date between '2014-07-24' and '2014-10-27' group by t.order_name";
List result = em.createNativeQuery(query).getResultList();
return result;
}
Now,This is working fine.it returns the data below:
[["Airtel",148], ["Expresso",8], ["Glo",49],
["MTN",110],["Select network",1],["Surfline",88],
["Tigo",35],["Vodafone",136],["Vouchers",30]]
My problem is I want this to return in the below format:
[{"order_name":"Airtel","number_of_transactions":148},
{"order_name":"Expresso","number_of_transactions":8},
{"order_name":"MTN","number_of_transactions":110},etc]
Then I can feed this into morris.js to plot a graph.
Any suggestion as to how to go about this.Thank much
You should probably just write some supporting code to transform the data into the format you want. Not sure you're going to get much traction trying to get JPA to produce the data in the format you want although it's arguably not out of the question.
Related
I am trying to move data from a SPARQL endpoint to a JSONObject. Using RDF4J.
RDF4J documentation does not address this directly (some info about using endpoints, less about converting to JSON, and nothing where these two cases meet up).
Sofar I have:
SPARQLRepository repo = new SPARQLRepository(<My Endpoint>);
Map<String, String> headers = new HashMap<String, String>();
headers.put("Accept", "SPARQL/JSON");
repo.setAdditionalHttpHeaders(headers);
try (RepositoryConnection conn = repo.getConnection())
{
String queryString = "SELECT * WHERE {GRAPH <urn:x-evn-master:mwadata> {?s ?p ?o}}";
GraphQuery query = conn.prepareGraphQuery(queryString);
debug("Mark 2");
try (GraphQueryResult result = query.evaluate())
this fails because "Server responded with an unsupported file format: application/sparql-results+json"
I figured a SPARQLGraphQuery should take the place of GraphQuery, but RepositoryConnection does not have a relevant prepare statement.
If I exchange
try (RepositoryConnection conn = repo.getConnection())
with
try (SPARQLConnection conn = (SPARQLConnection)repo.getConnection())
I run into the problem that SPARQLConnection does not generate a SPARQLGraphQuery. The closest I can get is:
SPARQLGraphQuery query = (SPARQLGraphQuery)conn.prepareQuery(QueryLanguage.SPARQL, queryString);
which gives a runtime error as these types cannot be cast to eachother.
I do not know how to proceed from here. Any help or advise much appreciated. Thank you
this fails because "Server responded with an unsupported file format: application/sparql-results+json"
In RDF4J, SPARQL SELECT queries are tuple queries, so named because each result is a set of bindings, which are tuples of the form (name, value). In contrast, CONSTRUCT (and DESCRIBE) queries are graph queries, so called because their result is a graph, that is, a collection of RDF statements.
Furthermore, setting additional headers for the response format as you have done here is not necessary (except in rare circumstances), the RDF4J client handles this for you automatically, based on the registered set of parsers.
So, in short, simplify your code as follows:
SPARQLRepository repo = new SPARQLRepository(<My Endpoint>);
try (RepositoryConnection conn = repo.getConnection()) {
String queryString = "SELECT * WHERE {GRAPH <urn:x-evn-master:mwadata> {?s ?p ?o}}";
TupleQuery query = conn.prepareTupleQuery(queryString);
debug("Mark 2");
try (TupleQueryResult result = query.evaluate()) {
...
}
}
If you want to write the result of the query in JSON format, you could use a TupleQueryResultHandler, for example the SPARQLResultsJSONWriter, as follows:
SPARQLRepository repo = new SPARQLRepository(<My Endpoint>);
try (RepositoryConnection conn = repo.getConnection()) {
String queryString = "SELECT * WHERE {GRAPH <urn:x-evn-master:mwadata> {?s ?p ?o}}";
TupleQuery query = conn.prepareTupleQuery(queryString);
query.evaluate(new SPARQLResultsJSONWriter(System.out));
}
This will write the result of the query (in this example to standard output) using the SPARQL Query Results JSON format. If you have a non-standard format in mind, you could of course also create your own TupleQueryResultHandler implementation.
For more details on the various ways in which you can process the result (including iterating, streaming, adding to a List, or just directly sending to a result handler), see the documentation on querying a repository. As an aside, the javadoc on the RDF4J APIs is pretty extensive too, so if your Java editing environment has support for displaying that, I'd advise you to make use of it.
I am trying to improve performance of querying a couchbase view by using async gets.
I have read their documentation about the proper way to do so, it goes something like:
Cluster cluster = CouchbaseCluster.create();
Bucket bucket = cluster.openBucket();
List<JsonDocument> foundDocs = Observable
.just("key1", "key2", "key3", "key4", "key5")
.flatMap(new Func1<String, Observable<JsonDocument>>() {
#Override
public Observable<JsonDocument> call(String id) {
return bucket.async().get(id);
}
})
.toList()
.toBlocking()
.single();
Which works great and fast, but since I rely on the order of the results, it seems that i need to do some extra work to keep the results ordered.
In the example above, the JsonDocument list contains all 5 documents but the order changes randomly from call to call.
Is there any ellegant way to order the result using JavaRx capabilities or couchbase Java SDK capabilities?
The only solution i can think of is saving the results in to a HashMap and then transform the original list of ids using this HashMap into an ordered list of JsonDocuments.
Instead of flatMap, you can either use:
concatMap: will retain order, but actually wait for each inner GET to complete before firing the next one (could revert to sequential execution with less performance)
concatMapEager: will immediately subscribe inner Observables (so trigger inner GET). Maintains the order by buffering responses that arrive out of order until they can be replayed at the correct index in the sequence. Best of both worlds in terms of ordering and performance.
I would use Zip operator to concat all your observables, and then once they finish add documents results into the list
#Test
public void zipObservables() {
Observable<String> oKey1 = Observable.just("key1").doOnNext(getDocument());
Observable<String> oKey2 = Observable.just("key2").doOnNext(getDocument());
Observable<String> oKey3 = Observable.just("key3").doOnNext(getDocument());
Observable<String> oKey4 = Observable.just("key4").doOnNext(getDocument());
List<Observable<String>> observables = Arrays.asList(oKey1,oKey2,oKey3,oKey4);
List<Object> foundDocs = Observable.zip(observables, Arrays::asList)
.toBlocking()
.single();
}
private Action1<String> getDocument() {
return id -> bucket.async().get(id);
}
You can see more Zip examples here https://github.com/politrons/reactive/blob/master/src/test/java/rx/observables/combining/ObservableZip.java
I am currently working on a ecommerce application where I have to show a list of available products using search functionality.
As with every search, I have to implement Pagination here.
I am using mybatis as my ORM tool and mysql as an underlying database.
Googling around I found following ways to accomplish this task :
Client Side paging
: Here I will have to fetch all the results from the database matching the search criteria in one stroke and handle the pagination at my code level (Possibly frond end code ).
Server Side Paging :
With mysql I can use the Limit and the offset of the resultset to construct a query like :
SELECT * FROM sampletable WHERE condition1>1 AND condition2>2 LIMIT 0,20
Here, I have to pass the offset and limit count everytime the user selects a new page while navigating in search results.
Can anyone tell,
which will be better way to implement paging ?
Do mybatis supports a better way to implement paging than just relying on above SQL queries ( like the hibernate criteria APIs).
Any inputs is highly appreaciated.
Thanks .
I myself use your second opion with LIMIT in sql query.
But there is range of methods that support pagination using RowBounds class.
This is well described in mybatis documentation here
Pay attention to correct result set type to use.
If you're using Mappers (much easier than using raw SqlSessions), the easiest way to apply a limit is by adding a RowBounds parameter to the mapping function's argument list, e.g:
// without limit
List<Foo> selectFooByExample(FooExample ex);
// with limit
List<Foo> selectFooByExample(FooExample ex, RowBounds rb);
This is mentioned almost as an afterthought in the link Volodymyr posted, under the Using Mappers heading, and could use some more emphasis:
You can also pass a RowBounds instance to the method to limit query results.
Note that support for RowBounds may vary by database. The Mybatis documentation implies that Mybatis will take care of using the appropriate query. However, for Oracle at least, this gets handled by very inefficient repeat calls to the database.
pagination has two types, physical and logical
logical means to retrieve all the data first then sort them in memory
physical means database level subset select
the default mybatis pagination is logical... thus when you select a massive database e.g 100GB of blobs, the rowbound method will still be very slow
the solution is to use the physical pagination
you can do your own way through the mybatis interceptor
or using plugins pre made by someone else
If you are using Spring MyBatis, you can achieve pagination manually using 2 MyBatis queries and the useful Spring Page and Pageable interfaces.
You create a higher level DAO interface e.g. UploadDao
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
public interface UploadDao {
Page<Upload> search(UploadSearch uploadSearch, Pageable pageable);
}
... where Upload maps to an upload table and UploadSearch is a parameter POJO e.g.
#Data // lombok
public class UploadSearch {
private Long userId;
private Long projectId;
...
}
An implementation of UploadDao (which injects a MyBatis UploadMapper mapper) is as follows:
public class DefaultUploadDao implements UploadDao {
#Autowired
private UploadMapper uploadMapper;
public Page<Upload> searchUploads(UploadSearch uploadSearch, Pageable pageable) {
List<Upload> content = uploadMapper.searchUploads(uploadSearch, pageable);
Long total = uploadMapper.countUploads(uploadSearch);
return new PageImpl<>(content, pageable, total);
}
}
The DAO implementation calls 2 methods of UploadMapper. These are:
UploadMapper.searchUploads - returns a page of results based on search param (UploadSearch) and Pageable param (contains offset / limit etc).
UploadMapper.countUploads - returns total count, again based on search param UploadSearch. NOTE - Pageable param is not required here as we're simply determining the total rows the search parameter filters to and don't care about page number / offset etc.
The injected UploadMapper interface looks like ...
#Mapper
public interface UploadMapper {
List<Upload> searchUploads(
#Param("search") UploadSearch search,
#Param("pageable") Pageable pageable);
long countUploads(
#Param("search") UploadSearch search);
}
... and the mapper XML file containing the dynamic SQL e.g. upload_mapper.xml contains ...
<mapper namespace="com.yourproduct.UploadMapper">
<select id="searchUploads" resultType="com.yourproduct.Upload">
select u.*
from upload u
<include refid="queryAndCountWhereStatement"/>
<if test="pageable.sort.sorted">
<trim prefix="order by">
<foreach item="order" index="i" collection="pageable.sort" separator=", ">
<if test="order.property == 'id'">id ${order.direction}</if>
<if test="order.property == 'projectId'">project_id ${order.direction}</if>
</foreach>
</trim>
</if>
<if test="pageable.paged">
limit #{pageable.offset}, #{pageable.pageSize}
</if>
<!-- NOTE: PostgreSQL has a slightly different syntax to MySQL i.e.
limit #{pageable.pageSize} offset #{pageable.offset}
-->
</select>
<select id="countUploads" resultType="long">
select count(1)
from upload u
<include refid="queryAndCountWhereStatement"/>
</select>
<sql id="queryAndCountWhereStatement">
<where>
<if test="search != null">
<if test="search.userId != null"> and u.user_id = #{search.userId}</if>
<if test="search.productId != null"> and u.product_id = #{search.productId}</if>
...
</if>
</where>
</sql>
</mapper>
NOTE - <sql> blocks (along with <include refid=" ... " >) are very useful here to ensure your count and select queries are aligned. Also, when sorting we are using conditions e.g. <if test="order.property == 'projectId'">project_id ${order.direction}</if> to map to a column (and stop SQL injection). The ${order.direction} is safe as the Spring Direction class is an enum.
The UploadDao could then be injected and used from e.g. a Spring controller:
#RestController("/upload")
public UploadController {
#Autowired
private UploadDao uploadDao; // Likely you'll have a service instead (which injects DAO) - here for brevity
#GetMapping
public Page<Upload>search (#RequestBody UploadSearch search, Pageable pageable) {
return uploadDao.search(search, pageable);
}
}
If you are using the MyBatis Generator, you may want to try the Row Bounds plugin from the official site: org.mybatis.generator.plugins.RowBoundsPlugin. This plugin will add a new version of the
selectByExample method that accepts a RowBounds parameter.
I am using Mahout to build an Item-based Cf recommendation engine.
I create an MahoutHelper class which has a constructor:
public MahoutHelper(String serverName, String user, String password,
String DatabaseName, String tableName) {
source = new MysqlConnectionPoolDataSource();
source.setServerName(serverName);
source.setUser(user);
source.setPassword(password);
source.setDatabaseName(DatabaseName);
source.setCachePreparedStatements(true);
source.setCachePrepStmts(true);
source.setCacheResultSetMetadata(true);
source.setAlwaysSendSetIsolation(true);
source.setElideSetAutoCommits(true);
DBmodel = new MySQLJDBCDataModel(source, tableName, "userId", "itemId",
"value", null);
similarity = new TanimotoCoefficientSimilarity(DBmodel);
}
and the recommend method is:
public List<RecommendedItem> recommendation() throws TasteException {
Recommender recommender = null;
recommender = new GenericItemBasedRecommender(DBmodel, similarity);
List<RecommendedItem> recommendations = null;
recommendations = recommender.recommend(userId, maxNum);
System.out.println("query completed");
return recommendations;
}
It's using datasource to build datamodel but the problem is that when mysql has only a few data (less than 100) the program works fine for me, while when the scale turns to be over 1,000,000, the program stacks at doing recommendation and never goes forward. I have no idea how it happens. By the way I used the same data to build a FileDataModel with a .dat file, and it takes only 2~3 second to complete analysis. I am confused.
Using the database directly will only work for tiny data sets, like maybe a hundred thousand data points. Beyond that the overhead of such data-intensive applications will never run quickly; a query takes thousands of SQL queries or more.
Instead you must load and re-load into memory. You can still pull from the database; look at ReloadFromJDBCDataModel as a wrapper.
I'm trying to select an object using values of another object in LINQ SQL,
I currently have this,
var result1 = (from s in pdc.ScanLogs
from ec in pdc.ExhibitsContacts
where s.ExhibitID == ec.ExhibitID
select ec.Contact);
I want to assign a value of ec.Contact.Note = ec.Comment;
Is there to a way to do this in LINQ SQL without writing multiple queries?
I read this blog article: http://blog.robvolk.com/2009/05/linq-select-object-but-change-some.html but it doesn't seem to work with LINQ SQL.
Basically you can't do this. LINQ is meant to be a query language, and what you want to do is mutate existing entities with your query. This means your query would have side effects and this is not something that is supported by LINQ to SQL.
While this won't work in a single query while returning LINQ to SQL entities, what will work is when you return simple DTO structues. For instance:
var result1 =
from s in pdc.ScanLogs
from ec in s.ExhibitsContacts
select new ContactDto
{
Id = ec.Contact.Id,
Note = ec.Comment,
SomeOtherFields = ec.Contact.SomeOtherFields
};
As a side note: also look at how I removed the where s.ExhibitID == ec.ExhibitID join from the query, by just using the ExhibitsContacts property of the ScanLog entity (which will be generated by LINQ to SQL for you when your database schema has the proper foreign keys defined).
Update:
When you need to return those DTO from several methods, you might consider centralizing the transformation from a collection of entities to a collection of DTO objects. What I tend to do is place this method on the DTO (which makes it easy to find). The code might look like this:
public class ContactDto
{
// Many public properties here
public static IQueryable<ContactDto> ToDto(
IQueryable<Contact> contacts)
{
return
from contact in contacts
select new ContactDto
{
Id = contact.Id,
Note = contact.ExhibitsContact.Comment,
ManyOtherFields = contact.ManyOtherFields
};
}
}
The trick with this static transformation method is that it takes an IQueryable and returns an IQueryable. This allows to to simply specify the transformation and let LINQ to SQL (or any other LINQ enabled O/RM) to efficiently execute that LINQ expression later on. The original code would now look like this:
IQueryable<Contact> contacts =
from s in pdc.ScanLogs
from ec in s.ExhibitsContacts
select ec.Contact;
IQuerable<ContactDto> result1 = ContactDto.ToDto(contacts);
the problem is that LINQ to SQL does not know how to interpret your extension method. The only way, other than using stored procedures from LINQ to SQL (which kind of defeats the ponit), is to get the object, update and then commit changes.