I am working on a database project using the Spring JDBC API with MySQL and Angular. I want to update the SQL query efficiently and get back that data from the DB. Right now i'm using StringBuilder to manually update the SQL query and getting the data back. Is there a better way to do this?
The initial state of the query is,
private String SQL_MZGTE = "SELECT * FROM `protein` WHERE `auroc_for_mixture_0` >= 0.65";
Receiving the name and dataset name from the controller (#RequestBody). This is what i'm doing at the repository level,
#Override
public void filter(Params[] params) {
if (!(params.length == 0)) {
var datasetList = new ArrayList<String>();
var SQL = new StringBuilder("SELECT * FROM `protein` WHERE `auroc_for_mixture_0` >= 0.65");
SQL.append(" AND ");
for (var dataset : params) {
if (!datasetList.contains(dataset.getOfDataset())) {
SQL.append("`dataset`").append(" = ").append("'").append(dataset.getOfDataset()).append("'").append(" OR ");
datasetList.add(dataset.getOfDataset());
}
}
SQL.replace(SQL.lastIndexOf("OR"), SQL.lastIndexOf("OR") + 2, "AND");
SQL.append("(");
for (var omicsFeature : params) {
SQL.append("`feature_type`").append(" = ").append("'").append(omicsFeature.getOmicsFeature()).append("'").append(" OR ");
}
this.SQL_MZGTE = SQL.substring(0, SQL.lastIndexOf("OR") - 1) + ")";
}
System.out.println("PARAMETERS --> " + Arrays.toString(params));
System.out.println("SQL --> " + this.SQL_MZGTE);
}
This is what i'm receiving through the #RequestBody,
#Data
#NoArgsConstructor
#AllArgsConstructor
public class Params {
private String omicsFeature;
private String ofDataset;
}
From another endpoint i'm getting this as the #RequestBody,
#Getter
#ToString
#NoArgsConstructor
#AllArgsConstructor
public class CancerTypeParam {
private String cancerType;
}
The goal is to combine omicsFeature as a union (or) if from the frontend multiple omicsFeature are being selected and same for the cancerType as well, but the there should be an intersection (and) between these two, so the query should be something like this,
SELECT * FROM `TABLE_NAME` WHERE `auroc_for_mixture_0` >= 0.65 AND `dataset` = 'DATASET_NAME' AND (`feature_type` = 'ft1' OR `feature_type` = 'ft2') AND (`cancer_type` = 'ct1' OR `cancer_type` = 'ct2');
And when the checkbox is unchecked at the frontend, the query should be changed or updated as well. If i'm unchecking the cancer_type 'ct1' the query should be like this,
SELECT * FROM `TABLE_NAME` WHERE `auroc_for_mixture_0` >= 0.65 AND `dataset` = 'DATASET_NAME' AND (`feature_type` = 'ft1' OR `feature_type` = 'ft2') AND (`cancer_type` = 'ct2');
These are just 2 constraints, there are many more in the DB. How can I do this efficiently?
Thanks in Advance!
Related
I want to retrieve all the trains with the given coach type that runs between the source and destination from the database in ascending order based on train number;
I am trying this query.This is a JDBC code to fetch the trains. I couldn't figure out what is wrong in this.
`import java.sql.*;
import java.util.*;
import java.io.*;
public class TrainManagementSystem {
public ArrayList <Train> viewTrain (String coachType, String source, String destination){
// Fill your code here
Connection myConn =null;
PreparedStatement myStmt = null;
ResultSet myRes = null;
ArrayList <Train> trainArr = new ArrayList<>();
try{
Properties props = new Properties();
FileInputStream in = new FileInputStream("database.properties");
props.load(in);
in.close();
String driver = props.getProperty("DB_DRIVER_CLASS");
if (driver != null) {
Class.forName(driver) ;
}
myConn = DriverManager.getConnection(
props.getProperty("DB_URL"),
props.getProperty("DB_USERNAME"),
props.getProperty("DB_PASSWORD"));
myStmt = myConn.prepareStatement("SELECT * from train WHERE source = ? AND destination = ? AND ? != 0 ORDER BY train_number");
myStmt.setString(1,source);
myStmt.setString(2,destination);
myStmt.setString(3,coachType);
myRes = myStmt.executeQuery();
while(myRes.next()){
trainArr.add(new Train(
myRes.getInt("train_number"),
myRes.getString("train_name"),
myRes.getString("source"),
myRes.getString("destination"),
myRes.getInt("ac1"),
myRes.getInt("ac2"),
myRes.getInt("ac3"),
myRes.getInt("sleeper"),
myRes.getInt("seater")));
}
return trainArr;
}catch(Exception exc){
exc.printStackTrace();
return null;
}
}
}
coachType=ac1,ac2,ac3,sleeper,seater;
In my opinion there are two approaches:
make a SELECT without the condition on the parametrized column (derived from coachType) and filter the results in your Java code. You'll get all the trains with the desired source and destination, ordered by train_number
myStmt = myConn.prepareStatement("SELECT * from train WHERE source = ? AND destination = ? ORDER BY train_number");
myStmt.setString(1,source);
myStmt.setString(2,destination);
myRes = myStmt.executeQuery();
and then, in your loop on the resultset,
while(myRes.next()) {
if (myRes.getInt(coachType) != 0) { // excludes records with column derived from coachType != 0
trainArr.add(new Train(
myRes.getInt("train_number"),
myRes.getString("train_name"),
myRes.getString("source"),
myRes.getString("destination"),
myRes.getInt("ac1"),
myRes.getInt("ac2"),
myRes.getInt("ac3"),
myRes.getInt("sleeper"),
myRes.getInt("seater")));
}
}
To avoid SQL Injection you can make an apparently useless query:
firstStmt = myConn.prepareStatement("SELECT column_name FROM information_schema.columns WHERE table_schema = ? AND table_name = ? AND column_name = ?");
firstStmt.setString(1, table_schema); // a variable with your schema name
firstStmt.setString(2, "train");
firstStmt.setString(3, coachType);
safeQueryRes = firstStmt.executeQuery();
safeCoachTypeColName = safeQueryRes.next().getString("column_name");
In this way you can use concatenation in your final query, avoiding SQL Injection, as if someone puts some hacking string in your coachType input variable, the first query will fail and nothing dangerous will happen. Otherwise, if the first query will be executed without a SQL Injection, you'll get the "real-and-SQL-Injection-safe" column name that you can use to create your final query with string concatenation:
myStmt = myConn.prepareStatement("SELECT * from train WHERE source = ? AND destination = ? AND " + safeCoachTypeColName + " != 0 ORDER BY train_number");
myStmt.setString(1,source);
myStmt.setString(2,destination);
myRes = myStmt.executeQuery();
I am trying to import the data from excel ile to different tables in my DB.
The import is done , the only problem is that some of the data is in the wrong order.
This is what my excel file loks like.
And this is what the data in my Bd lokks like.
My service file has this method:
#Override
public List<Task> getTasksFromExcel(MultipartFile files) throws IOException {
List<Task> taskList = new ArrayList<>();
XSSFWorkbook workbook = new XSSFWorkbook(files.getInputStream());
XSSFSheet worksheet = workbook.getSheetAt(0);
String a = "A";
for (int index = 0; index <= worksheet.getPhysicalNumberOfRows(); index++) {
if (index > 1) {
Task task = new Task();
Lot lot = new Lot();
String ref = a + index;
CellReference cr = new CellReference(ref);
XSSFRow row = worksheet.getRow(cr.getRow());
String lotName = row.getCell(0).getStringCellValue();
Lot existingLot = lotRepository.findByName(lotName);
if (existingLot == null) {
lot.setName(lotName);
lotRepository.save(lot);
} else {
lot = existingLot;
}
;
task.setName(row.getCell(1).getStringCellValue());
String email = row.getCell(2).getStringCellValue();
Collaborator collab = collaboratorRepository.findByEmail(email);
task.setCollaborator(collab);
List<Double> iC = new ArrayList<>();
for (int i = 3; i < 6; i++) {
iC.add((Double) row.getCell(i).getNumericCellValue());
}
Set<Double> charge = new HashSet<Double>();
charge.addAll(iC);
task.setInitialCharge(charge);
task.setLot(lot);
taskList.add(task);
taskRepository.save(task);
}
}
return taskList;
}
And for the entity definition I go tthis.
#Entity
#Table(name = "task")
public class Task {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private int id;
private String name;
private String description;
private Date assignment;
private Date deadline;
#ElementCollection
private Set<Double> initialCharge=new HashSet<Double>();
#Column(columnDefinition = "varchar(32)")
#Enumerated(EnumType.STRING)
private Status status = Status.TODO;
#ManyToOne(cascade = CascadeType.ALL)
#JoinColumn(name = "lot_id")
private Lot lot;
#ManyToOne()
#JoinColumn(name = "collaborator_id")
private Collaborator collaborator;
You mapped this differently then your excel spreadsheet, and so lose the S1,S2,S3 column name/ordering you had for the single Task row, and seemed to have assumed that the set positional would be consistent, giving you S1-S3 for free. It does not.
Normalizing this out to allow expanding lists of charges and still having an order would mean adding a positional column to the task_initial_charges table. JPA will populate this column behind the scenes if you simply annotate your element collection with the OrderColumn to specify it:
The task_initial_charge needs s1,s2,s3 columns so that a single task_id has 3 positional columns, or you need another column in there to allow writing out the position within your initialCharge Set.
#ElementCollection
#OrderColumn
private Set<Double> initialCharge=new HashSet<Double>();
The order of the initialCharge set when the entity is persisted will then be stored in the database, and should be used when fetching the entity.
I'm quite new to Apache ISIS, and I want to get a list via the dataNucleus with a legacy database(MYSQL), There are 300,000 of the data, But when I'm trying to use repositoryService.allInstances() method to get a List, returns the size of list is 2. I have other domain objects and those works fine.
here is the code and debug infos.
#PersistenceCapable(
identityType = IdentityType.DATASTORE,
schema = "public",
table = "tinstruction_parameter_value"
)
#DatastoreIdentity(
strategy = IdGeneratorStrategy.IDENTITY,
column = "id")
#Queries({
#Query(
name = "find", language = "JDOQL",
value = "SELECT "
+ "FROM domainapp.modules.simple.dom.impl.xfs.parameter.InstructionParameterValueTest "),
#Query(
name = "findByValueContains", language = "JDOQL",
value = "SELECT "
+ "FROM domainapp.modules.simple.dom.impl.xfs.parameter.InstructionParameterValueTest "
+ "WHERE value.indexOf(:value) >= 0 "),
#Query(
name = "findByValue", language = "JDOQL",
value = "SELECT "
+ "FROM domainapp.modules.simple.dom.impl.xfs.parameter.InstructionParameterValueTest "
+ "WHERE value == :value ")
})
#DomainObject(
editing = Editing.DISABLED
)
#DomainObjectLayout(
bookmarking = BookmarkPolicy.AS_ROOT
)
public class InstructionParameterValueTest implements Comparable<InstructionParameterValueTest> {
#Column(allowsNull = "true",jdbcType = "CLOB")
#Property()
#MemberOrder(sequence = "10")
#Getter #Setter
private String value;
//region > compareTo, toString
#Override
public int compareTo(final InstructionParameterValueTest other) {
return org.apache.isis.applib.util.ObjectContracts.compare(this, other, "value");
}
#Override
public String toString() {
return org.apache.isis.applib.util.ObjectContracts.toString(this, "value");
}
//endregion
}
public class InstructionParameterValueTestRepository {
#Programmatic
public java.util.List<InstructionParameterValueTest> listAll() {
return repositoryService.allInstances(InstructionParameterValueTest.class);
}
}
dataNucleus debug log
I donot know why the size of the list is 2, not all datas, the debug sql can execute and get all datas.
dataNucleus sql execute
can anyone tell me what I should do,
I am using Yii MCV with multiple dbs.
Why does it take 15 seconds to run this code?
How to improve?
I switch the db connection using a server_id integer value.
This is the class that switches the db connection.
class VillageSlaveM extends VillageM {
const UNVERIFIED = 0;
const VERIFIED = 1;
public static function model($className = __CLASS__) {
return parent::model($className);
}
public static $server_id;
public static $slave_db;
public function getDbConnection() {
self::$slave_db = Yii::app()->dbx;
if (self::$slave_db instanceof CDbConnection) {
self::$slave_db->active = false;
$config = require(Yii::app()->getBasePath() . '/config/main.php');
$connectionString = $config['components']['dbx']['connectionString'];
self::$slave_db->connectionString = sprintf($connectionString, self::$server_id);
self::$slave_db->setActive(true);
return self::$slave_db;
}
else
throw new CDbException(Yii::t('yii', 'Active Record requires a "db" CDbConnection application component.'));
}
}
This is the code that needs 15 seconds to execute, dont know why so much.
It selects the one village that has the oldest last_update_resource timestamp value.
I have set indexes for all the db table fields involved.
$criteria = new CDbCriteria();
$criteria->condition = 'last_update_resource <= ' . ($time_start - 60 * 60 * 8);
$criteria->order = 'last_update_resource asc';
$criteria->limit = 1;
VillageSlaveM::$server_id = $world_id;
$start_x = time();
$model_village = VillageSlaveM::model()->findByAttributes(array('map_type_id' => VillageM::$map_type_id['village'], 'status' => VillageM::ACTIVE), $criteria);
$stop_x = time();
$msg[] = 'start_x: ' . ($stop_x - $start_x);
ps: after this code runs, i have much more complex queries and they run instantly;
in my project , we use springmvc , spring and ibatis framework, the problem is :
in my dao code is :
#Override
public Integer insertAdzoneEnvInfoBatch(List<AdzoneEnvInfoDO> adzoneEnvInfoList) {
return executeInsertBatch("AdzoneEnvInfoDAO.insertAdzoneEnvInfo",adzoneEnvInfoList);
}
public Integer executeInsertBatch(final String sqlID, final List stList) {
Integer result = new Integer(-1);
if ((sqlID != null) && (stList != null) && !stList.isEmpty()) {
result = (Integer) getSqlMapClientTemplate().execute(
new SqlMapClientCallback() {
public Object doInSqlMapClient(SqlMapExecutor executor)
throws SQLException {
Integer result = new Integer(-1);
executor.startBatch();
for (int i = 0; i < stList.size(); i++) {
executor.insert(sqlID, stList.get(i));
}
result = new Integer(executor.executeBatch());
return result;
}
});
}
return result;
}
in my sqlmap file ,the sql is
<insert id="AdzoneEnvInfoDAO.insertAdzoneEnvInfo" parameterClass="adzoneEnvInfo">
insert into c_adzone_env_info(
url,adzoneid,pid,total_screen,screen_no,snapshot,adzone_num,ali_adzone_num,same_screen_num,same_screen_ali_num,
covered,ad_link,ad_snapshot,adzone_owner,
adzone_style,adzone_size,date_time,create_time,update_time
)
values(
#url#,#adzoneid#,#pid#,#totalScreen#,#screenNo#,#snapshot#,#adzoneNum#,#aliAdzoneNum#,
#sameScreenNum#,#sameScreenAliNum#,#covered#,#adLink#,#adSnapshot#,#adzoneOwner#,
#adzoneStyle#,#adzoneSize#,#dateTime# , now() , now()
)
<selectKey resultClass="long" keyProperty="id" type="post">
SELECT last_insert_id() as ID from c_adzone_env_info limit 1
</selectKey>
</insert>
and the dataobject has a property id respond to mysql autoincrement primary key
in my unittest ,code is
#Test
public void test(){
AdzoneEnvInfoDO adzoneEnvInfoDO = new AdzoneEnvInfoDO();
adzoneEnvInfoDO.setAdLink("adlink");
adzoneEnvInfoDO.setAdSnapshot("adsnapshot");
adzoneEnvInfoDO.setAdzoneid(99999999L);
adzoneEnvInfoDO.setAdzoneNum(434);
adzoneEnvInfoDO.setAdzoneOwner(11);
adzoneEnvInfoDO.setAdzoneSize("232下232");
adzoneEnvInfoDO.setAdzoneStyle(2);
adzoneEnvInfoDO.setAliAdzoneNum(334);
adzoneEnvInfoDO.setCovered(33);
adzoneEnvInfoDO.setUrl("sds");
adzoneEnvInfoDO.setUrlId(232323L);
adzoneEnvInfoDO.setTotalScreen(32423);
AdzoneEnvInfoDO adzoneEnvInfoDO1 = new AdzoneEnvInfoDO();
adzoneEnvInfoDO1.setAdLink("adlink");
adzoneEnvInfoDO1.setAdSnapshot("adsnapshot");
adzoneEnvInfoDO1.setAdzoneid(99999999L);
adzoneEnvInfoDO1.setAdzoneNum(434);
adzoneEnvInfoDO1.setAdzoneOwner(12);
adzoneEnvInfoDO1.setAdzoneSize("232下232");
adzoneEnvInfoDO1.setAdzoneStyle(22);
adzoneEnvInfoDO1.setAliAdzoneNum(334);
adzoneEnvInfoDO1.setCovered(33);
adzoneEnvInfoDO1.setUrl("sds");
adzoneEnvInfoDO1.setUrlId(232323L);
adzoneEnvInfoDO1.setTotalScreen(32423);
adzoneEnvInfoDAO.insertAdzoneEnvInfoBatch(Arrays.asList(adzoneEnvInfoDO, adzoneEnvInfoDO1));
System.out.println(adzoneEnvInfoDO.getId());
System.out.println(adzoneEnvInfoDO1.getId());
}
and in normal, the two object id should be parimary key in mysql ,but i found it is always null 0
and if i call a not batch method , it will be normal , the single data method is
public Long insertAdzoneEnvInfo(AdzoneEnvInfoDO adzoneEnvInfo) {
return (Long)executeInsert("AdzoneEnvInfoDAO.insertAdzoneEnvInfo",adzoneEnvInfo);
}
I got the same problem as you recently. I just read around the mybatis codes. This one is helpful to insert entities into MySQL. However it is a bit complicated because an ObjectWrapperFactory and a TypeHandler are registered in mybatis configuration file. See
https://github.com/jactive/java/tree/master/test/mybatis-demo
and the entry point
https://github.com/jactive/java/blob/master/test/mybatis-demo/java/com/jactive/mybatis/DaoTest.java