The most efficient way to store photo reference in a database - mysql

I'm currently looking to store approximately 3.5 million photo's from approximately 100/200k users. I'm only using a mysql database on aws. My question is in regards to the most efficient way to store the photo reference. I'm only aware of two ways and I'm looking for an expert opinion.
Choice A
A user table with a photo_url column, in that column I would build a comma separated list of photo's that both maintain the name and sort order. The business logic would handle extracting the path from the photo name and append photo size. The downside is the processing expense.
Database example
"0ea102, e435b9, etc"
Business logic would build the following urls from photo name
/0e/a1/02.jpg
/0e/a1/02_thumb.jpg
/e4/35/b9.jpg
/e4/35/b9_thumb.jpg
Choice B - Relational Table joined on user table with the following fields. I'm just concerned I may have potential database performance issues.
pk
user_id
photo_url_800
photo_url_150
photo_url_45
order
Does anybody have any suggestions on the better solution?

The best and most common answer would be: choice B - Relational Table joined on user table with the following fields.
id
order
user_id
desc
photo_url_800
photo_url_150
photo_url_45
date_uploaded
Or a hybrid, wherein, you store the file names individually and add the photo directory with your business logic layer.
My analysis, your first option is a bad practice. Comma separated fields are not advisable for database. It would be difficult for you to update these fields and add description on it.
Regarding the table optimization, you might want to see these articles:
Optimizing MyISAM Queries
Optimizing InnoDB Queries

Here is an example of my final solution using the hibernate ORM, Christian Mark, and my hybrid solution.
#Entity
public class Photo extends StatefulEntity {
private static final String FILE_EXTENSION_JPEG = ".jpg";
private static final String ROOT_PHOTO_URL = "/photo/";
private static final String PHOTO_SIZE_800 = "_800";
private static final String PHOTO_SIZE_150 = "_150";
private static final String PHOTO_SIZE_100 = "_100";
private static final String PHOTO_SIZE_50 = "_50";
#ManyToOne
#JoinColumn(name = "profile_id", nullable = false)
private Profile profile;
//Example "a1d2b0" which will later get parsed into "/photo/a1/d2/b0_size.jpg"
//using the generatePhotoUrl business logic below.
#Column(nullable = false, length = 6)
private String fileName;
private boolean temp;
#Column(nullable = false)
private int orderBy;
#Temporal(TemporalType.TIMESTAMP)
private Date dateUploaded;
public Profile getProfile() {
return profile;
}
public void setProfile(Profile profile) {
this.profile = profile;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public Date getDateUploaded() {
return dateUploaded;
}
public void setDateUploaded(Date dateUploaded) {
this.dateUploaded = dateUploaded;
}
public boolean isTemp() {
return temp;
}
public void setTemp(boolean temp) {
this.temp = temp;
}
public int getOrderBy() {
return orderBy;
}
public void setOrderBy(int orderBy) {
this.orderBy = orderBy;
}
public String getPhotoSize800() {
return generatePhotoURL(PHOTO_SIZE_800);
}
public String getPhotoSize150() {
return generatePhotoURL(PHOTO_SIZE_150);
}
public String getPhotoSize100() {
return generatePhotoURL(PHOTO_SIZE_100);
}
public String getPhotoSize50() {
return generatePhotoURL(PHOTO_SIZE_50);
}
private String generatePhotoURL(String photoSize) {
String firstDir = getFileName().substring(0, 2);
String secondDir = getFileName().substring(2, 4);
String photoName = getFileName().substring(4, 6);
StringBuilder sb = new StringBuilder();
sb.append(ROOT_PHOTO_URL);
sb.append("/");
sb.append(firstDir);
sb.append("/");
sb.append(secondDir);
sb.append("/");
sb.append(photoName);
sb.append(photoSize);
sb.append(FILE_EXTENSION_JPEG);
return sb.toString();
}
}

Related

JDBCTemplate : how to fetch values of MySQL variables

I want to get the value of a MySQL variable (example: max_allowed_packet) via jdbcTemplate. is this possible? if so, how ?
In SQL, I can do SHOW VARIABLES LIKE 'max_allowed_packet'; but how to do this via JDBCTemplate ?
Here is a solution
public List<Variable> findAllVariables() {
List<Variable> result = jdbcTemplate.query("SHOW GLOBAL VARIABLES", new VariableRowMapper());
//about 630 variables
return result;
}
Variable class:
public class Variable {
private String name;
private String value;
//getters and setters
}
VariableRowMapper class:
public class VariableRowMapper implements RowMapper<Variable> {
#Override
public Variable mapRow(ResultSet resultSet, int rowNum) throws SQLException {
String name = resultSet.getString("Variable_Name");
String value = resultSet.getString("Value");
return new Variable(name, value);
}
}
hope it helps.
I was particularly interested in getting the max_allowed_packet variable from the database. This below snippet does the trick.
private int fetchMaxAllowedPacketsFromDB(JdbcTemplate jdbcTemplate) {
final String sql = "SELECT ##GLOBAL.max_allowed_packet";
Integer maxAllowedPacketsFromDB = jdbcTemplate.queryForObject(sql, Integer.class);
log.info("##GLOBAL.max_allowed_packet : {}", maxAllowedPacketsFromDB);
return maxAllowedPacketsFromDB;
}
You can look at #Ali4j 's answer for a more generic/multi-variable requirement.
Or, You can refactor the snippet above to pass in a variable as argument, if you don't need the extra work of RowMappers

Room returns incorrect initialized object from generated query

I have three tables, one containing Cards, one containing CardDecks and third one implementing a many-to-many relation between the former two and additionally containg a symbol for every relation entry.
My task is to get three columns from the card-table and the symbol from the relation-table and save it in a data Object specifically designed for handling those inputs, the codition being, that all entries match the given deckId. Or in (hopefully correct) sql-language:
#Query("SELECT R.symbol, C.title, C.type, C.source " +
"FROM card_table C JOIN cards_to_card_deck R ON C.id = R.card_id"+
"WHERE R.card_deck_id = :cardDeckId")
LiveData<List<CardWithSymbol>> getCardsWithSymbolInCardDeckById(long cardDeckId);
But the room implementation class generates:
#Override
public LiveData<List<CardWithSymbol>> getCardsWithSymbolInCardDeckById(long
cardDeckId) {
final String _sql = "SELECT R.symbol, C.title, C.typ, C.source FROM
cards_to_card_deck R INNER JOIN card_table C ON R.card_id = C.id WHERE
R.card_deck_id = ?";
final RoomSQLiteQuery _statement = RoomSQLiteQuery.acquire(_sql, 1);
int _argIndex = 1;
_statement.bindLong(_argIndex, cardDeckId);
return new ComputableLiveData<List<CardWithSymbol>>() {
private Observer _observer;
#Override
protected List<CardWithSymbol> compute() {
if (_observer == null) {
_observer = new Observer("cards_to_card_deck","card_table") {
#Override
public void onInvalidated(#NonNull Set<String> tables) {
invalidate();
}
};
__db.getInvalidationTracker().addWeakObserver(_observer);
}
final Cursor _cursor = __db.query(_statement);
try {
final int _cursorIndexOfSymbol = _cursor.getColumnIndexOrThrow("symbol");
final List<CardWithSymbol> _result = new ArrayList<CardWithSymbol>(_cursor.getCount());
while(_cursor.moveToNext()) {
final CardWithSymbol _item;
final int _tmpSymbol;
_tmpSymbol = _cursor.getInt(_cursorIndexOfSymbol);
_item = new CardWithSymbol(_tmpSymbol,null,null,null);
_result.add(_item);
}
return _result;
} finally {
_cursor.close();
}
}
#Override
protected void finalize() {
_statement.release();
}
}.getLiveData();
}
Where
_item = new CardWithSymbol(_tmpSymbol,null,null,null);
should return my fully initialized object.
The CardWithSymbol class is declared as follows:
public class CardWithSymbol {
public int symbol;
public String cardName;
public String cardType;
public String cardSource;
public CardWithSymbol(int symbol, String cardName, String cardType, String cardSource){
this.symbol = symbol;
this.cardName = cardName;
this.cardType = cardType;
this.cardSource = cardSource;
}
And the types of the columns returned by the query are:
int symbol, String title, String type, String source
I already went through some debugging and the rest of the application works just fine. I can even read the symbol from the objects return by the query, but as mentioned above for some reason room ignores the other three parameters and just defaults them to null in the query-implementation.
So after some trial and error and reading through the dao-documentation once again i found my error:
When creating a class for handling subsets of columns in room, it is important to tell room which variable coresponds to which columns via #ColumnInfo(name = "name of the column goes here")-annotation.
So changing my CardWithSymbol class as follows solved the issue for me:
import android.arch.persistence.room.ColumnInfo;
public class CardWithSymbol {
#ColumnInfo(name = "symbol")
public int symbol;
#ColumnInfo(name = "title")
public String cardName;
#ColumnInfo(name = "type")
public String cardType;
#ColumnInfo(name = "source")
public String cardSource;
public CardWithSymbol(int symbol, String cardName, String cardType, String cardSource){
this.symbol = symbol;
this.cardName = cardName;
this.cardType = cardType;
this.cardSource = cardSource;
}
}

saving large object takes too long on hibernate

I have an object with a Blob column requestData and a Text Column "requestDataText" .
These two fields may hold large Data. In my example , the blob data is around 1.2 MBs and the Text column holds the text equivalent of that Data.
When i try to commit this single entity , it takes around 20 seconds .
DBUtil.beginTransaction();
session.saveOrUpdate(entity);
DBUtil.commitTransaction();
Is there something wrong or is there a way to shorten this period ?
package a.db.entity;
// Generated Feb 22, 2016 11:57:10 AM by Hibernate Tools 3.2.1.GA
/**
* Foo generated by hbm2java
*/
#Entity
#Table(name="foo"
,catalog="bar"
)
public class Foo implements java.io.Serializable {
private Long id;
private Date reqDate;
private byte[] requestData;
private String requestDataText;
private String functionName;
private boolean confirmed;
private boolean processed;
private boolean errorOnProcess;
private Date processStartedAt;
private Date processFinishedAt;
private String responseText;
private String processResult;
private String miscData;
public AsyncRequestLog() {
}
#Id #GeneratedValue(strategy=IDENTITY)
#Column(name="Id", unique=true, nullable=false)
public Long getId() {
return this.id;
}
public void setId(Long id) {
this.id = id;
}
...
}
I just noticed you're starting a transaction and then doing a saveOrUpdate() which might explain the slow down, as hibernate will try to retrieve the row from the DB first (as explained on this other SO answer).
If you know if the entity is new call save() and if you the entity has to be updated call update().
Another suggestion, but I'm not sure if this applies any more to MySQL, try to store the blobs/clobs in a different table from where you store the data, if you are intending to update the blob/clobs. In the past this mix made MySQL run slow as it had to resize the 'block' allocated to a row. So have one table with all the attributes and a different table just for the blob/clob. This is not the case if the table is read-only.

Hibernate: Storing an fixed length array in one database table row

I have been trying to find a solution to store a fixed length array as a property of an object using hibernate in the same DB table as the object not using a BLOB for the array.
I currently have a class ProductionQCSession which looks like
#Entity
public class ProductionQCSession extends IdEntity {
private Long id;
private Float velocity;
private Float velocityTarget;
private Float[] velocityProfile;
public ProductionQCSession() {
}
#Id #GeneratedValue(strategy=GenerationType.AUTO)
#Override
public Long getId() {
return id;
}
#SuppressWarnings("unused")
public void setId(Long id) {
this.id = id;
}
#Basic
public Float getVelocity() {
return velocity;
}
public void setVelocity(Float velocity) {
this.velocity = velocity;
}
#Basic
public Float[] getVelocityProfile() {
return velocityProfile;
}
public void setVelocityProfile(Float[] velocityProfile) {
this.velocityProfile = velocityProfile;
}
}
Ideally I would like the DB structure to be
id|velocity|VPValue0|VPValue1|VPValue2|VPValue3|...
21| 2.1| 0.1| 0.2| -0.1| 0.3|...
I know with a high certainty that we always have 15 items in the velocityProfile array and those values as just as much properties of the object as any other property therefore I think it makes sense to add them to the database table schema, if it's possible. I would prefer to have it this way as it would be easy to get a overview of the data just doing a raw table print.
The current code just stores the array data as a BLOB.
I have looked http://ndpsoftware.com/HibernateMappingCheatSheet.html mapping cheat sheet, but could not seem to find any good solution.
I'm I just trying to do something nobody else would do?
Essentially, you're trying to have a multi-value field, which is not a relational database concept. A normalized solution would put those into a child table, which Hibernate would let you access directly from the parent row (and return it as a collection).
If you are adamant that it should be in a single table, then you'll need to create 15 individual columns....and hope that in the future you don't suddenly need 16.
The solution ended up being using the standardised method of using a child table even though it makes the data analysis slightly more complicated. The following code was used.
#ElementCollection
#CollectionTable(name ="QCVelocityProfile")
public List<Float> getVelocityProfile() {
return velocityProfile;
}
public void setVelocityProfile(List<Float> velocityProfile) {
this.velocityProfile = velocityProfile;
}

How to handle unidirectional many-to-many relations with Ebean

I have a problem with Ebean. I have the usual Objects PsecUser, PsecRoles and PsecPermission.
A user can have many Permissions or Roles and a Role can have many Permission.
Here the code (extract):
#Entity
public class PsecPermission {
#Id
#GeneratedValue
private Long id;
#Column(unique=true, nullable=false)
private String name;
#Column(nullable=false)
private String type = PsecBasicPermission.class.getName();
#Column(nullable=false)
private String target;
#Column(nullable=false)
private String actions;
}
#Entity
public class PsecRole {
#Id
#GeneratedValue
private Long id;
#Column(unique=true, nullable=false)
private String name;
#Temporal(TemporalType.TIMESTAMP)
private Date lastUpdate;
#ManyToMany(fetch=FetchType.EAGER)
private List<PsecPermission> psecPermissions;
private boolean defaultRole = false;
}
I wrote the following helper-method:
public PsecRole createOrUpdateRole(String name, boolean defaultRole, String... permissions) {
PsecRole result = server.find(PsecRole.class).
where().eq("name", name).findUnique();
if (result == null) {
result = new PsecRole();
result.setName(name);
}
final List<PsecPermission> permissionObjects = server.find(PsecPermission.class).
where().in("name", (Object[])permissions).findList();
result.setPsecPermissions(permissionObjects);
result.setDefaultRole(defaultRole);
final Set <ConstraintViolation <PsecRole>> errors =
Validation.getValidator().validate(result);
if (errors.isEmpty()) {
server.save(result);
server.saveManyToManyAssociations(result, "psecPermissions");
} else {
log.error("Can't save role: " + name +"!");
for (ConstraintViolation <PsecRole> constraintViolation : errors) {
log.error(" " + constraintViolation);
}
}
return result;
}
and try the following test:
#Test
public void testCreateOrUpdateRole() {
String[] permNames = {"Test1", "Test2", "Test3"};
List <PsecPermission> permissions = new ArrayList <PsecPermission>();
for (int i = 0; i < permNames.length; i++) {
helper.createOrUpdatePermission(permNames[i], "target"+ i, "actions" +i);
PsecPermission perm = server.find(PsecPermission.class).where().eq("name", permNames[i]).findUnique();
assertThat(perm.getTarget()).isEqualTo("target" + i);
assertThat(perm.getActions()).isEqualTo("actions" + i);
permissions.add(perm);
}
PsecRole orgRole = helper.createOrUpdateRole(ROLE, false, permNames);
testRole(permNames, orgRole);
PsecRole role = server.find(PsecRole.class).where().eq("name", ROLE).findUnique();
testRole(permNames, role);
}
private void testRole(String[] permNames, PsecRole role) {
assertThat(role).isNotNull();
assertThat(role.getName()).isEqualTo(ROLE);
assertThat(role.isDefaultRole()).isEqualTo(false);
assertThat(role.getPermissions()).hasSize(permNames.length);
}
Which fails if it checks the number of permissions at the readed role. It's always 0.
I looked into the database and found that psec_role_psec_permission is alway empty.
Any idea what's wrong with the code?
You can get a pure Ebean-example from https://github.com/opensource21/ebean-samples/downloads it uses the eclipse-plugin from ebean.
There are two solutions for this problem:
Simply add cascade option at PsceRole
#ManyToMany(fetch=FetchType.EAGER, cascade=CascadeType.ALL)
private List<PsecPermission> psecPermissions;
and remove server.saveManyToManyAssociations(result, "psecPermissions"); you find it in the cascade-solution-branch.
The cleaner solution, because you don't need to define cascase- perhaps you don't want it:
Just don't replace the list, just add your entries to the list. Better is to add new and remove old one. This mean in createOrUpdateRole:
result.getPsecPermissions().addAll(permissionObjects);
instead of
result.setPsecPermissions(permissionObjects);