I'm trying to create a table that contains indexed virtual columns that are referencing a JSON.
I've created the table which contains a virtual column named "amount". The problem is that the JSON doesn't always contain the key "amount". Sometimes it's named "presentationAmount".
Is it possible to put a condition in place for this?
When the JSON contains the key "threeDSecure" that's when "presentationAmount" is used, otherwise "amount" is used.
Here's my create table code:
CREATE TABLE transactions (
id BIGINT(20) NOT NULL AUTO_INCREMENT PRIMARY KEY,
json JSON DEFAULT NULL,
type VARCHAR(12) GENERATED ALWAYS AS (json->>"$.transaction.type"),
uuid VARCHAR(32) GENERATED ALWAYS AS (json->>"$.transaction.payload.id"),
holder VARCHAR(20) GENERATED ALWAYS AS (json->>"$.transaction.payload.card.holder"),
amount DECIMAL(11,2) GENERATED ALWAYS AS (json->>"$.transaction.payload.amount"),
resultCode VARCHAR(11) GENERATED ALWAYS AS (json->>"$.transaction.payload.result.code"),
processingTime DATETIME GENERATED ALWAYS AS (json->>"$.transaction.payload.timestamp"),
paymentType VARCHAR(2) GENERATED ALWAYS AS (json->>"$.transaction.payload.paymentType"),
paymentBrand VARCHAR(20) GENERATED ALWAYS AS (json->>"$.transaction.payload.paymentBrand"),
eci INT(2) GENERATED ALWAYS AS (json->>"$.transaction.payload.eci"),
recurringType VARCHAR(9) GENERATED ALWAYS AS (json->>"$.transaction.payload.recurringType"),
clearingInstitute VARCHAR(30) GENERATED ALWAYS AS (json->>"$.transaction.payload.resultDetails.clearingInstituteName"),
merchantTransactionId VARCHAR(64) GENERATED ALWAYS AS (json->>"$.transaction.payload.merchantTransactionId"),
divisionName VARCHAR(32) GENERATED ALWAYS AS (json->>"$.division.name"),
divisionUuid VARCHAR(32) GENERATED ALWAYS AS (json->>"$.division.uuid"),
merchantName VARCHAR(32) GENERATED ALWAYS AS (json->>"$.merchant.name"),
merchantUuid VARCHAR(32) GENERATED ALWAYS AS (json->>"$.merchant.uuid"),
channelName VARCHAR(32) GENERATED ALWAYS AS (json->>"$.channel.name"),
channelUuid VARCHAR(32) GENERATED ALWAYS AS (json->>"$.channel.uuid"),
INDEX typeIndex (type),
INDEX idIndex (uuid),
INDEX holderIndex (holder),
INDEX amountIndex (amount),
INDEX resultCodeIndex (resultCode),
INDEX timestampIndex (processingTime),
INDEX paymentTypeIndex (paymentType),
INDEX paymentBrandIndex (paymentBrand),
INDEX recurringTypeIndex (recurringType),
INDEX clearingInstituteIndex (clearingInstitute),
INDEX merchantTransactionIdIndex (merchantTransactionId),
INDEX divisonNameIndex (divisionName),
INDEX divisionUuidIndex (divisionUuid),
INDEX merchantNameIndex (merchantName),
INDEX merchantUuidindex (merchantUuid),
INDEX channelNameIndex (channelName),
INDEX channelUuidIndex (channelUuid)
) ENGINE=INNODB;
You can use an IF() expression.
amount DECIMAL(11,2) GENERATED ALWAYS AS
(IF(JSON_CONTAINS_PATH(json, 'one', '$.transaction.payload.threeDSecure'),
json->>"$.transaction.payload.presentationAmount",
json->>"$.transaction.payload.amount")),
See the documentation of JSON_CONTAINS_PATH() for details of its use.
Related
I have a database. As you can see the primary key is an auto_increment and is also unique. I read that publically sharing a row's primary key of a table to the public is unsafe. I want to assign each row in customers a unique ID that I can publically share. How can I do this without having to specify each time what the public_id is in the INSERT statement? The database should automatically find a unique ID to assign to that row just like it does for id because of auto_increment.
CREATE TABLE customers (
id int primary key auto_increment,
name varchar(32) not null,
-- public_id (an ID I can give to the public to uniquely identify this row
);
INSERT INTO customers (name) VALUES ('Bob'), ('Sarah'), ('Bob');
Well, here's one way:
CREATE TABLE customers (
id int primary key auto_increment,
name varchar(32) not null,
public_id char(36) not null unique default uuid()
);
Note that the manual says:
Warning
Although UUID() values are intended to be unique, they are not necessarily unguessable or unpredictable. If unpredictability is required, UUID values should be generated some other way.
So this is simple, and maybe will float your goat, but we can also try better:
CREATE TABLE customers (
id int primary key auto_increment,
name varchar(32) not null,
public_id char(24) not null unique default to_base64(random_bytes(18))
);
This will be a nice and dense identifier, but it will have characters + and / which don't play well with URLs. You can encode them, of course, but if you want to go one lazier, you can also do this:
CREATE TABLE customers (
id int primary key auto_increment,
name varchar(32) not null,
public_id char(32) not null unique default hex(random_bytes(16))
);
Mind you, the identifier will get quite a bit longer this way.
To get the best of both worlds, we can do this, at the expense of a really long default value:
CREATE TABLE customers (
id int primary key auto_increment,
name varchar(32) not null,
public_id char(24) not null unique default replace(replace(to_base64(random_bytes(18)), '+', '_'), '/', '-')
);
Also note that messing around with MD5()/SHA()/SHA1()/SHA2() is no better than just generating a random hex string with a given length.
I am trying to run a CREATE TABLE script which has multiple INDEXES.
CREATE TABLE IF NOT EXISTS Equipment (
EquipmentID BIGINT UNSIGNED UNIQUE NOT NULL AUTO_INCREMENT,
Type VARCHAR(255) NOT NULL,
Make VARCHAR(255),
Model VARCHAR(255),
Description VARCHAR(255),
OperationNotes TEXT,
Damaged BOOLEAN DEFAULT 0,
PRIMARY KEY (EquipmentID),
INDEX ('EquipmentID'),
INDEX('Type'),
INDEX('Model'),
INDEX('Description')
INDEX('Damaged')
);
However I get a syntax error:
"(" is not valid at this position for this server version
On line:
INDEX ('EquipmentID'),
Single quote (') denote string literals. When referring to object names (such as columns), you shouldn't use single quotes. Remove them, and you should be OK. Also, note that a primary key implicitly creates an index, so you don't need to explicitly create an index on EquipmentID:
CREATE TABLE IF NOT EXISTS Equipment (
EquipmentID BIGINT UNSIGNED UNIQUE NOT NULL AUTO_INCREMENT,
Type VARCHAR(255) NOT NULL,
Make VARCHAR(255),
Model VARCHAR(255),
Description VARCHAR(255),
OperationNotes TEXT,
Damaged BOOLEAN DEFAULT 0,
PRIMARY KEY (EquipmentID),
INDEX (Type),
INDEX (Model),
INDEX (Description),
INDEX (Damaged)
);
I am using a MySQL database in my ASP.NET with C# web application. The MySQL Server version is 5.7 and there is 8 GB RAM in the PC. When I am executing the select query in MySQL database table, it takes more time in execution; a simple select query takes around 42 seconds. Across 1 crorerecord (10 million records) in the table. I have also done indexing for the table. How can I fix this?
The following is my table structure.
CREATE TABLE `smstable_read` (
`MessageID` int(11) NOT NULL AUTO_INCREMENT,
`ApplicationID` int(11) DEFAULT NULL,
`Api_userid` int(11) DEFAULT NULL,
`ReturnMessageID` varchar(255) DEFAULT NULL,
`Sequence_Id` int(11) DEFAULT NULL,
`messagetext` longtext,
`adtextid` int(11) DEFAULT NULL,
`mobileno` varchar(255) DEFAULT NULL,
`deliverystatus` int(11) DEFAULT NULL,
`SMSlength` int(11) DEFAULT NULL,
`DOC` varchar(255) DEFAULT NULL,
`DOM` varchar(255) DEFAULT NULL,
`BatchID` int(11) DEFAULT NULL,
`StudentID` int(11) DEFAULT NULL,
`SMSSentTime` varchar(255) DEFAULT NULL,
`SMSDeliveredTime` varchar(255) DEFAULT NULL,
`SMSDeliveredTimeTicks` decimal(28,0) DEFAULT '0',
`SMSSentTimeTicks` decimal(28,0) DEFAULT '0',
`Sent_SMS_Day` int(11) DEFAULT NULL,
`Sent_SMS_Month` int(11) DEFAULT NULL,
`Sent_SMS_Year` int(11) DEFAULT NULL,
`smssent` int(11) DEFAULT '1',
`Batch_Name` varchar(255) DEFAULT NULL,
`User_ID` varchar(255) DEFAULT NULL,
`Year_ID` int(11) DEFAULT NULL,
`Date_Time` varchar(255) DEFAULT NULL,
`IsGroup` double DEFAULT NULL,
`Date_Time_Ticks` decimal(28,0) DEFAULT NULL,
`IsNotificationSent` int(11) DEFAULT NULL,
`Module_Id` double DEFAULT NULL,
`Doc_Batch` decimal(28,0) DEFAULT NULL,
`SMS_Category_ID` int(11) DEFAULT NULL,
`SID` int(11) DEFAULT NULL,
PRIMARY KEY (`MessageID`),
KEY `index2` (`ReturnMessageID`),
KEY `index3` (`mobileno`),
KEY `BatchID` (`BatchID`),
KEY `smssent` (`smssent`),
KEY `deliverystatus` (`deliverystatus`),
KEY `day` (`Sent_SMS_Day`),
KEY `month` (`Sent_SMS_Month`),
KEY `year` (`Sent_SMS_Year`),
KEY `index4` (`ApplicationID`,`SMSSentTimeTicks`),
KEY `smslength` (`SMSlength`),
KEY `studid` (`StudentID`),
KEY `batchid_studid` (`BatchID`,`StudentID`),
KEY `User_ID` (`User_ID`),
KEY `Year_Id` (`Year_ID`),
KEY `IsNotificationSent` (`IsNotificationSent`),
KEY `isgroup` (`IsGroup`),
KEY `SID` (`SID`),
KEY `SMS_Category_ID` (`SMS_Category_ID`),
KEY `SMSSentTimeTicks` (`SMSSentTimeTicks`)
) ENGINE=MyISAM AUTO_INCREMENT=16513292 DEFAULT CHARSET=utf8;
The following is my select query:
SELECT messagetext, SMSSentTime, StudentID, batchid,
User_ID,MessageID,Sent_SMS_Day, Sent_SMS_Month,
Sent_SMS_Year,Module_Id,Year_ID,Doc_Batch
FROM smstable_read
WHERE StudentID=977 AND SID = 8582 AND MessageID>16013282
You need to learn about compound indexes and covering indexes. Read about those things.
Your query is slow because it's doing a half-scan of the table. It uses the primary key to find the first row with a qualifying MessageID, then looks at every row of the table to find matching rows.
Your filter criteria are StudentID = constant, SID = constant AND MessageID > constant. That means you need those three columns, in that order, in an index. The first two filter criteria will random-access your index to the correct place. The third criterion will scan the index starting right after the constant value in your query. It's called an Index Range Scan operation, and it's quite efficient.
ALTER TABLE smstable_read
ADD INDEX StudentSidMessage (StudentId, SID, MessageId);
This compound index should make your query efficient. Notice that in MyISAM, the primary key column of a table should appear in compound indexes. That's cool in this case because it's also part of your query criteria.
If this query is used very frequently, you could make a covering index: you could add the other columns of the query (the ones mentioned in your SELECT clause) to the index.
But, unfortunately you have defined your messageText column with a longtext data type. That allows for each message to contain up to four gigabytes. (Why? Is this really SMS data? There's a limit of 160 bytes per message in SMS. Four gigabytes >> 160 bytes.)
Now the point of a covering index is to allow the query to be satisfied entirely from the index, without referring back to the table. But when you include a longtext or any other LOB column in an index, it only contains a subset of the data. So the point of the covering index is lost.
If I were you I would change my table so messageText was a VARCHAR(255) data type, and then create this covering index:
ALTER TABLE smstable_read
ADD INDEX StudentSidMessage (StudentId, SID, MessageId,
SMSSentTime, batchid,
User_ID, Sent_SMS_Day, Sent_SMS_Month,
Sent_SMS_Year,Module_Id,Year_ID,Doc_Batch,
messageText);
(Notice that you should put variable-length items last in the index if you can.)
If you can't change your application to handle VARCHAR(255) then go with the first index I mentioned.
Pro tip: putting lots of single-column indexes on MySQL tables rarely helps SELECT performance and always harms INSERT and UPDATE performance. You need an index on your primary key, and you need indexes to support the queries you run. Extra indexes are harmful.
It looks like your database is not properly indexed and even not properly normalized. Normalizing your database will go a long way to speed up all your queries. Particularly in view of the fact that mysql used only one index per table in a query. Even though you have lot's of indexes, they cannot be used.
Your current query filters on StudentID,SID, and MessageID. The last is an inequality comparision so an index will not be very effective with that but the other two columns are equality comparisons. I suggest an index like this:
KEY `studid` (`StudentID`,`SID`)
Follow that up by dropping your existing index on SID. If you find that you don't want to drop it because it's used in another query, further evidence that your table is in desperate need of normalization.
Too many indexes slow down inserts and adds a little overhead to each SELECT because the query planner needs more effort to figure out which index to use.
I am working with mysql .
I have checked the CREATE table statement , and I saw there a KEY word
| pickupspc | CREATE TABLE `pickupspc` (
`McId` int(11) NOT NULL,
`Slot` int(11) NOT NULL,
`FromTime` datetime NOT NULL,
`ToTime` datetime NOT NULL,
`Head` int(11) NOT NULL,
`Nozzle` int(11) DEFAULT NULL,
`FeederID` int(11) DEFAULT NULL,
`CompName` varchar(64) DEFAULT NULL,
`CompID` varchar(32) DEFAULT NULL,
`PickUps` int(11) DEFAULT NULL,
`Errors` int(11) DEFAULT NULL,
`ErrorCode` varchar(32) DEFAULT NULL,
KEY `ndx_PickupSPC` (`McId`,`Slot`,`FromTime`,`ToTime`,`Head`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
But what is the meaning of it ?
It's not like a PRIMARY KEY right ?
Thanks .
It is simply a synonym for INDEX. It creates an index with the name ndx_PickupSPC on the columns specified in parenthesis.
See the CREATE TABLE syntax for more information.
It's just a non-unique index. From the manual
KEY is normally a synonym for INDEX. The key attribute PRIMARY KEY can
also be specified as just KEY when given in a column definition. This
was implemented for compatibility with other database systems.
Key and index are the same. The word Key in the table creation is used to create an index, which enables faster performance.
In the above code, Key ndx_PickupSPC means that it is creating an index by the name ndx_PickupSPC on the columns mentioned in parenthesis.
It's an INDEX on the table. Indexes enable fast lookups for specific queries which check the values of the columns the index is built on. The example uses a compound key.
They are a bit similar to the indexes you find at the end of the books. You can quickly find an entry with the index without searching through the whole book. Databases typically use B-Trees for indexes.
I am using a VARCHAR as my primary key. I want to auto increment it (base 62, lower/upper case, numbers), However, the below code fails (for obvious reasons):
CREATE TABLE IF NOT EXISTS `campaign` (
`account_id` BIGINT(20) NOT NULL,
`type` SMALLINT(5) NOT NULL,
`id` VARCHAR(16) NOT NULL AUTO_INCREMENT PRIMARY KEY
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
however, this works:
CREATE TABLE IF NOT EXISTS `campaign` (
`account_id` BIGINT(20) NOT NULL,
`type` SMALLINT(5) NOT NULL,
`id` VARCHAR(16) NOT NULL PRIMARY KEY
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
What is the best way to keep track of incrementation of 'id' myself? (Since auto_increment doesn't work). Do i need to make another table that contains the current iteration of ID? Or is there a better way to do this?
EDIT: I want to clarify that I know that using INT is a auto_increment primary key is the logical way to go. This question is in response to some previous dialogue I saw. Thanks
you have to use an INT field
and translate it to whatever format you want at select time
example of a solution to your problem:
create a file with a unique number and then increment with a function.
the filename can be the prefix and the file binary content represent a number.
when you need a new id to the reg invoque the function
Example
String generateID(string A_PREFIX){
int id_value = parsetoInt(readFile(A_PREFIX).getLine())
int return_id_value = id_value++
return return_id_value
}
where "A_PREFIX-" is the file name wich you use to generate the id for the field.
Or just create a sequence and maintain the pk field using the sequence to generate the primary key value with nextval function. And if perf is an issue, use cache on sequence.
But as others have stated, this is sub-optimal, if your primary key contains a numbered sequence then it's better to use int and auto-increment.
I don't see a use case where pk has to auto-increment but be a varchar data type, it doesn't make sense.
Assuming that for reasons external to the database, you do need that varchar column, and it needs to autoIncrement, then how about creating a trigger that grabs the existing autoIncrement value and uses Convert() to convert that value into a VarChar, dropping the VarChar into the field of interest. As mentioned in a previous answer, you could concatenate the table-name with the new varChar value, if there is some advantage to that.