I have around 2000 rows in my database at the moment, the structure previously was varchar(200). I have since changed the structure to varbinary.
Now when I insert data into my table I use
AES_ENCRYPT('Obama', 'sadhjksauejs') (just an example)
anyway, I want to use AES_ENCRYPT on all the data that is currently in the database, so get the data, encrypt it and put it back into the database, without losing the original data.
what is the best way to do AES_ENCRYPT on all the data that is currently there?
First you have to create new field in your table encryptedText.
UPDATE table SET encryptedText = AES_ENCRYPT(textField,'sadhjksauejs');
There are a couple of things you need to look out for here ...
Yes, VARBINARY is the appropriate data type, however, AES_ENCRYPT is a block-based and will pad your plaintext to the necessary length so your ciphertext will likely be longer than the original. The documentation gives this for calculating the correct column size:
16 * (trunc(string_length / 16) + 1)
You should check that the column is still going to be long enough to hold any value you want to store.
Second, as you are already adding new records in encrypted form you need to make sure that your update statement is restricted to only those records that are still in plaintext.
Once you have taken those things into consideration (and assuming some id or created column) your UPDATE statement will look something like this
UPDATE `yr_table` SET `col1` = AES_ENCRYPT(`col1`,'sadhjksauejs'),
`col2` = AES_ENCRYPT(`col2`,'sadhjksauejs')
WHERE `id` > whatever;
(i.e. Not unlike the suggestion by #Sadikhasan)
Related
I have a Mysql8 DB which contains JSON data. Unfortunately, the content is not always the same. To make it simple, the hierarchy is always the same, but sometimes part of the "tree" is missing or slightly different. For instance:
$.bilan.victimes.*.preview.TAGSAU (I use a star, since sometimes, it's '1', '2', etc... and sometimes it is only '$.bilan.victimes' (without further subkeys)
Now, I am using queries to lookup information in the JSON like:
SELECT
COUNT(fiche_id) AS USAGE_DSA,
JSON_VALUE(content, '$.bilan.victimes.*.preview.DSA') AS DSA
FROM bilan_json
WHERE STR_TO_DATE(JSON_VALUE(content, '$.bilan.victimes.*.preview.TAGSAU'),'%e/%c/%Y %H%#%i') >= '2021-01-01'
GROUP BY DSA;
This is working fine, but since there is a lot of records, and JSON could be very long, it takes an awful bunch of time to display the result. In this example, this is only key... I am supposed to retrieve multiples values from the JSON, sometimes in a single query.
I've read about virtual columns (https://stackoverflow.com/questions/68118107/how-to-create-a-virtual-column-to-index-json-column-in-mysql#:~:text=if%20table%20is%20already%20created%20and%20you%20want,%60jval%60%3B%20Dont%20forget%20to%20index%20the%20Generated%20Columns) and also improving performance for JSON object (https://blogit.create.pt/goncalomelo/2018/12/20/query-performance-for-json-objects-inside-sql-server/) but I can't really figure out if I should create a virtual column per key ? And, how can I create a virtual column with a transform ? In above case, I would create something like :
ALTER TABLE bilan_json
ADD COLUMN tagsau DATETIME
GENERATED ALWAYS AS STR_TO_DATE(JSON_VALUE(content, '$.bilan.victimes.*.preview.TAGSAU'),'%e/%c/%Y %H%#%i')
AFTER content;
What would be your advice ?
Simply put, If you expect to need a field in JSON for a WHERE or ORDER BY clause, that field should be in its own column.
3 approaches:
Redundantly store it in a column as you INSERT the rows.
Use a Virtual ("Generated") column (as you suggest).
Remove it from JSON as you put it in its own column.
Once it is in a column, it can be indexed. (It is unclear how useful an index would be for the SELECT you show.)
Did you try that ALTER? Did it work? We need SHOW CREATE TABLE in order to advise further.
How do I generate a range of numbers in one column in MySQL? I'm looking for any soluton to make numbers range that starts from 500000000 and ends on 889999999.
It seems that may want to use an AUTO_INCREMENT in the column value you want. You can set the starting value to the one you desire in this way.
Also, you can only have one AUTO_INCREMENT column in a given table.
CREATE TABLE your_table (
column_1 INT NOT NULL AUTO_INCREMENT = 500000000
--Add other columns
)
If you already have a table with the AUTO_INCREMENT column, just set the value to the one you want.
ALTER TABLE your_table AUTO_INCREMENT = 500000000;
If what you want is to insert rows with those numbers, use a loop.
Just for fun, generate the range in a text file, by any means available.
Unload to a text file.
Load that text file to your table. You don not say if you are constrained by how long this takes. It sounds like you just want a table of a single column of INT with lower and upper limits.
MySQL should just handle these numbers,this is not really a "big" range, seriously.
Do you want to constrain the values in the column to
{500000000..889999999}?
Or do you want to know how to define a column
to hold these values?
Do you want a written procedure to generate
these numbers for you?
Do you want us to size this for you?
Do you want us to write a script or program to load these?
All of these answers are available with minimal sweat. Keywords are MySQL,Integer, Types.
We cannot see your problem because your question does not describe a problem.
Tell us what you tried, and tell us what happened...
Otherwise just add them, you are still in INT territory (-2Gi..2Gi), not BIGINT yet.
Switch to MariaDB, then JOIN to a pseudo-table called seq_500000000_to_889999999.
I have a mysql table column (runs) which holds an integer, I want to increment it by another dynamic value (1,2,3,4,5) using an update query on click of a button. How should I do that?
(Without retrieving the original value from that table column, is there any direct way to increment the value, not like AUTO_INCREMENT since it does it using a static value)
I've checked the documentation on MySql here and it seems you can do it with an update query like this:
UPDATE t1 SET col1 = col1 + 1;
where you can change 1 to any numeric value as long as col1 is also numeric.
MySql will get the initial value of col1 and add your number to it.
Even if MySql can do this, I'm guessing this also has an impact on speed, it might be faster then 2 queries, but probably needs testing to determin the differences.
When confronted with software speeds issues saving a select query for only one value it's not the key modification that you can do to optimize your app. As Digvijay Yadav suggested, you should also check other solutions, and other optimizations on to make your app faster.
another example on that page that also shows that what your are trying to do can work is :
UPDATE t SET id = id + 1 ORDER BY id DESC;
please try it and let me know if it doesn't work.
If DB interaction is the main problem then in this case I would suggest not to read/write to DB on every user click. Instead you should retrieve the value once and store them in a DS (may be an array or list). On every click you perform operations on this list and after sometime you update the DB with updated values.
Also, if you are creating and destroying the DB connections on every read/write operation then I would not suggest that. A better solution IMO is to create a DB connection when your webapp starts up and store the connection object in the ServletContext as an attribute as you are using servlets and JSPs.
Further if your web app is too big and has thousand of users then there are several caching techniques for improving the performance of DB.
I am new to MySQL partitioning, therefore any example will be appreciated.
I am trying to create a sort of an ageing mechanism for a data that is distributed between several MyISAM tables.
My question will actually include several sub-questions.
The relevant tables are:
First table contains raw data with high input frequency (next to each record there is an auto incremented id).
Second table contains processed results, there is a result record per every raw data record (result record contains the source id record of the auto incremented field of raw data record)
Questions:
I need to be able to partition the raw data table and result data table similarly so that both of them will include only 10 weeks of data in single partition (each raw data record contains unixtimestamp field), how do i do it , can someone write small example case for two such tables?.
I want to be able to change the 10 weeks constraint on the fly.
I want that when ever the current partition will be filled or a new partition is created , the previous (10 weeks before) partition will be deleted automatically.
I don't want the auto increment id integer to be overflown, as much as i understand the ids are unique for the partition only, so if i am not wrong the auto increment id will start from zero for the next partition? but what if the previous partition still exist, will i have 2 duplicated ids , how i know to reference only for the last id when i present a result record?
I want to load raw data using LOAD DATA INTO... instead of multiple inserts , is MySQL partitioning functionality affected?
And the last question, would you suggest some other approach to implement aging mechanism (i am writing Java implementation product that processes around 1 GB or raw data per day and stores the results in MySQL)
It's hard to give a real answer on this question since it depends on your data. But let me give you some things to think about.
I assume we're talking about some kind of logs with recent data (so not spanning multiple years). You can partition by range. You could add one field to your table with the year/week number (ie 201201, 201202, etc). If this question is related to your question about importing into multiple tables, you can easily do this is that import script.
On the fly as in, repartition your data on the fly (70GB?). I would not recommend it. But you could do it if you had the weeknumber in there. If you later want to change it to 12 days, you could add a column for the date and partition by that.
Well it won't be deleted automatically but a cron job can handle that right? Just check how many partitions there are, and if there are 3(?) delete the first one.
The partition needs to have a primary index on the field that you partition (if you want to use auto increment). Therefor you can never fully rely on the auto increment id alone. I don't see a way around this.
I'm not sure what you mean.
If your data is just some logs in chronological order then you might just use separate tables for each period. Then before you start the new period (at 00:00) check the last id of the last table, create a new table and set the auto increment to that value +1. Then your import will decide when a new period will begin so it can be easily changed. Your import script can use a small table in where it can store the next period.
LOAD DATA is really quite fast. I would just have two steps(in no partic order) - LOAD DATA and then 'delete .. where date < 10 weeks'. Autoincrement will go on for as long as the datatype you're using. If you wanted to be super careful you could push it back to zero periodically.
Once the data is in the 'raw' table run your routine to create the 'processed' table. We use a v similar process where I work. We keep a separate table that has 'write' and 'parse' pointers to all of our 'raw' tables. As new data comes in and gets parsed the appropriate row pointers get set. If the 'raw' table gets truncated you can reset the 'write' pointer but leave the 'parse' pointer. (we store the offset in another table when this happens - just to be sure).
And if I recommend , creating the index column for each of the related columns can also enhanced the performance Delete old data from multiple related tables since we have just compared the index numbers rather than strings.
I wonder if your tables are being sorted or not.
I have a field called 'click_target' that stores a string of data similar to this:
http://domain.com/deals/244?utm_source=sendgrid.com&utm_medium=email&utm_campaign=website
Using a MySQL query, is it possible to pull the ID (244) from the string and use it to join another table?
You can certainly play games with expressions to pull the ID out of this string but I have a bigger worry - you're burning a dependency on the URL format into a query in the database. That's not really a good idea becuase when (I don't say IF) the URL's change your queries will suddenly fail silently - no errors, just empty (if you're lucky) or nonsensical results.
It's an ugly hack, but I believe this line will extract your ID for you in the above url.
REVERSE(LEFT(LOCATE('/', REVERSE(LEFT(click_target, LOCATE('?', click_target)-1)))-1))
Basically, I am getting the text between the first '?' and the last '/'. From there you can join on whatever table you want with that value, though I recommend aliasing it or storing it in a variable so that it is not recalculated frequently.
If you need the id, fix your database to store it porperly in a separate field.