Is there any built in metadata in mysql that I could query to find out when a row was created if I haven't implemented that myself?
Thanks in advance.
No there is no such way by which you can detect that. You have to create a timestamp field which will store the value of the row created.
The reason why MYSQL does not keep a track of the timestamp of the row in the metadata is because that would increase the general size of the table by 4 bytes per row.
Related
I am working on NIFI Data Flow where my usecase is fetch mysql table data and put into hdfs/local file system.
I have built a data flow pipeline where i used querydatabaseTable processor ------ ConvertRecord --- putFile processor.
My Table Schema ---> id,name,city,Created_date
I am able to receive files in destination even when i am inserting new records in table
But, but ....
When i am updating exsiting rows then processor is not fetching those records looks like it has some limitation.
My Question is ,How to handle this scenario? either by any other processor or need to update some property.
PLease someone help
#Bryan Bende
QueryDatabaseTable Processor needs to be informed which columns it can use to identify new data.
A serial id or created timestamp is not sufficient.
From the documentation:
Maximum-value Columns:
A comma-separated list of column names. The processor will keep track of the maximum value for each column that has been returned since the processor started running. Using multiple columns implies an order to the column list, and each column's values are expected to increase more slowly than the previous columns' values. Thus, using multiple columns implies a hierarchical structure of columns, which is usually used for partitioning tables. This processor can be used to retrieve only those rows that have been added/updated since the last retrieval. Note that some JDBC types such as bit/boolean are not conducive to maintaining maximum value, so columns of these types should not be listed in this property, and will result in error(s) during processing. If no columns are provided, all rows from the table will be considered, which could have a performance impact. NOTE: It is important to use consistent max-value column names for a given table for incremental fetch to work properly.
Judging be the table scheme, there is no sql-way of telling whether data was updated.
There are many ways to solve this. In your case, the easiest thing to do might be to rename column created to modified and set to now() on updates
or to work with a second timestamp column.
So for instance
| stamp_updated | timestamp | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
is the new column added. In the processor you use the stamp_updated column to identify new data
Don't forget to set Maximum-value Columns to those columns.
So what I am basically saying is:
If you cannot tell that it is a new record in sql yourself, nifi cannot either.
I have a base data table on SQL Server 2012 that is replaced every night with a new data set with identical columns. Once the data table is replaced, how do I flag a column whose row values have changed since the last data upload? The data type of the columns in questions are VARCHAR(255) and there can be multiple columns that require such a flag
The benefit I see in trying to implement the above solution is when I have to export data, I can reduce resource utilization by only pulling those rows that have a ISNOTNULL Column change flag
Thanks for your help!
Is there any way to get a datetime when the row was created? Is it stored anywhere? I just forgot to add a column which tells when the row was inserted, and lots of rows have been inserted and I don't know when they were inserted... Thanks.
Unfortunately there's no way to achieve that. MySQL doesn't store a timestamp of row creation, that's why you always have to create a column to do that, along with ON UPDATE clause which will update the column every time the row is updated
Nope. You need a column to record that, it's not stored in the system otherwise.
There isn't a way to get timestamp when the row was created unless you add a field that is populated with system time or has a default value.
My question is very similar to the one below, but on a informatica environment:
Retrieving the index of an inserted row
Here is a brief summary of the issue: I'm trying to figure out how I can insert a row into a table and then find out what the value of the auto_incremented id column was set to so that I can insert additional data into another table. Our target is SQL server 2008. We have a table which has to be populated by informatica ETLs and the application is also using the same table - so, we can't use informatica sequence generator.
In the past when I have used Oracle database, there was a Oracle sequence generator transformation available in Informatica - but for SQL server, I am not sure.
Any solutions please?
If the values to be populated are pure sequence values and have no other meaning you can use two sequence generators simultaneously. Use an Informatica sequence generator that generates values from -1 to negative infinity. At the same time the SQLserver auto-increment field will contain values from 1 to infinity. There would never be a collision.
You could use a sequence generator with the 'Reset' flag enabled so that it begins with 1 each session run, then use a Lookup to cache the current max sequence value from the target table. Then you can predict the sequence number that SQL Server will generate when a record is inserted by adding NEXTVAL to the max value.
Populate the first table (the table with auto incremented id) first. After that is done, run another mapping where you do a lookup to that table, get the ID value by some other identifiable key value in the table, and proceed to populate other tables (using the retrieved ID) as you wish.
I am using mysql. Now I want to write that query in which i will give mysql the row number then mysql will data from that column no only.
means..
if i have twenty rows in database but i want to get only 10th no of rows data? so how to write that kind query?
Create an INT field named "ID" that is set to "Auto Increment." This way each row will have a unique ID that you can use to select it. Details about how to do this are located here.
Add a column with the AUTO_INCREMENT flag on it, so that this columns contains the row number for all your rows. MySQL does not really have support for fetching a row by number, only by column value.