In the perspective of applying "custom retention" by doing some manual delete transaction on a dataset (Is it possible to apply manual retention on a dataset in Palantir Foundry?),
Is the "allow_retention" option in incremental decorator of the downstream transform enough to avoid triggering a snapshot on them? (Or is it only allowing delete transaction by the retention service?)
The 'allow_retention' option only prevents snapshot transactions when delete transactions are made by the retention service.
Related
Here is a rather common case for elasticsearch(ES): as ES is near realtime, we want to sync it with our MySQL as much close to real time as possible.
The initial tables are imported into index and it is working fine, but now we need to track inserting, updating and deleting new records and here is the place we have a question to. Import is done by elasticsearch-jdbc importer. It can have an interval parameter to periodically poll data, but:
we cannot add fields to db which will be responsible for fetching new/updated rows,
we want to have more atomic operations like DB insert row->ES insert document, DB update row->ES update document and so on.
Two options which might meet our needs:
Logstash with logstash-input-jdbc plugin
Databus for MySQL from LinkedIn
Other solutions are welcome!
The Logstash with logstash-input-jdbc plugin just can sync insert and update operation.
Bug delete operation can't be sync directly.
We talked more about How to sync Mysql delete operation with logstash-input-jdbc? in https://discuss.elastic.co/t/delete-elasticsearch-document-with-logstash-jdbc-input/47490/9.
I am using Microsoft SQL Server 2008 R2.I have copied database A(myproduction database) to database B(Myreportin database) by creating SSIS package.Both databases are in same server.I want to run a job so that If any change(data modifications like inserting new rows or updating values of any row in any table) take place in database A that will also take place in my B database and sql job will run and acomplish the changing automatically.I don't want that in database B table will be dropped and recreated (as its not our business rule )instead only the change will take place.
Can any one help me please.Thanks in Advance.
I would suggest that you investigate using replication. Specifically, transactional replication if you need constant updates. Here's a bit from MSDN:
Transactional replication typically starts with a snapshot of the publication database objects and data. As soon as the initial snapshot is taken, subsequent data changes and schema modifications made at the Publisher are usually delivered to the Subscriber as they occur (in near real time). The data changes are applied to the Subscriber in the same order and within the same transaction boundaries as they occurred at the Publisher; therefore, within a publication, transactional consistency is guaranteed.
If you don't need constant updating (that comes at a price in performance, of course), you can consider the alternatives of merge replication or snapshot replication. Here's a page to start examining those alternatives.
I have a Django app where the default "REPEATABLE READ" transaction isolation level in InnoDB is causing different processes to have different views of the data than that current in the database.
e.g. Process 1 has made a change but Process 2 isn't seeing it.
I don't need transactional integrity in the app; can I just turn off transactions altogether so that all processes doing a SELECT see the same data?
Any downside to doing this?
Is this what is meant by "READ UNCOMMITTED"?
Any pointers welcome
Rachel
I'd suggest that you just convert the InnoDB tables to myISAM. If your criteria is speed, you are wasting alot of potential by using a transaction oriented table type (InnoDB) and just disabling transactions. You would gain alot if you just converted the tables to myISAM. It's designed with lack of transactions in mind, while still being able to lock changes (i.e. table locks).
A clean
ALTER TABLE table_name ENGINE = MyISAM;
can do the trick for a single table, dumping, changing type and loading the table does the trick as well.
Autocommit is on by default in InnoDB. Transactions are still used for updates (which is necessary), but they are committed immediately after each statement.
The READ UNCOMMITTED isolation level allows a transaction to read rows which have been written by other transactions but haven't yet been committed. This point is irrelevant however if you're not explicitly using transactions and autocommit is on.
Unfortunately, I'm not too familiar with Django, but from the documentation I see:
How to globally deactivate transaction management
Control freaks can totally disable all transaction management by setting
DISABLE_TRANSACTION_MANAGEMENT to True in the Django settings file.
Hope that helps.
my question is about a database history or transaction log table which is currently updated by mysql procedure. The procedure is called by mysql trigger every time when we keep a history of an appropriate table in during insert, update or delete actions. As far as we have lots of tables for each of them we need to create a separate trigger e.g. for "accounts table" we need to create "accounts_insert, accounts_update and accounts_delete" triggers.
The problem is every time when we alter "accounts table" we have to modify appropriate triggers as well.
Is there any way to avoid that manual work? Would it be better to implement it in application layer/code?
There are no 'global' triggers if that's what you're thinking about.
Application side logging is one possible solution. You'll want to do this within transactions whenever possible.
Other possible approaches:
Create a script that will update your triggers for you. Can be fairly easy, if your triggers are generally similar to each other. Using information_schema database can be helpful here.
Parse general query log (careful, enabling this log can have large negative impact on server's performance)
Is there a way that if there's a change in records, that a query that changed the data (update, delete, insert) can be added to a "history" table transparently?
For example, if mySQL detects a change in a record or set of records, is there a way for mySQL to add that query statement into a separate table so that way, we can track the changes? That would make "rollback" possible since every query (other than SELECT) would be able to reconstruct database from its first row. Right?
I use PHP to interact with mySQL.
You need to enable the MySQL BinLog. This automatically logs all the alteration statements to a binary log which can be replied as needed.
The alternative is to use an auditing function through Triggers
Read about transaction logging in MySQL. This is built in to MySQL.
MySQL has logging functionality that can be used to log all queries. I usually leave this turned off since these logs can grow very rapidly, but it is useful to turn on when debugging.
If you are looking to track changes to records so that you can "roll back" a sequence of queries if some error condition presents itself, then you may want to look into MySQL's native support of transactions.