Can someone explain to me why you can't connect to a MySQL DB directly through dart from a security point of view?
There is no hard guideline on whether to connect frontend directly to backend or not. It is just a design practice that has been widely accepted and evolved over many years.
Typical app structure consists of
FRONTEND -> SOME MIDDLE LAYER -> BACKEND
Where your middle layer handles all the interactions/processing with the database and the frontend uses this functionality with some sort of API structure. Having this layer is extremely helpful when the application goes to scale, it gives an added abstraction to the frontend.
It is not advisable to directly fuse your frontend(your flutter app), to the DB(MySQL) because any efficient hacker might use basic man-in-middle attack to know your DB structure/connections/queries(There are some pretty effective decompilers present out there), and alter your data and you might not even get to know what caused the data to update unless you've applied some checks on DB layer.
Also, your frontend logic needs to be more of end-user centric than to handle the data of the user. Any backend system(java, node, etc) gives you added functionality & freedom to parse and present the data from either side.
You can use the sqlite package available to store basic data, like your session tokens, your app configurations etc, but it is advisable to keep the main user data like the logins, etc in a separate place, or better yet, you can use the firebase plugin to store data in document structure in the cloud.
I am coming from object relation database background, I understand Couchbase is schema-less, but data migration will still happen as the application develop.
In SQL we have management tool to alter table, or I can write migration script with SQL to do migration from version 1 table to version 2 table.
But in document, say we have json Document UserProfile:
UserProfile
{
"Owner": "Rich guy!",
"Car": "Cool car"
}
We might want to add a last visit field there, allow user have multiple car, so the new updated document will become follows:
UserProfile
{
"Owner": "Rich guy!",
"Car": ["Cool car", "Another car"],
"LastVisit": "2015-09-29"
}
But for easier maintenance, I want all other UserProfile documents to follow the same format, having "Car" field as an array.
From my experience in SQL, I could write migration script which support migrating different version of table. Migrate from version 1 table to version 2...N table.
So how can I should I write such migration code? I will have to really just writing an app (executable) using Couchbase SDK to migrate all the documents each time?
What will be the good way for doing migration like this?
Essentially, your problem breaks down into two parts:
Finding all the documents that need to be updated.
Retrieving and updating said documents.
You can do this in one of two ways: using a view that gives you the document ids, or using a DCP stream to get all the documents from the bucket. The view only gives you the ids of the documents, so you basically iterate over all the ids, and then retrieve, update and store each one using regular key-value methods. The DCP protocol, on the other hand, gives you the actual documents.
The advantage of using a view is that it's very simple to implement, works with any language SDK, and it lets you write your own logic around the process to make it more robust and safe. The disadvantage is having to build a view just for this, and also that if the data keeps changing, you must retrieve the ENTIRE view result at once, because if you try to page over the view with offsets, the ordering of results can change, thus giving you an inconsistent snapshot of the data.
The advantage of using DCP to stream all documents is that you're guaranteed to get a consistent snapshot of your data even if it's constantly changing, and also that you get the whole document directly as part of the stream, so you don't need to retrieve it separately - just update and store back to the database. The disadvantage is that it's currently only implemented in the Java SDK and is considered an experimental feature. See this blog for a simple implementation.
The third - and most convenient for an SQL user - way to do this is through the N1QL query language that's introduced in Couchbase 4. It has the same data manipulation commands as you would expect in SQL, so you could basically issue a command along the lines of UPDATE myBucket SET prop = {'field': 'value'} WHERE condition = 'something'. The advantage of this is pretty clear: it both finds and updates the documents all at once, without writing a single line of program code. The disadvantage is that the DML commands are considered "beta" in the 4.0 release of Couchbase, and that if the data set is too large, then it might not actually work due to timing out at some point. And of course, that fact that you need Couchbase 4.0 in the first place.
I don't know of any official tool currently to help with data model migrations, but there are some helpful code snippets depending on the SDK you use (see e.g. bulk updates in java).
For now you will have to write your own script. The basic process is as follow:
Make sure all your documents have a model_version attribute that you increment after each migration.
Before a migration update your application code so it can handle both the old and the new model_version, and so that new documents are written in the new model.
Write a script that iterate through all the old model documents in your bucket(you need a view that emits the document key), make the update you want, increment model_version and save the document back.
In a high concurrency environment it's important to have good error handling and monitoring, you could have for example a view that counts how many documents are in each model_version.
You can use Couchmove, which is a java migration tool working like Flyway DB.
You can execute N1QL queries with this tool to migrate your documents and keep tracking of your changes.
If I understood correctly, the crux here is getting and then 'update every CB docs'. This can be done with a view, provided that you understand that views are only 'eventually consistent' (unlike read/write actions which are strongly consistent).
If (at migration-time) no new documents are added to your bucket, then your view would be up-to-date and should return the entire set of documents to be migrated. easy.
On the other hand, if new documents continue to be written into your bucket, and these documents need to be migrated, then you will have to run your migration code continually to catch all these new docs (since the view wont return them until it is updated, a few seconds later).
In this 2nd scenario, while migration is happening, your bucket will contain a heterogeneous collection of docs: some that have been migrated already, some that are about to be migrated and some that your view has not 'seen' yet (because they were recently added) and would only be migrated once you re-run the migration code.
To make the migration process efficient, you'll need to find a way to differentiate between already-migrated items and yet-to-be-migrated items. You can add a field to each doc with its 'version number' and update it during the migration. Your view should be defined to only select documents with older 'version number' and ignore already-migrated items.
I suggest you read more about couchbase views - here and on their site.
Regarding your migration: There are two aspects here: (1) getting the list of document ids that need to be updated and (2) the actual update.
The actual update is simple: you retrieve the doc and save it again with the new format. There's no explicit schema. Where once you added column in SQL and populated it, you now just add a field in the json-doc (of all the docs). All migrated docs should have this field. Side note: Things get little more complicated if (while you're migrating) the document can be updated by another process. This requires special handling (read aboud CAS if that's the case).
Getting all the relevant doc-keys requires that you define a view and query it. Its beyond the scope of this answer (and is very well documented). Once you have all the keys, you simply iterate them one by one and update them.
With N1QL, Couchbase provides the same schema migration capabilities as you have in RDBMS or object-relational database. For the example in your question, you can place the following query in a migration script:
UPDATE UserProfile
SET Car = TO_ARRAY(Car),
LastVisit = NOW_STR();
This will migrate all the documents in your bucket to your new schema. Note that update statements in Couchbase provide document-level atomicity, not statement-level atomicity. But since this update is idempotent (repeatable), you can run it multiple times if you run into errors. Note: similar to the last paragraph of David's answer above.
I'm writing an intranet application (in a LAMP environment) that uses data from sections of an MSSQL 2012 database (used by another much larger application).
As I see it my options are to:
Directly query the database from the application.
Create a web service
Use Microsoft SQL Server Integration Services to have the data
automatically integrated into my applications database
I'm sure the best solution here would be using SSIS, however I've not done this before am on a deadline - so if that's the case could someone let me know
a) With my limited experience in that area would I be able to set that up, and
b) What are the pros and cons of the above options?
Any other suggestions outside of the options I've thought of would also be appreciated
Options:
Directly query the database from the application.
Upside:
Never any stale data
Downside:
Your application now contains specific code and is tied that
application
If you are in the common situation where the business
buys another application containing the same master data, you now
need special code to connect to two applications
Vendor might not like it
Might be performance impacts on source application
Use Windows Task Scheduler / SQL Agent to run a script or SSIS to replicate data at x minute intervals or so.
Upside:
Your application is only tied to your local copy of the database, which you can customise as required. If your source app gets moved to the cloud or something then you don't need to make application changes, just integration changes
If another source application appears with the same type of master data, you can now replicate that into your local DB rather than making application changes to connect to 2 databases.
Downside:
Possibility of stale data
Even worse: possibility of stale data without users realising it, with subsequent loss of confidence in the application
Another component to maintain
If you write a batch script, .Net app or SSIS, they are all pieces of logic that needs to be scheduled to run
Another option is to replicate the database using differential replication if your source database is Oracle or SQL, you can use replication to replicate it into another database.
You need to consider where you will be in a few years. The data copy method probably gives you more flexibility to adapt to changes in the source system as you only need to change your integration, not your whole app if something drastic changes with your source system.
You also need to consider: will you ever be asked to propogate changes back the other way, i.e. update data in your local copy and have it pushed back to the source systems.
I'm about to port an Android-Travellog App to other Plattforms using Sencha Touch.
The Problem is, that Sencha only has a Store System to store Data, but doesnt appear to have a possibilty to acctually make MySql queries.
And since most of the Mysql code in my previous app is already there, id would be quite a pain to redo everything with Senchas new System.
Is there a possibilty to use mysql (or any other sql) queries with Sencha to Store Data on the Phone?
Sencha stores and proxies abstract away the need to write raw query code. A store can use one of a number of different proxies for interfacing with different back-end data stores, one of which is the SQL proxy, which as you can see in the source code provides an API for basic data querying WebSQL databases.
If you want to gain the full benefit of the framework and do things the "Sencha way" you'll probably want to start from scratch and architect your app to use the stores API.
My web app uses mysql to store contact data. I'd like to sync this data via carddav with mobile devices. I understand carddav is based on a file system, not a database. What software is available to act as an interface or wrapper to make the carddav server work with mysql? or other relational database?
You might want to take a look at Bedework.
Baikal just added this feature!!!
Most dav servers are file system based. If you use SabreDav you can build a virtual filesystem based on your own backend. Baikal is a project that uses sabredav, and a virtual file system. Until recently it stored its data in sqllite. Now it supports both mysql and sqlite.
Its still not 100% mature, but its a great starting point. Playing around with it, I have been able to create contacts directly in the DB (by uploading vcard blobs to a table) and then having them show on my ipad addressbook.
After evaluating many systems, ones built on sabredav like baikal tend to be the simplest to build on. Fruxx is something else you may also check out. Its a hosted system, but will soon have an api.
Last if you are looking for a very elaborate system, then take a look at tine20. It supports activesync (illegally in the usa), carddav, caldav, and has a decent extjs web ui. It natively stores contact information in its mysql store, which is nice since you can update a contact through a sql statement without having to build a vcf file. Where tine doesnt make sense is that it uses a bit more resources because of all the features it offers, and the complexity has ensured that it has a VERY complicated database schema. In other words, you are probably better off creating a rest api on the tine source code rather than doing bare sql inserts.
http://baikal-server.com/