I'm able to create a new service which creates a table in the MySQL DB just fine, but the feathersJS model file simply creates 1 text field in the model by default,
when I modify the fields add more etc...this does not reflect in the database.
Is there a migration script i must run ? can someone show me how to run a command that will reflect the changes i make to the model so I can sync the two.
right now I have to manually modify the table to match the model.
In your src/sequelize.(js|ts), there should be a call like sequelize.sync(). This updates the structure of the database according to the model definitions every time the app gets started. So in theory, restarting the app should be enough.
If that doesn't work, try .sync({ alter: true }) or even .sync({ force: true }). (This might be fine during initial development, but is not recommended for production use as it can result in data loss. See also the docs about .sync() options.)
You already mentioned Migrations, which would be a better alternative for sure. With Umzug they can be automatically executed during startup as well.
Related
I've created a feeder script based on: https://github.com/jprante/elasticsearch-jdbc/wiki/jdbc-plugin-feeder-mode-as-an-alternative-to-the-deprecated-elasticsearch-river-api
I've not added any new attributes to the above example (ofcourse changed the DB settings etc). The table is indexed quite rapidly and then the feeder script terminates gracefully. What do i need to do make the feeder script persistent and get the updates from mysql db in (near) real time as and when a new record is added to the db?
Thank you
UPDATE:
The elastic search version is 1.4.4 with plugins river and head installed.
You have several options to get a near realtime behavior, but it comes with some cost.
Trigger index refresh explicitly
Could be done after all data has been written.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html
Configure index refresh time
You can set the index refresh tie to a short time span.
{
"settings": {
"refresh_interval": "1s"
}
}
For more information see: https://www.elastic.co/guide/en/elasticsearch/guide/current/near-real-time.html
Thanks all for the interest.
I finally decided to create the document both in DB and in ES. I'm refreshing the index before i get any value in the new doc (after the doc creation in the index). I know this is not an "ideal" situation but fixes my problem for now.
I need to setup demo sites quickly with demo data, including hourly reset of data for a public demo site. Since our data uses timestamps relative to "now" (e.g. archived_timestamp), we cannot just restore an sql dump with fixed timestamps.
My idea is to use Yii2 migrations for that task with PHP code generating timestamps and inserting the demo data.
How to achieve that?
Are Yii2 migrations the right tool for that?
Is it recommended to store the migration file in a seperate subdirectory that our demo setup does not interfere with ordinary "migrate/up" and "migrate/down" processes?
Is this migration bound to a file naming scheme or can this be e.g. demo-data-setup.php ?
Are Yii2 migrations the right tool for that?
Could be if you need a proper sequence of sql command and instruction for create and populate a specific set of table and data you can use the funtcion up for creation an popluation and the function down for drop delete (or delete) what you need.
I*s it recommended to store the migration file in a seperate subdirectory that our demo setup does not interfere with ordinary "migrate/up" and "migrate/down" processes? Of course
Is this migration bound to a file naming scheme or can this be e.g. demo-data-setup.php ? In yii2 (but also in the other migration tools ) the migration file are related to a proper template, tipically datetime_migration_name.php
But for my experience for a proper and recurrent create/populate and drop/update/delete could be useful in some situation use a controller, especially if these activities are to be launched by web page or a URL without having to launch console commands use a controller with the appropriate action can be even up and down and possibly a view to an appropriate echo the results of operations
Yes, you could assign command name using controller map, configure migration path & table and use it without interfering with the original migration command.
In the console app's config, add a controller map with demo
'controllerMap' => [
'demo-setup' => [
'class' => 'yii\console\controllers\MigrateController',
'migrationPath' => '#common/migrations/demo',
'migrationTable' => '{{%demo_setup_migration}}'
],
cd to your console app directory and use demo-setup command as you would use migrate command.
./yii demo-setup/create schema
./yii demo-setup/create sample-data
./yii demo-setup
You may wish to setup a cron job to re-create db and apply migration or re-apply migration to existing db to override demo data.
Are Yii2 migrations the right tool for that?
As long as you store your demo migrations in a separate location than your main migration. To be honest, when I start a new Yii2 project I always start with creating migrations for demo data (I also create the first user with migration). I usually use faker, and some of my own classes to generate demo data.
Is it recommended to store the migration file in a seperate subdirectory that our demo setup does not interfere with ordinary "migrate/up" and "migrate/down" processes? Yes, you should!
Is this migration bound to a file naming scheme or can this be e.g. demo-data-setup.php? From the official Yii2 documentation about naming migrations:
Note: Because the name argument will be used as part of the generated migration class name, it should only contain letters, digits, and/or underscore characters.
Ps.: don't overengineer your projects
Are Yii2 migrations the right tool for that?
No, migrations are for database structure changes (add a column, set index,..) more than fill tables with data. In your case, I would write a component, that have delete / create function for every model you need to restore. Then, you can call your component with a cron task.
You can use fixtures.
Fixtures are an important part of testing. Their main purpose is to set up the environment in a fixed/known state so that your tests are repeatable and run in an expected way. Yii provides a fixture framework that allows you to define your fixtures precisely and use them easily both when running your tests with Codeception and independently.
I have the following problem:
I Am using a Django framework.
One of the parts in a system (non-django) writes to the database, in the same database that django is using.
I want to have a signal when an object is being saved. It's a django model object but not saved via django, but directly in the mysql database.
Is there a way django can watch save-actions in his database when it's not being saved by django?
The neatest way would be: create an Api, and let the save action run through this api. The save signal can than be django default. (but this depends on some work of externals... so not the prefered route... for future development it sure is).
Another option is to implement celery and create a task that frequently looks whether one of the saved objects has had no follow up..... (also quit some puzzling I guess to get this up and running)
But there might be an easier... for me unknown?
I saw django watchdog solutions for file systems... not for databases (probably because django has this build in... when properly done through django)
to complex it: I test and develop locally with sqlite .... but the save signal I can put in my tests without needing to get this locally working.... as long as it works in mysql, I Am happy.
You can try this solution:
Create a new table 'django_watch' with one column 'object_id' (add other columns like 'created_datetime' etc according to your standards);
Lets say your main table is 'object'. Add a mysql trigger for the INSERT event on this table.
You should add an extra insert query inside the trigger to insert the object_id into 'django_watch' table.
Now you can have a cronjob that will be inpecting the new table 'django_watch' (for updations in Django objects) and perform necessary actions. You can run this cronjob continuously with some 1 minute delay (upto you).
In the end, I wrote an api that can be called by the thirdparty module. I delivered the code to logon on django using c-code to this api and call the GET of this api. (using django rest framework). This api just saves the object (the id given in the url), and from there on it's default django. The only thing the third party had to do is build in my code to call the api as well....
Maybe not the best solution, but the best to implement for my problem....
I created my models in a file and can successfully add and alter objects which successfully commit to the database when run in a function beneath the model declaration. When I import my Session and model classes into a separate script, I can successfully run a query, but if I update the object and run an update the db does not get updated. The same code works well when run beneath where I defined my models and I'm totally confused why this is. Am I confused as to how I should work with and update my database?
haha, I must have been looking at it too closely for too long. In the snippet of code that wasn't working I was calling session.commit, not session.commit().
I'm building a crossplatform HTML/Javascript app for iOS and Android using PhoneGap and jQueryMobile, and I am upgrading my app with (among others) a few new fields in one table of the local database (localdatabase/websql).
The challenge
I want to make sure that when the database is expanded with the new table fields, the existing user data, the user data will not be removed or become locked in an inaccesible older version of the database.
The background:
My app has a local database of the user's data (incomes and expenses, plus a few settings). These data need to be persitent, and the way to go, back when I started, was using the HTML5 localDatabase functionality, since that is both persistent, and available for the iOS and Android browsers as well as for most desktop browsers.
I am using a Javascript plugin/library/thingy called persistenceJS to make dealing with the localdb a little easier. But my question is not really specific to persistenceJS.
I am working on a new version of the app, which makes uses of a few new fields in the Settings table. So when these users download the new app and run it, it must test if their Settings table contains this field or not, and if not it must create the field.
How do I do this testing? I see two lines of thought:
Use the database label... that's used in the openDatabase function. This seems to be used by some developers to store a version number.
My trouble with this option is I only know how to use openDatabase to, well, open a database (and create a new one if none exists), and run a callback specifically if the database did not yet exist.
So if I open the table while specifying something like "v2" in the label, will it create a new table? If so, will it copy the old table's values into the new one?
Check for the existence of the table fields...
I could use openDatabase and then test for the existence of the table fields. If they don't, I could add them. The test would be run every time a user opens their app, which seems a little primitive.
By the way:
I know webSQL/localDb has been deprecated by the overlords, but it's still my tool and I want to stick to it for now.
I've found the answer here: http://blog.maxaller.name/2010/03/html5-web-sql-database-intro-to-versioning-and-migrations/.
Basically, you just apply the changeVersion method with the old and the new version label. If you didn't have a label, then the old label is "". While relabeling, webSQL quietly applies the new schema to the old database. Which in my case means adding the new fields.
The tutorial I linked to is really awesome (and so is the functionality).
I'm adding another answer because I've learned more about localDb opendatabase and migrating it.
As a reminder, openDatabase takes these parameters:
name - (string) name of the database
version label - (string) the version you want to open
display label - (string) a pretty useless display name that seems to be used nowhere
max size - (int) largest safe size is 5 * 1024 * 1024
newly created -= (function) to be fired if the db did not previously exist
It's wisest to assign the output of openDatabase to a variable. I.e.
myapp.db = openDatabase('mydb','','My database',5*1024*1024,newlyCreatedCallback);
First off, it seems wise to make use of the 'newly created' callback that's available as the fifth argument of openDatabase. It will fire only if there was no database with the parameters you specified. To prevent this callback from firing when your database did already exist, make sure you have the name, display label and maximum size set to exactly the values that were used to first create the database.
The reason to do this is that if the database was first created, you know for sure that you will not need to do any migrations. You can go straight to a function that adds tables and fields. I recommend using persistenceJS, a tool that helps you read and manipulate the local database.
Before calling openDatabase, it's wise to use jQuery to create a custom event 'dbopen' whose handler will execute migrations. This handler can be triggered by two events. The first is the 'newly created' callback we just discussed. The second is a setInterval that you define after call openDatabase. The interval must check for the existence of the myapp.db variable that you assigned the openDatabase output to.
The reason to create the dbopen custom event is that if you added a 'newly created' callback which triggers a whole bunch of events and continues the flow of your code afterwards, you will want a similar process for the 'not newly created' scenario. There is no callback for openDatabase that does this, so you will have to manually detect the creation of the local database and trigger 'dbopen' as soon as it has come into existence.
I use a window.setInterval for this. Make sure that you create the custom 'dbopen' event using jquery's .one() function, which will fire at most once. Otherwise if the database was newly created, you will fire the open event once when the 'newly created' callback fires, and once when the myapp.db variable comes into existence.