I'm building medium sized business web application, data is being saved on a MySQL database.
I'm trying to think of a way of adding certain selectable "widgets" to that application (e.g. a currency widget - which will show user specified currencies when the web app is visible) but having an hard time deciding how to save the widget data and settings per user since the widgets do not have a common base.
For example, the currency widget's settings is totally different than say, a weather widget.
One will require a list of desired currencies, and one would require the weather's target location.
I thought of solving the above by keeping all the widget's settings data encoded in the "widgetData" column of a db table which will contain the userId, widgetId and widgetData.
I chose JSON as my way of encoding, and each time a user tries to load it's page, I have to decode it's settings and hand the user the desired data based on the settings.
The same is true for saving the widget's actual data which does not have a common base itself.
Hopefully I can solve this by using a NO-SQL data structure next time, but this is not the case for the current project.
The Entity Attribute Value database model would be useful very to you in this scenario.
It's much more flexible than JSON or XML or other types of formats because it works within your standard SQL data storage, albeit in a different manner.
I voted up the EAV solution because this is one of the valid reasons for using it, but don't fall in love with it. An advantage of EAV is that it is database-native to the extent that you can write queries in SQL to query it (find me all widgets missing some setting and then add it), while most engines do not have JSON support.
On the other hand, if you want/need to query within a column which contains structured data, XML is a better option than JSON (right now): http://dev.mysql.com/doc/refman/5.1/en/xml-functions.html#function_extractvalue
If your widgets are rendered via Javascript in the browser, then your solution is perfectly fine. Your widgetData remains a JSON string, in Javascript you use JSON.parse() to turn it into an object and render it, and JSON.stringify() to turn it back into a string before posting it back to your server.
Related
I want to build an application that uses data from several endpoints.
Lets say I have:
JSON API for getting cinema data
XML Export for getting data about ???
Another JSON API for something else
A csv-file for some more shit ...
In my application I want to bring all this data together and build views for it and so on ...
MY idea was to set up a database by create schemas for all these data sources, so I can do some kind of "import scripts" which I can call whenever I want to get the latest data.
I thought of schemas because I want to be able to easily adept a new API with any kind of schema.
Please enlighten me of the possibilities and best practices out there (theory and practice if possible :P)
You are totally right on making a database. But the real problem is probably not going to be how to store your data. It's going to be how to make it fit together logically and semantically.
I suggest you first take a good look at what your enpoints can provide. Get several samples from every source and analyze them if you can. How will you know which data is new? How can you match it against existing data and against data from other sources? If existing data changes or gets deleted, how will you detect and handle that? What if sources disagree on something? How and when should you run the synchronization? What will you do if one of your sources goes down? Etc.
It is extremely difficult to make data consistent if your data sources are not. As a rule, if the sources are different, they are not consistent. Thus the proverb "garbage in, garbage out". We, humans, have no problem dealing with small inconsistencies, but algorithms cannot work correctly if there are discrepancies. Even if everything fits together on paper, one usually forgets that data can change over time...
At least that's my experience in such cases.
I'm not sure if in the application you want to display all the data in the same view or if you are going to be creating different views for each of the sources. If you want to display the data in the same view, like a grid, I would recommend using inheritance or an interface depending on your data and needs. I would recommend setting this structure up in the database too using different tables for the different sources and having a parent table related to all them that has a type associated with it.
Here's a good thread with discussion about choosing an interface or inheritance.
Inheritance vs. interface in C#
And here are some examples of representing inheritance in a database.
How can you represent inheritance in a database?
Background
I have a (glorified) CRUD application that I'd like to enable HTML5 offline support with. The cache-manifest system looks simple yet powerful, but I'm curious about how I can allow users to access data while offline.
For example, suppose I have these pages for the entity "Case" (i.e. this is CRM case-management software):
http://myapplication.com/Case
http://myapplication.com/Case/{id}
http://myapplication.com/Case/Create
The first URI contains a paged listing of all cases, using the querystring parameters pageIndex and pageSize, e.g. /Case?pageIndex=2&pageSize=20.
The second URI is the template for editing individual cases, e.g. /Case/1 or /Case/56.
Finally, /Case/Create is the form used to create cases.
The Problem
I would like all three to be available offline.
/Case
The simple way would be to add /Case to the cache-manifest, however that would break paging (as the links wouldn't work).
I think I could instead add something like /Case/AllData which is an XML resource, which is cached and if offline then a script on /Case would use this XML data to populate the list and provide for pagination.
If I go for the latter, how can I have this XML data stored in the in-browser SQL database instead of as a cached resource? I think using the SQL database would be more resilient.
/Case/{id}
This is more complicated. There is the simple solution of manually adding /Case/1, /Case/2, /Case/3 etc... to /Case/1234, but there can be hundreds or even thousands of cases so this isn't very practical.
I think the system should provide access to the 30 most recent cases, for example. As above, how can I store this data in the database?
Also, how would this work? If I don't explicitly add /Case/34 to the manifest and the user clicks on to /Case/34 how can I get the browser to load a page that my JavaScript will populate based on the browser's SQL database data and not display the offline message?
/Case/Create
This one is more simple - as it's just an empty page and on the <form>'s submit action my script would detect if it's offline, and if it is offline then it would add it to the browser's SQL database. Does this sound okay?
Thanks!
I think you need to be looking at a LocalStorage database (though it does have some downsides), but there are other alternatives such as WebSQL and IndexedDB.
Also I don't think you should be using numeric Id's if you are allowing people to create as you will get Primary Key conflicts, it is probably best to use something like a GUID.
Another thing you need is the ability to push those new cases onto the server. there could be multiple...
Can they be edited? If they can I think you really need to be thinking about synchronization and conflict resolution hard very hard if that is the case.
Shameless self promotion, I have a project that is designed to handle these very issues, though it's not done, it's close. You can see it (with an ugly but very functional) demo at https://github.com/forbesmyester/SyncIt
So I have this fileMakerPro7 database. As my senior project, I supposed to migrate the database to a MySQL database and than give it a PHP Based interface in 3N form...
Company allow us $200 tops to spend on the project, but if I pay for something, it has to work. However, I am having trouble finding a way of migrating the database. Any suggestions?
I have found "file maker pro migrator" (http://www.fmpromigrator.com), would the trial version be enough for us? In worst case, we will start from the beginning with throwing away the whole database that company has.
I can also download fileMakerPro12 and use it for a month with trial version for free. Would I be able to convert the db by using FMP12?
I am totally lost...open to any free suggestions...
+this is a non-profit-making company I'm doing the project for
If I had to do it, I'd look at the design of the FileMaker db and create something similar in mysql. Then I would export the Filemaker data to text and import it somehow. The details depend on foreign key values and such.
The PHP interface would be done separately.
MySQL Data Conversion:
Yes, if your database is small enough, the demo version of FmPro Migrator will convert the database and also build you a PHP web application - at no cost.
Here are the limitations of the demo version:
5 fields
5 scripts
5 layouts
PHP Web Application:
Most people don't realize it, but there is a wealth of FileMaker metadata available in XML format for performing these types of conversions. This XML info is available either thru copying the layout via the clipboard or reading it from the Database Design Report XML file. I have found the clipboard data to be the most reliable source of this info.
FmPro Migrator is able to parse in the XML and convert it into the PHP web application.
Each object on a layout is represented in XML, along with style and position info. This info can be used to create form files representing the same look as the original layout. In fact, it can be difficult to see the difference between the web application and the original database if you get all of the object properties implemented. This can be helpful for situations in which companies don't want to have to retrain their employees. They want the web application to look and work the same as the original desktop application.
I have done a few of these conversions recently into the CakePHP framework. Here a few techniques I used:
Auto-Enter Calculation Fields - Stored calculation fields are calculated and stored within the model saves a record to the database.
Unstored Calculation Fields - Unstored Calculation fields are calculated in real-time within the form controller - but only for fields actually displayed on the form. This prevents unnecessarily calculating these values if they aren't being used on a form, improving performance.
Global Fields - A Global field in FileMaker is used like a global variable in programming languages. It is important to know that each FileMaker user gets there own private copy of global field data. There is no equivalent feature MySQL or other SQL database servers, but this functionality can easily be simulated using SESSION variables. Therefore each web user will still get their own private SESSION data, simulating the same functionality originally present in the FileMaker database. I structure these globals in the model data array as if they were retrieved from the model, meaning that converted scripts and fields on forms can reference them easily. Just before the record gets written into the database, the results are saved into SESSION variables for persistence.
Global Variables in Scripts - Global variables within FileMaker scripts match up very well with the use of PHP SESSION variables, if you want to implement the same functionality.
Vector Graphic Objects - FileMaker layouts frequently include rectangles, ovals and line objects. These objects can be replaced with the RafaelJS library, providing high quality resolution independent graphics.
Value Lists - Custom and Field based value lists are implemented in a centralized location within the AppController.php file. Therefore making a change to the definition of the value list within the AppController, succeeds in changing the menu automatically throughout the whole application.
I have access to an web service that returns an XML or JSON. Since web service calls are expensive, I want to store the XML/JSON in my database so that I can access the data faster. The question I have is if I should just store the entire XML/JSON in a field or should I design a database model that represents the XML/JSON in a normalized way?
If I just needed to have the XML/JSON data available to me, then saving it as a string in a field would be OK.
However, I know that I'll be needed to extract only certain XML/JSON documents -- so I kindda need to be able to query this. For simple queries, maybe I can use something like LIKE %<title>hello world</title>% if I was search for "hello world" between the title tags within an XML document. But I think some of my queries might go beyond string matching (e.g. greater than a certain number or date, etc.). So I feel like I need to model this data properly in my database and populate it with the values from XML/JSON. This will become a painful exercise, though.
Any tips on what I should do? Maybe there is an option I didn't consider.
It really sounds like you have the requirement to translate the XML/JSON document into a distinct set of fields.
There are DBs out there that can return native JSON, but you normally would use the application layer.
I normally store pure XML/JSON when there's really no obvious need to access individual fields. For example, I'll store control data in XML format in a BLOB field, since I don't generally need to search for one particular control string.
One thing that you might want to think about is using a noSQL solution like MongoDB or Couch to store the JSON. You can put the JSON string and pull it out directly as well as access the individual fields.
A document database like MongoDB and Couch gives you the flexibility to natively store and retrieve the JSON document as well as access and search on the individual fields inside the JSON document, without needing to in advance know how the document is going to be used. It is really just eliminating a couple of steps in converting relational data into a structured document and converting a structured document into relational data.
When I do store XML directly in a BLOB, any of the searchable data would be in a field outside the BLOB. Disk space is relatively cheap, and this is a minor de-normalization. Sure, you'll have to make sure to to keep the field updated whenever the JSON/XML document is updated, but that's easy to do at the application layer.
I am developing a web application which has a typical layered architecture:
a DAO layer that retrieves domain model objects from a database;
this layer communicates with the service layer which does some business operations using those objects;
the web layer (Spring Controllers) use the service layer to retrieve the domain model objects (or collections for them) and pass them to the view layer;
the view layer is either simple JSPs which show the data using JSTL, or JSPs which retrieve some of the data through AJAX in the form of JSON objects (domain objects converted to JSON through Jackson library).
I have been researching about the following:
Very often I need to convert the db fields to a different format to show to the user. For example, a Date might be store as a Timestamp and I want it to show it as a formatted date (e.g. dd/mm/yyyy).
Also, I need to do the opposite, convert some value (usually user input) to the format of a domain model object's property.
My question is, where should I be doing this kind of conversions? Especially with JSON data, they should be already formatted on the AJAX response, I dont think I should format it with Javascript, am I right?
Thank you in advance.
Very often I need to convert the db fields to a different format to
show to the user. For example, a Date might be store as a Timestamp
and I want it to show it as a formatted date (e.g. dd/mm/yyyy).
Also, I need to do the opposite, convert some value (usually user
input) to the format of a domain model object's property.
My question is, where should I be doing this kind of conversions?
My opinion is that all data format conversions ought to be done in the view, or in collaborative objects. The model or the service layer should not be performing this activity. It would be poor design to perform this in the model, for it would couple the interfaces exposed by the model to only one view. Also, it would make it more difficult to make changes in the view, requiring changes in your model as well.
You can refer to well-written JSF applications (since you mentioned that you are using Java EE), where specific converters are written to ensure that the view would format the contents of the domain objects appropriately.
Especially with JSON data, they should be already formatted on the AJAX response, I dont think I should format it with Javascript, am I right?
That would depend on how you view the view. If your representation of the view is that the server must provide the client with formatted data, then the server ought to do this. But, most applications take the practical approach where in client-side logic is also treated as part of the view. Depending on the context in which the data is used, you may:
either format the data on the server, especially if the formatted data is used universally at the client side without any further modifications.
or, transmit data to the client in a canonical form, which is then formatted on the context in which the data used and/or is displayed to the user.