One application loads config values from external XML files. These XML files are not supposed to be modified by users directly. But should be modified within the application.
The question is, there's no way to prevent user from finding and modifying the XML files. And the modification can be incorrect/malicious.
Should I validate the values when loading into the application? Or any other best practice? Or should I just document it and blame user for mis-use.
My advice is that you should always validate the contents of a configuration file. Doing so means your application will be adhering to the Fail Fast principle. Whether you want to hand-code your own validation checks from scratch or use an off-the-shelf XML schema validation language to automate some/most/all the checks is of lesser importance.
Related
tl;dr
In my node.js application I create pdf documents. What is the best/right way to save them? Right now I use node.js fileserver and shell.js to do it.
I am working on a node.js web application to manage apartments and tenants for learning purpose and on some point I create PDF Documents that I want to save under a path
/documents/building_name/apartment_name/tenant_name/year/example.pfd
Now if the user wants to change the building, apartment or tenant name via an http PUT request I change the database but also the want to change the path.
Well both works but I can't write good tests for these functions.
Now a friend told me that it's a bad practice to save documents on a file server and I better should use BLOB.
On the other side google doesn't really agree on using blobs
So what is the right way to save documents?
Thanks
Amit
You should first define a source of truth. Unless you're legally obliged to keep copies of those files and they are not being accessed very often, I wouldn't even bother storing those at all and just generate them upon request.
If not, keep the DB clean, blobs will make it huge. Put them into cold storage (again assuming they are not being accessed too frequently) without those paths. If the paths are reliant on often changing information, that can't be performant for neither the file server nor your system.
Instead store a revision number in your DB that the file can be found under and limit the path structure to information that rarely change.
Like {building}/{apartment}/{tenant}_{revision}.pfd
That - depending on your backup structure - will allow you to time-travel if necessary and doesn't force a re-index all the time.
Note: I don't know too much about your use case.
One of the resources my app uses is data in two JSON files that are pulled from a third party and that are constantly updated with fresh content.
Each of these files have a specific structure that doesn't change.
However, sometimes the third party creates structural changes that may mess with my app.
My question is: how can I monitor their structure so I can detect changes as they occur?
Thanks!
For this reason you can use json schema and validate the files agains it. If they have the schema than you're, good just need to validate. If not than you have to do it based on a correct json. There are online generators for that.
Background
I have a (glorified) CRUD application that I'd like to enable HTML5 offline support with. The cache-manifest system looks simple yet powerful, but I'm curious about how I can allow users to access data while offline.
For example, suppose I have these pages for the entity "Case" (i.e. this is CRM case-management software):
http://myapplication.com/Case
http://myapplication.com/Case/{id}
http://myapplication.com/Case/Create
The first URI contains a paged listing of all cases, using the querystring parameters pageIndex and pageSize, e.g. /Case?pageIndex=2&pageSize=20.
The second URI is the template for editing individual cases, e.g. /Case/1 or /Case/56.
Finally, /Case/Create is the form used to create cases.
The Problem
I would like all three to be available offline.
/Case
The simple way would be to add /Case to the cache-manifest, however that would break paging (as the links wouldn't work).
I think I could instead add something like /Case/AllData which is an XML resource, which is cached and if offline then a script on /Case would use this XML data to populate the list and provide for pagination.
If I go for the latter, how can I have this XML data stored in the in-browser SQL database instead of as a cached resource? I think using the SQL database would be more resilient.
/Case/{id}
This is more complicated. There is the simple solution of manually adding /Case/1, /Case/2, /Case/3 etc... to /Case/1234, but there can be hundreds or even thousands of cases so this isn't very practical.
I think the system should provide access to the 30 most recent cases, for example. As above, how can I store this data in the database?
Also, how would this work? If I don't explicitly add /Case/34 to the manifest and the user clicks on to /Case/34 how can I get the browser to load a page that my JavaScript will populate based on the browser's SQL database data and not display the offline message?
/Case/Create
This one is more simple - as it's just an empty page and on the <form>'s submit action my script would detect if it's offline, and if it is offline then it would add it to the browser's SQL database. Does this sound okay?
Thanks!
I think you need to be looking at a LocalStorage database (though it does have some downsides), but there are other alternatives such as WebSQL and IndexedDB.
Also I don't think you should be using numeric Id's if you are allowing people to create as you will get Primary Key conflicts, it is probably best to use something like a GUID.
Another thing you need is the ability to push those new cases onto the server. there could be multiple...
Can they be edited? If they can I think you really need to be thinking about synchronization and conflict resolution hard very hard if that is the case.
Shameless self promotion, I have a project that is designed to handle these very issues, though it's not done, it's close. You can see it (with an ugly but very functional) demo at https://github.com/forbesmyester/SyncIt
I am developing a web application right now, where the user interacts with desktop software that exports files as XML.
I was curious if there was a way to take the data from the XML file and insert that data into a mySQL database using ColdFusion?
Of course you can, ColdFusion provides powerful tools for handling XML.
Typically you'll need to parse XML file into the XML document object with XmlParse and search through it using XPath language with XmlSearch. Fetched data you can easily use for inserting into the database or any other manipulations.
Please note that there are more useful XML functions present, for example you may be interested in validation XML before parsing it.
If you'll need help for specific situations -- please extend your question or ask another one.
If you are working with XML documents that fit into memory when parsed, #Sergii's answer is the right way to go. On the other hand, XML being verbose as it is, and ColdFusion's using a DOM XML parser, can easily lead to Out of Memory errors.
In that situation, given MySQL and ColdFusion, I see two alternative paths. One is exporting the data from the desktop application as CSV, if possible. Then use MySQL's LOAD DATA INFILE, which you can call from ColdFusion to import the data. This is probably the fastest performance.
If you cannot change the desktop application's export format, consider using a Java StAX parser instead. See my answer from another question for an example of how to do this with ColdFusion. This has the advantage of only pulling in part of the XML document into memory at any given time, but is somewhat more difficult to work with than a DOM parser. As such you will not get OOM errors.
Note, there is a third type of parser available as well from Java - SAX - that has the same quality as a StAX parser of not loading the whole document into memory. However, it's a more difficult approach IMO to work with, thus the StAX recommendation.
When I say "configure" I mean where to save those values that could change very often (constants values like taxes rates or something similar) and then when you need to change them you don't want to re-compile your application.
Where to save those values? Database? XML File? Flat File?
It depends on how often these change and who or what changes them. For some application specific settings, it's best to use an XML or config file, where the developers are the ones responsible for updating it. For other "businessy" values (like exchange rates, tax rates, etc), it's best to keep them in the database and provide a UI for users (not developers) to update.
It also depends on how many apps depend on this value, for example, if several applications depend on some setting (such as email server addres), it's best to put it in a database since it'll be easily accessible from any machine where the app is running.
I use INI files for potentially user-configurable files, and BIN files for data that save session state between runs.
But, it is very dependent upon what type of application you are developing.
it depends on how your app is architecture. you could design your app in such way that you could change the location of you configuration. by just injecting the provider.
Normally I use Ini files or XML if the data is structured.
For applications that already use a database and you don't want to have the user to change the data easily, you can use the database.
I almost never use binary data unless you want to obfuscate the data for the user.
Regardless of app, you're probably going to have at least 3 sources of configuration data:
Command line flags, usually for bootstrapping your run-time environment, e.g, finding config files, setting debug flags, include paths, class paths, etc
Config files, potentially more than one that may override each other. These usually boot strap your application: connection strings, cache settings, build-specific settings, etc
Control data in a database. Things like timezones, conversion rates, stable display values, etc. This data should also be versioned in the database (as in, a "Data Version" field, not living in a version control system). Versioning it will save a lot of headaches when you find you need to change a setting for a new release, but the old release will break if you change it.
Generally, anything that changes at run-time should go in the database. Anything that is sensitive and rarely changing should go into the config files, and any hacks should go on the command line (--[no]enable-bug-287438-hack can be very handy when you need it).
I prefer the simplicity of a flat ini file. Here's an example Setting class that you might find useful.