I'm interested in building a small offline webapp and I'm looking for some advice. Here's the basics of what I want it to do
Create reports that, initially, will just have a name and text field
List, edit, and delete these notes
Ideally I'd like to add more fields to the reports later
Is localstorage a good option for storing this type of data locally? If so, can anybody direct me to a complete list of the commands for interacting with it in javascript? e.g. setItem, getItem, etc.
Thanks.
localstorage will work just fine for this, but don't think of it as a robust solution.. It's just a basic key/value store and won't be very performant with thousands of complex things going on.
Check out the excellent Dive into HTML5 guide on localstorage:
http://diveintohtml5.info/storage.html
Link to the localstorage apis
Yes localstorage would be perfect for you application. It would allow your application to have no need to connect to a server at all. Keep in mind that local storage does have maximums on the amount of data that can be stored.
EDIT:
Using JSON.stringify() on can convert complex javascript objects to json which can be storage and retrieved with ease inside of local storage.
Related
I know this sounds crazy, but I had a thought and I was willing to try it out. I use GitLab pages for all my online projects, but a lot of them are ASP.NET MVC, which is an issue as I don't think you can run ASP.NET MVC sites on GitLab pages. I then thought, what if I make a site using something like angular or node.js, and have a central API for all my web projects? I thought that was a great idea, until I realized I couldn't use a database either. I guess what I'm asking is, would it be possible to create a REST API that uses JSON files for storage and node.js as the request pages, to create an API without a database?
Of course.
If you think about a database from the perspective of your application code, it is basically just a place to store and retrieve data.
Imagine the database library you are using has two simple methods, store and retrieve. In your application code, you could write db.store('here is the item') and the later on, db.retrieve().
However, those store and retrieve methods could be implemented in many different ways to provide the same effective behavior from the perspective of your application. Some examples:
Send/query the data to/from an external data store, such as PostgreSQL
Write it to a file on disk and read it back later
Store the data in memory
Make HTTP requests to an external system to store the data
Some of these options will be more or less appropriate depending on your exact requirements, however, the general idea is that given a database API, you could implement the exact same method signatures with a completely different approach.
I'm new to webdev and I'm trying to use passport for registration/authentication on a site I'm setting up. I'm also going to write an application in node later on that will be using some of the user data (users will need to provide an API key for an account on another site that I will use to pull data into the application).
At the moment, the main issue I'm having is figuring out what goes where. I've found plenty of resources that explain how to create an app using passport, but nothing shows how it would be incorporated into your website or where the files should be in relation to your website. I'm relatively new to Node.js, and while I've written a few small applications I have never hosted them anywhere.
Bonus question: I'm using MongoDB with passport and I was also planning to use it to store some JSON my application will be receiving from API calls. However, I wanted to use MySQL to store some data as well. More specifically, I'm planning to save the raw JSON then I'll create a relational database out of the data I need from the JSON and then keep the rest in MongoDB for easy access. Is this common/smart, or should I focus on keeping everything in my MongoDB? I'm relatively new to NoSQL.
Thanks in advance for any help.
I would reference this tutorial. I just recently used this to help myself with a new application. Also there is an example of the same thing but in SQL here. So not sure what you mean by " where the files should be in relation to your website". The information related to to authentication should go in your database.
To your "bonus question" you can use two databases. The key here is to ask yourself why and what are the true needs for data, and how is this data accessed and used. From ground up I would like one and stick with it. If at some point later you realize a certain type of data would be better in a different database then you can add it.
Side note: look into an IDE such as webstorm to help you out.
I'm building a Phonegap application and i'm planning to set it to work in both Offline and Online mode, the idea is to get JSON Data from a PHP server side script and show it in the application, these data contains text, images so my question is is there anyway to set a dynamic cache manifest or any other way in a way that the user can see the already loaded data when he's offline, like caching the json result itself or anything else you can help me with
thanks
What do you mean by "work in both Offline and Online mode"? Are the users able to only see the user created data or can they modify it too? Is it correct to use a Manifest to control caching of user created data? I would have thought you should only really use it for caching the application templates and code. In any case I think you'll probably need something that you can exert more control over, something based perhaps using WebSQL, IndexedDb or LocalStorage.
I have been working on the modify-it-too question for many months and have a solution in the form of javascript(phonegap) <-> server synchronization that works somewhat like a version control system, with version numbers and conflict resolution. There's some pretty great docs and demo on the GitHub page and you can even see a presentation courtesy of SkillsMatter / LondonAJAX.. Currently there is only a JS/Node based server but I will probably do PHP based server because that's my day job If you look at the server code it's pretty easy to implement.
BTW I would use a different method to store the images, why not just download them and store them in a File, I don't know the storage limits, but I bet that is the way that Phonegap will let you get away with most.
Background
I have a (glorified) CRUD application that I'd like to enable HTML5 offline support with. The cache-manifest system looks simple yet powerful, but I'm curious about how I can allow users to access data while offline.
For example, suppose I have these pages for the entity "Case" (i.e. this is CRM case-management software):
http://myapplication.com/Case
http://myapplication.com/Case/{id}
http://myapplication.com/Case/Create
The first URI contains a paged listing of all cases, using the querystring parameters pageIndex and pageSize, e.g. /Case?pageIndex=2&pageSize=20.
The second URI is the template for editing individual cases, e.g. /Case/1 or /Case/56.
Finally, /Case/Create is the form used to create cases.
The Problem
I would like all three to be available offline.
/Case
The simple way would be to add /Case to the cache-manifest, however that would break paging (as the links wouldn't work).
I think I could instead add something like /Case/AllData which is an XML resource, which is cached and if offline then a script on /Case would use this XML data to populate the list and provide for pagination.
If I go for the latter, how can I have this XML data stored in the in-browser SQL database instead of as a cached resource? I think using the SQL database would be more resilient.
/Case/{id}
This is more complicated. There is the simple solution of manually adding /Case/1, /Case/2, /Case/3 etc... to /Case/1234, but there can be hundreds or even thousands of cases so this isn't very practical.
I think the system should provide access to the 30 most recent cases, for example. As above, how can I store this data in the database?
Also, how would this work? If I don't explicitly add /Case/34 to the manifest and the user clicks on to /Case/34 how can I get the browser to load a page that my JavaScript will populate based on the browser's SQL database data and not display the offline message?
/Case/Create
This one is more simple - as it's just an empty page and on the <form>'s submit action my script would detect if it's offline, and if it is offline then it would add it to the browser's SQL database. Does this sound okay?
Thanks!
I think you need to be looking at a LocalStorage database (though it does have some downsides), but there are other alternatives such as WebSQL and IndexedDB.
Also I don't think you should be using numeric Id's if you are allowing people to create as you will get Primary Key conflicts, it is probably best to use something like a GUID.
Another thing you need is the ability to push those new cases onto the server. there could be multiple...
Can they be edited? If they can I think you really need to be thinking about synchronization and conflict resolution hard very hard if that is the case.
Shameless self promotion, I have a project that is designed to handle these very issues, though it's not done, it's close. You can see it (with an ugly but very functional) demo at https://github.com/forbesmyester/SyncIt
I'm writing a crawler which needs to get data from many websites. The problem is that every website has different structure. How can I easily write a crawler which downloads (correctly) data from (many) different websites? If the structure of a website will change will I need to rewrite the crawler, or are there other methods?
What logical and implemented tools can be used to improve the quality of data mined by an automatic web-crawler (many websites are involved with different structure)?
Thank You!
I presume you want to query it is some way, in which case you should store the data in a flexible data store. A relational database would not be fit for purpose as it has a strict schema, but something like mongodb which lets you store semi structured data without having to define a schema up front, but still provides a powerful query language.
The same goes for how you represent the data in the crawler code. Don't map the data to classes where the structure is defined up front, but use a flexible data structures that can change at runtime. If you are using Java then de-serialise the data into HashMaps. In other languages this might be called Dictionaries or Hashes.
If you're scraping data from websites that actually want to allow you to do that, chances are they will provide some sort of webservice to allow you to query their data in a structured way.
Otherwise, you're on your own, and you might even be violating their terms of use.
If the websites provide no APIs, then you're out cold and you have to write separate extraction module for each data format you're encountering. If the website changes the format, then you have to update your format module. A standard thing to do is to have plugins for every website you're crawling and have a testing framework which does regression testing with data you've already collected. When a test fails you know something went wrong and you can investigate whether you have to update your format plugin or if there is another issue.
Without knowing what kind of data you're collecting it will be very difficult to try to hypothesize about ways to improve the "quality" of the data that was mined.
Maybe you could find out whether the website allows you to access the data like API, if so, you could use this kind of structured data to your website directly. If not, you may need plugins for that. Or you could turn to other web crawlers with API access like Octoparse, to find the way to access their API to your own web crawler.