I am working on a mobile (windows phone) RSS Feed Reader pet project.
I had a few design decisions on which I needed guidance. Here they are:
Firstly, when the feed reader downloads the RSS feed how can I show which items are read by the user vs. those that are new or not-read.
Do I store the file contents locally in a DB along with the information of which articles are read/unread.
Secondly, when we download the RSS feed, do we download the entire file? I guess even with an approach like CONDITIONAL-GET to fetch feed contents only on update, there is no way to download a delta of the RSS file.
Thirdly, if the entire file is downloaded, do mobile feed readers show data from beginning or truncate it to show feeds only from last N days (where N is an integral number of days like 15 or 30)
Thanks in advance
Regards
Vikas
Since you're building a mobile phone application, there are 2 ways you could go: either have a server keep track of the updated content and let the devices connect to it to retrieve it or handle everything (fetching, parsing and diffing the feeds) from the phones.
For your questions:
You have to keep track locally (on the device) of what the user has read or not. You'll probably use the <guid> or <id>(Atom) elements in feed's entries to identify each of them and keep track of lists of items that were read.
Conditional GETs (if-Modified-Since and ETag) will not help you very much because when the content has been updated, they serve you the whole RSS/Atom document. So, yes, you'll have to download the whole file over and over again, and yes, that's a lot of waste.
This is really up to you. It may actually be relatively "cheap" to store everything on the device and let the user decide if they want to delete the past stories.
If you don't want to deal with the hassle of fetching, parsing and diffing RSS feeds, I suggest you check services like Superfeedr which can do that on your behalf either on the server or on the device!
Related
Hello Stack Overflow Community!
I am making a directory of many thousand custom mods for a game using HTML tables. When I started this project, I thought one HTML page would be slow, but adequate for the ~4k files I was expecting. As I progressed, I realized there are tens of thousands of files I need to have in these tables, and let the user search though to find what they are missing to load up a new scenario. Each entry has about 20 text entries and a small image (~3KB). I only need to be able to search through one column.
I'm thinking of dividing the tables across several pages on my website to help loading speeds and improve overall organization. But then a user would have to navigate to each page, and perform a search there. This could take a while and be very cumbersome.
I'm not great at website programming. Can someone advise a way to allow the user to search through several web pages and tables from one location? Ideally this would jump to the location in the table on the new webpage, or maybe highlight the entry like the browser's search function does.
You can see my current setup here : https://www.loco-dat-directory.site/
Hopefully someone can point me in the right direction, as I'm quite confused now :-)
This would be my steps,
Copy all my info into an excel spredsheet, then convert that to json, then make that an array for javascript (myarray), then can make an input field, and on click an if statement if input == myarray[0].propertyName
if you want something more than an exact match, you'd need https://lodash.com/
in your project.
Hacky Solution
There is a browser tool, called TableCapture, to capture data from html tables and load into excel/spreadsheets - where you are basically deferring to spreadsheet software to manage the searching.
You would have to see if:
This type of tool would solve your problem - maybe you can pull each HTML page's contents manually, then merge these pages into a document with multiple "sheets", and then let people download the "spreadsheet" from your website.
If you do not take on the labor above and just tell other people to do it, then you'd have to see if you can teach the people how to perform the search and do this method on their own. eg. "download this plugin, use it on these pages, search"
Why your question is difficult to answer
The reason why it will be hard for people to answer you in stackoverflow.com (usually code solutions) is that you need a more complicated solution (in my opinion) than hard coded tables and html/css/javascript.
This type of situation is exactly why people use databases and APIs to accept requests ("term": "something") for information and deliver responses ( "results": [...] ).
Thank you everyone for your great advice. I wasn't aware most of these potential solutions existed, and it was good to see how other people were tackling problems of similar scope.
I've decided to go with DataTables for their built-in sorting and filtering : https://datatables.net/
I'm also going to use a javascript array with an input field on the main page to allow users to search for which pack their mod is in. This will lead them to separate pages on my site, each with a unique datatable for a mod pack. Separate pages will load up much quicker than one gigantic page trying to show everything.
I tagged this question as "conceptual" because I'm not sure if creating Chrome Extension is the best way. In my opinion it's better to ask before spending few hours writing something and find out that some part is too much difficult or impossible.
The problem
I used to analyze my finance using csv files downloaded from bank account. But as it sometimes happens, my bank launched new GUI and csv feature disappeared. They do not know when and whether they will do it at all. So I have to grab transactions in some way and put them in CSV file.
Concept of solution
I think scrapping the page with transactions it's not good way, because whole data is looking like generated with totally random CSS classes and ids. I noticed that list of transactions is sent in JSON format by AJAX response. I analyzed that JSON and every interested field has name, so access to data is quite easy. Only one problem I see, it's that first JSON load shows only first 10 transactions. To load more I have to scroll down, then there is next AJAX request to the same URI, then next JSON load comes in response. So, if I want to get transactions from whole month I have to scroll few times down and my tool should catch first and also next responses.
I don't have experience with Chrome Extensions but they claim that if I know web technologies like JS, CSS, HTML it shouldn't be difficult to write simple extension. If I can take this JSON from AJAX response to my extension then generating CSV file shouldn't be problem.
The question
The main question is if my concept is possible to realize. Is there easy access to data loaded from AJAX response? If you see any better solution I'm open to suggestions.
you can create CSV file from Json in DOM (don't required to call any API).
Please refer this demo link
jsfiddle.net/hybrid13i/JXrwM/
if you care implemented this functionality in chrome extinction so you need to add download permission in manifist.json https://developer.chrome.com/extensions/downloads
I am using such MySQL request for measuring views count
UPDATE content SET views=views+1 WHERE id='$id'
For example if I want to check how many times some single page has been viewed I've just putting it on top of page code. Unfortunately I always receiving about 5-10x bigger amount than results in Google Analytics.
If I am correct one refresh should increase value in my data base about +1. Doesn't "Views" in Google Analytics works in the same way?
If e.g. Google Analytics provides me that single page has been viewed 100x times and my data base says it was e.g. 450x times. How such simple request could generate additional 350 views? And I don't mean visits or unique visits. Just regular views.
Is it possible that Google Analytics interprates such data in a little bit different way and my data base result is correct?
There are quite a few reasons why this could be occurring. The most usual culprit is bots and spiders. As soon as you use a third-party API like Google Analytics, or Facebook's API, you'll get their bots making hits to your page.
You need to examine each request in more detail. The user agent is a good place to start, although I do recommend researching this area further - discriminating between human and bot traffic is quite a deep subject.
In Google Analytics the data is provided by the user, for example:
A user view a page on your domain, now he is on charge to comunicate to Google The PageView, if something fails in the road, the data will no be included in the reports.
In the other case , the SQL sistem that you have is a Log Based Analytic, the data is collected by your system reducing the data collection failures.
If we see this in that way, that means taht some data can be missed with the slow conections and users that dont execute javascriopt (Adbloquers or bots), or the HTML page is not properly printed***.
Now 5x times more it's a huge discrepancy, in my experiences must be near 8-25% of discrepancy. (tested over transaction level, maybe in Pageview can be more)
What i recomend you is:
Save device, browser information, the ip, and some other metadata information that can be useful and dont forget the timesatmp, so in that way yo can isolate the problem, maybe are robots or users with adblock, in the worst case you code is not properly implemented ( located in the Footer as example)
*** i added this because one time i had a huge discrepancy, but it was a server error, the HTML code was not properly printed showing to the user a empty HTTP. The MYSQL was no so fast to save the information and process the HTML code. I notice it when the effort test (via Screaming frog) showed a lot of 500x errors. ( Wordpress Blog with no cache)
I need to fetch a list of all the files in a user's box account, such that the list of files can then be displayed in a table view (iOS).
I have successfully implemented this by recursively using /folders/{folder id}/items on all the folder's in my user's box.
However, while this works, it's kind of dirty, seeing as how a request is made for each of the users's folders, which could be quite a large number.
Is there any way to get a list of all the files (it's no issue if folders are included, I can ignore those manually) available?
I tried implementing this using search, but I couldn't identify a value for the query parameter that returned everything.
Any help would be appreciated.
Help me, Obi-Wan Kenobi. You're my only hope.
What you are looking for (recursive call through a Box account) is not available. We have enterprise customers will bajillions of files and millions of folders. Recursively asking for everything would take too long.
What we generally recommend is that you ask for as little as you can, and that you use multiple threads and anticipate what you'll need just a little bit, so that you can deliver a high-performance user-interface to your end-users.
For example ?fields=item_collection is expensive to retrieve, and can add a lot to a paylaod. It can double, or 10x the time that it takes to get back a payload from the Box API. Most UI's don't need to show all the items inside every folder. So they are better off asking for ?fields=.
You can make your application responsive to the user if you make the smallest possible call. Of course there is a balance. Mobile networks have high latency, and sometimes that next API call to show some extra thing is slow. But for a folder tree, you can get high performance by retrieving only the current level, displaying that, and then starting to fetch one-level down while the user is looking at the first level.
Same goes for displaying thumbnails. If a user drills into a folder and starts looking at thumbnails for pictures, there's a good chance they'll want to see other thumbnails in that same folder. Your app should anticipate that, and start to pull one or two extras down in the background. Yes, it means more API calls, but your users will give your app a higher rating for being fast.
There is a website that my company uses that updates information about 3 specific things throughout the day. We use the information from 1 of them and what we are wanting to do is pull this information as it is added to their site and add it to a page of our own to view easier. Is this even possible? Can anyone point me in the direction of setting this up? It is all text that we want to pull.
Pick a language (e.g. Perl). Find an HTTP library for it (e.g. LWP). Fetch the page and run it through an HTTP parser (e.g. HTML::TreeBuilder). Pull out the bits you want and shove them into a template (e.g. TT) then dump to a file. Stick the program in cron or Windows Scheduler.