I am a naive user.
There is this website which is a really important source of information for my business.
To monitor the websites, I convert them to RSS feeds using page2rss service and then monitor feeds in IFTTT.
However, this particular site does not use static web pages and generates data response to API Calls:
Here is a sample API Call:
https://www.mpeproc.gov.in/ROOTAPP/GetTenderFreeView.jsp?Department=Urban%20Administration%20and%20Development%20Department&company=MPSEDC
Is there a way by which I could record the response from this call to an html page on my server? or is there any other way to monitor such dynamic pages.
There are solutions but not simple ones. The first page uses JavaScript to create a FORM which it then submits. You can simulate this with the command line tool curl; see https://superuser.com/questions/149329/what-is-the-curl-command-line-syntax-to-do-a-post-request
But take note that many sites don't like scraping; if they notice what you're doing, you may end up on a blacklist. So it's better to ask the site's owner for permission before you aim automated tools at their precious data.
Related
have been with mvc for a little while. the usual case when an action returning json, it is initialized by ajax in the view and the view is expecting info inside the json.
is there a case the action returning json to the view and is caught by something else instead of javascript? Thanks.
Yes, a JSON API can be consumed by a large variety of clients. It can be the browser sending an AJAX request, but it can also be a desktop application fetching data from the Internet, a server-side job scraping the data for analysis, etc.
For example, let's say you're running a stock exchange website, and you're publishing current stock values as JSON. You can use that JSON on your website to display the data, but you (or any other developer) can also write a desktop application which will get that data and process it on a local machine (to, for example, show the user which stocks they should buy). Or aggregate data from different sources.
Many websites make their APIs public, so that third party developers can write alternative clients, integrate the API's functionality in their own products, and so on. For example, GitHub's APIs are public - the GitHub website can utilize them for the AJAX requests, and GitHub for Windows can show you the list of repositories you own by making a request to that API using C#'s WebClient.
Premise:
This question is more like "is this the correct approach?" rather then "how do I do this?"
Situation:
I have created a CherryPy REST application following this tutorial. The data are returned using the json module with a simple json.dumps. This example works. I have a mako template which I would like to use to create a HTML table with the data. Again, the template by itself works.
The question: In my head, it works this way:
The user asks for a URL
The REST API creates the JSON with the results
The mako template produces the HTML given the results
The HTML is returned to the user
My problem is, I'm not sure this is the right approach: in my (small) experience, the JSON should be returned to the client, which should render it in some way (using jQuery or some other client-side language); instead, the mako template works on the server-side, and that's where I got stuck.
Do you have any advice?
You don't need REST to solve this, this is a regular web application. Just to show you how this works by a REST service:
The user asks the REST client to do some-automated-stuff on the service
The client requests example.com/api/v1 to get the main page
The client looks for hyperlinks on the main page
The client finds hyperlink with relation=some-automated-stuff
The client follows that hyperlink and gets a JSON result from the REST API
The client uses the mako template and produces the HTML given the results
The HTML is returned to the user
A typical client is a 3rd party server side application, which is registered by the service and got a unique API key. The users allow to the clients to do some-automated-stuff on their account by giving permissions to API keys. So for example the service can be the facebook API and a client can be an application developed by Zynga e.g. Farmville. When you allow to the client to post on your news feed that you just harvested 100000 crops on your farm, then it sends a request about this to the facebook API and so some automated message will show up on your news feed.
Ofc. you can write a 1st party AJAX or server side client if you want, but if you don't want to support 3rd party clients, then it does not make much sense to develop a REST service. Ofc. your client won't break as easy as by regular applications, so it might worth the effort if that is important.
I'm making a Twitter account statistics program that reads tweets, retweet counts, and favorite counts. I could attempt to read the user's Twitter account URL line by line and parse the information from there, but I was wondering if there was a public API or part of Twitter that just spits out the raw data without formatting it all pretty for web browsers? Not only would this be more efficient in the program, but would also be much neater.
It seems as though API 1.1 uses JSON to fetch data, but I need to make a developer account and create unique identifiers in order to access such data. Is it worth it? Is there some sort of alternative that would be faster and easier?
All API calls to Twitter now require OAuth authentication, so there is unfortunately no way around signing up for a developer account and creating an app. It's not even possible to use a service that makes the requests on your behalf, as this is re-syndication which is forbidden by Twitter's API terms, so you need to make the calls yourself.
I've got an android and iphone app that both get the required data from a webserver. The data is sent via json to the client. Using this setup other people might simply retrieve the url the app is calling and this way could make use of the data that I gather with my scripts on the server. To make it short: I don't want that :)
My idea is to make the json unreadable for example by encrypting it. This would make it a little harder to retrieve the information since this way some who would like to use my service would had to decompile the app an lookup any decryption stuff I had implemented.
Therefore two questions:
Do there exist some libraries that already offer such a functionality (Server side is Java)?
Does anyone of you have any other suggestions how I could protect my api from unwanted guests?
Thanks in advance :)
I think the options available would be...
to lock down the API to Authorized/Authenticated users.
Using BSON to obfuscate the data.
You could always use oAuth to allow the users to authenticate based on an account they already have: Facebook, Twitter, Google etc.
For example, facebook. I have a list of different teams that shows overall data for that team, but the goal is the user will click on their team and send them to page that is detailed information about their team. My client wants the user's team name to be within the url so they can save the page as a favorite.
How I do this with out making a html file for every single team that gets made?
Im using Django for back end.
If you're using django then there is certainly no sense in making html for each team. You should make a template and populate this template with the data you're getting from your database (models.py) according to url (urls.py) and the appropriate view (views.py).
This is fairly basic django usage covered extensivly in the offical tutorials and the django documentation. Read it and use it, cause there is no shortcut. And last but not the least - enjoy cause such good tutorials and docs you won't see every day.