(I'm not sure the best way to word this, and I haven't found anything yet)
Is there a setting, or a plugin to allow me to isolate cookies to the site that originated the calls. I.e. when a script is called from a 3rd party site, that can include cookie information. I.e. this is how sites like FB track your usage across everything.
What I'm looking for is a way that when I go to BobsDomain.com who happens to be making calls to Facebook.com, instead of having cookies stores like this:
BobsDomain.com/Cookie1
Facebook.com/Cookie1
it would be stored like this:
BobsDomain.com/Cookie1
BobsDomain.com/Facebook.com/Cookie1.
That way when I go to another site, I get a new cookie from facebook, and it has no idea who I am. Of course It should also have the ability to allow the cookies to be taken from other sites in particular instances. (a user's choice)
Any one know of an existing plugin like this? For Chrome, IE, or some other browser?
Thanks.
Related
I have a chrome extension, and I am creating a website for it, that has info, and a way to test it. I would like to show the number of users currently, but have found no way to do this. It doesn't have to update every day, but if it updated weekly that would be great!
Also- I don't care if it is by a outside company rather than chrome itself. all that matters is that it works.
I am planning to show something similar on my extension's site so I have googled around a bit recently. I found a repo (https://github.com/petasittek/chrome-web-store-stats) that parses information about extensions in the Chrome Web Store. It could be useful for you. I think I will end up just parsing the Webstore page for the user numbers everyday in save it to a database. My extension is also available for Firefox and I will do the same there then aggregate the results.
I've been driving myself mad trying to get curl, wget, the python request module, and others, to simply get me logged in to a website and pull page text there. I can certainly request HTML from the site, but only as an anonymous user. I've spent a few hours with tricks like chrome's "copy cURL" feature, but the website in question is smart enough to defend against login playbacks.
All I want is a way, from the command-line, to do something like:
chrome.exe --output_to_file page.html https://www.endpoint.com/auth_access_only.html
Essentially, I'm looking for chrome to do for me what cURL does, but I want the command-line invocation to be executed as me. I can see how this might open a potential security issue, but I don't mind at all if I have to do something magical to authorize my script. I'm not looking to do anything evil - I just want to be able to write scripts that are as "me" as I am.
I guess that, if it's truly unavoidable, I could suck it up and dust off Internet Explorer. I'd really rather not do that. I'd feel so dirty.
This is possible, but it's not as simple as you're thinking.
You can use the Chrome Debugging Protocol to remote-control Chrome.
You will need to write some code to make this work - I have done similar tasks using the chrome-remote-interface library for Node.js.
Make sure you understand what a browser profile is and where your profile folder lives.
If Chrome is already running using your browser profile: make sure it was launched with --remote-debugging-port=9002 or similar.
If Chrome is not already running using your browser profile: launch it with --user-data-dir="C:\path\to\your\profile" --remote-debugging-port=9002 or similar.
The "running or not" part is a bit tricky - you cannot launch more than one Chrome instance with the same browser profile, but you need to use this user profile because your login data is stored there. It may actually be easiest to create a separate browser profile that is just used for this automated task, and log in to the site there too.
Then, at a high level, your Node.js code will need to connect to Chrome, load the page, wait for the response, and save it to a file. Have a look at the example code for the chrome-remote-interface library - you can definitely piece together what you need from there.
Another option which uses the same underlying technology is to use puppeteer which is another tool to automate Chrome. It is designed to start from a fresh profile every time. If you do this, you'll need to script more interactions:
Visit the site's login page
Type the login credentials into the form and click the login button
Visit the site's authenticated page and save it to a file.
The benefit of this approach is that the result should be more reliable, preventing issues like expired login sessions.
This all goes back to some of my original questions of trying to "index" a webpage. I was originally trying to do it specifically in java but now I'm opening it up to any language.
Before I tried using HTML unit and other methods in java to get the information I needed but wasn't successful.
The information I need to get from a webpage I can very easily find with firebug and I was wondering if there was anyway to duplicate what firebug was doing specifically for my needs. When I open up firebug I go to the NET tab, then to the XHR tab and it shows a constantly updating page with the information the server is updating. Then when I click on the request and look at the response it has the information I need, and this is all without ever refreshing the webpage which is what I am trying to do(not to mention the variables it is outputting do not show up in the html of the webpage)
So can anyone point me in the right direction of how they would go about this?
(I will be putting this information into a mysql database which is why i added it as a tag, still dont know what language would be best to use though)
Edit: These requests on the server are somewhat random and although it shows the url that they come from when I try to visit the url in firefox it comes up trying to open something called application/jos
Jon, I am fairly certain that you are confusing several technologies here, and the simple answer is that it doesn't work like that. Firebug works specifically because it runs as part of the browser, and (as far as I am aware) runs under a more permissive set of instructions than a JavaScript script embedded in a page.
JavaScript is, for the record, different from Java.
If you are trying to log AJAX calls, your best bet is for the serverside application to log the invoking IP, useragent, cookies, and complete URI to your database on receipt. It will be far better than any clientside solution.
On a note more related to your question, it is not good practice to assume that everyone has read other questions you have posted. Generally speaking, "we" have not. "We" is in quotes because, well, you know. :) It also wouldn't hurt for you to go back and accept a few answers to questions you've asked.
So, the problem is?:
With someone else's web-page, hosted on someone else's server, you want to extract select information?
Using cURL, Python, Java, etc. is too painful because the data is continually updating via AJAX (requires a JS interpreter)?
Plain jQuery or iFrame intercepts will not work because of XSS security.
Ditto, a bookmarklet -- which has the added disadvantage of needing to be manually triggered every time.
If that's all correct, then there are 3 other approaches:
Develop a browser plugin... More difficult, but has the power to do everything in one package.
Develop a userscript. This is much easier to do and technologies such as Greasemonkey deal with the XSS problem.
Use a browser macro technology such as Chickenfoot. These all have plusses and minuses -- which I won't get into.
Using Greasemonkey:
Depending on the site, this can be quite easy. The big drawback, if you want to record data, is that you need your own web-server and web-application. But this server can be locally hosted on an XAMPP stack, or whatever web-application technology you're comfortable with.
Sample code that intercepts a page's AJAX data is at: Using Greasemonkey and jQuery to intercept JSON/AJAX data from a page, and process it.
Note that if the target page does NOT use jQuery, the library in use (if any) usually has similar intercept capabilities. Or, listening for DOMSubtreeModified always works, too.
If you're using a library such as jQuery, you may have an option such as the jQuery ajaxSend and ajaxComplete callbacks. These could post requests to your server to log these events (being careful not to end up in an infinite loop).
I have an HTML5/jquery mobile web app at http://app.bluedot.mobi. It is used for long distance races to track competitors via SPOT satellite tracking. The issue I have not yet resolved is that when loading the app when no data connection exists, the browser throws a "no data connection" alert popup as it is attempting to fetch the manifest during the checking event. Even when a data connection is present, loading the app can take a very long time. There are ~ 500 files to check. The fastest way to load the app (from a phone) is to be in airplane mode and dismiss the browser's alert - not so elegant.
Rather than force an update on users who tend to be in the backcountry with a spotty connection, I want to use applicationCache.update() programmatically, giving the user control over the process and speeding up app load whether on or offline.
Is this currently possible with the HTML5 spec and respective browser implementations?
Sounds like you need the abort() method. Unfortunately it is very new, and it will probably be a while before it is implemented by the majority of mobile browsers.
There are ~ 500 files to check.
It sounds like you're implying that the browser checks each file to see if there's any of them which has changed. This is not correct. The browser only checks the manifest file if that has changed, and that is a simple byte check. If the manifest file has not changed, the browser believes nothing has changed.
So if your application is slow to start, it might be your because your application is complex and there's alot of HTML and Javascript to parse. I would advise you to take a look at the application and see if there's anything you can optimize. In that case, you might want to take a look at Yahoo's Best Practices for Speeding Up Your Web Site page.
For example, I noticed you have a lot of Javascript code in the HEAD section. The beforementioned article advices us to move all Javascript (To the extent of what's possible) to the bottom of the page, so that the browser can start rendering the page as soon as possible. And there's a lot of other sound advice in the article. So take a look, I'm sure you'll find it useful. :-)
I am looking for ways to browse sites that are blocked by proxy filters at my location.
One solution i came up with was to build a page that would take a input of a URL and display the site in an iframe. Thus i would have a window into a browser on a page that is being displayed by my proxy. I was going to host this on my personal web site and use it to access restricted content. this way i have access to blogs, and forums where there is a wealth of information that is blocked by a backwards blanketed restriction list.
How can i make a web page similar to this? Would it be simple html and javascript, do I need .Net?
What you aim to do has to be done server-side. When you put a page in an iframe, your web browser loads it, and will do so just as if you went directly to the URL.
There is no way around this via client-side code, such as JavaScript.
If you truly want to reinvent the wheel, pick a language and look into whatever functions download files. No need to do this though when there are plenty of web-based proxy services, such as http://www.hidemyass.com.
Even if you loaded it in an iframe, the request for the page in the iframe will still go through the proxy and so you will still be blocked.
You'd have to do something like open a socket to the site through your web host and then download the content and redisplay it. That's assuming your host isn't also blocked. Also, you'll lose the benefits of cookies and sessions this way (ie. you won't be able to be logged into things unless the session id is in the query string).
The fastest and simplest solution would be to create a free Log Me In account at www.logmein.com. then setup your host computer at home, login from work, and browse freely. I do this myself at work so no one can see my personal browsing history when I dont want them to. This of course would only work if logmein.com was not a blocked site at your work. good luck!
It depends upon the "filter" complexity. If you have your own website that you can reach through the proxy or if your computer can run as a webserver, you could try accessing via a proxy script such as "CGIProxy." There are online services that do this too. However, some proxy filters can detect these methods as well and you'd still be out of luck. No javascript or HTML tricks can overcome the proxy filter.