Multiple URLs copy in Sources/Network tab - google-chrome

Is it possible to extract multiple resources' URLs in Sources or Network tab of Chrome Dev Tools?
When I want to get URL of a single resource, I can do it with context menu function Copy link address
I can switch to this resource from Network to Sources tab and vice versa, but what if I have a need to get URLs of multiple resources at once? It is very cumbersome to copy them manually if resultset consists of 200-300 resources.
What I've tried so far:
To copy the whole folder from a sources tab, but from this answer I found out it is not possible for now.
To use $(selector) as specified in the Console reference, in a form of
$('img')
in case we need to fetch image URLs.
The complexity of this approach is that it's often hard to distinguish target images on a page that has hundreds of them, and furthermore, multiple versions of the same image (views, previews, small-sized icons and etc.) I.e. to match the element inside the tag with the needed resource is not that easy, as it seems. Also not all the file types have dedicated tags (as in the case with img).
Maybe I should use src tag with some modifiers? Any other suggestions?

make sure Network panel is active
switch devtools Dock side in the menu to a detached (floating) window
Next time you can press CtrlShiftD to toggle docking.
in the now detached devtools press CtrlShifti or ⌘⌥i on MacOS,
which will open devtools-on-devtools in a new window
Run the following code in this new window:
copy(UI.panels.network.networkLogView.dataGrid.rootNode().flatNodes.map(n => n.request().url()).join('\n'))
It'll copy the URLs of all requests that match current filter to clipboard.
Hint: save the code as a Snippet and run it in devtools-on-devtools window via the commands palette, CtrlP or ⌘P then type the snippet's name.
In old Chrome the code was different:
copy(UI.panels.network._networkLogView._dataGrid._rootNode._flatNodes.map(n => n._request._url).join('\n'))
copy(UI.panels.network.networkLogView.dataGrid.rootNode().flatNodes.map(n => n.request().urlInternal).join('\n'))

I found the above method too clunky, its way easier to use fiddler:
open fiddler (possibly install it)
filter on the domain name that you are interested in
clear the screen
refresh the web page
Select the fiddler output
right click and select copy just the URL's

Based on #wOxxOm, this worked for me:
var nodes = UI.panels.network._networkLogView._dataGrid._rootNode._flatNodes,
urls = [];
nodes.forEach(function() {
var req = arguments[0]._request;
if (req !== undefined) {
urls.push(req.url());
}
});

A selection and simple copy (Ctrl+C) work for me.
I select URLs in the Url column by the mouse.
Then I use the context menu to copy the list to the clipboard.
The clipboard contents then I can paste to Excel and get the URL list. It adds some empty lines though.

Related

Script error for "Multiple URLs copy in Sources/Network tab" [duplicate]

Is it possible to extract multiple resources' URLs in Sources or Network tab of Chrome Dev Tools?
When I want to get URL of a single resource, I can do it with context menu function Copy link address
I can switch to this resource from Network to Sources tab and vice versa, but what if I have a need to get URLs of multiple resources at once? It is very cumbersome to copy them manually if resultset consists of 200-300 resources.
What I've tried so far:
To copy the whole folder from a sources tab, but from this answer I found out it is not possible for now.
To use $(selector) as specified in the Console reference, in a form of
$('img')
in case we need to fetch image URLs.
The complexity of this approach is that it's often hard to distinguish target images on a page that has hundreds of them, and furthermore, multiple versions of the same image (views, previews, small-sized icons and etc.) I.e. to match the element inside the tag with the needed resource is not that easy, as it seems. Also not all the file types have dedicated tags (as in the case with img).
Maybe I should use src tag with some modifiers? Any other suggestions?
make sure Network panel is active
switch devtools Dock side in the menu to a detached (floating) window
Next time you can press CtrlShiftD to toggle docking.
in the now detached devtools press CtrlShifti or ⌘⌥i on MacOS,
which will open devtools-on-devtools in a new window
Run the following code in this new window:
copy(UI.panels.network.networkLogView.dataGrid.rootNode().flatNodes.map(n => n.request().url()).join('\n'))
It'll copy the URLs of all requests that match current filter to clipboard.
Hint: save the code as a Snippet and run it in devtools-on-devtools window via the commands palette, CtrlP or ⌘P then type the snippet's name.
In old Chrome the code was different:
copy(UI.panels.network._networkLogView._dataGrid._rootNode._flatNodes.map(n => n._request._url).join('\n'))
copy(UI.panels.network.networkLogView.dataGrid.rootNode().flatNodes.map(n => n.request().urlInternal).join('\n'))
I found the above method too clunky, its way easier to use fiddler:
open fiddler (possibly install it)
filter on the domain name that you are interested in
clear the screen
refresh the web page
Select the fiddler output
right click and select copy just the URL's
Based on #wOxxOm, this worked for me:
var nodes = UI.panels.network._networkLogView._dataGrid._rootNode._flatNodes,
urls = [];
nodes.forEach(function() {
var req = arguments[0]._request;
if (req !== undefined) {
urls.push(req.url());
}
});
A selection and simple copy (Ctrl+C) work for me.
I select URLs in the Url column by the mouse.
Then I use the context menu to copy the list to the clipboard.
The clipboard contents then I can paste to Excel and get the URL list. It adds some empty lines though.

Check if a new if a new tab was opened with puppeteer

I am using Puppeteer to gather a list of companies that hire remotely. The page I am trying to collect the data from has a call to action that is not an <a href="" blank"_tab", but a <button></button>. The button doesn't have any data attribute either, that gives a hint as to what the URL of the page that gets opened in a new tab, when I click the button, is. I am now trying to find a way to catch the opening of the new tab, by clicking the said button, to then fetch the URL of the newly opened tab. Is this possible with puppeteer?
The solution is to extract all pages of the browser via the browser.pages() method call.
In your puppeteer project you always start by initialising the browser. The puppeteer virtual browser does contain the method pages(). Usage of pages().
What you have to do now: At first get the current url of the current page you are on. Then stream the result of the browser.pages() method. While streaming the array you can just get the index of your current tab. The tab you just opened using the button is the page with that index + 1 (the next element in the array)...
How to handle multiple tabs
Puppeteer Documentation about pages()

Duplicate a tab in Chrome without Reloading the Page?

Is there any way to completely duplicate the state of a current tab in Google Chrome? I want an exact copy of the current state of the page without having to reload the page in another tab.
An example use case:
While browsing a "slideshow" on a news website, I want to preserve the current slide that I'm on, but create a duplicate so that I can continue viewing the next slide. If I simply Right-Click and "Duplicate" the tab, the new page will completely Reload, reprocessing all of the Javascript and running the pre-slideshow advertisement again.
In short "NO" you can't.
I am not expert on this
but a similar behavior can be achieved in some ways i know :
Dump the whole DOM
Never tried this though. You can convert the DOM to a string, pass it to the new window and then parse it as a document. This will let you lose your DOM events and State manipulation javascript. (But that's good for your case)
var dtab = window.open('about:blank', 'duplicate_a_tab');
dtab.document.open();
dtab.document.write("... yout html string ..");
dtab.document.close();
Develop an extension
Let the users continue on the current tab with the current state, your extension should be able to capture the screenshot of that area and open that screenshot in new tab. There are plenty of screenshot taking extensions are available in the market.
If that website is your own
You can develop your services that uses state locally like progressive web apps. Give a link separately to 'duplicate' which will eventually open the same URL in different tab with the same local state and with the flag do-not-sync.
This will not work when the user uses browser inbuilt duplicate
feature.

Chrome Extension Local Storage: Background.html doesn't have access to localstorage?

Writing and reading from LocalStorage is working fine from my popup and tab. However, when I attempt to add a value from my background page, it doesn't seem to write at all. I'm viewing local storage in Chrome Developer Tools by refreshing and looking for the value to show.
In the following example code for background.html 'lastId' is displayed correctly in the alert when a new bookmark is added. However, the value is not stored. Additionally, the request for a known value appears to fail with no alert displaying. (Results are the same attempting both syntaxes shown below.)
<html>
<script>
// Grab the id of newly created bookmarks
chrome.bookmarks.onCreated.addListener(function(id) {
var lastId = id;
alert(lastId);
localStorage['lastId'] = lastId;
var testvalue = localStorage['309'];
alert(testvalue);
localStorage.setItem('lastId', lastId);
var testvalue2 = localStorage.getItem('309');
alert(testvalue2);
});
</script>
</html>
I keep thinking I must just be missing some small syntax issue or something but can't see anything. If my manifest declaration was incorrect I don't think the alert would work for the id. Stumped...
UPDATE: Turns out that you have to force reload of the extension on updates to background pages since they are persistent in browser memory when opened. That is why my saved code appeared not to work. It wasn't refreshed and I am duly embarrassed.
Hm, I can think about couple things.
You say that it works in a tab and a popup. This is very strange because it shouldn't (if by tab you mean content script). Content scripts are able to access only localStorage that belongs to a site they are injected. Popup, background, option pages, and any othe page from extension's folder can only access extension's own localStorage. Those two local storages and completely separated. So maybe you are inspecting wrong localStorage?
To see extension's own localStorage you need to inspect background or popup page and check resources tab in the inspector. To inspect site or content script localStorage you need to open regular inspector on the page.
Second moment is your localStorage assignment might be not what you are expecting.
If you run:
var lastId = 5;
localStorage['lastId'] = lastId;
you will get value 5 assigned to lastId property. So to read written value you need to run:
alert(localStorage['lastId']); //not localStorage['5']
If you want to store arrays then you would need to serialize/unserialize them through JSON as localStorage can store only strings.

Is it possible to determine a tab's opener within a Google Chrome extension?

I am looking for a way to determine a given tab's opener (parent tab) within a Google Chrome extension.
I've looked at the documention for Tab but there doesn't really seem to be anything that would yield this information. http://code.google.com/chrome/extensions/tabs.html
I've tried injecting this content script into pages (thinking I could pass the value to my background page):
alert(window.opener);
.. but it just yields null.
The best thing I've come up with so far is to keep track of the currently focused tab, and whenever a new tab is created, just assume that the focused tab is the opener/parent of the new tab. I believe this would de facto identify the parent tab correctly most of the time since background tabs rarely (are allowed to) open new pages. However, it seems kludgey and potentially inaccurate at times -- for example, if another extension opened a new tab, this method may misidentify the new tab's opener.
Update: it is now possible to reliably determine a tab's opener tab within a Chrome extension natively using the newly added webNavigation API, and specifically by hooking the onCreatedNavigationTarget event.
https://code.google.com/chrome/extensions/trunk/webNavigation.html
Chrome has added an experimental extension API which can accomplish this -- specifically webNavigation.onBeforeRetarget. http://code.google.com/chrome/extensions/experimental.webNavigation.html
However since this is still experimental (not usable in Chrome stable versions or releasable on the Chrome Web Store) I have ended up using another approach.
Basically:
In content_script.js:
chrome.extension.sendRequest({
request: {
op: "pageLoadStarted",
url: document.location.href,
referrer: document.referrer
}
});
In background.html:
chrome.extension.onRequest.addListener(function(request, sender, sendResponse) {
console.log('onRequest: ' + JSON.stringify(request));
console.log(JSON.stringify(sender));
});
This approach allows me to obtain the referrer of a tab, which I can then correlate with an existing tab's url. This isn't always a one-to-one mapping, so I do some additional magic such as preferring the currently selected tab as the opener if its url matches the new tab's referrer.
This really is just a hack to approximate the functionality that would be provided more simply and accurately by webNavigation.onBeforeRetarget or window.opener.
Further investigation has revealed that onCreatedNavigationTarget() does not always fire when you think it would to indicate an opener-opened relationship.
An additional hint can sometimes be found in the Tab object returned by chrome.tabs.onCreated/onUpdated in the .openerTabId parameter.
A comprehensive solution will probably have to rely on multiple of the methods described in these answers.
port.onMessage.addListener(
function(msg) {
var tabid = port.sender.tab.openerTabId;
console.log("Received message from tab that was opened by tab id : " + tabid);
// reliably shows the tab id of the tab that opened
// the tab sending the message
});