I am using an online eduPad for collaboratively editing some texts. eduPad is a system that lets quickly create a text-only document that can be edited. It is based on Etherpad, but using the Etherpad Api fails because I could not figure out the Api key. Each document has a specific URL under which the textfile is shown. I would like to automatically download the eduPad page's content using a command line tool like wget. However, when I load the page with
wget <address of eduPad page>
I only get the HTML page with the scripts to display the data, but not my text. Is it possible to download (and refresh) the contents of an eduPad page using wget or a similar tool?
As an example, the page https://edupad.ch/p/xomgoEj4NY currently contains the text above.
If your site is generated by javascript, you need a JS aware tool, like puppeteer nodejs library:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({headless: true});
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'networkidle2' });
const html = await page.evaluate(() => document.documentElement.outerHTML);
console.log(html);
await browser.close();
})();
Usage
npm i puppeteer # install puppeteer library as a dependency for nodejs
nodejs wget.js
Related
I'm working on the video stream web app like youtube in firebase hosting and storage in nodejs.
I'm able to host the video player by getting the firebase storage download url location from client side and can play the video. But I don't want my users to see the url by inspecting the web page. For that I made a app.get("/video/videoID") in nodejs express and getting the download url server side, and I'm able to get the download URL. But I'm not able to send the URL to my video player. I tried res.sendFile("URL path"); but it is saying that the "Path should be absolute". I understand that the path should come from the server local machine.
Is there any way where I can send the URL to my video source path from server?
HTML file:
<video width="100%" controls><source src="/video/songID" type="video/mp4">Your browser does not support HTML video.</video>
Server code:
app.get("/video/:songKey", function(req, res, next) {
var fDb = admin.database();
var ref = fDb.ref("videos");
ref.orderByKey().equalTo(req.params.songKey).on('child_added', async (snapshot) => {
var refPath = snapshot.val().video_ref_path; // This is the firebase storage ref path stored in realtime database
var storage = admin.storage();
var songFile = storage.bucket("<bucket-name>").file(refPath);
var [meta] = await songFile.getMetadata();
var token = meta.metadata.firebaseStorageDownloadTokens;
var link = `https://firebasestorage.googleapis.com/v0/b/<bucket_name>/o/${encodeURIComponent(
refPath
)}?alt=media&token=${token}`
res.sendFile(link); //This line is giving error saying path needs absolute path
// If I use res.send(link) call the web location I'm able to get the file URL
});
});
Any help with this is appreciated and thankful. BTW I'm new to nodejs and learning it.
Thanks in Advance
res.sendFile() is meant to send the contents of locally (to the server) stored files.
You probably want to use res.redirect(), which is meant to redirect the browser to another URL.
Over the years on snapchat I have saved lots of photos that I would like to retrieve now, The problem is they do not make it easy to export, but luckily if you go online you can request all the data (thats great)
I can see all my photos download link and using the local HTML file if I click download it starts downloading.
Here's where the tricky part is, I have around 15,000 downloads I need to do and manually clicking each individual one will take ages, I've tried extracting all of the links through the download button and this creates lots of Urls (Great) but the problem is, if you past the url into the browser then ("Error: HTTP method GET is not supported by this URL") appears.
I've tried a multitude of different chrome extensions and none of them show the actually download, just the HTML which is on the left-hand side.
The download button is a clickable link that just starts the download in the tab. It belongs under Href A
I'm trying to figure out what the best way of bulk downloading each of these individual files is.
So, I just watched their code by downloading my own memories. They use a custom JavaScript function to download your data (a POST request with ID's in the body).
You can replicate this request, but you can also just use their method.
Open your console and use downloadMemories(<url>)
Or if you don't have the urls you can retrieve them yourself:
var links = document.getElementsByTagName("table")[0].getElementsByTagName("a");
eval(links[0].href);
UPDATE
I made a script for this:
https://github.com/ToTheMax/Snapchat-All-Memories-Downloader
Using the .json file you can download them one by one with python:
req = requests.post(url, allow_redirects=True)
response = req.text
file = requests.get(response)
Then get the correct extension and the date:
day = date.split(" ")[0]
time = date.split(" ")[1].replace(':', '-')
filename = f'memories/{day}_{time}.mp4' if type == 'VIDEO' else f'memories/{day}_{time}.jpg'
And then write it to file:
with open(filename, 'wb') as f:
f.write(file.content)
I've made a bot to download all memories.
You can download it here
It doesn't require any additional installation, just place the memories_history.json file in the same directory and run it. It skips the files that have already been downloaded.
Short answer
Download a desktop application that automates this process.
Visit downloadmysnapchatmemories.com to download the app. You can watch this tutorial guiding you through the entire process.
In short, the app reads the memories_history.json file provided by Snapchat and downloads each of the memories to your computer.
App source code
Long answer (How the app described above works)
We can iterate over each of the memories within the memories_history.json file found in your data download from Snapchat.
For each memory, we make a POST request to the URL stored as the memories Download Link. The response will be a URL to the file itself.
Then, we can make a GET request to the returned URL to retrieve the file.
Example
Here is a simplified example of fetching and downloading a single memory using NodeJS:
Let's say we have the following memory stored in fakeMemory.json:
{
"Date": "2022-01-26 12:00:00 UTC",
"Media Type": "Image",
"Download Link": "https://app.snapchat.com/..."
}
We can do the following:
// import required libraries
const fetch = require('node-fetch'); // Needed for making fetch requests
const fs = require('fs'); // Needed for writing to filesystem
const memory = JSON.parse(fs.readFileSync('fakeMemory.json'));
const response = await fetch(memory['Download Link'], { method: 'POST' });
const url = await response.text(); // returns URL to file
// We can now use the `url` to download the file.
const download = await fetch(url, { method: 'GET' });
const fileName = 'memory.jpg'; // file name we want this saved as
const fileData = download.body; // contents of the file
// Write the contents of the file to this computer using Node's file system
const fileStream = fs.createWriteStream(fileName);
fileData.pipe(fileStream);
fileStream.on('finish', () => {
console.log('memory successfully downloaded as memory.jpg');
});
I am using the Coverage tab at my Chrome Dev Tools and I have a really big file and after playing a lot with Coverage it's clear enough that only 15% enough of my CSS code is being used (I simulated button presses, hover menus...).
The problem is getting hat 15% of code OUT of the Coverage tab. I cant believe the Devs behind this really nice feature didnt think an easy way for the end user copy only the green part of the code. Check image attached.
Do you have any idea how I could do that? I read something about using Puppeteers but it requires lots of preparation. On latest Canary version it looks like I can export a JSON but it would require some time to code a parser to that JSON in order to extract only the needed part.
Thanks to an article by Phillip Kriegel (https://www.philkrie.me/2018/07/04/extracting-coverage.html) I managed to setup Puppeteer to extract the coverage CSS from a URL and output that CSS into a file.
Here's how to do it:
Step 1: Install node.js globally
Step 2: Create a folder on your desktop
Step 3: Inside the folder install the Node Package Manager (NPM) and the Puppeteer node module
Step 4: Create a JavaScript file inside the folder, name it coverage.js
Step 5: Put this code inside that js file:
const puppeteer = require('puppeteer');
// Include to be able to export files w/ node
const fs = require('fs');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
// Begin collecting CSS coverage data
await Promise.all([
page.coverage.startCSSCoverage()
]);
// Visit desired page
await page.goto('https://www.google.com');
//Stop collection and retrieve the coverage iterator
const cssCoverage = await Promise.all([
page.coverage.stopCSSCoverage(),
]);
//Investigate CSS Coverage and Extract Used CSS
const css_coverage = [...cssCoverage];
let css_used_bytes = 0;
let css_total_bytes = 0;
let covered_css = "";
for (const entry of css_coverage[0]) {
css_total_bytes += entry.text.length;
console.log(`Total Bytes for ${entry.url}: ${entry.text.length}`);
for (const range of entry.ranges){
css_used_bytes += range.end - range.start - 1;
covered_css += entry.text.slice(range.start, range.end) + "\n";
}
}
console.log(`Total Bytes of CSS: ${css_total_bytes}`);
console.log(`Used Bytes of CSS: ${css_used_bytes}`);
fs.writeFile("./exported_css.css", covered_css, function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
});
await browser.close();
})();
Step 6: BE SURE TO REPLACE the URL at this point in the code await page.goto('https://www.google.com'); with your desired URL
Step 7: In the command line tool (Git Bash) type node coverage.js
A file called exported_css.css will be created, it will contain all your coverage CSS for the URL you set in the code.
CAVEAT: This will extract the coverage CSS from ALL the CSS assets that are loaded from the URL you set. You will then have to further optimize that CSS (not covered in this example).
Open Chrome Tab --> Inspect Element (F12) --> Press Escape button
I'm in the process of creating a PHP script that parses the Coverage JSON exported file, and outputs the extracted used CSS/JS only. Unfortunately I have come across a snag, at some point the JSON parser loses the correct character number, and I end up with broken or incorrect CSS/JS syntax. It's only off by a few characters, but the amount of characters that it is off by is variable so it's almost impossible to predict it during parsing.
I'm not positive, but I think the issue results from PHP running on the server, and the server may read the characters in the original CSS file differently than a browser would. I'm going to attempt to write a Coverage JSON parser in JavaScript. When I do I'll be sure to post the code here for all to use.
Sorry I couldn't be of more help, I just wanted to warn away people from using PHP to do this as it seems to not read character numbers correctly in large CSS files.
I have a single page web application using React and materialize-css and I would like to export it as static HTML and CSS so that it is possible to easily edit HTML for the purpose of prototyping. Is it possible to export at least a snapshot of current state?
I tried "save page" in Firefox and Chrome, but it does not provide good results.
Follow the following steps :-
1. In brouser, got to the developer tools,
2. select Inspector(firefox)/Elements(chrome),
3. then select the tag HTML, right click on it,
4. then click Edit as HTML.
Now you can copy all the code and save it. While the color and shape of the document remains, you will miss the pictures.
Good luck ! :)
Probably not ideal, but you can store the entire page as a variable and download it. Run this in your browser console after the page has loaded:
var pageHTML = document.documentElement.outerHTML;
var tempEl = document.createElement('a');
tempEl.href = 'data:attachment/text,' + encodeURI(pageHTML);
tempEl.target = '_blank';
tempEl.download = 'thispage.html';
tempEl.click();
The ReactDOMServer module contains a function for rendering a React application to static HTML - it's designed for use on the server, but I don't think there's anything to stop you using it in the browser (don't do this in production though!)
import ReactDOMServer from "react-dom/server";
import App from "./yourComponent";
document.body.innerHTML = ReactDOMServer.renderToStaticMarkup(App);
var pageHTML = window.document.getElementById('divToPDF').innerHTML;
let data = new Blob([pageHTML], {type: 'data:attachment/text,'});
let csvURL = window.URL.createObjectURL(data);
let tempLink = document.createElement('a');
tempLink.href = csvURL;
tempLink.setAttribute('download', 'Graph.html');
tempLink.click();
You can build your code and host it on github.io. The following tutorial will help you achieve that. You can then use the generated code in the gh-pages branch as your exported HTML
This was the first thread I found on SW.. so I think it would be appropriate to copy my own answer from another thread: https://stackoverflow.com/a/72422258/1215913
async function saveToFile() {
const handle = await showSaveFilePicker({
suggestedName: 'index.html',
types: [{
description: 'HTML',
accept: {'text/html': ['.html']},
}]
});
const writable = await handle.createWritable();
await writable.write(document.body.parentNode.innerHTML);
writable.close();
}; saveToFile();
for more info check the source answer
I had done this before but was stuck and couldn't seem to find the documentation anywhere. My scenario was I had a react js SPA and needed to create a static build to run without a server (through an organisations SharePoint using a static doc repository).
It is pretty simple in the end, run
npm run build
in your project directory and it will create the static build in a 'build' folder ready for you to dump wherever needed.
Reference: https://create-react-app.dev/docs/production-build/
I'm currently developing my first Chrome app that we'll be used as a Kiosk app later.
I'm trying to read a file at the startup of the app, that file is a config file (.json). It contains values that will be passed inside a URL once the app has launched (ie: www.google.com/key=keyValueInTheJsonFile).
I used https://developer.chrome.com/apps/fileSystem (the method "chooseEntry" especially) to be able to read a file, but in my case I would like to directly specify the path/name of the file and not ask the user to select a file. Like that I can pass the values to the redirected URL at the startup.
Any idea of how I could possibly do that?
Thanks!
If your file is in the package you can read it using simple XHR or Fetch.
You can't use web filesystem since it has different purpose and Chrome filesystem (user's FS) won't work here either since it needs a user interaction.
Use function getURL to get a full URL to the resource and then make XHR call:
var rUrl = chrome.runtime.getURL('file.json');
fetch(rUrl).then((response) => {
return response.json();
})
.then((fileContent) => {
// the content
})
.catch((cause) => console.log(cause));