I am developing web crawler which could render Javascript websites and so I decided to use PupeeteerSharp, a .NET port of popular Node.JS headless Chrome browser Pupeeteer API. I am running Service Fabric's local development cluster on Windows 10 development machine and have one stateless service in my solution.
I've created Data folder under Service project's PackageRoot folder and put .local-chromium folder contents there (contains chrome.exe executable) so it deploys as independent data package of service.
I've also placed this XML config line in ServiceManifest.xml file:
<DataPackage Name="Data" Version="1.0.0" />
So far it looks good and headless browser content is copied to SFCluster Data package directory properly.
Then in my Stateless Service code I try to call Pupeeteer chromium executable as follows:
var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
Headless = true,
ExecutablePath = _chromiumPath // #$"{context.CodePackageActivationContext.GetDataPackageObject("Data").Path}\.local-chromium\Win64-706915\chrome-win\chrome.exe"
});
using (var page = (await browser.NewPageAsync()))
{
Response renderResponse;
try
{
renderResponse = await page.GoToAsync(webPage.AbsoluteUri, timeout);
if (renderResponse.Status != System.Net.HttpStatusCode.OK)
{
return new RenderResult(RenderStatus.OtherFailure);
}
// other code
}
catch (TimeoutException)
{
return new RenderResult(RenderStatus.Timeouted);
}
In this line: using (var page = (await browser.NewPageAsync())) my code (Thread) simply hangs without returning, in Debug console I see many thread exits, but no exception occurs. I was previously getting System.IO.FileNotFoundException when I was fixing some other errors regarding appropriate copying of chromium folder contents, but now these errors are gone so it seems that code find .exe but somehow cannot start headless mode of PupeeterSharp.
Does that mean that I cannot simply run external .exe chromium binary with Service Fabric's Native Application Model? Should I use Docker and Linux containers instead?
Related
I am programmatically starting an IPFS node using JS ipfs-core(npm package) with a custom repository using a different storage backend(similar to S3). Now once the node is started in the AWS instance, I want to send requests to the node using a remote client written in Java.
java-ipfs-http-client can connect to the API port. But, the API and gateway service does not get initiated when the node is started. The Java server will be running on a different machine.
Is it possible to access the ipfs node started using ipfs-core programmatically from a java server running on a different instance?
Found the solution.
When we initialize node programmatically, we need to manually start API/Gateway in the following way.
import * as IPFS from 'ipfs-core'
import { HttpApi } from 'ipfs-http-server'
import { HttpGateway } from 'ipfs-http-gateway'
async function startIpfsNode () {
const ipfs = await IPFS.create()
const httpApi = new HttpApi(ipfs)
await httpApi.start()
const httpGateway = new HttpGateway(ipfs)
await httpGateway.start()
}
startIpfsNode()
This will start the ipfs node along with the API and Gateway
The configuration of API and Gateway port can be changed programmatically in the following way
const ipfs = IPFS.create()
await ipfs.config.set('Addresses.API', '/ip4/127.0.0.1/tcp/5002');
await ipfs.config.set('Addresses.Gateway', '/ip4/127.0.0.1/tcp/9090');
Once the API is started, the IPFS node can accessed from a Java Program using java-ipfs-http-client
My unit test launch looks like this. As you can see I have exploited CLI options to install a VSIX my CICD has already produced, and then also tried to install ms-vscode-remote.remote-ssh because I want to re-run the tests on a remote workspace.
import * as path from 'path';
import * as fs from 'fs';
import { runTests } from '#vscode/test-electron';
async function main() {
try {
// The folder containing the Extension Manifest package.json
// Passed to `--extensionDevelopmentPath`
const extensionDevelopmentPath = path.resolve(__dirname, '../../');
// The path to the extension test runner script
// Passed to --extensionTestsPath
const extensionTestsPath = path.resolve(__dirname, './suite/index');
const vsixName = fs.readdirSync(extensionDevelopmentPath)
.filter(p => path.extname(p) === ".vsix")
.sort((a, b) => a < b ? 1 : a > b ? -1 : 0)[0];
const launchArgsLocal = [
path.resolve(__dirname, '../../src/test/test-docs'),
"--install-extension",
vsixName,
"--install-extension",
"ms-vscode-remote.remote-ssh"
];
const SSH_HOST = process.argv[2];
const SSH_WORKSPACE = process.argv[3];
const launchArgsRemote = [
"--folder-uri",
`vscode-remote://ssh-remote+testuser#${SSH_HOST}${SSH_WORKSPACE}`
];
// Download VS Code, unzip it and run the integration test
await runTests({ extensionDevelopmentPath, extensionTestsPath, launchArgs: launchArgsLocal });
await runTests({ extensionDevelopmentPath, extensionTestsPath, launchArgs: launchArgsRemote });
} catch (err) {
console.error(err);
console.error('Failed to run tests');
process.exit(1);
}
}
main();
runTests downloads and installs VS Code, and passes through the parameters I supply. For the local file system all the tests pass, so the extension from the VSIX is definitely installed.
But ms-vscode-remote.remote-ssh doesn't seem to be installed - I get this error:
Cannot get canonical URI because no extension is installed to resolve ssh-remote
and then the tests fail because there's no open workspace.
This may be related to the fact that CLI installation of multiple extensions repeats the --install-extension switch. I suspect the switch name is used as a hash key.
What to do? Well, I'm not committed to any particular course of action, just platform independence. If I knew how to do a platform independent headless CLI installation of VS Code:latest in a GitHub Action, that would certainly do the trick. I could then directly use the CLI to install the extensions before the tests, and pass the installation path. Which would also require a unified way to get the path for vs code.
Update 2022-07-20
Having figured out how to do a platform independent headless CLI installation of VS Code:latest in a GitHub Action followed by installation of the required extensions I face new problems.
The test framework options include a path to an existing installation of VS Code. According to the interface documentation, supplying this should cause the test to use the existing installation instead of installing VS Code; this is why I thought the above installation would solve my problems.
However, the option seems to be ignored.
My latest iteration uses an extension dependency on remote-ssh to install it. There's a new problem: how to get the correct version of my extension onto the remote host. By default the remote host uses the marketplace version, which obviously won't be the version we're trying to test.
I would first try with only one --install-extension option, just to check if any extension is installed.
I would also check if the same set of commands works locally (install VSCode and its remote SSH extension)
Testing it locally (with only one extension) also allows to check if that extension has any dependencies (like Remote SSH - Editing)
Problem Summary
Storybook snapshot test on static storybook returning blank screenshots even though they look fine on localhost:8080 when I ran npx http-server storybook-static
Tech stack and relevant code
Vue 3
Vite
Storybook
Jest
Storyshots
Puppeteer
I have components and their respective stories. npm run storybook works perfectly fine. My storybook.spec.js is as follows:
import { imageSnapshot } from "#storybook/addon-storyshots-puppeteer"
import initStoryshots from "#storybook/addon-storyshots"
initStoryshots({
suite: "Image storyshots",
test: imageSnapshot(
storybookUrl: 'file://absolute/path/to/my/storybook-static'
)
})
I ran the following. fyi, I did not modify any file in storybook-static after running npm run build-storybook.
npm run build-storybook
npm run test
npm run test constitutes jest --config=jest.config.js test
Problem
Unfortunately, the screenshots I get are all blank and fail the snapshot test.
I suspect it might be due to a CORS error just like other Storybook users when they click <project-root>/storybook-static/index.html after running npm run build-storybook, to which I want to ask for a solution as well, because I wanna run test remotely on a headless server.
Note
I used absolute path because relative path caused a resource not found error during the testing process.
The problem is that you're running the tests from file:// instead of http://. So the URI is file:// and the img url ends up like this after applying some url logic: path.resolve(window.location, '/your-image.png') file:///your-image.png.
If this is the case you could change to http://. You can start a express server and serve the storybook-static folder from setupGlobal and then shut it down in teardownGlobal. Then you will need to change your storybookUrl to http://localhost:<some-port>.
None of the images were loading within my pipeline but worked fine locally, ended up being because the components were fetching images using a relative path <img src="/my-image" /> which apparently is not allowed using the file protocol.
I ended up doing 2 things:
Updating the static dirs directory to use the root by updating the main.js file in storybook
module.exports = {
staticDirs: [{ from: '../static', to: '/' }],
}
Added a script to remove the leading slash of images in the preview-head.html file from storybook
<script>
document.addEventListener('DOMContentLoaded', () => {
Array.from(document.querySelectorAll('img')).forEach((img) => {
const original = img.getAttribute('src');
img.setAttribute('src', original.replace('/', ''));
});
});
</script>
Another (arguably better) approach would be to run the tests through a server where you can access the images
I am using the gulp-aemsync plugin to sync my css and js changes to a clientlib on an AEM instance. A have a gulp task watching the js and css that runs gulp-aemsync fine (changes are on the site when i refresh), but being a bit lazy as i am it would be nice to get live reload working so that i never have to manually refresh the page while working.
I have tried to follow both these 2 online guides:
https://adobe-consulting-services.github.io/acs-aem-tools/features/live-reload/index.html
https://www.cognifide.com/our-blogs/cq/up-and-running-with-livereload-in-adobe-aem6
Followed the steps of:
installing Netty package on AEM instance
installing ACS AEM tools package on the AEM instance
installing the RemoteLiveReload chrome extension (the AEM instance is hosted on AWS)
That didn't work, so i got one of our DevOps engineers to open port 35729 (which is the default for Livereload) on the AEM instance. That still doesn't work, and when i click the chrome browser extension to sync it i get the following message:
Could not connect to LiveReload server. Please make sure that LiveReload 2.3 (or later) or another compatible server is running.
Can anyone help me figure this out as i'd really like to get it working to streamline my workflow.
Thanks
DISCLAIMER: This answer is based on a setup I had working at some point, and by no means is a complete/working answer. But it should give you an alternative to the other tools that exist and get you half way there.
I have not used the tools you are mentioning, but since you are using gulp and aemsync, you could do the following:
In your gulp setup, create a websocket server and basically make that server publish messages everytime aemsync is triggered to push content to AEM.
// start a websocket server
const WebSocket = require('ws'); // requires "npm install ws"
const wss = new WebSocket.Server({ port: 8081 });
const connections = [];
wss.on('connection', function connection(ws) {
connections.push(ws); // keep track of all clients
// send any new messages that come to this server, to all connected clients
ws.on('message', (d) => connections.forEach(connection => connection.send(d)));
});
// create a new websocket to send messages to the websocket server above
const ws = new WebSocket('ws://localhost:8081');
// send a regex to the server every second
// NOTE: CHANGE this to run when aemsync is triggered in your build
setInterval( () => ws.send('reload'), 1000 );
Then in your JS code (on AEM) or really in a <script> tag that you make sure will NOT go beyond your local (or dev/prod) you can setup a websocket listener to refresh the page:
socket = new WebSocket('ws://localhost:8081');
socket.onopen = // add function for when ws is open
socket.onclose = // add function for when ws is closed
socket.onerror = // add function for when ws errors
// listen to messages and reload!
socket.addEventListener('message', function (event) {
location.reload();
});
Alternatively, you could use the chrome plugin I've developed:
https://github.com/ahmed-musallam/websocket-refresh-chrome-ext
It's not perfect by any means. However, for a basic setup, it should work great! an you don't need to touch your AEM JS.
My first Windows Store App is almost ready to be uploaded. It passed the Windows App Certification Kit test on my machine. However if I "build packages to upload to Windows Store", it fails the test due to "Crashes and hangs". I found out that the WACK deletes my package folder before running the test, and because the app needs to load data from there upon start, it will probably not find it and thus hangs.
My questions are:
What happens to the files in my Local folder (i.e. C:\Users\...\AppData\Local\Packages\packageName\LocalState) when the app is packaged?
If I need to read and write data from files upon app start, where should I keep these files?
Thank you a lot for your time!
I am using Visual Studio 2013 Update 2, WACK 3.4.
Here is some code that runs when my app starts, I try to deserialize an existing file in the Local folder:
StorageFolder localFolder = ApplicationData.Current.LocalFolder;
StorageFolder demoFolder = await localFolder.CreateFolderAsync("Demos", CreationCollisionOption.OpenIfExists);
StorageFolder gameRecordFolder = await demoFolder.CreateFolderAsync("GameRecords", CreationCollisionOption.OpenIfExists);
string filename = gameRecName+".xaml";
StorageFile gameFile = await gameRecordFolder.GetFileAsync(filename);
GameRecord g;
using (Stream fileStream = await gameFile.OpenStreamForReadAsync())
{
XmlSerializer serializer = new XmlSerializer(typeof(GameRecord));
g = (GameRecord)serializer.Deserialize(fileStream);
}