I am setting up github actions, and struggling to see how to handle the dependencies of a particular action.
Below is a simplified view of my repository setup, with the action and workflow YAML files as I currently understand them.
my-org/my-tool#master: a specific tool in my organisation that I want to enable as a github action
package.json
tool.js: the tool, depending on npm packages described in package.json
action.yml
name: my-tool
inputs:
file:
description: file to be processed by my-tool
required: true
runs:
using: node16
main: tool.js
my-org/my-code#branch: a code repository where I want to run the tool as an action
my-code.js: some code that needs to be processed by the tool
.github/workflows/use-tool.yml:
# ...
jobs:
tool:
steps:
- uses: my-org/my-tool#master
with:
file: my-code.js
# ...
If my-tool has no dependencies, this setup works well enough. But my-tool needs to have dependencies installed to function properly, i.e., a single npm install in its root directory. How do I specify this dependency in a github action setup?
I have multiple options, none of which are truly satisfying to me:
I can define a workflow that checks out my-tool, installs the dependencies, and then runs it. However, I don't want that logic to be in the workflow yaml of the my-code repository, as I have multiple similar repositories, all of which require my-tool, which will result in duplication. I don't see how to do this in the action.yml file itself. It seems I have to choose between a javascript action to run node tools, or a composite action to run shell tools.
I can bundle the node_modules directory in the my-tool repository; this bloats my repos unnecessarily.
I can bundle an ncc distribution of my-tool in its repo; same problem.
I can define and build a container image of my-tool with its dependencies installed, and use a container action to run the tool on my-code. This seems like overhead to me, and I have no idea how this container would access the files from the my-org/my-code repo (which would be its core purpose).
Options 2 and 3 are the ones described in the example published by github. I would be happy with using an artifact (either node_modules dir or ncc distribution) as the basis for an action, but there seems to be no way to do this.
My specific use case concerns npm dependencies, but there seems to be a generic use case for "action with dependencies", support for which is unclear to me from reading the documentation. I can't be the only dev in the world working with a toolset with dependencies of its own. Maybe I am looking at this setup in the wrong way. Is there a best practice for actions with dependencies? Is there a best practice to only have actions with zero dependencies?
I look forward to learn from any partial answer or suggestion.
Related
Editor's note: The question's original title was "Use npm install to install node modules stored on a local directory", which made the desire to transparently redefine the installation source less obvious. Therefore, some existing answers suggest solutions based on modifying the installation process.
I know this is a simple thing, but I'm quite new to anything in this area so after searching around and constantly finding answers that weren't really what I want I figured I'd just ask directly.
I currently have a process that is run in directory FOO that calls npm install. Directory FOO contains a package.json and a npm-shrinkwrap.json file to specify the modules (bluebird, extend, and mysql in this case but it doesn't really matter) and versions. This all works perfectly fine.
But now instead of reaching out to the internet to get the modules I want to have them stored in local directory BAR and have the process in foo use npm to install them from there. I can not store them permanently in FOO but I can in BAR for reasons outside my control. I know this is relatively simple but I can't seem to get the right set of commands down. Thanks for the help.
Note: This answer originally suggested only redefining the cache location. While that works in principle, npm still tries to contact the network for each package, causing excessive delays.
I assume your intent is to transparently change the installation source: in other words: you don't want to change your package, you simply want to call npm install as before, but have the packages be installed from your custom filesystem location, offline (without the need for an Internet connection).
There are two pieces to the puzzle:
Redefine npm's cache filesystem location (where previously downloaded packages are cached) to point to your custom location:
Note that cached packages are stored in a specific way: the package.json file is stored in subfolder package, and the zipped package as a whole as package.tgz. It's easiest to copy packages from your existing cache to your custom location, or to simply install additionally needed ones while you have an Internet connection, which caches them automatically.
For transparent use (npm install can be called as usual):
By setting the configuration item globally:
npm config set cache '/path/to/BAR'
Note that this will take effect for all npm operations, persistently.
Via an environment variable (which can be scoped to a script or even a single command):
export npm_config_cache='/path/to/BAR'
npm_config_cache='path/to/BAR' npm install
Ad-hoc use, via a command-line option:
npm install --cache /path/to/BAR
Force npm to use cached packages:
Currently, that requires a workaround via the cache-min configuration item.
A more direct feature, such as via an --offline switch has been a feature request for years - see https://github.com/npm/npm/issues/2568
The trick is to set cache-min to a very high value, so that all packages in the cache are considered fresh and served from there:
For transparent use (npm install can be called as usual):
By setting the configuration item globally:
npm config set cache-min 9999999999
Note that this will take effect for all npm operations, persistently.
Via an environment variable (which can be scoped to a script or even a single command):
export npm_config_cache_min=9999999999
npm_config_cache_min=9999999999 npm install
Ad-hoc use, via a command-line option:
npm install --cache-min 9999999999
Assuming you've set cache-min globally or through an environment variable,
running npm install should now serve the packages straight from your custom cache location.
Caveats:
This assumes that all packages your npm install needs are available in your custom location; trying to install a package that isn't in the cache will obviously fail without an Internet connection.
Conversely, if you do have Internet access but want to prevent npm from using it to fetch packages - which it still will attempt if a package is not found in the cache - you must change the registry configuration item to something invalid so as to force the online installation attempt to fail; e.g.:
export npm_config_registry=http://example.org
Note that the URL must exist to avoid delays while npm tries to connect to it; while you could set the value to something syntactically invalid (e.g., none), npm will then issue a warning on every use.
Sample bash script:
#!/usr/bin/env bash
# Set environment variables that set npm configuration items to:
# - redefine the location of the cache folder
# - make npm look in the cache only (assuming the packages are there)
# Note that by doing this inside a script the changes will only take effect
# in the script and NOT persist.
export npm_config_cache='/path/to/BAR' npm_config_cache_min=9999999999
# Now cd to your package and invoke `npm install` as usual.
cd '/path/to/project'
npm install
You might want to try npm link. You could:
download the dependency
run npm link from the dependency's directory
run npm link mycrazydependency from you project
Detail here: https://docs.npmjs.com/cli/link
If a shrink wrap file is present then package.json is ignored. What you need to do is change the URL they are being retrieved from using a find and replace operation like sed .... However I'm not sure changing the URL to a file:/// syntax is valid but give it a go.
I would like to optionally install dependencies for my deployments based on environment configuration of my node servers, is this possible?
Something like :
dependencies : env.prod == true ? {} : {something : "1.1" }
Are there any variables available at this stage in the npm install stage?
Many organizations run their own node repositories. This has the advantage that your production won't fail to deploy because some third party repository went down.
If you can run your own node repository, why not run two (or more). Then you can put mock packages in the "staging" repository and the real packages in the "production" repository.
It seems that this would achieve the effect you are trying for, but I fail to see the point.
I'm creating a yeoman generator for my web projects.
But I wonder how I can try and test my changes before publishing it?
Since I have installed it once, it will not run my local development version, instead it runs my installed version.
Any suggestions on how can test-run my local development version?
I finally found some information on how to accomplish this:
if you wish to develop on the generators code base, and debug locally, a common way to do so is to rely on npm link
git clone the generators repo locally
cd into that repository and run npm link. It'll install required dependencies and install the package globally, using a symbolic link to your local version.
If you want to install sub generators, you need to do so in the context of a yeoman-generator package linked earlier. Cd into the sub generators package you have cloned locally and run npm link.
We now have everything linked and known on the system, we now need to link the sub-generator repo into the parent one, yeoman-generator cloned and linked in step 1 & 2.
https://github.com/yeoman/generator/wiki/Testing-generators
EDIT:
Updated link for info: https://yeoman.io/authoring/index.html
If by "running locally" you mean the ability to test your generator and its flow you can simply do this.
In your project directory folder run npm link. If this passes in flying colors, go to step 2.
Open a terminal and cd into the folder you wish to initiate a project.
Run yo generator-theNameOfYourGenerator. This will run your generator.
I am currently creating a portable consolidation of my workflow using Node-Webkit which has node.js embedded. Now my problem is getting grunt/gulp inside the project itself as it depends on the cli somewhat(avoidable, granted), and also is confusing to me on the architecture. Is it possible to find just a .js with grunt in it to include much like Jquery/Handlebars?
Is this all I need to just include and run?
No before that make sure you environment is up, get the package.json, GruntFile.js file. In GruntFile.js you can specify what you want to pre-process. For example jade,Less,coffee. It looks very much like a node function, for sample you can refer to link
Now to make this work you also need to install various contrib plugins as per your requirement. Then register every single task in GruntFile.js. It really speeds up the development.
grunt.loadNpmTasks('grunt-contrib-less');
grunt.loadNpmTasks('grunt-contrib-jade');
grunt.loadNpmTasks('grunt-contrib-coffee');
grunt.registerTask('test', ['jade', 'less','coffee']);
So to process less,jade,coffee, we need to run the module installations such as
npm install grunt --save-dev
npm install grunt <module name> --save-dev
There are many more interesting configurations to learn and documentation is really nice, please refer to getting started guide
This adds the required Grunt and grunt plugins to package.json
when you generate a yeoman project, do you commit the node_modules that is generated into your code repository?
It seems like it is necessary for another developer to check out a project and develop from it, but it seems like a lot of files to commit which seem unrelated to a project itself.
You can just run npm install to get the dependencies installed. However there are multiple benefits to committing your dependencies, which you can read about in this blog post:
Checking in front-end dependencies (for Bower, but applies to npm too)