I can see in many templates of packer that, some files are attached as floppies, so that these are executed right when the VM creation completes in a builder, and some files are included in provisioner scripts so that they are executed serially later.
What's difference between the two? How to decide which script to include in floppies and which file to include in provioner scripts?
As per https://www.packer.io/docs/builders/vmware-vmx.html#floppy_files the floppy is useful for unattended installation. The floppy will contain files required for the same.
Whereas the provisioner will contain files that are required for configuring the system after it is installed.
Some operating systems no longer support the floppy images, but those that do typically use it for bootstrapping via kickstart or a Debian installer preseed. Many bootstrap methods support using a web server to host these files as well, so Packer provides a built in webserver to provide this functionality for building boxes if they don't have floppy support or the files are too large to fit on a 1.44M disk.
Related
I uploaded many GitHub artifacts, causing the GitHub free storage space (50 GB) to run out.
Most of these artifacts were copies or had very small changes.
Assuming that these were stored in layers as diffs from parent images (like docker images are stored), it was unlikely that 50 GB of space would run out.
Are GitHub artifacts stored as individual files every time a workflow is run ?
Are GitHub packages stored in the same way ?
Is the storage for packages and artifacts the same ?
GitHub's artifacts are usually linked with:
release: artifacts could be another term for "assets": files associated with a particular GitHub release (see Update a release asset for instance).
or workflow artifacts: An artifact is a file or collection of files produced during a workflow run. It allows you to persist data after a job has completed, and share that data with another job in the same workflow.
As Edward Thomson (from GitHub) notes: "because they're meant to be used to move data between jobs in a workflow, workflow assets are not permanent".
GitHub Container Registry is dedicated to store and manage Docker and OCI images.
Only the latter benefit from incremental storage through layers.
The former would be uploaded as a all for each file.
From the comments below:
A workflow where one authenticates to GHCR (GitHub Container Registry) and push to the registry an image (docker push ghcr.io/OWNER/IMAGE_NAME:tag) will benefit from an incremental layer-by-layer storage.
This differ from a regular asset upload, where the artifact is stored as a all.
I like to replace my default docker registry in OKD with an existing Nexus Docker registry in OKD. How do I replace the exist docker registry in OKD with the Nexus Docker Registry?
The idea is to host all my images in that Nexus repo. For example when I use S2I.
This topic (as #GhostCat indicates) leaves your options pretty open until you make some additional decisions. There are a couple of resources/topics that you may want to investigate to help hone on in your desired outcome:
1) Pushing/Pulling from an external registry
In this topic, I'd suggest that you don't "replace" the internal registry, but rather run a nexus registry alongside the internal registry. The internal registry can still be used to store system components, s2i image builders, etc etc., while Nexus can be the primary registry for your applications.
For image pulling: Each docker host can be configured with additional external registries to search through, and by simply editing your image streams or including the full image path in your deployment / statefulset / etc.
You can modify your build configs to store the built images in an external registry https://blog.openshift.com/pushing-application-images-to-an-external-registry/
2) Moving all registry features/components to Nexus
In this topic, you will need to read up on the disconnected installation notes around OpenShift and adapt them to your own Nexus config. Essentially, you need to load up all of the images in Nexus and modify all of the templates/imagestreams in openshift to point at the new repository.
Follow the guidance here and modify accordingly https://blog.openshift.com/pushing-application-images-to-an-external-registry/
This is a less common pattern and really only followed by teams in fully disconnected environments. Most teams we meet follow the guidance in #1 above.
Thanks for that article, its helpful but is there not a way to make this automatic select as default? So that always all my build images goes to my external Nexus Docker repo? Because if it goes to the internal registry then we lost the images by a reboot of a crash of the internal registry container.
Especially the S2I images.
I see a couple of ideas to discuss;
1) If you just want your build outputs to target another external repo instead of the default internal, this can be codified in any of your external ci/cd systems as part of the build config. The most common way in openshift would be to have Jenkins create the build configs through a Jenkinsfile in your code repo.
I'm not aware of a simple documented way to override a default configuration or setting for all build outputs across the system.
2) If you are losing your images on the internal registry, then you are not leveraging persistent storage, which should be considered. See [here] (https://docs.okd.io/3.10/install_config/registry/extended_registry_configuration.html#docker-registry-configuration-reference-storage)
I've been trying to follow the
Setting Up Stackdriver Debugger for Java applications on Google Compute Engine, but am running into issues with Stackdriver Debug.
I'm building my .war file from a separate build server, then deploying it to my GCE server. I added the agent to the start command via /etc/defaults, and my app appears in the https://console.cloud.google.com/debug control panel. The version I set in the run command matches the revision that shows up in the source-context(s).json files.
However when I click open the app, I see the message that
No source version information was provided by the deployed application
I connected the app's git repo as a mirrored cloud repository, and can browse the source files in the sidebar of the Stackdriver Debug page. But, If I browse to a file and add a breakpoint I get an error that the error "File was not found in the executable."
I have ran the gcloud preview app gen-repo-info-file command, which created two basic json files storing my git repo and revision. Is it supposed to do anything else?
I have tried running jetty using both normal and extracted modes. If I have jetty first extract the war file, I can see the source-context.json filesin the WEB-INF/classes directory.
What am I missing?
https://github.com/GoogleCloudPlatform/cloud-debug-java#extra-classpath mentions
you can update the agentPath showing your WEB-INF/class directory.
-agentpath:/opt/cdbg/cdbg_java_agent.so=--cdbg_extra_class_path=/opt/tomcat/webapps/myapp/WEB-INF/classes
For multiple class paths:
-agentpath:/opt/cdbg/cdbg_java_agent.so=--cdbg_extra_class_path=/opt/tomcat/webapps/myapp/WEB-INF/classes:/another/path/with/classes
There are a couple of things going on here.
First, it sounds like you are doing the correct thing with gen-repo-info-file. The debugger agent should pick up the json files from the WEB-INF/classes directory.
The debugger uses fuzzy matching to find source files, so as long as the name of the .java file matches a file in your executable, you should not get that error.
The most likely scenario given the information in your question is that you are attaching the debugger to a launcher process, rather than your actual application. Without further details, I can't absolutely confirm that, though.
If you send us more details at cdbg-feedback#google.com, we can look more closely at your case to see if we can understand exactly what's happening, and potentially improve our documentation, since it sounds like you followed the docs pretty closely.
So, I'm enjoying using composer, but I'm struggling to understand how others use it in relation to a deployment service. Currently I'm using deployhq, and yes, I can set it to deploy and run composer when there is an update to the repo, but this doesn't make sense to me now.
My main composer repo, containing just the json file of all of the packages I want to include in my build, only gets updated when I add a new package to the list.
When I update my theme, or custom extension (which is referenced in the json file), there is no "hook" to update my deployment service. So I have to log in to my server and manually run composer (which takes the site down until it's finished).
So how do others manage this? Should I only run composer locally and include the vendor folder in my repo?
Any answers would be greatly appreciated.
James
There will always be arguments as to the best way to do things such as this and there are different answers and different options - the trick is to find the one that works best for you.
Firstly
I would first take a step back and look at how you are managing your composer.json
I would recommend that all of your packages in composer.json be locked down to the exact version number of the item in Packagist. If you are using github repo's for any of the packages (or they are set to dev-master) then I would ensure that these packages are locked to a specific commit hash! It sounds like you are basically there with this as you say nothing updates out of the packages when you run it.
Why?
This is to ensure that when you run composer update on the server, these packages are taken from the cache if they exist and to ensure that you dont accidentally deploy untested code if one of the modules happens to get updated between you testing and your deployment.
Actual deployments
Possible Method 1
My opinion is slightly controversial in that when it comes to Composer for many of my projects that don't go through a CI system, I will commit the entire vendor directory to version control. This is quite simply to ensure that I have a completely deployable branch at any stage, it also makes deployments incredibly quick and easy (git pull).
There will already be people saying that this is unnecessary and that locking down the version numbers will be enough to ensure any remote system failures will be handled, it clogs up the VCS tree etc etc - I won't go into these now, there are arguments for and against (a lot of it opinion based), but as you mentioned it in your question I thought I would let you know that it has served me well on a lot of projects in the past and it is a viable option.
Possible Method 2
By using symlinks on your server to your document root you can ensure that the build completes before you switch over the symlink to the new directory once you have confirmed the build completed.
This is the least resistance path towards a safe deployment for a basic code set using composer update on the server. I actually use this method in conjunction with most of my deployments (including the ones above and below).
Possible Method 3
Composer can use "artifacts" rather than a remote server, this will mean that you will basically be creating a "repository folder" of your vendor files, this is an alternative to adding the entire vendor folder into your VCS - but it also protects you against Github / Packagist outages / files being removed and various other potential issues. The files are retrieved from the artifacts folder and installed directly from the zip file rather than being retrieved from a server - this folder can be stored remotely - think of it as a poor mans private packagist (another option btw).
IMO - The best method overall
Set up a CI system (like Jenkins), create some tests for your application and have them respond to push webhooks on your VCS so it builds each time something is pushed. In this build you will set up the system to:
run tests on your application (If they exist)
run composer update
generate an artifact of these files (if the above items succeed)
Jenkins can also do an actual deployment for you if you wish (and the build process doesn't fail), it can:
push the artifact to the server via SSH
deploy the artifact using a script
But if you already have a deployment system in place, having a tested artifact to be deployed will probably be one of its deployment scenarios.
Hope this helps :)
Question:
In the path HUDSON_HOME/jobs/<jobname>/builds/<timestamp>/workspace-files, there are a series of .tmp files. What are these files, and what feature of Hudson do they support?
Background
Using Hudson version 1.341, we have a continuous build task that runs on a slave instance. After the build is otherwise complete, including archiving the artifacts, task scanner, etc., the job appears to hang for a long period of time. In monitoring the master node, I noted that many .tmp files were being created and modified under builds//workspace=files, and that some of them were very large. This appears to be causing the delay, as the job completed at the same time that files in this path stopped changing.
Some key configuration points of the job:
It is tied to a specific slave node
It builds in a 'custom workspace'
It runs the Task Scanner plugin on a portion of the workspace to find "todo" items
It triggers a downstream job that builds in the same custom workspace on the same slave node
In this particular instance, the .tmp files were being created by the Task Scanner plugin. When tasks are found, the files in which they are found are copied back to the master node. This allows the master node to serve those files in the browser interface for Tasks.
Per this answer, it is likely that this same thing occurs with other plug-ins, too.
Plug-ins known to exhibit this behavior (feel free to add to this list)
Task Scanner
Warnings
FindBugs
There's an explanation on the hudson users mailing list:
...it looks like the warnings plugin copies any files that have compiler warnings from the workspace (possibly on a slave) into a "workspace-files" directory within HUDSON_HOME/jobs//builds/
The files then, I surmise, get processed resulting in a "compiler-warnings.xml" file within the HUDSON_HOME/jobs//builds/
I am using the "warnings" plugin, and I suspect it's related to that.