I want to see if a given string ("hello" in this case) is accessible via IPFS. I am trying this:
echo hello | ipfs add -nq | ipfs cat
I am getting this:
Error: lock /root/.ipfs/repo.lock: someone else has the lock
What's going on? Is there a global lock, or is IPFS just protecting itself from a race condition that could have happened if I hadn't specified the -n flag?
This seems to work:
HASH=`echo hithere | ipfs add -nq` && ipfs cat $HASH
However, I wonder whether there is a more idiomatic way. This must be a fairly standard thing to do.
What's going on?
Your daemon is not running, so both ipfs add and ipfs cat tries to start an offline daemon at the same time which doesn't work (only one daemon per IPFS_PATH can run).
You need to run ipfs daemon in the background first, so then the commands will just hit ipfs daemon's API instead of starting their own daemon.
This seems to work:
HASH=`echo hithere | ipfs add -nq` && ipfs cat $HASH
Yes because you are using && which schedule them to wait on each other first.
Related
After I add a file to ipfs using ipfs add hello - how to do I retrieve the hash for the file if I lose it?
I guess I'm expecting ipfs info <filepath> or something similar?
If you want to see what the hash of a file would be, without actually uploading it to IPFS, you can run ipfs add --only-hash, or ipfs add -n for short.
just run ipfs add hello again...
Make sure ipfs daemon is running before proceeding.
ipfs get theHashOfTheItem in the command line will print out the raw data/ text of the block corresponding to that particular hash
To get a list of objects with the hash that is hosted on your computer, you may run the daemon using ipfs daemon followed by going to and checking under Files.
http://localhost:5001/webui
I remember reading a way to get the list of hashes via the command line, but I can't seem to remember it. Once I get it, I shall post the details about that here as well.
Not ideal, but checking the information each hash stores, either by using the command shared on top or clicking on the files itself within the browser should let you find the hash you are looking for.
Since you've added the file/folder, it will be pinned to your ipfs repo. Run the command
ipfs pin ls
This will list all the objects of the files/folder pinned to your repo
Spent an hour doing this and turns out you can do a simple ipfs files stat /path/to/object to get an output like:
$ ipfs files stat /folder-2/text.txt
QmcNsPV7QZFHKb2DNn8GWsU5dtd8zH5DNRa31geC63ceb4
Size: 14
CumulativeSize: 72
ChildBlocks: 1
Type: file
If you want to script this, do a simple | head -n 1 to get the hash.
I am not a MySQL expert.
I have a script that installs MySQL, starts mysqld, and then uses mysql to do some initialization.
Currently, in order to have this work, I enter into a loop that (apologize for the pseudocode mixing multiple languages):
mysqld_safe /* ... */ & /* ampersand to start in background so we can continue */
while(fileDoesNotExist("/tmp/mysql.sock")) {
sleepFor100ms();
}
mysql -u root /* and so forth */ initialize.sql
This seems to work (!) but has multiple problems:
polling smells funny,
I am not smart enough about MySQL to know whether looking at that hard-coded pathname /tmp/mysql.sock is smart at all.
And yet it's a lot easier than trying to (for example) consume and parse the stdout (or is it stderr?) of mysqld_safe to figure out whether the server has started.
My narrow question is whether there's a way to issue a blocking start of mysqld: can I issue any command that blocks until the database has started, and then exits (and detaches, maybe leaving a PID file), and has a companion stop command? (Or maybe allows me to read the PID file and issue my own SIGTERM?)
My broader question is, am I on the right track, or is there some totally different and easier (to be "easier" for me it would have to be lightweight; I'm not that interested in installing a bunch of tools like Puppet or DbMaintain/Liquibase or whatever) approach to solving the problem I articulated? That is, starting with a .gz file containing MySQL, install a userland MySQL and initialize a database?
Check out the init shell script for mysqld. They do polling, in a function called wait_for_pid().
That function checks for the existence of the pid file, and if it doesn't exist yet, sleeps for 1 whole second, then tries again. There's a timeout that defaults to 900 seconds, at which point it gives up waiting and concludes that it's not going to start (and outputs a totally unhelpful message "The server quit without updating PID file").
You don't have to guess where the pid file is. If you're starting mysqld_safe, you should tell it where it should create the pid file, using the --pid-file option.
One tricky part is that the pid file isn't created until mysqld initializes. This can take a while if it has to perform crash recovery using the InnoDB log files, and the log files are large. So it could happen that 900 seconds of timeout isn't long enough, and you get a spurious error, even though mysqld successfully starts a moment after the timeout.
You can also read the error log or the console output of mysqld. It should eventually output a line that says "ready for connections."
To read until you get this line, and then terminate the read, you could use:
tail -f | sed -e '/ready for connections/q'
You can use
mysqladmin -h localhost status
or use a pure bash solution like wait-for-it
./wait-for-it.sh --timeout 10 -h localhost -p 3306
I'm launching instances using the following command:
gcutil addinstance \
--image=debian-7 \
--persistent_boot_disk \
--zone=us-central1-a \
--machine_type=n1-standard-1 \
--metadata_from_file=startup-script:install.sh \
instance-name
How can I detect when this instance has completed it's install script? I'd like to be able to place this launch command in a larger provisioning script that then goes on to issue commands to the server that depend on the install script having been successfully completed.
There is a number of ways: sending yourself an email, uploading to Cloud Storage, sending a jabber message, ...
One simple, observable way IMHO is to add a logger entry at the end of your install.sh script (I also tweak the beginning for symmetry). Something like:
#!/bin/bash
/usr/bin/logger "== Startup script START =="
#
# Your code goes here
#
/usr/bin/logger "== Startup script END =="
You can check then if the script started or ended in two ways:
From your Developer's Console, select "Projects" > "Compute" > "VM Instances" > your instance > "Serial console" > "View Output".
From CLI, by issuing a gcutil getserialportoutput instance-name.
I don't know of a way to do all of this within gcutil addinstance.
I'd suggest:
Adding the instance via gcutil addinstance, making sure to use the --wait_until_running flag to ensure that the instance is running before you continue
Copying your script over to the instance via something like gcutil push
Using gcutil ssh <instance-name> </path-to-script/script-to-run> to run your script manually.
This way, you can write your script in such a way that it blocks until it's finished, and the ssh command will not return until your script on the remote machine is done executing.
There really are a lot of ways to accomplish this goal. One that tickles my fancy is to use the metadata server associated with the instance. Have the startup script set a piece of metadata to "FINISHED" when the script is done. You can query the metadata server with a hanging GET that will only return when the metadata updates. Just use gcutil setmetadata
from within the script as the last command.
I like this method because the hanging GET just gives you one command to run, rather than a poll to run in a loop, and it doesn't involve any services besides Compute Engine.
One more hacky way:
startup_script_finished=false
while [[ "$startup_script_finished" = false ]]; do
pid=$(gcloud compute ssh $GCLOUD_USER#$GCLOUD_INSTANCE -- pgrep -f "\"/usr/bin/python /usr/bin/google_metadata_script_runner --script-type startup\"")
if [[ -z $pid ]]; then
startup_script_finished=true
else
sleep 2
fi
done
One possible solution would be to have your install script create a text file in a cloud storage bucket, as the last thing it does, using the host name as the filename.
Your main script that did the original gcutil addinstance command could then be periodically polling the contents of the bucket (using gsutil ls) until it sees a file with a matching name and then it would know the install had completed on that instance.
I am working on a project to stream HDTV from a personal computer to devices supporting HTTP Live Streaming (think iOS devices and some android). I have the formatting of the video and the streaming aspects down. What I am now trying to implement is an easy way to change the channel remotely.
My current method involves connecting via SSH to kill the old stream and begin a new stream. This works, but isn't pretty. I want something my Mom or girlfriend could use. I decided I would build an HTML5 app that would issue the channel switching over CGI scripts. I currently have a parent process with a form that calls a child process to decide if the stream is running and then a subchild process to actually tune the stream.
As I am streaming live video from my computer I need the subchild process to run indefinitely. Unfortunately it seems that when my parent process is finished the background process started in the subchild process terminates.
I have tried a simple &, using nohup, setsid, and daemon. daemon runs cleanest but still terminates when the parent finishes. even with a -r flag. I'll place my code below and maybe someone will have an idea on how I could implement this or a better way to achieve the same thing? Thanks! (oh and i know killing vlc is not a pretty way to kill the stream, if you have a better way i'm all ears)
parent invoking child:
----------------------
./ChangeChannel.sh $channel #passed from form submission
child (ChangeChannel.sh):
-------------------------
#!/bin/bash
directory=./Channels/
newchannel=$1
if [ $(pidof vlc) ]
then
sudo kill $(pidof vlc)
fi
daemon -r -v -d $directory$newchannel &
subchild example:
-----------------
vlc atsc://frequency=605029000 --intf=dummy --sout-transcode-audio-sync :live-cache=3000 --sout='#transcode{vcodec=h264,vb=150,fps=25,width=480,scale=1,venc=x264{aud,profile=baseline,level=30,keyint=15,bframes=0,ref=1},acodec=aac,ab=40,channels=2,samplerate=22050}:duplicate{dst=std{mux=ts,dst=-,access=livehttp{seglen=16,delsegs=true,numsegs=10,index=/var/www/stream/live.m3u8,index-url=content/live-######.ts},mux=ts{use-key-frames},dst=/var/www/stream/content/live-######.ts,ratecontrol=true}}'
how can i keep the subchild from terminating??? Running Apache on Ubuntu 12.04
I got it!
For anyone interested in how, i changed my tactics to use nohup, &, disown, and > /dev/null 2>&1.
Honestly, still not quite sure how I got it working... just a lot of trial and error with some educated guesses. My code follows:
parent invocation:
------------------
nohup ./ChangeChannel.sh $channel & disown
child invocation:
-----------------
sudo nohup su user $directory$newchannel &> /dev/null 2>&1
subchild invocation:
--------------------
vlc atsc://frequency=605029000 --intf=dummy --sout-transcode-audio-sync :live-cache=3000 --sout='#transcode{vcodec=h264,vb=150,fps=25,width=480,scale=1,venc=x264{aud,profile=baseline,level=30,keyint=15,bframes=0,ref=1},acodec=aac,ab=40,channels=2,samplerate=22050}:duplicate{dst=std{mux=ts,dst=-,access=livehttp{seglen=16,delsegs=true,numsegs=10,index=/var/www/stream/live.m3u8,index-url=content/live-######.ts},mux=ts{use-key-frames},dst=/var/www/stream/content/live-######.ts,ratecontrol=true}}' & disown
ChangeChannel.sh uses sudo to execute su via cgi in order to execute vlc as user other than root. It seems a little messy but hell it works.
I have deployed Jenkins in my CentOS machine, Jenkins was working well for 3 days, but yesterday there was a Disk space is too low. Only 1.019GB left. problem.
How can I solve this problem, it make my master offline for hours?
You can easily change the threshold from jenkins UI (my version is 1.651.3):
[]
Update: How to ensure high disk space
This feature is meant to prevent working on slaves with low free disk space. Lowering the threshold would not solve the fact that some jobs do not properly cleanup after they finish.
Depending on what you're building:
Make sure you understand what is the disk output of your build - if possible - restrict the output to happen only to the job workspace. Use workspace cleanup plugin to cleanup the workspace as post build step.
If the process must write some data to external folders - clean them up manually on post build steps.
Alternative1 - provision a new slave per job (use spot slaves - there are many plugins that integrate with different cloud provider to provision on the fly machines on demand)
Alternative2 - run the build inside a container. Everything will be discarded once the build is finished
Beside above solutions, there is a more "COMMON" way - directly delete the largest space consumer from Linux machine. You can follow the below steps:
Login to Jenkins machine (Putty)
cd to the Jenkins installation path
Using ls -lart to list out hidden folder also, normally jenkin
installation is placed in .jenkins/ folder
[xxxxx ~]$ ls -lart
drwxrwxr-x 12 xxxx 4096 Feb 8 02:08 .jenkins/
list out the folders spaces
Use df -h to show Disk space in high level
du -sh ./*/ to list out total memory for each subfolder in current path.
du -a /etc/ | sort -n -r | head -n 10 will list top 10 directories eating disk space in /etc/
Delete old build or other large size folder
Normally ./job/ folder or ./workspace/ folder can be the largest folder. Please go inside and delete base on you need (DO NOT
delete entire folder).
rm -rf theFolderToDelete
You can limit the reduce of disc space by discarding the old builds. There's a checkbox for this in the project configuration.
This is actually a legitimate question so I don't understand the downvotes, perhaps it belongs on Superuser or Serverfault. This is a soft warning threshold not hard limit where the disk is out of space.
For hudson see where to configure hudson node disk temp space thresholds - this is talking about the host, not nodes
Jenkins is the same. The conclusion is for many small projects the system property called hudson.diagnosis.HudsonHomeDiskUsageChecker.freeSpaceThreshold could be decreased.
In saying that I haven't tested it and there is a disclaimer
No compatibility guarantee
In general, these switches are often experimental in nature, and subject to change without notice. If you find some of those useful, please file a ticket to promote it to the official feature.
I got the same issue. My jenkins version is 2.3 and its UI is slightly different. Putting it here so that it may helps someone. Increasing both disk space thresholds to 5GB fixed the issue.
I have a cleanup job with the following build steps. You can schedule it #daily or #weekly.
Execute system groovy script build step to clean up old jobs:
import jenkins.model.Jenkins
import hudson.model.Job
BUILDS_TO_KEEP = 5
for (job in Jenkins.instance.items) {
println job.name
def recent = job.builds.limit(BUILDS_TO_KEEP)
for (build in job.builds) {
if (!recent.contains(build)) {
println "Preparing to delete: " + build
build.delete()
}
}
}
You'd need to have Groovy plugin installed.
Execute shell build step to clean cache directories
rm -r ~/.gradle/
rm -r ~/.m2/
echo "Disk space"
du -h -s /
To check the free space as Jenkins Job:
Parameters
FREE_SPACE: Needed free space in GB.
Job
#!/usr/bin/env bash
free_space="$(df -Ph . | awk 'NR==2 {print $4}')"
if [[ "${free_space}" = *G* ]]; then
free_space_gb=${x/[^0-9]*/}
if [[ ${free_space_gb} -lt ${FREE_SPACE} ]]; then
echo "Warning! Low space: ${free_space}"
exit 2
fi
else
echo "Warning! Unknown: ${free_space}"
exit 1
fi
echo "Free space: ${free_space}"
Plugins
Set build description
Post-Build Actions
Regular expression: Free space: (.*)
Description: Free space: \1
Regular expression for failed builds: Warning! (.*)
Description for failed builds: \1
For people who do not know where the configs are, download the tmpcleaner from
https://updates.jenkins-ci.org/download/plugins/tmpcleaner/
You will get an hpi file here. Go to Manage Jenkins-> Manage plugins-> Advanced and then upload the hpi file here and restart jenkins
You can immediately see a difference if you go to Manage Nodes.
Since my jenkins was installed in a debian server, I did not understand most of the answers related to this since i cannot find a /etc/default folder or jenkins file.
If someone knows where the /tmp folder is or how to configure it for debian , do let me know in comments