I am trying to instigate some configuration control tools for our mysql databases. To start with we trained and installed GIT on all dev machines including DBA. I got a php script written that extract the definition of all db, all tables, functions,SP, views, and triggers and puts this is a directory structure.
My initial intention was to automate the scrip to run at fixed interval and recreate the directory based on the state of the db server at that time and have git commit the differences.
During testing, I realized that every time I run the script and do a git status, all the files are found to be modified. When I do a git diff I find that it is due to white spaces. Mostly at the end of line. During the previous commit they are trimmed off I think and when the configuration script runs they all come back therefore git thinks all files have change. So I can't do a simple git add .
The configuration management script itself runs on a windows machine but the databases it analyses can be on any platform.
The current git config is autocrlf true and core.white-space all off
What should the configuration be so that all these white spaces do not appear as a diff to git. Remember this is to be completely automated.
I would recommending setting core.autocrlf to false. Having Git automatically try to convert line endings causes no end of problems.
Git does not automatically trim whitespace at the end of lines (unless you explicitly use --whitespace=fix) - from the information you have provided, it seems more likely that the problem is line ending handling.
Related
I am a little confused about start up scripts and the command line options. I am building a small raspberry pi based server for my node applications. In order to provide maximum protection against power failures and flash write corruption, the root file system is read only, and that embraces the home directory of my main user, were the production versions of my apps (two of them) are stored. Because the .pm2 directory here is no good for logs etc I currently set PM2_HOME environment variable to a place in /var (which has 512kb unused space around it to ensure writes to i. The eco-system.json file reads this environment variable also to determine where to place its logs.
In case I need to, I also have a secondary user with a read write home directory in another (protected by buffer space around it) partition. This contains development versions of my application code which because of the convenience of setting environments up etc I also want to monitor with PM2. If I need to investigate a problem I can log in to that user and run and test the application there.
Since this is a headless box, and with watchdog and kernel panic restarts built in, I want pm2 to start during boot and at minimum restart the two production apps. Ideally it should also starts the two development versions of the app also but I can live without that if its impossible.
I can switch the read only root partition to read/write - indeed it does so automatically when I ssh into my production user account. It switches back to read only automatically when I log out.
So I went to this account to try and create a startup script. It then said (unsurprisingly) that I had to run a sudo command like so:-
sudo su -c "env PATH=$PATH:/usr/local/bin pm2 startup ubuntu -u pi --hp /home/pi"
The key issue for me here is the --hp switch. I went searching for some clue as to what it means. Its clearly a home directory, but it doesn't match PM2_HOME - which is set to /var/pas in my case to take it out of the read only area. I don't want to try and and spray my home directory with files that shouldn't be there. So am asking for some guidance here
I found out by experiment what it does with an "ubuntu" start up script. It uses it to set PM2_HOME in the script by appending "/.pm2" to it.
However there is nothing stopping you editing the script once it has created it and setting PM2_HOME to whatever you want.
So effectively its a helper for the script, but only that and nothing more special.
So, I'm enjoying using composer, but I'm struggling to understand how others use it in relation to a deployment service. Currently I'm using deployhq, and yes, I can set it to deploy and run composer when there is an update to the repo, but this doesn't make sense to me now.
My main composer repo, containing just the json file of all of the packages I want to include in my build, only gets updated when I add a new package to the list.
When I update my theme, or custom extension (which is referenced in the json file), there is no "hook" to update my deployment service. So I have to log in to my server and manually run composer (which takes the site down until it's finished).
So how do others manage this? Should I only run composer locally and include the vendor folder in my repo?
Any answers would be greatly appreciated.
James
There will always be arguments as to the best way to do things such as this and there are different answers and different options - the trick is to find the one that works best for you.
Firstly
I would first take a step back and look at how you are managing your composer.json
I would recommend that all of your packages in composer.json be locked down to the exact version number of the item in Packagist. If you are using github repo's for any of the packages (or they are set to dev-master) then I would ensure that these packages are locked to a specific commit hash! It sounds like you are basically there with this as you say nothing updates out of the packages when you run it.
Why?
This is to ensure that when you run composer update on the server, these packages are taken from the cache if they exist and to ensure that you dont accidentally deploy untested code if one of the modules happens to get updated between you testing and your deployment.
Actual deployments
Possible Method 1
My opinion is slightly controversial in that when it comes to Composer for many of my projects that don't go through a CI system, I will commit the entire vendor directory to version control. This is quite simply to ensure that I have a completely deployable branch at any stage, it also makes deployments incredibly quick and easy (git pull).
There will already be people saying that this is unnecessary and that locking down the version numbers will be enough to ensure any remote system failures will be handled, it clogs up the VCS tree etc etc - I won't go into these now, there are arguments for and against (a lot of it opinion based), but as you mentioned it in your question I thought I would let you know that it has served me well on a lot of projects in the past and it is a viable option.
Possible Method 2
By using symlinks on your server to your document root you can ensure that the build completes before you switch over the symlink to the new directory once you have confirmed the build completed.
This is the least resistance path towards a safe deployment for a basic code set using composer update on the server. I actually use this method in conjunction with most of my deployments (including the ones above and below).
Possible Method 3
Composer can use "artifacts" rather than a remote server, this will mean that you will basically be creating a "repository folder" of your vendor files, this is an alternative to adding the entire vendor folder into your VCS - but it also protects you against Github / Packagist outages / files being removed and various other potential issues. The files are retrieved from the artifacts folder and installed directly from the zip file rather than being retrieved from a server - this folder can be stored remotely - think of it as a poor mans private packagist (another option btw).
IMO - The best method overall
Set up a CI system (like Jenkins), create some tests for your application and have them respond to push webhooks on your VCS so it builds each time something is pushed. In this build you will set up the system to:
run tests on your application (If they exist)
run composer update
generate an artifact of these files (if the above items succeed)
Jenkins can also do an actual deployment for you if you wish (and the build process doesn't fail), it can:
push the artifact to the server via SSH
deploy the artifact using a script
But if you already have a deployment system in place, having a tested artifact to be deployed will probably be one of its deployment scenarios.
Hope this helps :)
Is there a hack or a way out to run a hook on the server whenever a client pushes or pulls and gets "no changes found". I want to be able to run a script on a client's pull irrespective of whether the repository has changed or not.
Currently it is easily possible to use the standard hooks (pretxnchangegroup, changegroup, etc.) for all cases but they do not get triggered when there is no change.
A hook wont' be able to do what you're looking for. The hooks that run on the server side are all triggered by the arrival of things (changesets or pushkeys). There's no server-side hook for nothing-arrived nor for commands that never send things (incoming, outgoing, etc.).
One long shot I tried that didn't work was using a post-serve hook. When a client connects to a remote repository over ssh it's really running hg serve on that client, so I was hoping as post-serve hook would be executed at the end of a ssh session, but no such luck (indeed that hook likely doesn't exist at all).
Fortunately there are some other hackier-options than hooks. For repositories you access over ssh you can alter the .ssh/authorized-keys file to force the execution of some command:
command="/home/me/hg_and_somethingelse.sh" ssh-rsa AAAA....
then in /home/me/hg_and_something_else.sh you'd have:
#!/bin/bash
hg $#
echo ANY COMMAND FROM HERE ONWARD IS RUN FOR EVERY PUSH, PULL, CLONE, etc.
Similar for http-served repos you'd just tack whatever you want at the end of the wsgi file you're using.
As a caveat there are a lot of tools (IDEs for example) that check for changes often, so your script is going to be run more frequently than you're imagining.
I am using git to manage a dynamic website (PHP + MySQL) and I want to send my files from my localhost to my staging and development servers in the most efficient and hassle-free way.
I am currently convinced that the best way for me to approach this problem is to use this git branching model to organize my local git repo. From there, I will use the release branches to push to my staging server for testing. Once I am happy that the release code works on the staging server, I can then merge with my master branch and push that to my production server.
Pushing to Staging Server:
As noted in many introductory git posts, I could run into problems pushing into a non-bare repo, so, as suggested in this response, I plan to push the release branch to a bare repo on the server and have a post-receive hook that clones the bare repo to a non-bare repo that also acts as the web-hosted directory.
Pushing to Production Server:
Here's my newest source of confusion...
In the response that I cited above, it made me curious as to why #Paul states that it's a completely different story when pushing to a live, development server. I guess I don't see the problem. Would it be safe and hassle-free to follow the same steps as above, but for the master branch? Where are the potential pit-falls?
Config Files:
With respect to configuration files that are unique to each environment (.htaccess, config.php, etc), it seems simplest to .gitignore each of those files in their respective repos on their respective servers. Can you see anything immediately wrong with this? Better solutions?
Accessing Data:
Finally, as I initially stated, the site uses MySQL databases to store data. How would you suggest I access that data (for testing purposes) from the staging server and localhost?
I realize that I may have asked way too many questions for a single post, but since they're all related to the best way to set up this development scheme, I thought it was necessary.
Pushing to the production server
I assume that in the response you quote, the answer refers to pushing to the production server as "a different story", just because one can push any old commit to the staging server for testing, but you would be very careful only to push a thoroughly tested version to the production server.
I think the approach you refer to (of deploying by pushing to a bare repository with a post-receive that does git checkout -f with an appropriately set GIT_WORK_TREE) is a good one for deploying from git.
Config Files
That is a reasonable plan, but you have to be a somewhat careful about using .gitignore to ignore configuration files - you might want to look at this answer for more about this:
How can I have different versions of a file in the local working directory, remote working directory and git ftp target?
Accessing data
I think the question about data for your staging server really is a separate issue, since none of that data will be in your version control system - it might be worth adding another question here about that issue. You could have a script that dumps data on your live server and imports it to the staging server, but I can think of many situations in which that would be undesirable, particularly where customer details and data protections laws have to be considered.
The Git FAQ reccomends this post-receive hook script to reset the head of a non-bare repository after it is pushed to. It will save any changes that are uncommitted on the remote using a stash. Personally, I'd rather it reject the push in that case, but that can be cone
(please note: Lots of answers contain out of date links to the FAQ and the script - hopefully these will remain valid for some time at least)
I use git flow too. For config files in expressionengine, we use ee master config which basically determines the environment its in and applied a specific config. I imagine it could easily be modified for whatever you're doing.
For deployments, we use Beanstalk which allows you to add "[deploy:Environment]" to a commit message, which will make it upload (ftp) your specified branch (the one you commit to) to the specified environment, which you configure in their web interface, when you git push.
I've been trying to find an effective solution for .htaccess files that will allow me to htpasswd one of my environments, but not all. It looks like it's possible in Apache 2.3 with something like this:
<if "%{HTTP_HOST} == 'dev.example.com'">
# auth directives
</if>
but sadly, most of the production servers we use are running an earlier version, which doesn't support the directive :(
Somewhere I did something silly.
I was deploying my Rails app via cloning the Mercurial repo down onto my Ubuntu server. It worked the first time, and then...well, I made a small change on my dev machine, pushed the changes to the repo, and then deleted the copy on the Ubuntu server and re-cloned from the repo.
The clone operation (the second, and third, and 'n' times) works without error, but I don't have write access to the files that were cloned.
When I try to startup my mongrel - it can't create the /tmp folder, and because of no write access, fails to start the Rails app.
Fixed through work around stated in comment above.