Locked History Actions

HowTo/GitWorkflow

GIT Workflow

Getting started

Identify yourself

Find who GIT thinks you are using:

git config --get-regexp 'user.*'

If either of user.name or user.email is missing or wrong, set the correct value, for example:

git config --global user.name 'Albert Einstein'

These are the strings that will be used to identify any changes you make to a repository.

Get access to the server repository

Create a github account

Starting working on an existing package

Use the clone command, e.g.

git clone gitosis@darkmatter.ps.uci.edu:likely.git
cd likely

Create a new package

Create a new local package with some initial commits:

git init <pkg>
...

step1.png

For instructions on creating a corresponding package on the server, see here.

Working with topic branches

The basic principles are:

  • The head of the master branch must always be functional. We achieve this by only adding completed topic branches to master.
  • All development work is performed on a topic branch that starts from the head of the master branch. There is no problem with having multiple open topic branches per person.
  • Whenever someone closes a topic branch, resulting in an updated master, everyone else working on the same package must update their topic branches before they can be closed.
  • Any conflicts between topic branches are always resolved during the topic-branch update step and there should never be any conflicts when closing a completed topic branch.
  • Topic branches are routinely pushed to the server but this is for backup purposes only and you should never start a new branch from an open topic.

Images below are from SourceTree but command-line instructions are provided for each step. You will need to install the gitopic command before you can use the instructions below:

% git clone git@github.com:dkirkby/workflow.git
% cd workflow
% sudo cp gitopic /usr/local/bin/

The first step is to create a new topic branch in a package, for example:

% gitopic --open Topic_A
Created topic branch 'Topic_A'

If someone else working in their own repo creates a Topic_B, this is what your repo might look like after a few commits by each of you:

step2.png

You can see other people's topic branches in your local repo with names like origin/Topic_B. Your topic branch is also visible to them as origin/Topic_A.

Once you finish working on your topic branch, you are ready to merge it back into the master branch:

% gitopic --close changes.txt
Closed topic branch 'Topic_A'

Note that the current branch is assumed (use the git checkout command to switch between branches in your local repo). The file changes.txt should describe the new features that your topic branch adds following this recommended style. When you are preparing your new feature documentation, the following command provides a helpful summary of the changes you have made on the topic branch:

% git diff --stat master..HEAD
 Makefile.am            |    8 ++-
 Makefile.in            |   58 ++++++++++---
 bosslya/bosslya.h      |    3 +-
 bosslya/io/EnzoFile.cc |   73 +++++++++++++++
 bosslya/io/EnzoFile.h  |   27 ++++++
 configure              |   94 +++++++++++++++++++
 configure.ac           |    4 +
 script/enzo.cc         |   26 ++++++
 src/bossenzo.cc        |  236 ++++++++++++++++++++++++++++++++++++++++++++++++
 9 files changed, 515 insertions(+), 14 deletions(-)

To remind yourself how a particular file changed, the following commands are helpful:

% git log --oneline master..HEAD bosslya/bosslya.h
% git diff master..HEAD bosslya/bosslya.h

The first command shows the commit messages whenever the file changed, newest to oldest. The second command shows the net change to the file, line by line.

This is what your repo looks like now:

step3.png

Anyone can now see a log of the topic branches that have updated the master branch using:

% gitopic --log
04-Apr-2012, David Kirkby: Add feature set A

Note that Add feature set A is only the first line of the changes.txt you provided above. Use the option --llog instead of --log to see the whole documentation.

Now start a new topic branch for your next great idea:

gitopic --open Topic_C
...

In this case, you will automatically be building on top of your earlier topic and any other topics added by others:

step4.png

If the person working on topic B did not realize that a new topic had been merged back into the master and tried to close their branch, they would get the following error:

% gitopic --close changes.txt
gitopic: you need to --update your topic branch before you can --close it.

The update option requests that any changes to the master branch be folded into your ongoing work on a topic branch. This update step is where any conflicts will be detected (and when you will need to resolve them), but in this case there are no conflicts that cannot be automatically resolved and the update proceeds smoothly:

% gitopic --update
Successfully updated topic branch 'Topic_B'

The only rule is that you must update before you close your topic branch (unless there have been no changes to master, as above). However, it is generally best to update soon after any change to master to minimize the number of potential conflicts that you might have to resolve. If the master branch changes several times, you would generally do an update of your topic branches each time. Now, your repo looks like this and shows that the person working on topic B has updated to incorporate your completed topic A:

step5.png

The update command leaves your topic in the desired state in your local repo and on the server, but any other repos that had checked out your topic branch before the update will now be in a complicated "orphaned" state. This might happen, for example, if someone else (or perhaps you on a different computer) was testing your topic branch. If you try to checkout an orphaned branch, you will get warnings like this:

% git checkout Implement_unit_tests
Switched to branch 'Implement_unit_tests'
Your branch and 'origin/Implement_unit_tests' have diverged,
and have 2 and 15 different commit(s) each, respectively.

The following will also fail:

% git fetch
% git pull --ff-only

Assuming that you don't have any local modifications to the topic branch that need to be saved, you can either delete your local branch:

% git checkout master
% git branch -D <orphan>

or else, if you still need a local copy of the topic branch, do a hard reset to the current state:

% git reset --hard <orphan>

At any time, you can get an overview of all topic branches using:

% ../gitopic --list

== Local topic branches ==

        David Kirkby,  3 minutes ago: Analyze_enzo_simulation_outputs [+20,0]*

== Remote topic branches ==

        David Kirkby,  3 minutes ago: Analyze_enzo_simulation_outputs [+20,0]
   Michael Blomqvist,     5 days ago: Update_bosscont_QSO_information [+4,0]
   Michael Blomqvist,     5 days ago: Fit_fluxpdf_continuum [+17,-67]
      Daniel Margala,     7 days ago: Study_exposure_systematics [+69,-73]
      Daniel Margala,    3 weeks ago: Add_restframe_option_to_sky_analysis [+65,0]
      Daniel Margala,   10 weeks ago: adr_plate_guiding_corrections [+14,-88]

The first number in square brackets tell you how many commits you have made on your topic branch. The second number tells you how far behind the latest master branch you are (you will need to --update when this is negative, as described below).

Next, they close their topic B:

% gitopic --close changes.txt

so that the repo looks like this:

step6.png

Now it is your turn to update your topic C. However, this time there is a merge conflict and the update cannot proceed automatically:

% gitopic --update
gitopic: update found a merge conflict:

First, rewinding head to replay your work on top of it...
Applying: Commit 7
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging fileBC
CONFLICT (add/add): Merge conflict in fileBC
Failed to merge in the changes.
Patch failed at 0001 Commit 7

When you have resolved this problem run "git rebase --continue".
If you would prefer to skip this patch, instead run "git rebase --skip".
To restore the original branch and stop rebasing run "git rebase --abort".


gitopic: when the rebase is complete, do: git push --force origin Topic_C

If you are on a roll and really don't want to deal with this now, use the git rebase --abort command suggested above. Otherwise, you need to bite the bullet and resolve the conflict to finish the update. This is what an unresolved conflict looks like in your repo:

step7.png

The files with merge conflicts from commit 7 (fileBC in this example) need to be edited (lines with conflicts start with <<<<<<<) so that the topic C commits can be applied without conflicts. One solution that should always work is to force the merge to use the version of the conflicting files from topic C, which discards the conflicting lines from topic B:

% git checkout --theirs fileBC

You will normally want to do something better than this. When conflicts are resolved, add the updated files (but don't commit or push!) and continue to rebase:

% git add fileBC
% git rebase --continue

Finally, push your updated topic branch to the remote repo (using the command given in the last line of output from the gitopic --update ... above):

% git push --force origin Topic_C
Counting objects: 8, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 472 bytes, done.
Total 6 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
To /Users/david/git/sandbox/test.remote
 + 8bc7179...607cfec Topic_C -> Topic_C (forced update)

Now your repo looks like this:

step8.png

Finally, finish up and close your topic C:

% gitopic --close changes.txt

step9.png

Note that there should never be any conflicts when merging a topic branch back into the master, since these could only arise due to other topics being folded back into master and will have been caught and resolved during a rebase.

Working with remote repositories

The main commands for working with remote repos are remote, fetch, and push (there is also a pull command that you should generally avoid using).

To see what remotes your project already knows about:

% git remote -v
github  git@github-deepzot:deepzot/imengine.git (fetch)
github  git@github-deepzot:deepzot/imengine.git (push)
origin  gitosis@darkmatter.ps.uci.edu:imengine.git (fetch)
origin  gitosis@darkmatter.ps.uci.edu:imengine.git (push)

Note that each remote is listed twice (with separate fetch and push entries) and has a short alias (github and origin in this example). Use git remote add ... to add more remotes to your project. When you do not specify an alias with the fetch and push commands, origin will be used by default.

To synchronize your local repo with a remote, use git fetch [alias] which updates (and creates, if necessary) local branches with names like remotes/origin/master for each remote branch, without making any changes to your local branches (named master, etc). To see all your branches:

% git branch -a
* master
  remotes/github/master
  remotes/origin/master

You can generally omit the initial remotes/ when you refer to remote branches in commands. To see the differences between any two branches use:

git diff --stat <branch1>...<branch2>

The triple-dot syntax means that changes to <branch2> since it split off from <branch1> are displayed. Some particularly useful forms of this command are:

git diff --stat master...origin/master

to see any changes to the remote master. You can also review the changes on someone else's topic branch using:

git diff --stat origin/master...origin/<topic_branch>

To see changes to a specific file, remove the --stat option and add the filename at the end of the git diff command. To see how files changed with each commit, use git log --stat ... instead of git diff --stat ....

Adding a package to github/deepzot

Create the new package from the deepzot github account, then, e.g.

git remote add github git@github-deepzot:deepzot/likely.git
ssh-add ~/.ssh/deepzot-rsa
git push github master

Merging changes from a github fork

Declare the forked repo with a suitable alias (anze in this case):

git remote add anze https://github.com/slosar/baofit.git

Get a local copy of updates on the fork, e.g.:

git fetch anze

One approach is to cherry-pick specific commits, which effectively copies them over from the fork. The -x option adds some provenance to the commit message, e.g.:

git cherry-pick -x 19ee7a4b45ce

Otherwise, to merge an entire branch:

git merge anze/<...branch_name...>

New Workflow (slowly being replaced by sections above...)

The basic philosophy is described here. A slightly different workflow that is still useful to read is here.

Issues still to be decided:

  • Will we move to github hosting?
  • Will we use pull requests?
  • How and when are installed versions of packages updated on the server?
  • How do we use tags and implement version numbers?

I am expecting that we will usually work from our laptops via the SourceTree GUI, but command-line instructions below are provided for concreteness and reference.

The normal state of your laptop is probably that you have a local checkout of the master branch and any topic branches you are currently working on. List your local branches with (the asterisk shows your currently checked out branch):

 % git branch
  Update_pixel_mask_analysis
* Update_sky_residual_analysis
  master

The server repository should have all of these branches plus additional topic branches that others are working on. To see the current list, update your remote branches (this will not clobber your local work) and use:

 % git fetch
 % git branch -r
  origin/Update_pixel_mask_analysis
  origin/Update_sky_residual_analysis
  origin/adr_plate_guiding_corrections
  origin/master

See who is working on a topic branch:

 % git log origin/adr_plate_guiding_corrections -1
commit 44e1b50d9e6852b70e31ba419a8b8541bccb29ff
Author: Daniel Margala <dmargala@uci.edu>
Date:   Sun Nov 13 22:17:12 2011 -0800

    Provide throughput correction due to target centering offset (no star calibration yet)

Start a new topic branch

Start from the latest version of the master branch:

git checkout master
git pull --ff-only origin master

At this point, check that the package builds and runs correctly (this should always be true for the master branch, but check anyway).

Pick a short phrase describing what the new branch will provide and Turn_it_into_a_branch_name, then create your new branch locally:

 % git checkout -b Analyze_enzo_simulation_outputs
Switched to a new branch 'Analyze_enzo_simulation_outputs'

Finally, declare your new branch in the repo (this tells subsequent git push commands to propagate changes on your new branch):

 % git push origin Analyze_enzo_simulation_outputs

Work on a topic branch

Use git status, git add ... and git commit -m ... to incrementally develop in your local topic branch (this is much more intuitive via the SourceTree GUI). When you are done working on a topic for the day (or more often), push your changes to the repo using git push.

If you need to test a topic branch on a different computer (e.g., the server), make sure you have pushed the most recent version to the repo then, from the other computer:

cd <<pkg>>
git fetch
git checkout <<Topic_branch_name>>
git pull --ff-only
cd build
../configure
make

The first line is unnecessary (but harmless) if you are already using the topic branch on the other computer. If the second line fails, you probably made changes to the topic branch on the other computer that are now out of synch with the updated repo (try git reset ... to recover from this). Its ok to make changes on the second computer during your tests, but make sure to commit and push them back to the repo before you switch back to your laptop.

Test updates to master

Whenever the master branch is updated (presumably because a different topic branch has been completed), you should first update your master branch:

git checkout master
git pull --ff-only origin master

THIS PART STILL NOT FINALIZED Then rebase your topic branch with the new feature added to master (See here):

git checkout <topic-branch>
git rebase master

git pull --rebase

Finally, propagate the changes to the server:

git push origin <topic-branch>

If adding the new feature breaks your code, it is now your responsibility to make your code work with the new feature before adding your feature back to the master branch.

Switch between topic branches

Merge a topic branch back to master

(Should a version number tag be applied here? Need to understand the libtool conventions better).

First, merge your topic branch into the master branch on your local machine:

git checkout master
git merge --no-ff Analyze_enzo_simulation_outputs

Next delete your local copy of the branch:

git branch -d Analyze_enzo_simulation_outputs

Finally, push the updates to master (and the deletion of the topic branch) back to the repo:

git push origin master

The updated master is now the live production version, so install it into your system directories using sudo make install from your build directory and email everyone else who might be using this package so they can do the same. If you get such an email, you should:

cd <<pkgdir>>
git pull --ff-only origin master
cd build
../configure
make
sudo make install

If the git pull step fails, something is seriously wrong with your local master or the new update.

Abandon a topic branch

To abandon the current topic branch use:

gitopic --abandon

Working with GITHUB

Github identifies you via your sshkey, not a username. To work with more than one github account, see here.

Old Workflow

Initial Checkout

To make a local copy of a package from the server repository:

git clone gitosis@darkmatter.ps.uci.edu:<pkgname>.git

This will create a new subdirectory <pkgname>/ of your current working directory, containing the latest version of the master branch (and all of its history, via the contents of a .git/ subdirectory).

Working on the Server

New Instructions

Check that your local host is advertising the same key that you are registered with (its ok if it is also advertising other keys):

% ssh-add -L

Connect to the server with "agent-forwarding" enabled (-A). I usually add the -Y option also so that X-windows are directed back to my local host:

% ssh -A -Y my-user-id@darkmatter.ps.uci.edu

Check that the forwarding is working (you should see the same output you got from your local host):

% ssh-add -L

To update your branch on the server with changes pushed to the repo, use (master can be any branch name):

git fetch origin
git merge --ff-only origin/master

Old Instructions

To make a local copy of a package on the darkmatter server from the server's repository:

$ git clone -l /srv/gitosis/repositories/<pkgname>.git

Note Added 29-Oct-11: a local copy made this way appears to be read only, so you probably don't want to do this.

Upgrading an Old-Style Checkout

If you already have a local copy of a git repo, you can check how it was created using the following command from a local checkout directory:

git remote -v

The following output means that you have an old-style readonly copy (the name of the package at the end will generally be different):

origin  /srv/gitosis/repositories/bosslya.git (fetch)
origin  /srv/gitosis/repositories/bosslya.git (push)

If you have a working new-style read-write copy, you should see:

origin  gitosis@darkmatter.ps.uci.edu:bosslya.git (fetch)
origin  gitosis@darkmatter.ps.uci.edu:bosslya.git (push)

To convert a read-only copy to a read-write copy, use (make sure to change <<pkgname>> below):

git remote rm origin
git remote add origin gitosis@darkmatter.ps.uci.edu:<<pkgname>>.git

Update Local Version

Use git pull to pull changes from the repository into your working directory (and also update the history in .git/).

You can review the changes in the repository before pulling them in using:

git fetch
git diff ...origin/master
git pull origin master

Local Changes

Use git diff to see any changes you have made locally, including adding any files. Certain file types are ignored by GIT and so won't be flagged (.gitignore files are one way to configure this).

If you want to push all of the changes found by git diff back to the repository, use:

git add .
git commit -m '...some useful commit comment here...'
git push

To be more selective about which changed or new files you push, specify them in the git add command.

Tags

List existing tags:

git tag -n

Create a new signed tag (need to create a GPG identity first):

git tag -s -m '...tag comment here...' v<vers>

See here for recommendations on tag numbering.

To push only the new tag to a repo:

git push --tags origin

Make a tarball from an existing tag:

git archive --format=tar --prefix=<pkg>-<vers>/ v<vers> | gzip > <pkg>-<vers>.tar.gz

Repository Management

The darkmatter server runs gitosis. Currently only DK can edit admin files.

Register a new package

Update your local copy of the gitosis admin package:

cd ~/git/darkmatter/gitosis-admin
git pull

Edit gitosis.conf and append the new package name to the writable list under [group team].

Save the changes to the admin package:

git add .
git commit -m 'Registered new package'
git push

Push the package from a local git repository to the server, including any local tags:

git remote add origin gitosis@darkmatter.ps.uci.edu:<pkg>.git
git push --tags origin master

Register a new developer

Update your local copy of the gitosis admin package:

cd ~/git/darkmatter/gitosis-admin
git pull

Copy the new user's public ssh key into the admin package:

cp <userinfo>.pub keydir/
git add .

Edit gitosis.conf and append <userinfo> to the members list under [group team].

Save the changes to the admin package:

git add .
git commit -m 'Registered new developer'
git push

Debug SSH Problems

Check that your local ssh key is recognized by the GIT server:

% ssh gitosis@darkmatter.ps.uci.edu /bin/pwd
ERROR:gitosis.serve.main:Unknown command denied

The ERROR above is expected but if you get a different response (e.g., asking for a password) then your ssh public key is not registered with the server. In that case, check that your local host is actually advertising your public key using:

% ssh-add -L