progit.info-2

これは progit.info、progit.texi より makeinfo バージョン 4.8



File: progit.info,  Node: Ancestry References,  Next: Commit Ranges,  Prev: RefLog Shortnames,  Up: Revision Selection

6.1.6 Ancestry References
-------------------------

The other main way to specify a commit is via its ancestry. If you
place a `^' at the end of a reference, Git resolves it to mean the
parent of that commit. Suppose you look at the history of your project:

$ git log --pretty=format:'%h %s' --graph
* 734713b fixed refs handling, added gc auto, updated tests
*   d921970 Merge commit 'phedders/rdocs'
|\
| * 35cfb2b Some rdoc changes
* | 1c002dd added some blame and merge stuff
|/
* 1c36188 ignore *.gem
* 9b29157 add open3_detach to gemspec file list

   Then, you can see the previous commit by specifying `HEAD^', which
means "the parent of HEAD":

$ git show HEAD^
commit d921970aadf03b3cf0e71becdaab3147ba71cdef
Merge: 1c002dd... 35cfb2b...
Author: Scott Chacon <schacon@gmail.com>
Date:   Thu Dec 11 15:08:43 2008 -0800

    Merge commit 'phedders/rdocs'

   You can also specify a number after the `^' --- for example,
`d921970^2' means "the second parent of d921970." This syntax is only
useful for merge commits, which have more than one parent. The first
parent is the branch you were on when you merged, and the second is the
commit on the branch that you merged in:

$ git show d921970^
commit 1c002dd4b536e7479fe34593e72e6c6c1819e53b
Author: Scott Chacon <schacon@gmail.com>
Date:   Thu Dec 11 14:58:32 2008 -0800

    added some blame and merge stuff

$ git show d921970^2
commit 35cfb2b795a55793d7cc56a6cc2060b4bb732548
Author: Paul Hedderly <paul+git@mjr.org>
Date:   Wed Dec 10 22:22:03 2008 +0000

    Some rdoc changes

   The other main ancestry specification is the `~'. This also refers
to the first parent, so `HEAD~' and `HEAD^' are equivalent. The
difference becomes apparent when you specify a number. `HEAD~2' means
"the first parent of the first parent," or "the grandparent" --- it
traverses the first parents the number of times you specify. For
example, in the history listed earlier, `HEAD~3' would be

$ git show HEAD~3
commit 1c3618887afb5fbcbea25b7c013f4e2114448b8d
Author: Tom Preston-Werner <tom@mojombo.com>
Date:   Fri Nov 7 13:47:59 2008 -0500

    ignore *.gem

   This can also be written `HEAD^^^', which again is the first parent
of the first parent of the first parent:

$ git show HEAD^^^
commit 1c3618887afb5fbcbea25b7c013f4e2114448b8d
Author: Tom Preston-Werner <tom@mojombo.com>
Date:   Fri Nov 7 13:47:59 2008 -0500

    ignore *.gem

   You can also combine these syntaxes --- you can get the second
parent of the previous reference (assuming it was a merge commit) by
using `HEAD~3^2', and so on.


File: progit.info,  Node: Commit Ranges,  Prev: Ancestry References,  Up: Revision Selection

6.1.7 Commit Ranges
-------------------

Now that you can specify individual commits, let's see how to specify
ranges of commits. This is particularly useful for managing your
branches --- if you have a lot of branches, you can use range
specifications to answer questions such as, "What work is on this
branch that I haven't yet merged into my main branch?"

* Menu:

* Double Dot::
* Multiple Points::
* Triple Dot::


File: progit.info,  Node: Double Dot,  Next: Multiple Points,  Up: Commit Ranges

6.1.7.1 Double Dot
..................

The most common range specification is the double-dot syntax. This
basically asks Git to resolve a range of commits that are reachable
from one commit but aren't reachable from another. For example, say
you have a commit history that looks like Figure 6-1.

[image src="../figures/18333fig0601-tn.png" alt="Figure 6-1. Example history for range selection."]

Figure 6-1. Example history for range selection.

   You want to see what is in your experiment branch that hasn't yet
been merged into your master branch. You can ask Git to show you a log
of just those commits with `master..experiment' --- that means "all
commits reachable by experiment that aren't reachable by master." For
the sake of brevity and clarity in these examples, I'll use the
letters of the commit objects from the diagram in place of the actual
log output in the order that they would display:

$ git log master..experiment
D
C

   If, on the other hand, you want to see the opposite --- all commits
in `master' that aren't in `experiment' --- you can reverse the
branch names. `experiment..master' shows you everything in `master' not
reachable from `experiment':

$ git log experiment..master
F
E

   This is useful if you want to keep the `experiment' branch up to
date and preview what you're about to merge in. Another very frequent
use of this syntax is to see what you're about to push to a remote:

$ git log origin/master..HEAD

   This command shows you any commits in your current branch that
aren't in the `master' branch on your `origin' remote. If you run a
`git push' and your current branch is tracking `origin/master', the
commits listed by `git log origin/master..HEAD' are the commits that
will be transferred to the server. You can also leave off one side of
the syntax to have Git assume HEAD. For example, you can get the same
results as in the previous example by typing `git log origin/master..'
--- Git substitutes HEAD if one side is missing.


File: progit.info,  Node: Multiple Points,  Next: Triple Dot,  Prev: Double Dot,  Up: Commit Ranges

6.1.7.2 Multiple Points
.......................

The double-dot syntax is useful as a shorthand; but perhaps you want to
specify more than two branches to indicate your revision, such as
seeing what commits are in any of several branches that aren't in the
branch you're currently on. Git allows you to do this by using either
the `^' character or `--not' before any reference from which you
don't want to see reachable commits. Thus these three commands are
equivalent:

$ git log refA..refB
$ git log ^refA refB
$ git log refB --not refA

   This is nice because with this syntax you can specify more than two
references in your query, which you cannot do with the double-dot
syntax. For instance, if you want to see all commits that are reachable
from `refA' or `refB' but not from `refC', you can type one of these:

$ git log refA refB ^refC
$ git log refA refB --not refC

   This makes for a very powerful revision query system that should
help you figure out what is in your branches.


File: progit.info,  Node: Triple Dot,  Prev: Multiple Points,  Up: Commit Ranges

6.1.7.3 Triple Dot
..................

The last major range-selection syntax is the triple-dot syntax, which
specifies all the commits that are reachable by either of two
references but not by both of them. Look back at the example commit
history in Figure 6-1. If you want to see what is in `master' or
`experiment' but not any common references, you can run

$ git log master...experiment
F
E
D
C

   Again, this gives you normal `log' output but shows you only the
commit information for those four commits, appearing in the traditional
commit date ordering.

   A common switch to use with the `log' command in this case is
`--left-right', which shows you which side of the range each commit is
in. This helps make the data more useful:

$ git log --left-right master...experiment
< F
< E
> D
> C

   With these tools, you can much more easily let Git know what commit
or commits you want to inspect.


File: progit.info,  Node: Interactive Staging,  Next: Stashing,  Prev: Revision Selection,  Up: Git Tools

6.2 Interactive Staging
=======================

Git comes with a couple of scripts that make some command-line tasks
easier. Here, you'll look at a few interactive commands that can help
you easily craft your commits to include only certain combinations and
parts of files. These tools are very helpful if you modify a bunch of
files and then decide that you want those changes to be in several
focused commits rather than one big messy commit. This way, you can
make sure your commits are logically separate changesets and can be
easily reviewed by the developers working with you. If you run `git
add' with the `-i' or `--interactive' option, Git goes into an
interactive shell mode, displaying something like this:

$ git add -i
           staged     unstaged path
  1:    unchanged        +0/-1 TODO
  2:    unchanged        +1/-1 index.html
  3:    unchanged        +5/-1 lib/simplegit.rb

*** Commands ***
  1: status     2: update      3: revert     4: add untracked
  5: patch      6: diff        7: quit       8: help
What now>

   You can see that this command shows you a much different view of
your staging area --- basically the same information you get with `git
status' but a bit more succinct and informative. It lists the changes
you've staged on the left and unstaged changes on the right.

   After this comes a Commands section. Here you can do a number of
things, including staging files, unstaging files, staging parts of
files, adding untracked files, and seeing diffs of what has been staged.

* Menu:

* Staging and Unstaging Files::
* Staging Patches::


File: progit.info,  Node: Staging and Unstaging Files,  Next: Staging Patches,  Up: Interactive Staging

6.2.1 Staging and Unstaging Files
---------------------------------

If you type `2' or `u' at the `What now>' prompt, the script prompts
you for which files you want to stage:

What now> 2
           staged     unstaged path
  1:    unchanged        +0/-1 TODO
  2:    unchanged        +1/-1 index.html
  3:    unchanged        +5/-1 lib/simplegit.rb
Update>>

   To stage the TODO and index.html files, you can type the numbers:

Update>> 1,2
           staged     unstaged path
* 1:    unchanged        +0/-1 TODO
* 2:    unchanged        +1/-1 index.html
  3:    unchanged        +5/-1 lib/simplegit.rb
Update>>

   The `*' next to each file means the file is selected to be staged.
If you press Enter after typing nothing at the `Update>>' prompt, Git
takes anything selected and stages it for you:

Update>>
updated 2 paths

*** Commands ***
  1: status     2: update      3: revert     4: add untracked
  5: patch      6: diff        7: quit       8: help
What now> 1
           staged     unstaged path
  1:        +0/-1      nothing TODO
  2:        +1/-1      nothing index.html
  3:    unchanged        +5/-1 lib/simplegit.rb

   Now you can see that the TODO and index.html files are staged and
the simplegit.rb file is still unstaged. If you want to unstage the
TODO file at this point, you use the `3' or `r' (for revert) option:

*** Commands ***
  1: status     2: update      3: revert     4: add untracked
  5: patch      6: diff        7: quit       8: help
What now> 3
           staged     unstaged path
  1:        +0/-1      nothing TODO
  2:        +1/-1      nothing index.html
  3:    unchanged        +5/-1 lib/simplegit.rb
Revert>> 1
           staged     unstaged path
* 1:        +0/-1      nothing TODO
  2:        +1/-1      nothing index.html
  3:    unchanged        +5/-1 lib/simplegit.rb
Revert>> [enter]
reverted one path

   Looking at your Git status again, you can see that you've unstaged
the TODO file:

*** Commands ***
  1: status     2: update      3: revert     4: add untracked
  5: patch      6: diff        7: quit       8: help
What now> 1
           staged     unstaged path
  1:    unchanged        +0/-1 TODO
  2:        +1/-1      nothing index.html
  3:    unchanged        +5/-1 lib/simplegit.rb

   To see the diff of what you've staged, you can use the `6' or `d'
(for diff) command. It shows you a list of your staged files, and you
can select the ones for which you would like to see the staged diff.
This is much like specifying `git diff --cached' on the command line:

*** Commands ***
  1: status     2: update      3: revert     4: add untracked
  5: patch      6: diff        7: quit       8: help
What now> 6
           staged     unstaged path
  1:        +1/-1      nothing index.html
Review diff>> 1
diff --git a/index.html b/index.html
index 4d07108..4335f49 100644
--- a/index.html
+++ b/index.html
@@ -16,7 +16,7 @@ Date Finder

 <p id="out">...</p>

-<div id="footer">contact : support@github.com</div>
+<div id="footer">contact : email.support@github.com</div>

 <script type="text/javascript">

   With these basic commands, you can use the interactive add mode to
deal with your staging area a little more easily.


File: progit.info,  Node: Staging Patches,  Prev: Staging and Unstaging Files,  Up: Interactive Staging

6.2.2 Staging Patches
---------------------

It's also possible for Git to stage certain parts of files and not
the rest. For example, if you make two changes to your simplegit.rb
file and want to stage one of them and not the other, doing so is very
easy in Git. From the interactive prompt, type `5' or `p' (for patch).
Git will ask you which files you would like to partially stage; then,
for each section of the selected files, it will display hunks of the
file diff and ask if you would like to stage them, one by one:

diff --git a/lib/simplegit.rb b/lib/simplegit.rb
index dd5ecc4..57399e0 100644
--- a/lib/simplegit.rb
+++ b/lib/simplegit.rb
@@ -22,7 +22,7 @@ class SimpleGit
   end

   def log(treeish = 'master')
-    command("git log -n 25 #{treeish}")
+    command("git log -n 30 #{treeish}")
   end

   def blame(path)
Stage this hunk [y,n,a,d,/,j,J,g,e,?]?

   You have a lot of options at this point. Typing `?' shows a list of
what you can do:

Stage this hunk [y,n,a,d,/,j,J,g,e,?]? ?
y - stage this hunk
n - do not stage this hunk
a - stage this and all the remaining hunks in the file
d - do not stage this hunk nor any of the remaining hunks in the file
g - select a hunk to go to
/ - search for a hunk matching the given regex
j - leave this hunk undecided, see next undecided hunk
J - leave this hunk undecided, see next hunk
k - leave this hunk undecided, see previous undecided hunk
K - leave this hunk undecided, see previous hunk
s - split the current hunk into smaller hunks
e - manually edit the current hunk
? - print help

   Generally, you'll type `y' or `n' if you want to stage each hunk,
but staging all of them in certain files or skipping a hunk decision
until later can be helpful too. If you stage one part of the file and
leave another part unstaged, your status output will look like this:

What now> 1
           staged     unstaged path
  1:    unchanged        +0/-1 TODO
  2:        +1/-1      nothing index.html
  3:        +1/-1        +4/-0 lib/simplegit.rb

   The status of the simplegit.rb file is interesting. It shows you
that a couple of lines are staged and a couple are unstaged. You've
partially staged this file. At this point, you can exit the interactive
adding script and run `git commit' to commit the partially staged files.

   Finally, you don't need to be in interactive add mode to do the
partial-file staging --- you can start the same script by using `git
add -p' or `git add --patch' on the command line.


File: progit.info,  Node: Stashing,  Next: Rewriting History,  Prev: Interactive Staging,  Up: Git Tools

6.3 Stashing
============

Often, when you've been working on part of your project, things are
in a messy state and you want to switch branches for a bit to work on
something else. The problem is, you don't want to do a commit of
half-done work just so you can get back to this point later. The answer
to this issue is the `git stash' command.

   Stashing takes the dirty state of your working directory --- that
is, your modified tracked files and staged changes --- and saves it on
a stack of unfinished changes that you can reapply at any time.

* Menu:

* Stashing Your Work::
* Un-applying a Stash::
* Creating a Branch from a Stash::


File: progit.info,  Node: Stashing Your Work,  Next: Un-applying a Stash,  Up: Stashing

6.3.1 Stashing Your Work
------------------------

To demonstrate, you'll go into your project and start working on a
couple of files and possibly stage one of the changes. If you run `git
status', you can see your dirty state:

$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#      modified:   index.html
#
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#
#      modified:   lib/simplegit.rb
#

   Now you want to switch branches, but you don't want to commit what
you've been working on yet; so you'll stash the changes. To push a
new stash onto your stack, run `git stash':

$ git stash
Saved working directory and index state \
  "WIP on master: 049d078 added the index file"
HEAD is now at 049d078 added the index file
(To restore them type "git stash apply")

   Your working directory is clean:

$ git status
# On branch master
nothing to commit (working directory clean)

   At this point, you can easily switch branches and do work elsewhere;
your changes are stored on your stack. To see which stashes you've
stored, you can use `git stash list':

$ git stash list
stash@{0}: WIP on master: 049d078 added the index file
stash@{1}: WIP on master: c264051... Revert "added file_size"
stash@{2}: WIP on master: 21d80a5... added number to log

   In this case, two stashes were done previously, so you have access
to three different stashed works. You can reapply the one you just
stashed by using the command shown in the help output of the original
stash command: `git stash apply'. If you want to apply one of the older
stashes, you can specify it by naming it, like this: `git stash apply
stash@{2}'. If you don't specify a stash, Git assumes the most recent
stash and tries to apply it:

$ git stash apply
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#
#      modified:   index.html
#      modified:   lib/simplegit.rb
#

   You can see that Git re-modifies the files you uncommitted when you
saved the stash. In this case, you had a clean working directory when
you tried to apply the stash, and you tried to apply it on the same
branch you saved it from; but having a clean working directory and
applying it on the same branch aren't necessary to successfully apply
a stash. You can save a stash on one branch, switch to another branch
later, and try to reapply the changes. You can also have modified and
uncommitted files in your working directory when you apply a stash ---
Git gives you merge conflicts if anything no longer applies cleanly.

   The changes to your files were reapplied, but the file you staged
before wasn't restaged. To do that, you must run the `git stash
apply' command with a `--index' option to tell the command to try to
reapply the staged changes. If you had run that instead, you'd have
gotten back to your original position:

$ git stash apply --index
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#      modified:   index.html
#
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#
#      modified:   lib/simplegit.rb
#

   The apply option only tries to apply the stashed work --- you
continue to have it on your stack. To remove it, you can run `git stash
drop' with the name of the stash to remove:

$ git stash list
stash@{0}: WIP on master: 049d078 added the index file
stash@{1}: WIP on master: c264051... Revert "added file_size"
stash@{2}: WIP on master: 21d80a5... added number to log
$ git stash drop stash@{0}
Dropped stash@{0} (364e91f3f268f0900bc3ee613f9f733e82aaed43)

   You can also run `git stash pop' to apply the stash and then
immediately drop it from your stack.


File: progit.info,  Node: Un-applying a Stash,  Next: Creating a Branch from a Stash,  Prev: Stashing Your Work,  Up: Stashing

6.3.2 Un-applying a Stash
-------------------------

In some use case scenarios you might want to apply stashed changes, do
some work, but then un-apply those changes that originally came form
the stash. Git does not provide such a `stash unapply' command, but it
is possible to achieve the effect by simply retrieving the patch
associated with a stash and applying it in reverse:

$ git stash show -p stash@{0} | git apply -R

   Again, if you don't specify a stash, Git assumes the most recent
stash:

$ git stash show -p | git apply -R

   You may want to create an alias and effectively add a
`stash-unapply' command to your git. For example:

$ git config --global alias.stash-unapply '!git stash show -p | git apply -R'
$ git stash
$ #... work work work
$ git stash-unapply


File: progit.info,  Node: Creating a Branch from a Stash,  Prev: Un-applying a Stash,  Up: Stashing

6.3.3 Creating a Branch from a Stash
------------------------------------

If you stash some work, leave it there for a while, and continue on the
branch from which you stashed the work, you may have a problem
reapplying the work. If the apply tries to modify a file that you've
since modified, you'll get a merge conflict and will have to try to
resolve it. If you want an easier way to test the stashed changes
again, you can run `git stash branch', which creates a new branch for
you, checks out the commit you were on when you stashed your work,
reapplies your work there, and then drops the stash if it applies
successfully:

$ git stash branch testchanges
Switched to a new branch "testchanges"
# On branch testchanges
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#      modified:   index.html
#
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#
#      modified:   lib/simplegit.rb
#
Dropped refs/stash@{0} (f0dfc4d5dc332d1cee34a634182e168c4efc3359)

   This is a nice shortcut to recover stashed work easily and work on
it in a new branch.


File: progit.info,  Node: Rewriting History,  Next: Debugging with Git,  Prev: Stashing,  Up: Git Tools

6.4 Rewriting History
=====================

Many times, when working with Git, you may want to revise your commit
history for some reason. One of the great things about Git is that it
allows you to make decisions at the last possible moment. You can
decide what files go into which commits right before you commit with
the staging area, you can decide that you didn't mean to be working
on something yet with the stash command, and you can rewrite commits
that already happened so they look like they happened in a different
way. This can involve changing the order of the commits, changing
messages or modifying files in a commit, squashing together or
splitting apart commits, or removing commits entirely --- all before
you share your work with others.

   In this section, you'll cover how to accomplish these very useful
tasks so that you can make your commit history look the way you want
before you share it with others.

* Menu:

* Changing the Last Commit::
* Changing Multiple Commit Messages::
* Reordering Commits::
* Squashing a Commit::
* Splitting a Commit::
* The Nuclear Option filter-branch::


File: progit.info,  Node: Changing the Last Commit,  Next: Changing Multiple Commit Messages,  Up: Rewriting History

6.4.1 Changing the Last Commit
------------------------------

Changing your last commit is probably the most common rewriting of
history that you'll do. You'll often want to do two basic things to
your last commit: change the commit message, or change the snapshot you
just recorded by adding, changing and removing files.

   If you only want to modify your last commit message, it's very
simple:

$ git commit --amend

   That drops you into your text editor, which has your last commit
message in it, ready for you to modify the message. When you save and
close the editor, the editor writes a new commit containing that
message and makes it your new last commit.

   If you've committed and then you want to change the snapshot you
committed by adding or changing files, possibly because you forgot to
add a newly created file when you originally committed, the process
works basically the same way. You stage the changes you want by editing
a file and running `git add' on it or `git rm' to a tracked file, and
the subsequent `git commit --amend' takes your current staging area and
makes it the snapshot for the new commit.

   You need to be careful with this technique because amending changes
the SHA-1 of the commit. It's like a very small rebase --- don't
amend your last commit if you've already pushed it.


File: progit.info,  Node: Changing Multiple Commit Messages,  Next: Reordering Commits,  Prev: Changing the Last Commit,  Up: Rewriting History

6.4.2 Changing Multiple Commit Messages
---------------------------------------

To modify a commit that is farther back in your history, you must move
to more complex tools. Git doesn't have a modify-history tool, but
you can use the rebase tool to rebase a series of commits onto the HEAD
they were originally based on instead of moving them to another one.
With the interactive rebase tool, you can then stop after each commit
you want to modify and change the message, add files, or do whatever
you wish. You can run rebase interactively by adding the `-i' option to
`git rebase'. You must indicate how far back you want to rewrite
commits by telling the command which commit to rebase onto.

   For example, if you want to change the last three commit messages,
or any of the commit messages in that group, you supply as an argument
to `git rebase -i' the parent of the last commit you want to edit,
which is `HEAD~2^' or `HEAD~3'. It may be easier to remember the `~3'
because you're trying to edit the last three commits; but keep in
mind that you're actually designating four commits ago, the parent of
the last commit you want to edit:

$ git rebase -i HEAD~3

   Remember again that this is a rebasing command --- every commit
included in the range `HEAD~3..HEAD' will be rewritten, whether you
change the message or not. Don't include any commit you've already
pushed to a central server --- doing so will confuse other developers
by providing an alternate version of the same change.

   Running this command gives you a list of commits in your text editor
that looks something like this:

pick f7f3f6d changed my name a bit
pick 310154e updated README formatting and added blame
pick a5f4a0d added cat-file

# Rebase 710f0f8..a5f4a0d onto 710f0f8
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

   It's important to note that these commits are listed in the
opposite order than you normally see them using the `log' command. If
you run a `log', you see something like this:

$ git log --pretty=format:"%h %s" HEAD~3..HEAD
a5f4a0d added cat-file
310154e updated README formatting and added blame
f7f3f6d changed my name a bit

   Notice the reverse order. The interactive rebase gives you a script
that it's going to run. It will start at the commit you specify on
the command line (`HEAD~3') and replay the changes introduced in each
of these commits from top to bottom. It lists the oldest at the top,
rather than the newest, because that's the first one it will replay.

   You need to edit the script so that it stops at the commit you want
to edit. To do so, change the word pick to the word edit for each of
the commits you want the script to stop after. For example, to modify
only the third commit message, you change the file to look like this:

edit f7f3f6d changed my name a bit
pick 310154e updated README formatting and added blame
pick a5f4a0d added cat-file

   When you save and exit the editor, Git rewinds you back to the last
commit in that list and drops you on the command line with the
following message:

$ git rebase -i HEAD~3
Stopped at 7482e0d... updated the gemspec to hopefully work better
You can amend the commit now, with

       git commit --amend

Once you're satisfied with your changes, run

       git rebase --continue

   These instructions tell you exactly what to do. Type

$ git commit --amend

   Change the commit message, and exit the editor. Then, run

$ git rebase --continue

   This command will apply the other two commits automatically, and
then you're done. If you change pick to edit on more lines, you can
repeat these steps for each commit you change to edit. Each time, Git
will stop, let you amend the commit, and continue when you're
finished.


File: progit.info,  Node: Reordering Commits,  Next: Squashing a Commit,  Prev: Changing Multiple Commit Messages,  Up: Rewriting History

6.4.3 Reordering Commits
------------------------

You can also use interactive rebases to reorder or remove commits
entirely. If you want to remove the "added cat-file" commit and change
the order in which the other two commits are introduced, you can change
the rebase script from this

pick f7f3f6d changed my name a bit
pick 310154e updated README formatting and added blame
pick a5f4a0d added cat-file

   to this:

pick 310154e updated README formatting and added blame
pick f7f3f6d changed my name a bit

   When you save and exit the editor, Git rewinds your branch to the
parent of these commits, applies `310154e' and then `f7f3f6d', and then
stops. You effectively change the order of those commits and remove the
"added cat-file" commit completely.


File: progit.info,  Node: Squashing a Commit,  Next: Splitting a Commit,  Prev: Reordering Commits,  Up: Rewriting History

6.4.4 Squashing a Commit
------------------------

It's also possible to take a series of commits and squash them down
into a single commit with the interactive rebasing tool. The script
puts helpful instructions in the rebase message:

#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

   If, instead of "pick" or "edit", you specify "squash", Git applies
both that change and the change directly before it and makes you merge
the commit messages together. So, if you want to make a single commit
from these three commits, you make the script look like this:

pick f7f3f6d changed my name a bit
squash 310154e updated README formatting and added blame
squash a5f4a0d added cat-file

   When you save and exit the editor, Git applies all three changes and
then puts you back into the editor to merge the three commit messages:

# This is a combination of 3 commits.
# The first commit's message is:
changed my name a bit

# This is the 2nd commit message:

updated README formatting and added blame

# This is the 3rd commit message:

added cat-file

   When you save that, you have a single commit that introduces the
changes of all three previous commits.


File: progit.info,  Node: Splitting a Commit,  Next: The Nuclear Option filter-branch,  Prev: Squashing a Commit,  Up: Rewriting History

6.4.5 Splitting a Commit
------------------------

Splitting a commit undoes a commit and then partially stages and
commits as many times as commits you want to end up with. For example,
suppose you want to split the middle commit of your three commits.
Instead of "updated README formatting and added blame", you want to
split it into two commits: "updated README formatting" for the first,
and "added blame" for the second. You can do that in the `rebase -i'
script by changing the instruction on the commit you want to split to
"edit":

pick f7f3f6d changed my name a bit
edit 310154e updated README formatting and added blame
pick a5f4a0d added cat-file

   Then, when the script drops you to the command line, you reset that
commit, take the changes that have been reset, and create multiple
commits out of them. When you save and exit the editor, Git rewinds to
the parent of the first commit in your list, applies the first commit
(`f7f3f6d'), applies the second (`310154e'), and drops you to the
console. There, you can do a mixed reset of that commit with `git reset
HEAD^', which effectively undoes that commit and leaves the modified
files unstaged. Now you can stage and commit files until you have
several commits, and run `git rebase --continue' when you're done:

$ git reset HEAD^
$ git add README
$ git commit -m 'updated README formatting'
$ git add lib/simplegit.rb
$ git commit -m 'added blame'
$ git rebase --continue

   Git applies the last commit (`a5f4a0d') in the script, and your
history looks like this:

$ git log -4 --pretty=format:"%h %s"
1c002dd added cat-file
9b29157 added blame
35cfb2b updated README formatting
f3cc40e changed my name a bit

   Once again, this changes the SHAs of all the commits in your list,
so make sure no commit shows up in that list that you've already
pushed to a shared repository.


File: progit.info,  Node: The Nuclear Option filter-branch,  Prev: Splitting a Commit,  Up: Rewriting History

6.4.6 The Nuclear Option: filter-branch
---------------------------------------

There is another history-rewriting option that you can use if you need
to rewrite a larger number of commits in some scriptable way --- for
instance, changing your e-mail address globally or removing a file from
every commit. The command is `filter-branch', and it can rewrite huge
swaths of your history, so you probably shouldn't use it unless your
project isn't yet public and other people haven't based work off
the commits you're about to rewrite. However, it can be very useful.
You'll learn a few of the common uses so you can get an idea of some
of the things it's capable of.

* Menu:

* Removing a File from Every Commit::
* Making a Subdirectory the New Root::
* Changing E-Mail Addresses Globally::


File: progit.info,  Node: Removing a File from Every Commit,  Next: Making a Subdirectory the New Root,  Up: The Nuclear Option filter-branch

6.4.6.1 Removing a File from Every Commit
.........................................

This occurs fairly commonly. Someone accidentally commits a huge binary
file with a thoughtless `git add .', and you want to remove it
everywhere. Perhaps you accidentally committed a file that contained a
password, and you want to make your project open source.
`filter-branch' is the tool you probably want to use to scrub your
entire history. To remove a file named passwords.txt from your entire
history, you can use the `--tree-filter' option to `filter-branch':

$ git filter-branch --tree-filter 'rm -f passwords.txt' HEAD
Rewrite 6b9b3cf04e7c5686a9cb838c3f36a8cb6a0fc2bd (21/21)
Ref 'refs/heads/master' was rewritten

   The `--tree-filter' option runs the specified command after each
checkout of the project and then recommits the results. In this case,
you remove a file called passwords.txt from every snapshot, whether it
exists or not. If you want to remove all accidentally committed editor
backup files, you can run something like `git filter-branch
--tree-filter 'rm -f *~' HEAD'.

   You'll be able to watch Git rewriting trees and commits and then
move the branch pointer at the end. It's generally a good idea to do
this in a testing branch and then hard-reset your master branch after
you've determined the outcome is what you really want. To run
`filter-branch' on all your branches, you can pass `--all' to the
command.


File: progit.info,  Node: Making a Subdirectory the New Root,  Next: Changing E-Mail Addresses Globally,  Prev: Removing a File from Every Commit,  Up: The Nuclear Option filter-branch

6.4.6.2 Making a Subdirectory the New Root
..........................................

Suppose you've done an import from another source control system and
have subdirectories that make no sense (trunk, tags, and so on). If you
want to make the `trunk' subdirectory be the new project root for every
commit, `filter-branch' can help you do that, too:

$ git filter-branch --subdirectory-filter trunk HEAD
Rewrite 856f0bf61e41a27326cdae8f09fe708d679f596f (12/12)
Ref 'refs/heads/master' was rewritten

   Now your new project root is what was in the `trunk' subdirectory
each time. Git will also automatically remove commits that did not
affect the subdirectory.


File: progit.info,  Node: Changing E-Mail Addresses Globally,  Prev: Making a Subdirectory the New Root,  Up: The Nuclear Option filter-branch

6.4.6.3 Changing E-Mail Addresses Globally
..........................................

Another common case is that you forgot to run `git config' to set your
name and e-mail address before you started working, or perhaps you want
to open-source a project at work and change all your work e-mail
addresses to your personal address. In any case, you can change e-mail
addresses in multiple commits in a batch with `filter-branch' as well.
You need to be careful to change only the e-mail addresses that are
yours, so you use `--commit-filter':

$ git filter-branch --commit-filter '
        if [ "$GIT_AUTHOR_EMAIL" = "schacon@localhost" ];
        then
                GIT_AUTHOR_NAME="Scott Chacon";
                GIT_AUTHOR_EMAIL="schacon@example.com";
                git commit-tree "$@";
        else
                git commit-tree "$@";
        fi' HEAD

   This goes through and rewrites every commit to have your new
address. Because commits contain the SHA-1 values of their parents,
this command changes every commit SHA in your history, not just those
that have the matching e-mail address.


File: progit.info,  Node: Debugging with Git,  Next: Submodules,  Prev: Rewriting History,  Up: Git Tools

6.5 Debugging with Git
======================

Git also provides a couple of tools to help you debug issues in your
projects. Because Git is designed to work with nearly any type of
project, these tools are pretty generic, but they can often help you
hunt for a bug or culprit when things go wrong.

* Menu:

* File Annotation::
* Binary Search::


File: progit.info,  Node: File Annotation,  Next: Binary Search,  Up: Debugging with Git

6.5.1 File Annotation
---------------------

If you track down a bug in your code and want to know when it was
introduced and why, file annotation is often your best tool. It shows
you what commit was the last to modify each line of any file. So, if
you see that a method in your code is buggy, you can annotate the file
with `git blame' to see when each line of the method was last edited
and by whom. This example uses the `-L' option to limit the output to
lines 12 through 22:

$ git blame -L 12,22 simplegit.rb
^4832fe2 (Scott Chacon  2008-03-15 10:31:28 -0700 12)  def show(tree = 'master')
^4832fe2 (Scott Chacon  2008-03-15 10:31:28 -0700 13)   command("git show #{tree}")
^4832fe2 (Scott Chacon  2008-03-15 10:31:28 -0700 14)  end
^4832fe2 (Scott Chacon  2008-03-15 10:31:28 -0700 15)
9f6560e4 (Scott Chacon  2008-03-17 21:52:20 -0700 16)  def log(tree = 'master')
79eaf55d (Scott Chacon  2008-04-06 10:15:08 -0700 17)   command("git log #{tree}")
9f6560e4 (Scott Chacon  2008-03-17 21:52:20 -0700 18)  end
9f6560e4 (Scott Chacon  2008-03-17 21:52:20 -0700 19)
42cf2861 (Magnus Chacon 2008-04-13 10:45:01 -0700 20)  def blame(path)
42cf2861 (Magnus Chacon 2008-04-13 10:45:01 -0700 21)   command("git blame #{path}")
42cf2861 (Magnus Chacon 2008-04-13 10:45:01 -0700 22)  end

   Notice that the first field is the partial SHA-1 of the commit that
last modified that line. The next two fields are values extracted from
that commit---the author name and the authored date of that commit ---
so you can easily see who modified that line and when. After that come
the line number and the content of the file. Also note the `^4832fe2'
commit lines, which designate that those lines were in this file's
original commit. That commit is when this file was first added to this
project, and those lines have been unchanged since. This is a tad
confusing, because now you've seen at least three different ways that
Git uses the `^' to modify a commit SHA, but that is what it means here.

   Another cool thing about Git is that it doesn't track file renames
explicitly. It records the snapshots and then tries to figure out what
was renamed implicitly, after the fact. One of the interesting features
of this is that you can ask it to figure out all sorts of code movement
as well. If you pass `-C' to `git blame', Git analyzes the file
you're annotating and tries to figure out where snippets of code
within it originally came from if they were copied from elsewhere.
Recently, I was refactoring a file named `GITServerHandler.m' into
multiple files, one of which was `GITPackUpload.m'. By blaming
`GITPackUpload.m' with the `-C' option, I could see where sections of
the code originally came from:

$ git blame -C -L 141,153 GITPackUpload.m
f344f58d GITServerHandler.m (Scott 2009-01-04 141)
f344f58d GITServerHandler.m (Scott 2009-01-04 142) - (void) gatherObjectShasFromC
f344f58d GITServerHandler.m (Scott 2009-01-04 143) {
70befddd GITServerHandler.m (Scott 2009-03-22 144)         //NSLog(@"GATHER COMMI
ad11ac80 GITPackUpload.m    (Scott 2009-03-24 145)
ad11ac80 GITPackUpload.m    (Scott 2009-03-24 146)         NSString *parentSha;
ad11ac80 GITPackUpload.m    (Scott 2009-03-24 147)         GITCommit *commit = [g
ad11ac80 GITPackUpload.m    (Scott 2009-03-24 148)
ad11ac80 GITPackUpload.m    (Scott 2009-03-24 149)         //NSLog(@"GATHER COMMI
ad11ac80 GITPackUpload.m    (Scott 2009-03-24 150)
56ef2caf GITServerHandler.m (Scott 2009-01-05 151)         if(commit) {
56ef2caf GITServerHandler.m (Scott 2009-01-05 152)                 [refDict setOb
56ef2caf GITServerHandler.m (Scott 2009-01-05 153)

   This is really useful. Normally, you get as the original commit the
commit where you copied the code over, because that is the first time
you touched those lines in this file. Git tells you the original commit
where you wrote those lines, even if it was in another file.


File: progit.info,  Node: Binary Search,  Prev: File Annotation,  Up: Debugging with Git

6.5.2 Binary Search
-------------------

Annotating a file helps if you know where the issue is to begin with.
If you don't know what is breaking, and there have been dozens or
hundreds of commits since the last state where you know the code
worked, you'll likely turn to `git bisect' for help. The `bisect'
command does a binary search through your commit history to help you
identify as quickly as possible which commit introduced an issue.

   Let's say you just pushed out a release of your code to a
production environment, you're getting bug reports about something
that wasn't happening in your development environment, and you
can't imagine why the code is doing that. You go back to your code,
and it turns out you can reproduce the issue, but you can't figure
out what is going wrong. You can bisect the code to find out. First you
run `git bisect start' to get things going, and then you use `git
bisect bad' to tell the system that the current commit you're on is
broken. Then, you must tell bisect when the last known good state was,
using `git bisect good [good_commit]':

$ git bisect start
$ git bisect bad
$ git bisect good v1.0
Bisecting: 6 revisions left to test after this
[ecb6e1bc347ccecc5f9350d878ce677feb13d3b2] error handling on repo

   Git figured out that about 12 commits came between the commit you
marked as the last good commit (v1.0) and the current bad version, and
it checked out the middle one for you. At this point, you can run your
test to see if the issue exists as of this commit. If it does, then it
was introduced sometime before this middle commit; if it doesn't,
then the problem was introduced sometime after the middle commit. It
turns out there is no issue here, and you tell Git that by typing `git
bisect good' and continue your journey:

$ git bisect good
Bisecting: 3 revisions left to test after this
[b047b02ea83310a70fd603dc8cd7a6cd13d15c04] secure this thing

   Now you're on another commit, halfway between the one you just
tested and your bad commit. You run your test again and find that this
commit is broken, so you tell Git that with `git bisect bad':

$ git bisect bad
Bisecting: 1 revisions left to test after this
[f71ce38690acf49c1f3c9bea38e09d82a5ce6014] drop exceptions table

   This commit is fine, and now Git has all the information it needs to
determine where the issue was introduced. It tells you the SHA-1 of the
first bad commit and show some of the commit information and which
files were modified in that commit so you can figure out what happened
that may have introduced this bug:

$ git bisect good
b047b02ea83310a70fd603dc8cd7a6cd13d15c04 is first bad commit
commit b047b02ea83310a70fd603dc8cd7a6cd13d15c04
Author: PJ Hyett <pjhyett@example.com>
Date:   Tue Jan 27 14:48:32 2009 -0800

    secure this thing

:040000 040000 40ee3e7821b895e52c1695092db9bdc4c61d1730
f24d3c6ebcfc639b1a3814550e62d60b8e68a8e4 M  config

   When you're finished, you should run `git bisect reset' to reset
your HEAD to where you were before you started, or you'll end up in a
weird state:

$ git bisect reset

   This is a powerful tool that can help you check hundreds of commits
for an introduced bug in minutes. In fact, if you have a script that
will exit 0 if the project is good or non-0 if the project is bad, you
can fully automate `git bisect'. First, you again tell it the scope of
the bisect by providing the known bad and good commits. You can do this
by listing them with the `bisect start' command if you want, listing
the known bad commit first and the known good commit second:

$ git bisect start HEAD v1.0
$ git bisect run test-error.sh

   Doing so automatically runs `test-error.sh' on each checked-out
commit until Git finds the first broken commit. You can also run
something like `make' or `make tests' or whatever you have that runs
automated tests for you.


File: progit.info,  Node: Submodules,  Next: Subtree Merging,  Prev: Debugging with Git,  Up: Git Tools

6.6 Submodules
==============

It often happens that while working on one project, you need to use
another project from within it. Perhaps it's a library that a third
party developed or that you're developing separately and using in
multiple parent projects. A common issue arises in these scenarios: you
want to be able to treat the two projects as separate yet still be able
to use one from within the other.

   Here's an example. Suppose you're developing a web site and
creating Atom feeds. Instead of writing your own Atom-generating code,
you decide to use a library. You're likely to have to either include
this code from a shared library like a CPAN install or Ruby gem, or
copy the source code into your own project tree. The issue with
including the library is that it's difficult to customize the library
in any way and often more difficult to deploy it, because you need to
make sure every client has that library available. The issue with
vendoring the code into your own project is that any custom changes you
make are difficult to merge when upstream changes become available.

   Git addresses this issue using submodules. Submodules allow you to
keep a Git repository as a subdirectory of another Git repository. This
lets you clone another repository into your project and keep your
commits separate.

* Menu:

* Starting with Submodules::
* Cloning a Project with Submodules::
* Superprojects::
* Issues with Submodules::


File: progit.info,  Node: Starting with Submodules,  Next: Cloning a Project with Submodules,  Up: Submodules

6.6.1 Starting with Submodules
------------------------------

Suppose you want to add the Rack library (a Ruby web server gateway
interface) to your project, possibly maintain your own changes to it,
but continue to merge in upstream changes. The first thing you should
do is clone the external repository into your subdirectory. You add
external projects as submodules with the `git submodule add' command:

$ git submodule add git://github.com/chneukirchen/rack.git rack
Initialized empty Git repository in /opt/subtest/rack/.git/
remote: Counting objects: 3181, done.
remote: Compressing objects: 100% (1534/1534), done.
remote: Total 3181 (delta 1951), reused 2623 (delta 1603)
Receiving objects: 100% (3181/3181), 675.42 KiB | 422 KiB/s, done.
Resolving deltas: 100% (1951/1951), done.

   Now you have the Rack project under a subdirectory named `rack'
within your project. You can go into that subdirectory, make changes,
add your own writable remote repository to push your changes into,
fetch and merge from the original repository, and more. If you run `git
status' right after you add the submodule, you see two things:

$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#      new file:   .gitmodules
#      new file:   rack
#

   First you notice the `.gitmodules' file. This is a configuration
file that stores the mapping between the project's URL and the local
subdirectory you've pulled it into:

$ cat .gitmodules
[submodule "rack"]
      path = rack
      url = git://github.com/chneukirchen/rack.git

   If you have multiple submodules, you'll have multiple entries in
this file. It's important to note that this file is
version-controlled with your other files, like your `.gitignore' file.
It's pushed and pulled with the rest of your project. This is how
other people who clone this project know where to get the submodule
projects from.

   The other listing in the `git status' output is the rack entry. If
you run `git diff' on that, you see something interesting:

$ git diff --cached rack
diff --git a/rack b/rack
new file mode 160000
index 0000000..08d709f
--- /dev/null
+++ b/rack
@@ -0,0 +1 @@
+Subproject commit 08d709f78b8c5b0fbeb7821e37fa53e69afcf433

   Although `rack' is a subdirectory in your working directory, Git
sees it as a submodule and doesn't track its contents when you're
not in that directory. Instead, Git records it as a particular commit
from that repository. When you make changes and commit in that
subdirectory, the superproject notices that the HEAD there has changed
and records the exact commit you're currently working off of; that
way, when others clone this project, they can re-create the environment
exactly.

   This is an important point with submodules: you record them as the
exact commit they're at. You can't record a submodule at `master'
or some other symbolic reference.

   When you commit, you see something like this:

$ git commit -m 'first commit with submodule rack'
[master 0550271] first commit with submodule rack
 2 files changed, 4 insertions(+), 0 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 rack

   Notice the 160000 mode for the rack entry. That is a special mode in
Git that basically means you're recording a commit as a directory
entry rather than a subdirectory or a file.

   You can treat the `rack' directory as a separate project and then
update your superproject from time to time with a pointer to the latest
commit in that subproject. All the Git commands work independently in
the two directories:

$ git log -1
commit 0550271328a0038865aad6331e620cd7238601bb
Author: Scott Chacon <schacon@gmail.com>
Date:   Thu Apr 9 09:03:56 2009 -0700

    first commit with submodule rack
$ cd rack/
$ git log -1
commit 08d709f78b8c5b0fbeb7821e37fa53e69afcf433
Author: Christian Neukirchen <chneukirchen@gmail.com>
Date:   Wed Mar 25 14:49:04 2009 +0100

    Document version change


File: progit.info,  Node: Cloning a Project with Submodules,  Next: Superprojects,  Prev: Starting with Submodules,  Up: Submodules

6.6.2 Cloning a Project with Submodules
---------------------------------------

Here you'll clone a project with a submodule in it. When you receive
such a project, you get the directories that contain submodules, but
none of the files yet:

$ git clone git://github.com/schacon/myproject.git
Initialized empty Git repository in /opt/myproject/.git/
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 6 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (6/6), done.
$ cd myproject
$ ls -l
total 8
-rw-r--r--  1 schacon  admin   3 Apr  9 09:11 README
drwxr-xr-x  2 schacon  admin  68 Apr  9 09:11 rack
$ ls rack/
$

   The `rack' directory is there, but empty. You must run two commands:
`git submodule init' to initialize your local configuration file, and
`git submodule update' to fetch all the data from that project and
check out the appropriate commit listed in your superproject:

$ git submodule init
Submodule 'rack' (git://github.com/chneukirchen/rack.git) registered for path 'rack'
$ git submodule update
Initialized empty Git repository in /opt/myproject/rack/.git/
remote: Counting objects: 3181, done.
remote: Compressing objects: 100% (1534/1534), done.
remote: Total 3181 (delta 1951), reused 2623 (delta 1603)
Receiving objects: 100% (3181/3181), 675.42 KiB | 173 KiB/s, done.
Resolving deltas: 100% (1951/1951), done.
Submodule path 'rack': checked out '08d709f78b8c5b0fbeb7821e37fa53e69afcf433'

   Now your `rack' subdirectory is at the exact state it was in when
you committed earlier. If another developer makes changes to the rack
code and commits, and you pull that reference down and merge it in, you
get something a bit odd:

$ git merge origin/master
Updating 0550271..85a3eee
Fast forward
 rack |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
[master*]$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#      modified:   rack
#

   You merged in what is basically a change to the pointer for your
submodule; but it doesn't update the code in the submodule directory,
so it looks like you have a dirty state in your working directory:

$ git diff
diff --git a/rack b/rack
index 6c5e70b..08d709f 160000
--- a/rack
+++ b/rack
@@ -1 +1 @@
-Subproject commit 6c5e70b984a60b3cecd395edd5b48a7575bf58e0
+Subproject commit 08d709f78b8c5b0fbeb7821e37fa53e69afcf433

   This is the case because the pointer you have for the submodule
isn't what is actually in the submodule directory. To fix this, you
must run `git submodule update' again:

$ git submodule update
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 1), reused 2 (delta 0)
Unpacking objects: 100% (3/3), done.
From git@github.com:schacon/rack
   08d709f..6c5e70b  master     -> origin/master
Submodule path 'rack': checked out '6c5e70b984a60b3cecd395edd5b48a7575bf58e0'

   You have to do this every time you pull down a submodule change in
the main project. It's strange, but it works.

   One common problem happens when a developer makes a change locally
in a submodule but doesn't push it to a public server. Then, they
commit a pointer to that non-public state and push up the superproject.
When other developers try to run `git submodule update', the submodule
system can't find the commit that is referenced, because it exists
only on the first developer's system. If that happens, you see an
error like this:

$ git submodule update
fatal: reference isn't a tree: 6c5e70b984a60b3cecd395edd5b48a7575bf58e0
Unable to checkout '6c5e70b984a60b3cecd395edd5ba7575bf58e0' in submodule path 'rack'

   You have to see who last changed the submodule:

$ git log -1 rack
commit 85a3eee996800fcfa91e2119372dd4172bf76678
Author: Scott Chacon <schacon@gmail.com>
Date:   Thu Apr 9 09:19:14 2009 -0700

    added a submodule reference I will never make public. hahahahaha!

   Then, you e-mail that guy and yell at him.


File: progit.info,  Node: Superprojects,  Next: Issues with Submodules,  Prev: Cloning a Project with Submodules,  Up: Submodules

6.6.3 Superprojects
-------------------

Sometimes, developers want to get a combination of a large project's
subdirectories, depending on what team they're on. This is common if
you're coming from CVS or Subversion, where you've defined a module
or collection of subdirectories, and you want to keep this type of
workflow.

   A good way to do this in Git is to make each of the subfolders a
separate Git repository and then create superproject Git repositories
that contain multiple submodules. A benefit of this approach is that
you can more specifically define the relationships between the projects
with tags and branches in the superprojects.


File: progit.info,  Node: Issues with Submodules,  Prev: Superprojects,  Up: Submodules

6.6.4 Issues with Submodules
----------------------------

Using submodules isn't without hiccups, however. First, you must be
relatively careful when working in the submodule directory. When you
run `git submodule update', it checks out the specific version of the
project, but not within a branch. This is called having a detached head
--- it means the HEAD file points directly to a commit, not to a
symbolic reference. The issue is that you generally don't want to
work in a detached head environment, because it's easy to lose
changes. If you do an initial `submodule update', commit in that
submodule directory without creating a branch to work in, and then run
`git submodule update' again from the superproject without committing
in the meantime, Git will overwrite your changes without telling you.
Technically you won't lose the work, but you won't have a branch
pointing to it, so it will be somewhat difficult to retrieve.

   To avoid this issue, create a branch when you work in a submodule
directory with `git checkout -b work' or something equivalent. When you
do the submodule update a second time, it will still revert your work,
but at least you have a pointer to get back to.

   Switching branches with submodules in them can also be tricky. If
you create a new branch, add a submodule there, and then switch back to
a branch without that submodule, you still have the submodule directory
as an untracked directory:

$ git checkout -b rack
Switched to a new branch "rack"
$ git submodule add git@github.com:schacon/rack.git rack
Initialized empty Git repository in /opt/myproj/rack/.git/
...
Receiving objects: 100% (3184/3184), 677.42 KiB | 34 KiB/s, done.
Resolving deltas: 100% (1952/1952), done.
$ git commit -am 'added rack submodule'
[rack cc49a69] added rack submodule
 2 files changed, 4 insertions(+), 0 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 rack
$ git checkout master
Switched to branch "master"
$ git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#      rack/

   You have to either move it out of the way or remove it, in which
case you have to clone it again when you switch back---and you may lose
local changes or branches that you didn't push up.

   The last main caveat that many people run into involves switching
from subdirectories to submodules. If you've been tracking files in
your project and you want to move them out into a submodule, you must
be careful or Git will get angry at you. Assume that you have the rack
files in a subdirectory of your project, and you want to switch it to a
submodule. If you delete the subdirectory and then run `submodule add',
Git yells at you:

$ rm -Rf rack/
$ git submodule add git@github.com:schacon/rack.git rack
'rack' already exists in the index

   You have to unstage the `rack' directory first. Then you can add the
submodule:

$ git rm -r rack
$ git submodule add git@github.com:schacon/rack.git rack
Initialized empty Git repository in /opt/testsub/rack/.git/
remote: Counting objects: 3184, done.
remote: Compressing objects: 100% (1465/1465), done.
remote: Total 3184 (delta 1952), reused 2770 (delta 1675)
Receiving objects: 100% (3184/3184), 677.42 KiB | 88 KiB/s, done.
Resolving deltas: 100% (1952/1952), done.

   Now suppose you did that in a branch. If you try to switch back to a
branch where those files are still in the actual tree rather than a
submodule --- you get this error:

$ git checkout master
error: Untracked working tree file 'rack/AUTHORS' would be overwritten by merge.

   You have to move the `rack' submodule directory out of the way
before you can switch to a branch that doesn't have it:

$ mv rack /tmp/
$ git checkout master
Switched to branch "master"
$ ls
README  rack

   Then, when you switch back, you get an empty `rack' directory. You
can either run `git submodule update' to reclone, or you can move your
`/tmp/rack' directory back into the empty directory.


File: progit.info,  Node: Subtree Merging,  Next: Summary,  Prev: Submodules,  Up: Git Tools

6.7 Subtree Merging
===================

Now that you've seen the difficulties of the submodule system,
let's look at an alternate way to solve the same problem. When Git
merges, it looks at what it has to merge together and then chooses an
appropriate merging strategy to use. If you're merging two branches,
Git uses a _recursive_ strategy. If you're merging more than two
branches, Git picks the _octopus_ strategy. These strategies are
automatically chosen for you because the recursive strategy can handle
complex three-way merge situations --- for example, more than one
common ancestor --- but it can only handle merging two branches. The
octopus merge can handle multiple branches but is more cautious to
avoid difficult conflicts, so it's chosen as the default strategy if
you're trying to merge more than two branches.

   However, there are other strategies you can choose as well. One of
them is the _subtree_ merge, and you can use it to deal with the
subproject issue. Here you'll see how to do the same rack embedding
as in the last section, but using subtree merges instead.

   The idea of the subtree merge is that you have two projects, and one
of the projects maps to a subdirectory of the other one and vice versa.
When you specify a subtree merge, Git is smart enough to figure out
that one is a subtree of the other and merge appropriately --- it's
pretty amazing.

   You first add the Rack application to your project. You add the Rack
project as a remote reference in your own project and then check it out
into its own branch:

$ git remote add rack_remote git@github.com:schacon/rack.git
$ git fetch rack_remote
warning: no common commits
remote: Counting objects: 3184, done.
remote: Compressing objects: 100% (1465/1465), done.
remote: Total 3184 (delta 1952), reused 2770 (delta 1675)
Receiving objects: 100% (3184/3184), 677.42 KiB | 4 KiB/s, done.
Resolving deltas: 100% (1952/1952), done.
From git@github.com:schacon/rack
 * [new branch]      build      -> rack_remote/build
 * [new branch]      master     -> rack_remote/master
 * [new branch]      rack-0.4   -> rack_remote/rack-0.4
 * [new branch]      rack-0.9   -> rack_remote/rack-0.9
$ git checkout -b rack_branch rack_remote/master
Branch rack_branch set up to track remote branch refs/remotes/rack_remote/master.
Switched to a new branch "rack_branch"

   Now you have the root of the Rack project in your `rack_branch'
branch and your own project in the `master' branch. If you check out
one and then the other, you can see that they have different project
roots:

$ ls
AUTHORS        KNOWN-ISSUES   Rakefile      contrib        lib
COPYING        README         bin           example        test
$ git checkout master
Switched to branch "master"
$ ls
README

   You want to pull the Rack project into your `master' project as a
subdirectory. You can do that in Git with `git read-tree'. You'll
learn more about `read-tree' and its friends in Chapter 9, but for now
know that it reads the root tree of one branch into your current
staging area and working directory. You just switched back to your
`master' branch, and you pull the `rack' branch into the `rack'
subdirectory of your `master' branch of your main project:

$ git read-tree --prefix=rack/ -u rack_branch

   When you commit, it looks like you have all the Rack files under
that subdirectory --- as though you copied them in from a tarball. What
gets interesting is that you can fairly easily merge changes from one
of the branches to the other. So, if the Rack project updates, you can
pull in upstream changes by switching to that branch and pulling:

$ git checkout rack_branch
$ git pull

   Then, you can merge those changes back into your master branch. You
can use `git merge -s subtree' and it will work fine; but Git will also
merge the histories together, which you probably don't want. To pull
in the changes and prepopulate the commit message, use the `--squash'
and `--no-commit' options as well as the `-s subtree' strategy option:

$ git checkout master
$ git merge --squash -s subtree --no-commit rack_branch
Squash commit -- not updating HEAD
Automatic merge went well; stopped before committing as requested

   All the changes from your Rack project are merged in and ready to be
committed locally. You can also do the opposite --- make changes in the
`rack' subdirectory of your master branch and then merge them into your
`rack_branch' branch later to submit them to the maintainers or push
them upstream.

   To get a diff between what you have in your `rack' subdirectory and
the code in your `rack_branch' branch --- to see if you need to merge
them --- you can't use the normal `diff' command. Instead, you must
run `git diff-tree' with the branch you want to compare to:

$ git diff-tree -p rack_branch

   Or, to compare what is in your `rack' subdirectory with what the
`master' branch on the server was the last time you fetched, you can run

$ git diff-tree -p rack_remote/master


File: progit.info,  Node: Summary,  Prev: Subtree Merging,  Up: Git Tools

6.8 Summary
===========

You've seen a number of advanced tools that allow you to manipulate
your commits and staging area more precisely. When you notice issues,
you should be able to easily figure out what commit introduced them,
when, and by whom. If you want to use subprojects in your project,
you've learned a few ways to accommodate those needs. At this point,
you should be able to do most of the things in Git that you'll need
on the command line day to day and feel comfortable doing so.


File: progit.info,  Node: Customizing Git,  Next: Git and Other Systems,  Prev: Git Tools,  Up: Top

7 Customizing Git
*****************

So far, I've covered the basics of how Git works and how to use it,
and I've introduced a number of tools that Git provides to help you
use it easily and efficiently. In this chapter, I'll go through some
operations that you can use to make Git operate in a more customized
fashion by introducing several important configuration settings and the
hooks system. With these tools, it's easy to get Git to work exactly
the way you, your company, or your group needs it to.

* Menu:

* Git Configuration::
* Git Attributes::
* Git Hooks::
* An Example Git-Enforced Policy::
* Summary::


File: progit.info,  Node: Git Configuration,  Next: Git Attributes,  Up: Customizing Git

7.1 Git Configuration
=====================

As you briefly saw in the Chapter 1, you can specify Git configuration
settings with the `git config' command. One of the first things you did
was set up your name and e-mail address:

$ git config --global user.name "John Doe"
$ git config --global user.email johndoe@example.com

   Now you'll learn a few of the more interesting options that you
can set in this manner to customize your Git usage.

   You saw some simple Git configuration details in the first chapter,
but I'll go over them again quickly here. Git uses a series of
configuration files to determine non-default behavior that you may
want. The first place Git looks for these values is in an
`/etc/gitconfig' file, which contains values for every user on the
system and all of their repositories. If you pass the option `--system'
to `git config', it reads and writes from this file specifically.

   The next place Git looks is the `~/.gitconfig' file, which is
specific to each user. You can make Git read and write to this file by
passing the `--global' option.

   Finally, Git looks for configuration values in the config file in
the Git directory (`.git/config') of whatever repository you're
currently using. These values are specific to that single repository.
Each level overwrites values in the previous level, so values in
`.git/config' trump those in `/etc/gitconfig', for instance. You can
also set these values by manually editing the file and inserting the
correct syntax, but it's generally easier to run the `git config'
command.

* Menu:

* Basic Client Configuration::
* Colors in Git::
* External Merge and Diff Tools::
* Formatting and Whitespace::
* Server Configuration::


File: progit.info,  Node: Basic Client Configuration,  Next: Colors in Git,  Up: Git Configuration

7.1.1 Basic Client Configuration
--------------------------------

The configuration options recognized by Git fall into two categories:
client side and server side. The majority of the options are client
side---configuring your personal working preferences. Although tons of
options are available, I'll only cover the few that either are
commonly used or can significantly affect your workflow. Many options
are useful only in edge cases that I won't go over here. If you want
to see a list of all the options your version of Git recognizes, you
can run

$ git config --help

   The manual page for `git config' lists all the available options in
quite a bit of detail.

* Menu:

* coreeditor::
* committemplate::
* corepager::
* usersigningkey::
* coreexcludesfile::
* helpautocorrect::


File: progit.info,  Node: coreeditor,  Next: committemplate,  Up: Basic Client Configuration

7.1.1.1 core.editor
...................

By default, Git uses whatever you've set as your default text editor
or else falls back to the Vi editor to create and edit your commit and
tag messages. To change that default to something else, you can use the
`core.editor' setting:

$ git config --global core.editor emacs

   Now, no matter what is set as your default shell editor variable,
Git will fire up Emacs to edit messages.


File: progit.info,  Node: committemplate,  Next: corepager,  Prev: coreeditor,  Up: Basic Client Configuration

7.1.1.2 commit.template
.......................

If you set this to the path of a file on your system, Git will use that
file as the default message when you commit. For instance, suppose you
create a template file at `$HOME/.gitmessage.txt' that looks like this:

subject line

what happened

[ticket: X]

   To tell Git to use it as the default message that appears in your
editor when you run `git commit', set the `commit.template'
configuration value:

$ git config --global commit.template $HOME/.gitmessage.txt
$ git commit

   Then, your editor will open to something like this for your
placeholder commit message when you commit:

subject line

what happened

[ticket: X]
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
# modified:   lib/test.rb
#
~
~
".git/COMMIT_EDITMSG" 14L, 297C

   If you have a commit-message policy in place, then putting a
template for that policy on your system and configuring Git to use it
by default can help increase the chance of that policy being followed
regularly.


File: progit.info,  Node: corepager,  Next: usersigningkey,  Prev: committemplate,  Up: Basic Client Configuration

7.1.1.3 core.pager
..................

The core.pager setting determines what pager is used when Git pages
output such as `log' and `diff'. You can set it to `more' or to your
favorite pager (by default, it's `less'), or you can turn it off by
setting it to a blank string:

$ git config --global core.pager ''

   If you run that, Git will page the entire output of all commands, no
matter how long they are.


File: progit.info,  Node: usersigningkey,  Next: coreexcludesfile,  Prev: corepager,  Up: Basic Client Configuration

7.1.1.4 user.signingkey
.......................

If you're making signed annotated tags (as discussed in Chapter 2),
setting your GPG signing key as a configuration setting makes things
easier. Set your key ID like so:

$ git config --global user.signingkey <gpg-key-id>

   Now, you can sign tags without having to specify your key every time
with the `git tag' command:

$ git tag -s <tag-name>


File: progit.info,  Node: coreexcludesfile,  Next: helpautocorrect,  Prev: usersigningkey,  Up: Basic Client Configuration

7.1.1.5 core.excludesfile
.........................

You can put patterns in your project's `.gitignore' file to have Git
not see them as untracked files or try to stage them when you run `git
add' on them, as discussed in Chapter 2. However, if you want another
file outside of your project to hold those values or have extra values,
you can tell Git where that file is with the `core.excludesfile'
setting. Simply set it to the path of a file that has content similar
to what a `.gitignore' file would have.


File: progit.info,  Node: helpautocorrect,  Prev: coreexcludesfile,  Up: Basic Client Configuration

7.1.1.6 help.autocorrect
........................

This option is available only in Git 1.6.1 and later. If you mistype a
command in Git 1.6, it shows you something like this:

$ git com
git: 'com' is not a git-command. See 'git --help'.

Did you mean this?
     commit

   If you set `help.autocorrect' to 1, Git will automatically run the
command if it has only one match under this scenario.


File: progit.info,  Node: Colors in Git,  Next: External Merge and Diff Tools,  Prev: Basic Client Configuration,  Up: Git Configuration

7.1.2 Colors in Git
-------------------

Git can color its output to your terminal, which can help you visually
parse the output quickly and easily. A number of options can help you
set the coloring to your preference.

* Menu:

* colorui::
* color*::


File: progit.info,  Node: colorui,  Next: color*,  Up: Colors in Git

7.1.2.1 color.ui
................

Git automatically colors most of its output if you ask it to. You can
get very specific about what you want colored and how; but to turn on
all the default terminal coloring, set `color.ui' to true:

$ git config --global color.ui true

   When that value is set, Git colors its output if the output goes to
a terminal. Other possible settings are false, which never colors the
output, and always, which sets colors all the time, even if you're
redirecting Git commands to a file or piping them to another command.
This setting was added in Git version 1.5.5; if you have an older
version, you'll have to specify all the color settings individually.

   You'll rarely want `color.ui = always'. In most scenarios, if you
want color codes in your redirected output, you can instead pass a
`--color' flag to the Git command to force it to use color codes. The
`color.ui = true' setting is almost always what you'll want to use.


File: progit.info,  Node: color*,  Prev: colorui,  Up: Colors in Git

7.1.2.2 `color.*'
.................

If you want to be more specific about which commands are colored and
how, or you have an older version, Git provides verb-specific coloring
settings. Each of these can be set to `true', `false', or `always':

color.branch
color.diff
color.interactive
color.status

   In addition, each of these has subsettings you can use to set
specific colors for parts of the output, if you want to override each
color. For example, to set the meta information in your diff output to
blue foreground, black background, and bold text, you can run

$ git config --global color.diff.meta “blue black bold"

   You can set the color to any of the following values: normal, black,
red, green, yellow, blue, magenta, cyan, or white. If you want an
attribute like bold in the previous example, you can choose from bold,
dim, ul, blink, and reverse.

   See the `git config' manpage for all the subsettings you can
configure, if you want to do that.


File: progit.info,  Node: External Merge and Diff Tools,  Next: Formatting and Whitespace,  Prev: Colors in Git,  Up: Git Configuration

7.1.3 External Merge and Diff Tools
-----------------------------------

Although Git has an internal implementation of diff, which is what
you've been using, you can set up an external tool instead. You can
also set up a graphical merge conflict-resolution tool instead of
having to resolve conflicts manually. I'll demonstrate setting up the
Perforce Visual Merge Tool (P4Merge) to do your diffs and merge
resolutions, because it's a nice graphical tool and it's free.

   If you want to try this out, P4Merge works on all major platforms,
so you should be able to do so. I'll use path names in the examples
that work on Mac and Linux systems; for Windows, you'll have to
change `/usr/local/bin' to an executable path in your environment.

   You can download P4Merge here:

http://www.perforce.com/perforce/downloads/component.html

   To begin, you'll set up external wrapper scripts to run your
commands. I'll use the Mac path for the executable; in other systems,
it will be where your `p4merge' binary is installed. Set up a merge
wrapper script named `extMerge' that calls your binary with all the
arguments provided:

$ cat /usr/local/bin/extMerge
#!/bin/sh
/Applications/p4merge.app/Contents/MacOS/p4merge $*

   The diff wrapper checks to make sure seven arguments are provided
and passes two of them to your merge script. By default, Git passes the
following arguments to the diff program:

path old-file old-hex old-mode new-file new-hex new-mode

   Because you only want the `old-file' and `new-file' arguments, you
use the wrapper script to pass the ones you need.

$ cat /usr/local/bin/extDiff
#!/bin/sh
[ $# -eq 7 ] && /usr/local/bin/extMerge "$2" "$5"

   You also need to make sure these tools are executable:

$ sudo chmod +x /usr/local/bin/extMerge
$ sudo chmod +x /usr/local/bin/extDiff

   Now you can set up your config file to use your custom merge
resolution and diff tools. This takes a number of custom settings:
`merge.tool' to tell Git what strategy to use, `mergetool.*.cmd' to
specify how to run the command, `mergetool.trustExitCode' to tell Git
if the exit code of that program indicates a successful merge
resolution or not, and `diff.external' to tell Git what command to run
for diffs. So, you can either run four config commands

$ git config --global merge.tool extMerge
$ git config --global mergetool.extMerge.cmd \
    'extMerge "$BASE" "$LOCAL" "$REMOTE" "$MERGED"'
$ git config --global mergetool.trustExitCode false
$ git config --global diff.external extDiff

   or you can edit your `~/.gitconfig' file to add these lines:

[merge]
  tool = extMerge
[mergetool "extMerge"]
  cmd = extMerge "$BASE" "$LOCAL" "$REMOTE" "$MERGED"
  trustExitCode = false
[diff]
  external = extDiff

   After all this is set, if you run diff commands such as this:

$ git diff 32d1776b1^ 32d1776b1

   Instead of getting the diff output on the command line, Git fires up
P4Merge, which looks something like Figure 7-1.

[image src="../figures/18333fig0701-tn.png" alt="Figure 7-1. P4Merge."]

Figure 7-1. P4Merge.

   If you try to merge two branches and subsequently have merge
conflicts, you can run the command `git mergetool'; it starts P4Merge
to let you resolve the conflicts through that GUI tool.

   The nice thing about this wrapper setup is that you can change your
diff and merge tools easily. For example, to change your `extDiff' and
`extMerge' tools to run the KDiff3 tool instead, all you have to do is
edit your `extMerge' file:

$ cat /usr/local/bin/extMerge
#!/bin/sh
/Applications/kdiff3.app/Contents/MacOS/kdiff3 $*

   Now, Git will use the KDiff3 tool for diff viewing and merge
conflict resolution.

   Git comes preset to use a number of other merge-resolution tools
without your having to set up the cmd configuration. You can set your
merge tool to kdiff3, opendiff, tkdiff, meld, xxdiff, emerge, vimdiff,
or gvimdiff. If you're not interested in using KDiff3 for diff but
rather want to use it just for merge resolution, and the kdiff3 command
is in your path, then you can run

$ git config --global merge.tool kdiff3

   If you run this instead of setting up the `extMerge' and `extDiff'
files, Git will use KDiff3 for merge resolution and the normal Git diff
tool for diffs.


File: progit.info,  Node: Formatting and Whitespace,  Next: Server Configuration,  Prev: External Merge and Diff Tools,  Up: Git Configuration

7.1.4 Formatting and Whitespace
-------------------------------

Formatting and whitespace issues are some of the more frustrating and
subtle problems that many developers encounter when collaborating,
especially cross-platform. It's very easy for patches or other
collaborated work to introduce subtle whitespace changes because
editors silently introduce them or Windows programmers add carriage
returns at the end of lines they touch in cross-platform projects. Git
has a few configuration options to help with these issues.

* Menu:

* coreautocrlf::
* corewhitespace::


File: progit.info,  Node: coreautocrlf,  Next: corewhitespace,  Up: Formatting and Whitespace

7.1.4.1 core.autocrlf
.....................

If you're programming on Windows or using another system but working
with people who are programming on Windows, you'll probably run into
line-ending issues at some point. This is because Windows uses both a
carriage-return character and a linefeed character for newlines in its
files, whereas Mac and Linux systems use only the linefeed character.
This is a subtle but incredibly annoying fact of cross-platform work.

   Git can handle this by auto-converting CRLF line endings into LF
when you commit, and vice versa when it checks out code onto your
filesystem. You can turn on this functionality with the `core.autocrlf'
setting. If you're on a Windows machine, set it to `true' --- this
converts LF endings into CRLF when you check out code:

$ git config --global core.autocrlf true

   If you're on a Linux or Mac system that uses LF line endings, then
you don't want Git to automatically convert them when you check out
files; however, if a file with CRLF endings accidentally gets
introduced, then you may want Git to fix it. You can tell Git to
convert CRLF to LF on commit but not the other way around by setting
`core.autocrlf' to input:

$ git config --global core.autocrlf input

   This setup should leave you with CRLF endings in Windows checkouts
but LF endings on Mac and Linux systems and in the repository.

   If you're a Windows programmer doing a Windows-only project, then
you can turn off this functionality, recording the carriage returns in
the repository by setting the config value to `false':

$ git config --global core.autocrlf false


File: progit.info,  Node: corewhitespace,  Prev: coreautocrlf,  Up: Formatting and Whitespace

7.1.4.2 core.whitespace
.......................

Git comes preset to detect and fix some whitespace issues. It can look
for four primary whitespace issues --- two are enabled by default and
can be turned off, and two aren't enabled by default but can be
activated.

   The two that are turned on by default are `trailing-space', which
looks for spaces at the end of a line, and `space-before-tab', which
looks for spaces before tabs at the beginning of a line.

   The two that are disabled by default but can be turned on are
`indent-with-non-tab', which looks for lines that begin with eight or
more spaces instead of tabs, and `cr-at-eol', which tells Git that
carriage returns at the end of lines are OK.

   You can tell Git which of these you want enabled by setting
`core.whitespace' to the values you want on or off, separated by
commas. You can disable settings by either leaving them out of the
setting string or prepending a `-' in front of the value. For example,
if you want all but `cr-at-eol' to be set, you can do this:

$ git config --global core.whitespace \
    trailing-space,space-before-tab,indent-with-non-tab

   Git will detect these issues when you run a `git diff' command and
try to color them so you can possibly fix them before you commit. It
will also use these values to help you when you apply patches with `git
apply'. When you're applying patches, you can ask Git to warn you if
it's applying patches with the specified whitespace issues:

$ git apply --whitespace=warn <patch>

   Or you can have Git try to automatically fix the issue before
applying the patch:

$ git apply --whitespace=fix <patch>

   These options apply to the git rebase option as well. If you've
committed whitespace issues but haven't yet pushed upstream, you can
run a `rebase' with the `--whitespace=fix' option to have Git
automatically fix whitespace issues as it's rewriting the patches.


File: progit.info,  Node: Server Configuration,  Prev: Formatting and Whitespace,  Up: Git Configuration

7.1.5 Server Configuration
--------------------------

Not nearly as many configuration options are available for the server
side of Git, but there are a few interesting ones you may want to take
note of.

* Menu:

* receivefsckObjects::
* receivedenyNonFastForwards::
* receivedenyDeletes::


File: progit.info,  Node: receivefsckObjects,  Next: receivedenyNonFastForwards,  Up: Server Configuration

7.1.5.1 receive.fsckObjects
...........................

By default, Git doesn't check for consistency all the objects it
receives during a push. Although Git can check to make sure each object
still matches its SHA-1 checksum and points to valid objects, it
doesn't do that by default on every push. This is a relatively
expensive operation and may add a lot of time to each push, depending
on the size of the repository or the push. If you want Git to check
object consistency on every push, you can force it to do so by setting
`receive.fsckObjects' to true:

$ git config --system receive.fsckObjects true

   Now, Git will check the integrity of your repository before each
push is accepted to make sure faulty clients aren't introducing
corrupt data.


File: progit.info,  Node: receivedenyNonFastForwards,  Next: receivedenyDeletes,  Prev: receivefsckObjects,  Up: Server Configuration

7.1.5.2 receive.denyNonFastForwards
...................................

If you rebase commits that you've already pushed and then try to push
again, or otherwise try to push a commit to a remote branch that
doesn't contain the commit that the remote branch currently points
to, you'll be denied. This is generally good policy; but in the case
of the rebase, you may determine that you know what you're doing and
can force-update the remote branch with a `-f' flag to your push
command.

   To disable the ability to force-update remote branches to
non-fast-forward references, set `receive.denyNonFastForwards':

$ git config --system receive.denyNonFastForwards true

   The other way you can do this is via server-side receive hooks,
which I'll cover in a bit. That approach lets you do more complex
things like deny non-fast-forwards to a certain subset of users.


File: progit.info,  Node: receivedenyDeletes,  Prev: receivedenyNonFastForwards,  Up: Server Configuration

7.1.5.3 receive.denyDeletes
...........................

One of the workarounds to the `denyNonFastForwards' policy is for the
user to delete the branch and then push it back up with the new
reference. In newer versions of Git (beginning with version 1.6.1), you
can set `receive.denyDeletes' to true:

$ git config --system receive.denyDeletes true

   This denies branch and tag deletion over a push across the board ---
no user can do it. To remove remote branches, you must remove the ref
files from the server manually. There are also more interesting ways to
do this on a per-user basis via ACLs, as you'll learn at the end of
this chapter.


File: progit.info,  Node: Git Attributes,  Next: Git Hooks,  Prev: Git Configuration,  Up: Customizing Git

7.2 Git Attributes
==================

Some of these settings can also be specified for a path, so that Git
applies those settings only for a subdirectory or subset of files.
These path-specific settings are called Git attributes and are set
either in a `.gitattributes' file in one of your directories (normally
the root of your project) or in the `.git/info/attributes' file if you
don't want the attributes file committed with your project.

   Using attributes, you can do things like specify separate merge
strategies for individual files or directories in your project, tell
Git how to diff non-text files, or have Git filter content before you
check it into or out of Git. In this section, you'll learn about some
of the attributes you can set on your paths in your Git project and see
a few examples of using this feature in practice.

* Menu:

* Binary Files::
* Keyword Expansion::
* Exporting Your Repository::
* Merge Strategies::


File: progit.info,  Node: Binary Files,  Next: Keyword Expansion,  Up: Git Attributes

7.2.1 Binary Files
------------------

One cool trick for which you can use Git attributes is telling Git
which files are binary (in cases it otherwise may not be able to figure
out) and giving Git special instructions about how to handle those
files. For instance, some text files may be machine generated and not
diffable, whereas some binary files can be diffed --- you'll see how
to tell Git which is which.

* Menu:

* Identifying Binary Files::
* Diffing Binary Files::


File: progit.info,  Node: Identifying Binary Files,  Next: Diffing Binary Files,  Up: Binary Files

7.2.1.1 Identifying Binary Files
................................

Some files look like text files but for all intents and purposes are to
be treated as binary data. For instance, Xcode projects on the Mac
contain a file that ends in `.pbxproj', which is basically a JSON
(plain text javascript data format) dataset written out to disk by the
IDE that records your build settings and so on. Although it's
technically a text file, because it's all ASCII, you don't want to
treat it as such because it's really a lightweight database --- you
can't merge the contents if two people changed it, and diffs
generally aren't helpful. The file is meant to be consumed by a
machine. In essence, you want to treat it like a binary file.

   To tell Git to treat all `pbxproj' files as binary data, add the
following line to your `.gitattributes' file:

*.pbxproj -crlf -diff

   Now, Git won't try to convert or fix CRLF issues; nor will it try
to compute or print a diff for changes in this file when you run git
show or git diff on your project. In the 1.6 series of Git, you can
also use a macro that is provided that means `-crlf -diff':

*.pbxproj binary


File: progit.info,  Node: Diffing Binary Files,  Prev: Identifying Binary Files,  Up: Binary Files

7.2.1.2 Diffing Binary Files
............................

In the 1.6 series of Git, you can use the Git attributes functionality
to effectively diff binary files. You do this by telling Git how to
convert your binary data to a text format that can be compared via the
normal diff.

   Because this is a pretty cool and not widely known feature, I'll
go over a few examples. First, you'll use this technique to solve one
of the most annoying problems known to humanity: version-controlling
Word documents. Everyone knows that Word is the most horrific editor
around; but, oddly, everyone uses it. If you want to version-control
Word documents, you can stick them in a Git repository and commit every
once in a while; but what good does that do? If you run `git diff'
normally, you only see something like this:

$ git diff
diff --git a/chapter1.doc b/chapter1.doc
index 88839c4..4afcb7c 100644
Binary files a/chapter1.doc and b/chapter1.doc differ

   You can't directly compare two versions unless you check them out
and scan them manually, right? It turns out you can do this fairly well
using Git attributes. Put the following line in your `.gitattributes'
file:

*.doc diff=word

   This tells Git that any file that matches this pattern (.doc) should
use the "word" filter when you try to view a diff that contains
changes. What is the "word" filter? You have to set it up. Here
you'll configure Git to use the `strings' program to convert Word
documents into readable text files, which it will then diff properly:

$ git config diff.word.textconv strings

   Now Git knows that if it tries to do a diff between two snapshots,
and any of the files end in `.doc', it should run those files through
the "word" filter, which is defined as the `strings' program. This
effectively makes nice text-based versions of your Word files before
attempting to diff them.

   Here's an example. I put Chapter 1 of this book into Git, added
some text to a paragraph, and saved the document. Then, I ran `git
diff' to see what changed:

$ git diff
diff --git a/chapter1.doc b/chapter1.doc
index c1c8a0a..b93c9e4 100644
--- a/chapter1.doc
+++ b/chapter1.doc
@@ -8,7 +8,8 @@ re going to cover Version Control Systems (VCS) and Git basics
 re going to cover how to get it and set it up for the first time if you don
 t already have it on your system.
 In Chapter Two we will go over basic Git usage - how to use Git for the 80%
-s going on, modify stuff and contribute changes. If the book spontaneously
+s going on, modify stuff and contribute changes. If the book spontaneously
+Let's see if this works.

   Git successfully and succinctly tells me that I added the string
"Let's see if this works", which is correct. It's not perfect ---
it adds a bunch of random stuff at the end --- but it certainly works.
If you can find or write a Word-to-plain-text converter that works well
enough, that solution will likely be incredibly effective. However,
`strings' is available on most Mac and Linux systems, so it may be a
good first try to do this with many binary formats.

   Another interesting problem you can solve this way involves diffing
image files. One way to do this is to run JPEG files through a filter
that extracts their EXIF information --- metadata that is recorded with
most image formats. If you download and install the `exiftool' program,
you can use it to convert your images into text about the metadata, so
at least the diff will show you a textual representation of any changes
that happened:

$ echo '*.png diff=exif' >> .gitattributes
$ git config diff.exif.textconv exiftool

   If you replace an image in your project and run `git diff', you see
something like this:

diff --git a/image.png b/image.png
index 88839c4..4afcb7c 100644
--- a/image.png
+++ b/image.png
@@ -1,12 +1,12 @@
 ExifTool Version Number         : 7.74
-File Size                       : 70 kB
-File Modification Date/Time     : 2009:04:21 07:02:45-07:00
+File Size                       : 94 kB
+File Modification Date/Time     : 2009:04:21 07:02:43-07:00
 File Type                       : PNG
 MIME Type                       : image/png
-Image Width                     : 1058
-Image Height                    : 889
+Image Width                     : 1056
+Image Height                    : 827
 Bit Depth                       : 8
 Color Type                      : RGB with Alpha

   You can easily see that the file size and image dimensions have both
changed.


File: progit.info,  Node: Keyword Expansion,  Next: Exporting Your Repository,  Prev: Binary Files,  Up: Git Attributes

7.2.2 Keyword Expansion
-----------------------

SVN- or CVS-style keyword expansion is often requested by developers
used to those systems. The main problem with this in Git is that you
can't modify a file with information about the commit after you've
committed, because Git checksums the file first. However, you can
inject text into a file when it's checked out and remove it again
before it's added to a commit. Git attributes offers you two ways to
do this.

   First, you can inject the SHA-1 checksum of a blob into an `$Id$'
field in the file automatically. If you set this attribute on a file or
set of files, then the next time you check out that branch, Git will
replace that field with the SHA-1 of the blob. It's important to
notice that it isn't the SHA of the commit, but of the blob itself:

$ echo '*.txt ident' >> .gitattributes
$ echo '$Id$' > test.txt

   The next time you check out this file, Git injects the SHA of the
blob:

$ rm text.txt
$ git checkout -- text.txt
$ cat test.txt
$Id: 42812b7653c7b88933f8a9d6cad0ca16714b9bb3 $

   However, that result is of limited use. If you've used keyword
substitution in CVS or Subversion, you can include a datestamp --- the
SHA isn't all that helpful, because it's fairly random and you
can't tell if one SHA is older or newer than another.

   It turns out that you can write your own filters for doing
substitutions in files on commit/checkout. These are the "clean" and
"smudge" filters. In the `.gitattributes' file, you can set a filter
for particular paths and then set up scripts that will process files
just before they're checked out ("smudge", see Figure 7-2) and just
before they're committed ("clean", see Figure 7-3). These filters can
be set to do all sorts of fun things.

[image src="../figures/18333fig0702-tn.png" alt="Figure 7-2. The “smudge" filter is run on checkout."]

Figure 7-2. The “smudge" filter is run on checkout.

[image src="../figures/18333fig0703-tn.png" alt="Figure 7-3. The “clean" filter is run when files are staged."]

Figure 7-3. The “clean" filter is run when files are staged.

   The original commit message for this functionality gives a simple
example of running all your C source code through the `indent' program
before committing. You can set it up by setting the filter attribute in
your `.gitattributes' file to filter `*.c' files with the "indent"
filter:

*.c     filter=indent

   Then, tell Git what the "indent"" filter does on smudge and clean:

$ git config --global filter.indent.clean indent
$ git config --global filter.indent.smudge cat

   In this case, when you commit files that match `*.c', Git will run
them through the indent program before it commits them and then run
them through the `cat' program before it checks them back out onto
disk. The `cat' program is basically a no-op: it spits out the same
data that it gets in. This combination effectively filters all C source
code files through `indent' before committing.

   Another interesting example gets `$Date$' keyword expansion, RCS
style. To do this properly, you need a small script that takes a
filename, figures out the last commit date for this project, and
inserts the date into the file. Here is a small Ruby script that does
that:

#! /usr/bin/env ruby
data = STDIN.read
last_date = `git log --pretty=format:"%ad" -1`
puts data.gsub('$Date$', '$Date: ' + last_date.to_s + '$')

   All the script does is get the latest commit date from the `git log'
command, stick that into any `$Date$' strings it sees in stdin, and
print the results --- it should be simple to do in whatever language
you're most comfortable in. You can name this file `expand_date' and
put it in your path. Now, you need to set up a filter in Git (call it
`dater') and tell it to use your `expand_date' filter to smudge the
files on checkout. You'll use a Perl expression to clean that up on
commit:

$ git config filter.dater.smudge expand_date
$ git config filter.dater.clean 'perl -pe "s/\\\$Date[^\\\$]*\\\$/\\\$Date\\\$/"'

   This Perl snippet strips out anything it sees in a `$Date$' string,
to get back to where you started. Now that your filter is ready, you
can test it by setting up a file with your `$Date$' keyword and then
setting up a Git attribute for that file that engages the new filter:

$ echo '# $Date$' > date_test.txt
$ echo 'date*.txt filter=dater' >> .gitattributes

   If you commit those changes and check out the file again, you see
the keyword properly substituted:

$ git add date_test.txt .gitattributes
$ git commit -m "Testing date expansion in Git"
$ rm date_test.txt
$ git checkout date_test.txt
$ cat date_test.txt
# $Date: Tue Apr 21 07:26:52 2009 -0700$

   You can see how powerful this technique can be for customized
applications. You have to be careful, though, because the
`.gitattributes' file is committed and passed around with the project
but the driver (in this case, `dater') isn't; so, it won't work
everywhere. When you design these filters, they should be able to fail
gracefully and have the project still work properly.


File: progit.info,  Node: Exporting Your Repository,  Next: Merge Strategies,  Prev: Keyword Expansion,  Up: Git Attributes

7.2.3 Exporting Your Repository
-------------------------------

Git attribute data also allows you to do some interesting things when
exporting an archive of your project.

* Menu:

* export-ignore::
* export-subst::


File: progit.info,  Node: export-ignore,  Next: export-subst,  Up: Exporting Your Repository

7.2.3.1 export-ignore
.....................

You can tell Git not to export certain files or directories when
generating an archive. If there is a subdirectory or file that you
don't want to include in your archive file but that you do want
checked into your project, you can determine those files via the
`export-ignore' attribute.

   For example, say you have some test files in a `test/' subdirectory,
and it doesn't make sense to include them in the tarball export of
your project. You can add the following line to your Git attributes
file:

test/ export-ignore

   Now, when you run git archive to create a tarball of your project,
that directory won't be included in the archive.


File: progit.info,  Node: export-subst,  Prev: export-ignore,  Up: Exporting Your Repository

7.2.3.2 export-subst
....................

Another thing you can do for your archives is some simple keyword
substitution. Git lets you put the string `$Format:$' in any file with
any of the `--pretty=format' formatting shortcodes, many of which you
saw in Chapter 2. For instance, if you want to include a file named
`LAST_COMMIT' in your project, and the last commit date was
automatically injected into it when `git archive' ran, you can set up
the file like this:

$ echo 'Last commit date: $Format:%cd$' > LAST_COMMIT
$ echo "LAST_COMMIT export-subst" >> .gitattributes
$ git add LAST_COMMIT .gitattributes
$ git commit -am 'adding LAST_COMMIT file for archives'

   When you run `git archive', the contents of that file when people
open the archive file will look like this:

$ cat LAST_COMMIT
Last commit date: $Format:Tue Apr 21 08:38:48 2009 -0700$


File: progit.info,  Node: Merge Strategies,  Prev: Exporting Your Repository,  Up: Git Attributes

7.2.4 Merge Strategies
----------------------

You can also use Git attributes to tell Git to use different merge
strategies for specific files in your project. One very useful option
is to tell Git to not try to merge specific files when they have
conflicts, but rather to use your side of the merge over someone
else's.

   This is helpful if a branch in your project has diverged or is
specialized, but you want to be able to merge changes back in from it,
and you want to ignore certain files. Say you have a database settings
file called database.xml that is different in two branches, and you
want to merge in your other branch without messing up the database
file. You can set up an attribute like this:

database.xml merge=ours

   If you merge in the other branch, instead of having merge conflicts
with the database.xml file, you see something like this:

$ git merge topic
Auto-merging database.xml
Merge made by recursive.

   In this case, database.xml stays at whatever version you originally
had.


File: progit.info,  Node: Git Hooks,  Next: An Example Git-Enforced Policy,  Prev: Git Attributes,  Up: Customizing Git

7.3 Git Hooks
=============

Like many other Version Control Systems, Git has a way to fire off
custom scripts when certain important actions occur. There are two
groups of these hooks: client side and server side. The client-side
hooks are for client operations such as committing and merging. The
server-side hooks are for Git server operations such as receiving
pushed commits. You can use these hooks for all sorts of reasons, and
you'll learn about a few of them here.

* Menu:

* Installing a Hook::
* Client-Side Hooks::
* Server-Side Hooks::


File: progit.info,  Node: Installing a Hook,  Next: Client-Side Hooks,  Up: Git Hooks

7.3.1 Installing a Hook
-----------------------

The hooks are all stored in the `hooks' subdirectory of the Git
directory. In most projects, that's `.git/hooks'. By default, Git
populates this directory with a bunch of example scripts, many of which
are useful by themselves; but they also document the input values of
each script. All the examples are written as shell scripts, with some
Perl thrown in, but any properly named executable scripts will work
fine --- you can write them in Ruby or Python or what have you. For
post-1.6 versions of Git, these example hook files end with .sample;
you'll need to rename them. For pre-1.6 versions of Git, the example
files are named properly but are not executable.

   To enable a hook script, put a file in the `hooks' subdirectory of
your Git directory that is named appropriately and is executable. From
that point forward, it should be called. I'll cover most of the major
hook filenames here.


File: progit.info,  Node: Client-Side Hooks,  Next: Server-Side Hooks,  Prev: Installing a Hook,  Up: Git Hooks

7.3.2 Client-Side Hooks
-----------------------

There are a lot of client-side hooks. This section splits them into
committing-workflow hooks, e-mail-workflow scripts, and the rest of the
client-side scripts.

* Menu:

* Committing-Workflow Hooks::
* E-mail Workflow Hooks::
* Other Client Hooks::


File: progit.info,  Node: Committing-Workflow Hooks,  Next: E-mail Workflow Hooks,  Up: Client-Side Hooks

7.3.2.1 Committing-Workflow Hooks
.................................

The first four hooks have to do with the committing process. The
`pre-commit' hook is run first, before you even type in a commit
message. It's used to inspect the snapshot that's about to be
committed, to see if you've forgotten something, to make sure tests
run, or to examine whatever you need to inspect in the code. Exiting
non-zero from this hook aborts the commit, although you can bypass it
with `git commit --no-verify'. You can do things like check for code
style (run lint or something equivalent), check for trailing whitespace
(the default hook does exactly that), or check for appropriate
documentation on new methods.

   The `prepare-commit-msg' hook is run before the commit message
editor is fired up but after the default message is created. It lets
you edit the default message before the commit author sees it. This
hook takes a few options: the path to the file that holds the commit
message so far, the type of commit, and the commit SHA-1 if this is an
amended commit. This hook generally isn't useful for normal commits;
rather, it's good for commits where the default message is
auto-generated, such as templated commit messages, merge commits,
squashed commits, and amended commits. You may use it in conjunction
with a commit template to programmatically insert information.

   The `commit-msg' hook takes one parameter, which again is the path
to a temporary file that contains the current commit message. If this
script exits non-zero, Git aborts the commit process, so you can use it
to validate your project state or commit message before allowing a
commit to go through. In the last section of this chapter, I'll
demonstrate using this hook to check that your commit message is
conformant to a required pattern.

   After the entire commit process is completed, the `post-commit' hook
runs. It doesn't take any parameters, but you can easily get the last
commit by running `git log -1 HEAD'. Generally, this script is used for
notification or something similar.

   The committing-workflow client-side scripts can be used in just
about any workflow. They're often used to enforce certain policies,
although it's important to note that these scripts aren't
transferred during a clone. You can enforce policy on the server side
to reject pushes of commits that don't conform to some policy, but
it's entirely up to the developer to use these scripts on the client
side. So, these are scripts to help developers, and they must be set up
and maintained by them, although they can be overridden or modified by
them at any time.


File: progit.info,  Node: E-mail Workflow Hooks,  Next: Other Client Hooks,  Prev: Committing-Workflow Hooks,  Up: Client-Side Hooks

7.3.2.2 E-mail Workflow Hooks
.............................

You can set up three client-side hooks for an e-mail-based workflow.
They're all invoked by the `git am' command, so if you aren't using
that command in your workflow, you can safely skip to the next section.
If you're taking patches over e-mail prepared by `git format-patch',
then some of these may be helpful to you.

   The first hook that is run is `applypatch-msg'. It takes a single
argument: the name of the temporary file that contains the proposed
commit message. Git aborts the patch if this script exits non-zero. You
can use this to make sure a commit message is properly formatted or to
normalize the message by having the script edit it in place.

   The next hook to run when applying patches via `git am' is
`pre-applypatch'. It takes no arguments and is run after the patch is
applied, so you can use it to inspect the snapshot before making the
commit. You can run tests or otherwise inspect the working tree with
this script. If something is missing or the tests don't pass, exiting
non-zero also aborts the `git am' script without committing the patch.

   The last hook to run during a `git am' operation is
`post-applypatch'. You can use it to notify a group or the author of
the patch you pulled in that you've done so. You can't stop the
patching process with this script.


File: progit.info,  Node: Other Client Hooks,  Prev: E-mail Workflow Hooks,  Up: Client-Side Hooks

7.3.2.3 Other Client Hooks
..........................

The `pre-rebase' hook runs before you rebase anything and can halt the
process by exiting non-zero. You can use this hook to disallow rebasing
any commits that have already been pushed. The example `pre-rebase'
hook that Git installs does this, although it assumes that next is the
name of the branch you publish. You'll likely need to change that to
whatever your stable, published branch is.

   After you run a successful `git checkout', the `post-checkout' hook
runs; you can use it to set up your working directory properly for your
project environment. This may mean moving in large binary files that
you don't want source controlled, auto-generating documentation, or
something along those lines.

   Finally, the `post-merge' hook runs after a successful `merge'
command. You can use it to restore data in the working tree that Git
can't track, such as permissions data. This hook can likewise
validate the presence of files external to Git control that you may
want copied in when the working tree changes.


File: progit.info,  Node: Server-Side Hooks,  Prev: Client-Side Hooks,  Up: Git Hooks

7.3.3 Server-Side Hooks
-----------------------

In addition to the client-side hooks, you can use a couple of important
server-side hooks as a system administrator to enforce nearly any kind
of policy for your project. These scripts run before and after pushes
to the server. The pre hooks can exit non-zero at any time to reject
the push as well as print an error message back to the client; you can
set up a push policy that's as complex as you wish.

* Menu:

* pre-receive and post-receive::
* update::


File: progit.info,  Node: pre-receive and post-receive,  Next: update,  Up: Server-Side Hooks

7.3.3.1 pre-receive and post-receive
....................................

The first script to run when handling a push from a client is
`pre-receive'. It takes a list of references that are being pushed from
stdin; if it exits non-zero, none of them are accepted. You can use
this hook to do things like make sure none of the updated references
are non-fast-forwards; or to check that the user doing the pushing has
create, delete, or push access or access to push updates to all the
files they're modifying with the push.

   The `post-receive' hook runs after the entire process is completed
and can be used to update other services or notify users. It takes the
same stdin data as the `pre-receive' hook. Examples include e-mailing a
list, notifying a continuous integration server, or updating a
ticket-tracking system --- you can even parse the commit messages to
see if any tickets need to be opened, modified, or closed. This script
can't stop the push process, but the client doesn't disconnect
until it has completed; so, be careful when you try to do anything that
may take a long time.


File: progit.info,  Node: update,  Prev: pre-receive and post-receive,  Up: Server-Side Hooks

7.3.3.2 update
..............

The update script is very similar to the `pre-receive' script, except
that it's run once for each branch the pusher is trying to update. If
the pusher is trying to push to multiple branches, `pre-receive' runs
only once, whereas update runs once per branch they're pushing to.
Instead of reading from stdin, this script takes three arguments: the
name of the reference (branch), the SHA-1 that reference pointed to
before the push, and the SHA-1 the user is trying to push. If the
update script exits non-zero, only that reference is rejected; other
references can still be updated.


File: progit.info,  Node: An Example Git-Enforced Policy,  Next: Summary,  Prev: Git Hooks,  Up: Customizing Git

7.4 An Example Git-Enforced Policy
==================================

In this section, you'll use what you've learned to establish a Git
workflow that checks for a custom commit message format, enforces
fast-forward-only pushes, and allows only certain users to modify
certain subdirectories in a project. You'll build client scripts that
help the developer know if their push will be rejected and server
scripts that actually enforce the policies.

   I used Ruby to write these, both because it's my preferred
scripting language and because I feel it's the most
pseudocode-looking of the scripting languages; thus you should be able
to roughly follow the code even if you don't use Ruby. However, any
language will work fine. All the sample hook scripts distributed with
Git are in either Perl or Bash scripting, so you can also see plenty of
examples of hooks in those languages by looking at the samples.

* Menu:

* Server-Side Hook::
* Client-Side Hooks::


File: progit.info,  Node: Server-Side Hook,  Next: Client-Side Hooks,  Up: An Example Git-Enforced Policy

7.4.1 Server-Side Hook
----------------------

All the server-side work will go into the update file in your hooks
directory. The update file runs once per branch being pushed and takes
the reference being pushed to, the old revision where that branch was,
and the new revision being pushed. You also have access to the user
doing the pushing if the push is being run over SSH. If you've
allowed everyone to connect with a single user (like "git") via
public-key authentication, you may have to give that user a shell
wrapper that determines which user is connecting based on the public
key, and set an environment variable specifying that user. Here I
assume the connecting user is in the `$USER' environment variable, so
your update script begins by gathering all the information you need:

#!/usr/bin/env ruby

$refname = ARGV[0]
$oldrev  = ARGV[1]
$newrev  = ARGV[2]
$user    = ENV['USER']

puts "Enforcing Policies... \n(#{$refname}) (#{$oldrev[0,6]}) (#{$newrev[0,6]})"

   Yes, I'm using global variables. Don't judge me --- it's
easier to demonstrate in this manner.

* Menu:

* Enforcing a Specific Commit-Message Format::
* Enforcing a User-Based ACL System::
* Enforcing Fast-Forward-Only Pushes::


File: progit.info,  Node: Enforcing a Specific Commit-Message Format,  Next: Enforcing a User-Based ACL System,  Up: Server-Side Hook

7.4.1.1 Enforcing a Specific Commit-Message Format
..................................................

Your first challenge is to enforce that each commit message must adhere
to a particular format. Just to have a target, assume that each message
has to include a string that looks like "ref: 1234" because you want
each commit to link to a work item in your ticketing system. You must
look at each commit being pushed up, see if that string is in the
commit message, and, if the string is absent from any of the commits,
exit non-zero so the push is rejected.

   You can get a list of the SHA-1 values of all the commits that are
being pushed by taking the `$newrev' and `$oldrev' values and passing
them to a Git plumbing command called `git rev-list'. This is basically
the `git log' command, but by default it prints out only the SHA-1
values and no other information. So, to get a list of all the commit
SHAs introduced between one commit SHA and another, you can run
something like this:

$ git rev-list 538c33..d14fc7
d14fc7c847ab946ec39590d87783c69b031bdfb7
9f585da4401b0a3999e84113824d15245c13f0be
234071a1be950e2a8d078e6141f5cd20c1e61ad3
dfa04c9ef3d5197182f13fb5b9b1fb7717d2222a
17716ec0f1ff5c77eff40b7fe912f9f6cfd0e475

   You can take that output, loop through each of those commit SHAs,
grab the message for it, and test that message against a regular
expression that looks for a pattern.

   You have to figure out how to get the commit message from each of
these commits to test. To get the raw commit data, you can use another
plumbing command called `git cat-file'. I'll go over all these
plumbing commands in detail in Chapter 9; but for now, here's what
that command gives you:

$ git cat-file commit ca82a6
tree cfda3bf379e4f8dba8717dee55aab78aef7f4daf
parent 085bb3bcb608e1e8451d4b2432f8ecbe6306e7e7
author Scott Chacon <schacon@gmail.com> 1205815931 -0700
committer Scott Chacon <schacon@gmail.com> 1240030591 -0700

changed the version number

   A simple way to get the commit message from a commit when you have
the SHA-1 value is to go to the first blank line and take everything
after that. You can do so with the `sed' command on Unix systems:

$ git cat-file commit ca82a6 | sed '1,/^$/d'
changed the version number

   You can use that incantation to grab the commit message from each
commit that is trying to be pushed and exit if you see anything that
doesn't match. To exit the script and reject the push, exit non-zero.
The whole method looks like this:

$regex = /\[ref: (\d+)\]/

# enforced custom commit message format
def check_message_format
  missed_revs = `git rev-list #{$oldrev}..#{$newrev}`.split("\n")
  missed_revs.each do |rev|
    message = `git cat-file commit #{rev} | sed '1,/^$/d'`
    if !$regex.match(message)
      puts "[POLICY] Your message is not formatted correctly"
      exit 1
    end
  end
end
check_message_format

   Putting that in your `update' script will reject updates that
contain commits that have messages that don't adhere to your rule.


File: progit.info,  Node: Enforcing a User-Based ACL System,  Next: Enforcing Fast-Forward-Only Pushes,  Prev: Enforcing a Specific Commit-Message Format,  Up: Server-Side Hook

7.4.1.2 Enforcing a User-Based ACL System
.........................................

Suppose you want to add a mechanism that uses an access control list
(ACL) that specifies which users are allowed to push changes to which
parts of your projects. Some people have full access, and others only
have access to push changes to certain subdirectories or specific
files. To enforce this, you'll write those rules to a file named
`acl' that lives in your bare Git repository on the server. You'll
have the `update' hook look at those rules, see what files are being
introduced for all the commits being pushed, and determine whether the
user doing the push has access to update all those files.

   The first thing you'll do is write your ACL. Here you'll use a
format very much like the CVS ACL mechanism: it uses a series of lines,
where the first field is `avail' or `unavail', the next field is a
comma-delimited list of the users to which the rule applies, and the
last field is the path to which the rule applies (blank meaning open
access). All of these fields are delimited by a pipe (`|') character.

   In this case, you have a couple of administrators, some
documentation writers with access to the `doc' directory, and one
developer who only has access to the `lib' and `tests' directories, and
your ACL file looks like this:

avail|nickh,pjhyett,defunkt,tpw
avail|usinclair,cdickens,ebronte|doc
avail|schacon|lib
avail|schacon|tests

   You begin by reading this data into a structure that you can use. In
this case, to keep the example simple, you'll only enforce the
`avail' directives. Here is a method that gives you an associative
array where the key is the user name and the value is an array of paths
to which the user has write access:

def get_acl_access_data(acl_file)
  # read in ACL data
  acl_file = File.read(acl_file).split("\n").reject { |line| line == '' }
  access = {}
  acl_file.each do |line|
    avail, users, path = line.split('|')
    next unless avail == 'avail'
    users.split(',').each do |user|
      access[user] ||= []
      access[user] << path
    end
  end
  access
end

   On the ACL file you looked at earlier, this `get_acl_access_data'
method returns a data structure that looks like this:

{"defunkt"=>[nil],
 "tpw"=>[nil],
 "nickh"=>[nil],
 "pjhyett"=>[nil],
 "schacon"=>["lib", "tests"],
 "cdickens"=>["doc"],
 "usinclair"=>["doc"],
 "ebronte"=>["doc"]}

   Now that you have the permissions sorted out, you need to determine
what paths the commits being pushed have modified, so you can make sure
the user who's pushing has access to all of them.

   You can pretty easily see what files have been modified in a single
commit with the `--name-only' option to the `git log' command
(mentioned briefly in Chapter 2):

$ git log -1 --name-only --pretty=format:'' 9f585d

README
lib/test.rb

   If you use the ACL structure returned from the `get_acl_access_data'
method and check it against the listed files in each of the commits,
you can determine whether the user has access to push all of their
commits:

# only allows certain users to modify certain subdirectories in a project
def check_directory_perms
  access = get_acl_access_data('acl')

  # see if anyone is trying to push something they can't
  new_commits = `git rev-list #{$oldrev}..#{$newrev}`.split("\n")
  new_commits.each do |rev|
    files_modified = `git log -1 --name-only --pretty=format:'' #{rev}`.split("\n")
    files_modified.each do |path|
      next if path.size == 0
      has_file_access = false
      access[$user].each do |access_path|
        if !access_path  # user has access to everything
          || (path.index(access_path) == 0) # access to this path
          has_file_access = true
        end
      end
      if !has_file_access
        puts "[POLICY] You do not have access to push to #{path}"
        exit 1
      end
    end
  end
end

check_directory_perms

   Most of that should be easy to follow. You get a list of new commits
being pushed to your server with `git rev-list'. Then, for each of
those, you find which files are modified and make sure the user who's
pushing has access to all the paths being modified. One Rubyism that
may not be clear is `path.index(access_path) == 0', which is true if
path begins with `access_path' --- this ensures that `access_path' is
not just in one of the allowed paths, but an allowed path begins with
each accessed path.

   Now your users can't push any commits with badly formed messages
or with modified files outside of their designated paths.


File: progit.info,  Node: Enforcing Fast-Forward-Only Pushes,  Prev: Enforcing a User-Based ACL System,  Up: Server-Side Hook

7.4.1.3 Enforcing Fast-Forward-Only Pushes
..........................................

The only thing left is to enforce fast-forward-only pushes. In Git
versions 1.6 or newer, you can set the `receive.denyDeletes' and
`receive.denyNonFastForwards' settings. But enforcing this with a hook
will work in older versions of Git, and you can modify it to do so only
for certain users or whatever else you come up with later.

   The logic for checking this is to see if any commits are reachable
from the older revision that aren't reachable from the newer one. If
there are none, then it was a fast-forward push; otherwise, you deny it:

# enforces fast-forward only pushes
def check_fast_forward
  missed_refs = `git rev-list #{$newrev}..#{$oldrev}`
  missed_ref_count = missed_refs.split("\n").size
  if missed_ref_count > 0
    puts "[POLICY] Cannot push a non fast-forward reference"
    exit 1
  end
end

check_fast_forward

   Everything is set up. If you run `chmod u+x .git/hooks/update',
which is the file you into which you should have put all this code, and
then try to push a non-fast-forwarded reference, you get something like
this:

$ git push -f origin master
Counting objects: 5, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 323 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
Enforcing Policies...
(refs/heads/master) (8338c5) (c5b616)
[POLICY] Cannot push a non-fast-forward reference
error: hooks/update exited with error code 1
error: hook declined to update refs/heads/master
To git@gitserver:project.git
 ! [remote rejected] master -> master (hook declined)
error: failed to push some refs to 'git@gitserver:project.git'

   There are a couple of interesting things here. First, you see this
where the hook starts running.

Enforcing Policies...
(refs/heads/master) (fb8c72) (c56860)

   Notice that you printed that out to stdout at the very beginning of
your update script. It's important to note that anything your script
prints to stdout will be transferred to the client.

   The next thing you'll notice is the error message.

[POLICY] Cannot push a non fast-forward reference
error: hooks/update exited with error code 1
error: hook declined to update refs/heads/master

   The first line was printed out by you, the other two were Git
telling you that the update script exited non-zero and that is what is
declining your push. Lastly, you have this:

To git@gitserver:project.git
 ! [remote rejected] master -> master (hook declined)
error: failed to push some refs to 'git@gitserver:project.git'

   You'll see a remote rejected message for each reference that your
hook declined, and it tells you that it was declined specifically
because of a hook failure.

   Furthermore, if the ref marker isn't there in any of your commits,
you'll see the error message you're printing out for that.

[POLICY] Your message is not formatted correctly

   Or if someone tries to edit a file they don't have access to and
push a commit containing it, they will see something similar. For
instance, if a documentation author tries to push a commit modifying
something in the `lib' directory, they see

[POLICY] You do not have access to push to lib/test.rb

   That's all. From now on, as long as that `update' script is there
and executable, your repository will never be rewound and will never
have a commit message without your pattern in it, and your users will
be sandboxed.


File: progit.info,  Node: Client-Side Hooks,  Prev: Server-Side Hook,  Up: An Example Git-Enforced Policy

7.4.2 Client-Side Hooks
-----------------------

The downside to this approach is the whining that will inevitably
result when your users' commit pushes are rejected. Having their
carefully crafted work rejected at the last minute can be extremely
frustrating and confusing; and furthermore, they will have to edit
their history to correct it, which isn't always for the faint of
heart.

   The answer to this dilemma is to provide some client-side hooks that
users can use to notify them when they're doing something that the
server is likely to reject. That way, they can correct any problems
before committing and before those issues become more difficult to fix.
Because hooks aren't transferred with a clone of a project, you must
distribute these scripts some other way and then have your users copy
them to their `.git/hooks' directory and make them executable. You can
distribute these hooks within the project or in a separate project, but
there is no way to set them up automatically.

   To begin, you should check your commit message just before each
commit is recorded, so you know the server won't reject your changes
due to badly formatted commit messages. To do this, you can add the
`commit-msg' hook. If you have it read the message from the file passed
as the first argument and compare that to the pattern, you can force
Git to abort the commit if there is no match:

#!/usr/bin/env ruby
message_file = ARGV[0]
message = File.read(message_file)

$regex = /\[ref: (\d+)\]/

if !$regex.match(message)
  puts "[POLICY] Your message is not formatted correctly"
  exit 1
end

   If that script is in place (in `.git/hooks/commit-msg') and
executable, and you commit with a message that isn't properly
formatted, you see this:

$ git commit -am 'test'
[POLICY] Your message is not formatted correctly

   No commit was completed in that instance. However, if your message
contains the proper pattern, Git allows you to commit:

$ git commit -am 'test [ref: 132]'
[master e05c914] test [ref: 132]
 1 files changed, 1 insertions(+), 0 deletions(-)

   Next, you want to make sure you aren't modifying files that are
outside your ACL scope. If your project's `.git' directory contains a
copy of the ACL file you used previously, then the following
`pre-commit' script will enforce those constraints for you:

#!/usr/bin/env ruby

$user    = ENV['USER']

# [ insert acl_access_data method from above ]

# only allows certain users to modify certain subdirectories in a project
def check_directory_perms
  access = get_acl_access_data('.git/acl')

  files_modified = `git diff-index --cached --name-only HEAD`.split("\n")
  files_modified.each do |path|
    next if path.size == 0
    has_file_access = false
    access[$user].each do |access_path|
    if !access_path || (path.index(access_path) == 0)
      has_file_access = true
    end
    if !has_file_access
      puts "[POLICY] You do not have access to push to #{path}"
      exit 1
    end
  end
end

check_directory_perms

   This is roughly the same script as the server-side part, but with
two important differences. First, the ACL file is in a different place,
because this script runs from your working directory, not from your Git
directory. You have to change the path to the ACL file from this

access = get_acl_access_data('acl')

   to this:

access = get_acl_access_data('.git/acl')

   The other important difference is the way you get a listing of the
files that have been changed. Because the server-side method looks at
the log of commits, and, at this point, the commit hasn't been
recorded yet, you must get your file listing from the staging area
instead. Instead of

files_modified = `git log -1 --name-only --pretty=format:'' #{ref}`

   you have to use

files_modified = `git diff-index --cached --name-only HEAD`

   But those are the only two differences --- otherwise, the script
works the same way. One caveat is that it expects you to be running
locally as the same user you push as to the remote machine. If that is
different, you must set the `$user' variable manually.

   The last thing you have to do is check that you're not trying to
push non-fast-forwarded references, but that is a bit less common. To
get a reference that isn't a fast-forward, you either have to rebase
past a commit you've already pushed up or try pushing a different
local branch up to the same remote branch.

   Because the server will tell you that you can't push a
non-fast-forward anyway, and the hook prevents forced pushes, the only
accidental thing you can try to catch is rebasing commits that have
already been pushed.

   Here is an example pre-rebase script that checks for that. It gets a
list of all the commits you're about to rewrite and checks whether
they exist in any of your remote references. If it sees one that is
reachable from one of your remote references, it aborts the rebase:

#!/usr/bin/env ruby

base_branch = ARGV[0]
if ARGV[1]
  topic_branch = ARGV[1]
else
  topic_branch = "HEAD"
end

target_shas = `git rev-list #{base_branch}..#{topic_branch}`.split("\n")
remote_refs = `git branch -r`.split("\n").map { |r| r.strip }

target_shas.each do |sha|
  remote_refs.each do |remote_ref|
    shas_pushed = `git rev-list ^#{sha}^@ refs/remotes/#{remote_ref}`
    if shas_pushed.split(“\n").include?(sha)
      puts "[POLICY] Commit #{sha} has already been pushed to #{remote_ref}"
      exit 1
    end
  end
end

   This script uses a syntax that wasn't covered in the Revision
Selection section of Chapter 6. You get a list of commits that have
already been pushed up by running this:

git rev-list ^#{sha}^@ refs/remotes/#{remote_ref}

   The `SHA^@' syntax resolves to all the parents of that commit.
You're looking for any commit that is reachable from the last commit
on the remote and that isn't reachable from any parent of any of the
SHAs you're trying to push up --- meaning it's a fast-forward.

   The main drawback to this approach is that it can be very slow and
is often unnecessary --- if you don't try to force the push with
`-f', the server will warn you and not accept the push. However, it's
an interesting exercise and can in theory help you avoid a rebase that
you might later have to go back and fix.


File: progit.info,  Node: Summary,  Prev: An Example Git-Enforced Policy,  Up: Customizing Git

7.5 Summary
===========

You've covered most of the major ways that you can customize your Git
client and server to best fit your workflow and projects. You've
learned about all sorts of configuration settings, file-based
attributes, and event hooks, and you've built an example
policy-enforcing server. You should now be able to make Git fit nearly
any workflow you can dream up.


File: progit.info,  Node: Git and Other Systems,  Next: Git Internals,  Prev: Customizing Git,  Up: Top

8 Git and Other Systems
***********************

The world isn't perfect. Usually, you can't immediately switch
every project you come in contact with to Git. Sometimes you're stuck
on a project using another VCS, and many times that system is
Subversion. You'll spend the first part of this chapter learning
about `git svn', the bidirectional Subversion gateway tool in Git.

   At some point, you may want to convert your existing project to Git.
The second part of this chapter covers how to migrate your project into
Git: first from Subversion, then from Perforce, and finally via a
custom import script for a nonstandard importing case.

* Menu:

* Git and Subversion::
* Migrating to Git::
* Summary::


File: progit.info,  Node: Git and Subversion,  Next: Migrating to Git,  Up: Git and Other Systems

8.1 Git and Subversion
======================

Currently, the majority of open source development projects and a large
number of corporate projects use Subversion to manage their source
code. It's the most popular open source VCS and has been around for
nearly a decade. It's also very similar in many ways to CVS, which
was the big boy of the source-control world before that.

   One of Git's great features is a bidirectional bridge to
Subversion called `git svn'. This tool allows you to use Git as a valid
client to a Subversion server, so you can use all the local features of
Git and then push to a Subversion server as if you were using
Subversion locally. This means you can do local branching and merging,
use the staging area, use rebasing and cherry-picking, and so on, while
your collaborators continue to work in their dark and ancient ways.
It's a good way to sneak Git into the corporate environment and help
your fellow developers become more efficient while you lobby to get the
infrastructure changed to support Git fully. The Subversion bridge is
the gateway drug to the DVCS world.

* Menu:

* git svn::
* Setting Up::
* Getting Started::
* Committing Back to Subversion::
* Pulling in New Changes::
* Git Branching Issues::
* Subversion Branching::
* Switching Active Branches::
* Subversion Commands::
* Git-Svn Summary::


File: progit.info,  Node: git svn,  Next: Setting Up,  Up: Git and Subversion

8.1.1 git svn
-------------

The base command in Git for all the Subversion bridging commands is
`git svn'. You preface everything with that. It takes quite a few
commands, so you'll learn about the common ones while going through a
few small workflows.

   It's important to note that when you're using `git svn',
you're interacting with Subversion, which is a system that is far
less sophisticated than Git. Although you can easily do local branching
and merging, it's generally best to keep your history as linear as
possible by rebasing your work and avoiding doing things like
simultaneously interacting with a Git remote repository.

   Don't rewrite your history and try to push again, and don't push
to a parallel Git repository to collaborate with fellow Git developers
at the same time. Subversion can have only a single linear history, and
confusing it is very easy. If you're working with a team, and some
are using SVN and others are using Git, make sure everyone is using the
SVN server to collaborate --- doing so will make your life easier.


File: progit.info,  Node: Setting Up,  Next: Getting Started,  Prev: git svn,  Up: Git and Subversion

8.1.2 Setting Up
----------------

To demonstrate this functionality, you need a typical SVN repository
that you have write access to. If you want to copy these examples,
you'll have to make a writeable copy of my test repository. In order
to do that easily, you can use a tool called `svnsync' that comes with
more recent versions of Subversion --- it should be distributed with at
least 1.4. For these tests, I created a new Subversion repository on
Google code that was a partial copy of the `protobuf' project, which is
a tool that encodes structured data for network transmission.

   To follow along, you first need to create a new local Subversion
repository:

$ mkdir /tmp/test-svn
$ svnadmin create /tmp/test-svn

   Then, enable all users to change revprops --- the easy way is to add
a pre-revprop-change script that always exits 0:

$ cat /tmp/test-svn/hooks/pre-revprop-change
#!/bin/sh
exit 0;
$ chmod +x /tmp/test-svn/hooks/pre-revprop-change

   You can now sync this project to your local machine by calling
`svnsync init' with the to and from repositories.

$ svnsync init file:///tmp/test-svn http://progit-example.googlecode.com/svn/

   This sets up the properties to run the sync. You can then clone the
code by running

$ svnsync sync file:///tmp/test-svn
Committed revision 1.
Copied properties for revision 1.
Committed revision 2.
Copied properties for revision 2.
Committed revision 3.
...

   Although this operation may take only a few minutes, if you try to
copy the original repository to another remote repository instead of a
local one, the process will take nearly an hour, even though there are
fewer than 100 commits. Subversion has to clone one revision at a time
and then push it back into another repository --- it's ridiculously
inefficient, but it's the only easy way to do this.


File: progit.info,  Node: Getting Started,  Next: Committing Back to Subversion,  Prev: Setting Up,  Up: Git and Subversion

8.1.3 Getting Started
---------------------

Now that you have a Subversion repository to which you have write
access, you can go through a typical workflow. You'll start with the
`git svn clone' command, which imports an entire Subversion repository
into a local Git repository. Remember that if you're importing from a
real hosted Subversion repository, you should replace the
`file:///tmp/test-svn' here with the URL of your Subversion repository:

$ git svn clone file:///tmp/test-svn -T trunk -b branches -t tags
Initialized empty Git repository in /Users/schacon/projects/testsvnsync/svn/.git/
r1 = b4e387bc68740b5af56c2a5faf4003ae42bd135c (trunk)
      A    m4/acx_pthread.m4
      A    m4/stl_hash.m4
...
r75 = d1957f3b307922124eec6314e15bcda59e3d9610 (trunk)
Found possible branch point: file:///tmp/test-svn/trunk => \
    file:///tmp/test-svn /branches/my-calc-branch, 75
Found branch parent: (my-calc-branch) d1957f3b307922124eec6314e15bcda59e3d9610
Following parent with do_switch
Successfully followed parent
r76 = 8624824ecc0badd73f40ea2f01fce51894189b01 (my-calc-branch)
Checked out HEAD:
 file:///tmp/test-svn/branches/my-calc-branch r76

   This runs the equivalent of two commands --- `git svn init' followed
by `git svn fetch' --- on the URL you provide. This can take a while.
The test project has only about 75 commits and the codebase isn't
that big, so it takes just a few minutes. However, Git has to check out
each version, one at a time, and commit it individually. For a project
with hundreds or thousands of commits, this can literally take hours or
even days to finish.

   The `-T trunk -b branches -t tags' part tells Git that this
Subversion repository follows the basic branching and tagging
conventions. If you name your trunk, branches, or tags differently, you
can change these options. Because this is so common, you can replace
this entire part with `-s', which means standard layout and implies all
those options. The following command is equivalent:

$ git svn clone file:///tmp/test-svn -s

   At this point, you should have a valid Git repository that has
imported your branches and tags:

$ git branch -a
* master
  my-calc-branch
  tags/2.0.2
  tags/release-2.0.1
  tags/release-2.0.2
  tags/release-2.0.2rc1
  trunk

   It's important to note how this tool namespaces your remote
references differently. When you're cloning a normal Git repository,
you get all the branches on that remote server available locally as
something like `origin/[branch]' - namespaced by the name of the
remote. However, `git svn' assumes that you won't have multiple
remotes and saves all its references to points on the remote server
with no namespacing. You can use the Git plumbing command `show-ref' to
look at all your full reference names:

$ git show-ref
1cbd4904d9982f386d87f88fce1c24ad7c0f0471 refs/heads/master
aee1ecc26318164f355a883f5d99cff0c852d3c4 refs/remotes/my-calc-branch
03d09b0e2aad427e34a6d50ff147128e76c0e0f5 refs/remotes/tags/2.0.2
50d02cc0adc9da4319eeba0900430ba219b9c376 refs/remotes/tags/release-2.0.1
4caaa711a50c77879a91b8b90380060f672745cb refs/remotes/tags/release-2.0.2
1c4cb508144c513ff1214c3488abe66dcb92916f refs/remotes/tags/release-2.0.2rc1
1cbd4904d9982f386d87f88fce1c24ad7c0f0471 refs/remotes/trunk

   A normal Git repository looks more like this:

$ git show-ref
83e38c7a0af325a9722f2fdc56b10188806d83a1 refs/heads/master
3e15e38c198baac84223acfc6224bb8b99ff2281 refs/remotes/gitserver/master
0a30dd3b0c795b80212ae723640d4e5d48cabdff refs/remotes/origin/master
25812380387fdd55f916652be4881c6f11600d6f refs/remotes/origin/testing

   You have two remote servers: one named `gitserver' with a `master'
branch; and another named `origin' with two branches, `master' and
`testing'.

   Notice how in the example of remote references imported from `git
svn', tags are added as remote branches, not as real Git tags. Your
Subversion import looks like it has a remote named tags with branches
under it.


File: progit.info,  Node: Committing Back to Subversion,  Next: Pulling in New Changes,  Prev: Getting Started,  Up: Git and Subversion

8.1.4 Committing Back to Subversion
-----------------------------------

Now that you have a working repository, you can do some work on the
project and push your commits back upstream, using Git effectively as a
SVN client. If you edit one of the files and commit it, you have a
commit that exists in Git locally that doesn't exist on the
Subversion server:

$ git commit -am 'Adding git-svn instructions to the README'
[master 97031e5] Adding git-svn instructions to the README
 1 files changed, 1 insertions(+), 1 deletions(-)

   Next, you need to push your change upstream. Notice how this changes
the way you work with Subversion --- you can do several commits offline
and then push them all at once to the Subversion server. To push to a
Subversion server, you run the `git svn dcommit' command:

$ git svn dcommit
Committing to file:///tmp/test-svn/trunk ...
       M      README.txt
Committed r79
       M      README.txt
r79 = 938b1a547c2cc92033b74d32030e86468294a5c8 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk

   This takes all the commits you've made on top of the Subversion
server code, does a Subversion commit for each, and then rewrites your
local Git commit to include a unique identifier. This is important
because it means that all the SHA-1 checksums for your commits change.
Partly for this reason, working with Git-based remote versions of your
projects concurrently with a Subversion server isn't a good idea. If
you look at the last commit, you can see the new `git-svn-id' that was
added:

$ git log -1
commit 938b1a547c2cc92033b74d32030e86468294a5c8
Author: schacon <schacon@4c93b258-373f-11de-be05-5f7a86268029>
Date:   Sat May 2 22:06:44 2009 +0000

    Adding git-svn instructions to the README

    git-svn-id: file:///tmp/test-svn/trunk@79 4c93b258-373f-11de-be05-5f7a86268029

   Notice that the SHA checksum that originally started with `97031e5'
when you committed now begins with `938b1a5'. If you want to push to
both a Git server and a Subversion server, you have to push (`dcommit')
to the Subversion server first, because that action changes your commit
data.


File: progit.info,  Node: Pulling in New Changes,  Next: Git Branching Issues,  Prev: Committing Back to Subversion,  Up: Git and Subversion

8.1.5 Pulling in New Changes
----------------------------

If you're working with other developers, then at some point one of
you will push, and then the other one will try to push a change that
conflicts. That change will be rejected until you merge in their work.
In `git svn', it looks like this:

$ git svn dcommit
Committing to file:///tmp/test-svn/trunk ...
Merge conflict during commit: Your file or directory 'README.txt' is probably \
out-of-date: resource out of date; try updating at /Users/schacon/libexec/git-\
core/git-svn line 482

   To resolve this situation, you can run `git svn rebase', which pulls
down any changes on the server that you don't have yet and rebases
any work you have on top of what is on the server:

$ git svn rebase
       M      README.txt
r80 = ff829ab914e8775c7c025d741beb3d523ee30bc4 (trunk)
First, rewinding head to replay your work on top of it...
Applying: first user change

   Now, all your work is on top of what is on the Subversion server, so
you can successfully `dcommit':

$ git svn dcommit
Committing to file:///tmp/test-svn/trunk ...
       M      README.txt
Committed r81
       M      README.txt
r81 = 456cbe6337abe49154db70106d1836bc1332deed (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk

   It's important to remember that unlike Git, which requires you to
merge upstream work you don't yet have locally before you can push,
`git svn' makes you do that only if the changes conflict. If someone
else pushes a change to one file and then you push a change to another
file, your `dcommit' will work fine:

$ git svn dcommit
Committing to file:///tmp/test-svn/trunk ...
       M      configure.ac
Committed r84
       M      autogen.sh
r83 = 8aa54a74d452f82eee10076ab2584c1fc424853b (trunk)
       M      configure.ac
r84 = cdbac939211ccb18aa744e581e46563af5d962d0 (trunk)
W: d2f23b80f67aaaa1f6f5aaef48fce3263ac71a92 and refs/remotes/trunk differ, \
  using rebase:
:100755 100755 efa5a59965fbbb5b2b0a12890f1b351bb5493c18 \
  015e4c98c482f0fa71e4d5434338014530b37fa6 M   autogen.sh
First, rewinding head to replay your work on top of it...
Nothing to do.

   This is important to remember, because the outcome is a project
state that didn't exist on either of your computers when you pushed.
If the changes are incompatible but don't conflict, you may get
issues that are difficult to diagnose. This is different than using a
Git server --- in Git, you can fully test the state on your client
system before publishing it, whereas in SVN, you can't ever be
certain that the states immediately before commit and after commit are
identical.

   You should also run this command to pull in changes from the
Subversion server, even if you're not ready to commit yourself. You
can run `git svn fetch' to grab the new data, but `git svn rebase' does
the fetch and then updates your local commits.

$ git svn rebase
       M      generate_descriptor_proto.sh
r82 = bd16df9173e424c6f52c337ab6efa7f7643282f1 (trunk)
First, rewinding head to replay your work on top of it...
Fast-forwarded master to refs/remotes/trunk.

   Running `git svn rebase' every once in a while makes sure your code
is always up to date. You need to be sure your working directory is
clean when you run this, though. If you have local changes, you must
either stash your work or temporarily commit it before running `git svn
rebase' --- otherwise, the command will stop if it sees that the rebase
will result in a merge conflict.


File: progit.info,  Node: Git Branching Issues,  Next: Subversion Branching,  Prev: Pulling in New Changes,  Up: Git and Subversion

8.1.6 Git Branching Issues
--------------------------

When you've become comfortable with a Git workflow, you'll likely
create topic branches, do work on them, and then merge them in. If
you're pushing to a Subversion server via git svn, you may want to
rebase your work onto a single branch each time instead of merging
branches together. The reason to prefer rebasing is that Subversion has
a linear history and doesn't deal with merges like Git does, so git
svn follows only the first parent when converting the snapshots into
Subversion commits.

   Suppose your history looks like the following: you created an
`experiment' branch, did two commits, and then merged them back into
`master'. When you `dcommit', you see output like this:

$ git svn dcommit
Committing to file:///tmp/test-svn/trunk ...
       M      CHANGES.txt
Committed r85
       M      CHANGES.txt
r85 = 4bfebeec434d156c36f2bcd18f4e3d97dc3269a2 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
COPYING.txt: locally modified
INSTALL.txt: locally modified
       M      COPYING.txt
       M      INSTALL.txt
Committed r86
       M      INSTALL.txt
       M      COPYING.txt
r86 = 2647f6b86ccfcaad4ec58c520e369ec81f7c283c (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk

   Running `dcommit' on a branch with merged history works fine, except
that when you look at your Git project history, it hasn't rewritten
either of the commits you made on the `experiment' branch --- instead,
all those changes appear in the SVN version of the single merge commit.

   When someone else clones that work, all they see is the merge commit
with all the work squashed into it; they don't see the commit data
about where it came from or when it was committed.


File: progit.info,  Node: Subversion Branching,  Next: Switching Active Branches,  Prev: Git Branching Issues,  Up: Git and Subversion

8.1.7 Subversion Branching
--------------------------

Branching in Subversion isn't the same as branching in Git; if you
can avoid using it much, that's probably best. However, you can
create and commit to branches in Subversion using git svn.

* Menu:

* Creating a New SVN Branch::


File: progit.info,  Node: Creating a New SVN Branch,  Up: Subversion Branching

8.1.7.1 Creating a New SVN Branch
.................................

To create a new branch in Subversion, you run `git svn branch
[branchname]':

$ git svn branch opera
Copying file:///tmp/test-svn/trunk at r87 to file:///tmp/test-svn/branches/opera...
Found possible branch point: file:///tmp/test-svn/trunk => \
  file:///tmp/test-svn/branches/opera, 87
Found branch parent: (opera) 1f6bfe471083cbca06ac8d4176f7ad4de0d62e5f
Following parent with do_switch
Successfully followed parent
r89 = 9b6fe0b90c5c9adf9165f700897518dbc54a7cbf (opera)

   This does the equivalent of the `svn copy trunk branches/opera'
command in Subversion and operates on the Subversion server. It's
important to note that it doesn't check you out into that branch; if
you commit at this point, that commit will go to `trunk' on the server,
not `opera'.


File: progit.info,  Node: Switching Active Branches,  Next: Subversion Commands,  Prev: Subversion Branching,  Up: Git and Subversion

8.1.8 Switching Active Branches
-------------------------------

Git figures out what branch your dcommits go to by looking for the tip
of any of your Subversion branches in your history --- you should have
only one, and it should be the last one with a `git-svn-id' in your
current branch history.

   If you want to work on more than one branch simultaneously, you can
set up local branches to `dcommit' to specific Subversion branches by
starting them at the imported Subversion commit for that branch. If you
want an `opera' branch that you can work on separately, you can run

$ git branch opera remotes/opera

   Now, if you want to merge your `opera' branch into `trunk' (your
`master' branch), you can do so with a normal `git merge'. But you need
to provide a descriptive commit message (via `-m'), or the merge will
say "Merge branch opera" instead of something useful.

   Remember that although you're using `git merge' to do this
operation, and the merge likely will be much easier than it would be in
Subversion (because Git will automatically detect the appropriate merge
base for you), this isn't a normal Git merge commit. You have to push
this data back to a Subversion server that can't handle a commit that
tracks more than one parent; so, after you push it up, it will look
like a single commit that squashed in all the work of another branch
under a single commit. After you merge one branch into another, you
can't easily go back and continue working on that branch, as you
normally can in Git. The `dcommit' command that you run erases any
information that says what branch was merged in, so subsequent
merge-base calculations will be wrong --- the dcommit makes your `git
merge' result look like you ran `git merge --squash'. Unfortunately,
there's no good way to avoid this situation --- Subversion can't
store this information, so you'll always be crippled by its
limitations while you're using it as your server. To avoid issues,
you should delete the local branch (in this case, `opera') after you
merge it into trunk.


File: progit.info,  Node: Subversion Commands,  Next: Git-Svn Summary,  Prev: Switching Active Branches,  Up: Git and Subversion

8.1.9 Subversion Commands
-------------------------

The `git svn' toolset provides a number of commands to help ease the
transition to Git by providing some functionality that's similar to
what you had in Subversion. Here are a few commands that give you what
Subversion used to.

* Menu:

* SVN Style History::
* SVN Annotation::
* SVN Server Information::
* Ignoring What Subversion Ignores::


File: progit.info,  Node: SVN Style History,  Next: SVN Annotation,  Up: Subversion Commands

8.1.9.1 SVN Style History
.........................

If you're used to Subversion and want to see your history in SVN
output style, you can run `git svn log' to view your commit history in
SVN formatting:

$ git svn log
------------------------------------------------------------------------
r87 | schacon | 2009-05-02 16:07:37 -0700 (Sat, 02 May 2009) | 2 lines

autogen change

------------------------------------------------------------------------
r86 | schacon | 2009-05-02 16:00:21 -0700 (Sat, 02 May 2009) | 2 lines

Merge branch 'experiment'

------------------------------------------------------------------------
r85 | schacon | 2009-05-02 16:00:09 -0700 (Sat, 02 May 2009) | 2 lines

updated the changelog

   You should know two important things about `git svn log'. First, it
works offline, unlike the real `svn log' command, which asks the
Subversion server for the data. Second, it only shows you commits that
have been committed up to the Subversion server. Local Git commits that
you haven't dcommited don't show up; neither do commits that people
have made to the Subversion server in the meantime. It's more like
the last known state of the commits on the Subversion server.


File: progit.info,  Node: SVN Annotation,  Next: SVN Server Information,  Prev: SVN Style History,  Up: Subversion Commands

8.1.9.2 SVN Annotation
......................

Much as the `git svn log' command simulates the `svn log' command
offline, you can get the equivalent of `svn annotate' by running `git
svn blame [FILE]'. The output looks like this:

$ git svn blame README.txt
 2   temporal Protocol Buffers - Google's data interchange format
 2   temporal Copyright 2008 Google Inc.
 2   temporal http://code.google.com/apis/protocolbuffers/
 2   temporal
22   temporal C++ Installation - Unix
22   temporal =======================
 2   temporal
79    schacon Committing in git-svn.
78    schacon
 2   temporal To build and install the C++ Protocol Buffer runtime and the Protocol
 2   temporal Buffer compiler (protoc) execute the following:
 2   temporal

   Again, it doesn't show commits that you did locally in Git or that
have been pushed to Subversion in the meantime.


File: progit.info,  Node: SVN Server Information,  Next: Ignoring What Subversion Ignores,  Prev: SVN Annotation,  Up: Subversion Commands

8.1.9.3 SVN Server Information
..............................

You can also get the same sort of information that `svn info' gives you
by running `git svn info':

$ git svn info
Path: .
URL: https://schacon-test.googlecode.com/svn/trunk
Repository Root: https://schacon-test.googlecode.com/svn
Repository UUID: 4c93b258-373f-11de-be05-5f7a86268029
Revision: 87
Node Kind: directory
Schedule: normal
Last Changed Author: schacon
Last Changed Rev: 87
Last Changed Date: 2009-05-02 16:07:37 -0700 (Sat, 02 May 2009)

   This is like `blame' and `log' in that it runs offline and is up to
date only as of the last time you communicated with the Subversion
server.


File: progit.info,  Node: Ignoring What Subversion Ignores,  Prev: SVN Server Information,  Up: Subversion Commands

8.1.9.4 Ignoring What Subversion Ignores
........................................

If you clone a Subversion repository that has `svn:ignore' properties
set anywhere, you'll likely want to set corresponding `.gitignore'
files so you don't accidentally commit files that you shouldn't.
`git svn' has two commands to help with this issue. The first is `git
svn create-ignore', which automatically creates corresponding
`.gitignore' files for you so your next commit can include them.

   The second command is `git svn show-ignore', which prints to stdout
the lines you need to put in a `.gitignore' file so you can redirect
the output into your project exclude file:

$ git svn show-ignore > .git/info/exclude

   That way, you don't litter the project with `.gitignore' files.
This is a good option if you're the only Git user on a Subversion
team, and your teammates don't want `.gitignore' files in the project.


File: progit.info,  Node: Git-Svn Summary,  Prev: Subversion Commands,  Up: Git and Subversion

8.1.10 Git-Svn Summary
----------------------

The `git svn' tools are useful if you're stuck with a Subversion
server for now or are otherwise in a development environment that
necessitates running a Subversion server. You should consider it
crippled Git, however, or you'll hit issues in translation that may
confuse you and your collaborators. To stay out of trouble, try to
follow these guidelines:

   * Keep a linear Git history that doesn't contain merge commits
     made by `git merge'. Rebase any work you do outside of your
     mainline branch back onto it; don't merge it in.

   * Don't set up and collaborate on a separate Git server. Possibly
     have one to speed up clones for new developers, but don't push
     anything to it that doesn't have a `git-svn-id' entry. You may
     even want to add a `pre-receive' hook that checks each commit
     message for a `git-svn-id' and rejects pushes that contain commits
     without it.

   If you follow those guidelines, working with a Subversion server can
be more bearable. However, if it's possible to move to a real Git
server, doing so can gain your team a lot more.


File: progit.info,  Node: Migrating to Git,  Next: Summary,  Prev: Git and Subversion,  Up: Git and Other Systems

8.2 Migrating to Git
====================

If you have an existing codebase in another VCS but you've decided to
start using Git, you must migrate your project one way or another. This
section goes over some importers that are included with Git for common
systems and then demonstrates how to develop your own custom importer.

* Menu:

* Importing::
* Subversion::
* Perforce::
* A Custom Importer::


File: progit.info,  Node: Importing,  Next: Subversion,  Up: Migrating to Git

8.2.1 Importing
---------------

You'll learn how to import data from two of the bigger professionally
used SCM systems --- Subversion and Perforce --- both because they make
up the majority of users I hear of who are currently switching, and
because high-quality tools for both systems are distributed with Git.


File: progit.info,  Node: Subversion,  Next: Perforce,  Prev: Importing,  Up: Migrating to Git

8.2.2 Subversion
----------------

If you read the previous section about using `git svn', you can easily
use those instructions to `git svn clone' a repository; then, stop
using the Subversion server, push to a new Git server, and start using
that. If you want the history, you can accomplish that as quickly as
you can pull the data out of the Subversion server (which may take a
while).

   However, the import isn't perfect; and because it will take so
long, you may as well do it right. The first problem is the author
information. In Subversion, each person committing has a user on the
system who is recorded in the commit information. The examples in the
previous section show `schacon' in some places, such as the `blame'
output and the `git svn log'. If you want to map this to better Git
author data, you need a mapping from the Subversion users to the Git
authors. Create a file called `users.txt' that has this mapping in a
format like this:

schacon = Scott Chacon <schacon@geemail.com>
selse = Someo Nelse <selse@geemail.com>

   To get a list of the author names that SVN uses, you can run this:

$ svn log --xml | grep author | sort -u | perl -pe 's/.>(.?)<./$1 = /'

   That gives you the log output in XML format --- you can look for the
authors, create a unique list, and then strip out the XML. (Obviously
this only works on a machine with `grep', `sort', and `perl'
installed.) Then, redirect that output into your users.txt file so you
can add the equivalent Git user data next to each entry.

   You can provide this file to `git svn' to help it map the author
data more accurately. You can also tell `git svn' not to include the
metadata that Subversion normally imports, by passing `--no-metadata'
to the `clone' or `init' command. This makes your `import' command look
like this:

$ git-svn clone http://my-project.googlecode.com/svn/ \
      --authors-file=users.txt --no-metadata -s my_project

   Now you should have a nicer Subversion import in your `my_project'
directory. Instead of commits that look like this

commit 37efa680e8473b615de980fa935944215428a35a
Author: schacon <schacon@4c93b258-373f-11de-be05-5f7a86268029>
Date:   Sun May 3 00:12:22 2009 +0000

    fixed install - go to trunk

    git-svn-id: https://my-project.googlecode.com/svn/trunk@94 4c93b258-373f-11de-
    be05-5f7a86268029

   they look like this:

commit 03a8785f44c8ea5cdb0e8834b7c8e6c469be2ff2
Author: Scott Chacon <schacon@geemail.com>
Date:   Sun May 3 00:12:22 2009 +0000

    fixed install - go to trunk

   Not only does the Author field look a lot better, but the
`git-svn-id' is no longer there, either.

   You need to do a bit of `post-import' cleanup. For one thing, you
should clean up the weird references that `git svn' set up. First
you'll move the tags so they're actual tags rather than strange
remote branches, and then you'll move the rest of the branches so
they're local.

   To move the tags to be proper Git tags, run

$ cp -Rf .git/refs/remotes/tags/* .git/refs/tags/
$ rm -Rf .git/refs/remotes/tags

   This takes the references that were remote branches that started
with `tag/' and makes them real (lightweight) tags.

   Next, move the rest of the references under `refs/remotes' to be
local branches:

$ cp -Rf .git/refs/remotes/* .git/refs/heads/
$ rm -Rf .git/refs/remotes

   Now all the old branches are real Git branches and all the old tags
are real Git tags. The last thing to do is add your new Git server as a
remote and push to it. Because you want all your branches and tags to
go up, you can run this:

$ git push origin --all

   All your branches and tags should be on your new Git server in a
nice, clean import.


File: progit.info,  Node: Perforce,  Next: A Custom Importer,  Prev: Subversion,  Up: Migrating to Git

8.2.3 Perforce
--------------

The next system you'll look at importing from is Perforce. A Perforce
importer is also distributed with Git, but only in the `contrib'
section of the source code --- it isn't available by default like
`git svn'. To run it, you must get the Git source code, which you can
download from git.kernel.org:

$ git clone git://git.kernel.org/pub/scm/git/git.git
$ cd git/contrib/fast-import

   In this `fast-import' directory, you should find an executable
Python script named `git-p4'. You must have Python and the `p4' tool
installed on your machine for this import to work. For example,
you'll import the Jam project from the Perforce Public Depot. To set
up your client, you must export the P4PORT environment variable to
point to the Perforce depot:

$ export P4PORT=public.perforce.com:1666

   Run the `git-p4 clone' command to import the Jam project from the
Perforce server, supplying the depot and project path and the path into
which you want to import the project:

$ git-p4 clone //public/jam/src@all /opt/p4import
Importing from //public/jam/src@all into /opt/p4import
Reinitialized existing Git repository in /opt/p4import/.git/
Import destination: refs/remotes/p4/master
Importing revision 4409 (100%)

   If you go to the `/opt/p4import' directory and run `git log', you
can see your imported work:

$ git log -2
commit 1fd4ec126171790efd2db83548b85b1bbbc07dc2
Author: Perforce staff <support@perforce.com>
Date:   Thu Aug 19 10:18:45 2004 -0800

    Drop 'rc3' moniker of jam-2.5.  Folded rc2 and rc3 RELNOTES into
    the main part of the document.  Built new tar/zip balls.

    Only 16 months later.

    [git-p4: depot-paths = "//public/jam/src/": change = 4409]

commit ca8870db541a23ed867f38847eda65bf4363371d
Author: Richard Geiger <rmg@perforce.com>
Date:   Tue Apr 22 20:51:34 2003 -0800

    Update derived jamgram.c

    [git-p4: depot-paths = "//public/jam/src/": change = 3108]

   You can see the `git-p4' identifier in each commit. It's fine to
keep that identifier there, in case you need to reference the Perforce
change number later. However, if you'd like to remove the identifier,
now is the time to do so --- before you start doing work on the new
repository. You can use `git filter-branch' to remove the identifier
strings en masse:

$ git filter-branch --msg-filter '
        sed -e "/^\[git-p4:/d"
'
Rewrite 1fd4ec126171790efd2db83548b85b1bbbc07dc2 (123/123)
Ref 'refs/heads/master' was rewritten

   If you run `git log', you can see that all the SHA-1 checksums for
the commits have changed, but the `git-p4' strings are no longer in the
commit messages:

$ git log -2
commit 10a16d60cffca14d454a15c6164378f4082bc5b0
Author: Perforce staff <support@perforce.com>
Date:   Thu Aug 19 10:18:45 2004 -0800

    Drop 'rc3' moniker of jam-2.5.  Folded rc2 and rc3 RELNOTES into
    the main part of the document.  Built new tar/zip balls.

    Only 16 months later.

commit 2b6c6db311dd76c34c66ec1c40a49405e6b527b2
Author: Richard Geiger <rmg@perforce.com>
Date:   Tue Apr 22 20:51:34 2003 -0800

    Update derived jamgram.c

   Your import is ready to push up to your new Git server.


File: progit.info,  Node: A Custom Importer,  Prev: Perforce,  Up: Migrating to Git

8.2.4 A Custom Importer
-----------------------

If your system isn't Subversion or Perforce, you should look for an
importer online --- quality importers are available for CVS, Clear
Case, Visual Source Safe, even a directory of archives. If none of
these tools works for you, you have a rarer tool, or you otherwise need
a more custom importing process, you should use `git fast-import'. This
command reads simple instructions from stdin to write specific Git
data. It's much easier to create Git objects this way than to run the
raw Git commands or try to write the raw objects (see Chapter 9 for
more information). This way, you can write an import script that reads
the necessary information out of the system you're importing from and
prints straightforward instructions to stdout. You can then run this
program and pipe its output through `git fast-import'.

   To quickly demonstrate, you'll write a simple importer. Suppose
you work in current, you back up your project by occasionally copying
the directory into a time-stamped `back_YYYY_MM_DD' backup directory,
and you want to import this into Git. Your directory structure looks
like this:

$ ls /opt/import_from
back_2009_01_02
back_2009_01_04
back_2009_01_14
back_2009_02_03
current

   In order to import a Git directory, you need to review how Git
stores its data. As you may remember, Git is fundamentally a linked
list of commit objects that point to a snapshot of content. All you
have to do is tell `fast-import' what the content snapshots are, what
commit data points to them, and the order they go in. Your strategy
will be to go through the snapshots one at a time and create commits
with the contents of each directory, linking each commit back to the
previous one.

   As you did in the "An Example Git Enforced Policy" section of
Chapter 7, we'll write this in Ruby, because it's what I generally
work with and it tends to be easy to read. You can write this example
pretty easily in anything you're familiar with --- it just needs to
print the appropriate information to stdout. And, if you are running on
Windows, this means you'll need to take special care to not introduce
carriage returns at the end your lines --- git fast-import is very
particular about just wanting line feeds (LF) not the carriage return
line feeds (CRLF) that Windows uses.

   To begin, you'll change into the target directory and identify
every subdirectory, each of which is a snapshot that you want to import
as a commit. You'll change into each subdirectory and print the
commands necessary to export it. Your basic main loop looks like this:

last_mark = nil

# loop through the directories
Dir.chdir(ARGV[0]) do
  Dir.glob("*").each do |dir|
    next if File.file?(dir)

    # move into the target directory
    Dir.chdir(dir) do
      last_mark = print_export(dir, last_mark)
    end
  end
end

   You run `print_export' inside each directory, which takes the
manifest and mark of the previous snapshot and returns the manifest and
mark of this one; that way, you can link them properly. "Mark" is the
`fast-import' term for an identifier you give to a commit; as you
create commits, you give each one a mark that you can use to link to it
from other commits. So, the first thing to do in your `print_export'
method is generate a mark from the directory name:

mark = convert_dir_to_mark(dir)

   You'll do this by creating an array of directories and using the
index value as the mark, because a mark must be an integer. Your method
looks like this:

$marks = []
def convert_dir_to_mark(dir)
  if !$marks.include?(dir)
    $marks << dir
  end
  ($marks.index(dir) + 1).to_s
end

   Now that you have an integer representation of your commit, you need
a date for the commit metadata. Because the date is expressed in the
name of the directory, you'll parse it out. The next line in your
`print_export' file is

date = convert_dir_to_date(dir)

   where `convert_dir_to_date' is defined as

def convert_dir_to_date(dir)
  if dir == 'current'
    return Time.now().to_i
  else
    dir = dir.gsub('back_', '')
    (year, month, day) = dir.split('_')
    return Time.local(year, month, day).to_i
  end
end

   That returns an integer value for the date of each directory. The
last piece of meta-information you need for each commit is the
committer data, which you hardcode in a global variable:

$author = 'Scott Chacon <schacon@example.com>'

   Now you're ready to begin printing out the commit data for your
importer. The initial information states that you're defining a
commit object and what branch it's on, followed by the mark you've
generated, the committer information and commit message, and then the
previous commit, if any. The code looks like this:

# print the import information
puts 'commit refs/heads/master'
puts 'mark :' + mark
puts "committer #{$author} #{date} -0700"
export_data('imported from ' + dir)
puts 'from :' + last_mark if last_mark

   You hardcode the time zone (-0700) because doing so is easy. If
you're importing from another system, you must specify the time zone
as an offset. The commit message must be expressed in a special format:

data (size)\n(contents)

   The format consists of the word data, the size of the data to be
read, a newline, and finally the data. Because you need to use the same
format to specify the file contents later, you create a helper method,
`export_data':

def export_data(string)
  print "data #{string.size}\n#{string}"
end

   All that's left is to specify the file contents for each snapshot.
This is easy, because you have each one in a directory --- you can
print out the `deleteall' command followed by the contents of each file
in the directory. Git will then record each snapshot appropriately:

puts 'deleteall'
Dir.glob("**/*").each do |file|
  next if !File.file?(file)
  inline_data(file)
end

   Note: Because many systems think of their revisions as changes from
one commit to another, fast-import can also take commands with each
commit to specify which files have been added, removed, or modified and
what the new contents are. You could calculate the differences between
snapshots and provide only this data, but doing so is more complex ---
you may as well give Git all the data and let it figure it out. If this
is better suited to your data, check the `fast-import' man page for
details about how to provide your data in this manner.

   The format for listing the new file contents or specifying a
modified file with the new contents is as follows:

M 644 inline path/to/file
data (size)
(file contents)

   Here, 644 is the mode (if you have executable files, you need to
detect and specify 755 instead), and inline says you'll list the
contents immediately after this line. Your `inline_data' method looks
like this:

def inline_data(file, code = 'M', mode = '644')
  content = File.read(file)
  puts "#{code} #{mode} inline #{file}"
  export_data(content)
end

   You reuse the `export_data' method you defined earlier, because
it's the same as the way you specified your commit message data.

   The last thing you need to do is to return the current mark so it
can be passed to the next iteration:

return mark

   NOTE: If you are running on Windows you'll need to make sure that
you add one extra step. As metioned before, Windows uses CRLF for new
line characters while git fast-import expects only LF. To get around
this problem and make git fast-import happy, you need to tell ruby to
use LF instead of CRLF:

$stdout.binmode

   That's it. If you run this script, you'll get content that looks
something like this:

$ ruby import.rb /opt/import_from
commit refs/heads/master
mark :1
committer Scott Chacon <schacon@geemail.com> 1230883200 -0700
data 29
imported from back_2009_01_02deleteall
M 644 inline file.rb
data 12
version two
commit refs/heads/master
mark :2
committer Scott Chacon <schacon@geemail.com> 1231056000 -0700
data 29
imported from back_2009_01_04from :1
deleteall
M 644 inline file.rb
data 14
version three
M 644 inline new.rb
data 16
new version one
(...)

   To run the importer, pipe this output through `git fast-import'
while in the Git directory you want to import into. You can create a
new directory and then run `git init' in it for a starting point, and
then run your script:

$ git init
Initialized empty Git repository in /opt/import_to/.git/
$ ruby import.rb /opt/import_from | git fast-import
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects:       5000
Total objects:           18 (         1 duplicates                  )
      blobs  :            7 (         1 duplicates          0 deltas)
      trees  :            6 (         0 duplicates          1 deltas)
      commits:            5 (         0 duplicates          0 deltas)
      tags   :            0 (         0 duplicates          0 deltas)
Total branches:           1 (         1 loads     )
      marks:           1024 (         5 unique    )
      atoms:              3
Memory total:          2255 KiB
       pools:          2098 KiB
     objects:           156 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize =   33554432
pack_report: core.packedGitLimit      =  268435456
pack_report: pack_used_ctr            =          9
pack_report: pack_mmap_calls          =          5
pack_report: pack_open_windows        =          1 /          1
pack_report: pack_mapped              =       1356 /       1356
---------------------------------------------------------------------

   As you can see, when it completes successfully, it gives you a bunch
of statistics about what it accomplished. In this case, you imported 18
objects total for 5 commits into 1 branch. Now, you can run `git log'
to see your new history:

$ git log -2
commit 10bfe7d22ce15ee25b60a824c8982157ca593d41
Author: Scott Chacon <schacon@example.com>
Date:   Sun May 3 12:57:39 2009 -0700

    imported from current

commit 7e519590de754d079dd73b44d695a42c9d2df452
Author: Scott Chacon <schacon@example.com>
Date:   Tue Feb 3 01:00:00 2009 -0700

    imported from back_2009_02_03

   There you go --- a nice, clean Git repository. It's important to
note that nothing is checked out --- you don't have any files in your
working directory at first. To get them, you must reset your branch to
where `master' is now:

$ ls
$ git reset --hard master
HEAD is now at 10bfe7d imported from current
$ ls
file.rb  lib

   You can do a lot more with the `fast-import' tool --- handle
different modes, binary data, multiple branches and merging, tags,
progress indicators, and more. A number of examples of more complex
scenarios are available in the `contrib/fast-import' directory of the
Git source code; one of the better ones is the `git-p4' script I just
covered.


File: progit.info,  Node: Summary,  Prev: Migrating to Git,  Up: Git and Other Systems

8.3 Summary
===========

You should feel comfortable using Git with Subversion or importing
nearly any existing repository into a new Git one without losing data.
The next chapter will cover the raw internals of Git so you can craft
every single byte, if need be.


File: progit.info,  Node: Git Internals,  Prev: Git and Other Systems,  Up: Top

9 Git Internals
***************

You may have skipped to this chapter from a previous chapter, or you
may have gotten here after reading the rest of the book --- in either
case, this is where you'll go over the inner workings and
implementation of Git. I found that learning this information was
fundamentally important to understanding how useful and powerful Git
is, but others have argued to me that it can be confusing and
unnecessarily complex for beginners. Thus, I've made this discussion
the last chapter in the book so you could read it early or later in
your learning process. I leave it up to you to decide.

   Now that you're here, let's get started. First, if it isn't
yet clear, Git is fundamentally a content-addressable filesystem with a
VCS user interface written on top of it. You'll learn more about what
this means in a bit.

   In the early days of Git (mostly pre 1.5), the user interface was
much more complex because it emphasized this filesystem rather than a
polished VCS. In the last few years, the UI has been refined until
it's as clean and easy to use as any system out there; but often, the
stereotype lingers about the early Git UI that was complex and
difficult to learn.

   The content-addressable filesystem layer is amazingly cool, so
I'll cover that first in this chapter; then, you'll learn about the
transport mechanisms and the repository maintenance tasks that you may
eventually have to deal with.

* Menu:

* Plumbing and Porcelain::
* Git Objects::
* Git References::
* Packfiles::
* The Refspec::
* Transfer Protocols::
* Maintenance and Data Recovery::
* Summary::


File: progit.info,  Node: Plumbing and Porcelain,  Next: Git Objects,  Up: Git Internals

9.1 Plumbing and Porcelain
==========================

This book covers how to use Git with 30 or so verbs such as `checkout',
`branch', `remote', and so on. But because Git was initially a toolkit
for a VCS rather than a full user-friendly VCS, it has a bunch of verbs
that do low-level work and were designed to be chained together UNIX
style or called from scripts. These commands are generally referred to
as "plumbing" commands, and the more user-friendly commands are called
"porcelain" commands.

   The book's first eight chapters deal almost exclusively with
porcelain commands. But in this chapter, you'll be dealing mostly
with the lower-level plumbing commands, because they give you access to
the inner workings of Git and help demonstrate how and why Git does
what it does. These commands aren't meant to be used manually on the
command line, but rather to be used as building blocks for new tools
and custom scripts.

   When you run `git init' in a new or existing directory, Git creates
the `.git' directory, which is where almost everything that Git stores
and manipulates is located. If you want to back up or clone your
repository, copying this single directory elsewhere gives you nearly
everything you need. This entire chapter basically deals with the stuff
in this directory. Here's what it looks like:

$ ls
HEAD
branches/
config
description
hooks/
index
info/
objects/
refs/

   You may see some other files in there, but this is a fresh `git
init' repository --- it's what you see by default. The `branches'
directory isn't used by newer Git versions, and the `description'
file is only used by the GitWeb program, so don't worry about those.
The `config' file contains your project-specific configuration options,
and the `info' directory keeps a global exclude file for ignored
patterns that you don't want to track in a .gitignore file. The
`hooks' directory contains your client- or server-side hook scripts,
which are discussed in detail in Chapter 6.

   This leaves four important entries: the `HEAD' and `index' files and
the `objects' and `refs' directories. These are the core parts of Git.
The `objects' directory stores all the content for your database, the
`refs' directory stores pointers into commit objects in that data
(branches), the `HEAD' file points to the branch you currently have
checked out, and the `index' file is where Git stores your staging area
information. You'll now look at each of these sections in detail to
see how Git operates.


File: progit.info,  Node: Git Objects,  Next: Git References,  Prev: Plumbing and Porcelain,  Up: Git Internals

9.2 Git Objects
===============

Git is a content-addressable filesystem. Great. What does that mean? It
means that at the core of Git is a simple key-value data store. You can
insert any kind of content into it, and it will give you back a key
that you can use to retrieve the content again at any time. To
demonstrate, you can use the plumbing command `hash-object', which
takes some data, stores it in your `.git' directory, and gives you back
the key the data is stored as. First, you initialize a new Git
repository and verify that there is nothing in the `objects' directory:

$ mkdir test
$ cd test
$ git init
Initialized empty Git repository in /tmp/test/.git/
$ find .git/objects
.git/objects
.git/objects/info
.git/objects/pack
$ find .git/objects -type f
$

   Git has initialized the `objects' directory and created `pack' and
`info' subdirectories in it, but there are no regular files. Now, store
some text in your Git database:

$ echo 'test content' | git hash-object -w --stdin
d670460b4b4aece5915caf5c68d12f560a9fe3e4

   The `-w' tells `hash-object' to store the object; otherwise, the
command simply tells you what the key would be. `--stdin' tells the
command to read the content from stdin; if you don't specify this,
`hash-object' expects the path to a file. The output from the command
is a 40-character checksum hash. This is the SHA-1 hash --- a checksum
of the content you're storing plus a header, which you'll learn
about in a bit. Now you can see how Git has stored your data:

$ find .git/objects -type f
.git/objects/d6/70460b4b4aece5915caf5c68d12f560a9fe3e4

   You can see a file in the `objects' directory. This is how Git
stores the content initially --- as a single file per piece of content,
named with the SHA-1 checksum of the content and its header. The
subdirectory is named with the first 2 characters of the SHA, and the
filename is the remaining 38 characters.

   You can pull the content back out of Git with the `cat-file'
command. This command is sort of a Swiss army knife for inspecting Git
objects. Passing `-p' to it instructs the `cat-file' command to figure
out the type of content and display it nicely for you:

$ git cat-file -p d670460b4b4aece5915caf5c68d12f560a9fe3e4
test content

   Now, you can add content to Git and pull it back out again. You can
also do this with content in files. For example, you can do some simple
version control on a file. First, create a new file and save its
contents in your database:

$ echo 'version 1' > test.txt
$ git hash-object -w test.txt
83baae61804e65cc73a7201a7252750c76066a30

   Then, write some new content to the file, and save it again:

$ echo 'version 2' > test.txt
$ git hash-object -w test.txt
1f7a7a472abf3dd9643fd615f6da379c4acb3e3a

   Your database contains the two new versions of the file as well as
the first content you stored there:

$ find .git/objects -type f
.git/objects/1f/7a7a472abf3dd9643fd615f6da379c4acb3e3a
.git/objects/83/baae61804e65cc73a7201a7252750c76066a30
.git/objects/d6/70460b4b4aece5915caf5c68d12f560a9fe3e4

   Now you can revert the file back to the first version

$ git cat-file -p 83baae61804e65cc73a7201a7252750c76066a30 > test.txt
$ cat test.txt
version 1

   or the second version:

$ git cat-file -p 1f7a7a472abf3dd9643fd615f6da379c4acb3e3a > test.txt
$ cat test.txt
version 2

   But remembering the SHA-1 key for each version of your file isn't
practical; plus, you aren't storing the filename in your system ---
just the content. This object type is called a blob. You can have Git
tell you the object type of any object in Git, given its SHA-1 key,
with `cat-file -t':

$ git cat-file -t 1f7a7a472abf3dd9643fd615f6da379c4acb3e3a
blob

* Menu:

* Tree Objects::
* Commit Objects::
* Object Storage::


File: progit.info,  Node: Tree Objects,  Next: Commit Objects,  Up: Git Objects

9.2.1 Tree Objects
------------------

The next type you'll look at is the tree object, which solves the
problem of storing the filename and also allows you to store a group of
files together. Git stores content in a manner similar to a UNIX
filesystem, but a bit simplified. All the content is stored as tree and
blob objects, with trees corresponding to UNIX directory entries and
blobs corresponding more or less to inodes or file contents. A single
tree object contains one or more tree entries, each of which contains a
SHA-1 pointer to a blob or subtree with its associated mode, type, and
filename. For example, the most recent tree in the simplegit project
may look something like this:

$ git cat-file -p master^{tree}
100644 blob a906cb2a4a904a152e80877d4088654daad0c859      README
100644 blob 8f94139338f9404f26296befa88755fc2598c289      Rakefile
040000 tree 99f1a6d12cb4b6f19c8655fca46c3ecf317074e0      lib

   The `master^{tree}' syntax specifies the tree object that is pointed
to by the last commit on your `master' branch. Notice that the `lib'
subdirectory isn't a blob but a pointer to another tree:

$ git cat-file -p 99f1a6d12cb4b6f19c8655fca46c3ecf317074e0
100644 blob 47c6340d6459e05787f644c2447d2595f5d3a54b      simplegit.rb

   Conceptually, the data that Git is storing is something like Figure
9-1.

[image src="../figures/18333fig0901-tn.png" alt="Figure 9-1. Simple version of the Git data model."]

Figure 9-1. Simple version of the Git data model.

   You can create your own tree. Git normally creates a tree by taking
the state of your staging area or index and writing a tree object from
it. So, to create a tree object, you first have to set up an index by
staging some files. To create an index with a single entry --- the
first version of your text.txt file --- you can use the plumbing
command `update-index'. You use this command to artificially add the
earlier version of the test.txt file to a new staging area. You must
pass it the `--add' option because the file doesn't yet exist in your
staging area (you don't even have a staging area set up yet) and
`--cacheinfo' because the file you're adding isn't in your
directory but is in your database. Then, you specify the mode, SHA-1,
and filename:

$ git update-index --add --cacheinfo 100644 \
  83baae61804e65cc73a7201a7252750c76066a30 test.txt

   In this case, you're specifying a mode of `100644', which means
it's a normal file. Other options are `100755', which means it's an
executable file; and `120000', which specifies a symbolic link. The
mode is taken from normal UNIX modes but is much less flexible ---
these three modes are the only ones that are valid for files (blobs) in
Git (although other modes are used for directories and submodules).

   Now, you can use the `write-tree' command to write the staging area
out to a tree object. No `-w' option is needed --- calling `write-tree'
automatically creates a tree object from the state of the index if that
tree doesn't yet exist:

$ git write-tree
d8329fc1cc938780ffdd9f94e0d364e0ea74f579
$ git cat-file -p d8329fc1cc938780ffdd9f94e0d364e0ea74f579
100644 blob 83baae61804e65cc73a7201a7252750c76066a30      test.txt

   You can also verify that this is a tree object:

$ git cat-file -t d8329fc1cc938780ffdd9f94e0d364e0ea74f579
tree

   You'll now create a new tree with the second version of test.txt
and a new file as well:

$ echo 'new file' > new.txt
$ git update-index test.txt
$ git update-index --add new.txt

   Your staging area now has the new version of test.txt as well as the
new file new.txt. Write out that tree (recording the state of the
staging area or index to a tree object) and see what it looks like:

$ git write-tree
0155eb4229851634a0f03eb265b69f5a2d56f341
$ git cat-file -p 0155eb4229851634a0f03eb265b69f5a2d56f341
100644 blob fa49b077972391ad58037050f2a75f74e3671e92      new.txt
100644 blob 1f7a7a472abf3dd9643fd615f6da379c4acb3e3a      test.txt

   Notice that this tree has both file entries and also that the
test.txt SHA is the "version 2" SHA from earlier (`1f7a7a'). Just for
fun, you'll add the first tree as a subdirectory into this one. You
can read trees into your staging area by calling `read-tree'. In this
case, you can read an existing tree into your staging area as a subtree
by using the `--prefix' option to `read-tree':

$ git read-tree --prefix=bak d8329fc1cc938780ffdd9f94e0d364e0ea74f579
$ git write-tree
3c4e9cd789d88d8d89c1073707c3585e41b0e614
$ git cat-file -p 3c4e9cd789d88d8d89c1073707c3585e41b0e614
040000 tree d8329fc1cc938780ffdd9f94e0d364e0ea74f579      bak
100644 blob fa49b077972391ad58037050f2a75f74e3671e92      new.txt
100644 blob 1f7a7a472abf3dd9643fd615f6da379c4acb3e3a      test.txt

   If you created a working directory from the new tree you just wrote,
you would get the two files in the top level of the working directory
and a subdirectory named `bak' that contained the first version of the
test.txt file. You can think of the data that Git contains for these
structures as being like Figure 9-2.

[image src="../figures/18333fig0902-tn.png" alt="Figure 9-2. The content structure of your current Git data."]

Figure 9-2. The content structure of your current Git data.


File: progit.info,  Node: Commit Objects,  Next: Object Storage,  Prev: Tree Objects,  Up: Git Objects

9.2.2 Commit Objects
--------------------

You have three trees that specify the different snapshots of your
project that you want to track, but the earlier problem remains: you
must remember all three SHA-1 values in order to recall the snapshots.
You also don't have any information about who saved the snapshots,
when they were saved, or why they were saved. This is the basic
information that the commit object stores for you.

   To create a commit object, you call `commit-tree' and specify a
single tree SHA-1 and which commit objects, if any, directly preceded
it. Start with the first tree you wrote:

$ echo 'first commit' | git commit-tree d8329f
fdf4fc3344e67ab068f836878b6c4951e3b15f3d

   Now you can look at your new commit object with `cat-file':

$ git cat-file -p fdf4fc3
tree d8329fc1cc938780ffdd9f94e0d364e0ea74f579
author Scott Chacon <schacon@gmail.com> 1243040974 -0700
committer Scott Chacon <schacon@gmail.com> 1243040974 -0700

first commit

   The format for a commit object is simple: it specifies the top-level
tree for the snapshot of the project at that point; the
author/committer information pulled from your `user.name' and
`user.email' configuration settings, with the current timestamp; a
blank line, and then the commit message.

   Next, you'll write the other two commit objects, each referencing
the commit that came directly before it:

$ echo 'second commit' | git commit-tree 0155eb -p fdf4fc3
cac0cab538b970a37ea1e769cbbde608743bc96d
$ echo 'third commit'  | git commit-tree 3c4e9c -p cac0cab
1a410efbd13591db07496601ebc7a059dd55cfe9

   Each of the three commit objects points to one of the three snapshot
trees you created. Oddly enough, you have a real Git history now that
you can view with the `git log' command, if you run it on the last
commit SHA-1:

$ git log --stat 1a410e
commit 1a410efbd13591db07496601ebc7a059dd55cfe9
Author: Scott Chacon <schacon@gmail.com>
Date:   Fri May 22 18:15:24 2009 -0700

    third commit

 bak/test.txt |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

commit cac0cab538b970a37ea1e769cbbde608743bc96d
Author: Scott Chacon <schacon@gmail.com>
Date:   Fri May 22 18:14:29 2009 -0700

    second commit

 new.txt  |    1 +
 test.txt |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

commit fdf4fc3344e67ab068f836878b6c4951e3b15f3d
Author: Scott Chacon <schacon@gmail.com>
Date:   Fri May 22 18:09:34 2009 -0700

    first commit

 test.txt |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

   Amazing. You've just done the low-level operations to build up a
Git history without using any of the front ends. This is essentially
what Git does when you run the `git add' and `git commit' commands ---
it stores blobs for the files that have changed, updates the index,
writes out trees, and writes commit objects that reference the
top-level trees and the commits that came immediately before them.
These three main Git objects --- the blob, the tree, and the commit ---
are initially stored as separate files in your `.git/objects'
directory. Here are all the objects in the example directory now,
commented with what they store:

$ find .git/objects -type f
.git/objects/01/55eb4229851634a0f03eb265b69f5a2d56f341 # tree 2
.git/objects/1a/410efbd13591db07496601ebc7a059dd55cfe9 # commit 3
.git/objects/1f/7a7a472abf3dd9643fd615f6da379c4acb3e3a # test.txt v2
.git/objects/3c/4e9cd789d88d8d89c1073707c3585e41b0e614 # tree 3
.git/objects/83/baae61804e65cc73a7201a7252750c76066a30 # test.txt v1
.git/objects/ca/c0cab538b970a37ea1e769cbbde608743bc96d # commit 2
.git/objects/d6/70460b4b4aece5915caf5c68d12f560a9fe3e4 # 'test content'
.git/objects/d8/329fc1cc938780ffdd9f94e0d364e0ea74f579 # tree 1
.git/objects/fa/49b077972391ad58037050f2a75f74e3671e92 # new.txt
.git/objects/fd/f4fc3344e67ab068f836878b6c4951e3b15f3d # commit 1

   If you follow all the internal pointers, you get an object graph
something like Figure 9-3.

[image src="../figures/18333fig0903-tn.png" alt="Figure 9-3. All the objects in your Git directory."]

Figure 9-3. All the objects in your Git directory.


File: progit.info,  Node: Object Storage,  Prev: Commit Objects,  Up: Git Objects

9.2.3 Object Storage
--------------------

I mentioned earlier that a header is stored with the content. Let's
take a minute to look at how Git stores its objects. You'll see how
to store a blob object --- in this case, the string "what is up, doc?"
--- interactively in the Ruby scripting language. You can start up
interactive Ruby mode with the `irb' command:

$ irb
>> content = "what is up, doc?"
=> "what is up, doc?"

   Git constructs a header that starts with the type of the object, in
this case a blob. Then, it adds a space followed by the size of the
content and finally a null byte:

>> header = "blob #{content.length}\0"
=> "blob 16\000"

   Git concatenates the header and the original content and then
calculates the SHA-1 checksum of that new content. You can calculate
the SHA-1 value of a string in Ruby by including the SHA1 digest
library with the `require' command and then calling
`Digest::SHA1.hexdigest()' with the string:

>> store = header + content
=> "blob 16\000what is up, doc?"
>> require 'digest/sha1'
=> true
>> sha1 = Digest::SHA1.hexdigest(store)
=> "bd9dbf5aae1a3862dd1526723246b20206e5fc37"

   Git compresses the new content with zlib, which you can do in Ruby
with the zlib library. First, you need to require the library and then
run `Zlib::Deflate.deflate()' on the content:

>> require 'zlib'
=> true
>> zlib_content = Zlib::Deflate.deflate(store)
=> "x\234K\312\311OR04c(\317H,Q\310,V(-\320QH\311O\266\a\000_\034\a\235"

   Finally, you'll write your zlib-deflated content to an object on
disk. You'll determine the path of the object you want to write out
(the first two characters of the SHA-1 value being the subdirectory
name, and the last 38 characters being the filename within that
directory). In Ruby, you can use the `FileUtils.mkdir_p()' function to
create the subdirectory if it doesn't exist. Then, open the file with
`File.open()' and write out the previously zlib-compressed content to
the file with a `write()' call on the resulting file handle:

>> path = '.git/objects/' + sha1[0,2] + '/' + sha1[2,38]
=> ".git/objects/bd/9dbf5aae1a3862dd1526723246b20206e5fc37"
>> require 'fileutils'
=> true
>> FileUtils.mkdir_p(File.dirname(path))
=> ".git/objects/bd"
>> File.open(path, 'w') { |f| f.write zlib_content }
=> 32

   That's it --- you've created a valid Git blob object. All Git
objects are stored the same way, just with different types --- instead
of the string blob, the header will begin with commit or tree. Also,
although the blob content can be nearly anything, the commit and tree
content are very specifically formatted.


File: progit.info,  Node: Git References,  Next: Packfiles,  Prev: Git Objects,  Up: Git Internals

9.3 Git References
==================

You can run something like `git log 1a410e' to look through your whole
history, but you still have to remember that `1a410e' is the last
commit in order to walk that history to find all those objects. You
need a file in which you can store the SHA-1 value under a simple name
so you can use that pointer rather than the raw SHA-1 value.

   In Git, these are called "references" or "refs"; you can find the
files that contain the SHA-1 values in the `.git/refs' directory. In
the current project, this directory contains no files, but it does
contain a simple structure:

$ find .git/refs
.git/refs
.git/refs/heads
.git/refs/tags
$ find .git/refs -type f
$

   To create a new reference that will help you remember where your
latest commit is, you can technically do something as simple as this:

$ echo "1a410efbd13591db07496601ebc7a059dd55cfe9" > .git/refs/heads/master

   Now, you can use the head reference you just created instead of the
SHA-1 value in your Git commands:

$ git log --pretty=oneline  master
1a410efbd13591db07496601ebc7a059dd55cfe9 third commit
cac0cab538b970a37ea1e769cbbde608743bc96d second commit
fdf4fc3344e67ab068f836878b6c4951e3b15f3d first commit

   You aren't encouraged to directly edit the reference files. Git
provides a safer command to do this if you want to update a reference
called `update-ref':

$ git update-ref refs/heads/master 1a410efbd13591db07496601ebc7a059dd55cfe9

   That's basically what a branch in Git is: a simple pointer or
reference to the head of a line of work. To create a branch back at the
second commit, you can do this:

$ git update-ref refs/heads/test cac0ca

   Your branch will contain only work from that commit down:

$ git log --pretty=oneline test
cac0cab538b970a37ea1e769cbbde608743bc96d second commit
fdf4fc3344e67ab068f836878b6c4951e3b15f3d first commit

   Now, your Git database conceptually looks something like Figure 9-4.

[image src="../figures/18333fig0904-tn.png" alt="Figure 9-4. Git directory objects with branch head references included."]

Figure 9-4. Git directory objects with branch head references included.

   When you run commands like `git branch (branchname)', Git basically
runs that `update-ref' command to add the SHA-1 of the last commit of
the branch you're on into whatever new reference you want to create.

* Menu:

* The HEAD::
* Tags::
* Remotes::


File: progit.info,  Node: The HEAD,  Next: Tags,  Up: Git References

9.3.1 The HEAD
--------------

The question now is, when you run `git branch (branchname)', how does
Git know the SHA-1 of the last commit? The answer is the HEAD file. The
HEAD file is a symbolic reference to the branch you're currently on.
By symbolic reference, I mean that unlike a normal reference, it
doesn't generally contain a SHA-1 value but rather a pointer to
another reference. If you look at the file, you'll normally see
something like this:

$ cat .git/HEAD
ref: refs/heads/master

   If you run `git checkout test', Git updates the file to look like
this:

$ cat .git/HEAD
ref: refs/heads/test

   When you run `git commit', it creates the commit object, specifying
the parent of that commit object to be whatever SHA-1 value the
reference in HEAD points to.

   You can also manually edit this file, but again a safer command
exists to do so: `symbolic-ref'. You can read the value of your HEAD
via this command:

$ git symbolic-ref HEAD
refs/heads/master

   You can also set the value of HEAD:

$ git symbolic-ref HEAD refs/heads/test
$ cat .git/HEAD
ref: refs/heads/test

   You can't set a symbolic reference outside of the refs style:

$ git symbolic-ref HEAD test
fatal: Refusing to point HEAD outside of refs/


File: progit.info,  Node: Tags,  Next: Remotes,  Prev: The HEAD,  Up: Git References

9.3.2 Tags
----------

You've just gone over Git's three main object types, but there is a
fourth. The tag object is very much like a commit object --- it
contains a tagger, a date, a message, and a pointer. The main
difference is that a tag object points to a commit rather than a tree.
It's like a branch reference, but it never moves --- it always points
to the same commit but gives it a friendlier name.

   As discussed in Chapter 2, there are two types of tags: annotated
and lightweight. You can make a lightweight tag by running something
like this:

$ git update-ref refs/tags/v1.0 cac0cab538b970a37ea1e769cbbde608743bc96d

   That is all a lightweight tag is --- a branch that never moves. An
annotated tag is more complex, however. If you create an annotated tag,
Git creates a tag object and then writes a reference to point to it
rather than directly to the commit. You can see this by creating an
annotated tag (`-a' specifies that it's an annotated tag):

$ git tag -a v1.1 1a410efbd13591db07496601ebc7a059dd55cfe9 -m 'test tag'

   Here's the object SHA-1 value it created:

$ cat .git/refs/tags/v1.1
9585191f37f7b0fb9444f35a9bf50de191beadc2

   Now, run the `cat-file' command on that SHA-1 value:

$ git cat-file -p 9585191f37f7b0fb9444f35a9bf50de191beadc2
object 1a410efbd13591db07496601ebc7a059dd55cfe9
type commit
tag v1.1
tagger Scott Chacon <schacon@gmail.com> Sat May 23 16:48:58 2009 -0700

test tag

   Notice that the object entry points to the commit SHA-1 value that
you tagged. Also notice that it doesn't need to point to a commit;
you can tag any Git object. In the Git source code, for example, the
maintainer has added their GPG public key as a blob object and then
tagged it. You can view the public key by running

$ git cat-file blob junio-gpg-pub

   in the Git source code repository. The Linux kernel repository also
has a non-commit-pointing tag object --- the first tag created points
to the initial tree of the import of the source code.


File: progit.info,  Node: Remotes,  Prev: Tags,  Up: Git References

9.3.3 Remotes
-------------

The third type of reference that you'll see is a remote reference. If
you add a remote and push to it, Git stores the value you last pushed
to that remote for each branch in the `refs/remotes' directory. For
instance, you can add a remote called `origin' and push your `master'
branch to it:

$ git remote add origin git@github.com:schacon/simplegit-progit.git
$ git push origin master
Counting objects: 11, done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (7/7), 716 bytes, done.
Total 7 (delta 2), reused 4 (delta 1)
To git@github.com:schacon/simplegit-progit.git
   a11bef0..ca82a6d  master -> master

   Then, you can see what the `master' branch on the `origin' remote
was the last time you communicated with the server, by checking the
`refs/remotes/origin/master' file:

$ cat .git/refs/remotes/origin/master
ca82a6dff817ec66f44342007202690a93763949

   Remote references differ from branches (`refs/heads' references)
mainly in that they can't be checked out. Git moves them around as
bookmarks to the last known state of where those branches were on those
servers.


File: progit.info,  Node: Packfiles,  Next: The Refspec,  Prev: Git References,  Up: Git Internals

9.4 Packfiles
=============

Let's go back to the objects database for your test Git repository.
At this point, you have 11 objects --- 4 blobs, 3 trees, 3 commits, and
1 tag:

$ find .git/objects -type f
.git/objects/01/55eb4229851634a0f03eb265b69f5a2d56f341 # tree 2
.git/objects/1a/410efbd13591db07496601ebc7a059dd55cfe9 # commit 3
.git/objects/1f/7a7a472abf3dd9643fd615f6da379c4acb3e3a # test.txt v2
.git/objects/3c/4e9cd789d88d8d89c1073707c3585e41b0e614 # tree 3
.git/objects/83/baae61804e65cc73a7201a7252750c76066a30 # test.txt v1
.git/objects/95/85191f37f7b0fb9444f35a9bf50de191beadc2 # tag
.git/objects/ca/c0cab538b970a37ea1e769cbbde608743bc96d # commit 2
.git/objects/d6/70460b4b4aece5915caf5c68d12f560a9fe3e4 # 'test content'
.git/objects/d8/329fc1cc938780ffdd9f94e0d364e0ea74f579 # tree 1
.git/objects/fa/49b077972391ad58037050f2a75f74e3671e92 # new.txt
.git/objects/fd/f4fc3344e67ab068f836878b6c4951e3b15f3d # commit 1

   Git compresses the contents of these files with zlib, and you're
not storing much, so all these files collectively take up only 925
bytes. You'll add some larger content to the repository to
demonstrate an interesting feature of Git. Add the repo.rb file from
the Grit library you worked with earlier --- this is about a 12K source
code file:

$ curl http://github.com/mojombo/grit/raw/master/lib/grit/repo.rb > repo.rb
$ git add repo.rb
$ git commit -m 'added repo.rb'
[master 484a592] added repo.rb
 3 files changed, 459 insertions(+), 2 deletions(-)
 delete mode 100644 bak/test.txt
 create mode 100644 repo.rb
 rewrite test.txt (100%)

   If you look at the resulting tree, you can see the SHA-1 value your
repo.rb file got for the blob object:

$ git cat-file -p master^{tree}
100644 blob fa49b077972391ad58037050f2a75f74e3671e92      new.txt
100644 blob 9bc1dc421dcd51b4ac296e3e5b6e2a99cf44391e      repo.rb
100644 blob e3f094f522629ae358806b17daf78246c27c007b      test.txt

   You can then use `git cat-file' to see how big that object is:

$ git cat-file -s 9bc1dc421dcd51b4ac296e3e5b6e2a99cf44391e
12898

   Now, modify that file a little, and see what happens:

$ echo '# testing' >> repo.rb
$ git commit -am 'modified repo a bit'
[master ab1afef] modified repo a bit
 1 files changed, 1 insertions(+), 0 deletions(-)

   Check the tree created by that commit, and you see something
interesting:

$ git cat-file -p master^{tree}
100644 blob fa49b077972391ad58037050f2a75f74e3671e92      new.txt
100644 blob 05408d195263d853f09dca71d55116663690c27c      repo.rb
100644 blob e3f094f522629ae358806b17daf78246c27c007b      test.txt

   The blob is now a different blob, which means that although you
added only a single line to the end of a 400-line file, Git stored that
new content as a completely new object:

$ git cat-file -s 05408d195263d853f09dca71d55116663690c27c
12908

   You have two nearly identical 12K objects on your disk. Wouldn't
it be nice if Git could store one of them in full but then the second
object only as the delta between it and the first?

   It turns out that it can. The initial format in which Git saves
objects on disk is called a loose object format. However, occasionally
Git packs up several of these objects into a single binary file called
a packfile in order to save space and be more efficient. Git does this
if you have too many loose objects around, if you run the `git gc'
command manually, or if you push to a remote server. To see what
happens, you can manually ask Git to pack up the objects by calling the
`git gc' command:

$ git gc
Counting objects: 17, done.
Delta compression using 2 threads.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (17/17), done.
Total 17 (delta 1), reused 10 (delta 0)

   If you look in your objects directory, you'll find that most of
your objects are gone, and a new pair of files has appeared:

$ find .git/objects -type f
.git/objects/71/08f7ecb345ee9d0084193f147cdad4d2998293
.git/objects/d6/70460b4b4aece5915caf5c68d12f560a9fe3e4
.git/objects/info/packs
.git/objects/pack/pack-7a16e4488ae40c7d2bc56ea2bd43e25212a66c45.idx
.git/objects/pack/pack-7a16e4488ae40c7d2bc56ea2bd43e25212a66c45.pack

   The objects that remain are the blobs that aren't pointed to by
any commit --- in this case, the "what is up, doc?" example and the
"test content" example blobs you created earlier. Because you never
added them to any commits, they're considered dangling and aren't
packed up in your new packfile.

   The other files are your new packfile and an index. The packfile is
a single file containing the contents of all the objects that were
removed from your filesystem. The index is a file that contains offsets
into that packfile so you can quickly seek to a specific object. What
is cool is that although the objects on disk before you ran the `gc'
were collectively about 12K in size, the new packfile is only 6K.
You've halved your disk usage by packing your objects.

   How does Git do this? When Git packs objects, it looks for files
that are named and sized similarly, and stores just the deltas from one
version of the file to the next. You can look into the packfile and see
what Git did to save space. The `git verify-pack' plumbing command
allows you to see what was packed up:

$ git verify-pack -v \
  .git/objects/pack/pack-7a16e4488ae40c7d2bc56ea2bd43e25212a66c45.idx
0155eb4229851634a0f03eb265b69f5a2d56f341 tree   71 76 5400
05408d195263d853f09dca71d55116663690c27c blob   12908 3478 874
09f01cea547666f58d6a8d809583841a7c6f0130 tree   106 107 5086
1a410efbd13591db07496601ebc7a059dd55cfe9 commit 225 151 322
1f7a7a472abf3dd9643fd615f6da379c4acb3e3a blob   10 19 5381
3c4e9cd789d88d8d89c1073707c3585e41b0e614 tree   101 105 5211
484a59275031909e19aadb7c92262719cfcdf19a commit 226 153 169
83baae61804e65cc73a7201a7252750c76066a30 blob   10 19 5362
9585191f37f7b0fb9444f35a9bf50de191beadc2 tag    136 127 5476
9bc1dc421dcd51b4ac296e3e5b6e2a99cf44391e blob   7 18 5193 1
05408d195263d853f09dca71d55116663690c27c \
  ab1afef80fac8e34258ff41fc1b867c702daa24b commit 232 157 12
cac0cab538b970a37ea1e769cbbde608743bc96d commit 226 154 473
d8329fc1cc938780ffdd9f94e0d364e0ea74f579 tree   36 46 5316
e3f094f522629ae358806b17daf78246c27c007b blob   1486 734 4352
f8f51d7d8a1760462eca26eebafde32087499533 tree   106 107 749
fa49b077972391ad58037050f2a75f74e3671e92 blob   9 18 856
fdf4fc3344e67ab068f836878b6c4951e3b15f3d commit 177 122 627
chain length = 1: 1 object
pack-7a16e4488ae40c7d2bc56ea2bd43e25212a66c45.pack: ok

   Here, the `9bc1d' blob, which if you remember was the first version
of your repo.rb file, is referencing the `05408' blob, which was the
second version of the file. The third column in the output is the size
of the object in the pack, so you can see that `05408' takes up 12K of
the file but that `9bc1d' only takes up 7 bytes. What is also
interesting is that the second version of the file is the one that is
stored intact, whereas the original version is stored as a delta ---
this is because you're most likely to need faster access to the most
recent version of the file.

   The really nice thing about this is that it can be repacked at any
time. Git will occasionally repack your database automatically, always
trying to save more space. You can also manually repack at any time by
running `git gc' by hand.


File: progit.info,  Node: The Refspec,  Next: Transfer Protocols,  Prev: Packfiles,  Up: Git Internals

9.5 The Refspec
===============

Throughout this book, you've used simple mappings from remote
branches to local references; but they can be more complex. Suppose you
add a remote like this:

$ git remote add origin git@github.com:schacon/simplegit-progit.git

   It adds a section to your `.git/config' file, specifying the name of
the remote (`origin'), the URL of the remote repository, and the
refspec for fetching:

[remote "origin"]
       url = git@github.com:schacon/simplegit-progit.git
       fetch = +refs/heads/*:refs/remotes/origin/*

   The format of the refspec is an optional `+', followed by
`<src>:<dst>', where `<src>' is the pattern for references on the
remote side and `<dst>' is where those references will be written
locally. The `+' tells Git to update the reference even if it isn't a
fast-forward.

   In the default case that is automatically written by a `git remote
add' command, Git fetches all the references under `refs/heads/' on the
server and writes them to `refs/remotes/origin/' locally. So, if there
is a `master' branch on the server, you can access the log of that
branch locally via

$ git log origin/master
$ git log remotes/origin/master
$ git log refs/remotes/origin/master

   They're all equivalent, because Git expands each of them to
`refs/remotes/origin/master'.

   If you want Git instead to pull down only the `master' branch each
time, and not every other branch on the remote server, you can change
the fetch line to

fetch = +refs/heads/master:refs/remotes/origin/master

   This is just the default refspec for `git fetch' for that remote. If
you want to do something one time, you can specify the refspec on the
command line, too. To pull the `master' branch on the remote down to
`origin/mymaster' locally, you can run

$ git fetch origin master:refs/remotes/origin/mymaster

   You can also specify multiple refspecs. On the command line, you can
pull down several branches like so:

$ git fetch origin master:refs/remotes/origin/mymaster \
   topic:refs/remotes/origin/topic
From git@github.com:schacon/simplegit
 ! [rejected]        master     -> origin/mymaster  (non fast forward)
 * [new branch]      topic      -> origin/topic

   In this case, the master branch pull was rejected because it
wasn't a fast-forward reference. You can override that by specifying
the `+' in front of the refspec.

   You can also specify multiple refspecs for fetching in your
configuration file. If you want to always fetch the master and
experiment branches, add two lines:

[remote "origin"]
       url = git@github.com:schacon/simplegit-progit.git
       fetch = +refs/heads/master:refs/remotes/origin/master
       fetch = +refs/heads/experiment:refs/remotes/origin/experiment

   You can't use partial globs in the pattern, so this would be
invalid:

fetch = +refs/heads/qa*:refs/remotes/origin/qa*

   However, you can use namespacing to accomplish something like that.
If you have a QA team that pushes a series of branches, and you want to
get the master branch and any of the QA team's branches but nothing
else, you can use a config section like this:

[remote "origin"]
       url = git@github.com:schacon/simplegit-progit.git
       fetch = +refs/heads/master:refs/remotes/origin/master
       fetch = +refs/heads/qa/*:refs/remotes/origin/qa/*

   If you have a complex workflow process that has a QA team pushing
branches, developers pushing branches, and integration teams pushing
and collaborating on remote branches, you can namespace them easily
this way.

* Menu:

* Pushing Refspecs::
* Deleting References::


File: progit.info,  Node: Pushing Refspecs,  Next: Deleting References,  Up: The Refspec

9.5.1 Pushing Refspecs
----------------------

It's nice that you can fetch namespaced references that way, but how
does the QA team get their branches into a `qa/' namespace in the first
place? You accomplish that by using refspecs to push.

   If the QA team wants to push their `master' branch to `qa/master' on
the remote server, they can run

$ git push origin master:refs/heads/qa/master

   If they want Git to do that automatically each time they run `git
push origin', they can add a `push' value to their config file:

[remote "origin"]
       url = git@github.com:schacon/simplegit-progit.git
       fetch = +refs/heads/*:refs/remotes/origin/*
       push = refs/heads/master:refs/heads/qa/master

   Again, this will cause a `git push origin' to push the local
`master' branch to the remote `qa/master' branch by default.


File: progit.info,  Node: Deleting References,  Prev: Pushing Refspecs,  Up: The Refspec

9.5.2 Deleting References
-------------------------

You can also use the refspec to delete references from the remote
server by running something like this:

$ git push origin :topic

   Because the refspec is `<src>:<dst>', by leaving off the `<src>'
part, this basically says to make the topic branch on the remote
nothing, which deletes it.


File: progit.info,  Node: Transfer Protocols,  Next: Maintenance and Data Recovery,  Prev: The Refspec,  Up: Git Internals

9.6 Transfer Protocols
======================

Git can transfer data between two repositories in two major ways: over
HTTP and via the so-called smart protocols used in the `file://',
`ssh://', and `git://' transports. This section will quickly cover how
these two main protocols operate.

* Menu:

* The Dumb Protocol::
* The Smart Protocol::


File: progit.info,  Node: The Dumb Protocol,  Next: The Smart Protocol,  Up: Transfer Protocols

9.6.1 The Dumb Protocol
-----------------------

Git transport over HTTP is often referred to as the dumb protocol
because it requires no Git-specific code on the server side during the
transport process. The fetch process is a series of GET requests, where
the client can assume the layout of the Git repository on the server.
Let's follow the `http-fetch' process for the simplegit library:

$ git clone http://github.com/schacon/simplegit-progit.git

   The first thing this command does is pull down the `info/refs' file.
This file is written by the `update-server-info' command, which is why
you need to enable that as a `post-receive' hook in order for the HTTP
transport to work properly:

=> GET info/refs
ca82a6dff817ec66f44342007202690a93763949     refs/heads/master

   Now you have a list of the remote references and SHAs. Next, you
look for what the HEAD reference is so you know what to check out when
you're finished:

=> GET HEAD
ref: refs/heads/master

   You need to check out the `master' branch when you've completed
the process. At this point, you're ready to start the walking
process. Because your starting point is the `ca82a6' commit object you
saw in the `info/refs' file, you start by fetching that:

=> GET objects/ca/82a6dff817ec66f44342007202690a93763949
(179 bytes of binary data)

   You get an object back --- that object is in loose format on the
server, and you fetched it over a static HTTP GET request. You can
zlib-uncompress it, strip off the header, and look at the commit
content:

$ git cat-file -p ca82a6dff817ec66f44342007202690a93763949
tree cfda3bf379e4f8dba8717dee55aab78aef7f4daf
parent 085bb3bcb608e1e8451d4b2432f8ecbe6306e7e7
author Scott Chacon <schacon@gmail.com> 1205815931 -0700
committer Scott Chacon <schacon@gmail.com> 1240030591 -0700

changed the version number

   Next, you have two more objects to retrieve --- `cfda3b', which is
the tree of content that the commit we just retrieved points to; and
`085bb3', which is the parent commit:

=> GET objects/08/5bb3bcb608e1e8451d4b2432f8ecbe6306e7e7
(179 bytes of data)

   That gives you your next commit object. Grab the tree object:

=> GET objects/cf/da3bf379e4f8dba8717dee55aab78aef7f4daf
(404 - Not Found)

   Oops --- it looks like that tree object isn't in loose format on
the server, so you get a 404 response back. There are a couple of
reasons for this --- the object could be in an alternate repository, or
it could be in a packfile in this repository. Git checks for any listed
alternates first:

=> GET objects/info/http-alternates
(empty file)

   If this comes back with a list of alternate URLs, Git checks for
loose files and packfiles there --- this is a nice mechanism for
projects that are forks of one another to share objects on disk.
However, because no alternates are listed in this case, your object
must be in a packfile. To see what packfiles are available on this
server, you need to get the `objects/info/packs' file, which contains a
listing of them (also generated by `update-server-info'):

=> GET objects/info/packs
P pack-816a9b2334da9953e530f27bcac22082a9f5b835.pack

   There is only one packfile on the server, so your object is
obviously in there, but you'll check the index file to make sure.
This is also useful if you have multiple packfiles on the server, so
you can see which packfile contains the object you need:

=> GET objects/pack/pack-816a9b2334da9953e530f27bcac22082a9f5b835.idx
(4k of binary data)

   Now that you have the packfile index, you can see if your object is
in it --- because the index lists the SHAs of the objects contained in
the packfile and the offsets to those objects. Your object is there, so
go ahead and get the whole packfile:

=> GET objects/pack/pack-816a9b2334da9953e530f27bcac22082a9f5b835.pack
(13k of binary data)

   You have your tree object, so you continue walking your commits.
They're all also within the packfile you just downloaded, so you
don't have to do any more requests to your server. Git checks out a
working copy of the `master' branch that was pointed to by the HEAD
reference you downloaded at the beginning.

   The entire output of this process looks like this:

$ git clone http://github.com/schacon/simplegit-progit.git
Initialized empty Git repository in /private/tmp/simplegit-progit/.git/
got ca82a6dff817ec66f44342007202690a93763949
walk ca82a6dff817ec66f44342007202690a93763949
got 085bb3bcb608e1e8451d4b2432f8ecbe6306e7e7
Getting alternates list for http://github.com/schacon/simplegit-progit.git
Getting pack list for http://github.com/schacon/simplegit-progit.git
Getting index for pack 816a9b2334da9953e530f27bcac22082a9f5b835
Getting pack 816a9b2334da9953e530f27bcac22082a9f5b835
 which contains cfda3bf379e4f8dba8717dee55aab78aef7f4daf
walk 085bb3bcb608e1e8451d4b2432f8ecbe6306e7e7
walk a11bef06a3f659402fe7563abf99ad00de2209e6


File: progit.info,  Node: The Smart Protocol,  Prev: The Dumb Protocol,  Up: Transfer Protocols

9.6.2 The Smart Protocol
------------------------

The HTTP method is simple but a bit inefficient. Using smart protocols
is a more common method of transferring data. These protocols have a
process on the remote end that is intelligent about Git --- it can read
local data and figure out what the client has or needs and generate
custom data for it. There are two sets of processes for transferring
data: a pair for uploading data and a pair for downloading data.

* Menu:

* Uploading Data::
* Downloading Data::


File: progit.info,  Node: Uploading Data,  Next: Downloading Data,  Up: The Smart Protocol

9.6.2.1 Uploading Data
......................

To upload data to a remote process, Git uses the `send-pack' and
`receive-pack' processes. The `send-pack' process runs on the client
and connects to a `receive-pack' process on the remote side.

   For example, say you run `git push origin master' in your project,
and `origin' is defined as a URL that uses the SSH protocol. Git fires
up the `send-pack' process, which initiates a connection over SSH to
your server. It tries to run a command on the remote server via an SSH
call that looks something like this:

$ ssh -x git@github.com "git-receive-pack 'schacon/simplegit-progit.git'"
005bca82a6dff817ec66f4437202690a93763949 refs/heads/master report-status delete-refs
003e085bb3bcb608e1e84b2432f8ecbe6306e7e7 refs/heads/topic
0000

   The `git-receive-pack' command immediately responds with one line
for each reference it currently has --- in this case, just the `master'
branch and its SHA. The first line also has a list of the server's
capabilities (here, `report-status' and `delete-refs').

   Each line starts with a 4-byte hex value specifying how long the
rest of the line is. Your first line starts with 005b, which is 91 in
hex, meaning that 91 bytes remain on that line. The next line starts
with 003e, which is 62, so you read the remaining 62 bytes. The next
line is 0000, meaning the server is done with its references listing.

   Now that it knows the server's state, your `send-pack' process
determines what commits it has that the server doesn't. For each
reference that this push will update, the `send-pack' process tells the
`receive-pack' process that information. For instance, if you're
updating the `master' branch and adding an `experiment' branch, the
`send-pack' response may look something like this:

0085ca82a6dff817ec66f44342007202690a93763949  15027957951b64cf874c3557a0f3547bd83b3ff6 refs/heads/master report-status
00670000000000000000000000000000000000000000 cdfdb42577e2506715f8cfeacdbabc092bf63e8d refs/heads/experiment
0000

   The SHA-1 value of all '0's means that nothing was there before ---
because you're adding the experiment reference. If you were deleting
a reference, you would see the opposite: all '0's on the right side.

   Git sends a line for each reference you're updating with the old
SHA, the new SHA, and the reference that is being updated. The first
line also has the client's capabilities. Next, the client uploads a
packfile of all the objects the server doesn't have yet. Finally, the
server responds with a success (or failure) indication:

000Aunpack ok


File: progit.info,  Node: Downloading Data,  Prev: Uploading Data,  Up: The Smart Protocol

9.6.2.2 Downloading Data
........................

When you download data, the `fetch-pack' and `upload-pack' processes
are involved. The client initiates a `fetch-pack' process that connects
to an `upload-pack' process on the remote side to negotiate what data
will be transferred down.

   There are different ways to initiate the `upload-pack' process on
the remote repository. You can run via SSH in the same manner as the
`receive-pack' process. You can also initiate the process via the Git
daemon, which listens on a server on port 9418 by default. The
`fetch-pack' process sends data that looks like this to the daemon
after connecting:

003fgit-upload-pack schacon/simplegit-progit.git\0host=myserver.com\0

   It starts with the 4 bytes specifying how much data is following,
then the command to run followed by a null byte, and then the
server's hostname followed by a final null byte. The Git daemon
checks that the command can be run and that the repository exists and
has public permissions. If everything is cool, it fires up the
`upload-pack' process and hands off the request to it.

   If you're doing the fetch over SSH, `fetch-pack' instead runs
something like this:

$ ssh -x git@github.com "git-upload-pack 'schacon/simplegit-progit.git'"

   In either case, after `fetch-pack' connects, `upload-pack' sends
back something like this:

0088ca82a6dff817ec66f44342007202690a93763949 HEAD\0multi_ack thin-pack \
  side-band side-band-64k ofs-delta shallow no-progress include-tag
003fca82a6dff817ec66f44342007202690a93763949 refs/heads/master
003e085bb3bcb608e1e8451d4b2432f8ecbe6306e7e7 refs/heads/topic
0000

   This is very similar to what `receive-pack' responds with, but the
capabilities are different. In addition, it sends back the HEAD
reference so the client knows what to check out if this is a clone.

   At this point, the `fetch-pack' process looks at what objects it has
and responds with the objects that it needs by sending "want" and then
the SHA it wants. It sends all the objects it already has with "have"
and then the SHA. At the end of this list, it writes "done" to initiate
the `upload-pack' process to begin sending the packfile of the data it
needs:

0054want ca82a6dff817ec66f44342007202690a93763949 ofs-delta
0032have 085bb3bcb608e1e8451d4b2432f8ecbe6306e7e7
0000
0009done

   That is a very basic case of the transfer protocols. In more complex
cases, the client supports `multi_ack' or `side-band' capabilities; but
this example shows you the basic back and forth used by the smart
protocol processes.


File: progit.info,  Node: Maintenance and Data Recovery,  Next: Summary,  Prev: Transfer Protocols,  Up: Git Internals

9.7 Maintenance and Data Recovery
=================================

Occasionally, you may have to do some cleanup --- make a repository
more compact, clean up an imported repository, or recover lost work.
This section will cover some of these scenarios.

* Menu:

* Maintenance::
* Data Recovery::
* Removing Objects::


File: progit.info,  Node: Maintenance,  Next: Data Recovery,  Up: Maintenance and Data Recovery

9.7.1 Maintenance
-----------------

Occasionally, Git automatically runs a command called "auto gc". Most
of the time, this command does nothing. However, if there are too many
loose objects (objects not in a packfile) or too many packfiles, Git
launches a full-fledged `git gc' command. The `gc' stands for garbage
collect, and the command does a number of things: it gathers up all the
loose objects and places them in packfiles, it consolidates packfiles
into one big packfile, and it removes objects that aren't reachable
from any commit and are a few months old.

   You can run auto gc manually as follows:

$ git gc --auto

   Again, this generally does nothing. You must have around 7,000 loose
objects or more than 50 packfiles for Git to fire up a real gc command.
You can modify these limits with the `gc.auto' and `gc.autopacklimit'
config settings, respectively.

   The other thing `gc' will do is pack up your references into a
single file. Suppose your repository contains the following branches
and tags:

$ find .git/refs -type f
.git/refs/heads/experiment
.git/refs/heads/master
.git/refs/tags/v1.0
.git/refs/tags/v1.1

   If you run `git gc', you'll no longer have these files in the
`refs' directory. Git will move them for the sake of efficiency into a
file named `.git/packed-refs' that looks like this:

$ cat .git/packed-refs
# pack-refs with: peeled
cac0cab538b970a37ea1e769cbbde608743bc96d refs/heads/experiment
ab1afef80fac8e34258ff41fc1b867c702daa24b refs/heads/master
cac0cab538b970a37ea1e769cbbde608743bc96d refs/tags/v1.0
9585191f37f7b0fb9444f35a9bf50de191beadc2 refs/tags/v1.1
^1a410efbd13591db07496601ebc7a059dd55cfe9

   If you update a reference, Git doesn't edit this file but instead
writes a new file to `refs/heads'. To get the appropriate SHA for a
given reference, Git checks for that reference in the `refs' directory
and then checks the `packed-refs' file as a fallback. However, if you
can't find a reference in the `refs' directory, it's probably in
your `packed-refs' file.

   Notice the last line of the file, which begins with a `^'. This
means the tag directly above is an annotated tag and that line is the
commit that the annotated tag points to.


File: progit.info,  Node: Data Recovery,  Next: Removing Objects,  Prev: Maintenance,  Up: Maintenance and Data Recovery

9.7.2 Data Recovery
-------------------

At some point in your Git journey, you may accidentally lose a commit.
Generally, this happens because you force-delete a branch that had work
on it, and it turns out you wanted the branch after all; or you
hard-reset a branch, thus abandoning commits that you wanted something
from. Assuming this happens, how can you get your commits back?

   Here's an example that hard-resets the master branch in your test
repository to an older commit and then recovers the lost commits.
First, let's review where your repository is at this point:

$ git log --pretty=oneline
ab1afef80fac8e34258ff41fc1b867c702daa24b modified repo a bit
484a59275031909e19aadb7c92262719cfcdf19a added repo.rb
1a410efbd13591db07496601ebc7a059dd55cfe9 third commit
cac0cab538b970a37ea1e769cbbde608743bc96d second commit
fdf4fc3344e67ab068f836878b6c4951e3b15f3d first commit

   Now, move the `master' branch back to the middle commit:

$ git reset --hard 1a410efbd13591db07496601ebc7a059dd55cfe9
HEAD is now at 1a410ef third commit
$ git log --pretty=oneline
1a410efbd13591db07496601ebc7a059dd55cfe9 third commit
cac0cab538b970a37ea1e769cbbde608743bc96d second commit
fdf4fc3344e67ab068f836878b6c4951e3b15f3d first commit

   You've effectively lost the top two commits --- you have no branch
from which those commits are reachable. You need to find the latest
commit SHA and then add a branch that points to it. The trick is
finding that latest commit SHA --- it's not like you've memorized
it, right?

   Often, the quickest way is to use a tool called `git reflog'. As
you're working, Git silently records what your HEAD is every time you
change it. Each time you commit or change branches, the reflog is
updated. The reflog is also updated by the `git update-ref' command,
which is another reason to use it instead of just writing the SHA value
to your ref files, as we covered in the "Git References" section of
this chapter earlier. You can see where you've been at any time by
running `git reflog':

$ git reflog
1a410ef HEAD@{0}: 1a410efbd13591db07496601ebc7a059dd55cfe9: updating HEAD
ab1afef HEAD@{1}: ab1afef80fac8e34258ff41fc1b867c702daa24b: updating HEAD

   Here we can see the two commits that we have had checked out,
however there is not much information here. To see the same information
in a much more useful way, we can run `git log -g', which will give you
a normal log output for your reflog.

$ git log -g
commit 1a410efbd13591db07496601ebc7a059dd55cfe9
Reflog: HEAD@{0} (Scott Chacon <schacon@gmail.com>)
Reflog message: updating HEAD
Author: Scott Chacon <schacon@gmail.com>
Date:   Fri May 22 18:22:37 2009 -0700

    third commit

commit ab1afef80fac8e34258ff41fc1b867c702daa24b
Reflog: HEAD@{1} (Scott Chacon <schacon@gmail.com>)
Reflog message: updating HEAD
Author: Scott Chacon <schacon@gmail.com>
Date:   Fri May 22 18:15:24 2009 -0700

     modified repo a bit

   It looks like the bottom commit is the one you lost, so you can
recover it by creating a new branch at that commit. For example, you
can start a branch named `recover-branch' at that commit (ab1afef):

$ git branch recover-branch ab1afef
$ git log --pretty=oneline recover-branch
ab1afef80fac8e34258ff41fc1b867c702daa24b modified repo a bit
484a59275031909e19aadb7c92262719cfcdf19a added repo.rb
1a410efbd13591db07496601ebc7a059dd55cfe9 third commit
cac0cab538b970a37ea1e769cbbde608743bc96d second commit
fdf4fc3344e67ab068f836878b6c4951e3b15f3d first commit

   Cool --- now you have a branch named `recover-branch' that is where
your `master' branch used to be, making the first two commits reachable
again. Next, suppose your loss was for some reason not in the reflog
--- you can simulate that by removing `recover-branch' and deleting the
reflog. Now the first two commits aren't reachable by anything:

$ git branch -D recover-branch
$ rm -Rf .git/logs/

   Because the reflog data is kept in the `.git/logs/' directory, you
effectively have no reflog. How can you recover that commit at this
point? One way is to use the `git fsck' utility, which checks your
database for integrity. If you run it with the `--full' option, it
shows you all objects that aren't pointed to by another object:

$ git fsck --full
dangling blob d670460b4b4aece5915caf5c68d12f560a9fe3e4
dangling commit ab1afef80fac8e34258ff41fc1b867c702daa24b
dangling tree aea790b9a58f6cf6f2804eeac9f0abbe9631e4c9
dangling blob 7108f7ecb345ee9d0084193f147cdad4d2998293

   In this case, you can see your missing commit after the dangling
commit. You can recover it the same way, by adding a branch that points
to that SHA.


File: progit.info,  Node: Removing Objects,  Prev: Data Recovery,  Up: Maintenance and Data Recovery

9.7.3 Removing Objects
----------------------

There are a lot of great things about Git, but one feature that can
cause issues is the fact that a `git clone' downloads the entire
history of the project, including every version of every file. This is
fine if the whole thing is source code, because Git is highly optimized
to compress that data efficiently. However, if someone at any point in
the history of your project added a single huge file, every clone for
all time will be forced to download that large file, even if it was
removed from the project in the very next commit. Because it's
reachable from the history, it will always be there.

   This can be a huge problem when you're converting Subversion or
Perforce repositories into Git. Because you don't download the whole
history in those systems, this type of addition carries few
consequences. If you did an import from another system or otherwise
find that your repository is much larger than it should be, here is how
you can find and remove large objects.

   Be warned: this technique is destructive to your commit history. It
rewrites every commit object downstream from the earliest tree you have
to modify to remove a large file reference. If you do this immediately
after an import, before anyone has started to base work on the commit,
you're fine --- otherwise, you have to notify all contributors that
they must rebase their work onto your new commits.

   To demonstrate, you'll add a large file into your test repository,
remove it in the next commit, find it, and remove it permanently from
the repository. First, add a large object to your history:

$ curl http://kernel.org/pub/software/scm/git/git-1.6.3.1.tar.bz2 > git.tbz2
$ git add git.tbz2
$ git commit -am 'added git tarball'
[master 6df7640] added git tarball
 1 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 git.tbz2

   Oops --- you didn't want to add a huge tarball to your project.
Better get rid of it:

$ git rm git.tbz2
rm 'git.tbz2'
$ git commit -m 'oops - removed large tarball'
[master da3f30d] oops - removed large tarball
 1 files changed, 0 insertions(+), 0 deletions(-)
 delete mode 100644 git.tbz2

   Now, `gc' your database and see how much space you're using:

$ git gc
Counting objects: 21, done.
Delta compression using 2 threads.
Compressing objects: 100% (16/16), done.
Writing objects: 100% (21/21), done.
Total 21 (delta 3), reused 15 (delta 1)

   You can run the `count-objects' command to quickly see how much
space you're using:

$ git count-objects -v
count: 4
size: 16
in-pack: 21
packs: 1
size-pack: 2016
prune-packable: 0
garbage: 0

   The `size-pack' entry is the size of your packfiles in kilobytes, so
you're using 2MB. Before the last commit, you were using closer to 2K
--- clearly, removing the file from the previous commit didn't remove
it from your history. Every time anyone clones this repository, they
will have to clone all 2MB just to get this tiny project, because you
accidentally added a big file. Let's get rid of it.

   First you have to find it. In this case, you already know what file
it is. But suppose you didn't; how would you identify what file or
files were taking up so much space? If you run `git gc', all the
objects are in a packfile; you can identify the big objects by running
another plumbing command called `git verify-pack' and sorting on the
third field in the output, which is file size. You can also pipe it
through the `tail' command because you're only interested in the last
few largest files:

$ git verify-pack -v .git/objects/pack/pack-3f8c0...bb.idx | sort -k 3 -n | tail -3
e3f094f522629ae358806b17daf78246c27c007b blob   1486 734 4667
05408d195263d853f09dca71d55116663690c27c blob   12908 3478 1189
7a9eb2fba2b1811321254ac360970fc169ba2330 blob   2056716 2056872 5401

   The big object is at the bottom: 2MB. To find out what file it is,
you'll use the `rev-list' command, which you used briefly in Chapter
7. If you pass `--objects' to `rev-list', it lists all the commit SHAs
and also the blob SHAs with the file paths associated with them. You
can use this to find your blob's name:

$ git rev-list --objects --all | grep 7a9eb2fb
7a9eb2fba2b1811321254ac360970fc169ba2330 git.tbz2

   Now, you need to remove this file from all trees in your past. You
can easily see what commits modified this file:

$ git log --pretty=oneline -- git.tbz2
da3f30d019005479c99eb4c3406225613985a1db oops - removed large tarball
6df764092f3e7c8f5f94cbe08ee5cf42e92a0289 added git tarball

   You must rewrite all the commits downstream from `6df76' to fully
remove this file from your Git history. To do so, you use
`filter-branch', which you used in Chapter 6:

$ git filter-branch --index-filter \
   'git rm --cached --ignore-unmatch git.tbz2' -- 6df7640^..
Rewrite 6df764092f3e7c8f5f94cbe08ee5cf42e92a0289 (1/2)rm 'git.tbz2'
Rewrite da3f30d019005479c99eb4c3406225613985a1db (2/2)
Ref 'refs/heads/master' was rewritten

   The `--index-filter' option is similar to the `--tree-filter' option
used in Chapter 6, except that instead of passing a command that
modifies files checked out on disk, you're modifying your staging
area or index each time. Rather than remove a specific file with
something like `rm file', you have to remove it with `git rm --cached'
--- you must remove it from the index, not from disk. The reason to do
it this way is speed --- because Git doesn't have to check out each
revision to disk before running your filter, the process can be much,
much faster. You can accomplish the same task with `--tree-filter' if
you want. The `--ignore-unmatch' option to `git rm' tells it not to
error out if the pattern you're trying to remove isn't there.
Finally, you ask `filter-branch' to rewrite your history only from the
`6df7640' commit up, because you know that is where this problem
started. Otherwise, it will start from the beginning and will
unnecessarily take longer.

   Your history no longer contains a reference to that file. However,
your reflog and a new set of refs that Git added when you did the
`filter-branch' under `.git/refs/original' still do, so you have to
remove them and then repack the database. You need to get rid of
anything that has a pointer to those old commits before you repack:

$ rm -Rf .git/refs/original
$ rm -Rf .git/logs/
$ git gc
Counting objects: 19, done.
Delta compression using 2 threads.
Compressing objects: 100% (14/14), done.
Writing objects: 100% (19/19), done.
Total 19 (delta 3), reused 16 (delta 1)

   Let's see how much space you saved.

$ git count-objects -v
count: 8
size: 2040
in-pack: 19
packs: 1
size-pack: 7
prune-packable: 0
garbage: 0

   The packed repository size is down to 7K, which is much better than
2MB. You can see from the size value that the big object is still in
your loose objects, so it's not gone; but it won't be transferred
on a push or subsequent clone, which is what is important. If you
really wanted to, you could remove the object completely by running
`git prune --expire'.