GitMigration

From Wiki
Jump to: navigation, search

Do you want just want to work on DebConf and skip all the gory details? Then click here to skip down to the appropriate section…

Over the last couple of years, more and more people wanted to migrate the DebConf repositories away from Subversion, and it was unanimously agreed to use Git instead.

The following document aims to make the transition easier. If you are already familiar with Git and you just want to know the repository URLs, skip to the bottom of the page, section #URLs_and_repos.

Contents

[edit] Getting started

[edit] Requirements

Please apt-get install git.

You may also want to install gitk, gitg or tig, which can be used to graphically inspect the history of a repository. This may be useful especially when working with Git for the first time.

[edit] Quick overview of the differences between Git and Subversion

Git and Subversion are both revision control systems and fundamentally do the same task (keep track of changes to files over time). However, tthere are three main differences between the two tools:

  • When you issued the command svn commit, Subversion would record all the changes you made into one "revision". Git allows fine-grained control for what should become part of the commit transaction. If you want to commit all changes made to a file, type git add <filename>; if you want to select particular changes within a file, use git add -p, optionally with a filename.
  • Subversion automatically pushes a commit to the server when you run svn commit, Git does not. With Git, commits need to be explicitly pushed, which you do by running git push. Git can be used offline, while Subversion cannot.
  • As a consequence of Git's offline usability, it is possible that someone else pushes changes before you pushed your commits. In this case, you need to first pull the new commits, using git pull.

[edit] Getting you started quickly

The first thing to do is to identify yourself to Git:

git config --global user.name "Jane Q. Git"
git config --global user.email "jane.git@example.org"

Instead of specifying --global, you can leave it out and the configuration will be stored in the current repository only.

If you prefer not to dive right into the cold water, you can create yourself two aliases to mimic the way Subversion worked:

git config --global branch.autosetuprebase true
git config --global alias.scommit '!scommit() { git commit -a "$@" && git push; }; scommit'
git config --global alias.update pull
git config --global alias.up pull

With those in place, you can use git scommit just like you used svn commit: it will create a commit encompassing all changes and push it to the remote repository right away. You can also use git update just like svn update and it will fetch new revisions from the remote. The last setting ensures that local changes are added on top of any changes fetched from the remote.

Most of the other commands that are available for svn are identical in Git: add, rm, diff, log, status.

[edit] Getting help

Please feel free to ask any questions about this migration or about how to use Git on the #debconf-team channel on irc.oftc.net.

If you prefer to use the mailing list, send an e-mail to debconf-team@lists.debconf.org and make sure to include "[Git]" in the subject to help people identify the message.

[edit] More information

Here are a few links that may also be of interest:

[edit] URLs and repos

The URLs and description of the individual repos follows, and there's a table helping you where to find stuff below, but first, there are some important things:

[edit] Important information

Important news will be published here, so make sure to revisit this page often, especially in case of problems

If you do experience a problem, please send a message to debconf-team@lists.debconf.org with "[git migration]" in subject line.

[edit] Changes made since the migration

The following is a list of changes made since the announcement of the completion of the migration (2014-04-02-1403):

Timestamp Repository Re-clone? Description
2014-04-02-1616 debconf-data/dc11 Yes History rewritten to remove huge print-quality files (sponsorship brochure, now on media.dc.o).
2014-04-02-1632 debconf-data/dc13 Yes History rewritten to remove venue evaluation photos (now on gallery.dc.o).
2014-04-02-1718 debconf-data/dc13 Yes History rewritten to remove more venue evaluation photos (now on gallery.dc.o).
2014-04-02-2204 debconf-data/misc Yes History rewritten to split out individual repos:
  • 10years.git
  • blog.git
  • db.git
  • misc.git
  • penta.git
  • www.git

Instead of a re-clone, you can also run (this is for advanced users):

git tag -d debconf2git
git fetch
git checkout master
git reset --hard origin/master
git gc --prune=all

[edit] DebConf team repository

The debconf-team repository contains non-public information used by the team(s) to coordinate (except for accounting, see below). The same repository is used across different conferences to provide easier access to past information for new teams (this is an encouragement…).

You need to be a member of the debconf-team project on alioth.debian.org to be able to use this repository.

The following command is used to clone the respository:

git clone ssh://your-alioth-username@git.debian.org/git/debconf-team/debconf-team
cd debconf-team

Please try to refrain from committing huge files to the repository. Scans of invoices and receipts should go into a conference's accounting-data repository, and sensitive information that you need to keep around for the conference, and possibly for a while thereafter should live in the conference's private repository, see below:

For a list of available (sub-)repositories, please have a look in the directory /git/debconf-team on git.debian.org.

[edit] DebConfX accounting data repositories

Scans of invoices and receipts should be put into a separate, per-conference repository, dcX-accounting-data, which you can then access like so:

git clone ssh://your-alioth-username@git.debian.org/git/debconf-team/dc13-accounting-data
cd dc13-accounting-data

The main motivation here is to keep the size of the main team repository from exploding, and accounting data seems to be the worst offender here, based on past experience.

[edit] DebConfX private repositories

Personal information and other sensitive data, such as anything relating to visa applications of specific people, whould be put into dcX-private, and might be archived/purged at some point after the conference. This is to avoid infinite access to such data by every current and future team member.

git clone ssh://your-alioth-username@git.debian.org/git/debconf-team/dc14-private
cd dc14-private

[edit] DebConf data repositories

The debconf-data repositories contain public information for each conference. Contrary to the Subversion repository, each conference has its own repository. For a list of available (sub-)repositories, please visit GitWeb, the simple HTTP interface, or have a look in the directory /git/debconf-data on git.debian.org.

If you are a member of the debconf-data project on alioth.debian.org, you can and should clone each repository using SSH:

git clone ssh://your-alioth-username@git.debian.org/git/debconf-data/dc14
cd dc14

If you just require read-only access, use

git clone git://anonscm.debian.org/git/debconf-data/dc14
cd dc14

instead.

[edit] Where do I find …?

svn.debian.org/svn/… git.debian.org/git/… Notes
debconf-team debconf-team The debconf-team was mostly migrated as-is, with the exceptions of accounting and sensitive, personal data. Some of those have simply been removed, or split into their own repos:
debconf-team/…
  • dc6/money-matters/invoices
  • dc6/money-matters/payments*
  • dc7/sponsors/invoices
  • dc8/commitments
  • dc9/invoices
  • dc10/invoices
  • dc10/receipts
  • dc11/invoices
  • dc11/receipts
  • dc12/receipts
  • dc12/invoices
  • dc12/accounting
  • dc12/travel-reimbursements
  • dc13/finance
dcX-accounting-data/… Accounting data (mostly large scans of receipts and invoices, etc.) have been split out into repositories for each conference, e.g. dc6-accounting-data.
debconf-team/…
  • dc14/visa/people
dcX-private/… Personal data (such as used for visa applications) have been split out into repositories for each conference, e.g. dc14-private.
debconf-data/…
  • misc
  • reports
  • blog
  • penta
  • www
  • db
  • media.debconf.org
debconf-data/… Subdirectories have been split into their own repositories.
debconf-data/10year debconf-data/10years Dto., but renamed to match corresponding hostname.
debconf-data/…
  • dc*
  • *.mini
  • miniconf10
debconf-data/… Subdirectories for each conference have also been split into their own repositories

[edit] Help! Too many repositories!

If you don't like having 46 different repositories for your DebConf work and you wallow in memories of the days when there were just two, one for private and one for public data, here's a solution for you (which lets you have the best of all worlds: speed, size, Git, and simplicity).

[edit] Installing mr and basic setup

The tool to use is myrepos (mr), and the repositories have been enabled for its use:

% apt-get install myrepos

Before starting to use it, however, it helps to make sure that your SSH client knows how to log in to git.debian.org (e.g. which username to use, which key to present, etc.). So make sure there's a stanza like

Host git.debian.org
  User your_alioth_username
  #IdentityFile ~/.ssh/other_identity_if_not_id_rsa

in your ~/.ssh/config file.

Now we can start using mr:

Clone the debconf-team repository, as you normally would.

% git clone ssh://git.debian.org/git/debconf-team/debconf-team
[…]
% cd debconf-team

Tell mr that you trust the contents of the .mrconfig file contained therein:

% echo \~${PWD#$HOME}/.mrconfig | tee -a ~/.mrtrust
~/debian/debconf/team/.mrconfig

Clone the debconf-data meta-repository to a place of your choosing. I call it a meta-repository, because it is empty but only serves as a parent to all the sub-repositories:

% git clone ssh://git.debian.org/git/debconf-data/debconf-data
[…]

If you're like me, you want debconf-data to live inside the debconf-team repository, which you could tell mr like this (assuming you want it to live in the subdirectory pub-data, I chose this name instead of debconf-data to draw attention to the public nature of data contained therein):

% echo -e "[DEFAULT]\ninclude = cat .mrconfig.pub-data\n" >> .mrconfig.local
% mr up
[…]
mr checkout: ~/debian/debconf/debconf-team/pub-data
Cloning into '~/debian/debconf/debconf-team/pub-data'...
[…]
mr update: finished (1 ok; 11 skipped)

(The skip is needed due to #743576.)

Tell mr that you also trust the contents of the .mrconfig file contained in the repository just checked out:

% cd debconf-data    # or pub-data
% echo \~${PWD#$HOME}/.mrconfig | tee -a ~/.mrtrust
~/debian/debconf/team/pub-data/.mrconfig

I will assume you chose pub-data too in the following…

[edit] Interlude: directory structure

Providing automation with mr means that there are some choices that have been made which will affect you. In this case, there's a pre-defined directory structure, which can be described as follows:

  1. debconf-team is the base directory
  2. dcX-private and dcX-accounting-data repositories are placed inside the existing dcX directories, i.e. dcX/private and dcX/accounting-data.
  3. All the debconf-data sub-repositories become subdirectories of wherever you put the meta-repository, e.g. pub-data.

Here's what a complete tree would look like, at least at time of writing (abridged to only show relevant directories, and with debconf-data residing in debconf-team/pub-data):

.
├── dc1
├── dc10
│   ├── accounting-data
├── dc11
│   ├── accounting-data
├── dc12
│   ├── accounting-data
├── dc13
│   ├── accounting-data
├── dc14
│   ├── accounting-data
│   ├── private
├── dc15
├── dc2
├── dc3
├── dc4
├── dc5
├── dc6
│   ├── accounting-data
├── dc7
│   ├── accounting-data
├── dc8
│   ├── accounting-data
├── dc9
│   ├── accounting-data
├── dc-admin
├── debconf2git
├── kgb-client
├── pub-data
│   ├── 10years
│   ├── bcn2014.mini
│   ├── blog
│   ├── db
│   ├── dc0
│   ├── dc1
│   ├── dc10
│   ├── dc11
│   ├── dc12
│   ├── dc13
│   ├── dc14
│   ├── dc15
│   ├── dc2
│   ├── dc3
│   ├── dc4
│   ├── dc5
│   ├── dc6
│   ├── dc7
│   ├── dc8
│   ├── dc9
│   ├── fr2010.mini
│   ├── fr2012.mini
│   ├── in2010.mini
│   ├── in2011.mini
│   ├── in2013.mini
│   ├── media.debconf.org
│   ├── miniconf10
│   ├── misc
│   ├── ni2013.mini
│   ├── penta
│   ├── reports
│   └── www
└── video-hw

However, you need not check out all of these repositories to get work done! Keep reading…

[edit] Adding sub-repositories for mr to manage

What you have now is a complete debconf-team clone (without private and accounting-data) in the current directory, as well as ./pub-data prepared to hold debconf-data contents. If you run mr up now, it will tell you about "skipped" repositories:

% mr up
[…]
mr update: finished (2 ok; 43 skipped)

The reason for this high number of skips is that mr has been configured to only manage repositories that have been cloned beforehand. So say, for instance, you wanted to have access to debconf-team/dc14-private and debconf-data/blog, you would just do:

% mr -fd dc14/private checkout
mr checkout: ~/debian/debconf/debconf-team/dc14/private
Cloning into 'private'...
[…]
mr checkout: finished (1 ok)
% mr -fd pub-data/blog checkout
mr checkout: ~/debian/debconf/debconf-team/pub-data/blog
Cloning into 'blog'...
[…]
mr checkout: finished (1 ok)

Subsequent invocations of mr will now recurse over the two base repositories (debconf-team and pub-data), as well as the two additional repositories checked out:

% mr up
[…]
mr update: finished (4 ok; 41 skipped)

[edit] I want it all

If you don't want to select individual sub-repositories to clone and manage, but just let mr handle them all, put the following snippet into .mrconfig.local:

[DEFAULT]
lib = lazy() { false; }

Now running mr up will clone any repositories that aren't already present.

[edit] If you're already using mr

Those of you already using mr could just add the debconf-team repository to an existing .mrconfig file and enable chaining, e.g.

[debconf/team]
checkout = git clone ssh://git.debian.org/git/debconf-team/debconf-team $MR_REPO
chain = true

[edit] Next steps

And now you can use mr pretty much like you used svn before, as it does the recursion for you, e.g. the following commands all pretty much do what you expect:

  • mr update
  • mr status
  • mr commit (actually, in our case: mr record && mr push)
  • mr log and mr diff

Have a look at the mr(1) manpage for more information. You can define aliases, or new commands, hooks, run checkouts in parallel, and other things.

If you do want to make local changes to .mrconfig, please put them into .mrconfig.local (and do not commit that).

[edit] I just want to work on DC15!

You don't care about all of the above and you just want to get started? Alright! If you're okay using and learning about mr (possibly in addition to Git), read on. If you don't want to learn about mr, skip to the second-next section.

[edit] Using mr

[edit] Setup

sudo apt-get install git myrepos gitk
cd ~/path/to/my/debconf/workspace
git clone ssh://git.debian.org/git/debconf-team/debconf-team
cd debconf-team
echo -e '[DEFAULT]\ninclude = cat .mrconfig.pub-data\n' > .mrconfig.local
echo ${PWD}/.mrconfig | sed -e "s,$HOME,~," >> ~/.mrtrust
echo ${PWD}/pub-data/.mrconfig | sed -e "s,$HOME,~," >> ~/.mrtrust
mr up

[edit] Picking repos

All DC15 team-related stuff is already in ./dc15.

If you want to work on accounting or visa (or other personal stufF), you add the corresponding sub-repos like this:

mr -fd dc15/accounting-data co
mr -fd dc15/private co

You will also need the public data for DC15:

mr -fd pub-data/dc15 co

And depending on whether you will be working on the blog, media data, reports, do any or all of the following:

mr -fd pub-data/blog co
mr -fd pub-data/media.debconf.org co
mr -fd pub-data/reports co

[edit] The result

This is what your tree will now look like (abrigded):

.
├── dc10
├── dc11
│   […]
├── dc15
│   ├── accounting-data           # debconf-team/dc15-accounting-data
│   ├── contracts
│   ├── logos
│   ├── private                   # debconf-team/dc15-private
│   └── visa
├── dc-admin
├── pub-data
│   ├── blog                      # debconf-data/blog
│   ├── dc15                      # debconf-data/dc15
│   ├── media.debconf.org         # debconf-data/media.debconf.org
│   └── reports                   # debconf-data/reports
└── video-hw

[edit] I do not want to learn about mr

Fine!

sudo apt-get install git gitk
cd ~/path/to/my/debconf/workspace
git clone ssh://git.debian.org/git/debconf-team/debconf-team
git clone ssh://git.debian.org/git/debconf-team/dc15-private
git clone ssh://git.debian.org/git/debconf-team/dc15-accounting-data
git clone ssh://git.debian.org/git/debconf-data/dc15 dc15-data
git clone ssh://git.debian.org/git/debconf-data/blog
git clone ssh://git.debian.org/git/debconf-data/reports
git clone ssh://git.debian.org/git/debconf-team/logos
git clone ssh://git.debian.org/git/debconf-data/media.debconf.org

And now your tree will look like this:

.
├── blog
├── dc15-data
├── dc15-private
├── dc15-accounting-data
├── debconf-team
├── logos
├── media.debconf.org
└── reports

[edit] Alright, so how do I…?

[edit] … tell Git about a new file I want to add to the repo?

git add newfile
# git commit
# git push

[edit] … remove a file

git rm file-to-remove
# git commit
# git push

[edit] … obtain information on the status of things

git status

The output of git status also contains some helpful information about why certain files are listed where they are, along with relevant commands. It helps to run a recent version of Git!

[edit] … commit my changes? Git says "nothing to commit"

Git is slightly different from Subversion in that you have to "stage" changes. At first, this might seem like a nuisance, but over time, you'll come to appreciate this!

If you just want to commit all changes made (like Subversion would), you can do either of the following two:

git add -u && git commit
git commit -a

If you want to select only specific files to become part of a commit:

git add file1 file2
git commit

If you want to select specific lines (!) out of all the changes you have made (you can even edit the commits and effectively select words!):

git add -p [file1]
git commit

If you have staged changes and you have changed your mind, you can unstage them like so:

git reset HEAD -- file

[edit] … share my changes with the team?

"I committed changes, but noone else in the team can see them!"

Commiting is a local action with Git (meaning you can also do it offline!).

[edit] If you have an Alioth account with access to the repository…

To publish your commits, you can just push them to the repository:

git push

Quick note, in case you worry about pushing changes: don't; everything can be undone. However, if you are unsure and would prefer to send your changes as proposals via e-mail, please skip to the next section.

Git will report success with a bunch of lines ending e.g.

To ssh://git.debian.org/git/debconf-team/debconf-team
   067866e..cf54341  master -> master

It may also fail with the following:

To ssh://git.debian.org/git/debconf-team/debconf-team
 ! [rejected]        master -> master (non-fast-forward)
[…]

This simply means that someone else has pushed changes, which you need to integrate locally first:

git pull --rebase
# possibly resolve conflicts at this stage
git push

If you have branch.autosetuprebase set (see #Getting_you_started_quickly), then you can leave off --rebase. Rebasing on pull means that the history will be linearised, rather than different branches merged, you can read up on what that means e.g. here.

[edit] If you do not have an Alioth account with access to the repository…

Your local clone is a full-featured Git repository, meaning you can make regular commits. If you cannot push them directly to the repository, you can still share your changes, by e-mail. This is also a good way to share commits you want others to look at, or that should be discussed.

Let's assume you made a number of commits you would like to share. You can display those as follows:

git log --oneline origin/master..

If the output is not what you expect and you do not want to send all commits, then adjust the specification. For instance, to look at just the last commit, or the last two commits, you'd say:

git log --oneline HEAD^..
git log --oneline HEAD^^..

The ^ (caret) character denots the parent of a commit, and HEAD is the current commit. The .. afterwards just tell Git "everything since…". This is all explained with high complexity in the git-rev-parse(1) manpage.

Once you figure out which selector describes the set of commits to share, you just fire off another command:

git send-email --to debconf-team@lists.debconf.org origin/master

and follow the instructions. If you want to write an introductory message to your patch or patch series, throw a --compose in there:

git send-email --to debconf-team@lists.debconf.org --compose origin/master

If you do this often, you could configure the default recipient in your Git clone's configuration:

git config sendemail.to debconf-team@lists.debconf.org
git send-email origin/master

Have a look at the git-send-email(1) manpage for more settings, if you care.

You might also want to know that git send-email employes git format-patch to create the patches, so if you want to export your commits to files, have a look at that tool, as well.

[edit] … discard my changes

To completely discard your changes to a file, run

git checkout -- file

You can also just discard changes on a line-by-line basis:

git checkout -p [-- file]

[edit] … resolve conflicts?

After git pull, you may encounter some conflicts which you need to resolve. Unfortunately, the process is different depending on whether you rebased or not. While team policy is to rebase locally, we'll cover both cases for completeness:

[edit] Without rebasing

You should really be using git pull --rebase. To set that as default, run

git config branch.master.rebase true
git config branch.autosetuprebase true   # to set to true for new branches

Confllict resolution is required when git pull says:

CONFLICT (content): Merge conflict in a_file
Automatic merge failed; fix conflicts and then commit the result.

You can inspect this further like so:

% git status
On branch master
Your branch and 'origin/master' have diverged,
and have 1 and 1 different commit each, respectively.
  (use "git pull" to merge the remote branch into yours)

You have unmerged paths.
  (fix conflicts and run "git commit")

Unmerged paths:
  (use "git add <file>..." to mark resolution)

	both modified:      a_file
[…]

To resolve, edit a_file until it has the desired contents (make sure not to remove the changes someone else did!), and then commit the merge:

editor a_file
git add a_file
git commit
git push

You may also want to experiment with git mergetool.

[edit] With rebasing

A merge conflict using rebasing looks like this:

Falling back to patching base and 3-way merge...
Auto-merging a_file
CONFLICT (content): Merge conflict in a_file
Failed to merge in the changes.
Patch failed at 0001 .
The copy of the patch that failed is found in:
   /home/madduck/debian/debconf/team/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

and using git status:

rebase in progress; onto 37ab086
You are currently rebasing branch 'master' on '37ab086'.
  (fix conflicts and then run "git rebase --continue")
  (use "git rebase --skip" to skip this patch)
  (use "git rebase --abort" to check out the original branch)

Unmerged paths:
  (use "git reset HEAD <file>..." to unstage)
  (use "git add <file>..." to mark resolution)

	both modified:      a_file
[…]

To resolve, you edit the file, stage changes, but you do not commit; instead, you continue the rebase:

editor a_file         # or git mergetool
git add a_file
git rebase --continue

If instead you see a message like this, even though you used git add:

No changes - did you forget to use 'git add'?

then your changes may have been obsoleted by the changes someone else made. It's best to verify that, and then run

git rebase --skip

Finally, when the changes have been integrated, run

git push

[edit] … stash changes away temporarily?

"I have all these local changes, but I need to work on something else…?"

git stash

Now your changes have been "stashed away", and you can pull, edit, add, commit, push as you want.

When you want to resume your work, run

git stash pop

Weeeeh! \o/

[edit] Miscellaneous

[edit] Ignoring IRC notifications

Commits to the repositories will be reported to the debconf-team IRC channel. If you feel flooded at times (e.g. because you are working on DC14 and don't care about commits related to DC15), you could tell your IRC client to ignore those lines.

For instance, here is how you would tell irssi to stop bothering you with DC15-related commit notifications:

/ignore -regexp -pattern "debconf-(team |data/)dc15" -channels #debconf-team *!kgb@kgb*.bot.oftc.net PUBLIC
Personal tools