Videoteam/Archive

From Wiki
Jump to: navigation, search

Contents

[edit] Archive and mirrors

[edit] Information for mirror operators

Use rsync://meetings-archive.debian.net/pub/debian-meetings/ to mirror the archive. 2004-2013 is around 810G, growing at around 100GB per year.

Use this settings in your vhost.conf:

<Directory "/export/ftp/pub/debian-meetings">
         Options Indexes FollowSymLinks
         ReadmeName FOOTER.html
         DirectoryIndex index.html
</Directory>

[edit] git-annex

Please see the git-annex walkthrough for basic information about git-annex.

[edit] TODO

  • Make clear-text backends work
  • Create workflow for public read-only consumption
  • Use chunking for large files on Amazon S3 once Jessie is released

[edit] Add new data

git annex add $path
git annex sync # once Jessie releases, we can do git annex sync -m 'This is my message'
    # you can also git commit --amend for now

[edit] Edit data

git annex unlock $path
# edit what needs editing
git annex add $path
git annex sync # once Jessie releases, we can do git annex sync -m 'This is my message'
    # you can also git commit --amend for now


[edit] Sync data to Amazon S3

[edit] Normal every-day sync

git annex copy --to amazon_aws_s3-us_east_1-SHA512E --fast

[edit] Thourough sync

git annex copy --to amazon_aws_s3-us_east_1-SHA512E

[edit] Initial clone

git clone ssh://apu.debconf.org:/srv/video/video.debian.net
cd video.debian.net
git annex init '$yourname.localh/video.debian.net' # (or some other name which can be easily recognized by you and others. If in doubt, ask in #debconf-video. Please document your storage below. Ask RichiH about trust levels of your repo)
git annex get # if you want a full clone. Currently, this weighs in at less than 1TB
git annex sync

[edit] Consistency check & grabbing new data

Every X time, it would be nice if you ran:

git fsck && git annex fsck
##### check that there are no errors! If there are, ask in #debconf-video!
git annex sync
git annex get

[edit] Exisiting repos

[edit] Full data

  • apu.debconf.org/srv/video/video.debian.net - semitrusted
  • Amazon S3

[edit] Partial data

[edit] Git-only

[edit] Planned

  • Archive.org
Personal tools