hgext/largefiles/__init__.py
author Na'Tosha Bard <natosha@unity3d.com>
Sat, 09 Feb 2013 21:07:42 +0000
changeset 18704 d69585a5c5c0
parent 18599 5cd1dbf4c5d2
child 18975 aa8205a9f51a
permissions -rw-r--r--
largefiles: don't cache largefiles for pulled heads by default After discussion, we've agreed that largefiles for newly pulled heads should not be cached by default. The use case for this is using largefiles repos with multiple remote servers (and therefore multiple remote largefiles caches), where users will be pulling from non-default locations on a regular basis. We think this use case will be significantly less common than the use case where all largefiles are stored on the same central server, so the default should be no caching. The old behavior can be obtained by passing the --cache-largefiles flag to pull.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     1
# Copyright 2009-2010 Gregory P. Ward
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     2
# Copyright 2009-2010 Intelerad Medical Systems Incorporated
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     3
# Copyright 2010-2011 Fog Creek Software
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     4
# Copyright 2010-2011 Unity Technologies
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     5
#
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     6
# This software may be used and distributed according to the terms of the
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     7
# GNU General Public License version 2 or any later version.
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     8
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     9
'''track large binary files
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    10
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    11
Large binary files tend to be not very compressible, not very
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    12
diffable, and not at all mergeable. Such files are not handled
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    13
efficiently by Mercurial's storage format (revlog), which is based on
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    14
compressed binary deltas; storing large binary files as regular
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    15
Mercurial files wastes bandwidth and disk space and increases
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    16
Mercurial's memory usage. The largefiles extension addresses these
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    17
problems by adding a centralized client-server layer on top of
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    18
Mercurial: largefiles live in a *central store* out on the network
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    19
somewhere, and you only fetch the revisions that you need when you
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    20
need them.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    21
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    22
largefiles works by maintaining a "standin file" in .hglf/ for each
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    23
largefile. The standins are small (41 bytes: an SHA-1 hash plus
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    24
newline) and are tracked by Mercurial. Largefile revisions are
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    25
identified by the SHA-1 hash of their contents, which is written to
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    26
the standin. largefiles uses that revision ID to get/put largefile
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    27
revisions from/to the central store. This saves both disk space and
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    28
bandwidth, since you don't need to retrieve all historical revisions
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    29
of large files when you clone or pull.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    30
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    31
To start a new repository or add new large binary files, just add
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    32
--large to your :hg:`add` command. For example::
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    33
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    34
  $ dd if=/dev/urandom of=randomdata count=2000
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    35
  $ hg add --large randomdata
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    36
  $ hg commit -m 'add randomdata as a largefile'
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    37
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    38
When you push a changeset that adds/modifies largefiles to a remote
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    39
repository, its largefile revisions will be uploaded along with it.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    40
Note that the remote Mercurial must also have the largefiles extension
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    41
enabled for this to work.
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    42
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    43
When you pull a changeset that affects largefiles from a remote
18704
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    44
repository, the largefiles for the changeset won't be pulled down.
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    45
Instead, when you later update to such a revision, any largefiles
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    46
needed by that revision are downloaded and cached (if they have
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    47
never been downloaded before).  This means that network access may
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    48
be required to update to changesets you have previously updated to.
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    49
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    50
If you know you are pulling from a non-default location and want to
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    51
ensure that you will have the largefiles needed to merge or rebase
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    52
with new heads that you are pulling, then you can pull with the
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    53
--cache-largefiles flag to pre-emptively download any largefiles
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    54
that are new in the heads you are pulling.
18599
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    55
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    56
The one exception to the "largefiles won't be pulled until you update
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    57
to a revision that changes them" rule is when you pull new heads.
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    58
Because you could be pulling new heads (that you may later want to
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    59
merge with) from a non-default location (that Mercurial won't know
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    60
about later), when you pull new heads, largefiles revisions for those
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    61
heads are downloaded and cached locally.
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    62
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    63
If you already have large files tracked by Mercurial without the
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    64
largefiles extension, you will need to convert your repository in
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    65
order to benefit from largefiles. This is done with the
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    66
:hg:`lfconvert` command::
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    67
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    68
  $ hg lfconvert --size 10 oldrepo newrepo
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    69
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    70
In repositories that already have largefiles in them, any new file
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    71
over 10MB will automatically be added as a largefile. To change this
15304
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    72
threshold, set ``largefiles.minsize`` in your Mercurial config file
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    73
to the minimum size in megabytes to track as a largefile, or use the
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    74
--lfsize option to the add command (also in megabytes)::
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    75
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    76
  [largefiles]
15304
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    77
  minsize = 2
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    78
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    79
  $ hg add --lfsize 2
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    80
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    81
The ``largefiles.patterns`` config option allows you to specify a list
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    82
of filename patterns (see :hg:`help patterns`) that should always be
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    83
tracked as largefiles::
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    84
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    85
  [largefiles]
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    86
  patterns =
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    87
    *.jpg
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    88
    re:.*\.(png|bmp)$
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    89
    library.zip
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    90
    content/audio/*
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    91
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    92
Files that match one of these patterns will be added as largefiles
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    93
regardless of their size.
15743
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
    94
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
    95
The ``largefiles.minsize`` and ``largefiles.patterns`` config options
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
    96
will be ignored for any repositories not already containing a
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
    97
largefile. To add the first largefile to a repository, you must
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
    98
explicitly do so with the --large flag passed to the :hg:`add`
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
    99
command.
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   100
'''
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   101
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   102
from mercurial import commands
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   103
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   104
import lfcommands
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   105
import reposetup
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   106
import uisetup
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   107
17233
acea82757d8a largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents: 15743
diff changeset
   108
testedwith = 'internal'
acea82757d8a largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents: 15743
diff changeset
   109
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   110
reposetup = reposetup.reposetup
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   111
uisetup = uisetup.uisetup
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   112
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   113
commands.norepo += " lfconvert"
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   114
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   115
cmdtable = lfcommands.cmdtable