hgext/largefiles/__init__.py
author Augie Fackler <augie@google.com>
Tue, 23 Aug 2016 11:26:08 -0400
changeset 29841 d5883fd055c6
parent 29306 83cecc0b991f
child 32291 bd872f64a8ba
permissions -rw-r--r--
extensions: change magic "shipped with hg" string I've caught multiple extensions in the wild lying about being 'internal', so it's time to move the goalposts on people. Goalpost moving will continue until third party extensions stop trying to defeat the system.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     1
# Copyright 2009-2010 Gregory P. Ward
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     2
# Copyright 2009-2010 Intelerad Medical Systems Incorporated
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     3
# Copyright 2010-2011 Fog Creek Software
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     4
# Copyright 2010-2011 Unity Technologies
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     5
#
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     6
# This software may be used and distributed according to the terms of the
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     7
# GNU General Public License version 2 or any later version.
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     8
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     9
'''track large binary files
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    10
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    11
Large binary files tend to be not very compressible, not very
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    12
diffable, and not at all mergeable. Such files are not handled
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    13
efficiently by Mercurial's storage format (revlog), which is based on
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    14
compressed binary deltas; storing large binary files as regular
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    15
Mercurial files wastes bandwidth and disk space and increases
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    16
Mercurial's memory usage. The largefiles extension addresses these
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    17
problems by adding a centralized client-server layer on top of
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    18
Mercurial: largefiles live in a *central store* out on the network
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    19
somewhere, and you only fetch the revisions that you need when you
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    20
need them.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    21
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    22
largefiles works by maintaining a "standin file" in .hglf/ for each
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    23
largefile. The standins are small (41 bytes: an SHA-1 hash plus
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    24
newline) and are tracked by Mercurial. Largefile revisions are
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    25
identified by the SHA-1 hash of their contents, which is written to
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    26
the standin. largefiles uses that revision ID to get/put largefile
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    27
revisions from/to the central store. This saves both disk space and
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    28
bandwidth, since you don't need to retrieve all historical revisions
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    29
of large files when you clone or pull.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    30
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    31
To start a new repository or add new large binary files, just add
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    32
--large to your :hg:`add` command. For example::
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    33
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    34
  $ dd if=/dev/urandom of=randomdata count=2000
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    35
  $ hg add --large randomdata
28798
c065e0ec13d1 largefiles: use double quotes for arguments
timeless <timeless@mozdev.org>
parents: 28394
diff changeset
    36
  $ hg commit -m "add randomdata as a largefile"
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    37
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    38
When you push a changeset that adds/modifies largefiles to a remote
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    39
repository, its largefile revisions will be uploaded along with it.
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    40
Note that the remote Mercurial must also have the largefiles extension
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    41
enabled for this to work.
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    42
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    43
When you pull a changeset that affects largefiles from a remote
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    44
repository, the largefiles for the changeset will by default not be
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    45
pulled down. However, when you update to such a revision, any
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    46
largefiles needed by that revision are downloaded and cached (if
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    47
they have never been downloaded before). One way to pull largefiles
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    48
when pulling is thus to use --update, which will update your working
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    49
copy to the latest pulled revision (and thereby downloading any new
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    50
largefiles).
18704
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    51
18976
6734951e2d24 largefiles: introduce lfpull command for pulling missing largefiles
Mads Kiilerich <madski@unity3d.com>
parents: 18975
diff changeset
    52
If you want to pull largefiles you don't need for update yet, then
18978
8abaadab9abb largefiles: introduce pull --lfrev option
Mads Kiilerich <madski@unity3d.com>
parents: 18976
diff changeset
    53
you can use pull with the `--lfrev` option or the :hg:`lfpull` command.
18976
6734951e2d24 largefiles: introduce lfpull command for pulling missing largefiles
Mads Kiilerich <madski@unity3d.com>
parents: 18975
diff changeset
    54
19071
64ea454e7d76 largefiles: fix typos in documentation
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 18980
diff changeset
    55
If you know you are pulling from a non-default location and want to
64ea454e7d76 largefiles: fix typos in documentation
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 18980
diff changeset
    56
download all the largefiles that correspond to the new changesets at
18979
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    57
the same time, then you can pull with `--lfrev "pulled()"`.
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    58
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    59
If you just want to ensure that you will have the largefiles needed to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    60
merge or rebase with new heads that you are pulling, then you can pull
18979
1176832fc757 largefiles: introduce pulled() revset expression for use in --lfrev
Mads Kiilerich <madski@unity3d.com>
parents: 18978
diff changeset
    61
with `--lfrev "head(pulled())"` flag to pre-emptively download any largefiles
18704
d69585a5c5c0 largefiles: don't cache largefiles for pulled heads by default
Na'Tosha Bard <natosha@unity3d.com>
parents: 18599
diff changeset
    62
that are new in the heads you are pulling.
18599
5cd1dbf4c5d2 largefiles: document behavior of caching largefiles for new heads
Na'Tosha Bard <natosha@unity3d.com>
parents: 17233
diff changeset
    63
18975
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    64
Keep in mind that network access may now be required to update to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    65
changesets that you have not previously updated to. The nature of the
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    66
largefiles extension means that updating is no longer guaranteed to
aa8205a9f51a largefiles: update help
Mads Kiilerich <madski@unity3d.com>
parents: 18704
diff changeset
    67
be a local-only operation.
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    68
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    69
If you already have large files tracked by Mercurial without the
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    70
largefiles extension, you will need to convert your repository in
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    71
order to benefit from largefiles. This is done with the
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    72
:hg:`lfconvert` command::
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    73
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    74
  $ hg lfconvert --size 10 oldrepo newrepo
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    75
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    76
In repositories that already have largefiles in them, any new file
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    77
over 10MB will automatically be added as a largefile. To change this
15304
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    78
threshold, set ``largefiles.minsize`` in your Mercurial config file
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    79
to the minimum size in megabytes to track as a largefile, or use the
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    80
--lfsize option to the add command (also in megabytes)::
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    81
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    82
  [largefiles]
15304
9aa9d4bb3d88 largefiles: rename config setting 'size' to 'minsize'
Greg Ward <greg@gerg.ca>
parents: 15291
diff changeset
    83
  minsize = 2
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    84
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    85
  $ hg add --lfsize 2
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    86
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    87
The ``largefiles.patterns`` config option allows you to specify a list
15352
b74f74b482d8 largefiles: improve markup in module help text
Martin Geisler <mg@aragost.com>
parents: 15304
diff changeset
    88
of filename patterns (see :hg:`help patterns`) that should always be
15230
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    89
tracked as largefiles::
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    90
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    91
  [largefiles]
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    92
  patterns =
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    93
    *.jpg
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    94
    re:.*\.(png|bmp)$
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    95
    library.zip
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    96
    content/audio/*
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    97
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    98
Files that match one of these patterns will be added as largefiles
697289c5d415 largefiles: improve help
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
    99
regardless of their size.
15743
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   100
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   101
The ``largefiles.minsize`` and ``largefiles.patterns`` config options
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   102
will be ignored for any repositories not already containing a
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   103
largefile. To add the first largefile to a repository, you must
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   104
explicitly do so with the --large flag passed to the :hg:`add`
6266b1b970a5 largefiles: clarify help when options are ignored until first add is done
Michal Sznajder <michalsznajder@gmail.com>
parents: 15352
diff changeset
   105
command.
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   106
'''
29306
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   107
from __future__ import absolute_import
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   108
29306
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   109
from mercurial import (
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   110
    hg,
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   111
    localrepo,
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   112
)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   113
29306
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   114
from . import (
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   115
    lfcommands,
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   116
    overrides,
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   117
    proto,
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   118
    reposetup,
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   119
    uisetup as uisetupmod,
83cecc0b991f py3: make largefiles/__init__.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 28798
diff changeset
   120
)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   121
29841
d5883fd055c6 extensions: change magic "shipped with hg" string
Augie Fackler <augie@google.com>
parents: 29306
diff changeset
   122
# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
25186
80c5b2666a96 extensions: document that `testedwith = 'internal'` is special
Augie Fackler <augie@google.com>
parents: 21770
diff changeset
   123
# extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
80c5b2666a96 extensions: document that `testedwith = 'internal'` is special
Augie Fackler <augie@google.com>
parents: 21770
diff changeset
   124
# be specifying the version(s) of Mercurial they are tested with, or
80c5b2666a96 extensions: document that `testedwith = 'internal'` is special
Augie Fackler <augie@google.com>
parents: 21770
diff changeset
   125
# leave the attribute unspecified.
29841
d5883fd055c6 extensions: change magic "shipped with hg" string
Augie Fackler <augie@google.com>
parents: 29306
diff changeset
   126
testedwith = 'ships-with-hg-core'
17233
acea82757d8a largefiles: mark as a first party extension
Matt Harbison <matt_harbison@yahoo.com>
parents: 15743
diff changeset
   127
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   128
reposetup = reposetup.reposetup
19779
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   129
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   130
def featuresetup(ui, supported):
19928
d1ac3790e10a localrepo: invoke only feature setup functions for enabled extensions
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19779
diff changeset
   131
    # don't die on seeing a repo with the largefiles requirement
d1ac3790e10a localrepo: invoke only feature setup functions for enabled extensions
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19779
diff changeset
   132
    supported |= set(['largefiles'])
19779
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   133
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   134
def uisetup(ui):
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   135
    localrepo.localrepository.featuresetupfuncs.add(featuresetup)
20858
bc56ec9e64df hg: introduce "wirepeersetupfuncs" to setup wire peer by extensions (issue4109)
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19928
diff changeset
   136
    hg.wirepeersetupfuncs.append(proto.wirereposetup)
19779
fb6e87d93948 largefiles: setup "largefiles" feature in each repositories individually
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19071
diff changeset
   137
    uisetupmod.uisetup(ui)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   138
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   139
cmdtable = lfcommands.cmdtable
28394
dcb4209bd30d revset: replace extpredicate by revsetpredicate of registrar
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25186
diff changeset
   140
revsetpredicate = overrides.revsetpredicate