py3: use print as a function in test-convert-git.t Differential Revision: https://phab.mercurial-scm.org/D3278

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D3277

py3: prevent transformer from adding b'' by adding r'' prefix These are cases where we need to use str, therefore we add r'' prefix. # skip-blame because just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3276

py3: use pycompat.{strkwargs|byteskwargs} in infinitepush Differential Revision: https://phab.mercurial-scm.org/D3275

py3: make sure we open file in bytes mode Differential Revision: https://phab.mercurial-scm.org/D3274

diffhelpers: be more tolerant for stripped empty lines of CRLF ending Exchange. It appears to trim lines containing only whitespace as well as converting LF to CRLF.

diffhelpers: make return value of testhunk() more Pythonic It's no longer C.

patch: error out if reached to EOF while reading hunk This was where out-of-bounds read occurred in old C extension.

diffhelpers: remove unused return value from fixnewline() and addlines()

diffhelpers: move out of pure package

diffhelpers: naming and whitespace cleanup

diffhelpers: remove C implementation in favor of pure Python version

patch: stop using cext.diffhelpers The C implementation has a couple of memory bugs, and lacks error handling which could lead to SEGV. I could fix them one by one (and I mostly finished that), but the performance gain provided by cext.diffhelper is quite low. Besides, diffhelpers.addlines() calls back Python, linereader.readline(), from the innermost loop. $ hg export -R mozilla-central 0:100 > patch $ ls -lh patch -rw-r--r-- 184M patch $ hg init repo && hg -R repo import patch --time --bypass (cext) time: real 34.970 secs (user 32.720+0.000 sys 2.230+0.000) (pure) time: real 35.950 secs (user 33.600+0.000 sys 2.330+0.000) So, let's simply use the pure Python implementation.

diffhelpers: port docstrings from cext to pure I'll remove the C implementation.

py3: get rid of character access from pure.diffhelpers 's' is a result of readline(), so 'c == "\n"' means 's == "\n"'.

lfs: handle paths that don't end with '/' when inferring the blob store While here, I also checked the lfs.url config directly instead of testing the scheme, as requested by Yuya.

lfs: infer the blob store URL from an explicit push dest or default-push Unlike pull, the blobs are uploaded within the exchange.push() window, so simply wrap it and swap in a properly configured remote store. The '_subtoppath' field shouldn't be available during this window, but give the passed path priority for clarity. At one point I hit an AttributeError in one of the convert tests when trying to save the original remote blobstore when the swap was run unconditionally. I wrapped it in a util.safehasattr(), but then today I wasn't able to reproduce it. But now the whole thing is tucked under the requirement guard because without the requirement, there are no blobs in the repo, even if the extension is loaded.

lfs: infer the blob store URL from an explicit pull source I don't see any easier way to do this because the update part of `hg pull -u` happens outside exchange.pull(), and commands.postincoming() doesn't take a path. So (ab)use the mechanism used by subrepos to redirect where subrepos are pulled from when an explicit path is given. As a bonus, this should allow lfs blobs to be pulled into a subrepo when it is checked out. An explicit push path can be handled within exchange.push(). That can be done next, outside of this dirty hack.

lfs: special case the null:// usercache instead of treating it as a url The previous code worked on Windows, but not on Unix, and a pending patch's test failed. The url being used was something like "/tmp/.../client1/null://", courtesy of ui.configpath(). Looking at the doc comment, this seems like it's maybe not the right function to call (why should a relative cache path be expanded relative to the repo root or config file?), but largefiles has been using it since 8b8dd13295db (Oct 2011). It was introduced in 1b591f9b7fd2 (Jan 2011) without comment or callers. A grep over the whole history shows that only largefiles used it until lfs and infinitepush came along recently. It looks like if the `if not os.path.isabs(v) or "://" not in v` in configpath() is changed to an 'and', both Linux and Windows are happy. I'm guessing that "://" is to pick off URLs, so that seems reasonable. But I'm not sure why it isn't explicitly "file://", and I thought that "file://foo" is relative anyway. (At least, there are doctests for file:///tmp in util.url.) There is no mention of this setting in the help, but it is referenced on the wiki page for largefiles. (There's no mention that this is intended to be a URL, and the example uses an absolute path.) I don't want this blocking the rest of the lfs server discovery stuff. It was also wrong to allow a file:// URL here, but not in largefiles.

tests: add tests showing pulling from infinitepush works over wire The current tests in test-infinitepush-ci.t showed that `hg pull -r <rev>` does not work. Digging in code, I found that we have logic for pulling from bundlestore without having client side logic. This patch adds test demonstrating that pulling from bundlestore works when working over wire. Pulling from bundlestore when the peer is a localpeer still does not works. Differential Revision: https://phab.mercurial-scm.org/D3072

fix: use a portable python script instead of sed in test Differential Revision: https://phab.mercurial-scm.org/D2988

py3: use pycompat.bytestr() where repr in involved Differential Revision: https://phab.mercurial-scm.org/D3244

httppeer: support protocol upgrade With the new handshake defined and in place on the server, we can now implement it on the client. The HTTP handshake mechanism has been taught to add headers advertising its support for the new capabilities response. Response handling has been adjusted to allow CBOR responses through. And makepeer() has been taught to instantiate a mutually supported peer. The HTTPv2 peer class doesn't implement the full peer interface. So HTTPv2 is not yet usable as a peer. Like the server side, we support registering handlers for different API services. This allows extensions to easily implement API services and peers. A practical use case for this is to provide a previous implementation of the experimental version 2 wire protocol to a future version of Mercurial. We know there will be BC breaks after 4.6 ships. But someone could take the peer and server code from 4.6, drop it in an extension, and allow its use indefinitely. Differential Revision: https://phab.mercurial-scm.org/D3243

wireproto: define and implement HTTP handshake to upgrade protocol When clients connect to repositories over HTTP, they issue a request to the well-known URL "?cmd=capabilities" to fetch the repository capabilities. This is the handshake portion of the HTTP protocol. This commit defines a mechanism to use that HTTP request to return information about modern server features. If a client sends an X-HgUpgrade-* header containing a list of client-supported API names, the server responds with a response containing information about available services. This includes the normal capabilities string. So if the server doesn't support any newer services, the client can easily fall back. By advertising supported services from clients, server operators can see and log what client support exists in the wild. This will also help with debugging. The response contains the base path to API services. We know there are potential issues with the <repo>/api/ URL space conflicting with hgwebdir and subrepos. By making the API URL dynamic from the perspective of the client, the URL for APIs is not subject to backwards compatibility concerns - at least as long as a ?cmd=capabilities request is made. We've also defined the ``cbor`` client capability for the X-HgProto-* header. This MUST be sent in order to get the modern response from "?cmd=capabilities". During implementation, I initially always sent an application/mercurial-cbor response. However, the handshake mechanism will be more future compatible if the client is in charge of which formats to request. We already perform content negotiation from X-HgProto-*, so keying off this for the capabilities response feels appropriate. In addition, I initially used application/cbor. However, it is conceivable that a non-Mercurial server could serve application/cbor. To rule out this possibility, I've invented a new media type that is Mercurial specific and can't be confused for generic CBOR. Differential Revision: https://phab.mercurial-scm.org/D3242

httppeer: only advertise partial-pull if capabilities are known We don't need to be advertising client protocol parameters as part of the capabilities request during the handshake because nothing in version 1 of the wire protocol will use this data. i.e. the advertisement is wasteful. Differential Revision: https://phab.mercurial-scm.org/D3241

httppeer: always add x-hg* headers to Vary header Before, we manually updated the Vary header value for each header contributing to it. All X-Hg* headers are reserved for the Mercurial protocol and could have caching implications. So it makes sense to always add these headers to Vary. A test revealed that X-HgArgs-Post wasn't being added to Vary. This is only sent on POST requests. POST requests generally aren't cacheable. However, it is possible if the server sends the appropriate headers. Mercurial shouldn't be sending those headers. But let's not take any chances. Differential Revision: https://phab.mercurial-scm.org/D3240

httppeer: don't accept very old media types (BC) Versions of Mercurial older than 1.0 emitted the text/plain and application/hg-changegroup media types in response to wire protocol commands. Way back in 8760d0c83b9b in 2005, the code validating these media types was added, presumably for backwards compatibility. 0b245edec124 a short time before that commit changed things from text/plain and application/hg-changegroup to application/mercurial-0.1 and application/hg-0.1. 8760d0c83b9b seemed to indicate ("for now") that the BC compatibility was temporary. But that code has lived until this day. It has been more than 10 years and nobody should be running pre 1.0 servers. Pretty much the only risk to this is if there's a server somewhere advertising the old media types or server software is interfering and not letting Mercurial send the proper Content-Type header. I think the chances are rare. The wire protocol docs were created (by me) from reading existing code. So the deletions don't constitute a spec change as much as reflecting the reality of how things have been for years. .. bc:: The HTTP client no longer accepts text/plain and application/hg-changegroup Content-Type values as a valid Mercurial command response. These should only be encountered on pre 1.0 Mercurial servers. Differential Revision: https://phab.mercurial-scm.org/D3239

httppeer: allow opener to be passed to makepeer() This allows us to use makepeer() in `hg debugwireproto`. Differential Revision: https://phab.mercurial-scm.org/D3238

httppeer: perform capabilities request in makepeer() Previously, we constructed an httppeer then always ran _fetchcaps() to issue the capabilities command. We want to issue the capabilities command before constructing a peer instance so we can construct an appropriate peer instance depending on the capabilities result. With the code for making and sending requests moved out of httppeer, it is now possible to send command requests without an httppeer. This commit creates a new function for making the capabilities request and calls it as part of makepeer(). This code should be functionality equivalent to what existed before. Differential Revision: https://phab.mercurial-scm.org/D3237

httppeer: extract common response handling into own function This allows the common redirect detection, content type validation, and decompression wrapping to be usable outside of httppeer instances. Differential Revision: https://phab.mercurial-scm.org/D3236

httppeer: move error handling and response wrapping into sendrequest This is common for all HTTP requests. It should be part of sendrequest(). Differential Revision: https://phab.mercurial-scm.org/D3235

httppeer: extract code for creating a request into own function Some of this feels awkward, such as having to pass in a function to evaluate a capability. And this code is generally pretty difficult to read. I didn't want to perform too much refactoring as part of the code move since it would make review more difficult. Differential Revision: https://phab.mercurial-scm.org/D3234

httppeer: extract code for performing an HTTP request This is generic and doesn't need to live as a method of httppeer. Differential Revision: https://phab.mercurial-scm.org/D3233

httppeer: move requestbuilder defaults into makepeer() argument Upcoming commits will move the initial ?cmd=capabilities handshake request out of httppeer so the handshake can be performed before a peer instance is constructed. In order to do this, we'll need to refactor code for making HTTP requests. The type used to construct HTTP requests is configurable. If we'll be making HTTP requests outside of httppeer, we should be able to use a custom request builder. So move the definition of that type into makepeer(). Extensions can monkeypatch the function and override the argument value. Differential Revision: https://phab.mercurial-scm.org/D3232

wireproto: move version 2 command handlers to wireprotov2server This is relatively straightforward. As part of this, we introduced a local @wireprotocommand that wraps the main one and defines a v2 only policy by default. Because the hacky HTTPv2 peer isn't using capabilities response yet, we had to move some code around to force import of wireprotov2server so commands are registered. This is super hacky. But this code will go away once the HTTPv2 peer is using the capabilities response to derive permissions. Differential Revision: https://phab.mercurial-scm.org/D3231

wireproto: extract HTTP version 2 code to own module wireprotoserver has generic and version 1 server code. The wireproto module also has both version 1 and version 2 command implementations. Upcoming work I want to do will make it difficult for this code to live in the current locations. Plus, it kind of makes sense for the version 2 code to live in an isolated module. This commit copies the HTTPv2 bits from wireprotoserver into a new module. We do it as a file copy to preserve history. A future commit will be copying wire protocol commands into this module as well. But there is little history of that code, so it makes sense to take history for wireprotoserver. Differential Revision: https://phab.mercurial-scm.org/D3230

wireproto: client reactor support for receiving frames We can now feed received frames into the client reactor and it will validate their sanity, dispatch them appropriately. The hacky HTTP peer has been updated to use the new code. No existing tests changed, somewhat proving the code works as expected. Rudimentary unit tests for the new functionality have been implemented. Differential Revision: https://phab.mercurial-scm.org/D3224

wireproto: introduce a reactor for client-side state We have a nice state machine of sorts for reacting to server-side events. Now it is time to implement the client equivalent. We introduce a "clientreactor." It allows callers to request that commands be issued. It has multiple modes of operation to reflect what the underlying transport supports. e.g. for SSH, we can perform wire sends immediately but for HTTP we need to buffer sends until all command requests are received. In addition, SSH allows sending multiple requests as long as the connection is open. But HTTP/1.1 only allows sending request data once. For SSH, we'll have one reactor per connection. For HTTP, we'll have one reactor per HTTP request. But because code that calls wire protocol commands should not be aware of how the underlying transport works, this will all be abstracted away by the peer interface. Our crude HTTP peer has been updated to use the reactor instead of formulating frames directly. No behavior should have changed here and tests seem to confirm that. Basic unit tests for the reactor behavior have been added. Differential Revision: https://phab.mercurial-scm.org/D3223

tests: extract wire protocol framing tests to own file I was lazy when I put these in test-wireproto-serverreactor.py. Let's do it properly. Differential Revision: https://phab.mercurial-scm.org/D3222

wireproto: disallow commands handlers for multiple transport versions I think it will be more trouble than it is worth to code version 1 and version 2 command handlers to the same interface. It will feel awkward to shoehorn functionality into e.g. the version 1 protocol handler interface. This would likely constrain the ability for version 2 to evolve. Previous commits introduced a clean separation between command handlers for version 1 and version 2 transports. This commit reinforces that separation by dropping support for having a single command handler service both version 1 and version 2 transports. Differential Revision: https://phab.mercurial-scm.org/D3208

wireproto: make @wireprotocommand version 1 only by default For backwards compatibility reasons. We want extension provided commands to opt in to version 2 rather than get inherited automatically. This will facilitate a clean break between the protocols. As part of this, we duplicate some commands used in tests so there are different command handlers per transport. Differential Revision: https://phab.mercurial-scm.org/D3207

wireproto: only expose "getbundle" and "unbundle" to v1 transports These are the most complicated wire protocol commands. I don't want to deal with porting them just yet. Let's disable both of them on version 2 transports so we drive the final wedge between command handlers and start to evolve version 2 command handlers more. Differential Revision: https://phab.mercurial-scm.org/D3206

wireproto: port lookup to wire protocol v2 This is pretty straightforward. We don't yet handle errors because we don't have an error handling mechanism in place yet. I'm also tempted to fold this into `known`. We'll come back to this later. Differential Revision: https://phab.mercurial-scm.org/D3205

wireproto: port pushkey command to wire protocol version 2 It doesn't do output redirection yet. And I'd love to generally overhaul the pushkey protocol for wire protocol version 2. But this will be a bit of effort. Let's do it as a follow-up. Differential Revision: https://phab.mercurial-scm.org/D3204

wireproto: only expose "clonebundles" to version 1 transports This may make a comeback in wire protocol version 2. The feature definitely needs to be carried forward. But at this juncture, I'm flirting with the idea of implementing this via a "redirect" mechanism at the command response level itself rather than something that requires one-off client support for querying and handling. i.e. I want to make it so servers can say "fetch this first and then come back" and clients handle that automatically. This would not only support clone bundles, but would also support piece-meal "pull bundles." Whatever happens, we can deal with it down the road. Differential Revision: https://phab.mercurial-scm.org/D3203

wireproto: define and expose types of wire command arguments Exposing the set of argument names is cool. But with wire protocol version 2, we're using CBOR to transport arguments and this allows us to have typing for arguments. Typed arguments are much nicer because they will cut down on transfer overhead and processing overhead for decoding values. This commit teaches @wireprotocommand to accept a dictionary for arguments. The arguments registered for version 2 transports are canonically stored as dictionaries rather than a space-delimited string. It is an error to defined arguments with a dictionary for commands using version 1 transports. This reinforces my intent to fully decouple command handlers for version 2 transports. Differential Revision: https://phab.mercurial-scm.org/D3202

wireproto: only expose "stream_out" to version 1 transports I have plans to implement stream clone using a better mechanism than this existing command. Let's not carry it forward to wire protocol version 2. Differential Revision: https://phab.mercurial-scm.org/D3201

wireproto: implement capabilities for wire protocol v2 The capabilities mechanism for wire protocol version 2 represents a clean break from version 1. Instead of effectively exchanging a set of capabilities, we're exchanging a rich data structure. This data structure currently contains information about every available command, including its accepted arguments. It also contains information about supported compression formats. Exposing information about supported commands will allow clients to automatically generate bindings to the server. Clients will be able to do things like detect when they are attempting to run a command that isn't known to the server. Exposing the required permissions to run a command can be used by clients to determine if they have privileges to call a command before actually calling it. We could potentially even have clients send credentials preemptively without waiting for the server to deny the command request. Lots of potential here. The data returned by this command will likely evolve heavily. So we shouldn't bikeshed the implementation just yet. Differential Revision: https://phab.mercurial-scm.org/D3200

context: add deprecation warnings for deprecated types of changeids It's close to code freeze, and dropping support for repo['123'] and repo ['my-bookmark'] and repo['deadbeef'] is pretty dispruptive, so this just adds deprecation warnings so extensions can easily find the places they need to fix. Differential Revision: https://phab.mercurial-scm.org/D3197

revsymbol: stop delegating to repo.__getitem__ for unhandled symbols (API) The only remaining cases where we were delegating unhandled symbols to repo.__getitem__ should now be when the symbol could not be found. In that case we just delegated to repo.__getitem__ for the error message. Let's just copy the error message instead. If there were any cases where we got e.g. a binary nodeid or an integer revnum into revsymbol() (e.g. via repo.lookup()), we'd now start raising an exception instead. That is why this is marked (API). This affects one test case, but the new behavior seems better to me. I can't tell if the old behavior was desired or if the test was just there to document how it happened to work. Differential Revision: https://phab.mercurial-scm.org/D3196

context: handle partial nodeids in revsymbol() Similar reasoning as previous patches. Differential Revision: https://phab.mercurial-scm.org/D3195

context: handle namespaces in revsymbol() Similar reasoning as previous patches. Differential Revision: https://phab.mercurial-scm.org/D3194

scmutil: handle full hex nodeids in revsymbol() This is a bit unfortunate, but it enables moving other pieces out of changectx's constructor without affecting the order in which we look up things (e.g. hex nodeid before bookmark). We convert nodeid to revnum before calling repo.__getitem__, even though that will result in converting back to nodeid later. This is so we can handle the LookupError and attempt to interpret the string as something else (e.g. a bookmark). We also need to start handling WdirUnsupported now, since the full hex nodeid "ffff..." represents the working directory. The exception is raised by the revlog layer. Differential Revision: https://phab.mercurial-scm.org/D3193

context: handle stringified ints in revsymbol() This patch copies the handling of stringified ints from changectx's constructor. It then calls repo.__getitem__ with the int. Since that method only interprets integers as revnums the first thing it does, this will not be redoing any of the work already done. We leave the old code in place so we can later deprecate it instead of breaking extensions. Differential Revision: https://phab.mercurial-scm.org/D3146

py3: make sure we write bytes to file # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D3220

py3: workaround comparing NoneType and integers Comparing None with integers was fine in Python 2 but returns error in Python 3 which is nice. This patch replaces None with -1 where sorting is done and some related logic. Differential Revision: https://phab.mercurial-scm.org/D3219

py3: use sys.stdout instead of print in test-mq-qpush-fail.t Differential Revision: https://phab.mercurial-scm.org/D3218

py3: use '//' for integer division in tests/test-mq-qimport.t Differential Revision: https://phab.mercurial-scm.org/D3217

py3: use pycompat.byterepr() instead of repr() Differential Revision: https://phab.mercurial-scm.org/D3216

py3: use bytes instead of str in isinstance() We deal with bytes internally and things should be bytes Differential Revision: https://phab.mercurial-scm.org/D3215

py3: use .startswith() instead of bytes[0] Differential Revision: https://phab.mercurial-scm.org/D3214

py3: add b'' prefixes in tests/test-ui-config.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3221

lfs: infer the blob store URL from paths.default If `lfs.url` is specified, it takes precedence. However, now that we support serving blobs via hgweb, we shouldn't *require* this setting. Less configuration is better (things will work out of the box once this is sorted out), and git has similar functionality. This is not a complete solution- it isn't able to infer the blob store from an explicitly supplied path, and it should consider `paths.default-push` for push. The pull solution for that is a bit hacky, and this alone is an improvement for the vast majority of cases. Even though there are only a handful of references to the saved remote store, the location of them makes things complicated. 1) downloading files on demand in the revlog flag processor 2) copying to readonlyvfs with bundlerepo 3) downloading in the file prefetch hook 4) the canupload()/skipdownload() checks 5) uploading blobs Since revlog doesn't have a repo or ui reference, we can't avoid creating a remote store when the extension is loaded. While the long term goal is to make sure the prefetch hook is invoked early for every command for efficiency, this handling in the flag processor is needed as a last ditch fetch. In order to support the clone command, the remote store needs to be created later than when the extension loads, since `paths.default` isn't set until just before the files are checked out. Therefore, this patch changes the prefetch hook to ignore the saved reference, and build a new one. The canupload()/skipdownload() checks simply check if the stored instance is a `_nullremote`. Since this can only be set via `lfs.url` (which is reflected in the saved reference), checking only the instance created when the extension loaded is fine. The blob uploading function is called from several places: 1) a prepush hook 2) when writing a new bundle 3) from infinitepush The prepush hook gets an exchange.pushop, so it has a path to where the push is going. The bundle writer and infinitepush don't. Further, bundle creation for things like strip and amend are causing blobs to be uploaded. This seems wrong, but I don't want to side track this sorting that out, so punt on trying to handle explicit push paths or `paths.default-push`. I also think that sending blobs to a remote store when pushing to a local repo is wrong. This functionality predates the usercache, so perhaps that's the reason for it. I've got some patches floating around to stop sending blobs remotely in this case, and instead write directly to the other repo's blob store. But the tests for corruption handling weren't happy with this change, and I don't have time to rewrite them. So exclude filesystem based paths from this for now. I don't think there's much of a chance to implement `paths.remote:lfsurl` style configs, given how early these are resolved vs how late the remote store is created. But git has it, so I threw a TODO in there, in case anyone has ideas. I have no idea why this is now doing http auth twice when it wasn't before. I don't think the original blobstore's url is ever being used in these cases.

lfs: add the ability to disable the usercache While the usercache is important for real world uses, I've been tripped up more than a couple of times by it in tests- thinking a file was being downloaded, but it was simply linked from the local cache. The syntax for setting it is the same as for setting a null remote endpoint, and like that endpoint, is left undocumented. This may or may not be a useful feature in the real world (I'd expect any sane filesystem to support hardlinks at this point).

tests: stabilize test-pull-bundle.t for Windows See 594dd384803c.

hgweb: wrap {archives} with mappinglist No bare list of mappings should be put in a template mapping.

hgweb: forward archivelist() of hgweb to webutil self.configlist() is ui.configlist(untrusted=True), and url=None in templater effectively means 'url' is undefined.

hgweb: move archivelist() of hgwebdir to webutil

hgweb: drop archivespecs from requestcontext It's a constant.

hgweb: move archivespecs to webutil Move one step further from e38e7ea21987. Since 'archivespecs' is a constant shared with hgweb and hgwebdir, webutil is a better place to define it.

hgweb: wrap {labels} by hybridlist() This one is a simple list of strings, which can support map operation.

hgweb: wrap {pathdef} with mappinglist No bare list of mappings should be put in a template mapping.

hgwebdir: wrap {entries} with mappinggenerator No bare generator should be put in a template mapping.

eol: look up partial nodeid as partial nodeid Similar reasoning as the previous patch. For some reason the hook gives us a partial nodeid, so we need to resolve that to a full nodeid. Differential Revision: https://phab.mercurial-scm.org/D3159

histedit: look up partial nodeid as partial nodeid I'm about to remove support for repo[<partial hex nodeid>]. In the verify() method, we know that self.node is always a partial or full binary nodeid, so the most correct way to look up the revision is by using changelog._partialmatch(), so let's do that. (It's closer to the current code to do scmutil.revsymbol(), but that's less correct because it will match a bookmark or tag that happens to have the same prefix as the node.) Differential Revision: https://phab.mercurial-scm.org/D3158

histedit: drop unnecessary check for "self.node is not None" We are doing hex(self.node) just a few lines up, so it shouldn't be None. The only way it could be none is if it was reassigned in between. The only way that can happen is if the user had put a "ffff..." wdirhex revision in the histedit script. This code is much older than the "ffff..." identifier, so I'm confident it's not there to handle that case. I'll let someone else add proper checks for "ffff..." if they care enough. Differential Revision: https://phab.mercurial-scm.org/D3157

context: extract partial nodeid lookup method to scmutil We will add another caller soon, and there's a non-obvious reason to use the unfiltered repo that we don't want to copy across the code base. Differential Revision: https://phab.mercurial-scm.org/D3189

templatekw: fix return type of {succsandmarkers} (BC) A hybrid object represents a list/dict of values, but {succsandmarkers} returns a list of template mappings. This change means old-style list templates (e.g. "start_succsandmarkers") are no longer supported, but that should be okay since {succsandmarkers} is still experimental and relatively new.

formatter: remove template resources from nested items before generating JSON

templater: add public interface returning a set of resource keys The next patch depends on knownresourcekeys(), and there's no reason to keep availableresourcekeys() private.

formatter: make nested items somewhat readable in template output

templater: wrap result of '%' operation so it never looks like a thunk This fixes min/max()/json() of map result. Before, it was taken as a lazy byte string and stringified by evalfuncarg().

wireproto: support for pullbundles Pullbundles are similar to clonebundles, but served as normal inline bundle streams. They are almost transparent to the client -- the only visible effect is that the client might get less changes than what it asked for, i.e. not all requested head revisions are provided. The client announces support for the necessary retries with the partial-pull capability. After receiving a partial bundle, it updates the set of revisions shared with the server and drops all now-known heads from the request list. It will then rerun getbundle until no changes are received or all remote heads are present. Extend badserverext to support per-socket limit, i.e. don't assume that the same limits should be applied to all sockets. Differential Revision: https://phab.mercurial-scm.org/D1856

filelog: wrap revlog instead of inheriting it (API) The revlog base class exposes a ton of methods. Inheriting the revlog class for filelog will make it difficult to expose a clean interface. There will be abstraction violations. This commit breaks the inheritance of revlog by the filelog class. Filelog instances now contain a reference to a revlog instance. Various properties and methods are now proxied to that instance. There is precedence for doing this: manifestlog does something similar. Although, manifestlog has a cleaner interface than filelog. We'll get there with filelog... The new filelog class exposes a handful of extra properties and methods that aren't part of the declared filelog interface. Every extra item was added in order to get a test to pass. The set of tests that failed without these extra proxies has significant overlap with the set of tests that don't work with the simple store repo. There should be no surprise there. Hopefully the hardest part about this commit to review are the changes to bundlerepo and unionrepo. Both repository types define a custom revlog or revlog-like class and then have a custom filelog that inherits from both filelog and their custom revlog. This code has been changed so the filelog types don't inherit from revlog. Instead, they replace the revlog instance on the created filelog. This is super hacky. I plan to fix this in a future commit by parameterizing filelog.__init__. Because Python function call overhead is a thing, this change could impact performance by introducing a nearly empty proxy function for various methods and properties. I would gladly measure the performance impact of it, but I'm not sure what operations have tight loops over filelog attribute lookups or function calls. I know some of the DAG traversal code can be sensitive about the performance of e.g. parentrevs(). However, many of these functions are implemented on the revlog class and therefore have direct access to self.parentrevs() and aren't going through a proxy. .. api:: filelog.filelog is now a standalone class and doesn't inherit from revlog. Instead, it wraps a revlog instance at self._revlog. This change was made in an attempt to formalize storage APIs and prevent revlog implementation details leaking through to callers. Differential Revision: https://phab.mercurial-scm.org/D3154

util: drop write_content_size=True This is now the default in python-zstandard 0.9. While we're here, also add a comment about the ability to drop frame magic to save space. Differential Revision: https://phab.mercurial-scm.org/D3199

zstandard: vendor python-zstandard 0.9.0 This was just released. It features a number of goodies. More info at https://gregoryszorc.com/blog/2018/04/09/release-of-python-zstandard-0.9/. The clang-format ignore list was updated to reflect the new source of files. The project contains a vendored copy of zstandard 1.3.4. The old version was 1.1.3. One of the changes between those versions is that zstandard is now dual licensed BSD + GPLv2 and the patent rights grant has been removed. Good riddance. The API should be backwards compatible. So no changes in core should be needed. However, there were a number of changes in the library that we'll want to adapt to. Those will be addressed in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D3198

revlog: reset _nodepos after strip When using the pure revlog parser, _nodepos is used to keep track of the position during index scanning in the non-cached cache. If it is out of bounds, BaseIndexObject._fix_index will assert. Since strip can actually remove the position scanned last, make sure to reset it. Add an assertion in the place where the invariance is clearer. Differential Revision: https://phab.mercurial-scm.org/D3188

paper: make all source lines have the same minimum height Empty source lines in paper and coal themes used to have smaller height than every other line (because of the way line numbers are shown and because they are using smaller font). This wasn't very noticeable before the follow lines functionality was added, but after that just using the follow-lines button to select a block of code with empty lines would demonstrate the fact that empty lines didn't have enough height - there were white "gaps" in the selection block. Since this problem occurs when lines don't have any content inside, let's create a pseudo-element (it's unselectable because of that) which still doesn't have any content, but fills up empty lines to 100% of their height because of display: inline-block. This is the most natural way to solve this annoyance that I've found so far. Hardcoding height isn't useful because we can have wrapped lines, in which case multiple lines of text need to fit into a single <span>. Setting min-height or line-height doesn't remove the gaps when viewed in Chromium.

hgweb: make followlines button absolutely positioned It used to have position: absolute only on annotate page, but it makes sense to have it everywhere, because the button shouldn't affect other elements at all. Especially since the button has a set height, which meant that for certain smaller fonts source lines were changing their height on hover. Note that the button doesn't set any of the usual properties that accompany absolute position (top, right, bottom or left). These properties would position the button without any account for source line padding. Instead, margins are used (the button already has all margins defined, they do the job).

hgweb: insert followlines buttons before any children, including text nodes This way the buttons come before any other content, including text nodes. Because highlight extension replaces every line of text with some <span> elements that have CSS classes for highlighting, the placement of followlines buttons used to depend on if that extension was enabled or not. Let's make the placement more consistent, it'll help the next patch in this series.

wireproto: only expose "debugwireargs" to version 1 transports I'm not even sure this command should be enabled for version 1 transports. It is just a reflection endpoint for argument data. We definitely don't need to support it in version 2. Differential Revision: https://phab.mercurial-scm.org/D3184

wireproto: only expose "hello" command to version 1 transports This command is only ever used for the handshake in the SSH protocol. We probably don't even need for it to be a proper command. Let's not carry it forward to version 2 because I don't see a use for it there. Differential Revision: https://phab.mercurial-scm.org/D3183

wireproto: port branchmap to wire protocol v2 Differential Revision: https://phab.mercurial-scm.org/D3182

wireproto: port listkeys commands to wire protocol v2 Differential Revision: https://phab.mercurial-scm.org/D3181

wireproto: port keep command to wire protocol v2 This is pretty straightforward. Differential Revision: https://phab.mercurial-scm.org/D3180

wireproto: port heads command to wire protocol v2 After much thought and consideration, wire protocol version 2's commands will be defined in different functions from the existing commands. This will make it easier to implement these commands because it won't require shoehorning things like response formatting and argument declaration into the same APIs. For example, wire protocol version 1 requires that commands declare a fixed and ordered list of argument names. It isn't really possible to insert new arguments or have optional arguments without breaking backwards compatibility. Wire protocol version 2, however, uses CBOR maps for passing arguments. So arguments a) can be optional b) can be added without BC c) can be strongly typed. This commit starts our trek towards reimplementing the wire protocol for version 2 with the heads command. It is pretty similar to the existing heads command. One added feature is it can be told to operate on only public phase changesets. This is useful for making discovery faster when a repo has tens of thousands of draft phase heads (such as Mozilla's "try" repository). The HTTPv2 server-side protocol has had its `getargs()` implementation updated to reflect that arguments are a map and not a list. Differential Revision: https://phab.mercurial-scm.org/D3179

largefiles: wrap heads command handler more directly extensions.wrapfunction() is a more robust method for wrapping a function, since it allows multiple wrappers. While we're here, wrap the function registered with the command instead of installing a new command handler. Differential Revision: https://phab.mercurial-scm.org/D3178

wireproto: crude support for version 2 HTTP peer As part of implementing the server-side bits of the wire protocol command handlers for version 2, we want a way to easily test those commands. Currently, we use the "httprequest" action of `hg debugwireproto`. But this requires explicitly specifying the HTTP request headers, low-level frame details, and the data structure to encode with CBOR. That's a lot of boilerplate and a lot of it can change as the wire protocol evolves. `hg debugwireproto` has a mechanism to issue commands via the peer interface. That is *much* easier to use and we prefer to test with that going forward. This commit implements enough parts of the peer API to send basic requests via the HTTP version 2 transport. The peer code is super hacky. Again, the goal is to facilitate server testing, not robustly implement a client. The client code will receive love at a later time. Differential Revision: https://phab.mercurial-scm.org/D3177

tests: extract wire protocol shell helpers to standalone file This will make it easier for other tests to get up and running without the boilerplate. Differential Revision: https://phab.mercurial-scm.org/D3176

addbranchrevs: no longer accept revset as "revs" (API) The only caller was removed in 0b4692b9646d (bundle: avoid repo.lookup() for converting revnum to nodeid, 2018-04-02). Differential Revision: https://phab.mercurial-scm.org/D3192

clone: avoid using repo.lookup() with binary nodeid The code in hg.clone() is a bit of a mess, but it seems like the "checkout" variable is always a binary nodeid (tests pass when run with "assert len(checkout) == 20" before the repo.lookup()). repo.lookup() will soon work only with string inputs, so we need remove this use. Differential Revision: https://phab.mercurial-scm.org/D3191

merge: avoid unnecessary conversion from binary nodeid to binary nodeid "node" is already a binary nodeid here, so there's no need to convert it. repo.lookup() will soon lose support for binary nodeids as input. Differential Revision: https://phab.mercurial-scm.org/D3190

tests: stabilize test-push-http.t for Windows See 594dd384803c.

tests: add a substitution pattern for HTTP error log dates This will make tests with error log content less of a nuisance. I didn't roll it into to $LOGDATE$ because the regex got long and line wrapped. It didn't seem worth the obscurity.

wireproto: convert python literal to object without using unsafe eval() Follows up cc5a040fe150. At this point, I don't think we need a real eval(). If we want to support a set literal, maybe we can vendor ast.literal_eval(), which is relatively simple function.

tests: quote variable passed to shell test command

py3: system-stringify repr(frame) That's the Py3 requirement.

wireproto: show unknown id and flags in repr(frame) Perhaps we'll want it for debugging.

wireproto: fix repr(frame) to not crash by unknown type id Follows up 5ef2da00e935.

py3: use s.startswith() instead of s[n] while parsing patches I know 'bytes[n] in bytes' magically works, but I'm tired of finding which one breaks the tests.

py3: do not try to byte-stringify None in cmdutil.tryimportone() It's a debug message, so just use '' instead.

py3: work around weird handling of bytes/unicode in decode_header() Basically decode_header() works as follows, and on Python 3, email headers ARE UNICODE. def decode_header(header): if not ecre.search(header): # ecre is unicode regexp return [(header, None)] # so header is unicode string ... decode header into [(bytes_data, unicode_charset_name)] return collapsed

py3: use system string to access email headers

py3: fix string issues of email message in test-import.t - payload can be bytes - headers must be unicode on Python 3 - need to call msg.as_bytes() on Python 3, but msg.as_string() on Python 2, where bytes(msg) magic works

py3: use lower-cased module 'email.message' in test-import.t

py3: drop b'' from error message of fancyopts

procutil: drop unused 'newlines' option from popen*() (API) It's unlikely for us to use the universal_newlines option.

procutil: make explainexit() simply return a message (API) Almost all callers want it.

procutil: do not convert return code of signal exit to positive number (API) The docstring states that "codes from kill are negative", and it doesn't make sense to make exit/signal code ambiguous. .. api:: ``hook.hook()`` and ``hook.runhooks()`` may return a negative integer to denote that the process was killed by signal.

procutil: fix error message of tempfile filter First, we need to use procutil.system() to get an exit code compatible with explainexit(). Second, explainexit() returns (msg, code) tuple.