If you import Git refs, then rebase a commit pointed to by some Git
ref, and then re-import Git refs, you don't want the old commit to be
made a visible head again. That's particularly annoying when Git refs
are automatically updated by every command.
Since 94e03f5ac8, we lazily filter out non-heads from `View`'s set
of head. Accessing the `MutRepo::view()` triggers that filtering. This
patch makes `DescendantRebaser` not unnecessarily do that. That speeds
up the rebasing of 162 descendants in the git.git repo from ~3.6 s to
~330 ms. Rebasing 1272 descendants takes ~885 ms.
When you run e.g. `jj describe <some old commit>` or `jj squash -r
<some old commit>`, the descendants' tree objects will not change, so
we can avoid calculating them. This speeds up rebasing of 126 commits
in the git.git repo from ~9.8 s to ~3.6 s.
I recently (0c441d9558) made it so we don't create an operation when
nothing changed. Soon thereafter (94e03f5ac8), I broke that when I
introduced a cache-invalidation bug when I made the filtering-out of
non-heads be lazy. This patch fixes that and also adds a test to
prevent regressions.
When initializing a jj repo in the same directory as its backing git
repo, add `.jj/` to `.git/info/exclude` so it doesn't show up to `git`
commands.
This is part of #44.
It's very slow to remove non-heads from the set of heads every time we
add an head. For example, in the git.git repo, a no-op `jj git import`
takes ~15 s. This patch changes makes us just mark the set of heads
dirty when a commit has been added and then we remove non-heads when
needed. That cuts down the `jj git import` time to ~200 ms.
It's useful to know which commit is checked out in the underlying Git
repo (if there is one), so let's show that. This patch indicates that
commit with `HEAD@git` in the log output. It's probably not very
useful when the Git repo is "internal" (i.e. stored inside `.jj/`),
because then it's unlikely to change often. I therefore considered not
showing it when the Git repo is internal. However, it turned out that
`HEAD` points to a non-existent branch in the repo I use, so it won't
get imported anyway (by the function added in the previous patch). We
can always review this decision later.
This is part of #44.
This patch adds a place for tracking the current `HEAD` commit in the
underlying Git repo. It updates `git::import_refs()` to record it. We
don't use it anywhere yet.
This is part of #44.
I think these remaining implementations of `Drop` are for types that
are infrequently dropped (unlike `Transaction`), so it be fine to be
more strict about them.
If nothing changed in a transaction, it's rarely useful to commit it,
so let's avoid that. For example, if you run `jj git import` without
changing the anything in the Git repo, we now just print "Nothing
changed.".
I'm about to change `ReadonlyRepo::load()` to take the path to the
`.jj/` directory, so this patch prepares for that. It already works
because `ReadonlyRepo::load()` will search up the directory tree for
the `.jj/` entry.
`ReadonlyRepo::init_*()` currently calls `WorkingCopy::init()`. In
order to remove that dependency, this patch wraps the
`ReadonlyRepo::init_*()` functions in new `Workspace` functions. A
later patch will have those functions call `WorkspaceCopy::init()`.`
The `Repo` doesn't do anything with the `WorkingCopy` except keeping a
reference to it for its users to use. In fact, the entire lib crate
doesn't do antyhing with the `WorkingCopy`. It therefore seems simpler
to have the users of the crate manage the `WorkingCopy` instance. This
patch does that by letting `Workspace` own it. By not keeping an
instance in `Repo`, which is `Sync`, we can also drop the
`Arc<Mutex<>>` wrapping.
I left `Repo::working_copy()` for convenience for now, but now it
creates a new instance every time. It's only used in tests.
This further decoupling should help us add support for multiple
working copies (#13).
Having a concept of a "workspace" will be useful for adding support
for multiple workspaces (#13). You can think of the "workspace" as a
repo combined with a working copy. A workspace corresponds 1:1 with a
`.jj/` directory. It's pretty close to what other VCS simply call a
"repo", but I've ended up using the word "repo" for what Git calls a
"bare repo".
The recent e5dd93cbf7, whose description says "cleanup: make Vec
inside CommitId etc. non-public", made all ID types in the `backend`
module *except* for `CommitId` non-public :P This patch makes
A while ago, I replaced a call to git2-rs's `Remote::fetch()` by calls
to `Remote::download()` and `Remote::update_tips()`. The function is
documented to be a convenience for those function, but it turns out
that the pruning of deleted remote refs is a separate call
(`Remote::prune()`), so we need to call that too.
Since the working copy can now handle conflicts, we don't need to
materialize conflicts when checking out a commit.
Before this patch, we used to create a new commit on top whenever we
checked out a commit with conflicts. That new commit was intended just
for resolving the conflicts. The typical workflow was the resolve the
conflicts and then amend. To use the same workflow after this patch,
one needs to explicitly create a new commit on top with `jj new` after
checking out a commit with conflict.
I realized only recently that we can try to parse conflict markers in
files and leave them as conflicted if they haven't changed. If they
have changed and some conflict markers have been removed, we can even
update the conflict with that partial resolution.
This change teaches the working copy to write conflicts to the working
copy. It used to expect that the caller had already updated the tree
by materializing conflicts. With this change, we also start parsing
the conflict markers and leave the conflicts unresolved in the working
copy if the conflict markers remain.
There are some cases that we don't handle yet. For example, we don't
even try to set the executable bit correctly when we write
conflicts. OTOH, we didn't do that even before this change.
We still never actually write conflicts to the working copy (outside
of tests) because we currently materialize conflicts in
`MutRepo::check_out()`. I'll change that next.
I initially made the working copy materialize conflicts in its
`check_out()` method. Then I changed it later (exactly a year ago, on
Halloween of 2020, actually) so that the working copy expected
conflicts to already have been materalized, which happens in
`MutableRepo::check_out`().
I think my reasoning then was that the file system cannot represent a
conflict. While it's true that the file system itself doesn't have
information to know whether a file represents a conflict, we can
record that ourselves. We already record whether a file is executable
or not and then preserve that if we're on a file system that isn't
able to record it. It's not that different to do the same for
conflicts if we're on a file system that doesn't understand conflicts
(i.e. all file systems).
The plan is to have the working copy remember whether a file
represents a conflict. When we check if it has changed, we parse the
file, including conflict markers, and recreate the conflict from
it. We should be able to do that losslessly (and we should adjust
formats to make it possible if we find cases where it's not).
Having the working copy preserve conflict states has several
advantages:
* Because conflicts are not materialized in the working copy, you can
rebase the conflicted commit and the working copy without causing
more conflicts (that's currently a UX bug I run into every now and
then).
* If you don't change anything in the working copy, it will be
unchanged compared to its parent, which means we'll automatically
abandon it if you update away from it.
* The user can choose to resolve only some of the conflicts in a file
and squash those in, and it'll work they way you'd hope.
* It should make it easier to implement support for external merge
tools (#18) without having them treat the working copy differently.
This patch prepares for that work by adding support for parsing
materialized conflicts.
On Windows, we preserve the executable bit. I plan to also teach the
working copy to preserve conflict state. This refactoring prepares for
that by simplifying how we preserve parts of the current file state.
While working on demos, I noticed that `jj log` output in the
octocat/Hello-World repo was unstable: sometimes the first parent of
the merge was on the left and sometimes it was on the right. This
patch fixes that by sorting the edges by position in the index just
before returning them. It seems that most applications would want
stable output so I put it in the `RevsetGraphIterator` rather than
doing at the call site in the CLI. I ordered them with the reverse
index position rather than forward because it seemed to make the
graphs in the git.git repo slight nicer, with the left-most edge going
between subsequent releases.
There performance difference is within the noise level.
If you rewrite a commit that's also available on some remote, you'll
currently see both the old version and the new version in the view,
which means they're divergent. They're not logically divergent (the
rewritten version should replace the old version), so this is a UX
bug. I think it indicates that the set of current heads should be
redefined to be the *desired* heads. That's also what I had suspected
in the TODO removed by this change. I think another indication that
we should hide the old heads even if they have e.g. a remote branch
pointing to them is that we don't want them to be rebased if we
rewrite an ancestor.
So that's what I decided to do: let the view's heads be the desired
heads. The user can still define revsets for showing non-current
commits pointed to by e.g. remote branches.
This fixes a bug I've run into somewhat frequently. What happens is
that if you have a conflict on top of another conflict and you resolve
the conflict in the bottom commit, we just simplify the `Conflict`
object in the second commit, but we don't try to resolve the new
conflict. That shows up as an unexpected "conflict" in `jj log`
output, and when you check out the commit, there are actually no
conflicts, so you can just `jj squash` right away.
This patch fixes that bug. It also teaches the code to work with more
than 3 parts in the merge, so if there's a 5-way conflict, for
example, we still try to resolve it if possible.
Git notes (at least as implemented by libgit2) quickly gets really
slow, as noted in issue #7. This patch replaces it by a custom storage
format.
I tested the performance in the git.git repo with just a few hundred
annotated commits (~450, I think) and no sharding. I listed the first
~2900 commits there using `jj log --no-graph -r ,,v1.0.0 -T 'author
"\n"' | wc -l`. That took about 882ms. After this patch, it dropped to
108ms.
I did a similar test in this repo with 12700 annotated commits and
sharding, listing all visible commits. That took 142ms before this
patch (the sharding helps a lot!) and 55ms after.
Closes#3.
Closes#7.
The new store works the same way as the `OpHeadsStore`. It keeps track
of the current head file(s) by recording their names in a
directory. When a write happens, it adds the new head and then removes
the old head. There will be generally be a single head at a time. The
only exception is when there's been concurrent operations (locally, or
remotely, in the case of a distributed file system). When there are
multiple heads files, they are automatically merged. No guarantee is
given about which value wins if the key exists in several heads; the
store is meant to be used for data that's immutable once written. As
long as different keys are written, this is a CRDT. That makes it fit
for solving both #3 and #7.
I'm trying to replace the Git backend's use of Git notes for storing
metadata (#7). This patch adds a file format that I hope can be used
for that. It's a simple generic format for storing fixed-size keys and
associated variable-size values. The keys are stored in sorted
order. Each key is followed by an offset to the value. The offset is
relative to the first value. All values are concatenated after each
other. I suppose it's a bit like Git's pack files but lacking both
delta-encoding and compression.
Each file can also have a parent pointer (just like the index files
have), so we don't have to rewrite the whole file each time. As with
the index files, the new format squashes a file into its parent if it
contains more than half the number of entries of the parent. The code
is also based on `index.rs`.
Perhaps we can alo replace the default operation storage with this
format. Maybe also the native local backend's storage. We'll need
delta-encoding and compression soon then.
I'm about to change the index format (to remove predecessor
information), which will break the format. Let's prepare for that by
having `IndexStore` reindex the repo if it fails to read the index..
I think this is just cleaner, and it gives us room to put other
store-related data in the `.jj/store/` directory. I may want to use
that place for writing the metadata we currently write in Git notes
(#7).
With this change, you can do e.g. `heads(remote_branches())`. That
should currently be the same as `public_heads()`, except that we don't
yet remove public heads when remote branches have been updated. Having
this support should be generally useful, but I may use it in the short
term specifically for depending less on the public heads, until I get
around to keeping them up to date.
I noticed while working on support for unified diffs (#33) that
`Diff::for_tokenizer(..., &find_line_ranges)` would return a
`DiffHunk::Matching` for each matching line instead of a single
`DiffHunk::Matching` for all the matching lines. That's different from
what you get from `Diff::default_refinement()` and seems less
convenient to work with.
My SSH keys are password-protected, so I haven't been able to test
this patch completely, but I believe it should work. We now use
ssh-agent if `$SSH_AGENT_PID` is set, otherwise we check if
`$HOME/.ssh/id_rsa` exists and assume it's a password-less key. That's
quite hacky but I think it's good enough for now. We eventually need
to move this out of the library crate just like libgit2 has done.
Closes#25.
It's been a lot of work, but now we're finally able to remove the
`Evolution` state! `jj obslog` still works as before (it just walks
the predecessor pointers).
The removal of hidden heads was just there to help with the transition
away from evolution (#32). Now that we no longer depend on evolution
for removing old heads, we can remove the hack.
This rewrites the code for resolving a change id to simply walk the
entire index. That's obviously not optimal, but it's not worse than
what we did in the evolution-based resolution. This is yet another
step towards removing support for evolution (#32).
This patch teaches `DescendantRebaser` to also update heads. That's
done at the end of the rebase (when `rebase_next()` starts returning
`None`), which is a little weird. We should probably change the
interface, but this will do for now.
With this change, we should no longer need to remove hidden heads when
the transaction commits. That will remove one of the last bits of
dependence on evolution from most commands (#32).
Now that we no longer have to be careful whether we mean "all heads"
or "non-obsolete heads", there's no need to pass them as
arguments. It's still possible to get a DAG range to a hidden commit
by using `RevsetExpression::dag_range_to()`, as long as the hidden
commit is indexed.
Now that we remove hidden heads whenever a transaction commits,
`non_obsolete_heads()` should always be the same as `all_heads()`,
except during a transaction. I don't think we depend on the difference
even during a transaction. Let's simplify a bit by removing the revset
function `all_heads()` and renaming `non_obsolete_heads()` to
`heads()`. This is part of issue #32.
This is similar to how a recent change taught `DescendantRebaser` to
update branches pointing to rewritten commits. Now we also update the
checkout if it pointed to a rewritten commit.
This patch moves the logic for updating branches from
`update_branches_after_rewrite()` into `DescendantRebaser`. The
branches are now updated along with each rebased commit rather than
all being updated at the end. The new code uses the information about
rewritten and abandoned commits that `DescendantRebaser` gets from
`MutableRepo`. That is different from the old code, which used the
evolution state. This patch thus moves us one step closer to removing
evolution (#32).
I'm going to teach `DescendantRebaser` to also update local branches
pointing to rewritten commits, taking over the responsibility from
`rewrite::update_branches_after_rewrite()`. For commits that have been
rewritten as multiple new commits (divergent, not split), that
function makes local branches pointing to the old commit point to all
the new commits. To replicate that behavior in `DescendantRebaser`, it
needs to know about divergent changes. This change addresses that.
I recently made the CLI remove hidden heads when a transaction is
committed (38474a9). Let's move that to `Transaction::commit()`, so
the library crate becomes more similar to how the CLI behaves and more
similar to our evolution-less future (#32).
The next patch would otherwise make this test fail because
"transaction 2" tries to point a branch to a commit that's not visible
(because it's created by the concurrent "transaction 1").
Same reasoning as the previous change.
With this change, I believe we now record all rewritten and abandoned
commits correctly. We're now almost ready to switch the CLI away from
using evolution for automatically rebasing commits.
This is part of removing support for evolution (#32). Since
`CommitBuilder` now records rewritten commits in `MutableRepo`, we can
use that recorded information to automatically rebase descendants.
When we remove support for evolution (#32), we need to still make it
easy for application code to rebase descendants of rewritten and
abandoned commits. The way applications currently do that is by using
e.g. `CommitBuilder::for_rewrite_from()` followed by
`evolve_orphans()`. This patch puts some bookkeeping in `MutableRepo`
for rewritten and abandoned commits, along with a function for
creating a `DescendantRebaser` based on it. I'll then make
`CommitBuilder` record rewritten commits there.
If you rewrite a change twice, from A to A' to A'', then undo the
operation that created A', you'll end up with a repo where A'' refers
to commit (A') that's not reachable from any head in the view. We
currently crash when that happens. This change fixes the
crash. Undoing the A' operation now instead produces a state where A
and A'' are divergent. That at least makes some sense.
This may not seem important since I'm working on removing support for
evolution (#32), but I wanted to get it fixed in order to help with
the transition off of evolution. Specifically, I want to be able to
start removing old heads more freely.
This closes#28.
In the recent switch away from `git2::Remote::fetch()`, I passed
`git2::AutotagOption::All`, which caused cloning of e.g. the `clap`
repo to fail like this:
```
Error: Fetch failed: Error { code: -1, klass: 4, message: "target OID for the reference doesn't exist on the repository" }
```
This commit changes from `All` to `Unspecified`, which respects the
remote's configuration.
The default branch relies on checking the value of `HEAD`. The `empty_git_commit` function updates the ref `refs/heads/main`, but since `HEAD` was never updated to point to that ref, the default branch can't be determined. The fix is to explicitly set `HEAD`.
Personally, this test failed reliably for me on macOS. I don't know why this behavior would be non-deterministic on other platforms.
It seems it wasn't Windows that behaved differently when it comes
getting the remote's default branch; the test failed on Ubuntu
too.
The documentation for `Remote::default_branch()` says that it can be
called even after the connection has been closed, but let's see if
calling it while the connection is open helps anyway. To do that, we
have to replicate what `Remote::fetch()` does.
Descendants of abandoned commits should be rebased onto their parents,
or the rewritten parents if they had been rewritten. This patch
teaches `DescendantRebaser` to do that. It updates `jj rebase -r` to
use the functionality. I plan to also use it in `jj abandon`
(naturally, given the name), and for rebasing descendants of deleted
refs imported from `jj git refresh/fetch/push`.
The fact that `DescendantRebaser` visits some commits that don't need
to be rebased is mostly an implementation detail. I can't think of a
reason that callers would care about these commits.
The command's help text says "Abandon a revision", which I think is a
good indication that the command's name should be `abandon`. This
patch renames the command and other user-facing occurrences of the
word. The remaining occurrences should be removed when I remove
support for evolution.
This patch moves the function for updating branches after rewrite from
`commands.rs` into `rewrite.rs`.
It also changes the function to update branches even if they were
conflicted or become conflicted. I think that seems better than
leaving branches on old commits. For example, let's say you have start
with this:
```
C main
|
B origin@main
|
A
```
You now pull from origin, which has updated the main branch from B to
B'. We apply that change to both the remote branch and the local
branch, which results in a conflict in the local branch:
```
C main?
|
B B' main? origin@main
|/
A
```
If you now rewrite C to C', the conflicted main branch will still
point to C, which is just weird. This patch changes that so the
conflicted side of main gets repointed to C'.
I also refactored the code to reuse our existing
`MutableRepo::merge_single_ref()`, which improves the behavior in
several cases, such as the conflict-resolution case in the last test
case.
As the updates test case shows, when rebasing forward, we missed
commits that fork off from the section between the source and the
destination.
As part of the fix, I also restructured the code a bit to prepare for
support for rebasing descendants of multiple rewritten commits.
It turns out that `FETCH_HEAD` is not the remote's `HEAD` (it's
actually not even a normal symbolic ref; it contains many lines of
commits and names). We're supposed to ask the remote for its default
branch instead. That's what this patch does.
It's annoying to have to add `--branch main` every time I push to
GitHub.
Maybe we should make it push only the current branch by default, but
we don't even have a concept of a current branch yet...
Before this change, you could end up with an index segment with 10
commits, then a child segment with 9 commits, then another child with
8 commits, and so on. That's not what I had intended. This changes
makes it so we squash if a segment has more than half as many commits
as its parent instead.
We have had support for ignores via `.gitignore` files since
3b326a942c, and we haven't had the problem with the temporary
`.git/` directory created by libgit2 since 88f7f4732b.
Git doesn't want `.git` entries in its trees, so at least when using
the Git backend, we need to ignore such paths. Let's just ignore
`.git` paths regardless of backend to keep it simple.
Closes#24.
When I added the function for rebasing descendants, I forgot to call
the existing `rebase()` function and instead simply created a new
commit with the new parents but the old contents.
This should be useful in lots of places. For example, `jj rebase -r`
currently rebases all descendants, because that's what the auto-evolve
feature does. I think it would be nice to instead copy from
Mercurial's `-s` flag for also rebasing descendants. Then `jj rebase
-r` can be made to pull a commit out of a stack, rebasing descendants
onto the rebased commit's parents. I also intend to use this
functionality for rebasing descendants when remote branches have been
rewritten.
