ok/jj
1
0
Fork 0
forked from mirrors/jj
Commit graph

213 commits

Author SHA1 Message Date
Martin von Zweigbergk
779db67f8f index_store: avoid passing whole repo into get_index_at_op()
I want to be able to load the index at an operation before the repo
has been loaded.
2021-03-06 23:06:35 -08:00
Martin von Zweigbergk
d1509ffdd4 index: extract a function for adding all commits from a segment
I also restructured the loop in `maybe_squash_with_ancestors()` to
hopefully make it a little clearer.
2021-03-06 20:30:12 -08:00
Martin von Zweigbergk
3fc35288c0 index: remove dir field from ReadonlyIndex and MutableIndex 2021-03-06 10:02:19 -08:00
Martin von Zweigbergk
502ba895f5 index: move ReadonlyFile::associate_with_operation() to IndexStore
After this patch, the `index` module no longer knows about the
".jj/index/operations/" directory; that knowledge is now only in
`IndexStore`.
2021-03-06 10:00:30 -08:00
Martin von Zweigbergk
c4fe7aab10 index: move ReadonlyRepo::load_at_operation() to IndexStore 2021-03-06 09:52:44 -08:00
Martin von Zweigbergk
12bfbc489c index: move ReadonlyIndex::index() to IndexStore 2021-03-06 09:52:38 -08:00
Martin von Zweigbergk
2fdf9721c0 index: move load() to IndexStore 2021-03-06 09:52:27 -08:00
Martin von Zweigbergk
403e86c138 index: introduce IndexStore, which owns ReadonlyIndex files
This patch introduces a new `IndexStore` struct. The idea is that it
will know about the directory in which the index files are stored, the
associations with operations. It may also cache `Arc<ReadonlyIndex>`
instances so if multiple `ReadonlyIndex` instances are loaded, they
can be returned from the cache. That may be useful when merging
operations because the operations are likely to share a large parent
index file. For now, however, all the new type has is `init()`,
`load()`, and `reinit()`.
2021-03-06 09:52:16 -08:00
Martin von Zweigbergk
0a4ef1030f repo: add support for loading at given operation without loading head op first
The only way to load the repo at a current operation (as with
`--at-op`) is currently to first load it at the head operation and
then call `reload()` on the repo. This patch makes it so we can load
the repo directly at the requested operation.
2021-03-06 09:52:10 -08:00
Martin von Zweigbergk
df53871daf repo: extract a type for loading the repo in two stages
We'll want to be able to load the repo at a given operation without
first loading the head operation as we do today. This patch introduces
a struct for keeping the state of a half-loaded repo. In that
half-loaded state, the store and the op-store have been loaded, but
the view has not yet been loaded. That makes it possible for callers
to use the loaded op-store for looking up an operation to load the
view at.
2021-03-06 09:52:10 -08:00
Martin von Zweigbergk
5a32118af1 repo: move creation of OpStore out of View
We want to support loading the repo at a specific operation without
first loading the head operation (like we currently do). One reason
for that is of course efficiency. A possibly more important reason is
that the head operation may be conflicted, depending on how we decide
to deal with operation-level conflicts. In order to do that, it makes
sense to move the creation of the `OpStore` outside of the
`View`. That also suggests that the `.jj/view/op_store/` directory
should move to `.jj/op_store/`, so this patch also does that. That's
consistent with how `.jj/store/` is outside of `.jj/working_copy/`.
2021-03-06 09:52:00 -08:00
Martin von Zweigbergk
e2e9fe8f0d index: add stats for number of change ids and pruned commits 2021-03-06 09:50:22 -08:00
Martin von Zweigbergk
bc64cf02c7 index: don't use an all-0 change id in tests
It was weird to have the same change id for all commits. I think that
was a leftover from a me just quickly getting tests to pass.
2021-03-06 09:30:52 -08:00
Martin von Zweigbergk
031a39ecba cleanup: fix lots of issues found in the lib crate by clippy
I had forgotten to pass `--workspace` to clippy all this time :P
2021-02-26 23:15:43 -08:00
Martin von Zweigbergk
48b800e5c4 cleanup: fix a few things reported by upgraded rustc and clippy 2021-02-26 22:48:12 -08:00
Martin von Zweigbergk
d961f61623 evolution: calculate state using index
All the information needed for calculating the evolution state is now
in the index, so let's use it. This speeds up calculation of the
evolution state from 1.53s to 150ms in the git.git repo. In the Linux
repo, it was sped up from 28.9s to 3.07s. That's still unbearably slow
(and still pretty slow in the git.git repo too). We may need to keep a
persistent cache of the evolution state, but that will have to come
later; this improvement is good enough for now.
2021-02-26 21:19:18 -08:00
Martin von Zweigbergk
190899fe76 index: make rev_walk() iterate over IndexEntry instead of CommitId
We have the `IndexEntry` available in the iterator, so let's return it
so the caller doesn't need to look it up themselves.
2021-02-26 16:49:27 -08:00
Martin von Zweigbergk
8be84c345b index: also index change id 2021-02-26 10:33:34 -08:00
Martin von Zweigbergk
3a53a187ff index: add flag indicating pruned commit 2021-02-26 10:33:34 -08:00
Martin von Zweigbergk
d80903ce48 index: also index predecessors
Evolution needs to have fast access to the predecessors. This change
adds that information to the commit index.

Evolution also needs fast access to the change id and the bit saying
whether a commit is pruned. We'll add those soon.

Some tests changed because they previously added commits with
predecessors that were not indexed, which is no longer allowed from
this change. (We'll probably eventually want to allow that again, so
that the user can prune predecessors they no longer care about from
the repo.)
2021-02-26 10:33:34 -08:00
Martin von Zweigbergk
afc59a210a index: make tests more focused, and add tests of octopus merge 2021-02-26 10:33:32 -08:00
Martin von Zweigbergk
ab57584281 cli: allow -R and --at-op anywhere in command line 2021-02-26 10:30:14 -08:00
Martin von Zweigbergk
2a531832d6 rewrite: make merge_commit_trees() use index for finding common ancestors
The index is now always kept up to date and it has functionality for
finding common ancestors, so let's use it! This should make merging
commits a little faster if their common ancestor is far away (which is
rare). It's probably much more important that the index-based
algorithm is more correct. Also, it returns multiple common ancestors
in the criss-cross case, which lets us do a recursive merge like git
does. I'm leaving the recursive merge for later, though.
2021-02-23 20:49:18 -08:00
Martin von Zweigbergk
bb94516175 index: add support for finding common ancestors
We currently need to read the commit objects for finding common
ancestors. That can be very slow when the common ancestor is far back
in history. This patch adds a function for finding common ancestors
using the index instead.

Unlike the current algorithm, which only returns one common ancestor,
the new index-based one correctly handles criss-cross merges.

Here are some timings for finding the common ancestors in the git.git
repo:

                          |      Without index     |       With Index       |
                          | First run | Subsequent | First run | Subsequent |
v2.30.0-rc0 v2.30.0-rc1   |   5.68 ms |    5.94 us |   40.3 us |    4.77 us |
v2.25.4 v2.26.1           |   1.75 ms |    1.42 us |   13.8 ms |    4.29 ms |
v1.0.0 v2.0.0             |    492 ms |    2.79 ms |   23.4 ms |    6.41 ms |

Finding ancestors of v2.25.4 and v2.26.1 got much slower because the
new algorithm finds all common ancestors. Therefore, it also finds
v2.24.2, v2.23.2, v2.22.3, v2.21.2, v2.20.3, v2.19.4, v2.18.3, and
v2.17.4, which it then filters out because they're all ancestors of
v2.25.3.

Also note that the result was incorrect before, because the old
algorithm would return as soon as it had found a common ancestor, even
if it's not the latest common ancestor. For example, for the common
ancestor between v1.0.0 and v2.0.0, it returned an ancestor of v1.0.0
because it happened to get there by following some side branch that
led there more quickly.

The only place we currently need to find the common ancestor is when
merging trees, which we only do when the user runs `jj merge`, as well
as when operating on existing merge commits (e.g. to diff or rebase
them). That means that this change won't be very noticeable. However,
it's something we clearly want to do sooner or later, so we might as
well get it done.
2021-02-23 17:29:23 -08:00
Martin von Zweigbergk
422d333d4b index: make heads() return result in index order instead of hash order
It's nice to have a non-random order for tests (we can revisit later
if it shows up in profiling). I'm changing the order to be the index
order so the future caller of `heads_pos()` (not `heads()`) will also
get consistent order.
2021-02-23 17:24:55 -08:00
Martin von Zweigbergk
1481935472 index: extract a function for removing ancestors of set based on positions
We already have the `heads()` function, which works on
`CommitId`s. This just extracts a function that works on
positions. I'll use it soon.
2021-02-21 22:28:44 -08:00
Martin von Zweigbergk
def1a2de95 bench: also print time of first iteration, to show effect of caching
The `StoreWrapper` currently caches all objects it returns. That lead
to e.g. `common_ancestors()` being very fast once all commits have
been read in. For example, in the git.git repo `jj bench
commonancestors` with v1.0.0 and v2.0.0 reports 2.8ms, but the first
iteration takes 480ms. This commit highlights such differences by
adding a printout of the time it took to run the timed routine the
first time.
2021-02-21 22:28:30 -08:00
Martin von Zweigbergk
5aadbcf6fc evolve: pass Transaction to listener functions, so they see the updated state 2021-02-21 22:27:13 -08:00
Martin von Zweigbergk
62ce5782b5 index: when writing incremental index, squash into parent file if smaller
We currently write a new incremental index file every time. That means
that the stack of index files quickly gets deep, which makes it slow
to read the index. This commit makes it so that we squash the new
index segment into its parent if the parent has fewer commits. That
means we'll limit the number of files to O(log n). Writes time will
also be O(log n) on average.
2021-02-16 23:47:43 -08:00
Martin von Zweigbergk
a51543b752 index: make first level in stats be the root index
I've confused myself a few times already thinking that level 0 is the
root, so that's probably more intuitive. It also makes tests simpler
because the initial part of the list is unchanged when a new
transaction commits.
2021-02-16 23:45:54 -08:00
Martin von Zweigbergk
b122f33312 index: don't write empty incremental index file 2021-02-16 23:45:52 -08:00
Martin von Zweigbergk
a7b6bcfd79 transaction: write incremental index on commit
With this change, we start writing the incremental index to disk, so
the next reader won't have to re-read the commits and create the
index.

As of this change, we simply write a new index file for each
transaction. That will clearly mean that the stack of files gets deep
pretty quickly. For now, the user will have to do `jj debug reindex`
when things get slow. I plan to change it so instead of writing an
incremental index file every time, we first check if the new index
file would have at least as many commits as the parent file, and if it
will, we write a combined one instead. That should apply recursively,
so we'd have O(log n) index files.
2021-02-15 11:03:41 -08:00
Martin von Zweigbergk
86915f0a6f index: fix check for adding existing commit to index
The check for adding an existing commit to the index only checked if
the commit was already in the `MutableIndex`, not if it was already in
the parent `ReadonlyIndex`.
2021-02-15 10:28:18 -08:00
Martin von Zweigbergk
37cf6a8395 transaction: don't walk to root when adding on top of non-head
I don't know why I made the walk stop at heads instead of indexed
commits before. Perhaps I did it because it's cheap to check in the
set of head. However, it gets very expensive to walk all the way back
to the root if the parents are not in the set of heads.
2021-02-15 10:28:18 -08:00
Martin von Zweigbergk
0f56e014b7 tests: some fixups to test_transaction as a result of reordering commits 2021-02-15 10:28:07 -08:00
Martin von Zweigbergk
3c832cbbbe index: let index structs keep track of the index directory
This matches how it's done for the other struct (View, WorkingCopy).
2021-02-14 01:03:49 -08:00
Martin von Zweigbergk
b77740e58a index: move function for saving MutableIndex onto the struct 2021-02-14 01:03:49 -08:00
Martin von Zweigbergk
713d32d803 index: keep up to date within transaction
With tons of groundwork done, wee can now finally keep the index up to
date within a transaction! That means that we can start relying on the
index to always be valid, so we can use it e.g. for finding common
ancestors within a transaction. That should help speed up `jj evolve`
immensely on large repos.

We still don't write the updated index to disk when the transaction
closes. That will come later.
2021-02-14 00:58:11 -08:00
Martin von Zweigbergk
e19a65cf14 transaction: make add_head() use incremental update of evolution in common case
`Transaction::add_head()` currently invalidates the whole evolution
state. We've had support for incrementally updating evolution since
4619942a57. We should start taking advantage of that. Let's add a
fast-path in `Transaction::add_head()` for the common case where we
add a single commit on top of an existing head. That cheap an simple
to check for. However, it won't cover the case of adding a child off
of a non-head. It's still a good start.
2021-02-14 00:56:34 -08:00
Martin von Zweigbergk
f05a12d301 index: make CompositeIndex non-public and add new IndexRef enum instead
We're getting close to finally having a `RepoRef::index()` method.
2021-02-13 13:56:26 -08:00
Martin von Zweigbergk
face4d637f index: define methods from CompositeIndex directly on {Readonly,Mutable}Index
This is one step towards making `CompositeIndex` non-public (and maybe
deleting it). Next, we'll add an `IndexRef` enum similar to `RepoRef`
etc.
2021-02-13 13:46:58 -08:00
Martin von Zweigbergk
8dda4b05e4 index: add "segment_" prefix to methods in IndexSegment
I'm about to move the functions from `CompositeIndex` to an new
`Index` trait implemeted by `ReadonlyIndex` and `MutableIndex`. Since
those types already implement `IndexSegment`, the names would conflict
and it would get annoying to have to disambiguate them. This commit
therefore prepares for that by adding a `segment_` prefix to the
functions in `IndexSegment`.
2021-02-13 13:45:31 -08:00
Martin von Zweigbergk
3066381d57 transaction: add accessors for view and evolution directly on transaction 2021-02-13 13:43:48 -08:00
Martin von Zweigbergk
72aebc9da3 view: replace View trait by enum with Readonly and Mutable variants 2021-02-13 08:31:41 -08:00
Martin von Zweigbergk
d1e5f46969 evolution: replace Evolution trait by enum with Readonly and Mutable variants 2021-02-13 08:31:41 -08:00
Martin von Zweigbergk
f1666375bd repo: replace Repo trait by enum with readonly and mutable variants
I want to keep the index updated within the transaction. I tried doing
that by adding a `trait Index`, implemented by `ReadonlyIndex` and
`MutableIndex`. However, `ReadonlyRepo::index` is of type
`Mutex<Option<Arc<IndexFile>>>` (because it is lazily initialized),
and we cannot get a `&dyn Index` that lives long enough to be returned
from a `Repo::index()` from that. It seems the best solution is to
instead create an `Index` enum (instead of a trait), with one readonly
and one mutable variant. This commit starts the migration to that
design by replacing the `Repo` trait by an enum. I never intended for
there there to be more implementations of `Repo` than `ReadonlyRepo`
and `MutableRepo` anyway.
2021-02-13 08:31:23 -08:00
Martin von Zweigbergk
a1983ebe96 git: add a ref to each commit we create
I just learned that attaching a git note is not enough to keep a
commit from being GC'd. I had read `git help gc` before but it was
quite misleading (I just sent a patch to clarify it). Since the git
note is not enough, we need to create some other reference. This patch
makes it so we write refs in `refs/jj/keep/` for every commit we
create. We will probably want to remove unnecessary refs (ancestors of
commits pointed to by other refs) once we have a `jj gc` command.
2021-02-13 08:16:18 -08:00
Martin von Zweigbergk
dd98f0564e git: remove git note pointing to conflicts
We store conflicts as blobs with JSON data and with a git note
pointing to them to prevent GC. These are stored in the git tree as
regular files. The only thing that distinguishes them is that their
filename ends with `.jjconflict`. Since they are referenced from the
tree, there's no need for the git note to prevent GC (which doesn't
work anyway, as I just learned), and we don't store any additional
data in the note either, so let's just remove it.
2021-02-13 08:15:12 -08:00
Martin von Zweigbergk
fa30cf768f index: rename UnsavedIndexData to MutableIndex 2021-02-07 23:35:37 -08:00
Martin von Zweigbergk
8170c06573 index: rename IndexFile to ReadonlyIndex 2021-02-07 23:35:22 -08:00