Commit graph

88 commits

Author SHA1 Message Date
jyn
d66fcf2ca0 compile integration tests as a single binary
this greatly speeds up the time to run all tests, at the cost of slightly larger recompile times for individual tests.

this unfortunately adds the requirement that all tests are listed in `runner.rs` for the crate.
to avoid forgetting, i've added a new test that ensures the directory is in sync with the file.

 ## benchmarks

before this change, recompiling all tests took 32-50 seconds and running a single test took 3.5 seconds:

```
; hyperfine 'touch lib/src/lib.rs && cargo t --test test_working_copy'
  Time (mean ± σ):      3.543 s ±  0.168 s    [User: 2.597 s, System: 1.262 s]
  Range (min … max):    3.400 s …  3.847 s    10 runs
```

after this change, recompiling all tests take 4 seconds:
```
;  hyperfine 'touch lib/src/lib.rs ; cargo t --test runner --no-run'
  Time (mean ± σ):      4.055 s ±  0.123 s    [User: 3.591 s, System: 1.593 s]
  Range (min … max):    3.804 s …  4.159 s    10 runs
```
and running a single test takes about the same:
```
; hyperfine 'touch lib/src/lib.rs && cargo t --test runner -- test_working_copy'
  Time (mean ± σ):      4.129 s ±  0.120 s    [User: 3.636 s, System: 1.593 s]
  Range (min … max):    3.933 s …  4.346 s    10 runs
```

about 1.4 seconds of that is the time for the runner, of which .4 is the time for the linker. so
there may be room for further improving the times.
2024-02-06 18:19:41 -08:00
Yuya Nishihara
351487b9f5 backend: pass Index and keep_newer timestamp parameters to gc()
GitBackend::gc() will need to check if a commit is reachable from any
historical operations. This could be calculated from the view and commit
objects, but the Index will do a better job.
2024-01-27 10:18:11 +09:00
Yuya Nishihara
4e54021930 backend: have gc() return BackendError instead of opaque error type
The gc() implementation is likely to call other backend functions, which
return BackendError.
2024-01-27 10:18:11 +09:00
Yuya Nishihara
fa5e40719c object_id: extract ObjectId trait and macros to separate module
I'm going to add a prefix resolution method to OpStore, but OpStore is
unrelated to the index. I think ObjectId, HexPrefix, and PrefixResolution can
be extracted to this module.
2024-01-05 10:20:57 +09:00
Ilya Grigoriev
45cd0bf11b test_rewrite.rs: stop using DescendantRebaser when testing EmptyBehavior
This completes the process of removing DescendantRebaser-related APIs from
tests. It requires creating some new test utils and a new
`rebase_descendants_with_option_return_map`.
2024-01-01 18:51:36 -08:00
Ilya Grigoriev
b2abba07e9 tests: (mostly) stop using soon-to-be-private DescendantRebaser-related APIs
This removes uses of `DescendantRebaser::new` or
`MutRepo::create_descendant_rebaser` from most tests. The exceptions  are the
tests having to do with abandoning empty commits on rebase, since adjusting
those is a bit more elaborate (see follow-up commits).
2024-01-01 18:51:36 -08:00
Yuya Nishihara
51691ea22c tests: add lib tests for op id resolution, migrate some from cli
CLI testing is slow and harder to set up crafted environment.
2024-01-02 10:30:08 +09:00
Yuya Nishihara
f9e9058b9b index: show bad operation id if commit lookup failed during reindexing
My jj repo contains such head commits, and "jj debug reindex" fails. To address
this problem, we'll probably need to implement GC, and the user will discard
operations before the first bad op id.
2023-12-29 13:05:58 +09:00
Martin von Zweigbergk
60fae3114e transaction: take description at end instead of start
It seems better to have the caller pass the transaction description
when we finish the transaction than when we start it. That way we have
all the information we want to include more readily available.
2023-12-13 08:12:49 -08:00
Martin von Zweigbergk
1cc271441f gc: implement basic GC for Git backend
This adds an initial `jj util gc` command, which simply calls `git gc`
when using the Git backend. That should already be useful in
non-colocated repos because it's not obvious how to GC (repack) such
repos. In my own jj repo, it shrunk `.jj/repo/store/` from 2.4 GiB to
780 MiB, and `jj log --ignore-working-copy` was sped up from 157 ms to
86 ms.

I haven't added any tests because the functionality depends on having
`git` binary on the PATH, which we don't yet depend on anywhere
else. I think we'll still be able to test much of the future parts of
garbage collection without a `git` binary because the interesting
parts are about manipulating the Git repo before calling `git gc` on
it.
2023-12-03 07:40:12 -08:00
Yuya Nishihara
d747879aee signing: pass SigningFn by reference
write_commit() doesn't need ownership of the signing function.
2023-12-01 22:55:04 +09:00
Anton Bulakh
eb1c0ab4a2 sign: Implement a test signing backend and add a few basic tests 2023-11-30 23:36:56 +02:00
Anton Bulakh
d7229a3f90 sign: Define signing backend API and integrate it
Finished everything except actual signing backend implementation(s) and
the UI.
2023-11-30 23:36:56 +02:00
Yuya Nishihara
28ab9593c3 repo_path: split RepoPath into owned and borrowed types
This enables cheap str-to-RepoPath cast, which is useful when sorting and
filtering a large Vec<(String, _)> list by using matcher for example. It
will also eliminate temporary allocation by repo_path.parent().
2023-11-28 07:33:28 +09:00
Yuya Nishihara
0a1bc2ba42 repo_path: add stub RepoPathBuf type, update callers
Most RepoPath::from_internal_string() callers will be migrated to the function
that returns &RepoPath, and cloning &RepoPath won't work.
2023-11-28 07:33:28 +09:00
Yuya Nishihara
f5938985f0 repo_path: make RepoPath::from_internal_string() accept owned string
I'm going to add borrowed RepoPath type, and most from_internal_string()
callers will be migrated to it. For the remaining callers, it makes more
sense to move the ownership of String to RepoPathBuf.
2023-11-28 07:33:28 +09:00
Anton Bulakh
5c3c0e9f6e sign: Implement generic commit signing on the backend 2023-11-23 22:52:20 +02:00
Anton Bulakh
e3a1e5b80e sign: Implement storage for digital commit signatures
Recognize signature metadata from git commit objects, implement
a basic version of that for the native backend.
Extract the signed data (a commit binary repr without the signature) to
be verified later.
2023-11-12 03:37:13 +02:00
Yuya Nishihara
ea32c0cb9e git_backend: pass UserSettings to GitBackend constructors 2023-11-11 22:35:54 +09:00
Yuya Nishihara
8a2048a0e5 repo: pass UserSettings to store factories and initializers
GitBackend will use it to configure gix::Repository. I think UserSettings
is generally useful to pass store-specific parameters, so I've updated all
factory functions.
2023-11-11 22:35:54 +09:00
Martin von Zweigbergk
d989d4093d merged_tree: let backend influence whether to use new diff algo
Since the concurrent diff algorithm is significantly slower when using
the Git backend, I think we'll have to use switch between the two
algorithms depending on backend. Even if the concurrent version always
performed as well as the sequential version, exactly how concurrent it
should be probably still depends on the backend. This commit therefore
adds a function to the `Backend` trait, so each backend can say how
much concurrency they deal well with. I then use that number for
choosing between the sequential and concurrent versions in
`MergedTree::diff_stream()`, and also to decide the number of
concurrent reads to do in the concurrent version.
2023-11-06 23:12:02 -08:00
Yuya Nishihara
9a86b77e38 tests: force gitoxide to not load config nor use "main" as default branch
AFAIK, there's no global config state for gitoxide. We can use
Config::isolated() in tests, but GitBackend should load config files in a
normal way.

https://docs.rs/gix/0.55.2/gix/open/permissions/struct.Config.html#method.isolated
https://docs.rs/gix/0.55.2/gix/init/constant.DEFAULT_BRANCH_NAME.html
2023-11-02 19:33:06 +09:00
Martin von Zweigbergk
cfcdd71865 backend: make read_conflict synchronous again
This avoids https://github.com/rust-lang/futures-rs/issues/2090. I
don't think we need to worry about reading legacy conflicts
asynchronously - async is really only useful for Google's backend
right now, and we don't use the legacy format at Google. In
particular, I don't want `MergedTree::value()` to have to be async.
2023-10-28 16:45:40 -07:00
Martin von Zweigbergk
e1f00d9426 working copy: pass commit instead of tree into check_out()
Our internal working copy implementations at Google will need the
commit so they can walk history backwards until they get to a "public"
commit. They'll then use that to tell build tools and virtual file
systems to present that as a base.

I'm not sure if we'll need to update `reset()` too. It's currently
only used by `jj untrack`, which doesn't change the commit's parent,
so it wouldn't affect any history walks.
2023-10-16 22:33:44 -07:00
Martin von Zweigbergk
7c8a0a18f9 repo: define types for backend initializer functions
`ReadonlyRepo::init()` takes callbacks for initializing each kind of
backend. We called these things like `op_store_initializer`. I found
that confusing because it is not a `OpStoreFactory` (which is for
loading an existing backend). This patch tries to clarify that by
renaming the arguments and adding types for each kind of callback
function.
2023-10-16 22:33:44 -07:00
Martin von Zweigbergk
0582893144 working copy: return Box<dyn LockedWorkingCopy> from start_mutation() 2023-10-15 16:13:19 -07:00
Martin von Zweigbergk
781859cb51 working copy: add snapshot() function to the backend trait
This includes documenting the new function and the other types moved
to the `working_copy` module.
2023-10-15 15:59:49 -07:00
Martin von Zweigbergk
c8184845d7 workspace: replace working_copy_mut() by wrapper type
I'm about to make `LockedLocalWorkingCopy` not borrow from
`LocalWorkingCopy`. That will make it easier to forget to update any
`LocalWorkingCopy` variables when the modifications have been
committed. This patch introduces a wrapper around
`LockedLocalWorkingCopy` to help prevent that.

Thanks to Yuya for the suggestion.
2023-10-15 15:59:49 -07:00
Martin von Zweigbergk
5174489959 backend: make read functions async
The commit backend at Google is cloud-based (and so are the other
backends); it reads and writes commits from/to a server, which stores
them in a database. That makes latency much higher than for disk-based
backends. To reduce the latency, we have a local daemon process that
caches and prefetches objects. There are still many cases where
latency is high, such as when diffing two uncached commits. We can
improve that by changing some of our (jj's) algorithms to read many
objects concurrently from the backend. In the case of tree-diffing, we
can fetch one level (depth) of the tree at a time. There are several
ways of doing that:

 * Make the backend methods `async`
 * Use many threads for reading from the backend
 * Add backend methods for batch reading

I don't think we typically need CPU parallelism, so it's wasteful to
have hundreds of threads running in order to fetch hundreds of objects
in parallel (especially when using a synchronous backend like the Git
backend). Batching would work well for the tree-diffing case, but it's
not as composable as `async`. For example, if we wanted to fetch some
commits at the same time as we were doing a diff, it's hard to see how
to do that with batching. Using async seems like our best bet.

I didn't make the backend interface's write functions async because
writes are already async with the daemon we have at Google. That
daemon will hash the object and immediately return, and then send the
object to the server in the background. I think any cloud-based
solution will need a similar daemon process. However, we may need to
reconsider this if/when jj gets used on a server with a custom backend
that writes directly to a database (i.e. no async daemon in between).

I've tried to measure the performance impact. That's the largest
difference I've been able to measure was on `jj diff
--ignore-working-copy -s --from v5.0 --to v6.0` in the Linux repo,
which increases from 749 ms to 773 ms (3.3%). In most cases I've
tested, there's no measurable difference. I've tried diffing from the
root commit, as well as `jj --ignore-working-copy log --no-graph -r
'::v3.0 & author(torvalds)' -T 'commit_id ++ "\n"'` (to test a
commit-heavy load).
2023-10-08 23:36:49 -07:00
Martin von Zweigbergk
187ba9430a working_copy: rename to local_working_copy
It's about time we make the working copy a pluggable backend like we
have for the other storage. We will use it at Google for at least two
reasons:

 * To support our virtual file system. That will be a completely
   separate working copy backend, which will interact with the virtual
   file system to update and snapshot the working copy.

 * On local disk, we need to tell our build system where to find the
   paths that are not in the sparse patterns. We plan to do that by
   wrapping the standard local working copy backend (the one moved in
   this commit), writing a symlink that points to the mainline commit
   where the "background" files can be read from.

Let's start by renaming the exising implementation to
`local_working_copy`.
2023-10-07 08:19:03 -07:00
Martin von Zweigbergk
380e204e73 test: use test backend in most remaining tests too
I don't think the backend should matter for any of these tests, so
let's test with only one, and let's make that the strictest one - the
new test backend.

This reduces the number of tests by 74 (from 974 to 900), but saves no
measurable run time.
2023-09-24 21:24:01 -07:00
Martin von Zweigbergk
f39b0d24c8 tests: use test backend in working copy tests, fix MergedTree bug
Only tests dealing with Git submodules care about the backend type.

Switching the tests to use the test backend also uncovered another bug
in `MergedTree`, so I fixed that too. The bug only happens with legacy
trees (path-level conflicts) and backends that care about the conflict
path, so it wouldn't happen with Git backends, and it wouldn't happen
at Google either (because we use tree-level conflicts).
2023-09-19 20:49:41 -07:00
Martin von Zweigbergk
0f7054e8c3 tests: wherever we test with only one backend, use the test backend
I don't think there's any reason to use the local backend in tests
instead of using the stricter test backend.

I think we should generally use the test backend in tests and only use
the local backend or git backend when there's a particular reason to
do so (such as in `test_bad_locking` where the on-disk directory
structure matters). But this patch only deals with the simpler cases
where we were only testing with the local backend.
2023-09-19 20:49:41 -07:00
Martin von Zweigbergk
63ba2a6346 tests: add a strict backend for use in tests
We ran into a bug in `MergedTree` with our commit backend at
Google. The problem there was that `MergedTree` sometimes uses the
wrong path when reading files and trees. We didn't catch the bug in
our tests (outside of Google) because both our backends let you read
files and trees at any path.

This commit introduces a stricter backend that we can use in tests to
catch this kind of bug. For simplicity, it stores all data in
memory. Since tests are short-lived, I think that should be fine.

For now, this backend is stricter only in that it doesn't mix objects
written to different paths. We can make it strict/lossy in other ways
later (e.g. modifying written commit objects).

I think having a backend designed for tests can also be useful for
later making it possible to control the backend, e.g. to inject
errors.

We may want to replace almost all uses of the local backend in tests
with uses of this new test backend.
2023-09-18 07:53:19 -07:00
Martin von Zweigbergk
9c30d7500b testutils: delete bool-typed init() in favor of enum-typed version
It makes the call sites clearer if we pass the `TestRepoBackend` enum
instead of the boolean `use_git` value. It's also more extensible (I
plan to add another backend for tests).
2023-09-18 07:15:37 -07:00
Martin von Zweigbergk
50596c499e testutils: allow passing TestRepoBackend to TestWorkspace too 2023-09-18 07:15:37 -07:00
Martin von Zweigbergk
c6cf9d54f6 testutils: add an enum for TestRepo backend
I plan to add another backend for use in tests.
2023-09-18 07:15:37 -07:00
Martin von Zweigbergk
79527d707c testutils: use .jj-internal git repos in most tests
I don't think there's much reason to run most tests with a `.git`
directory outside of `.jj`. I think it's just that way for historical
reasons. It's been that way since I added support for `.jj`-internal
repos in a8a9f7dedd.

The reason I want to switch is to make it a little easier to create
test repos for different backends. The problem with `.jj`-external git
repos is that they depend on an additional path.

I had to update `test_bad_locking.rs` to make the code merging
directories able handle missing directories on some side, because
git's loose objects result in directories getting created on one or
both sides.
2023-09-18 07:15:37 -07:00
Martin von Zweigbergk
9d9b2cd057 tests: leverage create_tree() in a few more tests 2023-08-30 19:58:42 -07:00
Martin von Zweigbergk
962da1947e tests: make dump_tree() work with merged trees
My goal is to minimize impact on tests when we start using the new
format.
2023-08-30 06:17:21 -07:00
Martin von Zweigbergk
d9ce70c176 tests: make create_tree() return MergedTree
I think most tests want a `MergedTree`, so this makes `create_tree()`
return that. I kept the old function as `create_single_tree()`. That's
now only used in `test_merge_trees` and `test_merged_tree`.

I also consistently imported the functions now, something I've
considered doing for a long time.
2023-08-29 07:01:52 -07:00
Martin von Zweigbergk
e4c6595620 tests: make create_random_tree() return a MergedTreeId 2023-08-29 07:01:52 -07:00
Martin von Zweigbergk
1674a421ec commit_builder: take MergedTreeId for root id argument 2023-08-28 15:58:34 -07:00
Martin von Zweigbergk
a7e5ea06c0 tests: make test helper for snapshotting working copy return MergedTree 2023-08-27 06:49:45 -07:00
Martin von Zweigbergk
abf3853717 working_copy: return MergedTreeId on snapshot 2023-08-27 06:49:45 -07:00
Waleed Khan
1633eccdca Use { workspace = true } to appease VS Code's Cargo.toml parser
The VS Code "Better TOML" plugin (which I think most of our VS Code developers use?) doesn't support the `x.y = z` syntax at the top level, even though it's valid TOML. 

This is also useful if we ever want to add additional properties in different sub-crates (although unlikely for the near future).
2023-08-22 21:38:53 -07:00
Benjamin Saunders
417035cb20 tests: validate snapshot.max-new-file-size behavior 2023-08-17 19:29:38 -07:00
Benjamin Saunders
54f1d310c4 testutils: propagate snapshot errors 2023-08-17 19:29:38 -07:00
Yuya Nishihara
552c71ed36 tests: move commit_transactions() helper to testutils 2023-08-10 06:27:16 +09:00
Austin Seipp
d858db7e85 cargo: unify a lot of crate metadata in the workspace
Summary: There's no need to go around specifying `rust-version` or `edition` or
`version` several times, now that we have a global workspace. Instead, inherit
workspace metadata from the top-level Cargo.toml file.

Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: Iaf905445978ed2b3377239dcdb8a6c32
2023-08-06 16:44:33 -05:00