If a file gets replaced by a directory right after list files in a
directory but before we stat the file, we currently crash. Let's
instead treat it as a missing file, using the mechanism introduced for
#258.
This patch makes us treat special files (e.g. Unix sockets) as absent
when snapshotting the working copy. We can consider later reporting
such files back to the caller (possibly via callback) so it can inform
the user about them.
Closes#258
This patch is essentially f6a516ff6d taken further, to also apply to
when we write a symlink or a conflict. As with regular files, these
races seem very unlikely to happen, but I found these cases while
working on #258, so let's fix. Fixing it also means that we don't need
to handle these transition cases in the next patch (when
`file_states()` can indicate that the file is e.g. a socket).
The regular `Display` format is (not surprisingly) more user-friendly,
as pointed out by @yuja.
I also switched to using format strings for these cases, and some
nearby strings for consistency.
We can easily make the `DirEntry` available here, so we can call
`.metadata()` on that instead of on the `Path`. I think that avoids
walking the path. I'm sure this has no significant impact on
performance, but it's also almost as readable.
When we have just written a file to disk on checkout, let's record the
size we expected instead of what we got from `fstat()`. This should
address a race where the file was modified between the time we wrote
it and the time we requested its stat. I just happened to notice this
while going through the code; it seems very unlikely to be noticed in
practice.
When we have just written a file or conflict, we can get metadata for
it via the open file descriptor instead of using the path. That
removes the risk of a race where the file got removed or replaced by
another file type (at least on Unix).
These assertions were there to catch bugs, but when the bugs happen,
the assertions can obsure the underlying error (as @tp-woven found out
on #258). Let's just print errors instead.
The biggest difference in the API is that fields are now public. The
exception from that is `oneof` fields, which still require setters and
getters.
I couldn't measure any difference in performance. I didn't expect any
difference either, but it's good that it didn't seem to regress. I
timed `jj debug operation <some hash prefix>`, which will read the
whole operation log (to check that the prefix is unambiguous).
I think I copied the name `write_tree()` from Git, but I find it quite
confusing, since it's not clear if it write a tree to the working copy
or reads the working copy and writes a tree to the store (it's the
former).
When committing the working copy, we try to not visit ignored
directories (as e.g. `target/` often is), but we need to visit it if
there are already tracked files in it. I initially missed that in
c1060610bd and then fixed it in a028f33e3b. The fix works by
checking if the next path after the ignored path is inside the ignore
path (viewed as a directory). However, I forgot to handle the case
where there are no paths at all after the ignored path. So, for
example, if the `target/` directory should be ignored and it there
were no tracked paths after `target/` in alphabetical order, we would
still visit the directory. That's why the bug reproduced in the
`git-branchless` repo but not in the `jj` repo (because there are
files under `testing/` and `tests/` here).
Closes#247.
This patch makes room for sparse patterns in the `TreeState` proto
message. We also start setting that value to a list of just the
pattern `.` when we create new working copies. Old working copies
without the sparse patterns are also interpreted as having that single
pattern. Note that this absence of sparse patterns is different from a
present list of no patterns. The latter is a valid state and means
that no paths are included in the sparse checkout.
We do it for all the other kinds of objects already. It's useful to
have the path for backends that store objects by path (we don't have
any such backends yet). I think the reason I didn't do it from the
beginning was because we had separate `RepoPath` types for files and
directories back then.
The library crate shouldn't look up the user's `$HOME` directory
(maybe the library is used by a server process), so let's have the
caller pass it into the library crate instead.
We no longer need the commit ID, so we shouldn't make the callers pass
it. This lets us simplify several tests, because they no longer to
create commits just to check out a tree in the working copy.
We used to use the value to detect races, but we use the tree ID and
the operation ID these days, so we don't need the commit ID.
By changing this, we can avoid creating some commit IDs in tests,
which is why I tackled this issue now.
There are only two callers of `LockedWorkingCopy::check_out()`. One is
in `commands.rs`. That caller already checks after taking the lock
that the old commit ID is as expected. The other caller is
`WorkingCopy::check_out()`. We can simply move the check to that level
since it's the only caller that cares now.
`LockedWorkingCopy::discard()` shouldn't result in changes to the
on-disk state, but `LockedWorkingCopy::check_out()` may have already
written a state file, which is surprising. The changes also remain in
memory, which is also surprising. Let's fix both of those issues.
When there are concurrent operations that want to update the working
copy, it's useful to know which operation was the last to successfully
update the working copy. That can help use decide how to resolve a
mismatch between the repo view's record and the working copy's
record. If we detect such a difference, we can look at the working
copy's operation ID to see if it was updated by an operation before or
after we loaded the repo.
If the working copy's record says that it was updated at operation A
and we have loaded the repo at operation B (after A), we know that the
working copy is stale, so we can automatically update it (or tell the
user to run some command to update it if we think that's more
user-friendly).
Conversely, if we have loaded the repo at operation A and the working
copy's record says that it was updated at operation B, we know that
there was some concurrent operation that updated it. We can then
decide to print a warning telling the user that we skipped updating
because of the conflict. We already have logic for not updating the
working copy if the repo is loaded at an earlier operation, but maybe
we can drop that if we record the operation in the working copy (as
this patch does).
Having the checkout functionality in `LockedWorkingCopy` makes it a
little more flexible (one could imagine using it for udating working
copy files and then discarding the state changes, for example). It
also lets us reuse a few lines of code for locking. I left
`WorkingCopy::check_out()` for convenience because that's what all
current users want.
`WorkingCopy::check_out()` currently fails if the commit recorded on
disk has changed since it was last read. It fails with a "concurrent
checkout" error. That usually works well in practice, but one can
imagine cases where it's not correct. For an example where the current
behavior is wrong, consider this sequence of events:
1. Process A loads the repo and working copy.
2. Process B loads the repo at operation A. It has not loaded the
working copy yet.
3. Process A writes an operation and updates the working copy.
4. Process B loads the working copy and sees that it is checked out
to the commit process B set it to. We don't currently have any
checks that the working copy commit matches the view's checkout
(though I plan to add that).
5. Process B finishes its operation (which is now divergent with the
operation written by process A). It updates the working copy to
the checkout set in the repo view by process B. There's no data
loss here, but the behavior is surprising because we would usually
tell the user that we detected a concurrent update to the working
copy.
We should instead check that the working copy's commit on disk matches
what the previous repo view said, i.e. the view at the start of the
operation we just committed. This patch does that by having the caller
pass in the expected old commit ID.
We already have two usecases that can be modeled as updating the
`TreeState` without touching the working copy:
1. `jj untrack` can be implemented as removing paths from the tree
object and then doing a reset of the working copy state.
2. Importing Git HEAD when sharing the working copy with a Git repo.
This patch adds that functionality to `TreeState`.
I was surprised that we save the `TreeState` before
`LockedWorkingCopy::finish()`. That means that even if the caller
instead decides to discard the changes, some changes will already have
been written.
This patch changes the interface for making changes to the working
copy by replacing `write_tree()` and `untrack()` by a single
`start_mutation()` method. The two functions now live on the returned
`LockedWorkingCopy` object instead. That is more flexible because the
caller can make multiple changes while the working copy is locked. It
also helps us reduce the risk of buggy callers that read the commit ID
before taking the lock, because we can now make it accessible only on
`LockedWorkingCopy`.
The working copy object knows the currently checked out commit ID. It
is set to `None` when the object is initialized. It is also set to
`None` when an existing working copy is loaded. In that case, it's
used only to facilitate lazy loading. However, that means that
`WorkingCopy::current_commit_id()` fails if the working copy has been
initalized but no checkout has been specified. I've never run into
that case, but it's ugly that it can happen. This patch fixes it by
having `WorkingCopy::init()` take a `CommitId`.
`WorkingCopy::current_commit()` has been there from the beginning. It
has made less sense since we made the repo view keep track of the
current checkout. Let's remove it.
I think these remaining implementations of `Drop` are for types that
are infrequently dropped (unlike `Transaction`), so it be fine to be
more strict about them.
The recent e5dd93cbf7, whose description says "cleanup: make Vec
inside CommitId etc. non-public", made all ID types in the `backend`
module *except* for `CommitId` non-public :P This patch makes
I realized only recently that we can try to parse conflict markers in
files and leave them as conflicted if they haven't changed. If they
have changed and some conflict markers have been removed, we can even
update the conflict with that partial resolution.
This change teaches the working copy to write conflicts to the working
copy. It used to expect that the caller had already updated the tree
by materializing conflicts. With this change, we also start parsing
the conflict markers and leave the conflicts unresolved in the working
copy if the conflict markers remain.
There are some cases that we don't handle yet. For example, we don't
even try to set the executable bit correctly when we write
conflicts. OTOH, we didn't do that even before this change.
We still never actually write conflicts to the working copy (outside
of tests) because we currently materialize conflicts in
`MutRepo::check_out()`. I'll change that next.
On Windows, we preserve the executable bit. I plan to also teach the
working copy to preserve conflict state. This refactoring prepares for
that by simplifying how we preserve parts of the current file state.
We have had support for ignores via `.gitignore` files since
3b326a942c, and we haven't had the problem with the temporary
`.git/` directory created by libgit2 since 88f7f4732b.
Git doesn't want `.git` entries in its trees, so at least when using
the Git backend, we need to ignore such paths. Let's just ignore
`.git` paths regardless of backend to keep it simple.
Closes#24.
The auto-rebasing of descendants doesn't work if you have an open
commit checked out, which means that you may still end up with orphans
in that case (though that's usually a short-lived problem since they
get rebased when you close the commit). I'm also about to make
branches update to successors, but that also doesn't work when the
branch is on a working copy commit that gets rewritten. To fix this
problem, I've decided to let the caller of `WorkingCopy::commit()`
responsible for the transaction.
I expect that some of the code that this change moves from the lib
crate to the cli crate will later move back into the lib crate in some
form.
Unlike the other places I fixed in 134940d2bb, the calls in
`working_copy.rs` should not simply use an existing file if the target
file was open. They should probably try again instead, but I'll leave
that for later.
On Windows, it seems that you can't rename a file if the target file
is open (Stebalien/tempfile#131). I think that's the reason for our
failing tests on Windows. This patch adds a simple wrapper around
`NamedTempFile::persist()` that returns the existing file instead of
failing, if there is one.
This change teaches `Tree::diff()` to filter by a matcher. It only
filters the result so far; it does not restrict the tree walk to what
`Matcher::visit()` says is necessary yet. It also doesn't teach the
CLI to create a matcher and pass it in.
I had initially hoped that the type-safety provided by the separate
`FileRepoPath` and `DirRepoPath` types would help prevent bugs. I'm
not sure if it has prevented any bugs so far. It has turned out that
there are more cases than I had hoped where it's unknown whether a
path is for a directory or a file. One such example is for the path of
a conflict. Since it can be conflict between a directory and a file,
it doesn't make sense to use either. Instead we end up with quite a
bit of conversion between the types. I feel like they are not worth
the extra complexity. This patch therefore starts simplifying it by
replacing uses of `FileRepoPath` by `RepoPath`. `DirRepoPath` is a
little more complicated because its string form ends with a '/'. I'll
address that in separate patches.
This makes the workging copy walk skip an entire ignored directory if
there are no negative patterns later in the ignore file. That speeds
up `jj st` in this repo with ~13k files in `target/` from ~100 ms to
~25 ms (6.0dB). This closes issue #8.
This is to address issue #8. I haven't added the optimization to avoid
walking all the files in `target/` yet. Even so, this patch still
speeds up `jj st` in this repo, with ~13k files in `target/`, from
~320 ms to ~100 ms (-5.1dB). The time actually checking if paths match
gitignores seems to go down from 116 ms to 6 ms. I think that's mostly
because libgit2 has to look for `.gitignore` files in every parent
directory every time we ask it about a file, while the rewritten code
looks for a `.gitignore` file only when visiting a new directory.
I've wanted the API to look like this for a while. It seems like a
good API to me. It means that the caller won't have to reload the repo
after committing. The cost seems relatively small. It involves copying
potentially a lot of data in memory (at least the View object), but it
shouldn't involve reading from disk or any other processing. To reduce
the amount of data to copy, it may be worth switching to persistent
data types. I've also wanted to do that for the copying we do when
start a transaction.
I couldn't measure any slowdown caused by this change.
This is yet another step towards making it easy to propagate
`BrokenPipe` errors. The `jj diff` code (naturally) diffs two trees
and prints the diffs. If the printing fails, we shouldn't just crash
like we do today.
The new code is probably slower since it does more copying (the
callback got references to the `FileRepoPath` and `TreeValue`). I hope
that won't make a noticeable difference. At least `jj diff -r
334afbc76fbd --summary` didn't seem to get measurably slower.
I just changed my `~/.gitignore` and some tests started failing
because the working copy respects the user's `~/.gitignore`. We should
probably not depend on `$HOME` in the library crate. For now, this
patch just makes sure we set it to an arbitrary directory in the tests
where it matters.
I only noticed that there was a newer version when running `cargo
install --path .`, which resulted in warnings about deprecated
functions. There's no other reason I'm aware of to upgrade now.
TreeState::write_tree leaves a ".git" file in the working copy. This is
undesirable but more problematic on Windows - The second time
TreeState::write_tree would panic because Repository::init_opts will fail
with a Permission Denied error.
This seems to be a libgit2 defect. But for now let's just remove ".git"
automatically. This makes `cargo test --test smoke_test` pass on Windows.
Evolution needs to have fast access to the predecessors. This change
adds that information to the commit index.
Evolution also needs fast access to the change id and the bit saying
whether a commit is pruned. We'll add those soon.
Some tests changed because they previously added commits with
predecessors that were not indexed, which is no longer allowed from
this change. (We'll probably eventually want to allow that again, so
that the user can prune predecessors they no longer care about from
the repo.)
Windows doesn't support recording the executable bit in the file
system. Before this commit, the code for reading and writing the
executable wouldn't even compile on Windows. This commit at least
makes it so we preserve whatever bit has been recorded in the repo.
At least I hope that's what it does -- I don't have access to a
Windows machine right now.
The project doesn't currently build on Windows. One reason is because
we had a `unimplemented!()` when trying to write a symlink. Let's
print a warning instead, so the project can start building on
Windows. (The next patch will fix another build problem on Windows.)
This adds an interactive mode for `jj restore`. It works by first
creating two temporary directories with the contents of the subset of
files that differ between the two trees, and then letting the user
edit the directory representing the right/after side. This has some
advantages compared to the interactive modes in Git and Mercurial:
* It lets the user edit the final state as opposed to the diff itself
(depending on the diff tool, of course). I think most users find it
easier to edit the file contents than to edit the patch
format.
* It delegates the hard work to a tool that is already written (this
is a big advantage for an immature tool like Jujube, but it is not
an advantage from the user's point of view).
Almost all of the work in this commit went into adding a function that
takes two trees, lets the user edit the diff, and returns a new tree
id. I plan to reuse that function for other interactive commands. One
planned command is `jj edit`, which will let the user edit the changes
in a commit. `jj edit -r abc123` will be mostly about providing a more
intuitive name for `jj restore --source abc123^ --destination abc123`,
plus it will be different for merge commits (it will edit only the
changes in the merge commit). I also plan to add `jj split` by letting
the user edit the full diff, leaving only the parts that should go
into the first commit. Perhaps there will also be commands for moving
part of a commit out of or into a parent commit.
I had tried to generate the protobuf code at build time many months
ago, but decided against it because it slowed down the build too
much. I didn't realize there was the
"cargo:rerun-if-changed=<filename>" feature that time. Given that that
exists, it seems like an obvious win to generate the source code at
build time.
I put the generated sources in `$OUT_DIR` (where [1] says they should
be), then include them in the `protos` module by using the `include!`
macro. The biggest problem with that is that I couldn't get IntelliJ
to understand it, even after enabling the experimental features
described in [2].
[1] https://doc.rust-lang.org/cargo/reference/build-script-examples.html#code-generation
[2] https://github.com/intellij-rust/intellij-rust/issues/1908#issuecomment-592773865
The project's source of truth is now in Git and I really miss support
for anonymous heads and evolution (compared to when the code was in
Mercurial). I'm therefore more motivated to make the tool useful for
day-to-day work on small repos, so I can use it myself. Until now, I
had been more focused on improving performance when it was used as a
read-only client for medium-to-large repos.
One important feature for my day-to-day work is support for
ignores. This commit adds simple and effective, but somewhat hacky
support for that. libgit2 requires a repo to check if a file should be
ignored (presumably so it can respect `.git/info/excludes`). To work
around that, we create a temporary git repo in `/tmp/` whenever the
working copy is committed. We set that temporary git repo's working
copy to be shared with our own working copy. Due to
https://github.com/libgit2/libgit2sharp/issues/1716 (which seems to
apply to the non-.NET version as well), this workaround unfortunately
leaves a .git file (pointing to the deleted temporary git repo) around
in every Jujube repo. That's always ignored by libgit2, so it's not
much of a problem.
