We're more likely to filter out empty commits, so this should be slightly
faster in practice.
The extra Option<> isn't needed, but it should clarify that "prefix([])"
is not "everything".
This basically transforms 's1 & (f() | s2)' to
's1.iter().filter(all && f || s2)'. Still the predicate part includes "all",
the filter function doesn't need to load commit data for every entry since
's1.iter().filter(all)' is tested first. To optimize "all" predicate out,
maybe we can add a wrapper that returns '|_: &IndexEntry| true'.
Instead of inserting AsFilter(_) node, I could add a recursive is_filter()
function. That would also work so long as the height of RevsetExpression tree
is limited. I chose node insertion just for ease of snapshot testing.
The idea behind this is to extend RevWalk to track generation (or depth from
the initial wanted items.) Basic DAG walk doesn't need such data, but a query
like 'rev---' could be translated to a RevWalk yielding nth ancestors.
The default log revset can also be expressed as 0/1-th ancestors of
'(remote_branches() | tags())..'.
Also, this appears to be faster than using boundary sets, based on the
bench extracted from test_index_commits_criss_cross().
In order to optimize a query like '(author(_) | @) & main..', we'll probably
need a predicate form of an iterable set so that the query can be evaluated
to '(main..).iter().filter(author(_) | @)'. And if a predicate function can
terminate the source iterator early (by returning true/false/false_forever),
complexity of a filtered revset is basically the same as an intersection of
iterator pair. This means we can eventually merge IntersectionRevset with
FilterRevset.
With that in mind, this patch removes the redundant 'candidates' field from
the filter node, which would otherwise appear in the predicate function as
'candidates.contains(entry)'. A filter node with candidates was somewhat
useful while rewriting the tree, but that can be dealt with a view function
like as_filter_intersection() in this patch.
This also simplify the subsequent filter transformation as we no longer need
to test if candidates == All.
Previously we only have a test for the left recursion. The added test
contains right recursion path, which should have caught the error I made
while working on the next "unary filter node" patch.
The doc comment summarizes what I'm going to implement. I'm not sure if
we'll add all of them because revset evaluation isn't the key performance
bottleneck at the moment. Anyway, I don't think any of these ideas would
logically conflict with segmented changelog adaptation unless we decide to
replace the whole revset stack with Eden/Sapling's.
To prevent git's GC from breaking a repo, we already add a git ref to
commits we create in the git backend. However, we don't add refs to
commits we import from git. This fixes that.
Closes#815.
There's no need to update our record of the ref if it didn't
change. This is just about making it clearer; I doubt it will have
measurable performance impact.
This patch adds a `legacy-thrift` Cargo feature that's enabled by
default. If it's disabled, the upgrade from Thrift-based operation log
does not happen, and the `thrift` depdendency is not included.
With this patch, we auto-upgrade existing repos that use Thrift format
for the operation log to use Protobuf format. That would only be repos
used with an unreleased version of jj after 0.5.1 (which may be the
majority of repos?).
The upgrade from Thrift is simpler because we now use the same hashing
scheme for the Protobuf-based storage, so the operation and view IDs
remain the same as they were in the Thrift-based storage. We could
simplify the code a bit more as a result, but since this code is
supposed to be short-lived, I didn't bother.
Since the change from the Protobuf format with the old hashing scheme
to a the (same) Protobuf format with the new hashing scheme shouldn't
impact users, I removed the entry we had in the changelog about the
format change.
The code for migrating from ProtoBuf to Thrift is almost completely
independent of which direction the upgrade goes, so we can very easily
reuse it for migrating from Thrift to Protobuf. This patch renames
some variables to "old/new" instead of "proto/thrift", making the next
patch even simpler.
Since we're now allowed to use the `protobuf` crate, I'm going to make
`SimpleOpStore` use it again. This moves the `ThriftOpStore` into a
new `legacy_thrift_op_store.rs` file.
Since we now have approval to use the `protobuf` crate at Google, it's
no longer the "legacy" format, so we should remove it. I'll almost
definitely soon add `legacy_thrift` feature instead.
This refactors `conflicts.rs` to:
1. Make `describe_conflict` public
2. Extract the functionality to create text version of
a conflict as the `materialize_merge_result` function.
3. Extract the functionality to turn a conflicted file
into the complete contents of each version of the file
"added" or removed" (when possible). This becomes the
`extract_file_conflict_data` function.
This is useful in order to present these text versions
in a merge tool.
The function doesn't do much at all now and there's a single caller,
so let's inline it.
I tried to clean up the code a bit futher so it wouldn't even create
the `old_view`, but it was harder than I had hoped. I might get back
to it later.
@yuja asked on #701 about the difference between the state in the
`git_export_view` and what we have in `mut_repo.view()`. It's true
that the branches in `mut_repo.view().git_refs()` should match what we
wrote to disk. We can therefore remove the on-disk storage and
simplify quite a bit. For now, I create the `last_export_view` from
the `mut_repo.view().git_refs()` before calling
`export_changes()`. I'll clean up a bit more next.
I think this is correct even considering e.g. undo. Let's consider
what would happen in a non-colocated Git repo (not because tricky
cases cannot happen there but because the explicit exports and imports
make it easier to discuss, and more cases can occur). If the user
moved a branch and then did `jj git export`, `jj undo`, and then `jj
git export` again, we would think on the second export that we should
perform the same changes to the Git repo, which should have no effect.
This patch also fixes the bug we were forced to work around in the
test case in the previous patch.
This removes one of our uses of Thrift.
This fixes the bugs shown by the tests added in the previous patch by
checking that the git branches we're about to update have not been
updated by git since our last export. If they have, we fail those
branches. The user can then re-import from the git repo and resolve
any conflicts before exporting again.
I had to update the `test_export_import_sequence` to make it
pass. That shows a new bug, which I'll fix next. The problem is that
the exported view doesn't get updated on import, so we would try to
export changes compared to an earlier export, even though we actually
knew (because of the `jj git import`) that the state in git had
changed.
If you update a branch using regular `git` (or some Git-based tool)
between two `jj git export`, we will overwrite that change if you had
also changed the branch in jj land. There's a similar problem if you
delete the branch in jj land. Let's have a test for that. I'm going to
make us not overwrite it soon. This patch adds a test for those cases,
plus many other cases in consistent way. Since the new test covers
some cases tested by existing tests, I removed those tests.
The Protobuf team at Google decided to let us use Protobufs internally
after all. That will make things a little easier for us with the
Google-internal adapations, and the `protobuf` crate is noticeably
faster than the `thrift` crate.
This effectively rolls back commit 5b10c9aa0a. I resolved some
conflicts caused by the rename from `NormalFile` to `File`. I also
kept the changelog entry, but I changed it to say that the hashing
scheme has changed (not the format), but since the hashes are just
used for identity, existing repos should still work.
It seems that we didn't have a test for this simple case. I wrote this
test case while working on #111 but I don't know why I didn't push it
back then.
A new FileType, GitSubmodule is added which is ignored. Files or
directories having this type are not added to the work queue and
are ignored in snapshot. Submodules are not created by jujutsu
when resetting or checking out a tree, they should be currently
managed using git.
Writing double negates is silly, but it might be hidden by revset alias
if we added such feature.
I made fold_redundant_expression() a separate step from fold_difference()
since I'll probably want to apply the cleanup step before rewriting filter
expressions.
Because a unary negation node '~y' is more primitive than the corresponding
difference node 'x~y', '~y' is easier to deal with while rewriting the tree.
That's the main reason to add RevsetExpression::NotIn node.
As we have a NotIn node, it makes sense to add an operator for that. This
patch reuses '~' token, which I feel intuitive since the other set operators
looks like bitwise ops. Another option is '!'.
The unary '~' operator has the highest precedence among the set operators,
but they are lower than the ranges. This might be counter intuitive, but
useful because a prefix range ':x' can be negated without parens.
Maybe we can remove the redundant infix operator 'x ~ y', but it isn't
decided yet.
Function parameters are processed as local symbols while substituting
alias expression. This isn't as efficient as Mercurial which caches
a tree of fully-expanded function template, but that wouldn't matter in
practice.
Let's acknowledge everyone's contributions by replacing "Google LLC"
in the copyright header by "The Jujutsu Authors". If I understand
correctly, it won't have any legal effect, but maybe it still helps
reduce concerns from contributors (though I haven't heard any
concerns).
Google employees can read about Google's policy at
go/releasing/contributions#copyright.
Follows up c5ed3e1477. Now change/commit ids are resolved at the same
precedence, which means there are at least three types of ambiguity.
I don't think we would need to discriminate these.
Because the use of the change id is recommended, any operation should abort
if a valid change id happens to match a commit id. We still try the commit
id lookup first as the change id lookup is more costly.
Ambiguous change/commit id is reported as AmbiguousCommitIdPrefix for now.
Maybe we can merge AmbiguousCommit/ChangeIdPrefix errors into one?
Closes#799
84b924946f switched to requiring both
SSH_AUTH_SOCK and SSH_AGENT_PID for an agent to be used. This doesn't
seem to be a typical situation, so perhaps it was not intended.
Since syntactic information like symbol or function name is lost after
parse(), alias substitution is inserted to the middle of the post-parsing
stage, not after the whole RevsetExpression tree is built. This is the main
difference from Mercurial. Mercurial also caches parsed aliases, but I don't
think that would have a measurable impact.
The CLI will load aliases from config, insert them one by one, and warn if
declaration part is invalid. That's why RevsetAliasesMap is a public struct
and needs to be instantiated by the caller.
I'll add aliases map, substitution stack (to detect recursion), and locals
(for function aliases) there. Fortunately, we can avoid shared mutables
so a copyable struct should be good.
parse_function_argument_to_string() doesn't need a workspace_ctx, but there
should be no reason to explicitly nullify it either.
To reduce conflicts between branches like `main` and `main/sub`, it's
better to first delete refs in git that have been deleted in jj, and
then add/update refs that have been added/updated in jj.
Since we now write a (partial) view object of the exported branches to
disk (since 7904474320), we can safely skip exporting some
branches. We already skip conflicted branches. This commit makes us
also skip branches that we fail to write to the backing Git repo,
instead of failing the whole operation (after possibly updating some
Git refs).
I made the `export_refs()` function return the branches that
failed. We should probably make that a struct later and have a
separate field for branches that we skipped due to conflicts.
Closes#493.
When skipping branches we fail to update in the backing Git repo, we
must also skip updating the `exported_view` object, so we don't trick
ourselves into thinking the branch was already updated in the Git repo
on the next export.
I'm going to make the export skip branches that we fail to update in
the Git repo. For that, we need to know the branch name while
interacting with the `git2::Repository` object. This little
refactoring prepares for that.
The comment says that we collect the changes to make before making
them, in order to reduce the risk of making some changes before
failing. However, there is nothing in the code that collects changes
that can fail, and it's all doing comparisons in memory, so it should
be very fast. It's been like that since I added it in 47b3abd0f7. We
still need to preserve the structure to avoid mutating `mut_repo`
while iterating over branches, however, so I just updated the comment.
This adds a test for attempting to export both a branch called `main`
and one called `main/sub` (#493), as well as for exporting a branch
with an empty string as name (reported directly to me by @lkorinth).
I'm thinking of adding alias expansion at this stage, and it would be a bit
tedious to pass around mutable context by function parameter. So let's reduce
the number of the intermediate functions.
This also produces a better error message.
It would be nice to be able to use snapshot testing and not have to
parse the output of `jj op log`. This patch lets us do that by
providing a new environment variable and config for overriding the
timestamps. Unlike `operation.hostname` and `operation.username`,
these are only meant for tests.
This makes the tests more hermetic, even though I don't think the
default values (taken from `whoami`) can break any tests (then we
would have already seen them break). Now we just need to make the
operation log's timestamps predictable and then we can start using
operation IDs in snapshot tests.
The expression 'x ~ empty()' is identical to 'x & file(".")', but more
intuitive.
Note that 'x ~ empty()' is slower than 'x & file(".")' since the negative
intersection isn't optimized right now. I think that can be handled as
follows: 'x ~ filter(f)' -> 'x & filter(!f)' -> 'filter(!f, x)'
There are no "non-normal" files, so "normal" is not needed. We have
symlinks and conflicts, but they are not files, so I think just "file"
is unambiguous.
I left `testutils::write_normal_file()` because there it's used to
mean "not executable file" (there's also a `write_executable_file()`).
I left `working_copy::FileType::Normal` since renaming `Normal` there
to `File` would also suggest we should rename `FileType`, and I don't
know what would be a better name for that type.
We currently get the hostname and username from the `whoami` crate. We
do that in lib crate, without giving the caller a way to override
them. That seems wrong since it might be used in a server and
performing operations on behalf of some other user. This commit makes
the hostname and username configurable, so the calling crate can pass
them in. If they have not been passed in, we still default to the
values from the `whoami` crate.
This migrates the native backend from Protobuf to Thrift since
Google's Protobuf team does let us import jj into Google's monorepo if
it uses a third-party Protobuf library.
Since the native backend is not supported, I didn't write any
migration code for it.
We can't remove `lib/src/protos/store.proto` yet, because it's also
used by the Git backend (only the `predecessors` and `change_id`
fields).
When we export branches to Git, we didn't update our own record of
Git's refs. This frequently led to spurious conflicts in these refs
(e.g. #463). This is typically what happened:
1. Import a branch pointing to commit A from Git
2. Modify the branch in jj to point to commit B
3. Export the branch to Git
4. Update the branch in Git to point to commit C
5. Import refs from Git
In step 3, we forgot to update our record of the branch in the repo
view's `git_refs` field. That led to the import in step 5 to think
that the branch moved from A to C in Git, which conflicts with the
internal branch target of B.
This commit fixes the bug by updating the refs in the `MutableRepo`.
Closes#463.
As I said in the previous patch, I don't know why I made the initial
export to Git a no-op. Exporting everything makes more sense to
(current-)me. It will make it slightly easier to skip exporting
conflicted branches (#463). It also lets us remove a `jj export` call
from `test_templater.rs`.
The first time we export to Git, we don't actually export
anything. That's a little weird and I don't know why I did it that
way. It seems more natural to export the current state. I'd like to
change it to do that. However, that currently means we'll detach the
current HEAD if it points to any of the branches we export. This patch
restructures the code a bit and skips the detach step if the target
branch already points to the right commit.
We could write the view object to the operation store instead, but
then we would need to make sure we don't GC it (once we add support
for GC of the operation store).
I'm going to add other error variants for when we fail to read/write
to disk, and I don't think we need to have a custom message for each
in the CLI crate. It's easier to pass along the message from the lib
crate.
(`ConflictedBranch`, on the other hand, is an expected error so I
think we should continue to let let the CLI crate define the error
message for it.)
In order to fix#463, I'm going to make us export to Git a little
earlier, before finishing the transation. That means we won't have an
operation yet, but we don't need that anyway.
To fix#463, I think we want to skip conflicted branches when we
export instead of erroring out. It seems we didn't have test case for
the current behavior, so let's add one.
This is a test case for #463. It's not exactly the same case, but I'm
confident that the root cause is the same (that the
`.jj/repo/git_export_operation_id` doesn't include the git refs we
just updated).
These calls often appear in expressions long enough that not having to
qualify it means that we can sometimes avoid wrapping a line. I
noticed because IntelliJ told me that `test_git.rs` had some
unnecessary qualificiations (the function was already imported there).
As mentioned in the previous commit, we need to remove the Protobuf
dependency in order to be allowed to import jj into Google's
repo. This commit makes `SimpleOpStore` store its data using Thrift
instead of Protobufs. It also adds automatic upgrade of existing
repos. The upgrade process took 18 s in my repo, which has 22k
operations. The upgraded storage uses practically the same amount of
space. `jj op log` (the full outut) in my repo slowed down from 1.2 s
to 3.4 s. Luckily that's an uncommon operation. I couldn't measure any
difference in `jj status` (loading a single operation).
In order to allow building jj inside of Google, our Protobuf team
doesn't want to us to use a Google-unsupported implementation. Since
there is no supported implementation in Rust, we have to migrate off
of Protobufs. I'm starting with the operation store. This commit moves
the current implementation to a separate file so it can easily be
disabled by a Caargo feature.
Decouples view/operation IDs from serialized forms, which are not
necessarily stable. Not breaking as these IDs are persistent, never
recomputed or used for integrity checking.
This should help us reason about the safety implication. New inner module
is added to encapsulate unsafe access.
DirtyCell provides .with_ref(callback) instead of .borrow(). This isn't
strictly needed, but should clarify the intent of the temporary reference.
This also allows us to rewrite DirtyCell without unsafe code, if needed,
by leveraging OnceCell<T> x RefCell<Option<T>> pair.
The `testutils` module should ideally not be part of the library
dependencies. Since they're used by the integration tests (and the CLI
tests), we need to move them to a separate crate to achieve that.
Unfortunately, TOML requires quotes around the argument. So, the
usage is `jj --config-toml ui.color=\"always\"` in bash. The plan is
to eventually have a `--config` option with simpler syntax for
simple cases.
As discussed in https://github.com/martinvonz/jj/discussions/688.
If I'm reading this attribute correctly, it says that if the
`map_first_last` feature is enabled, then we should enable the
`map_first_last` feature, which seems like it would not have any
effect. We started getting warnings from the nightly compiler about
this line because it tries to enable a feature that's stable in that
version.
