Code at the speed of thought – Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.
Find a file
Thorsten Ball 0533923f91
Reduce memory usage to represent buffers by up to 50% (#10321)
This should help with some of the memory problems reported in
https://github.com/zed-industries/zed/issues/8436, especially the ones
related to large files (see:
https://github.com/zed-industries/zed/issues/8436#issuecomment2037442695),
by **reducing the memory required to represent a buffer in Zed by
~50%.**

### How?

Zed's memory consumption is dominated by the in-memory representation of
buffer contents.

On the lowest level, the buffer is represented as a
[Rope](https://en.wikipedia.org/wiki/Rope_(data_structure)) and that's
where the most memory is used. The layers above — buffer, syntax map,
fold map, display map, ... — basically use "no memory" compared to the
Rope.

Zed's `Rope` data structure is itself implemented as [a `SumTree` of
`Chunks`](8205c52d2b/crates/rope/src/rope.rs (L35-L38)).

An important constant at play here is `CHUNK_BASE`:

`CHUNK_BASE` is the maximum length of a single text `Chunk` in the
`SumTree` underlying the `Rope`. In other words: It determines into how
many pieces a given buffer is split up.

By changing `CHUNK_BASE` we can adjust the level of granularity
withwhich we index a given piece of text. Theoretical maximum is the
length of the text, theoretical minimum is 1. Sweet spot is somewhere
inbetween, where memory use and performance of write & read access are
optimal.

We started with `16` as the `CHUNK_BASE`, but that wasn't the result of
extensive benchmarks, more the first reasonable number that came to
mind.

### What

This changes `CHUNK_BASE` from `16` to `64`. That reduces the memory
usage, trading it in for slight reduction in performance in certain
benchmarks.

### Benchmarks

I added a benchmark suite for `Rope` to determine whether we'd regress
in performance as `CHUNK_BASE` goes up. I went from `16` to `32` and
then to `64`. While `32` increased performance and reduced memory usage,
`64` had one slight drop in performance, increases in other benchmarks
and substantial memory savings.

| `CHUNK_BASE` from `16` to `32` | `CHUNK_BASE` from `16` to `64` |
|-------------------|--------------------|
|
![chunk_base_16_to_32](https://github.com/zed-industries/zed/assets/1185253/fcf1f9c6-4f43-4e44-8ef5-29c1e5d8e2b9)
|
![chunk_base_16_to_64](https://github.com/zed-industries/zed/assets/1185253/d82a0478-eeef-43d0-9240-e0aa9df8d946)
|

### Real World Results

We tested this by loading a 138 MB `*.tex` file (parsed as plain text)
into Zed and measuring in `Instruments.app` the allocation.

#### standard allocator
Before, with `CHUNK_BASE: 16`, the memory usage was ~827MB after loading
the buffer.

| `CHUNK_BASE: 16` |
|---------------------|
|
![memory_consumption_chunk_base_16_std_alloc](https://github.com/zed-industries/zed/assets/1185253/c1e04c34-7d1a-49fa-bb3c-6ad10aec6e26)
|


After, with `CHUNK_BASE: 64`, the memory usage was ~396MB after loading
the buffer.

| `CHUNK_BASE: 64` |
|---------------------|
|
![memory_consumption_chunk_base_64_std_alloc](https://github.com/zed-industries/zed/assets/1185253/c728e134-1846-467f-b20f-114a582c7b5a)
|


#### `mimalloc`

`MiMalloc` by default and that seems to be pretty aggressive when it
comes to growing memory. Whereas the std allocator would go up to
~800mb, MiMalloc would jump straight to 1024MB.

I also can't get `MiMalloc` to work properly with `Instruments.app` (it
always shows 15MB of memory usage) so I had to use these `Activity
Monitor` screenshots:

| `CHUNK_BASE: 16` |
|---------------------|
|
![memory_consumption_chunk_base_16_mimalloc](https://github.com/zed-industries/zed/assets/1185253/1e6e05e9-80c2-4ec7-9b0e-8a6fa78836eb)
|

| `CHUNK_BASE: 64` |
|---------------------|
|
![memory_consumption_chunk_base_64_mimalloc](https://github.com/zed-industries/zed/assets/1185253/8a47e982-a675-4db0-b690-d60f1ff9acc8)
|

### Release Notes

Release Notes:

- Reduced memory usage for files by up to 50%.

---------

Co-authored-by: Antonio <antonio@zed.dev>
2024-04-09 18:07:53 +02:00
.cargo Enable tokio-console (#8897) 2024-03-05 10:56:14 -07:00
.config
.github Bump Python version in actions 2024-04-05 08:24:00 -04:00
.zed Format prettier_server.js (#9583) 2024-03-20 19:49:14 +01:00
assets Switch to the system UI font on macOS (#10317) 2024-04-09 07:44:57 -06:00
crates Reduce memory usage to represent buffers by up to 50% (#10321) 2024-04-09 18:07:53 +02:00
docs chore: Remove tasks.md (#10273) 2024-04-08 16:21:24 +02:00
extensions Use v0.0.6 of the zed_extension_api for extensions that need it (#10324) 2024-04-09 10:57:26 -04:00
script Bump PyGithub 2024-04-07 01:13:34 -04:00
tooling/xtask Windows: Enable clippy deny warnings (#9920) 2024-03-28 11:55:35 -04:00
.dockerignore Add serialized versions of themes (#6885) 2024-01-27 13:35:43 -05:00
.gitattributes Prevent GitHub from displaying comments within JSON files as errors (#7043) 2024-01-29 23:11:25 -05:00
.gitignore windows: Add file dialog using IFileOpenDialog (#8919) 2024-03-08 20:07:48 -08:00
.gitmodules
.mailmap Update .mailmap (#7871) 2024-02-15 18:35:43 -05:00
Cargo.lock Reduce memory usage to represent buffers by up to 50% (#10321) 2024-04-09 18:07:53 +02:00
Cargo.toml WIP: Refactor Linux platform implementation (#10227) 2024-04-08 16:40:35 -07:00
CODE_OF_CONDUCT.md Add CODE_OF_CONDUCT.md (#4239) 2024-01-23 22:31:39 -05:00
CONTRIBUTING.md Style crate names in CONTRIBUTING.md (#6939) 2024-01-28 11:07:37 -05:00
debug.plist
docker-compose.sql
docker-compose.yml Add LiveKit server to Docker Compose (#7907) 2024-02-16 10:49:48 -05:00
Dockerfile Revert "Revert "chore: Bump Rust version to 1.77 (#9631)"" (#9672) 2024-03-22 11:17:16 +01:00
LICENSE-AGPL chore: Add crate licenses. (#4158) 2024-01-23 16:56:22 +01:00
LICENSE-APACHE chore: Add crate licenses. (#4158) 2024-01-23 16:56:22 +01:00
LICENSE-GPL Licenses: change license fields in Cargo.toml to AGPL-3.0-or-later. (#5535) 2024-01-27 13:51:16 +01:00
livekit.yaml Add LiveKit server to Docker Compose (#7907) 2024-02-16 10:49:48 -05:00
Procfile Make collab quieter on startup (#8685) 2024-03-01 13:39:13 -07:00
README.md Update Homebrew installation instructions (#9356) 2024-03-14 14:00:31 -04:00
rust-toolchain.toml Revert "Revert "chore: Bump Rust version to 1.77 (#9631)"" (#9672) 2024-03-22 11:17:16 +01:00
typos.toml Remove basic.conf (#10120) 2024-04-03 09:38:36 -04:00

Zed

CI

Welcome to Zed, a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.

Installation

You can download Zed today for macOS (v10.15+).

Support for additional platforms is on our roadmap:

For macOS users, you can also install Zed using Homebrew:

brew install zed

Alternatively, to install the Preview release:

brew tap homebrew/cask-versions
brew install zed-preview

Developing Zed

Contributing

See CONTRIBUTING.md for ways you can contribute to Zed.

Licensing

License information for third party dependencies must be correctly provided for CI to pass.

We use cargo-about to automatically comply with open source licenses. If CI is failing, check the following:

  • Is it showing a no license specified error for a crate you've created? If so, add publish = false under [package] in your crate's Cargo.toml.
  • Is the error failed to satisfy license requirements for a dependency? If so, first determine what license the project has and whether this system is sufficient to comply with this license's requirements. If you're unsure, ask a lawyer. Once you've verified that this system is acceptable add the license's SPDX identifier to the accepted array in script/licenses/zed-licenses.toml.
  • Is cargo-about unable to find the license for a dependency? If so, add a clarification field at the end of script/licenses/zed-licenses.toml, as specified in the cargo-about book.