The new constructors are shorter and omit the bare `None` in the `anon`
call sites which gave no clues to the reader what the effect of that
`None` was. This should improve readability.
TEST=./build_test
BUG=None
Change-Id: I2e34e7df9a4ccc5da50edf4e963a6a42e3d84b22
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1797188
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Use the standard byte order conversion functions rather than the
byteorder crate.
BUG=None
TEST=./build_test
TEST=cargo build -p qcow_utils
TEST=cargo test -p qcow
Change-Id: I9ff7368cc54c539db1996f81d4220cabf7e6e301
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1761151
Reviewed-by: Zach Reizner <zachr@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
If the header puts the refcount table outside the file size or if it
specifies a table much larger than needed, fail to open the file.
These might not be hard qcow errors, but they are situations that crosvm
will never encounter.
BUG=986061
TEST=fuzzer with new test cases completes in less than 5 seconds.
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Change-Id: If048c96f6255ca81740e20f3f4eb7669467dbb7b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1716365
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Zeroing a cluster will be done from more than one place in qcow.rs soon,
add a helper to reduce duplication.
Change-Id: Idb40539f8e4ed2338fc84c0d53b37c913f2d90fe
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1697122
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
There are many corner cases when handling sizes that approach u64::max.
Limit the files to 16TB.
BUG=979458
TEST=Added unittest to check large disks fail
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Change-Id: I93a87c17267ae69102f8d46ced9dbea8c686d093
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1679892
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
The extra % operation will be slower, but none of these divisions are in
hot paths. They are only used during setup. Many of these operations
take untrusted input from the disk file, so need to be hardened.
BUG=979458
TEST=unit tests still pass
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Change-Id: I0e93c73b345faf643da53ea41bde3349d756bdc7
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1679891
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Before this change, a corrupt or malicious qcow file could cause crosvm
to allocate absurd amounts of memory. The fuzzer found this case,
limit the L1 table size so it can't cause issues.
BUG=chromium:974123
TEST=run fuzzer locally, add unit test
Change-Id: Ieb6db6c87f71df726b3cc9a98404581fe32fb1ce
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1660890
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Start with a valid header so the invalid cluster bits field is tested in
isolation. Before this change the test would pass even if the cluster
bits check was removed from the code because the header was invalid for
other reasons.
BUG=none
TEST=this is a test
Change-Id: I5c09417ae3f974522652a50cb0fdc5dc0e10dd44
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1660889
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
qcow currently makes assumptions about being able to fit the L1 and
refcount tables in memory. This isn't necessarily true for very large
files. The new limits to the size of in-memory buffers still allow 1TB
files with default cluster sizes. Because there aren't any 1 TB
chromebooks available yet, limit disks to that size.
This works around issues found by clusterfuzz related to large files
built with small clusters.
BUG=972165,972175,972160,972172
TEST=fuzzer locally + new unit tests
Change-Id: I15d5d8e3e61213780ff6aea5759b155c63d49ea3
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1651460
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
u32's get multiplied together and can overflow. A usize was being
returned, make everything a u64 to make sure it fits.
Change-Id: I87071d294f4e62247c9ae72244db059a7b528b62
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1651459
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Add a lower limit because cases such as eight byte clusters aren't
practical and aren't worth handling, tracking a cluster costs 16 bytes.
Also put an upper limit on the cluster size, choose 21 bits to match
qemu.
Change-Id: Ifcab081d0e630b5d26b0eafa552bd7c695821686
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1651458
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
This manifested itself in a couple places that were turning shared
memory buffers into slices for the purposes of passing these slices to
`Read` and `Write` trait methods.
However, this required the removal of the methods that took `Read` and
`Write` instances. This was a convenient interface but impossible to
implement safely because making slices from raw pointers without
enforcing safety guarantees causes undefined behaviour in Rust. It turns
out lots of code in crosvm was using these interfaces indirectly, which
explains why this CL touches so much.
TEST=crosvm run
BUG=chromium:938767
Change-Id: I4ff40c98da6ed08a4a42f4c31f0717f81b1c5863
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1636685
Reviewed-by: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Zach Reizner <zachr@chromium.org>
In Rust 2018 edition, `extern crate` is no longer required for importing
from other crates. Instead of writing:
extern crate dep;
use dep::Thing;
we write:
use dep::Thing;
In this approach, macros are imported individually from the declaring
crate rather than through #[macro_use]. Before:
#[macro_use]
extern crate sys_util;
After:
use sys_util::{debug, error};
The only place that `extern crate` continues to be required is in
importing the compiler's proc_macro API into a procedural macro crate.
This will hopefully be fixed in a future Rust release.
extern crate proc_macro;
TEST=cargo check
TEST=cargo check --all-features
TEST=cargo check --target aarch64-unknown-linux-gnu
TEST=local kokoro
Change-Id: I0b43768c0d81f2a250b1959fb97ba35cbac56293
Reviewed-on: https://chromium-review.googlesource.com/1565302
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
Macros were previously imported through `#[macro_use] extern crate`,
which is basically a glob import of all macros from the crate. As of
2018 edition of Rust, `extern crate` is no longer required and macros
are imported individually like any other item from a dependency. This CL
fills in all the appropriate macro imports that will allow us to remove
our use of `extern crate` in a subsequent CL.
TEST=cargo check --all-features --tests
TEST=kokoro
Change-Id: If2ec08b06b743abf5f62677c6a9927c3d5d90a54
Reviewed-on: https://chromium-review.googlesource.com/1565546
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
To avoid wasting time re-sorting these things (CL:1492612).
https://docs.rs/remain
Disclaimer: I wrote the macro.
This CL adds #[sorted] attributes to those Error enums that seemed to
have made some effort to be in sorted order.
TEST=cargo check
TEST=cargo check --all-features
TEST=cargo check --target aarch64-unknown-linux-gnu
TEST=emerge-nami crosvm
TEST=local kokoro
CQ-DEPEND=CL:1524247
Change-Id: I89685ced05e2f149fa189ca509bc14c70aebb531
Reviewed-on: https://chromium-review.googlesource.com/1515998
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
Separated out of CL:1513058 to make it possible to land parts
individually while the affected crate has no other significant CLs
pending. This avoids repeatedly introducing non-textual conflicts with
new code that adds `use` statements.
TEST=cargo check
TEST=cargo check --all-features
TEST=cargo check --target aarch64-unknown-linux-gnu
Change-Id: Ief3fb967df340a99b8263ac185207e30e096105a
Reviewed-on: https://chromium-review.googlesource.com/1519704
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
This is an easy step toward adopting 2018 edition eventually, and will
make any future CL that sets `edition = "2018"` this much smaller.
The module system changes in Rust 2018 are described here:
https://doc.rust-lang.org/edition-guide/rust-2018/module-system/path-clarity.html
Generated by running:
cargo fix --edition --all
in each workspace, followed by bin/fmt.
TEST=cargo check
TEST=cargo check --all-features
TEST=cargo check --target aarch64-unknown-linux-gnu
Change-Id: I000ab5e69d69aa222c272fae899464bbaf65f6d8
Reviewed-on: https://chromium-review.googlesource.com/1513054
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
I have been running into Debug-printed error messages too often and
needing to look up in the source code each level of nested errors to
find out from the comment on the error variant what the short name of
the variant means in human terms. Worse, many errors (like the one shown
below) already had error strings written but were being printed from the
calling code in the less helpful Debug representation anyway.
Before:
[ERROR:src/main.rs:705] The architecture failed to build the vm: NoVarEmpty
After:
[ERROR:src/main.rs:705] The architecture failed to build the vm: /var/empty doesn't exist, can't jail devices.
TEST=cargo check --all-features
TEST=FEATURES=test emerge-amd64-generic crosvm
Change-Id: I77122c7d6861b2d610de2fff718896918ab21e10
Reviewed-on: https://chromium-review.googlesource.com/1469225
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
We won't support resizing a QcowFile for now, since it would require
resizing refcount and L1 tables.
BUG=chromium:858815
TEST=build_test
Change-Id: Ia108e07be8b06b6fbe81838831a8ecbffdfb341c
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1394149
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
We updated the production toolchain from 1.30 to 1.31 in CL:1366446.
This CL does the same upgrade for the local developer toolchain and
Kokoro.
The relevant changes are in rust-toolchain and kokoro/Dockerfile.
The rest are from rustfmt.
TEST=cargo fmt --all -- --check
TEST=as described in kokoro/README.md
Change-Id: I3b4913f3e237baa36c664b4953be360c09efffd4
Reviewed-on: https://chromium-review.googlesource.com/1374376
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
During runtime, we track unreferenced clusters (via unref_clusters and
avail_clusters) and reuse them before extending the disk image.
However, across boots, we did not previously recover the list of
unreferenced clusters, so the disk file could grow beyond the range that
the reference table count represent. This patch adds a boot-time scan
for all unreferenced clusters so that they get reused.
BUG=chromium:899273
TEST=Boot with qcow2 image, fill the disk with dd, delete the dd'd file,
refill with dd, and so on, repeatedly. Ensure that the disk image does
not grow beyond the expected max size and that no clusters beyond the
size of the refcount table are used.
Change-Id: Idd21b08bb4c55b8244e7ecaccafc4ccc46b7b17a
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1327822
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Hopefully the changes are self-explanatory and uncontroversial. This
eliminates much of the noise from `cargo clippy` and, for my purposes,
gives me a reasonable way to use it as a tool when writing and reviewing
code.
Here is the Clippy invocation I was using:
cargo +nightly clippy -- -W clippy::correctness -A renamed_and_removed_lints -Aclippy::{blacklisted_name,borrowed_box,cast_lossless,cast_ptr_alignment,enum_variant_names,identity_op,if_same_then_else,mut_from_ref,needless_pass_by_value,new_without_default,new_without_default_derive,or_fun_call,ptr_arg,should_implement_trait,single_match,too_many_arguments,trivially_copy_pass_by_ref,unreadable_literal,unsafe_vector_initialization,useless_transmute}
TEST=cargo check --features wl-dmabuf,gpu,usb-emulation
TEST=boot linux
Change-Id: I55eb1b4a72beb2f762480e3333a921909314a0a2
Reviewed-on: https://chromium-review.googlesource.com/1356911
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This adds the ability to regenerate the reference counts by walking all
of the L1/L2 tables and headers to find all reachable clusters. This is
necessary for the next patch, which will use the reference count tables
to find unused clusters to reuse.
BUG=chromium:899273
TEST=cargo test -p cqow
Change-Id: I93dd00d381d8d33010fddfc10aa18ca32586e1f4
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1327821
Reviewed-by: Dylan Reid <dgreid@chromium.org>
The refcount table needs to include not only the data clusters and
reftable clusters but also the L1 and L2 tables and main qcow2 header.
Also add sanity checking to prevent allocating a cluster that cannot be
indexed with the current reference count table size.
BUG=chromium:899273
TEST=cargo test -p qcow
Change-Id: I9da4515db3dccbabdeee4f60dc392b5b42d62cb2
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1308833
Reviewed-by: Dylan Reid <dgreid@chromium.org>
File exposes sync_all() and sync_data() functions, which map to fsync()
and fdatasync(), but these functions are not in a trait (they are just
implemented directly on File), so they can't be implemented and used in
a generic way for QcowFile.
Add a new trait, FileSync, that exposes a fsync() function that may be
used in the virtio block model. Previously, we were translating a block
flush request into a call to File's flush() function, but this just
flushes internal Rust library buffers to the file descriptor; it didn't
actually result in a fsync() call. Using the new trait, we can cause an
actual fsync() to occur for raw files, as intended. QcowFile was
already safe, since its flush() function actually calls sync_all() under
the hood.
BUG=None
TEST=sync with raw disk and verify fsync() in strace output
Change-Id: I9bee2c0d2df3747aac1e7d9ec7d9b46a7862dc48
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1297839
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Make --disk and --rwdisk automatically distinguish between qcow2 and raw
disk images. --qcow and --rwqcow are kept as aliases for compatibility.
BUG=chromium:893380
TEST=Boot crosvm with both raw and qcow2 disk images.
Change-Id: I5b572626ca5ab894c78454f59355f27d552cbf7d
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1275185
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Our branch of the 3.18 kernel has FALLOC_FL_PUNCH_HOLE disabled for the
ext4 filesystem, which means that systems running that kernel always
take the fallback path of writing buffers full of zeroes. This is not
necessary for the Discard command, since it is just a hint and is not
required to actually zero the blocks.
Split the WriteZeroes trait up into a new PunchHole trait, which
corresponds to fallocate() with FALLOC_FL_PUNCH_HOLE, and use the new
trait to implement the virtio block Discard command.
BUG=chromium:896314
TEST=`mkfs.btrfs /dev/vdb` and verify the desired fallocate() is used
and no write() calls are issued when inducing a failure
Change-Id: I67fd9a132758d8d766531ccca8358c7fe67b0460
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1286224
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This will be used in vm_concierge's ExportDiskImage function in order to
allow a minimal qcow2 image to be written on the fly (containing only
the required clusters in a tightly-packed image file). It also allows
future flexibility to change the underlying disk image file format while
still exporting qcow2 images (e.g. via `vmc export`).
For testing, add a qcow_img `convert` command, which can convert
between raw and qcow2 as both source and destination.
BUG=None
TEST=Use qcow_img to convert a raw image to qcow2 and back and verify
its contents are the same as the original.
Change-Id: I74167aca9a9c857d892e24adf5ee17afc0f6e6b5
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1272060
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Allow seeking to the next hole or data region in File and QcowFile.
BUG=None
TEST=None
Change-Id: I16e77e4791aa85b4cc96f38327026cd93f02b7e1
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1274147
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Rather than using the default BufWriter capacity (currently 8K),
calculate the exact size required to contain the table. This further
consolidates the write() system calls to the minimum required.
BUG=chromium:891335
TEST=Manually verify expected write() calls with strace
Change-Id: I9ade77024e60a92d30b65beb07c2385e5dacc167
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1250035
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Now that cargo fmt has landed, run it over everything at once to bring
rust source to the standard formatting.
TEST=cargo test
BUG=None
Change-Id: Ic95a48725e5a40dcbd33ba6d5aef2bd01e91865b
Reviewed-on: https://chromium-review.googlesource.com/1259287
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
In deallocate_cluster(), we call set_cluster_refcount() to unref the
cluster that is being deallocated, but we never actually added the
deallocated cluster to the unref_clusters list. Add clusters whose
refcounts reach 0 to the unref_clusters list as well.
Also add mremap() to the seccomp whitelist for the block device, since
this is being triggered by libc realloc() and other devices already
include it in the whitelist.
BUG=chromium:850998
TEST=cargo test -p qcow; test crosvm on nami and verify that qcow file
size stays bounded when creating a 1 GB file and deleting it
repeatedly
Change-Id: I1bdd96b2176dc13069417e0ac77f0768f9f26012
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1259404
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Track the clean/dirty state of the L1 table and the refcount table to
avoid writing them out and doing an extra fsyncdata() if nothing has
changed.
BUG=None
TEST=Manually verify strace output contains only the expected fsyncs
Change-Id: I20bdd250024039a5b4142605462a8977ced1efcc
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1247442
Reviewed-by: Dylan Reid <dgreid@chromium.org>
When reading and writing refcount blocks and pointer tables, the
QcowRawFile implementation was performing many individual read() and
write() system calls (one per table entry), which is quite inefficient.
Use the read_*_into functions from ReadBytesExt for reads and BufWriter
for writes to buffer the I/O into larger chunks.
BUG=None
TEST=Manually verify larger reads/writes with strace
Change-Id: I276963db0a4e91b22335c26c799ae8fb55bf6ad3
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1247441
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Being able to access the state of the qcow file makes debugging easier.
These functions will be used from a helper program in the following
commit.
Change-Id: I1db7ddaeaff1c83363513a2c55c44a1825833634
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207454
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Cache the address lookup and refcount tables in RAM. This removes an
absurd number of system calls required for accessing the qcow image as
previously each lookup required reading from at least three locations on
disk. Now the disk will only be accessed when there is a cache miss or
when the structure of the disk need to be updated; when a cluster is
added or removed.
The L1 address lookup table and the refcount table are both read at
creation time and kept in memory. For now all caches are committed to
disk only when the file is flushed. Later a timer will be added to
periodically flush the caches to disk so there is less window for data
loss.
The L2 and refcount blocks are cached as full clusters. Those clusters
are kept in the cache until a flush or they need to be evicted to make
room for a new entry. The eviction is currently very simple, writing
back the first entry in the cache in whatever order a hashmap iterator
uses. This can be improved, but the speedup is already enough that it
seems acceptable to start.
Change-Id: Ifcc55f243961d54eb1c6255b975a1529e2e987af
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207453
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The refcounting helper breaks out management of the refcounts and
caching the refcount blocks.
Change-Id: I6e75fbe0eb47277ccf7a93af026b5020089875db
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207452
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The `VecCache` struct will be used to represent the file clusters in
caches. It ties a vector to a state of dirty or clean.
Change-Id: I474eb67d2ad9f086da638ecc385ccce74737d3b9
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207451
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The raw file struct will be used to hold enough state for basic
operations. This will allow mutating the file without taking a mutable
reference to an entire QcowFile.
Change-Id: Ia0a86537915da039274923df2f85c22d191b9969
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207450
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
When a write_zeroes call covers a whole cluster, we can deallocate the
storage for that cluster rather than writing zeroes.
This is currently implemented by removing the cluster allocation from
the mapping tables, then attempting to release the backing storage using
fallocate() with FALLOC_FL_PUNCH_HOLE.
BUG=chromium:850998
TEST=cargo test -p qcow
Change-Id: Ie4edb2e02bfaa1df9a19919b77eeb3c58c112d1c
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1187019
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Add a simple implementation of WriteZeroes for QcowFile that just
writes zeroes to allocated clusters and skips clusters that are already
unallocated (since they already read back as zeroes).
BUG=chromium:850998
TEST=cargo test -p qcow
Change-Id: I8f26c8cc4016c129850aaf08c7188dfe08d6dacb
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1187018
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
The bit that indicates a compressed cluster in the qcow2 format is
stored in the L2 table entry (cluster descriptor), but the code was
checking for it in the L1 table entry.
Spotted by inspection based on the qcow2 format documentation (QEMU
docs/interop/qcow2.txt).
This has no impact in normal crosvm use cases, since crosvm never writes
compressed clusters; the only case where it could have any effect would
be when using a QEMU-created qcow2 image with compressed clusters, which
would have returned incorrect data from a read instead of failing the
read request as intended.
BUG=chromium:874699
Change-Id: Ic247490dcac5440fab8e4d34d24ffe6fe9ab3110
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1176733
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
All qcow clusters need to have their refcounts set. Add a `new` method
to `Qcowfile` and use it instead of just headers from the library.
The new method will loop over the initial clusters and initialize their
refcounts.
Add a `create_qcow2` option to the main executable so there is a way to
test image creation that doesn't require DBUS and Concierge.
BUG=none
TEST='crosvm create_qcow2 /tmp/file.qcow2 1000000'
'qemu-img check /tmp/file.qcow2'
no errors reported.
Change-Id: I8798df5942fb23f79cc7ca86820d0783d1f2b608
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1136900
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Make sure that the refcount update is commited to disk before we start
to write data to the cluster.
Change-Id: Ibdf5dde7a75c3582f87df46af1095c7dc0007d98
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1136019
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Signal to the OS that we want these writes committed all the way to
disk. Replace an existing call to flush as that's not sufficient.
Change-Id: I9df9e55d2182e283e15eebc02a54c1ce08434f42
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1060696
Reviewed-by: Zach Reizner <zachr@chromium.org>
The u64 offsets could be truncated when running on a 32 bit machine.
Do the math in 64 bit, limit to usize::MAX, then truncate.
BUG=837453
TEST=run crosvm and read/write files
Change-Id: If44ec94cf730ca7c1e580eeddd202e54e2de1081
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1031301
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
Having qcow clippy clean will make us more likely to see important
warnings.
Change-Id: I3ce08f50eb73f2291c56dc8fe06dd378eb3dcf93
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/954404
Reviewed-by: Zach Reizner <zachr@chromium.org>
