We updated the production toolchain from 1.30 to 1.31 in CL:1366446.
This CL does the same upgrade for the local developer toolchain and
Kokoro.
The relevant changes are in rust-toolchain and kokoro/Dockerfile.
The rest are from rustfmt.
TEST=cargo fmt --all -- --check
TEST=as described in kokoro/README.md
Change-Id: I3b4913f3e237baa36c664b4953be360c09efffd4
Reviewed-on: https://chromium-review.googlesource.com/1374376
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
During runtime, we track unreferenced clusters (via unref_clusters and
avail_clusters) and reuse them before extending the disk image.
However, across boots, we did not previously recover the list of
unreferenced clusters, so the disk file could grow beyond the range that
the reference table count represent. This patch adds a boot-time scan
for all unreferenced clusters so that they get reused.
BUG=chromium:899273
TEST=Boot with qcow2 image, fill the disk with dd, delete the dd'd file,
refill with dd, and so on, repeatedly. Ensure that the disk image does
not grow beyond the expected max size and that no clusters beyond the
size of the refcount table are used.
Change-Id: Idd21b08bb4c55b8244e7ecaccafc4ccc46b7b17a
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1327822
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Hopefully the changes are self-explanatory and uncontroversial. This
eliminates much of the noise from `cargo clippy` and, for my purposes,
gives me a reasonable way to use it as a tool when writing and reviewing
code.
Here is the Clippy invocation I was using:
cargo +nightly clippy -- -W clippy::correctness -A renamed_and_removed_lints -Aclippy::{blacklisted_name,borrowed_box,cast_lossless,cast_ptr_alignment,enum_variant_names,identity_op,if_same_then_else,mut_from_ref,needless_pass_by_value,new_without_default,new_without_default_derive,or_fun_call,ptr_arg,should_implement_trait,single_match,too_many_arguments,trivially_copy_pass_by_ref,unreadable_literal,unsafe_vector_initialization,useless_transmute}
TEST=cargo check --features wl-dmabuf,gpu,usb-emulation
TEST=boot linux
Change-Id: I55eb1b4a72beb2f762480e3333a921909314a0a2
Reviewed-on: https://chromium-review.googlesource.com/1356911
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This adds the ability to regenerate the reference counts by walking all
of the L1/L2 tables and headers to find all reachable clusters. This is
necessary for the next patch, which will use the reference count tables
to find unused clusters to reuse.
BUG=chromium:899273
TEST=cargo test -p cqow
Change-Id: I93dd00d381d8d33010fddfc10aa18ca32586e1f4
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1327821
Reviewed-by: Dylan Reid <dgreid@chromium.org>
The refcount table needs to include not only the data clusters and
reftable clusters but also the L1 and L2 tables and main qcow2 header.
Also add sanity checking to prevent allocating a cluster that cannot be
indexed with the current reference count table size.
BUG=chromium:899273
TEST=cargo test -p qcow
Change-Id: I9da4515db3dccbabdeee4f60dc392b5b42d62cb2
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1308833
Reviewed-by: Dylan Reid <dgreid@chromium.org>
File exposes sync_all() and sync_data() functions, which map to fsync()
and fdatasync(), but these functions are not in a trait (they are just
implemented directly on File), so they can't be implemented and used in
a generic way for QcowFile.
Add a new trait, FileSync, that exposes a fsync() function that may be
used in the virtio block model. Previously, we were translating a block
flush request into a call to File's flush() function, but this just
flushes internal Rust library buffers to the file descriptor; it didn't
actually result in a fsync() call. Using the new trait, we can cause an
actual fsync() to occur for raw files, as intended. QcowFile was
already safe, since its flush() function actually calls sync_all() under
the hood.
BUG=None
TEST=sync with raw disk and verify fsync() in strace output
Change-Id: I9bee2c0d2df3747aac1e7d9ec7d9b46a7862dc48
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1297839
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Make --disk and --rwdisk automatically distinguish between qcow2 and raw
disk images. --qcow and --rwqcow are kept as aliases for compatibility.
BUG=chromium:893380
TEST=Boot crosvm with both raw and qcow2 disk images.
Change-Id: I5b572626ca5ab894c78454f59355f27d552cbf7d
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1275185
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Our branch of the 3.18 kernel has FALLOC_FL_PUNCH_HOLE disabled for the
ext4 filesystem, which means that systems running that kernel always
take the fallback path of writing buffers full of zeroes. This is not
necessary for the Discard command, since it is just a hint and is not
required to actually zero the blocks.
Split the WriteZeroes trait up into a new PunchHole trait, which
corresponds to fallocate() with FALLOC_FL_PUNCH_HOLE, and use the new
trait to implement the virtio block Discard command.
BUG=chromium:896314
TEST=`mkfs.btrfs /dev/vdb` and verify the desired fallocate() is used
and no write() calls are issued when inducing a failure
Change-Id: I67fd9a132758d8d766531ccca8358c7fe67b0460
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1286224
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This will be used in vm_concierge's ExportDiskImage function in order to
allow a minimal qcow2 image to be written on the fly (containing only
the required clusters in a tightly-packed image file). It also allows
future flexibility to change the underlying disk image file format while
still exporting qcow2 images (e.g. via `vmc export`).
For testing, add a qcow_img `convert` command, which can convert
between raw and qcow2 as both source and destination.
BUG=None
TEST=Use qcow_img to convert a raw image to qcow2 and back and verify
its contents are the same as the original.
Change-Id: I74167aca9a9c857d892e24adf5ee17afc0f6e6b5
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1272060
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Allow seeking to the next hole or data region in File and QcowFile.
BUG=None
TEST=None
Change-Id: I16e77e4791aa85b4cc96f38327026cd93f02b7e1
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1274147
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Rather than using the default BufWriter capacity (currently 8K),
calculate the exact size required to contain the table. This further
consolidates the write() system calls to the minimum required.
BUG=chromium:891335
TEST=Manually verify expected write() calls with strace
Change-Id: I9ade77024e60a92d30b65beb07c2385e5dacc167
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1250035
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Now that cargo fmt has landed, run it over everything at once to bring
rust source to the standard formatting.
TEST=cargo test
BUG=None
Change-Id: Ic95a48725e5a40dcbd33ba6d5aef2bd01e91865b
Reviewed-on: https://chromium-review.googlesource.com/1259287
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
In deallocate_cluster(), we call set_cluster_refcount() to unref the
cluster that is being deallocated, but we never actually added the
deallocated cluster to the unref_clusters list. Add clusters whose
refcounts reach 0 to the unref_clusters list as well.
Also add mremap() to the seccomp whitelist for the block device, since
this is being triggered by libc realloc() and other devices already
include it in the whitelist.
BUG=chromium:850998
TEST=cargo test -p qcow; test crosvm on nami and verify that qcow file
size stays bounded when creating a 1 GB file and deleting it
repeatedly
Change-Id: I1bdd96b2176dc13069417e0ac77f0768f9f26012
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1259404
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Track the clean/dirty state of the L1 table and the refcount table to
avoid writing them out and doing an extra fsyncdata() if nothing has
changed.
BUG=None
TEST=Manually verify strace output contains only the expected fsyncs
Change-Id: I20bdd250024039a5b4142605462a8977ced1efcc
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1247442
Reviewed-by: Dylan Reid <dgreid@chromium.org>
When reading and writing refcount blocks and pointer tables, the
QcowRawFile implementation was performing many individual read() and
write() system calls (one per table entry), which is quite inefficient.
Use the read_*_into functions from ReadBytesExt for reads and BufWriter
for writes to buffer the I/O into larger chunks.
BUG=None
TEST=Manually verify larger reads/writes with strace
Change-Id: I276963db0a4e91b22335c26c799ae8fb55bf6ad3
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1247441
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Being able to access the state of the qcow file makes debugging easier.
These functions will be used from a helper program in the following
commit.
Change-Id: I1db7ddaeaff1c83363513a2c55c44a1825833634
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207454
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Cache the address lookup and refcount tables in RAM. This removes an
absurd number of system calls required for accessing the qcow image as
previously each lookup required reading from at least three locations on
disk. Now the disk will only be accessed when there is a cache miss or
when the structure of the disk need to be updated; when a cluster is
added or removed.
The L1 address lookup table and the refcount table are both read at
creation time and kept in memory. For now all caches are committed to
disk only when the file is flushed. Later a timer will be added to
periodically flush the caches to disk so there is less window for data
loss.
The L2 and refcount blocks are cached as full clusters. Those clusters
are kept in the cache until a flush or they need to be evicted to make
room for a new entry. The eviction is currently very simple, writing
back the first entry in the cache in whatever order a hashmap iterator
uses. This can be improved, but the speedup is already enough that it
seems acceptable to start.
Change-Id: Ifcc55f243961d54eb1c6255b975a1529e2e987af
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207453
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The refcounting helper breaks out management of the refcounts and
caching the refcount blocks.
Change-Id: I6e75fbe0eb47277ccf7a93af026b5020089875db
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207452
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The `VecCache` struct will be used to represent the file clusters in
caches. It ties a vector to a state of dirty or clean.
Change-Id: I474eb67d2ad9f086da638ecc385ccce74737d3b9
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207451
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The raw file struct will be used to hold enough state for basic
operations. This will allow mutating the file without taking a mutable
reference to an entire QcowFile.
Change-Id: Ia0a86537915da039274923df2f85c22d191b9969
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207450
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
When a write_zeroes call covers a whole cluster, we can deallocate the
storage for that cluster rather than writing zeroes.
This is currently implemented by removing the cluster allocation from
the mapping tables, then attempting to release the backing storage using
fallocate() with FALLOC_FL_PUNCH_HOLE.
BUG=chromium:850998
TEST=cargo test -p qcow
Change-Id: Ie4edb2e02bfaa1df9a19919b77eeb3c58c112d1c
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1187019
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Add a simple implementation of WriteZeroes for QcowFile that just
writes zeroes to allocated clusters and skips clusters that are already
unallocated (since they already read back as zeroes).
BUG=chromium:850998
TEST=cargo test -p qcow
Change-Id: I8f26c8cc4016c129850aaf08c7188dfe08d6dacb
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1187018
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
The bit that indicates a compressed cluster in the qcow2 format is
stored in the L2 table entry (cluster descriptor), but the code was
checking for it in the L1 table entry.
Spotted by inspection based on the qcow2 format documentation (QEMU
docs/interop/qcow2.txt).
This has no impact in normal crosvm use cases, since crosvm never writes
compressed clusters; the only case where it could have any effect would
be when using a QEMU-created qcow2 image with compressed clusters, which
would have returned incorrect data from a read instead of failing the
read request as intended.
BUG=chromium:874699
Change-Id: Ic247490dcac5440fab8e4d34d24ffe6fe9ab3110
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1176733
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
All qcow clusters need to have their refcounts set. Add a `new` method
to `Qcowfile` and use it instead of just headers from the library.
The new method will loop over the initial clusters and initialize their
refcounts.
Add a `create_qcow2` option to the main executable so there is a way to
test image creation that doesn't require DBUS and Concierge.
BUG=none
TEST='crosvm create_qcow2 /tmp/file.qcow2 1000000'
'qemu-img check /tmp/file.qcow2'
no errors reported.
Change-Id: I8798df5942fb23f79cc7ca86820d0783d1f2b608
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1136900
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Make sure that the refcount update is commited to disk before we start
to write data to the cluster.
Change-Id: Ibdf5dde7a75c3582f87df46af1095c7dc0007d98
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1136019
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Signal to the OS that we want these writes committed all the way to
disk. Replace an existing call to flush as that's not sufficient.
Change-Id: I9df9e55d2182e283e15eebc02a54c1ce08434f42
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1060696
Reviewed-by: Zach Reizner <zachr@chromium.org>
The u64 offsets could be truncated when running on a 32 bit machine.
Do the math in 64 bit, limit to usize::MAX, then truncate.
BUG=837453
TEST=run crosvm and read/write files
Change-Id: If44ec94cf730ca7c1e580eeddd202e54e2de1081
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1031301
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
Having qcow clippy clean will make us more likely to see important
warnings.
Change-Id: I3ce08f50eb73f2291c56dc8fe06dd378eb3dcf93
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/954404
Reviewed-by: Zach Reizner <zachr@chromium.org>
This should be the number of bits (1 << order) divided by bits per byte.
BUG=none
TEST=cargo test
Change-Id: I201dbaf21d13a82af6bcb493ba5f3b62c89286fa
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/954348
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Clippy only has 6 lints that are defaulted to deny, which this CL fixes.
The first step to running clippy and establishing our own set of default
deny lints, is to ensure we aren't running afoul of the most common
deny lints.
BUG=None
TEST=cargo clippy --all
Change-Id: I225801357d76a8a9e246e3842bc9bf550fcd7207
Reviewed-on: https://chromium-review.googlesource.com/952045
Commit-Ready: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Check that two address calculations don't overflow. Return an error if
they do. Thanks cargo fuzz.
Change-Id: I9dd301c4e08516cea8456f1ea313d62582979da0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/932646
Reviewed-by: Stephen Barber <smbarber@chromium.org>
The reference count table must be continuous in the file, not
preallocating it when creating a file causes errors when setting
reference counts for new blocks later.
BUG=809847
TEST=Replay formatting and downloading container events.
Add unit test that catches error with default files.
Change-Id: I08840958a1180a73f32f42d520517bcf88a158b0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/909915
Check the correct offset for divided writes. This was causing failures
with btrfs.
Add a unit test that checks for errors like this.
Change-Id: Ic29e6f02edf42aea19c4cbced2994c2259baf823
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/894469
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
For creating qcow files there needs to be the ability to save headers to
disk so they can be opened later.
Change-Id: Icf0134dd5ad3910c09f6e3766aca17ee003956c0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/875115
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Allow an empty QcowHeader to be created. Later, this allows QcowFiles to
be created in addition to opened.
Change-Id: Ifcc2f8ed2a92054fb7b60999d401fb573e98aa73
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/875114
Reviewed-by: Stephen Barber <smbarber@chromium.org>
QCOW2 files will be used to store the VM's data on ChromeOS. Add
support for basic reading and writing of qcow files.
CQ-DEPEND=CL:872451
BUG=none
TEST=qcow unit tests and use "qemu-image check" to verify generated qcow
files.
Change-Id: I4b9e77e58edca69f894ede19e79ebd2cd7b2623e
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/862628
Reviewed-by: Stephen Barber <smbarber@chromium.org>
