Rather than using the default BufWriter capacity (currently 8K),
calculate the exact size required to contain the table. This further
consolidates the write() system calls to the minimum required.
BUG=chromium:891335
TEST=Manually verify expected write() calls with strace
Change-Id: I9ade77024e60a92d30b65beb07c2385e5dacc167
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1250035
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Now that cargo fmt has landed, run it over everything at once to bring
rust source to the standard formatting.
TEST=cargo test
BUG=None
Change-Id: Ic95a48725e5a40dcbd33ba6d5aef2bd01e91865b
Reviewed-on: https://chromium-review.googlesource.com/1259287
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
In deallocate_cluster(), we call set_cluster_refcount() to unref the
cluster that is being deallocated, but we never actually added the
deallocated cluster to the unref_clusters list. Add clusters whose
refcounts reach 0 to the unref_clusters list as well.
Also add mremap() to the seccomp whitelist for the block device, since
this is being triggered by libc realloc() and other devices already
include it in the whitelist.
BUG=chromium:850998
TEST=cargo test -p qcow; test crosvm on nami and verify that qcow file
size stays bounded when creating a 1 GB file and deleting it
repeatedly
Change-Id: I1bdd96b2176dc13069417e0ac77f0768f9f26012
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1259404
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Track the clean/dirty state of the L1 table and the refcount table to
avoid writing them out and doing an extra fsyncdata() if nothing has
changed.
BUG=None
TEST=Manually verify strace output contains only the expected fsyncs
Change-Id: I20bdd250024039a5b4142605462a8977ced1efcc
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1247442
Reviewed-by: Dylan Reid <dgreid@chromium.org>
When reading and writing refcount blocks and pointer tables, the
QcowRawFile implementation was performing many individual read() and
write() system calls (one per table entry), which is quite inefficient.
Use the read_*_into functions from ReadBytesExt for reads and BufWriter
for writes to buffer the I/O into larger chunks.
BUG=None
TEST=Manually verify larger reads/writes with strace
Change-Id: I276963db0a4e91b22335c26c799ae8fb55bf6ad3
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1247441
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Being able to access the state of the qcow file makes debugging easier.
These functions will be used from a helper program in the following
commit.
Change-Id: I1db7ddaeaff1c83363513a2c55c44a1825833634
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207454
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Cache the address lookup and refcount tables in RAM. This removes an
absurd number of system calls required for accessing the qcow image as
previously each lookup required reading from at least three locations on
disk. Now the disk will only be accessed when there is a cache miss or
when the structure of the disk need to be updated; when a cluster is
added or removed.
The L1 address lookup table and the refcount table are both read at
creation time and kept in memory. For now all caches are committed to
disk only when the file is flushed. Later a timer will be added to
periodically flush the caches to disk so there is less window for data
loss.
The L2 and refcount blocks are cached as full clusters. Those clusters
are kept in the cache until a flush or they need to be evicted to make
room for a new entry. The eviction is currently very simple, writing
back the first entry in the cache in whatever order a hashmap iterator
uses. This can be improved, but the speedup is already enough that it
seems acceptable to start.
Change-Id: Ifcc55f243961d54eb1c6255b975a1529e2e987af
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207453
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The refcounting helper breaks out management of the refcounts and
caching the refcount blocks.
Change-Id: I6e75fbe0eb47277ccf7a93af026b5020089875db
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207452
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The `VecCache` struct will be used to represent the file clusters in
caches. It ties a vector to a state of dirty or clean.
Change-Id: I474eb67d2ad9f086da638ecc385ccce74737d3b9
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207451
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
The raw file struct will be used to hold enough state for basic
operations. This will allow mutating the file without taking a mutable
reference to an entire QcowFile.
Change-Id: Ia0a86537915da039274923df2f85c22d191b9969
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207450
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
When a write_zeroes call covers a whole cluster, we can deallocate the
storage for that cluster rather than writing zeroes.
This is currently implemented by removing the cluster allocation from
the mapping tables, then attempting to release the backing storage using
fallocate() with FALLOC_FL_PUNCH_HOLE.
BUG=chromium:850998
TEST=cargo test -p qcow
Change-Id: Ie4edb2e02bfaa1df9a19919b77eeb3c58c112d1c
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1187019
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Add a simple implementation of WriteZeroes for QcowFile that just
writes zeroes to allocated clusters and skips clusters that are already
unallocated (since they already read back as zeroes).
BUG=chromium:850998
TEST=cargo test -p qcow
Change-Id: I8f26c8cc4016c129850aaf08c7188dfe08d6dacb
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1187018
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
The bit that indicates a compressed cluster in the qcow2 format is
stored in the L2 table entry (cluster descriptor), but the code was
checking for it in the L1 table entry.
Spotted by inspection based on the qcow2 format documentation (QEMU
docs/interop/qcow2.txt).
This has no impact in normal crosvm use cases, since crosvm never writes
compressed clusters; the only case where it could have any effect would
be when using a QEMU-created qcow2 image with compressed clusters, which
would have returned incorrect data from a read instead of failing the
read request as intended.
BUG=chromium:874699
Change-Id: Ic247490dcac5440fab8e4d34d24ffe6fe9ab3110
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1176733
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
All qcow clusters need to have their refcounts set. Add a `new` method
to `Qcowfile` and use it instead of just headers from the library.
The new method will loop over the initial clusters and initialize their
refcounts.
Add a `create_qcow2` option to the main executable so there is a way to
test image creation that doesn't require DBUS and Concierge.
BUG=none
TEST='crosvm create_qcow2 /tmp/file.qcow2 1000000'
'qemu-img check /tmp/file.qcow2'
no errors reported.
Change-Id: I8798df5942fb23f79cc7ca86820d0783d1f2b608
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1136900
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Make sure that the refcount update is commited to disk before we start
to write data to the cluster.
Change-Id: Ibdf5dde7a75c3582f87df46af1095c7dc0007d98
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1136019
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Signal to the OS that we want these writes committed all the way to
disk. Replace an existing call to flush as that's not sufficient.
Change-Id: I9df9e55d2182e283e15eebc02a54c1ce08434f42
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1060696
Reviewed-by: Zach Reizner <zachr@chromium.org>
The u64 offsets could be truncated when running on a 32 bit machine.
Do the math in 64 bit, limit to usize::MAX, then truncate.
BUG=837453
TEST=run crosvm and read/write files
Change-Id: If44ec94cf730ca7c1e580eeddd202e54e2de1081
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1031301
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
Having qcow clippy clean will make us more likely to see important
warnings.
Change-Id: I3ce08f50eb73f2291c56dc8fe06dd378eb3dcf93
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/954404
Reviewed-by: Zach Reizner <zachr@chromium.org>
This should be the number of bits (1 << order) divided by bits per byte.
BUG=none
TEST=cargo test
Change-Id: I201dbaf21d13a82af6bcb493ba5f3b62c89286fa
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/954348
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Clippy only has 6 lints that are defaulted to deny, which this CL fixes.
The first step to running clippy and establishing our own set of default
deny lints, is to ensure we aren't running afoul of the most common
deny lints.
BUG=None
TEST=cargo clippy --all
Change-Id: I225801357d76a8a9e246e3842bc9bf550fcd7207
Reviewed-on: https://chromium-review.googlesource.com/952045
Commit-Ready: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Check that two address calculations don't overflow. Return an error if
they do. Thanks cargo fuzz.
Change-Id: I9dd301c4e08516cea8456f1ea313d62582979da0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/932646
Reviewed-by: Stephen Barber <smbarber@chromium.org>
The reference count table must be continuous in the file, not
preallocating it when creating a file causes errors when setting
reference counts for new blocks later.
BUG=809847
TEST=Replay formatting and downloading container events.
Add unit test that catches error with default files.
Change-Id: I08840958a1180a73f32f42d520517bcf88a158b0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/909915
Check the correct offset for divided writes. This was causing failures
with btrfs.
Add a unit test that checks for errors like this.
Change-Id: Ic29e6f02edf42aea19c4cbced2994c2259baf823
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/894469
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
For creating qcow files there needs to be the ability to save headers to
disk so they can be opened later.
Change-Id: Icf0134dd5ad3910c09f6e3766aca17ee003956c0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/875115
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Allow an empty QcowHeader to be created. Later, this allows QcowFiles to
be created in addition to opened.
Change-Id: Ifcc2f8ed2a92054fb7b60999d401fb573e98aa73
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/875114
Reviewed-by: Stephen Barber <smbarber@chromium.org>
QCOW2 files will be used to store the VM's data on ChromeOS. Add
support for basic reading and writing of qcow files.
CQ-DEPEND=CL:872451
BUG=none
TEST=qcow unit tests and use "qemu-image check" to verify generated qcow
files.
Change-Id: I4b9e77e58edca69f894ede19e79ebd2cd7b2623e
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/862628
Reviewed-by: Stephen Barber <smbarber@chromium.org>
