From b9e16070434545713ddb0b5a324afc55036714ad Mon Sep 17 00:00:00 2001 From: Pekka Enberg Date: Mon, 29 Jul 2019 19:37:39 +0300 Subject: [PATCH 1/6] Add a Docker sandbox for building Wasmer Building Wasmer is bit of a pain, because it requires Rust nightly, and bunch of OS specific packages. To make building easier, this adds a "build" script at the top-level directory, which can be used to build Wasmer within a Docker sandbox that has all the necessary dependencies installed. The build environment is based on latest Ubuntu 19.04. You first need to build a Docker image of the sandbox: docker build --file Dockerfile.build --tag wasmer-build . Then, to build Wasmer, run: ./build make To test Wasmer, run: ./build make test and so on. You can also drop into a shell within the Docker with: ./build The "build" script bind mounts current directory as "/wasmer" in the Docker container, which allows inspecting the build contents like you had built them on your local machine. For future improvements, we should consider: - Consolidation with existing Dockerfile (that is used for Circle CI) - Publishing the build sandbox image on Docker Hub so that people don't have to build it themselves - Moving dependency installation to separate script, which can be reused outside of the Docker sandbox. The work has been inspired by "devtool" in the Firecracker project: https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool and "dbuild" in the Scylla project: https://github.com/scylladb/scylla/blob/master/tools/toolchain/dbuild --- Dockerfile.build | 13 ++++++++ build | 79 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 Dockerfile.build create mode 100755 build diff --git a/Dockerfile.build b/Dockerfile.build new file mode 100644 index 000000000..f4440ae8b --- /dev/null +++ b/Dockerfile.build @@ -0,0 +1,13 @@ +FROM ubuntu:19.04 + +ARG RUST_TOOLCHAIN="nightly" + +ENV CARGO_HOME=/usr/local/rust +ENV RUSTUP_HOME=/usr/local/rust +ENV PATH="$PATH:$CARGO_HOME/bin" + +RUN apt-get update \ + && apt-get -y install sudo strace curl cmake pkg-config python libssl-dev llvm-dev libz-dev gnuplot-nox \ + && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers \ + && echo '%wheel ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers \ + && curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain "$RUST_TOOLCHAIN" diff --git a/build b/build new file mode 100755 index 000000000..eee434dbb --- /dev/null +++ b/build @@ -0,0 +1,79 @@ +#!/bin/bash + +# Wasmer build tool +# +# This is a script to build Wasmer in a Docker sandbox. +# +# To use the script, first make sure Docker is installed. Then build the +# sandbox image with: +# +# docker build --file Dockerfile.build --tag wasmer-build . +# +# After the sandbox image is built successfully, you can run commands in it +# with this script. +# +# For example, to build Wasmer, run: +# +# ./build make +# +# To test Wasmer, run: +# +# ./build make test +# +# and so on. + +docker_hostname="wasmer-build" + +docker_img="wasmer-build" + +docker_workdir="/wasmer" + +docker_args=( + # + # General config. + # + --hostname=${docker_hostname} + --interactive + --network=host + --rm + --tty + + # + # User and group config. + # + # Use the same user and group permissions as host to make integration + # between host and container simple. + # + --user "$(id --user):$(id --group)" + --volume "/etc/group:/etc/group:ro" + --volume "/etc/passwd:/etc/passwd:ro" + --volume "/etc/shadow:/etc/shadow:ro" + + # + # Time zone config. + # + # Use the same time zone as the host. + # + --volume "/etc/localtime:/etc/localtime:ro" + + # + # Linux capabilities. + # + # Add SYS_PTRACE capability to the container so that people can run + # `strace'. + # + --cap-add SYS_PTRACE + + # + # Source directory. + # + --workdir=${docker_workdir} + --volume "$(pwd):${docker_workdir}:z" + + # + # Environment variables. + # + --env "CARGO_HOME=${docker_workdir}/.cargo" +) + +docker run ${docker_args[@]} ${docker_img} $* From e076568168851107f4e5c7efe0db38b4aed6ed12 Mon Sep 17 00:00:00 2001 From: Nick Lewycky Date: Mon, 29 Jul 2019 11:12:23 -0700 Subject: [PATCH 2/6] Add instructions on how to seed the fuzzer corpus from the spec tests. --- fuzz/README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/fuzz/README.md b/fuzz/README.md index eeb43ebd5..cac0a320a 100644 --- a/fuzz/README.md +++ b/fuzz/README.md @@ -33,6 +33,19 @@ INFO: seed corpus: files: 8 min: 1b max: 1b total: 8b rss: 133Mb ``` It will continue to generate random inputs forever, until it finds a bug or is terminated. The testcases for bugs it finds go into `fuzz/artifacts/simple_instantiate` and you can rerun the fuzzer on a single input by passing it on the command line `cargo fuzz run simple_instantiate my_testcase.wasm`. +## Seeding the corpus, optional + +The fuzzer works best when it has examples of small Wasm files to start with. Using `wast2json` from [wabt](https://github.com/WebAssembly/wabt), we can easily produce `.wasm` files out of the WebAssembly spec tests. + +```sh +mkdir spec-test-corpus +for i in lib/spectests/spectests/*.wast; do wast2json $i -o spec-test-corpus/$(basename $i).json; done +mv spec-test-corpus/*.wasm fuzz/corpus/simple_instantiate/ +rm -r spec-test-corpus +``` + +The corpus directory is created on the first run of the fuzzer. If it doesn't exist, run it first and then seed the corpus. The fuzzer will pick up new files added to the corpus while it is running. + ## Trophy case - [x] https://github.com/wasmerio/wasmer/issues/558 From 86316c474a9b4bac144c1b85320e863e97ec8ba6 Mon Sep 17 00:00:00 2001 From: Nick Lewycky Date: Sat, 27 Jul 2019 11:13:13 -0700 Subject: [PATCH 3/6] Use --enable-simd to control whether SIMD is enabled in the wasmparser. Before this change, 'wasmer run --backend=llvm some-simd.wasm' would run without complaint. Also, note that the flag is not part of the cache key, so after any successful run, we can run it again without passing the flag. --- lib/runtime-core/src/codegen.rs | 12 ++++++------ lib/runtime-core/src/parse.rs | 6 ++++-- src/bin/wasmer.rs | 3 +++ 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/lib/runtime-core/src/codegen.rs b/lib/runtime-core/src/codegen.rs index 47207fafd..0641fae84 100644 --- a/lib/runtime-core/src/codegen.rs +++ b/lib/runtime-core/src/codegen.rs @@ -1,6 +1,6 @@ use crate::{ backend::RunnableModule, - backend::{Backend, CacheGen, Compiler, CompilerConfig, Token}, + backend::{Backend, CacheGen, Compiler, CompilerConfig, Features, Token}, cache::{Artifact, Error as CacheError}, error::{CompileError, CompileResult}, module::{ModuleInfo, ModuleInner}, @@ -137,12 +137,12 @@ impl< } } -pub fn default_validating_parser_config() -> wasmparser::ValidatingParserConfig { +pub fn validating_parser_config(features: &Features) -> wasmparser::ValidatingParserConfig { wasmparser::ValidatingParserConfig { operator_config: wasmparser::OperatorValidatorConfig { enable_threads: false, enable_reference_types: false, - enable_simd: true, + enable_simd: features.simd, enable_bulk_memory: false, enable_multi_value: false, }, @@ -150,9 +150,9 @@ pub fn default_validating_parser_config() -> wasmparser::ValidatingParserConfig } } -fn validate(bytes: &[u8]) -> CompileResult<()> { +fn validate(bytes: &[u8], features: &Features) -> CompileResult<()> { let mut parser = - wasmparser::ValidatingParser::new(bytes, Some(default_validating_parser_config())); + wasmparser::ValidatingParser::new(bytes, Some(validating_parser_config(features))); loop { let state = parser.read(); match *state { @@ -180,7 +180,7 @@ impl< _: Token, ) -> CompileResult { if requires_pre_validation(MCG::backend_id()) { - validate(wasm)?; + validate(wasm, &compiler_config.features)?; } let mut mcg = MCG::new(); diff --git a/lib/runtime-core/src/parse.rs b/lib/runtime-core/src/parse.rs index 282168d26..51be05641 100644 --- a/lib/runtime-core/src/parse.rs +++ b/lib/runtime-core/src/parse.rs @@ -85,8 +85,10 @@ pub fn read_module< custom_sections: HashMap::new(), })); - let mut parser = - wasmparser::ValidatingParser::new(wasm, Some(default_validating_parser_config())); + let mut parser = wasmparser::ValidatingParser::new( + wasm, + Some(validating_parser_config(&compiler_config.features)), + ); let mut namespace_builder = Some(StringTableBuilder::new()); let mut name_builder = Some(StringTableBuilder::new()); diff --git a/src/bin/wasmer.rs b/src/bin/wasmer.rs index 0ac460dfc..b0ba2bde6 100644 --- a/src/bin/wasmer.rs +++ b/src/bin/wasmer.rs @@ -440,6 +440,9 @@ fn execute_wasm(options: &Run) -> Result<(), String> { CompilerConfig { symbol_map: em_symbol_map, track_state, + features: Features { + simd: options.features.simd || options.features.all, + }, ..Default::default() }, &*compiler, From 7a1ddc0b5a7e33ee7aef77307c012dcba6ab808b Mon Sep 17 00:00:00 2001 From: Nick Lewycky Date: Mon, 29 Jul 2019 12:01:16 -0700 Subject: [PATCH 4/6] Rename validate to validate_with_features. --- lib/runtime-core/src/codegen.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/runtime-core/src/codegen.rs b/lib/runtime-core/src/codegen.rs index 0641fae84..66fe50c41 100644 --- a/lib/runtime-core/src/codegen.rs +++ b/lib/runtime-core/src/codegen.rs @@ -150,7 +150,7 @@ pub fn validating_parser_config(features: &Features) -> wasmparser::ValidatingPa } } -fn validate(bytes: &[u8], features: &Features) -> CompileResult<()> { +fn validate_with_features(bytes: &[u8], features: &Features) -> CompileResult<()> { let mut parser = wasmparser::ValidatingParser::new(bytes, Some(validating_parser_config(features))); loop { @@ -180,7 +180,7 @@ impl< _: Token, ) -> CompileResult { if requires_pre_validation(MCG::backend_id()) { - validate(wasm, &compiler_config.features)?; + validate_with_features(wasm, &compiler_config.features)?; } let mut mcg = MCG::new(); From 5f0ce6ef2f3962564adf99f5446b971d57c2538f Mon Sep 17 00:00:00 2001 From: Nick Lewycky Date: Mon, 29 Jul 2019 15:47:38 -0700 Subject: [PATCH 5/6] Enable SIMD when parsing the spec tests. --- lib/spectests/build/spectests.rs | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/spectests/build/spectests.rs b/lib/spectests/build/spectests.rs index e3fdde530..52fbe4e7c 100644 --- a/lib/spectests/build/spectests.rs +++ b/lib/spectests/build/spectests.rs @@ -86,7 +86,7 @@ use wasmer_runtime_core::types::Value; use wasmer_runtime_core::{{Instance, module::Module}}; use wasmer_runtime_core::error::Result; use wasmer_runtime_core::vm::Ctx; -use wasmer_runtime_core::backend::Compiler; +use wasmer_runtime_core::backend::{Compiler, CompilerConfig, Features}; static IMPORT_MODULE: &str = r#" (module @@ -136,7 +136,7 @@ pub fn generate_imports() -> ImportObject { let mut features = wabt::Features::new(); features.enable_simd(); let wasm_binary = wat2wasm_with_features(IMPORT_MODULE.as_bytes(), features).expect("WAST not valid or malformed"); - let module = wasmer_runtime_core::compile_with(&wasm_binary[..], &get_compiler()) + let module = wasmer_runtime_core::compile_with_config(&wasm_binary[..], &get_compiler(), CompilerConfig { features: Features { simd: true }, ..Default::default() }) .expect("WASM can't be compiled"); let instance = module .instantiate(&ImportObject::new()) @@ -408,7 +408,7 @@ fn test_module_{}() {{ let module_str = \"{}\"; println!(\"{{}}\", module_str); let wasm_binary = wat2wasm(module_str.as_bytes()).expect(\"WAST not valid or malformed\"); - let module = wasmer_runtime_core::compile_with(&wasm_binary[..], &get_compiler()).expect(\"WASM can't be compiled\"); + let module = wasmer_runtime_core::compile_with_config(&wasm_binary[..], &get_compiler(), CompilerConfig {{ features: Features {{ simd: true }}, ..Default::default() }}).expect(\"WASM can't be compiled\"); module.instantiate(&generate_imports()).expect(\"WASM can't be instantiated\") }}\n", self.last_module, @@ -431,7 +431,7 @@ fn test_module_{}() {{ "#[test] fn {}_assert_invalid() {{ let wasm_binary = {:?}; - let module = wasmer_runtime_core::compile_with(&wasm_binary, &get_compiler()); + let module = wasmer_runtime_core::compile_with_config(&wasm_binary, &get_compiler(), CompilerConfig {{ features: Features {{ simd: true }}, ..Default::default() }}); assert!(module.is_err(), \"WASM should not compile as is invalid\"); }}\n", command_name, @@ -562,7 +562,7 @@ fn {}_assert_invalid() {{ "#[test] fn {}_assert_malformed() {{ let wasm_binary = {:?}; - let compilation = wasmer_runtime_core::compile_with(&wasm_binary, &get_compiler()); + let compilation = wasmer_runtime_core::compile_with_config(&wasm_binary, &get_compiler(), CompilerConfig {{ features: Features {{ simd: true }}, ..Default::default() }}); assert!(compilation.is_err(), \"WASM should not compile as is malformed\"); }}\n", command_name, From c21a91e2fe53130737fc31476f418e4469e7cfab Mon Sep 17 00:00:00 2001 From: Syrus Akbary Date: Tue, 30 Jul 2019 12:40:39 -0700 Subject: [PATCH 6/6] Updated CHANGELOG with missing singlepass changes --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3cdbc36b9..bb9af1487 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,8 @@ Blocks of changes will separated by version increments. - makes wasitests-generate output stdout/stderr by default & adds function to print stdout and stderr for a command if it fails - compiles wasm with size optimizations & strips generated wasm with wasm-strip - [#554](https://github.com/wasmerio/wasmer/pull/554) Finish implementation of `wasi::fd_seek`, fix bug in filestat +- [#550](https://github.com/wasmerio/wasmer/pull/550) Fix singlepass compilation error with `imul` instruction + ## 0.5.5 - 2019-07-10 - [#541](https://github.com/wasmerio/wasmer/pull/541) Fix dependency graph by making separate test crates; ABI implementations should not depend on compilers. Add Cranelift fork as git submodule of clif-backend @@ -41,6 +43,7 @@ Blocks of changes will separated by version increments. - [#523](https://github.com/wasmerio/wasmer/pull/523) Update wapm version to fix bug related to signed packages in the global namespace and locally-stored public keys ## 0.5.2 - 2019-07-02 +- [#516](https://github.com/wasmerio/wasmer/pull/516) Add workaround for singlepass miscompilation on GetLocal - [#521](https://github.com/wasmerio/wasmer/pull/521) Update Wapm-cli, bump version numbers - [#518](https://github.com/wasmerio/wasmer/pull/518) Update Cranelift and WasmParser - [#514](https://github.com/wasmerio/wasmer/pull/514) [#519](https://github.com/wasmerio/wasmer/pull/519) Improved Emscripten network related calls, added a null check to `WasmPtr`