Fluence services are composed from Wasm modules written in Rust and configuration files. The examples in this directory illustrate how to construct a variety of modules and services. If you are not accessing the examples from the Fluence devcontainer, see the [Setup Environment](https://doc.fluence.dev/docs/tutorials_tutorials/recipes_setting_up) section of the Fluence documentation.
**Services** are logical constructs derived from Wasm modules and the associated linking and configuration data, which describe how modules are linked with each other and how they interact, if at all, with the outside world. The instantiation of a service from Wasm modules takes place on the peer-to-peer network, i.e. at the node level. More specifically, **blueprints** are json documents that provide the necessary information to build, i.e. link, a service from Wasm modules. See Figure 1 for a stylized representation of module and service creation.
Figure 1: Module And Service Creation With The Marine Toolchain
```mermaid
stateDiagram
state "Rust Marine Code" as Code
state "Build wasm32-wasi Module" as Build
state "Test Wasm App With Cargo" as Test
state "REPL Locally Interact With Modules" as Repl
state "Module Configuration" as Config
state "Service Configuration" as Service
state "Deploy To Network" as Deploy
[*] --> Code
Code --> Build
Build --> Repl
Config --> Repl
Config --> Test
Build --> Test
Build --> Deploy
Service --> Deploy
```
Wasm modules can also be accessed locally through the Marine REPL and tested with `cargo test`.
## Greeting Example
This is a very simple service comprised of a single Wasam module, where the module only has one function, `greeting`. Fundamentally, this example illustrates how to build a function as a service (FaaS) for peer-to-peer deployment, with the exciting purpose of providing a *hello world* experience.
The Rust code for our greeting function is straight forward with the exception of the `#[marine]` macro (1). This macro, among other things, ensures for Wasm Interface Type checking and compilation and is critical for Rust code to compile to the desired wasm32-wasi target.
The test section is also pretty standard except for the macro (2). But looks can be deceiving. While the tests are run with the familiar `cargo test`, the actual code tested is **not** the Rust code from the body but from the Wasm module. The macro facilitates the import and execution of the Wasm module in the artifacts directory and the Config.toml file and then accesses the test function through the module namespace (3), which allows the prefixing of the Wasm function with the namespace (4). Hence, the code needs to be compiled to the Wasm module **before** running the test.
Let's have a look at the Config.toml file, which describes that only one module is linked by specifying the location of the module (1), the name (2), the desired memory allocation (#) and the logging (4) preference.
```toml
modules_dir = "artifacts/" # <-1
[[module]]
name = "greeting" # <-2
mem_pages_count = 1 # <-3
logger_enabled = false # <-4
```
In order to compile the example, which copies the Wasm module to the artifacts directory.
```bash
cd greeting
./build.sh
```
Now that we have a module and a config file, we can locally inspect and test:
```bash
mrepl Config.toml
Welcome to the Marine REPL (version 0.8.0)
Minimal supported versions
sdk: 0.6.0
interface-types: 0.20.0
app service was created with service id = 05af8670-7168-4ecd-b3e4-9579e0c66a25
For a detailed introduction to and review of Fluence, see the [documentation](https://doc.fluence.dev/docs/).
## Records Example
This example illustrates the use of the [records type](https://github.com/fluencelabs/interface-types/blob/master/crates/it-types/src/types.rs) both as a pure and an effector module. Pure modules are comprised of functions without side-effects, whereas effector modules contain at least one function with side-effect. The TestRecord struct imported by both the pure and effector code is defined in the `test-record`.
Since we have multiple modules comprising the service, the config file needs to reflect the multi module dependency:
```toml
# Config.toml
modules_dir = "artifacts/"
[[module]]
name = "records_effector"
mem_pages_count = 1
logger_enabled = true
[[module]]
name = "records_pure"
mem_pages_count = 1
logger_enabled = true
```
Please note that since the `TestRecord` struct is imported by both the pure and effector module there is no need to include it directly in the config specification.
Our linking scheme defined in Config.toml is fully reflected when we run the modules in the REPL:
```bash
mrepl Config.toml
Welcome to the Marine REPL (version 0.8.0)
Minimal supported versions
sdk: 0.6.0
interface-types: 0.20.0
app service was created with service id = 502d7df7-24fd-43ad-a2ba-204b9506376b
[SQLite](https://www.sqlite.org/index.html) is an ubiquitous persistence solution also available on the Fluence stack. Unlike basic compute modules, creating an SQLite service is a little more intricate. In order to create the service, we need two dependencies: An SQLite Wasm module and an SQLite adapter, where the adapter is a [WASM IT compliant](https://crates.io/crates/marine-sqlite-connector) [sqlite](https://github.com/stainless-steel/sqlite) package implementation. The SQLite Wasm module can be obtain from the repo as a [release](https://github.com/fluencelabs/sqlite/releases/tag/v0.15.0_w) or compiled from [code](https://github.com/fluencelabs/sqlite). For the purpose of this example, we use the release version -- see `build.sh`.
As a result, our service is comprised of two modules: the SQLite Wasm module and our sqlite module. The Rust code, see `src/main.rs` is pretty standard fare but our config file changes signficiantly:
```toml
# Config.toml
modules_dir = "artifacts/"
[[module]] <-1
name = "sqlite3"
mem_pages_count = 100
logger_enabled = false
[module.wasi]
preopened_files = ["/tmp"]
mapped_dirs = { "tmp" = "/tmp" }
[[module]] <-2
name = "sqlite_test"
mem_pages_count = 1
logger_enabled = false
[module.wasi]
preopened_files = ["/tmp"]
mapped_dirs = { "tmp" = "/tmp" }
```
1. The first module specification is the SQLIte Wasm module we downloaded from the repo. We allocate sizable mem-pages and disable our logger. We furst specify the wasi submodule that allows the Wasm module to write outside the sandbox, i.e., to the host node, for the SQLite db file. We also map our host directory to the Wasm module
2. We list our `sqlite_test` Wasm module and here too add the wasi submodule with parameters identical to the ones user in the prior SQLite Wasm module specification.
The local file access is critical if we want to use SQLite for or ethan just an in-memory solution and the mapped directory `tmp` needs to be used in our code module, `sqlite_test`. For example,
```rust
// src/main.rs
// <snip>
#[marine]
pub fn test3() {
let db_path = "/tmp/users.sqlite"; // <-NotethatthepathdirneedstomatchthemappeddirfromConfig.toml
let connection = marine_sqlite_connector::open(db_path).expect("db should be opened");
connection
.execute(
"
CREATE TABLE IF NOT EXISTS users (name TEXT, age INTEGER);
INSERT INTO users VALUES ('Alice', 42);
INSERT INTO users VALUES ('Bob', 69);
",
)
.expect("table should be created successfully");
let connection = marine_sqlite_connector::open(db_path).expect("db should be opened");
let cursor = connection.prepare("SELECT * FROM users").unwrap().cursor();
println!("table size is: {:?}", cursor.count());
}
// <snip>
```
You can build, inspect and test the project as outlined in the [Greeting Example](#Greeting-Example).
## URL Downloader Example
Our service makes a url call and persists the response to a file. Pretty straight forward, right? Due to the limitations of Wasm, sockets are not available and storage is also not part of the Wasm sandbox per se. In this example, we illustrate the necessary steps required to achieve our goal. For this we need a
* Facade module
* cUrl module
* Storage module
A facade module is the main module
### cUrl Module
While we don't have a socket available, we can use a wrapper to access the node's cUrl binary:
```rust
// curl_adapter
// main.rs
use marine_rs_sdk::marine;
use marine_rs_sdk::module_manifest;
use marine_rs_sdk::MountedBinaryResult;
use marine_rs_sdk::WasmLoggerBuilder;
module_manifest!();
/// Log level can be changed by `RUST_LOG` env as well.
pub fn main() {
WasmLoggerBuilder::new().build().unwrap();
}
#[marine]
pub fn download(url: String) -> String { // <-2
log::info!("download called with url {}\n", url);
let result = curl(vec![url]);
String::from_utf8(result.stdout).unwrap()
}
/// Permissions in `Config.toml` should exist to use host functions.
#[marine]
#[link(wasm_import_module = "host")] // <- 1
extern "C" {
fn curl(cmd: Vec<String>) -> MountedBinaryResult;
}
```
By
1. linking an external binary, cUrl, c-style into a marine interface
2. using the `download` function to wrap curl binary and expose it as a public function
Compiling this crate gives us a `curl_adapter.wasm` module exposing the `download` function we can use either directly or as part of a multi-module service. For the `download` function to actually work, we need to reflect the unique structure of our solution in the config file as well:
```toml
# Config.toml
# <snip>
[[module]] # <-1
name = "curl_adapter"
logger_enabled = true
[module.mounted_binaries] # <-2
curl = "/usr/bin/curl"
# <snip>
```
In our config file we
1. specify our basic module information including module name and logging preferences and
2. add a sub module referencing the mounted binary and command line call for cUrl, which specifies the Wasm module has access to the named host binary.
You can build, inspect and test the project as outlined in the [Greeting Example](#Greeting-Example).
### Local Storage
Sometimes in-memory storage is not enough, a database solution too much and file storage just right. Since we're again have to use the Wasm sandbox to write to the node, we need to again have that reflected in our Rust code and configuration. Unlike the cUrl example, we don't need to rely on an external binary to write to file but need to rely on an available read/write directory (1).
```rust
// local-storage dir
// main.rs
use marine_rs_sdk::marine;
use marine_rs_sdk::module_manifest;
use marine_rs_sdk::WasmLoggerBuilder;
use std::fs;
use std::path::PathBuf;
module_manifest!();
const SITES_DIR: &str = "/sites/"; // <-1
/// Log level can be changed by `RUST_LOG` env as well.
pub fn main() {
WasmLoggerBuilder::new().build().unwrap();
}
/// You can read or write files from the file system if there is permission to use directories described in `Config.toml`.
log::info!("put called with file name {}\n", name);
let rpc_tmp_filepath = format!("{}{}", SITES_DIR, name);
let result = fs::write(PathBuf::from(rpc_tmp_filepath.clone()), file_content);
if let Err(e) = result {
return format!("file can't be written: {}", e);
}
String::from("Ok")
}
#[marine]
pub fn get(file_name: String) -> Vec<u8> {
log::info!("get called with file name: {}\n", file_name);
let tmp_filepath = format!("{}{}", SITES_DIR, file_name);
fs::read(tmp_filepath).unwrap_or_else(|_| b"error while reading file".to_vec())
}
```
The majority of the heavy lifting for read-write operations is happening in the configuration file:
```toml
# Config.toml
# <snip>
[[module]] # <-1
name = "local_storage"
logger_enabled = true
[module.wasi] # <-2
preopened_files = ["./sites"]
# this is where files will be stored
mapped_dirs = { "sites" = "./sites" }
#<snip>
```
As always, we declare our module name and logging preference but then add the Wasi submodule not only specifying the directory mapping for the node's file system to the Wasm module, remember `const SITES_DIR: &str = "/sites/";` ?, but also permissioning the Wasm module to access host storage.
### Facade Module
The facade module is our entry to our download-n-save feature, thus being the main entry point to our
```rust
use marine_rs_sdk::marine;
use marine_rs_sdk::module_manifest;
use marine_rs_sdk::WasmLoggerBuilder;
module_manifest!();
pub fn main() {
WasmLoggerBuilder::new().build().unwrap();
}
/// Combining of modules: `curl` and `local_storage`.
/// Calls `curl` and stores returned result into a file.
Basically, `get_n_save` uses the `download` function from the `curl_adapter` module and the `get` and `put` functions from the `local_storage` module to accomplish our goal. Since our Wasm modules [share nothing'](https://en.wikipedia.org/wiki/Shared-nothing_architecture), we need to somehow make available what's needed. Since we are dealing with Wasm modules, the usual Rust import process clearly won't work. Instead, we link public functions from other Wasm modules with the `#[link(wasm_import_module = "???")]` macro. Please note the (1) and (2) are not (directly) dealing with the facade module's access to host functionality. That is, we specify explicitly what's shared from one module to another.
Of course, that makes for a simple config specification for the facade module:
```toml
# Config.toml
# <snip>
[[module]]
name = "facade"
logger_enabled = true
```
With our facade module in place, we can explore the fully linked solution in the REPL:
```bash
mrepl Config.toml
Welcome to the Marine REPL (version 0.8.0)
Minimal supported versions
sdk: 0.6.0
interface-types: 0.20.0
app service was created with service id = 4d8ff671-a4da-4ff6-b757-0498aa0b14ce
We see our three modules (1, 2, 3) and their respective public functions, which we can test on a per module basis and as the facade service aggregate. For example:
In the examples above we used the `marine_test` macro to test a service. This example illustrates another ability of the `marine_test` macro: testing several services at once. To show that we will create couple of services:
a producer service which will create some data, and a consumer service which will process the data. Then write a test that ensures that the consumer properly processed data from the producer.
The only new thing here is `#[marine]` on a struct. This adds the structure into module interface and requires all the fields to be supported by default or be another `#[marine]` structure. There is more information about it in [docs](https://doc.fluence.dev/docs/knowledge_aquamarine/marine/marine-rs-sdk#function-export).
Consumer:
```rust
use marine_rs_sdk::marine;
use marine_rs_sdk::module_manifest;
module_manifest!();
pub fn main() {}
#[marine]
pub struct Data {
pub name: String,
}
#[marine]
pub fn consume(data: Data) -> String {
data.name
}
```
There is nothing special.
The test will show what we can do with them:
```rust
fn main() {}
#[cfg(test)]
mod tests {
use marine_rs_sdk_test::marine_test;
#[marine_test(
producer(
config_path = "../producer/Config.toml", // <-1
modules_dir = "../producer/artifacts"
),
consumer(
config_path = "../consumer/Config.toml",
modules_dir = "../consumer/artifacts"
)
)]
fn test() {
let mut producer = marine_test_env::producer::ServiceInterface::new(); // <-2
let mut consumer = marine_test_env::consumer::ServiceInterface::new();
let input = marine_test_env::producer::Input { // <-3
first_name: String::from("John"),
last_name: String::from("Doe"),
};
let data = producer.produce(input); // <-4
let result = consumer.consume(data); // <-5
assert_eq!(result, "John Doe")
}
}
```
As the `marine_test` needs only compiled .wasm files and not the code, the test can be in any project. In this case the test is in a separate crate.
We describe the services as named pairs of config file and directory with .wasm files(1), then in test function we create services(2). Please note, that each service creates its own `marine` runtime. Then, we create a structure(3) to pass it to a function from interface of producer service(4) and finally pass its result to the consumer service. The `ServiceInterface` and interface structures are accessed through `marine_test_env` — the module defined by the `marine_test` macro. The functions in turn are accessed through the `ServiceInterace` instance.
Also we can rewrite tests by applying `marine_test` to a `mod`:
```rust
#[cfg(test)]
#[marine_rs_sdk_test::marine_test(
producer(
config_path = "../producer/Config.toml",
modules_dir = "../producer/artifacts"
),
consumer(
config_path = "../consumer/Config.toml",
modules_dir = "../consumer/artifacts"
)
)]
mod tests_on_mod {
#[test]
fn test() {
let mut producer = marine_test_env::producer::ServiceInterface::new();
let mut consumer = marine_test_env::consumer::ServiceInterface::new();
let input = marine_test_env::producer::Input {
first_name: String::from("John"),
last_name: String::from("Doe"),
};
let data = producer.produce(input);
let result = consumer.consume(data);
assert_eq!(result, "John Doe")
}
}
```
It defines one `marine_test_env` for the whole `mod`, so we can write utility functions that use types from `marine_test_env`, and we do not have to wrap each function in the macro.
There is another way to generate `marine_test_env` and use in tests: [cargo build scripts](https://doc.rust-lang.org/cargo/reference/build-scripts.html). In this example we will show how to use `marine-rs-sdk-test` to generate test environment and how to enable IDE support for generated files.
Assume that we have Greeting example compiled, and we want to write tests for it using build.rs. Firstly, we need to write a build.rs file:
```rust
use marine_rs_sdk_test::generate_marine_test_env;
use marine_rs_sdk_test::ServiceDescription;
fn main() {
let services = vec![ // <-1
("greeting".to_string(), ServiceDescription {
config_path: "Config.toml".to_string(),
modules_dir: Some("artifacts".to_string()),
})
];
let target = std::env::var("CARGO_CFG_TARGET_ARCH").unwrap();
and add `marine-rs-sdk-test` to build-dependencies section of `Cargo.toml`:
```toml
[build-dependencies]
marine-rs-sdk-test = "0.4.0"
```
So, what do we see here:
1. We created pairs (name, description) for services
2. We checked if we are compiling to wasm32 target. It is needed when service and tests are in one crate: `build.rs` runs before the crate compilation, and it depends on compilation result. So, we work around this issue using that service is compiled to wasm32 and tests are compiled to the native target: we generate `marine_test_env` only when we are compiling to non-wasm target. This will not work great if you need to compile service to non-wasm target not only for tests. Another option is to use a separate crate for tests.
3. We pass services, generated file name, and path to the build.rs file to the generator. The last argument will always be `file!()`.
4. We signal when to re-run `build.rs`. This part must be customized for every project: for the projects where tests are in a separate crate, `build.rs` must be re-run when wasm files or `Config.toml` are changed.
Then we need to actually write a test in the `src/main.rs`:
let mut greeting = marine_test_env::greeting::ServiceInterface::new(); // <-2
let actual = greeting.greeting("name".to_string());
assert_eq!(actual, "Hi, name");
}
}
```
1. We include generated file to out module with tests. `include_test_env!` is just a wrapper over `include!(concat!(env!("OUT_DIR"), $filename))`.
2. We use `marine_test_env` as usual.
And the last thing to do: setting up IDE, as the main advantage of `build.rs` approach is the IDE support.
### CLion:
* in the `Help -> Actions -> Experimental Futures` enable `org.rust.cargo.evaluate.build.scripts`
* refresh cargo project in order to update generated code: change `Cargo.toml` and build from IDE or press `Refresh Cargo Project` in `Cargo` tab.
### VS Code:
* install `rust-analyzer` plugin
* change `Cargo.toml` to let plugin update code from generated files
After that IDEs will provide code completion for `marine_test_env`. In order to update the env you should build service to wasm, then build tests (with `build.rs` re-run).
In each of the examples we created modules and services configurations and tested and inspected them with the Marine REPL. Feels like we're all dressed up and nowhere to go. In this section we briefly discuss how to deploy our modules and configurations to the network as services using the *Greeting* example.
Before we begin, you need to have the `fldist` tool installed. See the [Tools documentation](https://doc.fluence.dev/docs/knowledge_tools) for more information.
We use the `fldist new_service` command to do our bidding:
```zsh
fldist new_service --help
fldist new_service
Create service from a list of modules
Options:
--help Show help [boolean]
--version Show version number [boolean]
-v, --verbose Display verbose information such as created client
seed + peer Id and relay peer id
[boolean] [default: false]
-s, --seed Client seed [string]
--sk, --secret-key Clients ed25519 private key in base64 (32 byte)
--ms, --modules array of path:config pairs; meaning <pathtowasm
module>:<pathtoconfig> [array] [required]
-n, --name name of the service; will be set in the blueprint
[string] [required]
```
Aside from our modules and configuration, we also want to supply the peer id of the node we want to host our service. You cna find all available nodes on the [Fluence Dashboard](https://dash.fluence.dev/nodes). Please note that for all of our examples we will use peer `12D3KooWKnEqMfYo9zvfHmqTLpLdiHXPe4SVqUWcWHDJdFGrSmcA`. Alternatively, we can deploy to a local node -- see [Deploy A Local Node](https://doc.fluence.dev/docs/tutorials_tutorials/tutorial_run_local_node) for instructions.
To create our greeting service on peer `12D3KooWKnEqMfYo9zvfHmqTLpLdiHXPe4SVqUWcWHDJdFGrSmcA`:
2. Module and config file location with the `--ms` flag
3. Service name with the `--name` flag
Which results in a success message and more importantly, the unique id for the deployed service:
```zsh
service id: 810b8abc-6b9d-4227-a228-a53145086464
service created successfully
```
The (peer id, service id) tuple is going to be useful once you start putting the service to work with, say, [Aqua](https://doc.fluence.dev/aqua-book/) and you should hold on the data for future use. Also, you can look up your service on the [Fluence Dashboard](https://dash.fluence.dev/blueprint/d67cdb81c416bff902e0b5847a39b84e2cb42d21dde6426de14bb75d1d61236d/). For more detailed information regarding the Fluence solution see the [Fluence documentation](https://doc.fluence.dev/docs/) and if you have any questions, comments or suggestions for improvements, please open create Issue or open a PR.
