This content will be ignored.

Getting Started

This guide will take you through the fundamentals of the Picard Framework, starting with the basics of the Actor Model through to more advanced topics such as application resilience.

If you're familiar with the Actor Model and have used other actor frameworks, you will find Picard familiar. In particular, Picard has been inspired by Scala's Akka and has adopted some of the core concepts and terminology from that project.

If you've never used actors in application development before, this documentation aims to be concise and easy to understand. No prior knowledge of actors is necessary.

We welcome and encourage contributions to this guide. Please feel free to create Github issues with feedback or create PRs with changes. This documentation's source lives at: https://github.com/picard-rs/picard.

Note: Picard is pre-1.0, and as such the framework is evolving. You can expect some API-level changes, but we do strive to keep breaking changes to an absolute minimum. Between versions 0.1 to 0.2.3 there was only one minor API change, for example.

Actors

The actor model is a conceptual model to deal with concurrent computation¹. At the core of the Picard framework are four main components:

ActorSystem - Every Picard application has an ActorSystem that manages actor lifecycles
Actor - Rust types that implement the Actor trait so they may receive messages
Props - A configuration object that is Send and can be used when creating new actors
ActorRef - A lightweight type that is inexpensive to clone and can be used to interact with its underlying Actor, such as sending messages to it

Let's look at each of these and see how a simple application is created.

Defining Actors

An Actor is the fundamental unit of computation. Actors communicate solely through messages in an asynchronous fashion. An actor can perform three distinct actions based on the message it receives:

send a finite number of messages to other actors
create a finite number of new actors
change its state or designate the behavior to be used for the next message it receives

Actors interact with each other by passing messages. There is no assumed order to the above actions, and they could be carried out concurrently. Two messages that are sent concurrently can arrive in either order.

To define an actor, the system needs to understand how an actor should handle the messages it receives. To do this, implement the Actor trait on your data type and, at a minimum, provide a recv method.

Here's the Rust code:

#![allow(unused)]
fn main() {
struct MyActor;

impl Actor for MyActor {
    type Msg = String;

    fn recv(&mut self,
            ctx: &Context<String>,
            msg: String,
            sender: Sender) {

        println!("received {}", msg);
    }
}
}

In this example, a simple struct MyActor implements the Actor trait. When a message is sent to MyActor, it is scheduled by the system for immediate execution. The recv function is invoked and the message is printed to stdout.

Creating Actors

Every application has an ActorSystem. The actor system provides actor management and the runtime to execute actors when they're sent messages. It also provides essential services such as starting actors and exposing system services.

To start the actor system:

#![allow(unused)]
fn main() {
let sys = ActorSystem::new().unwrap();
}

Here we see that the actor is started using ActorSystem::new. Once we've started the actor system we're ready to create some actors.

We can also configure the system with a custom name using the SystemBuilder:

#![allow(unused)]
fn main() {
let sys = SystemBuilder::new()
                        .name("my-app")
                        .create()
                        .unwrap();
}

Once the actor system is started, we can begin to create actors:

#![allow(unused)]
fn main() {
let my_actor = sys.actor_of::<MyActor>("my-actor").unwrap();
}

Every actor has a name that is required to be unique among its singlings (those actors sharing the same parent actor). A name is also required so that we can interact with the actor using its actor path.

Although this is just a single line of code, a lot is happening behind the scenes. Actor lifecycles and state are managed by the system. When an actor starts, it keeps the properties in case it needs to restart the actor if it fails. When an actor is created, it gets its own mailbox for receiving messages and other interested actors are notified about the new actor joining the system.

Actor References

When an actor is started using variants of actor_of, the system returns a reference to the actor, an ActorRef. The actual actor instance remains inaccessible directly, its lifecycle being managed and protected by the system. In Rust terms, the system has and always maintains 'ownership' of the actor instance. When you're interacting with actors, you're actually interacting with the actor's ActorRef! This is a core concept of the actor model.

An ActorRef always refers to a specific instance of an actor. When two instances of the same Actor are started, they're still considered separate actors, each with different ActorRefs.

Note: ActorRefs are inexpensive and can be cloned (they implement Clone) without too much concern about resources. References can also be used in Props as a field in another actor's factory method, a pattern known as endowment. ActorRefs can be sent as a message to another actor, a pattern known as introduction.

Sending Messages

Actors communicate only through sending and receiving messages. They are isolated and never expose their state or behavior.

If we want to send a message to an actor, we use the tell method on the actor's ActorRef:

#![allow(unused)]
fn main() {
let my_actor = sys.actor_of::<MyActor>("my-actor").unwrap();
my_actor.tell("Hello my actor!".to_string(), None);
}

Here, we've sent a message of type String to our MyActor actor. The second parameter lets us specify a sender as an Option<BasicActorRef> (type alias Sender). Since we're sending the message from main and not from an actor, we're setting the sender as None.

Example

Let's go back to our MyActor and combine what we've seen so far into a complete example:

Cargo.toml dependencies:

[dependencies]
picard = "0.4"

main.rs:

use std::time::Duration;
use picard::actors::*;

struct MyActor;

// implement the Actor trait
impl Actor for MyActor {
    type Msg = String;

    fn recv(&mut self,
            _ctx: &Context<String>,
            msg: String,
            _sender: Sender) {

        println!("Received: {}", msg);
    }
}

// start the system and create an actor
fn main() {
    let sys = ActorSystem::new().unwrap();

    let my_actor = sys.actor_of::<MyActor>("my-actor").unwrap();

    my_actor.tell("Hello my actor!".to_string(), None);

    // force main to wait before exiting program
    std::thread::sleep(Duration::from_millis(500));
}

Here, we've started the actor system and an instance of MyActor. Then, we sent a message to the actor.

To explore this example project, click here.

Actor Parameters

So far we've seen simple actors that have no constuctor and arguments. Actors can support argument passing by implmenting the ActorFactoryArgs trait:

#![allow(unused)]
fn main() {
struct Drone {
    battery_mAh: u32,
    max_range: Option<u32>,
}

// single argument
impl ActorFactoryArgs<u32> for Drone {
    fn create_args(battery_mAh: u32) -> Self {
        Drone {
            battery_mAh,
            max_range: None
        }
    }
}

// multiple arguments - using tuple
impl ActorFactoryArgs<(u32, u32)> for Drone {
    fn create_args((battery_mAh, max_range): (u32, u32)) -> Self {
        Drone {
            battery_mAh,
            max_range: Some(max_range)
        }
    }
}

...

let d1 = sys.actor_of_args::<Drone, _>("drone-1", 100_000).unwrap();
let d2 = sys.actor_of_args::<Drone, _>("drone-2", (200_000, 50)).unwrap();
}

Props

In systems that have multi-level actor hierarchies, actor parameters can be encapsulated into a configuration object Props. This allows actor configuration and actor starting to be decoupled. Configuration can be done once and shared with multiple actor starting sites. It also enables common actor patterns.

#![allow(unused)]
fn main() {
struct ClientWebhookManager {
    customer: Uuid,
    max_retries: u32,
}

impl ActorFactoryArgs<(Uuid, u32)> for ClientWebhookManager {
    fn create_args((customer, max_retries): (Uuid, u32)) -> Self {
        ClientWebhookManager {
            customer,
            max_retries
        }
    }
}

...
// create props, `Clone` and `Send` to anywhere that needs to start
// the respective actor at any time
let props = Props::new_args::<ClientWebhookManager, _>((cid, 3));

// use props to start actor
let w1 = sys.actor_of_props("manager-1", props.clone()).unwrap();
let w2 = sys.actor_of_props("manager-2", props).unwrap();

}

Message Guarantees

Picard provides certain guarantees when handling messages:

Message delivery is 'at-most-once'. A message will either fail to be delivered, or be delivered one time. There is no repeat delivery of the same message.
An actor handles one message at any time.
Messages are stored in an actor's mailbox in the order that they are received.

On this page, you learned the basics of creating a Picard application using actors. Let's move on to the next section to see more comprehensive example using multiple message types:

Sending multiple message types

Multi-type Messaging

In real world applications, actors will typically receive different message types and execute different behaviors based on the type received.

So far you've seen a simple example where an actor's message type is defined in the Actor::Msg associated type. More specifically, this defines an actor's mailbox type. To allow an actor to receive multiple message types, Picard provides a Receive<T> trait and the #[actor] attribute.

Let's see how these are used:

// Define the messages we'll use
#[derive(Clone, Debug)]
pub struct Add;

#[derive(Clone, Debug)]
pub struct Sub;

#[derive(Clone, Debug)]
pub struct Print;

// Define the Actor and use the 'actor' attribute
// to specify which messages it will receive
#[actor(Add, Sub, Print)]
struct Counter {
    count: u32,
}

impl Actor for Counter {
    // we used the #[actor] attribute so CounterMsg is the Msg type
    type Msg = CounterMsg;

    fn recv(&mut self,
                ctx: &Context<Self::Msg>,
                msg: Self::Msg,
                sender: Sender) {

        // Use the respective Receive<T> implementation
        self.receive(ctx, msg, sender);
    }
}

impl Receive<Add> for Counter {
    type Msg = CounterMsg;

    fn receive(&mut self,
                _ctx: &Context<Self::Msg>,
                _msg: Add,
                _sender: Sender) {
        self.count += 1;
    }
}

impl Receive<Sub> for Counter {
    type Msg = CounterMsg;

    fn receive(&mut self,
                _ctx: &Context<Self::Msg>,
                _msg: Sub,
                _sender: Sender) {
        self.count -= 1;
    }
}

impl Receive<Print> for Counter {
    type Msg = CounterMsg;

    fn receive(&mut self,
                _ctx: &Context<Self::Msg>,
                _msg: Print,
                _sender: Sender) {
        println!("Total counter value: {}", self.count);
    }
}

fn main() {
    let sys = ActorSystem::new().unwrap();

    let props = Props::produce::<Counter>();
    let actor = sys.actor_of_props(props, "counter").unwrap();
    actor.tell(Add, None);
    actor.tell(Add, None);
    actor.tell(Sub, None);
    actor.tell(Print, None);

    // force main to wait before exiting program
    std::thread::sleep(Duration::from_millis(500));
}

In this example, we've used #actor[Add, Sub, Print] to set up the actor to receive Add, Sub and Print types. For each of these, the Receive<T> trait is implemented on the actor, defining how each message should be handled.

Note: When using the #[actor()] attribute, the actor's Msg associated type should be set to '[DataType]Msg'. E.g. if an actor is a struct named MyActor, then the Actor::Msg associated type will be MyActorMsg.

By utilizing Receive<T> and #[actor], complex message handling can be defined clearly and concisely. For more advanced messaging examples see Advanced Messaging.

Serializable messages

Enable the serde feature to require every actor message to implement serde::Serialize and serde::de::DeserializeOwned, for example to send messages to actors on another machine:

[dependencies]
picard = { version = "0.4", features = ["serde"] }
serde = { version = "1", features = ["derive"] }

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Add;
}

Creating an actor whose Msg type doesn't implement both traits is a compile error. The message enum generated by #[actor] derives them automatically, so only the types listed in #[actor(...)] need the derives. picard-cqrs's EntityMsg does the same.

A serializable message can't carry process-local values such as an std::sync::mpsc::Sender or an ActorRef. Give the actor those values when you create it, for example with actor_of_args, and send plain data in the message. When you need the sender's address, use the sender argument of recv.

Picard's own actors (channels, system actors) keep their local messages and are unaffected. Their messages use the LocalMessage bound, which is what the runtime requires internally. Message is LocalMessage plus the serde traits when the feature is enabled.

Cargo features are unified across a build, so enabling picard/serde in one crate applies it to every crate that uses Picard in the same build.

In the next section, we'll explore the relationship between actors and how actors form a hierarchy.

Actor Hierarchy

Actors in Picard form a hierarchy with each actor addressable by a path. An actor's place in the hierarchy is determined by the location of its parent. Let's take a look at what the actor hierarchy looks like immediately after the actor system has started:

picard
└─ user
└─ system
   └─ logger
   └─ sys_events
   └─ dead_letters
   └─ dl_logger
└─ temp

We can see that without starting any actors ourselves, there's already a number of actors running. At the base of the hierarchy is our application which by default is named picard unless a name was provided using SystemBuilder.

There's then three root actors, user, system and temp. These are guardian actors under which all other actors are created. Perhaps the most important of these is user, since most actors created as part of the application are created in this branch.

If we start an actor (e.g. using system.actor_of::<MyActorType>("my-actor")) we can see it added under user:

my-app
└─ user
   └─ my-actor      <-- our new actor is added
└─ system
   └─ logger
   └─ sys_events
   └─ dead_letters
   └─ dl_logger
└─ temp

In this case the newly created my-actor has a path of /user/my-actor. Since it was started on ActorSystem it is considered a top-level actor.

Let's look at how the hierarchy changes when another actor is started, this time from within /user/my-actor's recv method using Context.actor_of.

#![allow(unused)]
fn main() {
impl Actor for MyActor {
    type Msg = String;

    fn recv(&mut self,
            ctx: &Context<String>,
            msg: String,
            sender: Sender) {

        ctx.actor_of::<MyActor>("my-child").unwrap();
    }
}
}

Here MyActor will start another actor, which is also an instance of MyActor.

my-app
└─ user
   └─ my-actor
      └─ my-child   <-- our new actor is added
└─ system
   └─ logger
   └─ sys_events
   └─ dead_letters
   └─ dl_logger
└─ temp

Since the new actor was started using my-actor's context it is added to the hierarchy as a child of my-actor. my-child's path becomes /user/my-actor/my-child.

Let's move on the next section where we'll look at how the actor hierarchy is used in supervision to build resilient, self healing applications.

Fault Tolerance

Picard applications exhibit fault tolerant behavior through self-healing. This is achieved by supervision - each actor has a supervisor that is responsible for determining what to do if the actor panics. In Picard, an actor's parent is its supervisor. This 'parental supervision' is a natural fit since the actor system is a hierarchy.

When an actor fails we can't guarantee its state is not corrupted. Its parent has three choices (strategies):

Restart the actor
Escalate up to the next supervisor
Stop the actor

Supervision isolates failures and errors don't leak or cascade. Instead the system can decide the best way to restore to a clean, working state, or to gracefully stop.

The supervision strategy an actor should use to supervise its children can be set in its supervisor_strategy method:

#![allow(unused)]
fn main() {
fn supervisor_strategy(&self) -> Strategy {
    Strategy::Stop
}
}

In this case, if a child fails it will choose to stop it.

Note: If supervisor_strategy is not set the default implementation is Strategy::Restart.

Mailboxes

An actor has its own mailbox that messages are queued to during message delivery. When a message is sent to an actor it is added to the actor's mailbox and the actor is then scheduled to run. If during handling of a message the actor fails (panics) messages can still continue to be sent to the actor since the mailbox is separate. This allows the supervisor to handle the failure without losing messages - a restarted actor will then continue handling the queued messages once it restarts.

An actor's mailbox continues to exist until its actor is stopped or the system is stopped.

Restart Strategy

#![allow(unused)]
fn main() {
fn supervisor_strategy(&self) -> Strategy {
    Strategy::Restart
}
}

The restart strategy attempts to restart the actor in its initial state, which is considered to be uncorrupted.

The sequence followed is:

The actor's mailbox is suspended. Messages can be received but they won't be handled
All children of the failed actor are sent termination requests
Wait for all children to terminate - a non-blocking operation
Restart the failed actor
Resume the actor's mailbox and message handling

Escalate Strategy

#![allow(unused)]
fn main() {
fn supervisor_strategy(&self) -> Strategy {
    Strategy::Escalate
}
}

The escalate strategy moves the decison of how to handle the failure up to the supervisor's parent. This works by failing the current supervisor and its parent will determine how to handle the failure.

The sequence followed is:

The actor's mailbox is suspended. Messages can be received but they won't be handled
The supervisor escalates and its mailbox is suspended
The new supervisor decides which supervision strategy to follow

Stop Strategy

#![allow(unused)]
fn main() {
fn supervisor_strategy(&self) -> Strategy {
    Strategy::Stop
}
}

The stop strategy stops the failed actor, removing it and its mailbox from the system.

Dead letters

When an actor is terminated all existing ActorRefs are invalidated. Messages sent (using tell) are instead rerouted to dead letters, a dedicated channel that publishes undeliverable messages to any interested actors. Picard has a default subscriber, dl_logger, that simply logs dead letter messages using info!.

Supervisor Design

Good supervisor design is key to designing resilient, fault tolerant systems. At the core of this is creating an actor hierarchy that matches message flow and dependency.

Next we'll see how actor paths can be utilized to message actors without an actor reference and broadcast to entire segments of the actor hierarchy.

Actor Selection

The primary means to interact with an actor is through it's actor reference (ActorRef). Since every actor also has a path it's possible to 'lookup' an actor by that path. It's also possible to interact with all actors that are part of a path.

For example, if an actor is known to live at /user/comms/high_gain_1, but we don't have the actor reference for this actor, we can perform a selection:

#![allow(unused)]
fn main() {
let hga = ctx.select("/user/comms/high_gain_1").unwrap();
}

This will return an ActorSelection. In some ways an ActorSelection behaves like an ActorRef but represents a collection of actors. When sending a message to a selection all the actors in the selection that accept the sent message type will receive the message.

To send messages to a selection:

#![allow(unused)]
fn main() {
let hga = ctx.select("/user/comms/high_gain_1").unwrap();
hga.try_tell("I've arrived safely".into(), None)?;
}

try_tell returns the number of unique actors whose mailboxes accepted the message; if traversal reaches the same actor by multiple paths, it receives the message only once. The method does not wait for actor processing to complete. It returns SelectionSendError::NoRecipients when the selection matches no actors, and SelectionSendError::DeliveryFailed { delivered, failed } when delivery fails for one or more selected actors. An actor that rejects the message type is counted as a failed delivery.

While this example highlights how it's possible to message actors based on their path in practice it should be carefully considered. ActorRef (and even BasicActorRef) is almost always the better choice for actor interaction since messages are directly sent to the actor's mailbox without any preprocessing or cloning. However there are several use cases where ActorSelection makes sense:

You know the path of an actor but due to design you don't have its ActorRef
You want to broadcast a message to all actors within a path

It is possible to select all actors under an actor path and send the same message the actors in the selection:

#![allow(unused)]
fn main() {
let sel = ctx.selection("/user/home-control/lighting/*").unwrap();
let delivered = sel.try_tell(Protocol::Off, None)?;
println!("Message accepted by {delivered} lights");
}

In this example an actor responsible for lighting in a home has a child actor for each individual light. If we want to turn off all lights a control message (Protocol::Off) could be sent to /user/home-control/lighting/*. Each child actor will receive the same message.

Note: Paths are relative to the location where the selection is being made. E.g. from the actor lighting's context, all children could be selected using ctx.selection("*").

We've seen that ActorSelection provides flexibility for certain use cases such as when an ActorRef isn't known at compile time, but more specifically for messaging multiple actors. This comes at the cost of traversing part of the actor hierarchy and cloning messages.

Next we'll see how Channels provide publish/subscribe features to enable actor choreography.

Channels

Picard channels allow for messages to be published to interested subscribers. Channels are actors so messaging a channel works the same way as any othe actor.

Starting a channel

The picard::actors::channel function returns a channel:

#![allow(unused)]
fn main() {
let chan: ChannelRef<PowerStatus> = channel("power-status", &sys).unwrap();
}

Subscribing

To subscribe to and receive messages from a channel an actor must support the message type of the channel.

In the above example we created a channel that publishes PowerStatus messages, notifying components in an autonomous vehicle of changes in available battery energy. Let's create two actors that will subscribe to the channel so they can receive this message:

#![allow(unused)]
fn main() {
#[actor(PowerStatus)]
struct GpsActor;

#[actor(PowerStatus)]
struct NavigationActor;

...

// Each actor would send a Subscribe message to the
// channel, typically in `pre_start`. E.g.:
let sub = Box::new(ctx.myself());
chan.tell(Subscribe { actor: sub, topic: "*".into() }, None);

}

Here we have two actors that each need to receive status changes in available battery power so they can adapt their behavior. They both must support the PowerStatus message that the channel publishes. You'll notice that we need to use Box to create a trait object of Tell<PowerStatus>.

The Subscribe message is used to subscribe an actor to a channel, which you'll notice requires a topic. A channel consists of one or more topics, typically that have a common theme. When a message is published it is published to a specific topic on the channel.

Note: When subscribing to a topic, if it does't already exist it will be created and any future messages published to it will be sent to the subscriber.

Publishing

The Publish message is used to publish to a channel:

#![allow(unused)]
fn main() {
let stat = PowerStatus { ... };
chan.tell(Publish { msg: stat, topic: "power".into() }, None);
}

This message will be cloned and sent to each subscriber of my-topic on the channel chan.

In this case, it may be that the GpsActor will choose to lower the GPS sampling rate if the battery level falls below a certain percentage, thus lowering the power used. The NavigationActor might override any active mission and force the vehicle to return to base if the power level drops to a critical level. The same use of channels could be applied to ecommerce platforms, payments systems, warehouse logistics, shipping tracking, etc.

Common channels

When the actor system starts serveral channels are created. These channels help developers receive messages about system events and failed messages.

System events

The System Events channel provides events including ActorCreated, ActorRestarted and ActorTerminated events. Each of these are reprented as topic actor.created, actor.restarted and actor.terminated topics respectively. The message type is SystemEvent enum

Example:

#![allow(unused)]
fn main() {
sys.sys_events()
    .tell(Subscribe { actor: sub1, topic: "actor.created".into() }, None);
}

Note: System events are considered system messages and therefore a subscriber's Actor::sys_recv method will be invoked instead of Actor::recv.

Dead letters

The Dead Letters channel publishes messages that failed to be delivered to their destination actor. This channel can be subscribed to to handle those messages. Note: Dead letters use Debug representation of the original undelivered message, limiting the use of dead letters to logging of failed messages rather than actually acting upon them.

Example:

#![allow(unused)]
fn main() {
sys.dead_letters().tell(Subscribe { actor: sub1, topic: "*".into() }, None);
}

Channels form an integral part of the Picard system and provide essential services to creating dynamic applications where actors collaborate to achieve a common goal.

Next we'll look at scheduling messages to be sent at a time in the future.

Scheduling Messages

Remote messaging

Picard's opt-in remoting feature provides versioned JSON messaging over an authenticated WebSocket connection. The built-in transport uses WebSocket over TCP, so a trusted reverse proxy can terminate TLS. Picard still authenticates each connecting peer with a bearer token; TLS termination does not replace peer authentication.

This is remote message delivery, not transparent clustering. There is no discovery, failover, exactly-once guarantee, or automatic remote-reference conversion. UDP and distributed data (including CRDTs) are out of scope.

Enable remote messaging

Enable the feature in the application:

picard = { version = "0.4", features = ["remoting"] }

The feature enables Picard's serde message bound. Actor messages delivered remotely must implement Serialize and DeserializeOwned.

Create a receiver actor and start its authenticated WebSocket listener:

let listener = system.listen_remote_actors(
    "127.0.0.1:9000".parse()?,
    PeerAuthenticator::new([("orders-client".into(), token.into())]),
)?;

The listener binds plain HTTP. Bind it to a private interface or loopback when a trusted reverse proxy terminates TLS. The proxy must prevent untrusted clients from reaching the backend listener directly.

Connect and deliver a serialized message:

let peer = RemotePeer::new(
    "orders",
    "wss://actors.example.test/remoting",
    token,
)?;
let remote = system.connect_remote(peer);
let address = ActorAddress::remote(
    remote_system_id,
    "wss://actors.example.test/remoting",
    "/user/orders",
);
let ack = remote
    .send(address, None, serde_json::to_value(&message)?)
    .await?;

listen_remote_actors resolves the address's system ID and actor path against the local actor registry, deserializes the payload using that actor's message type, and acknowledges the request after it enters the mailbox. Acknowledgment does not mean the actor has processed the message. Unknown actors, invalid payloads, unsupported protocol versions, failed authentication, and transport errors are reported to the caller.

Use RemoteConnection::ask when the actor must reply through the usual Sender:

let reply: String = remote
    .ask(address, request, std::time::Duration::from_secs(2))
    .await?;

The destination actor receives a temporary BasicActorRef as its Sender; the first serializable message it sends through that reference is returned to the caller. This temporary reference supports typed try_tell replies; it does not support watches, system messages, or type-erased try_tell_any. A reply received after the timeout is rejected. If the actor drops its sender without replying, the call returns RemoteError::NoReply; if no reply arrives before the deadline, it returns RemoteError::Timeout. A connection or peer failure is reported as RemoteError::Transport. The ask timeout bounds the remote actor's reply wait after the request reaches its system; transport setup failures are reported separately. Timeouts are limited to 60 seconds to bound the time a listener worker waits on any one request.

For a delivery barrier, use RemoteConnection::send_processed:

remote
    .send_processed(address, message, std::time::Duration::from_secs(2))
    .await?;

This waits until the destination actor's recv call returns, not merely until the message is enqueued. If the timeout expires, the processing outcome is unknown: the actor may still be processing the message. The timeout is capped at 60 seconds.

send remains fire-and-forget with a mailbox-acceptance acknowledgment. Application-defined exchanges may also use listen_remote with a custom RemoteHandler, which can return a RemoteResponse payload.

Custom transports

Applications can implement RemoteTransport to use their own communication system. A transport must support both send and listen, authenticate peers before dispatch, enforce the protocol frame-size limit, and preserve request correlation IDs. Use ActorSystem::connect_remote_with and ActorSystem::listen_remote_with to install it; inbound requests use the same handler interface as the built-in WebSocket transport.

Protocol and security boundaries

Protocol version 3 uses JSON messages and limits frames to 1 MiB. It adds processed-delivery acknowledgments and is not wire-compatible with v2 peers.
Peer bearer tokens are sent in the HTTP upgrade Authorization header, not in URLs, and are redacted from Debug output.
Use wss:// or a private network for transport confidentiality and integrity. When a proxy terminates TLS, trust and secure the proxy-to-Picard link.
Allow-listed peers may address any actor registered for remote delivery. Authorization by actor path or message type is not yet configurable.
A disconnect can leave the caller uncertain whether the remote mailbox accepted a message. The protocol does not promise exactly-once delivery.
The built-in transport opens a WebSocket connection per send. Connection pooling, retries, discovery, and remote peer lifecycle management are future work.

WebSockets are a pragmatic fit when deployments already use HTTP reverse proxies for TLS termination. Compared with a raw TCP protocol, they provide standard proxy traversal and HTTP-level peer authentication headers, at the cost of an HTTP upgrade and WebSocket framing. The proxy is part of the trusted computing base; Picard's bearer-token check remains mandatory regardless.

Persistence and CQRS

Picard does not provide a built-in event store or persistent actor lifecycle. Actor state lives in memory; when an actor is restarted or recreated, the application is responsible for restoring any state that must survive. This can be done by loading state from an application-owned database or by implementing an event-sourcing design in the application.

CQRS entity management

The separate picard-cqrs crate provides an entity manager that routes commands by entity ID and starts an actor for an entity when needed. After an entity actor has been idle for the configured interval, the manager stops it; a later command creates a fresh actor instance. The crate does not store or replay events, and the application must arrange any required state recovery.

The default idle interval is 120 seconds. Configure it under [cqrs] in picard.toml:

[cqrs]
sleep_after_secs = 120

See the picard-cqrs README for an entity manager example. For messaging and actor lifecycle, see Actors and Message Types.

Message Scheduling

Picard's timer module provides scheduling features that allow messages to be sent after a given duration or at a specific time. Timer methods are exposed on both ActorSystem and Context.

One-time scheduling

There are two methods that provide one-time scheduling:

schedule_once schedules a message to be sent after a given delay.
schedule_at_time schedules a message to be sent at a given specific time.

Examples:

#![allow(unused)]
fn main() {
let delay = Duration::from_secs(1);
let actor = ctx.actor_of::<MyActor>("my-actor").unwrap();

ctx.schedule_once(delay,
                actor,
                None,
                "that's one small step for man".into());
}

Here a message is scheduled to be sent to an actor after 20 seconds.

#![allow(unused)]
fn main() {
let time = SystemTime::now();
let actor = ctx.actor_of::<MyActor>("my-actor").unwrap();

ctx.schedule_at_time(time,
                    actor,
                    None,
                    "one giant leap for mankind".into());
}

Here a message is scheduled to be sent at the specific time time.

Repeat scheduling

Messages can be scheduled to be repeatedly sent at specific intervals:

schedule method schedules a message to be repeatedly sent at a given interval.

Example:

#![allow(unused)]
fn main() {
let delay = Duration::from_millis(100);
let iterv = Duration::from_millis(500);
let actor = ctx.actor_of::<MyActor>("my-actor").unwrap();

ctx.schedule(delay,
            interv,
            actor,
            None,
            "a scheduled msg".into());
}

Here a message is scheduled to be repeated every 500 milliseconds. There is also a 100 millisecond initial delay, i.e. the duration before repeating of the message begins.

Note: Picard's default timer module is not persistent meaning that any scheduling is lost when an application is stopped. It's optimized for short dated durations from a few milliseconds to 48 hours, or your average time between deployments.

Cancelling

When scheduling a message a schedule ID is returned which can be used at a later time to cancel the schedule.

Example:

#![allow(unused)]
fn main() {
let id = ctx.schedule(delay,
            interv,
            actor,
            None,
            "a scheduled msg".into());

ctx.cancel_schedule(id);
}

Canceling a schedule removes it from the timer and the message will no longer be sent.

Some example use cases of message scheduling include:

To wait for a specific time for other actors to provide input, such as bidding systems
As part of workflow to provide default messages after a timeout interval
To periodically wake up an actor to check a resource, such as a queue, IO or sensor
To periodically publish or broadcast messages to other actors

Message scheduling is a core feature of concurrent systems and can drive applications to complete their objectives.

We've covered the basics of the Picard Framework. Other topics include:

Configuration

Running Futures

Logging

Configuration

Basics

Picard makes configuration straightforward by using two config files:

picard.toml can be modified to change the behavior of Picard and various modules
app.*, such as app.toml or app.yml for application specific settings

Configuration is loaded with the config crate, so any format it supports can be used.

When the system starts picard.toml is loaded and if an application config file exists it will be merged and deserialized into a single typed PicardConfig. This keeps settings separate for the purposes of maintenance and CI/CD, but makes it uniformly available at runtime.

The PicardConfig is accessible via the ActorSystem instance. Framework settings are typed fields (e.g. sys.config().mailbox.msg_process_limit), and application settings from the [app] section are in app:

#![allow(unused)]
fn main() {
let myval = sys.config().app["myval"].clone().into_string().unwrap();
}

This expects the following entry in app.*:

[app]
myval = "five by five"

File Paths

By default Picard looks for picard.toml and app.* in the config directory relative to the current directory of execution.

It's standard practice to place the config directory in the root directory of your Rust application, i.e. on the same level as src. This way, cargo run and cargo test will use the files located in config.

Configuration file paths can be overridden using environment variables:

PICARD_CONF to override picard.toml file path
APP_CONF to override app.* file path

Best Practises

Default Values

Framework settings missing from the config files fall back to the defaults of PicardConfig. Invalid values (e.g. a zero pool size) are rejected when the system starts. Use ActorSystem::with_config with a PicardConfig to bypass file loading.

Using .config()

It is highly encouraged in terms of function signatures to not pass around instances of PicardConfig. Not only is it slower but it makes understanding what a function does difficult. It is always better to extract values from configuration early on in your application and then pass those to functions. If you have many settings to pass to a function a dedicated settings struct could be used.

Application Settings

PicardConfig accepts framework settings such as [log], [mailbox], [dispatcher], and [cqrs]. Put application-specific settings under [app]; they are available through sys.config().app and are not interpreted by Picard. For example:

[app.storage]
url = "redis://127.0.0.1"

The application is responsible for reading and applying its own settings.

Logging

Picard provides logging out of the box which is built on top of the Log crate logging facade. The logger module starts during the actor system's start, making log macros such as info!, debug!, error! available immediately after ActorSystem::new:

#![allow(unused)]
fn main() {
let sys = ActorSystem::new().unwrap();
info!("My first log message!");
}

The only requirement is importing the Log crate into your application.

Base log level

The base log level is the level at which the Log crate's macros determine whether to ignore a message or forward it to the logger module. This is configured in picard.toml:

[log]
level = "debug"

Default logger

Log messages will be routed to the logging module configured in the model. The rest of this page refers primarily to the default Picard logger, the picard-logger crate.

Since the default logger is an actor log messages are handled concurrently, in a non-blocking manner. This means the impact of logging on an application's response time is reduced.

The default logger has the following features:

Log entry format is configurable using named parameters
Date and time formats can be configured
Log entries can be further filtered at the module level
An optional filter can be configured to omit logs from certain modules

Let's take a look at an example log configuration in picard.toml:

[log]
# max level to log
level = "debug"

# Uncomment this to enable filters on the logger.  The {module} field
# of every log line will be checked, and if the {module} field contains
# any item in this list, the entire log line will be omitted from the
# logging output.
#
# This example will omit any logging output from any module with
# "test" in the name and any module whose name contains "debug".
#
filter = [ "test", "debug" ]

# log format to use
# correlates to format!(log_format, date=, time=, level=, module=, body=);
# since named parameters are used the order of the fields is flexible
# the formatting of each field can be changed also
# e.g. to completely hide a field: {module:.0}
# See: https://doc.rust-lang.org/std/fmt/#syntax

# {date}    the calendar day
# {time}    the calendar time
# {level}   the level for the entry
# {module}  the module path originating the entry
# {body}    the message body
log_format = "{date} {time} {level} [{module}] {body}"
date_format = "%Y-%m-%d"
time_format = "%H:%M:%S%:z"

This configuration will produce log entries formatted like:

2018-06-11 08:31:58+00:00 DEBUG [picard::system::system] Actor system started
2018-06-11 08:31:58+00:00 DEBUG [main] My first log message!

and the log lines which would have otherwise been printed:

2018-06-11 08:31:58+00:00 DEBUG [testSystem] A test line which should be filtered out!
2018-06-11 08:31:58+00:00 DEBUG [debug] A debugger module's log message

will be omitted from the log output.

Diagnostics

Picard has optional Cargo features for observing a running system. They are off by default and add no code when disabled. The runtime console provides authenticated WebSocket inspection and management; tracing and stats are described below.

[dependencies]
picard = { version = "0.4", features = ["tracing", "stats"] }

Tracing

The tracing feature emits tracing spans and events. Install any subscriber, for example tracing-subscriber, to collect them:

#![allow(unused)]
fn main() {
tracing_subscriber::fmt().with_max_level(tracing::Level::DEBUG).init();
let sys = ActorSystem::new().unwrap();
}

Events have separate targets so subscriber filters can silence frequent lifecycle and message diagnostics independently of supervision events. For example, with an EnvFilter-enabled subscriber, RUST_LOG=picard::actor::lifecycle=warn,picard::messaging=warn,picard::actor::supervision=info keeps supervision warnings while reducing routine event volume.

Level	Target	Kind	Name / message	Fields
`INFO`	`picard::system`	event	`actor system started`, `actor system shutting down`	`system`, `id`
`DEBUG`	`picard::actor::lifecycle`	event	`actor created`	`actor`, `actor_type`
`DEBUG`	`picard::actor::lifecycle`	event	`actor terminated`	`actor`
`INFO`	`picard::actor::lifecycle`	event	`actor restarted`	`actor`
`WARN`	`picard::actor::lifecycle`	event	`actor failed to restart`	`actor`
`ERROR`	`picard::actor::supervision`	event	`actor panicked; reporting failure to parent`	`actor`
`WARN`	`picard::actor::supervision`	event	`supervising failed child`	`supervisor`, `actor`, `strategy`
`DEBUG`	`picard::messaging`	event	`message could not be delivered...`	`actor`, `msg_type` (typed sends)
`TRACE`	`picard::messaging`	span	`recv`, entered while an actor handles a message	`actor`, `msg_type`

Events raised inside an actor's recv (including your own tracing calls) are nested in its recv span. The recv span is at TRACE level, so filter it out in production if its per-message cost matters.

The tracing feature is independent of Picard's slog-based logger; both can be used at the same time.

Runtime statistics

The stats feature keeps cumulative counters and exposes them through ActorSystem::stats():

#![allow(unused)]
fn main() {
let stats = sys.stats();
println!(
    "live actors: {}, messages sent: {}, failed: {}",
    stats.live_actors(),
    stats.messages_sent,
    stats.messages_failed,
);
}

RuntimeStats contains:

Field	Counts
`actors_created`	Actors whose instance was produced and kernel started
`actors_terminated`	Actors that were stopped and removed
`actors_restarted`	Restarts after a failure (not counted as creations)
`messages_sent`	Messages enqueued in a user mailbox
`messages_failed`	Messages (user or system) that could not be enqueued, e.g. the recipient had stopped
`system_messages_sent`	Lifecycle commands, events and failure notifications enqueued

live_actors() returns actors_created - actors_terminated.

Counts include Picard's own actors (root, guardians, system channels and the dead letter logger) and messages sent to them, such as channel publishes. Each field is read separately, so a snapshot of a busy system is not atomic across fields. Counters are relaxed atomics on separate cache lines; each send costs one extra atomic increment.

Runtime console

Enable console to inspect and manage a native Picard application over a WebSocket connection. The feature alone does not open a port. It is independent of stats and tracing.

[dependencies]
picard = { version = "0.4", features = ["console"] }

Also enable serde to send JSON messages and inspect their JSON payloads:

picard = { version = "0.4", features = ["console", "serde"] }

Starting an endpoint

use picard::actors::*;
use picard::console::ConsoleOptions;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let sys = ActorSystem::new()?;
    let console = sys.start_console(ConsoleOptions {
        bind: "127.0.0.1:9000".parse()?,
        token: std::env::var("PICARD_CONSOLE_TOKEN")?,
    })?;
    println!("Console listening at ws://{}/console", console.local_addr());

    // Create actors and run the application here, keeping `console` alive.

    console.stop()?;
    futures::executor::block_on(sys.shutdown())?;
    Ok(())
}

Use port 0 to request an available port, then read it with local_addr(). Only loopback addresses are accepted. The token must contain at least 32 ASCII letters, digits, - or _; generate an unpredictable token in your application or deployment environment. Do not put it in source control.

The handle owns the listener and client threads. stop() reports shutdown errors; dropping the handle also stops and joins them, logging failures. Stopping the actor system does not automatically drop the console handle. There is no console listener or terminal client on WebAssembly.

The endpoint exposes application internals and destructive controls. Treat the token as an administrative credential. Remote use should go through a secure tunnel to the loopback port; the endpoint does not provide TLS.

Custom transports

Applications that already run a server can expose the console through their own endpoint instead of calling start_console. ActorSystem::console() returns a cloneable, transport-agnostic Console service that executes protocol requests:

#![allow(unused)]
fn main() {
use picard::{actors::ActorSystem, console::Console};

fn console_service(sys: &ActorSystem) -> Console {
    sys.console()
}

fn on_console_text(console: &Console, text: &str) -> String {
    // `text` comes from an authenticated client of your own server.
    console.handle_text(text)
}
}

handle_text(&str) -> String parses one JSON request and returns the JSON response. Malformed requests and requests larger than MAX_REQUEST_BYTES (64 KiB) produce invalid_request; responses above MAX_RESPONSE_BYTES are replaced by response_too_large.
handle(Request) -> Response is the typed equivalent. Response::to_json applies the same response size limit.
The service is synchronous and may run user serializers, so async servers should call it from a blocking task (for example tokio::task::spawn_blocking).

The service performs no authentication. Your transport must authenticate clients before routing requests, cap request sizes, and limit concurrent clients. console::is_valid_token checks a bearer token against the built-in endpoint's policy. To stay compatible with picard-console, serve WebSocket JSON text frames, require Authorization: Bearer <token>, and reject browser upgrades that carry an Origin header. Living City's --console-route (in demo/src/server.rs) is an axum example.

Terminal client

The picard-console workspace crate provides a terminal client:

cargo run -p picard-console -- ws://127.0.0.1:9000/console

Set PICARD_CONSOLE_TOKEN in the client's environment to the same token used by the server. The client does not accept tokens as command-line arguments or URL parameters.

When stdin and stdout are terminals, the client opens a full-screen TUI. Pass --repl for the line-oriented REPL, which is also used automatically when input or output is redirected (for example in scripts).

TUI

The TUI shows the actor tree with queue lengths and mailbox state, the selected actor's details, and its recent message events (newest first). It refreshes every second. The tree starts collapsed below the root; filtering shows all matches.

Registered peers appear as collapsed @peer nodes. Expanding one fetches that peer's actors through the connected actor; they behave like local rows (details, messages, pause, clear, restart, stop, and s, which prefills @peer send PATH ). The peer's root / is hidden.

Up/Down, j/k   Select an actor (PgUp/PgDn, Home/End jump)
Right/l, Left/h  Expand / collapse the actor's children (Left on a leaf selects its parent)
/              Filter actors by path (Enter keeps, Esc clears)
:              Enter a command (Tab completes, Up/Down history)
p              Pause or resume the selected actor's mailbox
c              Clear the paused mailbox (asks for confirmation)
s              Send a JSON message to the selected actor
r              Restart the selected actor (asks for confirmation)
x              Stop the selected actor (asks for confirmation)
F5, Ctrl-L     Refresh now
?              Toggle help
q, Ctrl-C      Quit

s opens the command bar prefilled with send PATH ; type the JSON payload and press Enter. The command bar accepts the REPL commands below. Its history is kept in memory only and never includes send commands. Colors follow NO_COLOR.

REPL

Commands:

actors
inspect /user/worker
messages /user/worker
pause /user/worker
clear /user/worker
resume /user/worker
restart /user/worker
send /user/worker {"Greet":{"name":"Picard"}}
peers
send @district-1 /user/supervisor {"SetVehicles":100}
stop /user/worker
help
quit

Tab completes commands and actor paths. Up/Down recall history, Ctrl-C cancels input, and Ctrl-D disconnects. Prompts, hints, results, and errors use terminal colors; NO_COLOR disables them, and redirected output is uncolored.

History is stored at $XDG_STATE_HOME/picard-console/history, or $HOME/.local/state/picard-console/history. send commands are never saved because payloads may be sensitive.

Errors

Connection and protocol errors are explicit; there is no automatic reconnect or retry of destructive operations. If no endpoint is listening, the client reports Unable to connect with a hint: check that the application was built with console and that it started an endpoint (the feature alone opens no port).

Living City

The Living City demo forwards its console and serde features to Picard. Start its optional endpoint with --console-listen 127.0.0.1:9000 and set PICARD_CONSOLE_TOKEN in the environment:

cargo run -p living-city --release --features console,serde -- \
  --console-listen 127.0.0.1:9000

The same option works with loadtest. Alternatively, --console-route serves the console at /console on the demo's own web server (loopback --host only):

cargo run -p living-city --release --features console,serde -- --console-route
cargo run -p picard-console -- ws://127.0.0.1:8981/console

See the Living City guide for Clap environment options and its JSON message compatibility.

Inspection and management semantics

actors lists registered actors, including runtime actors. Each entry has its absolute path, parent path, Rust message type, JSON-send capability, management capability, and mailbox state. The client follows actor-list pages automatically and refreshes path completion before each prompt (REPL) or on every refresh (TUI).
inspect returns one actor's metadata, queued user/system message counts, and paused, suspended, and scheduled flags. Snapshots are best-effort views of a concurrent runtime, not an atomic snapshot of the whole application.
messages returns at most 64 recent user-message events per actor: enqueued, received, cleared, and delivery_failed. Each event includes the message type and optional sender path. This is a bounded history, not a nondestructive copy of the current queue. received means dequeued for processing, not that the handler successfully completed.
With serde, enqueue/receive events include a JSON message payload for actors created through APIs with a Message bound. Captured payloads are limited to 4096 serialized bytes. Oversized or unserializable payloads produce payload_error metadata without preventing normal delivery. Clear events contain metadata only. Runtime message types need not implement serde.
pause prevents further user-message dequeues. An already dequeued message can still finish. System messages continue, so supervision, restart, stop, and system shutdown remain available. Pausing through the console is separate from failure-induced suspension and persists across restarts.
resume removes the console pause and wakes the actor. It does not bypass suspension caused by actor failure.
clear requires a paused mailbox. It discards its queued user messages, returning a count; it does not clear system messages or an in-flight message. Explicitly cleared messages are not sent to dead letters.
stop and restart enqueue normal lifecycle commands; accepted: true means the command was queued, not that the lifecycle operation completed. Normal descendant termination and supervision semantics apply.
Runtime actors created through internal/local APIs (guardians, system channels, and the dead-letter logger) are inspectable but cannot be managed. Actors created through public user, child, system, or temporary actor factories can be managed. An actor disappears from the registry on termination.

With console disabled, the registry, message history, and mailbox control locks are not compiled. Enabling it adds per-message inspection overhead and bounded history storage even before an endpoint is started. JSON serialization adds further overhead when both console and serde are enabled.

Sending to remote actors

send @PEER PATH JSON forwards the JSON message to the actor at PATH on a remote system; peers lists the available @PEER names. The console only reaches peers the application registers with ActorSystem::register_console_peer(name, sender), where sender hands the message to your remoting layer (for example a RemoteConnection). The remote actor must be registered for remote delivery, and the response only says the message was queued, not that the actor accepted it. In the wire protocol this is {"op":"send","peer":"district-1","path":"/user/gateway","message":{...}} and {"op":"peers"}. The Living City demo registers every --connect peer as district-N.

Querying and managing remote actors

@PEER COMMAND runs any inspection or management command (actors, inspect, messages, pause, resume, clear, stop, restart) on the console of a remote system, through the system you are connected to: @district-1 inspect /user/supervisor. The application registers the peer with ActorSystem::register_console_peer_query(name, query), where query carries a request as JSON text to the remote system and returns the text of Console::handle_text there. Forwarded requests cannot be forwarded again, and the remote console applies its own rules (for example, runtime actors stay protected). The wire form is {"op":"forward","peer":"district-1","command":{"op":"inspect","path":"/user/x"}}. A forwarded actors returns one page (the client paginates). A forwarded send runs on the remote console with its own typed decoder. Whoever holds a peer's shared token can use this path, so it grants the same trust as sending messages to that peer.

Typed JSON sends

send looks up the target actor's registered decoder and deserializes the JSON into that actor's own message type, then uses the normal message-send path. Messages do not need to implement Clone. The JSON representation follows the message type's serde derives or custom implementation.

For example, an enum declared as:

#![allow(unused)]
fn main() {
#[derive(Debug, serde::Serialize, serde::Deserialize)]
enum WorkerMsg {
    Greet { name: String },
    Count(u32),
}
}

accepts {"Greet":{"name":"Picard"}} or {"Count":42}. A u32 actor accepts 42, and a String actor accepts "hello". Messages queued while paused are processed only after resume. A missing actor, unavailable decoder, invalid JSON message type, or failed delivery produces an error response. serde is not required merely to inspect or manage actors.

WebSocket protocol

Connect to ws://HOST:PORT/console with the HTTP upgrade header:

Authorization: Bearer TOKEN

The listener accepts at most 16 concurrent clients. Upgrade authentication has a two-second read timeout. Requests must be text frames containing JSON and are limited to 64 KiB. Responses are limited to one MiB. Binary requests receive an error. Oversized WebSocket messages disconnect the client. Ping/pong and close frames follow the normal WebSocket protocol.

Unknown paths, query parameters, missing/incorrect credentials, and upgrade requests with an Origin header are rejected. This is a native administrative client protocol, not a directly browser-accessible API.

Every request has an unsigned integer id and an op:

{"id":1,"op":"actors","offset":0,"limit":100}
{"id":2,"op":"inspect","path":"/user/worker"}
{"id":3,"op":"send","path":"/user/worker","message":{"Count":42}}

Supported operations are actors, inspect, messages, pause, resume, clear, stop, restart, send, forward, and peers. All except actors and peers require path; send additionally requires message and accepts an optional peer.

actors defaults to offset: 0 and limit: 100. Limits must be in 1..=100. Its result contains actors, total, and next_offset (null for the final page). Actor creation/termination during paging can change the result.

Responses echo the request ID and contain exactly one of result and error:

{"id":3,"result":{"accepted":true}}
{"id":4,"error":{"code":"invalid_message","message":"..."}}

Malformed requests use id: null if a valid ID cannot be recovered. Error codes include invalid_request, not_found, protected_actor, not_paused, json_unavailable, invalid_message, delivery_failed, and response_too_large.

Running Futures

Picard can execute and drive futures to completion. Internally, actors are also executed as futures by the dispatcher. ActorSystem and Context both provide a run method.

On native targets, Picard uses a futures thread pool. On WebAssembly it dispatches tasks to the browser event loop instead. This example shows the native API:

use futures::executor::block_on;
use picard::actors::*;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let system = ActorSystem::new()?;
    let handle = system.run(async { "someval".to_string() })?;
    assert_eq!(block_on(handle), "someval");
    Ok(())
}

run returns a futures::future::RemoteHandle for the future's output, or a SpawnError if the executor cannot accept the task.

In the next section we'll see how to to test Picard applications.

Testing

Testing concurrent systems can be more difficult than single threaded applications, since the test itself and the application are running on separate threads. Moreover, because of Picard's resilient self-healing approach where panics are isolated, handled and the failed component restarted, detecting failures in tests proves challenging.

To help make testing easier, the picard-testkit introduces a 'probe' that can be sent to actors either through messaging or as part of an actor's Props. Probes can then emit values back to the test thread.

Here's an example of testing an actor restart:

#![allow(unused)]
fn main() {
use picard::actors::*;

use picard_testkit::probe::{Probe, ProbeReceive};
use picard_testkit::probe::channel::{probe, ChannelProbe};

#[derive(Clone, Debug)]
pub struct Add;

#[derive(Clone, Debug)]
pub struct TestProbe(ChannelProbe<(), ()>);

#[actor(TestProbe, Add)]
struct Counter {
    probe: Option<TestProbe>,
    count: u32,
}

impl Actor for Counter {
    // we used the #[actor] attribute so CounterMsg is the Msg type
    type Msg = CounterMsg;

    fn recv(&mut self,
                ctx: &Context<Self::Msg>,
                msg: Self::Msg,
                sender: Sender) {
        self.receive(ctx, msg, sender);
    }
}

impl Receive<TestProbe> for Counter {
    type Msg = CounterMsg;

    fn receive(&mut self,
                _ctx: &Context<Self::Msg>,
                msg: TestProbe,
                _sender: Sender) {
        self.probe = Some(msg)
    }
}

impl Receive<Add> for Counter {
    type Msg = CounterMsg;

    fn receive(&mut self,
                _ctx: &Context<Self::Msg>,
                _msg: Add,
                _sender: Sender) {
        self.count += 1;
        if self.count == 1_000_000 {
            self.probe.as_ref().unwrap().0.event(())
        }
    }
}

#[test]
fn actor_tell() {
    let sys = ActorSystem::new().unwrap();

    let actor = sys.actor_of::<Counter>("counter-1").unwrap();

    let (probe, listen) = probe();
    actor.tell(TestProbe(probe), None);

    for _ in 0..1_000_000 {
        actor.tell(Add, None);
    }

    p_assert_eq!(listen, ());
}
}

This test sends a test probe to the test actor, which is keeps and uses to signal back after one million test messages were received. The macro p_assert_eq! waits (blocks) on the listener until a value is received from the probe.

Patterns

Ask

The Ask pattern allows values to be sent by actors to outside of the actor system. The value is delivered as a Future.

Let's look at how this works:

use picard_patterns::ask;

struct MyActor;

impl Actor for MyActor {
    type Msg = u32;

    fn recv(&mut self,
                ctx: &Context<Self::Msg>,
                msg: Self::Msg,
                sender: Sender) {

        // sender is the Ask, waiting for a message to be sent back to it
        sender.as_ref()
                .unwrap()
                .try_tell(msg * 2, Some(ctx.myself().into()));
    }
}

fn main() {
    let sys = ActorSystem::new().unwrap();

    let props = Props::produce::<MyActor>();
    let my_actor = sys.actor_of_props(props, "my-actor");

    // ask returns a future that automatically is driven
    // to completion by the system.
    let res: RemoteHandle<String> = ask(&sys, &my_actor, 100);

    // the result future can be passed to a library or fuction that
    // expects a future, or it can be extracted locally using `block_on`.
    let res = block_on(res);

    println!("The result value is: {}", res);
}

In the background Ask sets up a temporary intermediate actor that lives for the lifetime of the ask. Other actors see this temporary actor as the sender and can send a message back to it. When the temporary ask actor receives a message it fulfills the outstanding future and performs a stop on itself to cleanup.

Ask is particularly useful when you have part of an application that runs outside of the actor system, or in another actor system, such as a web server (e.g. Hyper) serving API requests. The resulting future can then be chained as part of the future stack.

What is Picard's goal?

Picard's principle goal is to be a framework to make concurrent, resilient software development easy and to make the Actor Model applicable to a wide spectrum of platforms.

Picard achieves this by:

Exposing a simple Actor trait
Actor supervision based on the actor hierarchy
Allowing Actors to own their state so Sync is not required
Providing mailbox delivery within a running process (not durable or exactly-once delivery)
Providing a configurable, modular system
Optional CQRS entity management through the separate picard-cqrs crate
Low memory footprint
Fast execution
No unsafe code (with the exception of impl Send and impl Sync)

Is Picard a remake of Akka for Rust?

Picard is inspired by Akka, the JVM's popular Actor Model implementation. However it's not Picard's goal to recreate Akka for Rust. Other environments have done this, for example C#'s Akka.NET but the differences of Rust and Scala/Java and others require a fresh Rust-specific approach. And since Picard works well on resource limited hardware such as single board computers there are new challenges and problems to solve that are not applicable to Akka.

How do actors compare to Futures?

Both Actors and Futures in general work to solve the challenges of concurrent software design. However they are complementary since they are solving different issues.

Futures allow software developers to write concurrent code in an almost single-threaded manner. Simple, short-lived workload processing that is non-blocking can be easily written. So Futures can be seen as building blocks for non-blocking application design.

Actors on the other hand are used when state exists and must be maintained between cycles of when an actor is running and when it is idle. Actors are well suited for long running processes that can even span the lifetime of the application itself. Actors also form a hierarchy, with parent and child actors, which communicate with each other via a message based API. Since Actors are aware of other Actors in the system they can heal each other when unexpected things happen, such as panics.

Under the covers, Picard actors are futures. Native builds use a thread pool; WebAssembly builds dispatch tasks on the browser event loop.

How do actors compare to threads?

Actors are scheduled when they have messages to process. On native targets, the dispatcher runs actor tasks on its thread pool; idle actors do not occupy a thread. The browser event loop is used on WebAssembly.

It's possible to have thousands or millions of actors that share just two CPU core/threads.

Can Picard communicate across hosts?

The optional remoting feature provides authenticated WebSocket message delivery between Picard systems. It is not transparent clustering: there is no discovery, failover, remote actor-reference conversion, or cross-host coordination to ensure one active actor per entity. The separate picard-cqrs crate manages entities within one actor system; applications that distribute entity ownership must provide that coordination themselves.

Can multiple ActorSystem instances run in the same app?

It is possible to create two or more instances using ActorSystem::new but there is little benefit in doing so. A single instance is already optimized to run actors on the configured dispatcher which generally uses all cores of a CPU.

An exception to this advice is when an application is handling greatly varying workloads and it is not possible due to architecture constraints to break into multiple applications.

Does Picard use unsafe code?

The Picard runtime crates currently contain no unsafe blocks or unsafe trait implementations. This describes the project source, not its dependencies.

Are there any benchmarks?

The repository includes Criterion benchmarks for message throughput, actor round-trip latency, actor creation, mailbox workloads, scheduling, and other scenarios. Run them with cargo bench -p picard --bench runtime or cargo bench -p picard --bench actor_suite. Results depend on hardware and workload; benchmark numbers should not be treated as a general throughput guarantee.