Oliver
Kowalke
2013
Oliver Kowalke

Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

C++ Library to cooperatively schedule and synchronize micro-threads

# Fiber

# Overview

Boost.Fiber provides a framework for micro-/userland-threads (fibers) scheduled cooperatively. The API contains classes and functions to manage and synchronize fibers similiarly to standard thread support library.

Each fiber has its own stack.

A fiber can save the current execution state, including all registers and CPU flags, the instruction pointer, and the stack pointer and later restore this state. The idea is to have multiple execution paths running on a single thread using cooperative scheduling (versus threads, which are preemptively scheduled). The running fiber decides explicitly when it should yield to allow another fiber to run (context switching). Boost.Fiber internally uses call/cc from Boost.Context; the classes in this library manage, schedule and, when needed, synchronize those execution contexts. A context switch between threads usually costs thousands of CPU cycles on x86, compared to a fiber switch with less than a hundred cycles. A fiber runs on a single thread at any point in time.

In order to use the classes and functions described here, you can either include the specific headers specified by the descriptions of each class or function, or include the master library header:

    #include <boost/fiber/all.hpp>

which includes all the other headers in turn.

The namespaces used are:

    namespace boost::fibers
    namespace boost::this_fiber

Fibers and Threads

Control is cooperatively passed between fibers launched on a given thread. At a given moment, on a given thread, at most one fiber is running.

Spawning additional fibers on a given thread does not distribute your program across more hardware cores, though it can make more effective use of the core on which it's running.

On the other hand, a fiber may safely access any resource exclusively owned by its parent thread without explicitly needing to defend that resource against concurrent access by other fibers on the same thread. You are already guaranteed that no other fiber on that thread is concurrently touching that resource. This can be particularly important when introducing concurrency in legacy code. You can safely spawn fibers running old code, using asynchronous I/O to interleave execution.

In effect, fibers provide a natural way to organize concurrent code based on asynchronous I/O. Instead of chaining together completion handlers, code running on a fiber can make what looks like a normal blocking function call. That call can cheaply suspend the calling fiber, allowing other fibers on the same thread to run. When the operation has completed, the suspended fiber resumes, without having to explicitly save or restore its state. Its local stack variables persist across the call.

A fiber can be migrated from one thread to another, though the library does not do this by default. It is possible for you to supply a custom scheduler that migrates fibers between threads. You may specify custom fiber properties to help your scheduler decide which fibers are permitted to migrate. Please see Migrating fibers between threads and Customization for more details.

A fiber launched on a particular thread continues running on that thread unless migrated. It might be unblocked (see Blocking below) by some other thread, but that only transitions the fiber from blocked to ready on its current thread — it does not cause the fiber to resume on the thread that unblocked it.

thread-local storage

Unless migrated, a fiber may access thread-local storage; however that storage will be shared among all fibers running on the same thread. For fiber-local storage, please see `fiber_specific_ptr`.

BOOST_FIBERS_NO_ATOMICS

The fiber synchronization objects provided by this library will, by default, safely synchronize fibers running on different threads. However, this level of synchronization can be removed (for performance) by building the library with `BOOST_FIBERS_NO_ATOMICS` defined. When the library is built with that macro, you must ensure that all the fibers referencing a particular synchronization object are running in the same thread. Please see Synchronization.

Blocking

Normally, when this documentation states that a particular fiber blocks (or equivalently, suspends), it means that it yields control, allowing other fibers on the same thread to run. The synchronization mechanisms provided by Boost.Fiber have this behavior.

A fiber may, of course, use normal thread synchronization mechanisms; however a fiber that invokes any of these mechanisms will block its entire thread, preventing any other fiber from running on that thread in the meantime. For instance, when a fiber wants to wait for a value from another fiber in the same thread, using `std::future` would be unfortunate: `std::future::get()` would block the whole thread, preventing the other fiber from delivering its value. Use `future<>` instead.

Similarly, a fiber that invokes a normal blocking I/O operation will block its entire thread. Fiber authors are encouraged to consistently use asynchronous I/O. Boost.Asio and other asynchronous I/O operations can straightforwardly be adapted for Boost.Fiber: see Integrating Fibers with Asynchronous Callbacks.

Boost.Fiber depends upon Boost.Context. Boost version 1.61.0 or greater is required.

This library requires C++11!

## Implementations: fcontext_t, ucontext_t and WinFiber

Boost.Fiber uses call/cc from Boost.Context as building-block.

fcontext_t

The implementation uses `fcontext_t` per default. fcontext_t is based on assembler and not available for all platforms. It provides a much better performance than `ucontext_t` (the context switch takes two magnitudes of order less CPU cycles; see section performance) and `WinFiber`.

ucontext_t

As an alternative, `ucontext_t` can be used by compiling with `BOOST_USE_UCONTEXT` and b2 property `context-impl=ucontext`. `ucontext_t` might be available on a broader range of POSIX-platforms but has some disadvantages (for instance deprecated since POSIX.1-2003, not C99 conform).

call/cc supports Segmented stacks only with `ucontext_t` as its implementation.

WinFiber

With `BOOST_USE_WINFIB` and b2 property `context-impl=winfib` Win32-Fibers are used as implementation for call/cc.

Because the TIB (thread information block) is not fully described in the MSDN, it might be possible that not all required TIB-parts are swapped.

The first call of call/cc converts the thread into a Windows fiber by invoking `ConvertThreadToFiber()`. If desired, `ConvertFiberToThread()` has to be called by the user explicitly in order to release resources allocated by `ConvertThreadToFiber()` (e.g. after using boost.context).

# Fiber management

Synopsis

    #include <boost/fiber/all.hpp>

    namespace boost {
    namespace fibers {

    class fiber;
    bool operator<( fiber const& l, fiber const& r) noexcept;
    void swap( fiber & l, fiber & r) noexcept;

    template< typename SchedAlgo, typename ... Args >
    void use_scheduling_algorithm( Args && ... args);
    bool has_ready_fibers();

    namespace algo {

    struct algorithm;
    template< typename PROPS >
    struct algorithm_with_properties;
    class round_robin;
    class shared_round_robin;

    }

    namespace this_fiber {

    fibers::id get_id() noexcept;
    void yield();
    template< typename Clock, typename Duration >
    void sleep_until( std::chrono::time_point< Clock, Duration > const& abs_time)
    template< typename Rep, typename Period >
    void sleep_for( std::chrono::duration< Rep, Period > const& rel_time);
    template< typename PROPS >
    PROPS & properties();

    }}

Tutorial

Each `fiber` represents a micro-thread which will be launched and managed cooperatively by a scheduler. Objects of type `fiber` are move-only.

    boost::fibers::fiber f1; // not-a-fiber

    void f() {
        boost::fibers::fiber f2( some_fn);

        f1 = std::move( f2); // f2 moved to f1
    }

Launching

A new fiber is launched by passing an object of a callable type that can be invoked with no parameters. If the object must not be copied or moved, then std::ref can be used to pass in a reference to the function object. In this case, the user must ensure that the referenced object outlives the newly-created fiber.

    struct callable {
        void operator()();
    };

    boost::fibers::fiber copies_are_safe() {
        callable x;
        return boost::fibers::fiber( x);
    } // x is destroyed, but the newly-created fiber has a copy, so this is OK

    boost::fibers::fiber oops() {
        callable x;
        return boost::fibers::fiber( std::ref( x) );
    } // x is destroyed, but the newly-created fiber still has a reference
      // this leads to undefined behaviour

The spawned `fiber` does not immediately start running. It is enqueued in the list of ready-to-run fibers, and will run when the scheduler gets around to it.

Exceptions

An exception escaping from the function or callable object passed to the `fiber` constructor calls `std::terminate()`. If you need to know which exception was thrown, use `future<>` or `packaged_task<>`.

Detaching

A `fiber` can be detached by explicitly invoking the `fiber::detach()` member function. After `fiber::detach()` is called on a fiber object, that object represents not-a-fiber. The fiber object may then safely be destroyed.

    boost::fibers::fiber( some_fn).detach();

Boost.Fiber provides a number of ways to wait for a running fiber to complete. You can coordinate even with a detached fiber using a `mutex`, or `condition_variable`, or any of the other synchronization objects provided by the library.

If a detached fiber is still running when the thread’s main fiber terminates, the thread will not shut down.

Joining

In order to wait for a fiber to finish, the `fiber::join()` member function of the `fiber` object can be used. `fiber::join()` will block until the `fiber` object has completed.

    void some_fn() {
        ...
    }

    boost::fibers::fiber f( some_fn);
    ...
    f.join();

If the fiber has already completed, then `fiber::join()` returns immediately and the joined `fiber` object becomes not-a-fiber.

Destruction

When a `fiber` object representing a valid execution context (the fiber is `fiber::joinable()`) is destroyed, the program terminates. If you intend the fiber to outlive the `fiber` object that launched it, use the `fiber::detach()` method.

    {
        boost::fibers::fiber f( some_fn);
    } // std::terminate() will be called

    {
        boost::fibers::fiber f(some_fn);
        f.detach();
    } // okay, program continues

Fiber IDs

Objects of class `fiber::id` can be used to identify fibers. Each running `fiber` has a unique `fiber::id` obtainable from the corresponding `fiber` by calling the `fiber::get_id()` member function. Objects of class `fiber::id` can be copied, and used as keys in associative containers: the full range of comparison operators is provided. They can also be written to an output stream using the stream insertion operator, though the output format is unspecified.

Each instance of `fiber::id` either refers to some fiber, or not-a-fiber. Instances that refer to not-a-fiber compare equal to each other, but not equal to any instances that refer to an actual fiber. The comparison operators on `fiber::id` yield a total order for every non-equal `fiber::id`.

Enumeration `launch`

`launch` specifies whether control passes immediately into a newly-launched fiber.

    enum class launch {
        dispatch,
        post
    };

`dispatch`

##

Effects:

  A fiber launched with `launch == dispatch` is entered immediately. In other words, launching a fiber with `dispatch` suspends the caller (the previously-running fiber) until the fiber scheduler has a chance to resume it later.

`post`

##

Effects:

  A fiber launched with `launch == post` is passed to the fiber scheduler as ready, but it is not yet entered. The caller (the previously-running fiber) continues executing. The newly-launched fiber will be entered when the fiber scheduler has a chance to resume it later.

Note:

  If `launch` is not explicitly specified, `post` is the default.

## Class `fiber`

    #include <boost/fiber/fiber.hpp>

    namespace boost {
    namespace fibers {

    class fiber {
    public:
        class id;

        constexpr fiber() noexcept;

        template< typename Fn, typename ... Args >
        fiber( Fn &&, Args && ...);

        template< typename Fn, typename ... Args >
        fiber( launch, Fn &&, Args && ...);

        template< typename StackAllocator, typename Fn, typename ... Args >
        fiber( std::allocator_arg_t, StackAllocator, Fn &&, Args && ...);

        template< typename StackAllocator, typename Fn, typename ... Args >
        fiber( launch, std::allocator_arg_t, StackAllocator, Fn &&, Args && ...);

        ~fiber();

        fiber( fiber const&) = delete;

        fiber & operator=( fiber const&) = delete;

        fiber( fiber &&) noexcept;

        fiber & operator=( fiber &&) noexcept;

        void swap( fiber &) noexcept;

        bool joinable() const noexcept;

        id get_id() const noexcept;

        void detach();

        void join();

        template< typename PROPS >
        PROPS & properties();
    };

    bool operator<( fiber const&, fiber const&) noexcept;

    void swap( fiber &, fiber &) noexcept;

    template< typename SchedAlgo, typename ... Args >
    void use_scheduling_algorithm( Args && ...) noexcept;

    bool has_ready_fibers() noexcept;

    }}

Default constructor

    constexpr fiber() noexcept;

###

Effects:

  Constructs a `fiber` instance that refers to not-a-fiber.

Postconditions:

  `this->get_id() == fiber::id()`

Throws:

  Nothing

Constructor

    template< typename Fn, typename ... Args >
    fiber( Fn && fn, Args && ... args);

    template< typename Fn, typename ... Args >
    fiber( launch policy, Fn && fn, Args && ... args);

    template< typename StackAllocator, typename Fn, typename ... Args >
    fiber( std::allocator_arg_t, StackAllocator salloc, Fn && fn, Args && ... args);

    template< typename StackAllocator, typename Fn, typename ... Args >
    fiber( launch policy, std::allocator_arg_t, StackAllocator salloc,
           Fn && fn, Args && ... args);

###

Preconditions:

  `Fn` must be copyable or movable.

Effects:

  `fn` is copied or moved into internal storage for access by the new fiber. If `launch` is specified (or defaulted) to `post`, the new fiber is marked ready and will be entered at the next opportunity. If `launch` is specified as `dispatch`, the calling fiber is suspended and the new fiber is entered immediately.

Postconditions:

  `*this` refers to the newly created fiber of execution.

Throws:

  `fiber_error` if an error occurs.

Note:

  `StackAllocator` is required to allocate a stack for the internal __econtext__. If `StackAllocator` is not explicitly passed, the default stack allocator depends on `BOOST_USE_SEGMENTED_STACKS`: if defined, you will get a `segmented_stack`, else a `fixedsize_stack`.

See also:

  `std::allocator_arg_t`, Stack allocation

Move constructor

    fiber( fiber && other) noexcept;

###

Effects:

  Transfers ownership of the fiber managed by `other` to the newly constructed `fiber` instance.

Postconditions:

  `other.get_id() == fiber::id()` and `get_id()` returns the value of `other.get_id()` prior to the construction

Throws:

  Nothing

Move assignment operator

    fiber & operator=( fiber && other) noexcept;

###

Effects:

  Transfers ownership of the fiber managed by `other` (if any) to `*this`.

Postconditions:

  `other->get_id() == fiber::id()` and `get_id()` returns the value of `other.get_id()` prior to the assignment.

Throws:

  Nothing

Destructor

    ~fiber();

###

Effects:

  If the fiber is `fiber::joinable()`, calls std::terminate. Destroys `*this`.

Note:

  The programmer must ensure that the destructor is never executed while the fiber is still `fiber::joinable()`. Even if you know that the fiber has completed, you must still call either `fiber::join()` or `fiber::detach()` before destroying the `fiber` object.

Member function `joinable`()

    bool joinable() const noexcept;

###

Returns:

  `true` if `*this` refers to a fiber of execution, which may or may not have completed; otherwise `false`.

Throws:

  Nothing

Member function `join`()

    void join();

###

Preconditions:

  the fiber is `fiber::joinable()`.

Effects:

  Waits for the referenced fiber of execution to complete.

Postconditions:

  The fiber of execution referenced on entry has completed. `*this` no longer refers to any fiber of execution.

Throws:

  `fiber_error`

Error Conditions:

  resource_deadlock_would_occur: if `this->get_id() == boost::this_fiber::get_id()`. invalid_argument: if the fiber is not `fiber::joinable()`.

Member function `detach`()

    void detach();

###

Preconditions:

  the fiber is `fiber::joinable()`.

Effects:

  The fiber of execution becomes detached, and no longer has an associated `fiber` object.

Postconditions:

  `*this` no longer refers to any fiber of execution.

Throws:

  `fiber_error`

Error Conditions:

  invalid_argument: if the fiber is not `fiber::joinable()`.

Member function `get_id`()

    fiber::id get_id() const noexcept;

###

Returns:

  If `*this` refers to a fiber of execution, an instance of `fiber::id` that represents that fiber. Otherwise returns a default-constructed `fiber::id`.

Throws:

  Nothing

See also:

  `this_fiber::get_id()`

Templated member function `properties`()

    template< typename PROPS >
    PROPS & properties();

###

Preconditions:

  `*this` refers to a fiber of execution. `use_scheduling_algorithm()` has been called from this thread with a subclass of `algorithm_with_properties<>` with the same template argument `PROPS`.

Returns:

  a reference to the scheduler properties instance for `*this`.

Throws:

  `std::bad_cast` if `use_scheduling_algorithm()` was called with a `algorithm_with_properties` subclass with some other template parameter than `PROPS`.

Note:

  `algorithm_with_properties<>` provides a way for a user-coded scheduler to associate extended properties, such as priority, with a fiber instance. This method allows access to those user-provided properties.

See also:

  Customization

Member function `swap`()

    void swap( fiber & other) noexcept;

###

Effects:

  Exchanges the fiber of execution associated with `*this` and `other`, so `*this` becomes associated with the fiber formerly associated with `other`, and vice-versa.

Postconditions:

  `this->get_id()` returns the same value as `other.get_id()` prior to the call. `other.get_id()` returns the same value as `this->get_id()` prior to the call.

Throws:

  Nothing

Non-member function `swap()`

    void swap( fiber & l, fiber & r) noexcept;

###

Effects:

  Same as `l.swap( r)`.

Throws:

  Nothing

Non-member function `operator<()`

    bool operator<( fiber const& l, fiber const& r) noexcept;

###

Returns:

  `true` if `l.get_id() < r.get_id()` is `true`, false otherwise.

Throws:

  Nothing.

Non-member function `use_scheduling_algorithm()`

    template< typename SchedAlgo, typename ... Args >
    void use_scheduling_algorithm( Args && ... args) noexcept;

###

Effects:

  Directs Boost.Fiber to use `SchedAlgo`, which must be a concrete subclass of `algorithm`, as the scheduling algorithm for all fibers in the current thread. Pass any required `SchedAlgo` constructor arguments as `args`.

Note:

  If you want a given thread to use a non-default scheduling algorithm, make that thread call `use_scheduling_algorithm()` before any other Boost.Fiber entry point. If no scheduler has been set for the current thread by the time Boost.Fiber needs to use it, the library will create a default `round_robin` instance for this thread.

Throws:

  Nothing

See also:

  Scheduling, Customization

Non-member function `has_ready_fibers()`

    bool has_ready_fibers() noexcept;

###

Returns:

  `true` if scheduler has fibers ready to run.

Throws:

  Nothing

Note:

  Can be used for work-stealing to find an idle scheduler.

## Class fiber::id

    #include <boost/fiber/fiber.hpp>

    namespace boost {
    namespace fibers {

    class id {
    public:
        constexpr id() noexcept;

        bool operator==( id const&) const noexcept;

        bool operator!=( id const&) const noexcept;

        bool operator<( id const&) const noexcept;

        bool operator>( id const&) const noexcept;

        bool operator<=( id const&) const noexcept;

        bool operator>=( id const&) const noexcept;

        template< typename charT, class traitsT >
        friend std::basic_ostream< charT, traitsT > &
        operator<<( std::basic_ostream< charT, traitsT > &, id const&);
    };

    }}

Constructor

    constexpr id() noexcept;

###

Effects:

  Represents an instance of not-a-fiber.

Throws:

  Nothing.

Member function `operator==`()

    bool operator==( id const& other) const noexcept;

###

Returns:

  `true` if `*this` and `other` represent the same fiber, or both represent not-a-fiber, `false` otherwise.

Throws:

  Nothing.

Member function `operator!=`()

    bool operator!=( id const& other) const noexcept;

###

Returns:

  `! (other == * this)`

Throws:

  Nothing.

Member function `operator<`()

    bool operator<( id const& other) const noexcept;

###

Returns:

  `true` if `*this != other` is true and the implementation-defined total order of `fiber::id` values places `*this` before `other`, false otherwise.

Throws:

  Nothing.

Member function `operator>`()

    bool operator>( id const& other) const noexcept;

###

Returns:

  `other < * this`

Throws:

  Nothing.

Member function `operator<=`()

    bool operator<=( id const& other) const noexcept;

###

Returns:

  `! (other < * this)`

Throws:

  Nothing.

Member function `operator>=`()

    bool operator>=( id const& other) const noexcept;

###

Returns:

  `! (* this < other)`

Throws:

  Nothing.

operator<<

    template< typename charT, class traitsT >
    std::basic_ostream< charT, traitsT > &
    operator<<( std::basic_ostream< charT, traitsT > & os, id const& other);

###

Efects:

  Writes the representation of `other` to stream `os`. The representation is unspecified.

Returns:

  `os`

## Namespace this_fiber

In general, `this_fiber` operations may be called from the main fiber — the fiber on which function `main()` is entered — as well as from an explicitly-launched thread’s thread-function. That is, in many respects the main fiber on each thread can be treated like an explicitly-launched fiber.

    namespace boost {
    namespace this_fiber {

    fibers::fiber::id get_id() noexcept;
    void yield() noexcept;
    template< typename Clock, typename Duration >
    void sleep_until( std::chrono::time_point< Clock, Duration > const&);
    template< typename Rep, typename Period >
    void sleep_for( std::chrono::duration< Rep, Period > const&);
    template< typename PROPS >
    PROPS & properties();

    }}

Non-member function `this_fiber::get_id()`

    #include <boost/fiber/operations.hpp>

    namespace boost {
    namespace fibers {

    fiber::id get_id() noexcept;

    }}

###

Returns:

  An instance of `fiber::id` that represents the currently executing fiber.

Throws:

  Nothing.

Non-member function `this_fiber::sleep_until()`

    #include <boost/fiber/operations.hpp>

    namespace boost {
    namespace fibers {

    template< typename Clock, typename Duration >
    void sleep_until( std::chrono::time_point< Clock, Duration > const& abs_time);

    }}

###

Effects:

  Suspends the current fiber until the time point specified by `abs_time` has been reached.

Throws:

  timeout-related exceptions.

Note:

  The current fiber will not resume before `abs_time`, but there are no guarantees about how soon after `abs_time` it might resume.

Note:

  timeout-related exceptions are as defined in the C++ Standard, section 30.2.4 Timing specifications [thread.req.timing]: A function that takes an argument which specifies a timeout will throw if, during its execution, a clock, time point, or time duration throws an exception. Such exceptions are referred to as timeout-related exceptions.

Non-member function `this_fiber::sleep_for()`

    #include <boost/fiber/operations.hpp>

    namespace boost {
    namespace fibers {

    template< class Rep, class Period >
    void sleep_for( std::chrono::duration< Rep, Period > const& rel_time);

    }}

###

Effects:

  Suspends the current fiber until the time duration specified by `rel_time` has elapsed.

Throws:

  timeout-related exceptions.

Note:

  The current fiber will not resume before `rel_time` has elapsed, but there are no guarantees about how soon after that it might resume.

Non-member function `this_fiber::yield()`

    #include <boost/fiber/operations.hpp>

    namespace boost {
    namespace fibers {

    void yield() noexcept;

    }}

###

Effects:

  Relinquishes execution control, allowing other fibers to run.

Throws:

  Nothing.

Note:

  A fiber that calls `yield()` is not suspended: it is immediately passed to the scheduler as ready to run.

Non-member function `this_fiber::properties()`

    #include <boost/fiber/operations.hpp>

    namespace boost {
    namespace fibers {

    template< typename PROPS >
    PROPS & properties();

    }}

###

Preconditions:

  `use_scheduling_algorithm()` has been called from this thread with a subclass of `algorithm_with_properties<>` with the same template argument `PROPS`.

Returns:

  a reference to the scheduler properties instance for the currently running fiber.

Throws:

  `std::bad_cast` if `use_scheduling_algorithm()` was called with an `algorithm_with_properties` subclass with some other template parameter than `PROPS`.

Note:

  `algorithm_with_properties<>` provides a way for a user-coded scheduler to associate extended properties, such as priority, with a fiber instance. This function allows access to those user-provided properties.

Note:

  The first time this function is called from the main fiber of a thread, it may internally yield, permitting other fibers to run.

See also:

  Customization

# Scheduling

The fibers in a thread are coordinated by a fiber manager. Fibers trade control cooperatively, rather than preemptively: the currently-running fiber retains control until it invokes some operation that passes control to the manager. Each time a fiber suspends (or yields), the fiber manager consults a scheduler to determine which fiber will run next.

Boost.Fiber provides the fiber manager, but the scheduler is a customization point. (See Customization.)

Each thread has its own scheduler. Different threads in a process may use different schedulers. By default, Boost.Fiber implicitly instantiates `round_robin` as the scheduler for each thread.

You are explicitly permitted to code your own `algorithm` subclass. For the most part, your `algorithm` subclass need not defend against cross-thread calls: the fiber manager intercepts and defers such calls. Most `algorithm` methods are only ever directly called from the thread whose fibers it is managing — with exceptions as documented below.

Your `algorithm` subclass is engaged on a particular thread by calling `use_scheduling_algorithm()`:

    void thread_fn() {
        boost::fibers::use_scheduling_algorithm< my_fiber_scheduler >();
        ...
    }

A scheduler class must implement interface `algorithm`. Boost.Fiber provides schedulers: `round_robin`, `work_stealing`, `numa::work_stealing` and `shared_work`.

    void thread( std::uint32_t thread_count) {
        // thread registers itself at work-stealing scheduler
        boost::fibers::use_scheduling_algorithm< boost::fibers::algo::work_stealing >( thread_count);
        ...
    }

    // count of logical cpus
    std::uint32_t thread_count = std::thread::hardware_concurrency();
    // start-thread registers itself at work-stealing scheduler
    boost::fibers::use_scheduling_algorithm< boost::fibers::algo::work_stealing >( thread_count);
    std::vector< std::thread > threads;
    for ( std::uint32_t i = 1 /* count start-thread */; i < thread_count; ++i) {
        // spawn thread
        threads.emplace_back( thread, thread_count);
    }
    ...

The example spawns as many threads as `std::thread::hardware_concurrency()` returns. Each thread runs a `work_stealing` scheduler. Each instance of this scheduler needs to know how many threads run the work-stealing scheduler in the program. If the local queue of one thread runs out of ready fibers, the thread tries to steal a ready fiber from another thread running this scheduler.

Class `algorithm`

`algorithm` is the abstract base class defining the interface that a fiber scheduler must implement.

    #include <boost/fiber/algo/algorithm.hpp>

    namespace boost {
    namespace fibers {
    namespace algo {

    struct algorithm {
        virtual ~algorithm();

        virtual void awakened( context *) noexcept = 0;

        virtual context * pick_next() noexcept = 0;

        virtual bool has_ready_fibers() const noexcept = 0;

        virtual void suspend_until( std::chrono::steady_clock::time_point const&) noexcept = 0;

        virtual void notify() noexcept = 0;
    };

    }}}

Member function `awakened`()

    virtual void awakened( context * f) noexcept = 0;

##

Effects:

  Informs the scheduler that fiber `f` is ready to run. Fiber `f` might be newly launched, or it might have been blocked but has just been awakened, or it might have called `this_fiber::yield()`.

Note:

  This method advises the scheduler to add fiber `f` to its collection of fibers ready to run. A typical scheduler implementation places `f` into a queue.

See also:

  `round_robin`

Member function `pick_next`()

    virtual context * pick_next() noexcept = 0;

##

Returns:

  the fiber which is to be resumed next, or `nullptr` if there is no ready fiber.

Note:

  This is where the scheduler actually specifies the fiber which is to run next. A typical scheduler implementation chooses the head of the ready queue.

See also:

  `round_robin`

Member function `has_ready_fibers`()

    virtual bool has_ready_fibers() const noexcept = 0;

##

Returns:

  `true` if scheduler has fibers ready to run.

Member function `suspend_until`()

    virtual void suspend_until( std::chrono::steady_clock::time_point const& abs_time) noexcept = 0;

##

Effects:

  Informs the scheduler that no fiber will be ready until time-point `abs_time`.

Note:

  This method allows a custom scheduler to yield control to the containing environment in whatever way makes sense. The fiber manager is stating that `suspend_until()` need not return until `abs_time` — or `algorithm::notify()` is called — whichever comes first. The interaction with `notify()` means that, for instance, calling `std::this_thread::sleep_until(abs_time)` would be too simplistic. `round_robin::suspend_until()` uses a `std::condition_variable` to coordinate with `round_robin::notify()`.

Note:

  Given that `notify()` might be called from another thread, your `suspend_until()` implementation — like the rest of your `algorithm` implementation — must guard any data it shares with your `notify()` implementation.

Member function `notify`()

    virtual void notify() noexcept = 0;

##

Effects:

  Requests the scheduler to return from a pending call to `algorithm::suspend_until()`.

Note:

  Alone among the `algorithm` methods, `notify()` may be called from another thread. Your `notify()` implementation must guard any data it shares with the rest of your `algorithm` implementation.

Class `round_robin`

This class implements `algorithm`, scheduling fibers in round-robin fashion.

    #include <boost/fiber/algo/round_robin.hpp>

    namespace boost {
    namespace fibers {
    namespace algo {

    class round_robin : public algorithm {
        virtual void awakened( context *) noexcept;

        virtual context * pick_next() noexcept;

        virtual bool has_ready_fibers() const noexcept;

        virtual void suspend_until( std::chrono::steady_clock::time_point const&) noexcept;

        virtual void notify() noexcept;
    };

    }}}

Member function `awakened`()

    virtual void awakened( context * f) noexcept;

##

Effects:

  Enqueues fiber `f` onto a ready queue.

Throws:

  Nothing.

Member function `pick_next`()

    virtual context * pick_next() noexcept;

##

Returns:

  the fiber at the head of the ready queue, or `nullptr` if the queue is empty.

Throws:

  Nothing.

Note:

  Placing ready fibers onto the tail of a queue, and returning them from the head of that queue, shares the thread between ready fibers in round-robin fashion.

Member function `has_ready_fibers`()

    virtual bool has_ready_fibers() const noexcept;

##

Returns:

  `true` if scheduler has fibers ready to run.

Throws:

  Nothing.

Member function `suspend_until`()

    virtual void suspend_until( std::chrono::steady_clock::time_point const& abs_time) noexcept;

##

Effects:

  Informs `round_robin` that no ready fiber will be available until time-point `abs_time`. This implementation blocks in `std::condition_variable::wait_until()`.

Throws:

  Nothing.

Member function `notify`()

    virtual void notify() noexcept = 0;

##

Effects:

  Wake up a pending call to `round_robin::suspend_until()`, some fibers might be ready. This implementation wakes `suspend_until()` via `std::condition_variable::notify_all()`.

Throws:

  Nothing.

Class `work_stealing`

This class implements `algorithm`; if the local ready-queue runs out of ready fibers, ready fibers are stolen from other schedulers. The victim scheduler (from which a ready fiber is stolen) is selected at random.

Worker-threads are stored in a static variable, dynamically adding/removing worker threads is not supported.

    #include <boost/fiber/algo/work_stealing.hpp>

    namespace boost {
    namespace fibers {
    namespace algo {

    class work_stealing : public algorithm {
    public:
        work_stealing( std::uint32_t thread_count, bool suspend = false);

        work_stealing( work_stealing const&) = delete;
        work_stealing( work_stealing &&) = delete;

        work_stealing & operator=( work_stealing const&) = delete;
        work_stealing & operator=( work_stealing &&) = delete;

        virtual void awakened( context *) noexcept;

        virtual context * pick_next() noexcept;

        virtual bool has_ready_fibers() const noexcept;

        virtual void suspend_until( std::chrono::steady_clock::time_point const&) noexcept;

        virtual void notify() noexcept;
    };

    }}}

Constructor

    work_stealing( std::uint32_t thread_count, bool suspend = false);

##

Effects:

  Constructs work-stealing scheduling algorithm. `thread_count` represents the number of threads running this algorithm.

Throws:

  `system_error`

Note:

  If `suspend` is set to `true`, then the scheduler suspends if no ready fiber could be stolen. The scheduler will by woken up if a sleeping fiber times out or it was notified from remote (other thread or fiber scheduler).

Member function `awakened`()

    virtual void awakened( context * f) noexcept;

##

Effects:

  Enqueues fiber `f` onto the shared ready queue.

Throws:

  Nothing.

Member function `pick_next`()

    virtual context * pick_next() noexcept;

##

Returns:

  the fiber at the head of the ready queue, or `nullptr` if the queue is empty.

Throws:

  Nothing.

Note:

  Placing ready fibers onto the tail of the sahred queue, and returning them from the head of that queue, shares the thread between ready fibers in round-robin fashion.

Member function `has_ready_fibers`()

    virtual bool has_ready_fibers() const noexcept;

##

Returns:

  `true` if scheduler has fibers ready to run.

Throws:

  Nothing.

Member function `suspend_until`()

    virtual void suspend_until( std::chrono::steady_clock::time_point const& abs_time) noexcept;

##

Effects:

  Informs `work_stealing` that no ready fiber will be available until time-point `abs_time`. This implementation blocks in `std::condition_variable::wait_until()`.

Throws:

  Nothing.

Member function `notify`()

    virtual void notify() noexcept = 0;

##

Effects:

  Wake up a pending call to `work_stealing::suspend_until()`, some fibers might be ready. This implementation wakes `suspend_until()` via `std::condition_variable::notify_all()`.

Throws:

  Nothing.

Class `shared_work`

Because of the non-locality of data, shared_work is less performant than `work_stealing`.

This class implements `algorithm`, scheduling fibers in round-robin fashion. Ready fibers are shared between all instances (running on different threads) of shared_work, thus the work is distributed equally over all threads.

Worker-threads are stored in a static variable, dynamically adding/removing worker threads is not supported.

    #include <boost/fiber/algo/shared_work.hpp>

    namespace boost {
    namespace fibers {
    namespace algo {

    class shared_work : public algorithm {
        virtual void awakened( context *) noexcept;

        virtual context * pick_next() noexcept;

        virtual bool has_ready_fibers() const noexcept;

        virtual void suspend_until( std::chrono::steady_clock::time_point const&) noexcept;

        virtual void notify() noexcept;
    };

    }}}

Member function `awakened`()

    virtual void awakened( context * f) noexcept;

##

Effects:

  Enqueues fiber `f` onto the shared ready queue.

Throws:

  Nothing.

Member function `pick_next`()

    virtual context * pick_next() noexcept;

##

Returns:

  the fiber at the head of the ready queue, or `nullptr` if the queue is empty.

Throws:

  Nothing.

Note:

  Placing ready fibers onto the tail of the shared queue, and returning them from the head of that queue, shares the thread between ready fibers in round-robin fashion.

Member function `has_ready_fibers`()

    virtual bool has_ready_fibers() const noexcept;

##

Returns:

  `true` if scheduler has fibers ready to run.

Throws:

  Nothing.

Member function `suspend_until`()

    virtual void suspend_until( std::chrono::steady_clock::time_point const& abs_time) noexcept;

##

Effects:

  Informs `shared_work` that no ready fiber will be available until time-point `abs_time`. This implementation blocks in `std::condition_variable::wait_until()`.

Throws:

  Nothing.

Member function `notify`()

    virtual void notify() noexcept = 0;

##

Effects:

  Wake up a pending call to `shared_work::suspend_until()`, some fibers might be ready. This implementation wakes `suspend_until()` via `std::condition_variable::notify_all()`.

Throws:

  Nothing.

Custom Scheduler Fiber Properties

A scheduler class directly derived from `algorithm` can use any information available from `context` to implement the `algorithm` interface. But a custom scheduler might need to track additional properties for a fiber. For instance, a priority-based scheduler would need to track a fiber’s priority.

Boost.Fiber provides a mechanism by which your custom scheduler can associate custom properties with each fiber.

Class `fiber_properties`

A custom fiber properties class must be derived from `fiber_properties`.

    #include <boost/fiber/properties.hpp>

    namespace boost {
    namespace fibers {

    class fiber_properties {
    public:
        fiber_properties( context *) noexcept;

        virtual ~fiber_properties();

    protected:
        void notify() noexcept;
    };

    }}

Constructor

    fiber_properties( context * f) noexcept;

##

Effects:

  Constructs base-class component of custom subclass.

Throws:

  Nothing.

Note:

  Your subclass constructor must accept a `context*` and pass it to the base-class `fiber_properties` constructor.

Member function `notify`()

    void notify() noexcept;

##

Effects:

  Pass control to the custom `algorithm_with_properties<>` subclass’s `algorithm_with_properties::property_change()` method.

Throws:

  Nothing.

Note:

  A custom scheduler’s `algorithm_with_properties::pick_next()` method might dynamically select from the ready fibers, or `algorithm_with_properties::awakened()` might instead insert each ready fiber into some form of ready queue for `pick_next()`. In the latter case, if application code modifies a fiber property (e.g. priority) that should affect that fiber’s relationship to other ready fibers, the custom scheduler must be given the opportunity to reorder its ready queue. The custom property subclass should implement an access method to modify such a property; that access method should call `notify()` once the new property value has been stored. This passes control to the custom scheduler’s `property_change()` method, allowing the custom scheduler to reorder its ready queue appropriately. Use at your discretion. Of course, if you define a property which does not affect the behavior of the `pick_next()` method, you need not call `notify()` when that property is modified.

Template `algorithm_with_properties<>`

A custom scheduler that depends on a custom properties class `PROPS` should be derived from `algorithm_with_properties<PROPS>`. `PROPS` should be derived from `fiber_properties`.

    #include <boost/fiber/algorithm.hpp>

    namespace boost {
    namespace fibers {
    namespace algo {

    template< typename PROPS >
    struct algorithm_with_properties {
        virtual void awakened( context *, PROPS &) noexcept = 0;

        virtual context * pick_next() noexcept;

        virtual bool has_ready_fibers() const noexcept;

        virtual void suspend_until( std::chrono::steady_clock::time_point const&) noexcept = 0;

        virtual void notify() noexcept = 0;

        PROPS & properties( context *) noexcept;

        virtual void property_change( context *, PROPS &) noexcept;

        virtual fiber_properties * new_properties( context *);
    };

    }}}

Member function `awakened`()

    virtual void awakened( context * f, PROPS & properties) noexcept;

##

Effects:

  Informs the scheduler that fiber `f` is ready to run, like `algorithm::awakened()`. Passes the fiber’s associated `PROPS` instance.

Throws:

  Nothing.

Note:

  An `algorithm_with_properties<>` subclass must override this method instead of `algorithm::awakened()`.

Member function `pick_next`()

    virtual context * pick_next() noexcept;

##

Returns:

  the fiber which is to be resumed next, or `nullptr` if there is no ready fiber.

Throws:

  Nothing.

Note:

  same as `algorithm::pick_next()`

Member function `has_ready_fibers`()

    virtual bool has_ready_fibers() const noexcept;

##

Returns:

  `true` if scheduler has fibers ready to run.

Throws:

  Nothing.

Note:

  same as `algorithm::has_ready_fibers()`

Member function `suspend_until`()

    virtual void suspend_until( std::chrono::steady_clock::time_point const& abs_time) noexcept = 0;

##

Effects:

  Informs the scheduler that no fiber will be ready until time-point `abs_time`.

Note:

  same as `algorithm::suspend_until()`

Member function `notify`()

    virtual void notify() noexcept = 0;

##

Effects:

  Requests the scheduler to return from a pending call to `algorithm_with_properties::suspend_until()`.

Note:

  same as `algorithm::notify()`

Member function `properties`()

    PROPS& properties( context * f) noexcept;

##

Returns:

  the `PROPS` instance associated with fiber `f`.

Throws:

  Nothing.

Note:

  The fiber’s associated `PROPS` instance is already passed to `algorithm_with_properties::awakened()` and `algorithm_with_properties::property_change()`. However, every `algorithm` subclass is expected to track a collection of ready `context` instances. This method allows your custom scheduler to retrieve the `fiber_properties` subclass instance for any `context` in its collection.

Member function `property_change`()

    virtual void property_change( context * f, PROPS & properties) noexcept;

##

Effects:

  Notify the custom scheduler of a possibly-relevant change to a property belonging to fiber `f`. `properties` contains the new values of all relevant properties.

Throws:

  Nothing.

Note:

  This method is only called when a custom `fiber_properties` subclass explicitly calls `fiber_properties::notify()`.

Member function `new_properties`()

    virtual fiber_properties * new_properties( context * f);

##

Returns:

  A new instance of `fiber_properties` subclass `PROPS`.

Note:

  By default, `algorithm_with_properties<>::new_properties()` simply returns `new PROPS(f)`, placing the `PROPS` instance on the heap. Override this method to allocate `PROPS` some other way. The returned `fiber_properties` pointer must point to the `PROPS` instance to be associated with fiber `f`.

Class `context`

While you are free to treat `context*` as an opaque token, certain `context` members may be useful to a custom scheduler implementation.

Of particular note is the fact that `context` contains a hook to participate in a `boost::intrusive::list` `typedef`’ed as `boost::fibers::scheduler::ready_queue_t`. This hook is reserved for use by `algorithm` implementations. (For instance, `round_robin` contains a `ready_queue_t` instance to manage its ready fibers.) See `context::ready_is_linked()`, `context::ready_link()`, `context::ready_unlink()`.

Your `algorithm` implementation may use any container you desire to manage passed `context` instances. `ready_queue_t` avoids some of the overhead of typical STL containers.

    #include <boost/fiber/context.hpp>

    namespace boost {
    namespace fibers {

    enum class type {
      none               = unspecified,
      main_context       = unspecified, // fiber associated with thread's stack
      dispatcher_context = unspecified, // special fiber for maintenance operations
      worker_context     = unspecified, // fiber not special to the library
      pinned_context     = unspecified  // fiber must not be migrated to another thread
    };

    class context {
    public:
        class id;

        static context * active() noexcept;

        context( context const&) = delete;
        context & operator=( context const&) = delete;

        id get_id() const noexcept;

        void detach() noexcept;
        void attach( context *) noexcept;

        bool is_context( type) const noexcept;

        bool is_terminated() const noexcept;

        bool ready_is_linked() const noexcept;
        bool remote_ready_is_linked() const noexcept;
        bool wait_is_linked() const noexcept;

        template< typename List >
        void ready_link( List &) noexcept;
        template< typename List >
        void remote_ready_link( List &) noexcept;
        template< typename List >
        void wait_link( List &) noexcept;

        void ready_unlink() noexcept;
        void remote_ready_unlink() noexcept;
        void wait_unlink() noexcept;

        void suspend() noexcept;
        void set_ready( context *) noexcept;
    };

    bool operator<( context const& l, context const& r) noexcept;

    }}

Static member function `active`()

    static context * active() noexcept;

##

Returns:

  Pointer to instance of current fiber.

Throws:

  Nothing

Member function `get_id`()

    context::id get_id() const noexcept;

##

Returns:

  If `*this` refers to a fiber of execution, an instance of `fiber::id` that represents that fiber. Otherwise returns a default-constructed `fiber::id`.

Throws:

  Nothing

See also:

  `fiber::get_id()`

Member function `attach`()

    void attach( context * f) noexcept;

##

Precondition:

  `this->get_scheduler() == nullptr`

Effects:

  Attach fiber `f` to scheduler running `*this`.

Postcondition:

  `this->get_scheduler() != nullptr`

Throws:

  Nothing

Note:

  A typical call: `boost::fibers::context::active()->attach(f);`

Note:

  `f` must not be the running fiber’s context. It must not be blocked or terminated. It must not be a `pinned_context`. It must be currently detached. It must not currently be linked into an `algorithm` implementation’s ready queue. Most of these conditions are implied by `f` being owned by an `algorithm` implementation: that is, it has been passed to `algorithm::awakened()` but has not yet been returned by `algorithm::pick_next()`. Typically a `pick_next()` implementation would call `attach()` with the `context*` it is about to return. It must first remove `f` from its ready queue. You should never pass a `pinned_context` to `attach()` because you should never have called its `detach()` method in the first place.

Member function `detach`()

    void detach() noexcept;

##

Precondition:

  `(this->get_scheduler() != nullptr) && ! this->is_context(pinned_context)`

Effects:

  Detach fiber `*this` from its scheduler running `*this`.

Throws:

  Nothing

Postcondition:

  `this->get_scheduler() == nullptr`

Note:

  This method must be called on the thread with which the fiber is currently associated. `*this` must not be the running fiber’s context. It must not be blocked or terminated. It must not be a `pinned_context`. It must not be detached already. It must not already be linked into an `algorithm` implementation’s ready queue. Most of these conditions are implied by `*this` being passed to `algorithm::awakened()`; an `awakened()` implementation must, however, test for `pinned_context`. It must call `detach()` before linking `*this` into its ready queue.

Note:

  In particular, it is erroneous to attempt to migrate a fiber from one thread to another by calling both `detach()` and `attach()` in the `algorithm::pick_next()` method. `pick_next()` is called on the intended destination thread. `detach()` must be called on the fiber’s original thread. You must call `detach()` in the corresponding `awakened()` method.

Note:

  Unless you intend make a fiber available for potential migration to a different thread, you should call neither `detach()` nor `attach()` with its `context`.

Member function `is_context`()

    bool is_context( type t) const noexcept;

##

Returns:

  `true` if `*this` is of the specified type.

Throws:

  Nothing

Note:

  `type::worker_context` here means any fiber not special to the library. For `type::main_context` the `context` is associated with the main fiber of the thread: the one implicitly created by the thread itself, rather than one explicitly created by Boost.Fiber. For `type::dispatcher_context` the `context` is associated with a dispatching fiber, responsible for dispatching awakened fibers to a scheduler’s ready-queue. The dispatching fiber is an implementation detail of the fiber manager. The context of the main or dispatching fiber — any fiber for which `is_context(pinned_context)` is `true` — must never be passed to `context::detach()`.

Member function `is_terminated`()

    bool is_terminated() const noexcept;

##

Returns:

  `true` if `*this` is no longer a valid context.

Throws:

  Nothing

Note:

  The `context` has returned from its fiber-function and is no longer considered a valid context.

Member function `ready_is_linked`()

    bool ready_is_linked() const noexcept;

##

Returns:

  `true` if `*this` is stored in an `algorithm` implementation’s ready-queue.

Throws:

  Nothing

Note:

  Specifically, this method indicates whether `context::ready_link()` has been called on `*this`. `ready_is_linked()` has no information about participation in any other containers.

Member function `remote_ready_is_linked`()

    bool remote_ready_is_linked() const noexcept;

##

Returns:

  `true` if `*this` is stored in the fiber manager’s remote-ready-queue.

Throws:

  Nothing

Note:

  A `context` signaled as ready by another thread is first stored in the fiber manager’s remote-ready-queue. This is the mechanism by which the fiber manager protects an `algorithm` implementation from cross-thread `algorithm::awakened()` calls.

Member function `wait_is_linked`()

    bool wait_is_linked() const noexcept;

##

Returns:

  `true` if `*this` is stored in the wait-queue of some synchronization object.

Throws:

  Nothing

Note:

  The `context` of a fiber waiting on a synchronization object (e.g. `mutex`, `condition_variable` etc.) is stored in the wait-queue of that synchronization object.

Member function `ready_link`()

    template< typename List >
    void ready_link( List & lst) noexcept;

##

Effects:

  Stores `*this` in ready-queue `lst`.

Throws:

  Nothing

Note:

  Argument `lst` must be a doubly-linked list from Boost.Intrusive, e.g. an instance of `boost::fibers::scheduler::ready_queue_t`. Specifically, it must be a `boost::intrusive::list` compatible with the `list_member_hook` stored in the `context` object.

Member function `remote_ready_link`()

    template< typename List >
    void remote_ready_link( List & lst) noexcept;

##

Effects:

  Stores `*this` in remote-ready-queue `lst`.

Throws:

  Nothing

Note:

  Argument `lst` must be a doubly-linked list from Boost.Intrusive.

Member function `wait_link`()

    template< typename List >
    void wait_link( List & lst) noexcept;

##

Effects:

  Stores `*this` in wait-queue `lst`.

Throws:

  Nothing

Note:

  Argument `lst` must be a doubly-linked list from Boost.Intrusive.

Member function `ready_unlink`()

    void ready_unlink() noexcept;

##

Effects:

  Removes `*this` from ready-queue: undoes the effect of `context::ready_link()`.

Throws:

  Nothing

Member function `remote_ready_unlink`()

    void remote_ready_unlink() noexcept;

##

Effects:

  Removes `*this` from remote-ready-queue.

Throws:

  Nothing

Member function `wait_unlink`()

    void wait_unlink() noexcept;

##

Effects:

  Removes `*this` from wait-queue.

Throws:

  Nothing

Member function `suspend`()

    void suspend() noexcept;

##

Effects:

  Suspends the running fiber (the fiber associated with `*this`) until some other fiber passes `this` to `context::set_ready()`. `*this` is marked as not-ready, and control passes to the scheduler to select another fiber to run.

Throws:

  Nothing

Note:

  This is a low-level API potentially useful for integration with other frameworks. It is not intended to be directly invoked by a typical application program.

Note:

  The burden is on the caller to arrange for a call to `set_ready()` with a pointer to `this` at some future time.

Member function `set_ready`()

    void set_ready( context * ctx ) noexcept;

##

Effects:

  Mark the fiber associated with context `*ctx` as being ready to run. This does not immediately resume that fiber; rather it passes the fiber to the scheduler for subsequent resumption. If the scheduler is idle (has not returned from a call to `algorithm::suspend_until()`), `algorithm::notify()` is called to wake it up.

Throws:

  Nothing

Note:

  This is a low-level API potentially useful for integration with other frameworks. It is not intended to be directly invoked by a typical application program.

Note:

  It is explicitly supported to call `set_ready(ctx)` from a thread other than the one on which `*ctx` is currently suspended. The corresponding fiber will be resumed on its original thread in due course.

Non-member function `operator<()`

    bool operator<( context const& l, context const& r) noexcept;

##

Returns:

  `true` if `l.get_id() < r.get_id()` is `true`, `false` otherwise.

Throws:

  Nothing.

# Stack allocation

A `fiber` uses internally an __econtext__ which manages a set of registers and a stack. The memory used by the stack is allocated/deallocated via a stack_allocator which is required to model a stack-allocator concept.

A stack_allocator can be passed to `fiber::fiber()` or to `fibers::async()`.

stack-allocator concept

A stack_allocator must satisfy the stack-allocator concept requirements shown in the following table, in which `a` is an object of a stack_allocator type, `sctx` is a `stack_context`, and `size` is a `std::size_t`:

expression | return type | notes
--- | --- | ---
`a(size)` |  | creates a stack allocator
`a.allocate()` | `stack_context` | creates a stack
`a.deallocate( sctx)` | `void` | deallocates the stack created by `a.allocate()`

The implementation of `allocate()` might include logic to protect against exceeding the context's available stack size rather than leaving it as undefined behaviour.

Calling `deallocate()` with a `stack_context` not obtained from `allocate()` results in undefined behaviour.

The memory for the stack is not required to be aligned; alignment takes place inside __econtext__.

See also Boost.Context stack allocation. In particular, `traits_type` methods are as described for `boost::context::stack_traits`.

Class `protected_fixedsize_stack`

Boost.Fiber provides the class `protected_fixedsize_stack` which models the stack-allocator concept. It appends a guard page at the end of each stack to protect against exceeding the stack. If the guard page is accessed (read or write operation) a segmentation fault/access violation is generated by the operating system.

Using `protected_fixedsize_stack` is expensive. Launching a new fiber with a stack of this type incurs the overhead of setting the memory protection; once allocated, this stack is just as efficient to use as `fixedsize_stack`.

The appended `guard page` is not mapped to physical memory, only virtual addresses are used.

    #include <boost/fiber/protected_fixedsize.hpp>

    namespace boost {
    namespace fibers {

    struct protected_fixedsize {
        protected_fixesize(std::size_t size = traits_type::default_size());

        stack_context allocate();

        void deallocate( stack_context &);
    }

    }}

Member function `allocate`()

    stack_context allocate();

##

Preconditions:

  `traits_type::minimum_size() <= size` and `traits_type::is_unbounded() || ( size <= traits_type::maximum_size() )`.

Effects:

  Allocates memory of at least `size` bytes and stores a pointer to the stack and its actual size in `sctx`. Depending on the architecture (the stack grows downwards/upwards) the stored address is the highest/lowest address of the stack.

Member function `deallocate`()

    void deallocate( stack_context & sctx);

##

Preconditions:

  `sctx.sp` is valid, `traits_type::minimum_size() <= sctx.size` and `traits_type::is_unbounded() || ( sctx.size <= traits_type::maximum_size() )`.

Effects:

  Deallocates the stack space.

Class `pooled_fixedsize_stack`

Boost.Fiber provides the class `pooled_fixedsize_stack` which models the stack-allocator concept. In contrast to `protected_fixedsize_stack` it does not append a guard page at the end of each stack. The memory is managed internally by `boost::pool<>`.

    #include <boost/fiber/pooled_fixedsize_stack.hpp>

    namespace boost {
    namespace fibers {

    struct pooled_fixedsize_stack {
        pooled_fixedsize_stack(std::size_t stack_size = traits_type::default_size(), std::size_t next_size = 32, std::size_t max_size = 0);

        stack_context allocate();

        void deallocate( stack_context &);
    }

    }}

Constructor

    pooled_fixedsize_stack(std::size_t stack_size, std::size_t next_size, std::size_t max_size);

##

Preconditions:

  `traits_type::is_unbounded() || ( traits_type::maximum_size() >= stack_size)` and `0 < next_size`.

Effects:

  Allocates memory of at least `stack_size` bytes and stores a pointer to the stack and its actual size in `sctx`. Depending on the architecture (the stack grows downwards/upwards) the stored address is the highest/lowest address of the stack. Argument `next_size` determines the number of stacks to request from the system the first time that `*this` needs to allocate system memory. The third argument `max_size` controls how much memory might be allocated for stacks — a value of zero means no upper limit.

Member function `allocate`()

    stack_context allocate();

##

Preconditions:

  `traits_type::is_unbounded() || ( traits_type::maximum_size() >= stack_size)`.

Effects:

  Allocates memory of at least `stack_size` bytes and stores a pointer to the stack and its actual size in `sctx`. Depending on the architecture (the stack grows downwards/upwards) the stored address is the highest/lowest address of the stack.

Member function `deallocate`()

    void deallocate( stack_context & sctx);

##

Preconditions:

  `sctx.sp` is valid, `traits_type::is_unbounded() || ( traits_type::maximum_size() >= sctx.size)`.

Effects:

  Deallocates the stack space.

This stack allocator is not thread safe.

Class `fixedsize_stack`

Boost.Fiber provides the class `fixedsize_stack` which models the stack-allocator concept. In contrast to `protected_fixedsize_stack` it does not append a guard page at the end of each stack. The memory is simply managed by `std::malloc()` and `std::free()`.

    #include <boost/context/fixedsize_stack.hpp>

    namespace boost {
    namespace fibers {

    struct fixedsize_stack {
        fixedsize_stack(std::size_t size = traits_type::default_size());

        stack_context allocate();

        void deallocate( stack_context &);
    }

    }}

Member function `allocate`()

    stack_context allocate();

##

Preconditions:

  `traits_type::minimum_size() <= size` and `traits_type::is_unbounded() || ( traits_type::maximum_size() >= size)`.

Effects:

  Allocates memory of at least `size` bytes and stores a pointer to the stack and its actual size in `sctx`. Depending on the architecture (the stack grows downwards/upwards) the stored address is the highest/lowest address of the stack.

Member function `deallocate`()

    void deallocate( stack_context & sctx);

##

Preconditions:

  `sctx.sp` is valid, `traits_type::minimum_size() <= sctx.size` and `traits_type::is_unbounded() || ( traits_type::maximum_size() >= sctx.size)`.

Effects:

  Deallocates the stack space.

Class `segmented_stack`

Boost.Fiber supports usage of a `segmented_stack`, i.e. the stack grows on demand. The fiber is created with a minimal stack size which will be increased as required. Class `segmented_stack` models the stack-allocator concept. In contrast to `protected_fixedsize_stack` and `fixedsize_stack` it creates a stack which grows on demand.

Segmented stacks are currently only supported by gcc from version 4.7 and clang from version 3.4 onwards. In order to use a `segmented_stack` Boost.Fiber must be built with property `segmented-stacks`, e.g. toolset=gcc segmented-stacks=on and applying BOOST_USE_SEGMENTED_STACKS at b2/bjam command line.

Segmented stacks can only be used with callcc() using property `context-impl=ucontext`.

    #include <boost/fiber/segmented_stack.hpp>

    namespace boost {
    namespace fibers {

    struct segmented_stack {
        segmented_stack(std::size_t stack_size = traits_type::default_size());

        stack_context allocate();

        void deallocate( stack_context &);
    }

    }}

Member function `allocate`()

    stack_context allocate();

##

Preconditions:

  `traits_type::minimum_size() <= size` and `traits_type::is_unbounded() || ( traits_type::maximum_size() >= size)`.

Effects:

  Allocates memory of at least `size` bytes and stores a pointer to the stack and its actual size in `sctx`. Depending on the architecture (the stack grows downwards/upwards) the stored address is the highest/lowest address of the stack.

Member function `deallocate`()

    void deallocate( stack_context & sctx);

##

Preconditions:

  `sctx.sp` is valid, `traits_type::minimum_size() <= sctx.size` and `traits_type::is_unbounded() || ( traits_type::maximum_size() >= sctx.size)`.

Effects:

  Deallocates the stack space.

If the library is compiled for segmented stacks, `segmented_stack` is the only available stack allocator.

## Support for valgrind

Running programs that switch stacks under valgrind causes problems. Property (b2 command-line) `valgrind=on` let valgrind treat the memory regions as stack space which suppresses the errors.

# Synchronization

In general, Boost.Fiber synchronization objects can neither be moved nor copied. A synchronization object acts as a mutually-agreed rendezvous point between different fibers. If such an object were copied somewhere else, the new copy would have no consumers. If such an object were moved somewhere else, leaving the original instance in an unspecified state, existing consumers would behave strangely.

The fiber synchronization objects provided by this library will, by default, safely synchronize fibers running on different threads. However, this level of synchronization can be removed (for performance) by building the library with `BOOST_FIBERS_NO_ATOMICS` defined. When the library is built with that macro, you must ensure that all the fibers referencing a particular synchronization object are running in the same thread.

## Mutex Types

Class `mutex`

    #include <boost/fiber/mutex.hpp>

    namespace boost {
    namespace fibers {

    class mutex {
    public:
        mutex();
        ~mutex();

        mutex( mutex const& other) = delete;
        mutex & operator=( mutex const& other) = delete;

        void lock();
        bool try_lock();
        void unlock();
    };

    }}

`mutex` provides an exclusive-ownership mutex. At most one fiber can own the lock on a given instance of `mutex` at any time. Multiple concurrent calls to `lock()`, `try_lock()` and `unlock()` shall be permitted.

Any fiber blocked in `lock()` is suspended until the owning fiber releases the lock by calling `unlock()`.

Member function `lock`()

    void lock();

###

Precondition:

  The calling fiber doesn't own the mutex.

Effects:

  The current fiber blocks until ownership can be obtained.

Throws:

  `lock_error`

Error Conditions:

  resource_deadlock_would_occur: if `boost::this_fiber::get_id()` already owns the mutex.

Member function `try_lock`()

    bool try_lock();

###

Precondition:

  The calling fiber doesn't own the mutex.

Effects:

  Attempt to obtain ownership for the current fiber without blocking.

Returns:

  `true` if ownership was obtained for the current fiber, `false` otherwise.

Throws:

  `lock_error`

Error Conditions:

  resource_deadlock_would_occur: if `boost::this_fiber::get_id()` already owns the mutex.

Member function `unlock`()

    void unlock();

###

Precondition:

  The current fiber owns `*this`.

Effects:

  Releases a lock on `*this` by the current fiber.

Throws:

  `lock_error`

Error Conditions:

  operation_not_permitted: if `boost::this_fiber::get_id()` does not own the mutex.

Class `timed_mutex`

    #include <boost/fiber/timed_mutex.hpp>

    namespace boost {
    namespace fibers {

    class timed_mutex {
    public:
        timed_mutex();
        ~timed_mutex();

        timed_mutex( timed_mutex const& other) = delete;
        timed_mutex & operator=( timed_mutex const& other) = delete;

        void lock();
        bool try_lock();
        void unlock();

        template< typename Clock, typename Duration >
        bool try_lock_until( std::chrono::time_point< Clock, Duration > const& timeout_time);
        template< typename Rep, typename Period >
        bool try_lock_for( std::chrono::duration< Rep, Period > const& timeout_duration);
    };

    }}

`timed_mutex` provides an exclusive-ownership mutex. At most one fiber can own the lock on a given instance of `timed_mutex` at any time. Multiple concurrent calls to `lock()`, `try_lock()`, `try_lock_until()`, `try_lock_for()` and `unlock()` shall be permitted.

Member function `lock`()

    void lock();

###

Precondition:

  The calling fiber doesn't own the mutex.

Effects:

  The current fiber blocks until ownership can be obtained.

Throws:

  `lock_error`

Error Conditions:

  resource_deadlock_would_occur: if `boost::this_fiber::get_id()` already owns the mutex.

Member function `try_lock`()

    bool try_lock();

###

Precondition:

  The calling fiber doesn't own the mutex.

Effects:

  Attempt to obtain ownership for the current fiber without blocking.

Returns:

  `true` if ownership was obtained for the current fiber, `false` otherwise.

Throws:

  `lock_error`

Error Conditions:

  resource_deadlock_would_occur: if `boost::this_fiber::get_id()` already owns the mutex.

Member function `unlock`()

    void unlock();

###

Precondition:

  The current fiber owns `*this`.

Effects:

  Releases a lock on `*this` by the current fiber.

Throws:

  `lock_error`

Error Conditions:

  operation_not_permitted: if `boost::this_fiber::get_id()` does not own the mutex.

Templated member function `try_lock_until`()

    template< typename Clock, typename Duration >
    bool try_lock_until( std::chrono::time_point< Clock, Duration > const& timeout_time);

###

Precondition:

  The calling fiber doesn't own the mutex.

Effects:

  Attempt to obtain ownership for the current fiber. Blocks until ownership can be obtained, or the specified time is reached. If the specified time has already passed, behaves as `timed_mutex::try_lock()`.

Returns:

  `true` if ownership was obtained for the current fiber, `false` otherwise.

Throws:

  `lock_error`, timeout-related exceptions.

Error Conditions:

  resource_deadlock_would_occur: if `boost::this_fiber::get_id()` already owns the mutex.

Templated member function `try_lock_for`()

    template< typename Rep, typename Period >
    bool try_lock_for( std::chrono::duration< Rep, Period > const& timeout_duration);

###

Precondition:

  The calling fiber doesn't own the mutex.

Effects:

  Attempt to obtain ownership for the current fiber. Blocks until ownership can be obtained, or the specified time is reached. If the specified time has already passed, behaves as `timed_mutex::try_lock()`.

Returns:

  `true` if ownership was obtained for the current fiber, `false` otherwise.

Throws:

  `lock_error`, timeout-related exceptions.

Error Conditions:

  resource_deadlock_would_occur: if `boost::this_fiber::get_id()` already owns the mutex.

Class `recursive_mutex`

    #include <boost/fiber/recursive_mutex.hpp>

    namespace boost {
    namespace fibers {

    class recursive_mutex {
    public:
        recursive_mutex();
        ~recursive_mutex();

        recursive_mutex( recursive_mutex const& other) = delete;
        recursive_mutex & operator=( recursive_mutex const& other) = delete;

        void lock();
        bool try_lock() noexcept;
        void unlock();
    };

    }}

`recursive_mutex` provides an exclusive-ownership recursive mutex. At most one fiber can own the lock on a given instance of `recursive_mutex` at any time. Multiple concurrent calls to `lock()`, `try_lock()` and `unlock()` shall be permitted. A fiber that already has exclusive ownership of a given `recursive_mutex` instance can call `lock()` or `try_lock()` to acquire an additional level of ownership of the mutex. `unlock()` must be called once for each level of ownership acquired by a single fiber before ownership can be acquired by another fiber.

Member function `lock`()

    void lock();

###

Effects:

  The current fiber blocks until ownership can be obtained.

Throws:

  Nothing

Member function `try_lock`()

    bool try_lock() noexcept;

###

Effects:

  Attempt to obtain ownership for the current fiber without blocking.

Returns:

  `true` if ownership was obtained for the current fiber, `false` otherwise.

Throws:

  Nothing.

Member function `unlock`()

    void unlock();

###

Effects:

  Releases a lock on `*this` by the current fiber.

Throws:

  `lock_error`

Error Conditions:

  operation_not_permitted: if `boost::this_fiber::get_id()` does not own the mutex.

Class `recursive_timed_mutex`

    #include <boost/fiber/recursive_timed_mutex.hpp>

    namespace boost {
    namespace fibers {

    class recursive_timed_mutex {
    public:
        recursive_timed_mutex();
        ~recursive_timed_mutex();

        recursive_timed_mutex( recursive_timed_mutex const& other) = delete;
        recursive_timed_mutex & operator=( recursive_timed_mutex const& other) = delete;

        void lock();
        bool try_lock() noexcept;
        void unlock();

        template< typename Clock, typename Duration >
        bool try_lock_until( std::chrono::time_point< Clock, Duration > const& timeout_time);
        template< typename Rep, typename Period >
        bool try_lock_for( std::chrono::duration< Rep, Period > const& timeout_duration);
    };

    }}

`recursive_timed_mutex` provides an exclusive-ownership recursive mutex. At most one fiber can own the lock on a given instance of `recursive_timed_mutex` at any time. Multiple concurrent calls to `lock()`, `try_lock()`, `try_lock_for()`, `try_lock_until()` and `unlock()` shall be permitted. A fiber that already has exclusive ownership of a given `recursive_timed_mutex` instance can call `lock()`, `try_lock()`, `try_lock_for()` or `try_lock_until()` to acquire an additional level of ownership of the mutex. `unlock()` must be called once for each level of ownership acquired by a single fiber before ownership can be acquired by another fiber.

Member function `lock`()

    void lock();

###

Effects:

  The current fiber blocks until ownership can be obtained.

Throws:

  Nothing

Member function `try_lock`()

    bool try_lock() noexcept;

###

Effects:

  Attempt to obtain ownership for the current fiber without blocking.

Returns:

  `true` if ownership was obtained for the current fiber, `false` otherwise.

Throws:

  Nothing.

Member function `unlock`()

    void unlock();

###

Effects:

  Releases a lock on `*this` by the current fiber.

Throws:

  `lock_error`

Error Conditions:

  operation_not_permitted: if `boost::this_fiber::get_id()` does not own the mutex.

Templated member function `try_lock_until`()

    template< typename Clock, typename Duration >
    bool try_lock_until( std::chrono::time_point< Clock, Duration > const& timeout_time);

###

Effects:

  Attempt to obtain ownership for the current fiber. Blocks until ownership can be obtained, or the specified time is reached. If the specified time has already passed, behaves as `recursive_timed_mutex::try_lock()`.

Returns:

  `true` if ownership was obtained for the current fiber, `false` otherwise.

Throws:

  Timeout-related exceptions.

Templated member function `try_lock_for`()

    template< typename Rep, typename Period >
    bool try_lock_for( std::chrono::duration< Rep, Period > const& timeout_duration);

###

Effects:

  Attempt to obtain ownership for the current fiber. Blocks until ownership can be obtained, or the specified time is reached. If the specified time has already passed, behaves as `recursive_timed_mutex::try_lock()`.

Returns:

  `true` if ownership was obtained for the current fiber, `false` otherwise.

Throws:

  Timeout-related exceptions.

## Condition Variables

Synopsis

    enum class cv_status; {
        no_timeout,
        timeout
    };

    class condition_variable;
    class condition_variable_any;

The class `condition_variable` provides a mechanism for a fiber to wait for notification from another fiber. When the fiber awakens from the wait, then it checks to see if the appropriate condition is now true, and continues if so. If the condition is not true, then the fiber calls `wait` again to resume waiting. In the simplest case, this condition is just a boolean variable:

    boost::fibers::condition_variable cond;
    boost::fibers::mutex mtx;
    bool data_ready = false;

    void process_data();

    void wait_for_data_to_process() {
        {
            std::unique_lock< boost::fibers::mutex > lk( mtx);
            while ( ! data_ready) {
                cond.wait( lk);
            }
        }   // release lk
        process_data();
    }

Notice that the `lk` is passed to `condition_variable::wait()`: `wait()` will atomically add the fiber to the set of fibers waiting on the condition variable, and unlock the `mutex`. When the fiber is awakened, the `mutex` will be locked again before the call to `wait()` returns. This allows other fibers to acquire the `mutex` in order to update the shared data, and ensures that the data associated with the condition is correctly synchronized.

`wait_for_data_to_process()` could equivalently be written:

    void wait_for_data_to_process() {
        {
            std::unique_lock< boost::fibers::mutex > lk( mtx);
            // make condition_variable::wait() perform the loop
            cond.wait( lk, [](){ return data_ready; });
        }   // release lk
        process_data();
    }

In the meantime, another fiber sets `data_ready` to `true`, and then calls either `condition_variable::notify_one()` or `condition_variable::notify_all()` on the `condition_variable` `cond` to wake one waiting fiber or all the waiting fibers respectively.

    void retrieve_data();
    void prepare_data();

    void prepare_data_for_processing() {
        retrieve_data();
        prepare_data();
        {
            std::unique_lock< boost::fibers::mutex > lk( mtx);
            data_ready = true;
        }
        cond.notify_one();
    }

Note that the same `mutex` is locked before the shared data is updated, but that the `mutex` does not have to be locked across the call to `condition_variable::notify_one()`.

Locking is important because the synchronization objects provided by Boost.Fiber can be used to synchronize fibers running on different threads.

Boost.Fiber provides both `condition_variable` and `condition_variable_any`. `boost::fibers::condition_variable` can only wait on `std::unique_lock` `< boost::fibers::` `mutex` `>` while `boost::fibers::condition_variable_any` can wait on user-defined lock types.

No Spurious Wakeups

Neither `condition_variable` nor `condition_variable_any` are subject to spurious wakeup: `condition_variable::wait()` can only wake up when `condition_variable::notify_one()` or `condition_variable::notify_all()` is called. Even so, it is prudent to use one of the `wait( lock, predicate )` overloads.

Consider a set of consumer fibers processing items from a `std::queue`. The queue is continually populated by a set of producer fibers.

The consumer fibers might reasonably wait on a `condition_variable` as long as the queue remains `empty()`.

Because producer fibers might `push()` items to the queue in bursts, they call `condition_variable::notify_all()` rather than `condition_variable::notify_one()`.

But a given consumer fiber might well wake up from `condition_variable::wait()` and find the queue `empty()`, because other consumer fibers might already have processed all pending items.

(See also spurious wakeup.)

Enumeration `cv_status`

A timed wait operation might return because of timeout or not.

    enum class cv_status {
        no_timeout,
        timeout
    };

`no_timeout`

###

Effects:

  The condition variable was awakened with `notify_one` or `notify_all`.

`timeout`

###

Effects:

  The condition variable was awakened by timeout.

Class `condition_variable_any`

    #include <boost/fiber/condition_variable.hpp>

    namespace boost {
    namespace fibers {

    class condition_variable_any {
    public:
        condition_variable_any();
        ~condition_variable_any();

        condition_variable_any( condition_variable_any const&) = delete;
        condition_variable_any & operator=( condition_variable_any const&) = delete;

        void notify_one() noexcept;
        void notify_all() noexcept;

        template< typename LockType >
        void wait( LockType &);

        template< typename LockType, typename Pred >
        void wait( LockType &, Pred);

        template< typename LockType, typename Clock, typename Duration >
        cv_status wait_until( LockType &,
                              std::chrono::time_point< Clock, Duration > const&);

        template< typename LockType, typename Clock, typename Duration, typename Pred >
        bool wait_until( LockType &,
                         std::chrono::time_point< Clock, Duration > const&,
                         Pred);

        template< typename LockType, typename Rep, typename Period >
        cv_status wait_for( LockType &,
                            std::chrono::duration< Rep, Period > const&);

        template< typename LockType, typename Rep, typename Period, typename Pred >
        bool wait_for( LockType &,
                       std::chrono::duration< Rep, Period > const&,
                       Pred);
    };

    }}

Constructor

    condition_variable_any()

###

Effects:

  Creates the object.

Throws:

  Nothing.

Destructor

    ~condition_variable_any()

###

Precondition:

  All fibers waiting on `*this` have been notified by a call to `notify_one` or `notify_all` (though the respective calls to `wait`, `wait_for` or `wait_until` need not have returned).

Effects:

  Destroys the object.

Member function `notify_one`()

    void notify_one() noexcept;

###

Effects:

  If any fibers are currently blocked waiting on `*this` in a call to `wait`, `wait_for` or `wait_until`, unblocks one of those fibers.

Throws:

  Nothing.

Note:

  It is arbitrary which waiting fiber is resumed.

Member function `notify_all`()

    void notify_all() noexcept;

###

Effects:

  If any fibers are currently blocked waiting on `*this` in a call to `wait`, `wait_for` or `wait_until`, unblocks all of those fibers.

Throws:

  Nothing.

Note:

  This is why a waiting fiber must also check for the desired program state using a mechanism external to the `condition_variable_any`, and retry the wait until that state is reached. A fiber waiting on a `condition_variable_any` might well wake up a number of times before the desired state is reached.

Templated member function `wait`()

    template< typename LockType >
        void wait( LockType & lk);

    template< typename LockType, typename Pred >
    void wait( LockType & lk, Pred pred);

###

Precondition:

  `lk` is locked by the current fiber, and either no other fiber is currently waiting on `*this`, or the execution of the `mutex()` member function on the `lk` objects supplied in the calls to `wait` in all the fibers currently waiting on `*this` would return the same value as `lk->mutex()` for this call to `wait`.

Effects:

  Atomically call `lk.unlock()` and blocks the current fiber. The fiber will unblock when notified by a call to `this->notify_one()` or `this->notify_all()`. When the fiber is unblocked (for whatever reason), the lock is reacquired by invoking `lk.lock()` before the call to `wait` returns. The lock is also reacquired by invoking `lk.lock()` if the function exits with an exception. The member function accepting `pred` is shorthand for:

      while ( ! pred() ) {
          wait( lk);
      }

Postcondition:

  `lk` is locked by the current fiber.

Throws:

  `fiber_error` if an error occurs.

Note:

  The Precondition is a bit dense. It merely states that all the fibers concurrently calling `wait` on `*this` must wait on `lk` objects governing the same `mutex`. Three distinct objects are involved in any `condition_variable_any::wait()` call: the `condition_variable_any` itself, the `mutex` coordinating access between fibers and a local lock object (e.g. `std::unique_lock`). In general, you can partition the lifespan of a given `condition_variable_any` instance into periods with one or more fibers waiting on it, separated by periods when no fibers are waiting on it. When more than one fiber is waiting on that `condition_variable_any`, all must pass lock objects referencing the same `mutex` instance.

Templated member function `wait_until`()

    template< typename LockType, typename Clock, typename Duration >
    cv_status wait_until( LockType & lk,
                          std::chrono::time_point< Clock, Duration > const& abs_time);

    template< typename LockType, typename Clock, typename Duration, typename Pred >
    bool wait_until( LockType & lk,
                     std::chrono::time_point< Clock, Duration > const& abs_time,
                     Pred pred);

###

Precondition:

  `lk` is locked by the current fiber, and either no other fiber is currently waiting on `*this`, or the execution of the `mutex()` member function on the `lk` objects supplied in the calls to `wait`, `wait_for` or `wait_until` in all the fibers currently waiting on `*this` would return the same value as `lk.mutex()` for this call to `wait_until`.

Effects:

  Atomically call `lk.unlock()` and blocks the current fiber. The fiber will unblock when notified by a call to `this->notify_one()` or `this->notify_all()`, when the system time would be equal to or later than the specified `abs_time`. When the fiber is unblocked (for whatever reason), the lock is reacquired by invoking `lk.lock()` before the call to `wait_until` returns. The lock is also reacquired by invoking `lk.lock()` if the function exits with an exception. The member function accepting `pred` is shorthand for:

      while ( ! pred() ) {
          if ( cv_status::timeout == wait_until( lk, abs_time) )
              return pred();
      }
      return true;

  That is, even if `wait_until()` times out, it can still return `true` if `pred()` returns `true` at that time.
Postcondition:

  `lk` is locked by the current fiber.

Throws:

  `fiber_error` if an error occurs or timeout-related exceptions.

Returns:

  The overload without `pred` returns `cv_status::no_timeout` if awakened by `notify_one()` or `notify_all()`, or `cv_status::timeout` if awakened because the system time is past `abs_time`.

Returns:

  The overload accepting `pred` returns `false` if the call is returning because the time specified by `abs_time` was reached and the predicate returns `false`, `true` otherwise.

Note:

  See Note for `condition_variable_any::wait()`.

Templated member function `wait_for`()

    template< typename LockType, typename Rep, typename Period >
    cv_status wait_for( LockType & lk,
                        std::chrono::duration< Rep, Period > const& rel_time);

    template< typename LockType, typename Rep, typename Period, typename Pred >
    bool wait_for( LockType & lk,
                   std::chrono::duration< Rep, Period > const& rel_time,
                   Pred pred);

###

Precondition:

  `lk` is locked by the current fiber, and either no other fiber is currently waiting on `*this`, or the execution of the `mutex()` member function on the `lk` objects supplied in the calls to `wait`, `wait_for` or `wait_until` in all the fibers currently waiting on `*this` would return the same value as `lk.mutex()` for this call to `wait_for`.

Effects:

  Atomically call `lk.unlock()` and blocks the current fiber. The fiber will unblock when notified by a call to `this->notify_one()` or `this->notify_all()`, when a time interval equal to or greater than the specified `rel_time` has elapsed. When the fiber is unblocked (for whatever reason), the lock is reacquired by invoking `lk.lock()` before the call to `wait` returns. The lock is also reacquired by invoking `lk.lock()` if the function exits with an exception. The `wait_for()` member function accepting `pred` is shorthand for:

      while ( ! pred() ) {
          if ( cv_status::timeout == wait_for( lk, rel_time) ) {
              return pred();
          }
      }
      return true;

  (except of course that `rel_time` is adjusted for each iteration). The point is that, even if `wait_for()` times out, it can still return `true` if `pred()` returns `true` at that time.
Postcondition:

  `lk` is locked by the current fiber.

Throws:

  `fiber_error` if an error occurs or timeout-related exceptions.

Returns:

  The overload without `pred` returns `cv_status::no_timeout` if awakened by `notify_one()` or `notify_all()`, or `cv_status::timeout` if awakened because at least `rel_time` has elapsed.

Returns:

  The overload accepting `pred` returns `false` if the call is returning because at least `rel_time` has elapsed and the predicate returns `false`, `true` otherwise.

Note:

  See Note for `condition_variable_any::wait()`.

Class `condition_variable`

    #include <boost/fiber/condition_variable.hpp>

    namespace boost {
    namespace fibers {

    class condition_variable {
    public:
        condition_variable();
        ~condition_variable();

        condition_variable( condition_variable const&) = delete;
        condition_variable & operator=( condition_variable const&) = delete;

        void notify_one() noexcept;
        void notify_all() noexcept;

        void wait( std::unique_lock< mutex > &);

        template< typename Pred >
        void wait( std::unique_lock< mutex > &, Pred);

        template< typename Clock, typename Duration >
        cv_status wait_until( std::unique_lock< mutex > &,
                              std::chrono::time_point< Clock, Duration > const&);

        template< typename Clock, typename Duration, typename Pred >
        bool wait_until( std::unique_lock< mutex > &,
                         std::chrono::time_point< Clock, Duration > const&,
                         Pred);

        template< typename Rep, typename Period >
        cv_status wait_for( std::unique_lock< mutex > &,
                            std::chrono::duration< Rep, Period > const&);

        template< typename Rep, typename Period, typename Pred >
        bool wait_for( std::unique_lock< mutex > &,
                       std::chrono::duration< Rep, Period > const&,
                       Pred);
    };

    }}

Constructor

    condition_variable()

###

Effects:

  Creates the object.

Throws:

  Nothing.

Destructor

    ~condition_variable()

###

Precondition:

  All fibers waiting on `*this` have been notified by a call to `notify_one` or `notify_all` (though the respective calls to `wait`, `wait_for` or `wait_until` need not have returned).

Effects:

  Destroys the object.

Member function `notify_one`()

    void notify_one() noexcept;

###

Effects:

  If any fibers are currently blocked waiting on `*this` in a call to `wait`, `wait_for` or `wait_until`, unblocks one of those fibers.

Throws:

  Nothing.

Note:

  It is arbitrary which waiting fiber is resumed.

Member function `notify_all`()

    void notify_all() noexcept;

###

Effects:

  If any fibers are currently blocked waiting on `*this` in a call to `wait`, `wait_for` or `wait_until`, unblocks all of those fibers.

Throws:

  Nothing.

Note:

  This is why a waiting fiber must also check for the desired program state using a mechanism external to the `condition_variable`, and retry the wait until that state is reached. A fiber waiting on a `condition_variable` might well wake up a number of times before the desired state is reached.

Templated member function `wait`()

    void wait( std::unique_lock< mutex > & lk);

    template< typename Pred >
    void wait( std::unique_lock< mutex > & lk, Pred pred);

###

Precondition:

  `lk` is locked by the current fiber, and either no other fiber is currently waiting on `*this`, or the execution of the `mutex()` member function on the `lk` objects supplied in the calls to `wait` in all the fibers currently waiting on `*this` would return the same value as `lk->mutex()` for this call to `wait`.

Effects:

  Atomically call `lk.unlock()` and blocks the current fiber. The fiber will unblock when notified by a call to `this->notify_one()` or `this->notify_all()`. When the fiber is unblocked (for whatever reason), the lock is reacquired by invoking `lk.lock()` before the call to `wait` returns. The lock is also reacquired by invoking `lk.lock()` if the function exits with an exception. The member function accepting `pred` is shorthand for:

      while ( ! pred() ) {
          wait( lk);
      }

Postcondition:

  `lk` is locked by the current fiber.

Throws:

  `fiber_error` if an error occurs.

Note:

  The Precondition is a bit dense. It merely states that all the fibers concurrently calling `wait` on `*this` must wait on `lk` objects governing the same `mutex`. Three distinct objects are involved in any `condition_variable::wait()` call: the `condition_variable` itself, the `mutex` coordinating access between fibers and a local lock object (e.g. `std::unique_lock`). In general, you can partition the lifespan of a given `condition_variable` instance into periods with one or more fibers waiting on it, separated by periods when no fibers are waiting on it. When more than one fiber is waiting on that `condition_variable`, all must pass lock objects referencing the same `mutex` instance.

Templated member function `wait_until`()

    template< typename Clock, typename Duration >
    cv_status wait_until( std::unique_lock< mutex > & lk,
                          std::chrono::time_point< Clock, Duration > const& abs_time);

    template< typename Clock, typename Duration, typename Pred >
    bool wait_until( std::unique_lock< mutex > & lk,
                     std::chrono::time_point< Clock, Duration > const& abs_time,
                     Pred pred);

###

Precondition:

  `lk` is locked by the current fiber, and either no other fiber is currently waiting on `*this`, or the execution of the `mutex()` member function on the `lk` objects supplied in the calls to `wait`, `wait_for` or `wait_until` in all the fibers currently waiting on `*this` would return the same value as `lk.mutex()` for this call to `wait_until`.

Effects:

  Atomically call `lk.unlock()` and blocks the current fiber. The fiber will unblock when notified by a call to `this->notify_one()` or `this->notify_all()`, when the system time would be equal to or later than the specified `abs_time`. When the fiber is unblocked (for whatever reason), the lock is reacquired by invoking `lk.lock()` before the call to `wait_until` returns. The lock is also reacquired by invoking `lk.lock()` if the function exits with an exception. The member function accepting `pred` is shorthand for:

      while ( ! pred() ) {
          if ( cv_status::timeout == wait_until( lk, abs_time) )
              return pred();
      }
      return true;

  That is, even if `wait_until()` times out, it can still return `true` if `pred()` returns `true` at that time.
Postcondition:

  `lk` is locked by the current fiber.

Throws:

  `fiber_error` if an error occurs or timeout-related exceptions.

Returns:

  The overload without `pred` returns `cv_status::no_timeout` if awakened by `notify_one()` or `notify_all()`, or `cv_status::timeout` if awakened because the system time is past `abs_time`.

Returns:

  The overload accepting `pred` returns `false` if the call is returning because the time specified by `abs_time` was reached and the predicate returns `false`, `true` otherwise.

Note:

  See Note for `condition_variable::wait()`.

Templated member function `wait_for`()

    template< typename Rep, typename Period >
    cv_status wait_for( std::unique_lock< mutex > & lk,
                        std::chrono::duration< Rep, Period > const& rel_time);

    template< typename Rep, typename Period, typename Pred >
    bool wait_for( std::unique_lock< mutex > & lk,
                   std::chrono::duration< Rep, Period > const& rel_time,
                   Pred pred);

###

Precondition:

  `lk` is locked by the current fiber, and either no other fiber is currently waiting on `*this`, or the execution of the `mutex()` member function on the `lk` objects supplied in the calls to `wait`, `wait_for` or `wait_until` in all the fibers currently waiting on `*this` would return the same value as `lk.mutex()` for this call to `wait_for`.

Effects:

  Atomically call `lk.unlock()` and blocks the current fiber. The fiber will unblock when notified by a call to `this->notify_one()` or `this->notify_all()`, when a time interval equal to or greater than the specified `rel_time` has elapsed. When the fiber is unblocked (for whatever reason), the lock is reacquired by invoking `lk.lock()` before the call to `wait` returns. The lock is also reacquired by invoking `lk.lock()` if the function exits with an exception. The `wait_for()` member function accepting `pred` is shorthand for:

      while ( ! pred() ) {
          if ( cv_status::timeout == wait_for( lk, rel_time) ) {
              return pred();
          }
      }
      return true;

  (except of course that `rel_time` is adjusted for each iteration). The point is that, even if `wait_for()` times out, it can still return `true` if `pred()` returns `true` at that time.
Postcondition:

  `lk` is locked by the current fiber.

Throws:

  `fiber_error` if an error occurs or timeout-related exceptions.

Returns:

  The overload without `pred` returns `cv_status::no_timeout` if awakened by `notify_one()` or `notify_all()`, or `cv_status::timeout` if awakened because at least `rel_time` has elapsed.

Returns:

  The overload accepting `pred` returns `false` if the call is returning because at least `rel_time` has elapsed and the predicate returns `false`, `true` otherwise.

Note:

  See Note for `condition_variable::wait()`.

## Barriers

A barrier is a concept also known as a rendezvous, it is a synchronization point between multiple contexts of execution (fibers). The barrier is configured for a particular number of fibers (`n`), and as fibers reach the barrier they must wait until all `n` fibers have arrived. Once the `n`-th fiber has reached the barrier, all the waiting fibers can proceed, and the barrier is reset.

The fact that the barrier automatically resets is significant. Consider a case in which you launch some number of fibers and want to wait only until the first of them has completed. You might be tempted to use a `barrier(2)` as the synchronization mechanism, making each new fiber call its `barrier::wait()` method, then calling `wait()` in the launching fiber to wait until the first other fiber completes.

That will in fact unblock the launching fiber. The unfortunate part is that it will continue blocking the remaining fibers.

Consider the following scenario:

1. Fiber main launches fibers A, B, C and D, then calls `barrier::wait()`.

2. Fiber C finishes first and likewise calls `barrier::wait()`.

3. Fiber main is unblocked, as desired.

4. Fiber B calls `barrier::wait()`. Fiber B is blocked!

5. Fiber A calls `barrier::wait()`. Fibers A and B are unblocked.

6. Fiber D calls `barrier::wait()`. Fiber D is blocked indefinitely.

(See also when_any, simple completion.)

It is unwise to tie the lifespan of a barrier to any one of its participating fibers. Although conceptually all waiting fibers awaken simultaneously, because of the nature of fibers, in practice they will awaken one by one in indeterminate order. The current implementation wakes fibers in FIFO order: the first to call `wait()` wakes first, and so forth. But it is perilous to rely on the order in which the various fibers will reach the `wait()` call. The rest of the waiting fibers will still be blocked in `wait()`, which must, before returning, access data members in the barrier object.

Class `barrier`

    #include <boost/fiber/barrier.hpp>

    namespace boost {
    namespace fibers {

    class barrier {
    public:
        explicit barrier( std::size_t);

        barrier( barrier const&) = delete;
        barrier & operator=( barrier const&) = delete;

        bool wait();
    };

    }}

Instances of `barrier` are not copyable or movable.

Constructor

    explicit barrier( std::size_t initial);

###

Effects:

  Construct a barrier for `initial` fibers.

Throws:

  `fiber_error`

Error Conditions:

  invalid_argument: if `initial` is zero.

Member function `wait`()

    bool wait();

###

Effects:

  Block until `initial` fibers have called `wait` on `*this`. When the `initial`-th fiber calls `wait`, all waiting fibers are unblocked, and the barrier is reset.

Returns:

  `true` for exactly one fiber from each batch of waiting fibers, `false` otherwise.

Throws:

  `fiber_error`

## Channels

A channel is a model to communicate and synchronize `Threads of Execution` The smallest ordered sequence of instructions that can be managed independently by a scheduler is called a `Thread of Execution`. via message passing.

Enumeration `channel_op_status`

channel operations return the state of the channel.

    enum class channel_op_status {
        success,
        empty,
        full,
        closed,
        timeout
    };

`success`

###

Effects:

  Operation was successful.

`empty`

###

Effects:

  channel is empty, operation failed.

`full`

###

Effects:

  channel is full, operation failed.

`closed`

###

Effects:

  channel is closed, operation failed.

`timeout`

###

Effects:

  The operation did not become ready before specified timeout elapsed.

### Buffered Channel

Boost.Fiber provides a bounded, buffered channel (MPMC queue) suitable to synchonize fibers (running on same or different threads) via asynchronouss message passing.

    typedef boost::fibers::buffered_channel< int > channel_t;

    void send( channel_t & chan) {
        for ( int i = 0; i < 5; ++i) {
            chan.push( i);
        }
        chan.close();
    }

    void recv( channel_t & chan) {
        int i;
        while ( boost::fibers::channel_op_status::success == chan.pop(i) ) {
            std::cout << "received " << i << std::endl;
        }
    }

    channel_t chan{ 2 };
    boost::fibers::fiber f1( std::bind( send, std::ref( chan) ) );
    boost::fibers::fiber f2( std::bind( recv, std::ref( chan) ) );

    f1.join();
    f2.join();

Class `buffered_channel` supports range-for syntax:

    typedef boost::fibers::buffered_channel< int > channel_t;

    void foo( channel_t & chan) {
        chan.push( 1);
        chan.push( 1);
        chan.push( 2);
        chan.push( 3);
        chan.push( 5);
        chan.push( 8);
        chan.push( 12);
        chan.close();
    }

    void bar( channel_t & chan) {
        for ( unsigned int value : chan) {
            std::cout << value << " ";
        }
        std::cout << std::endl;
    }

Template `buffered_channel<>`

    #include <boost/fiber/buffered_channel.hpp>

    namespace boost {
    namespace fibers {

    template< typename T >
    class buffered_channel {
    public:
        typedef T   value_type;

        explicit buffered_channel( std::size_t capacity);

        buffered_channel( buffered_channel const& other) = delete;
        buffered_channel & operator=( buffered_channel const& other) = delete;

        void close() noexcept;

        channel_op_status push( value_type const& va);
        channel_op_status push( value_type && va);
        template< typename Rep, typename Period >
        channel_op_status push_wait_for(
            value_type const& va,
            std::chrono::duration< Rep, Period > const& timeout_duration);
        channel_op_status push_wait_for( value_type && va,
            std::chrono::duration< Rep, Period > const& timeout_duration);
        template< typename Clock, typename Duration >
        channel_op_status push_wait_until(
            value_type const& va,
            std::chrono::time_point< Clock, Duration > const& timeout_time);
        template< typename Clock, typename Duration >
        channel_op_status push_wait_until(
            value_type && va,
            std::chrono::time_point< Clock, Duration > const& timeout_time);
        channel_op_status try_push( value_type const& va);
        channel_op_status try_push( value_type && va);

        channel_op_status pop( value_type & va);
        value_type value_pop();
        template< typename Rep, typename Period >
        channel_op_status pop_wait_for(
            value_type & va,
            std::chrono::duration< Rep, Period > const& timeout_duration);
        template< typename Clock, typename Duration >
        channel_op_status pop_wait_until(
            value_type & va,
            std::chrono::time_point< Clock, Duration > const& timeout_time);
        channel_op_status try_pop( value_type & va);
    };

    }}

Constructor

    explicit buffered_channel( std::size_t capacity);

####

Preconditions:

  `2<=capacity && 0==(capacity & (capacity-1))`

Effects:

  The constructor constructs an object of class `buffered_channel` with an internal buffer of size `capacity`.

Throws:

  `fiber_error`

Error Conditions:

  invalid_argument: if `0==capacity || 0!=(capacity & (capacity-1))`.

Notes:

  A `push()`, `push_wait_for()` or `push_wait_until()` will not block until the number of values in the channel becomes equal to `capacity`. The channel can hold only `capacity - 1` elements, otherwise it is considered to be full.

Member function `close`()

    void close() noexcept;

####

Effects:

  Deactivates the channel. No values can be put after calling `this->close()`. Fibers blocked in `this->pop()`, `this->pop_wait_for()` or `this->pop_wait_until()` will return `closed`. Fibers blocked in `this->value_pop()` will receive an exception.

Throws:

  Nothing.

Note:

  `close()` is like closing a pipe. It informs waiting consumers that no more values will arrive.

Member function `push`()

    channel_op_status push( value_type const& va);
    channel_op_status push( value_type && va);

####

Effects:

  If channel is closed, returns `closed`. Otherwise enqueues the value in the channel, wakes up a fiber blocked on `this->pop()`, `this->value_pop()`, `this->pop_wait_for()` or `this->pop_wait_until()` and returns `success`.

Throws:

  Exceptions thrown by copy- or move-operations.

Member function `try_push`()

    channel_op_status try_push( value_type const& va);
    channel_op_status try_push( value_type && va);

####

Effects:

  If channel is closed, returns `closed`. Otherwise enqueues the value in the channel, wakes up a fiber blocked on `this->pop()`, `this->value_pop()`, `this->pop_wait_for()` or `this->pop_wait_until()` and returns `success`.

Throws:

  Exceptions thrown by copy- or move-operations.

Member function `pop`()

    channel_op_status pop( value_type & va);

####

Effects:

  Dequeues a value from the channel. If the channel is empty, the fiber gets suspended until at least one new item is `push()`ed (return value `success` and `va` contains dequeued value) or the channel gets `close()`d (return value `closed`).

Throws:

  Exceptions thrown by copy- or move-operations.

Member function `value_pop`()

    value_type value_pop();

####

Effects:

  Dequeues a value from the channel. If the channel is empty, the fiber gets suspended until at least one new item is `push()`ed or the channel gets `close()`d (which throws an exception).

Throws:

  `fiber_error` if `*this` is closed or by copy- or move-operations.

Error conditions:

  `std::errc::operation_not_permitted`

Member function `try_pop`()

    channel_op_status try_pop( value_type & va);

####

Effects:

  If channel is empty, returns `empty`. If channel is closed, returns `closed`. Otherwise it returns `success` and `va` contains the dequeued value.

Throws:

  Exceptions thrown by copy- or move-operations.

Member function `pop_wait_for`()

    template< typename Rep, typename Period >
    channel_op_status pop_wait_for(
        value_type & va,
        std::chrono::duration< Rep, Period > const& timeout_duration)

####

Effects:

  Accepts `std::chrono::duration` and internally computes a timeout time as (system time + `timeout_duration`). If channel is not empty, immediately dequeues a value from the channel. Otherwise the fiber gets suspended until at least one new item is `push()`ed (return value `success` and `va` contains dequeued value), or the channel gets `close()`d (return value `closed`), or the system time reaches the computed timeout time (return value `timeout`).

Throws:

  timeout-related exceptions or by copy- or move-operations.

Member function `pop_wait_until`()

    template< typename Clock, typename Duration >
    channel_op_status pop_wait_until(
        value_type & va,
        std::chrono::time_point< Clock, Duration > const& timeout_time)

####

Effects:

  Accepts a `std::chrono::time_point< Clock, Duration >`. If channel is not empty, immediately dequeues a value from the channel. Otherwise the fiber gets suspended until at least one new item is `push()`ed (return value `success` and `va` contains dequeued value), or the channel gets `close()`d (return value `closed`), or the system time reaches the passed `time_point` (return value `timeout`).

Throws:

  timeout-related exceptions or by copy- or move-operations.

### Unbuffered Channel

Boost.Fiber provides template `unbuffered_channel` suitable to synchonize fibers (running on same or different threads) via synchronous message passing. A fiber waiting to consume an value will block until the value is produced. If a fiber attempts to send a value through an unbuffered channel and no fiber is waiting to receive the value, the channel will block the sending fiber.

The unbuffered channel acts as an `rendezvous point`.

    typedef boost::fibers::unbuffered_channel< int > channel_t;

    void send( channel_t & chan) {
        for ( int i = 0; i < 5; ++i) {
            chan.push( i);
        }
        chan.close();
    }

    void recv( channel_t & chan) {
        int i;
        while ( boost::fibers::channel_op_status::success == chan.pop(i) ) {
            std::cout << "received " << i << std::endl;
        }
    }

    channel_t chan{ 1 };
    boost::fibers::fiber f1( std::bind( send, std::ref( chan) ) );
    boost::fibers::fiber f2( std::bind( recv, std::ref( chan) ) );

    f1.join();
    f2.join();

Range-for syntax is supported:

    typedef boost::fibers::unbuffered_channel< int > channel_t;

    void foo( channel_t & chan) {
        chan.push( 1);
        chan.push( 1);
        chan.push( 2);
        chan.push( 3);
        chan.push( 5);
        chan.push( 8);
        chan.push( 12);
        chan.close();
    }

    void bar( channel_t & chan) {
        for ( unsigned int value : chan) {
            std::cout << value << " ";
        }
        std::cout << std::endl;
    }

Template `unbuffered_channel<>`

    #include <boost/fiber/unbuffered_channel.hpp>

    namespace boost {
    namespace fibers {

    template< typename T >
    class unbuffered_channel {
    public:
        typedef T   value_type;

        unbuffered_channel();

        unbuffered_channel( unbuffered_channel const& other) = delete;
        unbuffered_channel & operator=( unbuffered_channel const& other) = delete;

        void close() noexcept;

        channel_op_status push( value_type const& va);
        channel_op_status push( value_type && va);
        template< typename Rep, typename Period >
        channel_op_status push_wait_for(
            value_type const& va,
            std::chrono::duration< Rep, Period > const& timeout_duration);
        channel_op_status push_wait_for( value_type && va,
            std::chrono::duration< Rep, Period > const& timeout_duration);
        template< typename Clock, typename Duration >
        channel_op_status push_wait_until(
            value_type const& va,
            std::chrono::time_point< Clock, Duration > const& timeout_time);
        template< typename Clock, typename Duration >
        channel_op_status push_wait_until(
            value_type && va,
            std::chrono::time_point< Clock, Duration > const& timeout_time);

        channel_op_status pop( value_type & va);
        value_type value_pop();
        template< typename Rep, typename Period >
        channel_op_status pop_wait_for(
            value_type & va,
            std::chrono::duration< Rep, Period > const& timeout_duration);
        template< typename Clock, typename Duration >
        channel_op_status pop_wait_until(
            value_type & va,
            std::chrono::time_point< Clock, Duration > const& timeout_time);
    };

    }}

Constructor

    unbuffered_channel();

####

Effects:

  The constructor constructs an object of class `unbuffered_channel`.

Member function `close`()

    void close() noexcept;

####

Effects:

  Deactivates the channel. No values can be put after calling `this->close()`. Fibers blocked in `this->pop()`, `this->pop_wait_for()` or `this->pop_wait_until()` will return `closed`. Fibers blocked in `this->value_pop()` will receive an exception.

Throws:

  Nothing.

Note:

  `close()` is like closing a pipe. It informs waiting consumers that no more values will arrive.

Member function `push`()

    channel_op_status push( value_type const& va);
    channel_op_status push( value_type && va);

####

Effects:

  If channel is closed, returns `closed`. Otherwise enqueues the value in the channel, wakes up a fiber blocked on `this->pop()`, `this->value_pop()`, `this->pop_wait_for()` or `this->pop_wait_until()` and returns `success`.

Throws:

  Exceptions thrown by copy- or move-operations.

Member function `pop`()

    channel_op_status pop( value_type & va);

####

Effects:

  Dequeues a value from the channel. If the channel is empty, the fiber gets suspended until at least one new item is `push()`ed (return value `success` and `va` contains dequeued value) or the channel gets `close()`d (return value `closed`).

Throws:

  Exceptions thrown by copy- or move-operations.

Member function `value_pop`()

    value_type value_pop();

####

Effects:

  Dequeues a value from the channel. If the channel is empty, the fiber gets suspended until at least one new item is `push()`ed or the channel gets `close()`d (which throws an exception).

Throws:

  `fiber_error` if `*this` is closed or by copy- or move-operations.

Error conditions:

  `std::errc::operation_not_permitted`

Member function `pop_wait_for`()

    template< typename Rep, typename Period >
    channel_op_status pop_wait_for(
        value_type & va,
        std::chrono::duration< Rep, Period > const& timeout_duration)

####

Effects:

  Accepts `std::chrono::duration` and internally computes a timeout time as (system time + `timeout_duration`). If channel is not empty, immediately dequeues a value from the channel. Otherwise the fiber gets suspended until at least one new item is `push()`ed (return value `success` and `va` contains dequeued value), or the channel gets `close()`d (return value `closed`), or the system time reaches the computed timeout time (return value `timeout`).

Throws:

  timeout-related exceptions or by copy- or move-operations.

Member function `pop_wait_until`()

    template< typename Clock, typename Duration >
    channel_op_status pop_wait_until(
        value_type & va,
        std::chrono::time_point< Clock, Duration > const& timeout_time)

####

Effects:

  Accepts a `std::chrono::time_point< Clock, Duration >`. If channel is not empty, immediately dequeues a value from the channel. Otherwise the fiber gets suspended until at least one new item is `push()`ed (return value `success` and `va` contains dequeued value), or the channel gets `close()`d (return value `closed`), or the system time reaches the passed `time_point` (return value `timeout`).

Throws:

  timeout-related exceptions or by copy- or move-operations.

## Futures

Overview

The futures library provides a means of handling asynchronous future values, whether those values are generated by another fiber, or on a single fiber in response to external stimuli, or on-demand.

This is done through the provision of four class templates: `future<>` and `shared_future<>` which are used to retrieve the asynchronous results, and `promise<>` and `packaged_task<>` which are used to generate the asynchronous results.

An instance of `future<>` holds the one and only reference to a result. Ownership can be transferred between instances using the move constructor or move-assignment operator, but at most one instance holds a reference to a given asynchronous result. When the result is ready, it is returned from `future::get()` by rvalue-reference to allow the result to be moved or copied as appropriate for the type.

On the other hand, many instances of `shared_future<>` may reference the same result. Instances can be freely copied and assigned, and `shared_future::get()` returns a `const` reference so that multiple calls to `shared_future::get()` are safe. You can move an instance of `future<>` into an instance of `shared_future<>`, thus transferring ownership of the associated asynchronous result, but not vice-versa.

`fibers::async()` is a simple way of running asynchronous tasks. A call to `async()` spawns a fiber and returns a `future<>` that will deliver the result of the fiber function.

Creating asynchronous values

You can set the value in a future with either a `promise<>` or a `packaged_task<>`. A `packaged_task<>` is a callable object with `void` return that wraps a function or callable object returning the specified type. When the `packaged_task<>` is invoked, it invokes the contained function in turn, and populates a future with the contained function's return value. This is an answer to the perennial question: How do I return a value from a fiber? Package the function you wish to run as a `packaged_task<>` and pass the packaged task to the fiber constructor. The future retrieved from the packaged task can then be used to obtain the return value. If the function throws an exception, that is stored in the future in place of the return value.

    int calculate_the_answer_to_life_the_universe_and_everything() {
        return 42;
    }

    boost::fibers::packaged_task<int()> pt(calculate_the_answer_to_life_the_universe_and_everything);
    boost::fibers::future<int> fi=pt.get_future();
    boost::fibers::fiber(std::move(pt)).detach(); // launch task on a fiber

    fi.wait(); // wait for it to finish

    assert(fi.is_ready());
    assert(fi.has_value());
    assert(!fi.has_exception());
    assert(fi.get()==42);

A `promise<>` is a bit more low level: it just provides explicit functions to store a value or an exception in the associated future. A promise can therefore be used where the value might come from more than one possible source.

    boost::fibers::promise<int> pi;
    boost::fibers::future<int> fi;
    fi=pi.get_future();

    pi.set_value(42);

    assert(fi.is_ready());
    assert(fi.has_value());
    assert(!fi.has_exception());
    assert(fi.get()==42);

### Future

A future provides a mechanism to access the result of an asynchronous operation.

shared state

Behind a `promise<>` and its `future<>` lies an unspecified object called their shared state. The shared state is what will actually hold the async result (or the exception).

The shared state is instantiated along with the `promise<>`.

Aside from its originating `promise<>`, a `future<>` holds a unique reference to a particular shared state. However, multiple `shared_future<>` instances can reference the same underlying shared state.

As `packaged_task<>` and `fibers::async()` are implemented using `promise<>`, discussions of shared state apply to them as well.

Enumeration `future_status`

Timed wait-operations (`future::wait_for()` and `future::wait_until()`) return the state of the future.

    enum class future_status {
        ready,
        timeout,
        deferred  // not supported yet
    };

`ready`

####

Effects:

  The shared state is ready.

`timeout`

####

Effects:

  The shared state did not become ready before timeout has passed.

Deferred futures are not supported.

Template `future<>`

A `future<>` contains a shared state which is not shared with any other future.

    #include <boost/fiber/future/future.hpp>

    namespace boost {
    namespace fibers {

    template< typename R >
    class future {
    public:
        future() noexcept;

        future( future const& other) = delete;

        future & operator=( future const& other) = delete;

        future( future && other) noexcept;

        future & operator=( future && other) noexcept;

        ~future();

        bool valid() const noexcept;

        shared_future< R > share();

        R get();    // member only of generic future template
        R & get();  // member only of future< R & > template specialization
        void get(); // member only of future< void > template specialization

        std::exception_ptr get_exception_ptr();

        void wait() const;

        template< class Rep, class Period >
        future_status wait_for(
            std::chrono::duration< Rep, Period > const& timeout_duration) const;

        template< typename Clock, typename Duration >
        future_status wait_until(
            std::chrono::time_point< Clock, Duration > const& timeout_time) const;
    };

    }}

Default constructor

    future() noexcept;

####

Effects:

  Creates a future with no shared state. After construction `false == valid()`.

Throws:

  Nothing.

Move constructor

    future( future && other) noexcept;

####

Effects:

  Constructs a future with the shared state of other. After construction `false == other.valid()`.

Throws:

  Nothing.

Destructor

    ~future();

####

Effects:

  Destroys the future; ownership is abandoned.

Note:

  `~future()` does not block the calling fiber.

Consider a sequence such as:

1. instantiate `promise<>`

2. obtain its `future<>` via `promise::get_future()`

3. launch `fiber`, capturing `promise<>`

4. destroy `future<>`

5. call `promise::set_value()`

The final `set_value()` call succeeds, but the value is silently discarded: no additional `future<>` can be obtained from that `promise<>`.

Member function `operator=`()

    future & operator=( future && other) noexcept;

####

Effects:

  Moves the shared state of other to `this`. After the assignment, `false == other.valid()`.

Throws:

  Nothing.

Member function `valid`()

    bool valid() const noexcept;

####

Effects:

  Returns `true` if future contains a shared state.

Throws:

  Nothing.

Member function `share`()

    shared_future< R > share();

####

Effects:

  Move the state to a `shared_future<>`.

Returns:

  a `shared_future<>` containing the shared state formerly belonging to `*this`.

Postcondition:

  `false == valid()`

Throws:

  `future_error` with error condition `future_errc::no_state`.

Member function `get`()

    R get();    // member only of generic future template
    R & get();  // member only of future< R & > template specialization
    void get(); // member only of future< void > template specialization

####

Precondition:

  `true == valid()`

Returns:

  Waits until `promise::set_value()` or `promise::set_exception()` is called. If `promise::set_value()` is called, returns the value. If `promise::set_exception()` is called, throws the indicated exception.

Postcondition:

  `false == valid()`

Throws:

  `future_error` with error condition `future_errc::no_state`, `future_errc::broken_promise`. Any exception passed to `promise::set_exception()`.

Member function `get_exception_ptr`()

    std::exception_ptr get_exception_ptr();

####

Precondition:

  `true == valid()`

Returns:

  Waits until `promise::set_value()` or `promise::set_exception()` is called. If `set_value()` is called, returns a default-constructed `std::exception_ptr`. If `set_exception()` is called, returns the passed `std::exception_ptr`.

Throws:

  `future_error` with error condition `future_errc::no_state`.

Note:

  `get_exception_ptr()` does not invalidate the `future`. After calling `get_exception_ptr()`, you may still call `future::get()`.

Member function `wait`()

    void wait();

####

Effects:

  Waits until `promise::set_value()` or `promise::set_exception()` is called.

Throws:

  `future_error` with error condition `future_errc::no_state`.

Templated member function `wait_for`()

    template< class Rep, class Period >
    future_status wait_for( std::chrono::duration< Rep, Period > const& timeout_duration) const;

####

Effects:

  Waits until `promise::set_value()` or `promise::set_exception()` is called, or `timeout_duration` has passed.

Result:

  A `future_status` is returned indicating the reason for returning.

Throws:

  `future_error` with error condition `future_errc::no_state` or timeout-related exceptions.

Templated member function `wait_until`()

    template< typename Clock, typename Duration >
    future_status wait_until( std::chrono::time_point< Clock, Duration > const& timeout_time) const;

####

Effects:

  Waits until `promise::set_value()` or `promise::set_exception()` is called, or `timeout_time` has passed.

Result:

  A `future_status` is returned indicating the reason for returning.

Throws:

  `future_error` with error condition `future_errc::no_state` or timeout-related exceptions.

Template `shared_future<>`

A `shared_future<>` contains a shared state which might be shared with other `shared_future<>` instances.

    #include <boost/fiber/future/future.hpp>

    namespace boost {
    namespace fibers {

    template< typename R >
    class shared_future {
    public:
        shared_future() noexcept;

        ~shared_future();

        shared_future( shared_future const& other);

        shared_future( future< R > && other) noexcept;

        shared_future( shared_future && other) noexcept;

        shared_future & operator=( shared_future && other) noexcept;

        shared_future & operator=( future< R > && other) noexcept;

        shared_future & operator=( shared_future const& other) noexcept;

        bool valid() const noexcept;

        R const& get(); // member only of generic shared_future template
        R & get();      // member only of shared_future< R & > template specialization
        void get();     // member only of shared_future< void > template specialization

        std::exception_ptr get_exception_ptr();

        void wait() const;

        template< class Rep, class Period >
        future_status wait_for(
            std::chrono::duration< Rep, Period > const& timeout_duration) const;

        template< typename Clock, typename Duration >
        future_status wait_until(
            std::chrono::time_point< Clock, Duration > const& timeout_time) const;
    };

    }}

Default constructor

    shared_future();

####

Effects:

  Creates a shared_future with no shared state. After construction `false == valid()`.

Throws:

  Nothing.

Move constructor

    shared_future( future< R > && other) noexcept;
    shared_future( shared_future && other) noexcept;

####

Effects:

  Constructs a shared_future with the shared state of other. After construction `false == other.valid()`.

Throws:

  Nothing.

Copy constructor

    shared_future( shared_future const& other) noexcept;

####

Effects:

  Constructs a shared_future with the shared state of other. After construction `other.valid()` is unchanged.

Throws:

  Nothing.

Destructor

    ~shared_future();

####

Effects:

  Destroys the shared_future; ownership is abandoned if not shared.

Note:

  `~shared_future()` does not block the calling fiber.

Member function `operator=`()

    shared_future & operator=( future< R > && other) noexcept;
    shared_future & operator=( shared_future && other) noexcept;
    shared_future & operator=( shared_future const& other) noexcept;

####

Effects:

  Moves or copies the shared state of other to `this`. After the assignment, the state of `other.valid()` depends on which overload was invoked: unchanged for the overload accepting `shared_future const&`, otherwise `false`.

Throws:

  Nothing.

Member function `valid`()

    bool valid() const noexcept;

####

Effects:

  Returns `true` if shared_future contains a shared state.

Throws:

  Nothing.

Member function `get`()

    R const& get(); // member only of generic shared_future template
    R & get();      // member only of shared_future< R & > template specialization
    void get();     // member only of shared_future< void > template specialization

####

Precondition:

  `true == valid()`

Returns:

  Waits until `promise::set_value()` or `promise::set_exception()` is called. If `promise::set_value()` is called, returns the value. If `promise::set_exception()` is called, throws the indicated exception.

Postcondition:

  `false == valid()`

Throws:

  `future_error` with error condition `future_errc::no_state`, `future_errc::broken_promise`. Any exception passed to `promise::set_exception()`.

Member function `get_exception_ptr`()

    std::exception_ptr get_exception_ptr();

####

Precondition:

  `true == valid()`

Returns:

  Waits until `promise::set_value()` or `promise::set_exception()` is called. If `set_value()` is called, returns a default-constructed `std::exception_ptr`. If `set_exception()` is called, returns the passed `std::exception_ptr`.

Throws:

  `future_error` with error condition `future_errc::no_state`.

Note:

  `get_exception_ptr()` does not invalidate the `shared_future`. After calling `get_exception_ptr()`, you may still call `shared_future::get()`.

Member function `wait`()

    void wait();

####

Effects:

  Waits until `promise::set_value()` or `promise::set_exception()` is called.

Throws:

  `future_error` with error condition `future_errc::no_state`.

Templated member function `wait_for`()

    template< class Rep, class Period >
    future_status wait_for( std::chrono::duration< Rep, Period > const& timeout_duration) const;

####

Effects:

  Waits until `promise::set_value()` or `promise::set_exception()` is called, or `timeout_duration` has passed.

Result:

  A `future_status` is returned indicating the reason for returning.

Throws:

  `future_error` with error condition `future_errc::no_state` or timeout-related exceptions.

Templated member function `wait_until`()

    template< typename Clock, typename Duration >
    future_status wait_until( std::chrono::time_point< Clock, Duration > const& timeout_time) const;

####

Effects:

  Waits until `promise::set_value()` or `promise::set_exception()` is called, or `timeout_time` has passed.

Result:

  A `future_status` is returned indicating the reason for returning.

Throws:

  `future_error` with error condition `future_errc::no_state` or timeout-related exceptions.

Non-member function `fibers::async()`

    #include <boost/fiber/future/async.hpp>

    namespace boost {
    namespace fibers {

    template< class Function, class ... Args >
    future<
        std::result_of_t<
            std::decay_t< Function >( std::decay_t< Args > ... )
        >
    >
    async( Function && fn, Args && ... args);

    template< class Function, class ... Args >
    future<
        std::result_of_t<
            std::decay_t< Function >( std::decay_t< Args > ... )
        >
    >
    async( launch policy, Function && fn, Args && ... args);

    template< typename StackAllocator, class Function, class ... Args >
    future<
        std::result_of_t<
            std::decay_t< Function >( std::decay_t< Args > ... )
        >
    >
    async( launch policy, std::allocator_arg_t, StackAllocator salloc,
           Function && fn, Args && ... args);

    template< typename StackAllocator, typename Allocator, class Function, class ... Args >
    future<
        std::result_of_t<
            std::decay_t< Function >( std::decay_t< Args > ... )
        >
    >
    async( launch policy, std::allocator_arg_t, StackAllocator salloc,
           Allocator alloc, Function && fn, Args && ... args);

    }}

####

Effects:

  Executes `fn` in a `fiber` and returns an associated `future<>`.

Result:

      future<
          std::result_of_t<
              std::decay_t< Function >( std::decay_t< Args > ... )
          >
      >

  representing the shared state associated with the asynchronous execution of `fn`.
Throws:

  `fiber_error` or `future_error` if an error occurs.

Notes:

  The overloads accepting `std::allocator_arg_t` use the passed `StackAllocator` when constructing the launched `fiber`. The overloads accepting `launch` use the passed `launch` when constructing the launched `fiber`. The default `launch` is `post`, as for the `fiber` constructor.

Deferred futures are not supported.

### Template `promise<>`

A `promise<>` provides a mechanism to store a value (or exception) that can later be retrieved from the corresponding `future<>` object. `promise<>` and `future<>` communicate via their underlying shared state.

    #include <boost/fiber/future/promise.hpp>

    namespace boost {
    namespace fibers {

    template< typename R >
    class promise {
    public:
        promise();

        template< typename Allocator >
        promise( std::allocator_arg_t, Allocator);

        promise( promise &&) noexcept;

        promise & operator=( promise &&) noexcept;

        promise( promise const&) = delete;

        promise & operator=( promise const&) = delete;

        ~promise();

        void swap( promise &) noexcept;

        future< R > get_future();

        void set_value( R const&);  // member only of generic promise template
        void set_value( R &&);      // member only of generic promise template
        void set_value( R &);       // member only of promise< R & > template
        void set_value();           // member only of promise< void > template

        void set_exception( std::exception_ptr p);
    };

    template< typename R >
    void swap( promise< R > &, promise< R > &) noexcept;

    }

Default constructor

    promise();

####

Effects:

  Creates a promise with an empty shared state.

Throws:

  Exceptions caused by memory allocation.

Constructor

    template< typename Allocator >
    promise( std::allocator_arg_t, Allocator alloc);

####

Effects:

  Creates a promise with an empty shared state by using `alloc`.

Throws:

  Exceptions caused by memory allocation.

See also:

  `std::allocator_arg_t`

Move constructor

    promise( promise && other) noexcept;

####

Effects:

  Creates a promise by moving the shared state from `other`.

Postcondition:

  `other` contains no valid shared state.

Throws:

  Nothing.

Destructor

    ~promise();

####

Effects:

  Destroys `*this` and abandons the shared state if shared state is ready; otherwise stores `future_error` with error condition `future_errc::broken_promise` as if by `promise::set_exception()`: the shared state is set ready.

Member function `operator=`()

    promise & operator=( promise && other) noexcept;

####

Effects:

  Transfers the ownership of shared state to `*this`.

Postcondition:

  `other` contains no valid shared state.

Throws:

  Nothing.

Member function `swap`()

    void swap( promise & other) noexcept;

####

Effects:

  Swaps the shared state between other and `*this`.

Throws:

  Nothing.

Member function `get_future`()

    future< R > get_future();

####

Returns:

  A `future<>` with the same shared state.

Throws:

  `future_error` with `future_errc::future_already_retrieved` or `future_errc::no_state`.

Member function `set_value`()

    void set_value( R const& value);  // member only of generic promise template
    void set_value( R && value);      // member only of generic promise template
    void set_value( R & value);       // member only of promise< R & > template
    void set_value();                 // member only of promise< void > template

####

Effects:

  Store the result in the shared state and marks the state as ready.

Throws:

  `future_error` with `future_errc::future_already_satisfied` or `future_errc::no_state`.

Member function `set_exception`()

    void set_exception( std::exception_ptr);

####

Effects:

  Store an exception pointer in the shared state and marks the state as ready.

Throws:

  `future_error` with `future_errc::future_already_satisfied` or `future_errc::no_state`.

Non-member function `swap()`

    template< typename R >
    void swap( promise< R > & l, promise< R > & r) noexcept;

####

Effects:

  Same as `l.swap( r)`.

### Template `packaged_task<>`

A `packaged_task<>` wraps a callable target that returns a value so that the return value can be computed asynchronously.

Conventional usage of `packaged_task<>` is like this:

1. Instantiate `packaged_task<>` with template arguments matching the signature of the callable. Pass the callable to the constructor.

2. Call `packaged_task::get_future()` and capture the returned `future<>` instance.

3. Launch a `fiber` to run the new `packaged_task<>`, passing any arguments required by the original callable.

4. Call `fiber::detach()` on the newly-launched `fiber`.

5. At some later point, retrieve the result from the `future<>`.

This is, in fact, pretty much what `fibers::async()` encapsulates.

    #include <boost/fiber/future/packaged_task.hpp>

    namespace boost {
    namespace fibers {

    template< class R, typename ... Args >
    class packaged_task< R( Args ... ) > {
    public:
        packaged_task() noexcept;

        template< typename Fn >
        explicit packaged_task( Fn &&);

        template< typename Fn, typename Allocator >
        packaged_task( std::allocator_arg_t, Allocator const&, Fn &&);

        packaged_task( packaged_task &&) noexcept;

        packaged_task & operator=( packaged_task &&) noexcept;

        packaged_task( packaged_task const&) = delete;

        packaged_task & operator=( packaged_task const&) = delete;

        ~packaged_task();

        void swap( packaged_task &) noexcept;

        bool valid() const noexcept;

        future< R > get_future();

        void operator()( Args ...);

        void reset();
    };

    template< typename Signature >
    void swap( packaged_task< Signature > &, packaged_task< Signature > &) noexcept;

    }}

Default constructor `packaged_task()`

    packaged_task() noexcept;

####

Effects:

  Constructs an object of class `packaged_task` with no shared state.

Throws:

  Nothing.

Templated constructor `packaged_task()`

    template< typename Fn >
    explicit packaged_task( Fn && fn);

    template< typename Fn, typename Allocator >
    packaged_task( std::allocator_arg_t, Allocator const& alloc, Fn && fn);

####

Effects:

  Constructs an object of class `packaged_task` with a shared state and copies or moves the callable target `fn` to internal storage.

Throws:

  Exceptions caused by memory allocation.

Note:

  The signature of `Fn` should have a return type convertible to `R`.

See also:

  `std::allocator_arg_t`

Move constructor

    packaged_task( packaged_task && other) noexcept;

####

Effects:

  Creates a packaged_task by moving the shared state from `other`.

Postcondition:

  `other` contains no valid shared state.

Throws:

  Nothing.

Destructor

    ~packaged_task();

####

Effects:

  Destroys `*this` and abandons the shared state if shared state is ready; otherwise stores `future_error` with error condition `future_errc::broken_promise` as if by `promise::set_exception()`: the shared state is set ready.

Member function `operator=`()

    packaged_task & operator=( packaged_task && other) noexcept;

####

Effects:

  Transfers the ownership of shared state to `*this`.

Postcondition:

  `other` contains no valid shared state.

Throws:

  Nothing.

Member function `swap`()

    void swap( packaged_task & other) noexcept;

####

Effects:

  Swaps the shared state between other and `*this`.

Throws:

  Nothing.

Member function `valid`()

    bool valid() const noexcept;

####

Effects:

  Returns `true` if `*this` contains a shared state.

Throws:

  Nothing.

Member function `get_future`()

    future< R > get_future();

####

Returns:

  A `future<>` with the same shared state.

Throws:

  `future_error` with `future_errc::future_already_retrieved` or `future_errc::no_state`.

Member function `operator()`()

    void operator()( Args && ... args);

####

Effects:

  Invokes the stored callable target. Any exception thrown by the callable target `fn` is stored in the shared state as if by `promise::set_exception()`. Otherwise, the value returned by `fn` is stored in the shared state as if by `promise::set_value()`.

Throws:

  `future_error` with `future_errc::no_state`.

Member function `reset`()

    void reset();

####

Effects:

  Resets the shared state and abandons the result of previous executions. A new shared state is constructed.

Throws:

  `future_error` with `future_errc::no_state`.

Non-member function `swap()`

    template< typename Signature >
    void swap( packaged_task< Signature > & l, packaged_task< Signature > & r) noexcept;

####

Effects:

  Same as `l.swap( r)`.

# Fiber local storage

Synopsis

Fiber local storage allows a separate instance of a given data item for each fiber.

Cleanup at fiber exit

When a fiber exits, the objects associated with each `fiber_specific_ptr` instance are destroyed. By default, the object pointed to by a pointer `p` is destroyed by invoking `delete p`, but this can be overridden for a specific instance of `fiber_specific_ptr` by providing a cleanup routine `func` to the constructor. In this case, the object is destroyed by invoking `func(p)`. The cleanup functions are called in an unspecified order.

Class `fiber_specific_ptr`

    #include <boost/fiber/fss.hpp>

    namespace boost {
    namespace fibers {

    template< typename T >
    class fiber_specific_ptr {
    public:
        typedef T   element_type;

        fiber_specific_ptr();

        explicit fiber_specific_ptr( void(*fn)(T*) );

        ~fiber_specific_ptr();

        fiber_specific_ptr( fiber_specific_ptr const&) = delete;
        fiber_specific_ptr & operator=( fiber_specific_ptr const&) = delete;

        T * get() const noexcept;

        T * operator->() const noexcept;

        T & operator*() const noexcept;

        T * release();

        void reset( T *);
    };

    }}

Constructor

    fiber_specific_ptr();
    explicit fiber_specific_ptr( void(*fn)(T*) );

##

Requires:

  `delete this->get()` is well-formed; `fn(this->get())` does not throw

Effects:

  Construct a `fiber_specific_ptr` object for storing a pointer to an object of type `T` specific to each fiber. When `reset()` is called, or the fiber exits, `fiber_specific_ptr` calls `fn(this->get())`. If the no-arguments constructor is used, the default `delete`-based cleanup function will be used to destroy the fiber-local objects.

Throws:

  `fiber_error` if an error occurs.

Destructor

    ~fiber_specific_ptr();

##

Requires:

  All the fiber specific instances associated to this `fiber_specific_ptr` (except maybe the one associated to this fiber) must be nullptr.

Effects:

  Calls `this->reset()` to clean up the associated value for the current fiber, and destroys `*this`.

Remarks:

  The requirement is an implementation restriction. If the destructor promised to delete instances for all fibers, the implementation would be forced to maintain a list of all the fibers having an associated specific ptr, which is against the goal of fiber specific data. In general, a `fiber_specific_ptr` should outlive the fibers that use it.

Care needs to be taken to ensure that any fibers still running after an instance of `fiber_specific_ptr` has been destroyed do not call any member functions on that instance.

Member function `get`()

    T * get() const noexcept;

##

Returns:

  The pointer associated with the current fiber.

Throws:

  Nothing.

The initial value associated with an instance of `fiber_specific_ptr` is `nullptr` for each fiber.

Member function `operator->`()

    T * operator->() const noexcept;

##

Requires:

  `this->get()` is not `nullptr`.

Returns:

  `this->get()`

Throws:

  Nothing.

Member function `operator*`()

    T & operator*() const noexcept;

##

Requires:

  `this->get()` is not `nullptr`.

Returns:

  `*(this->get())`

Throws:

  Nothing.

Member function `release`()

    T * release();

##

Effects:

  Return `this->get()` and store `nullptr` as the pointer associated with the current fiber without invoking the cleanup function.

Postcondition:

  `this->get()==nullptr`

Throws:

  Nothing.

Member function `reset`()

    void reset( T * new_value);

##

Effects:

  If `this->get()!=new_value` and `this->get()` is not `nullptr`, invoke `delete this->get()` or `fn(this->get())` as appropriate. Store `new_value` as the pointer associated with the current fiber.

Postcondition:

  `this->get()==new_value`

Throws:

  Exception raised during cleanup of previous value.

# Migrating fibers between threads

Overview

Each fiber owns a stack and manages its execution state, including all registers and CPU flags, the instruction pointer and the stack pointer. That means, in general, a fiber is not bound to a specific thread. The main fiber on each thread, that is, the fiber on which the thread is launched, cannot migrate to any other thread. Also Boost.Fiber implicitly creates a dispatcher fiber for each thread — this cannot migrate either. , Of course it would be problematic to migrate a fiber that relies on thread-local storage.

Migrating a fiber from a logical CPU with heavy workload to another logical CPU with a lighter workload might speed up the overall execution. Note that in the case of NUMA-architectures, it is not always advisable to migrate data between threads. Suppose fiber f is running on logical CPU cpu0 which belongs to NUMA node node0. The data of f are allocated on the physical memory located at node0. Migrating the fiber from cpu0 to another logical CPU cpuX which is part of a different NUMA node nodeX might reduce the performance of the application due to increased latency of memory access.

Only fibers that are contained in `algorithm`’s ready queue can migrate between threads. You cannot migrate a running fiber, nor one that is blocked. You cannot migrate a fiber if its `context::is_context()` method returns `true` for `pinned_context`.

In Boost.Fiber a fiber is migrated by invoking `context::detach()` on the thread from which the fiber migrates and `context::attach()` on the thread to which the fiber migrates.

Thus, fiber migration is accomplished by sharing state between instances of a user-coded `algorithm` implementation running on different threads. The fiber’s original thread calls `algorithm::awakened()`, passing the fiber’s `context` `*`. The `awakened()` implementation calls `context::detach()`.

At some later point, when the same or a different thread calls `algorithm::pick_next()`, the `pick_next()` implementation selects a ready fiber and calls `context::attach()` on it before returning it.

As stated above, a `context` for which `is_context(pinned_context) == true` must never be passed to either `context::detach()` or `context::attach()`. It may only be returned from `pick_next()` called by the same thread that passed that context to `awakened()`.

Example of work sharing

In the example work_sharing.cpp multiple worker fibers are created on the main thread. Each fiber gets a character as parameter at construction. This character is printed out ten times. Between each iteration the fiber calls `this_fiber::yield()`. That puts the fiber in the ready queue of the fiber-scheduler shared_ready_queue, running in the current thread. The next fiber ready to be executed is dequeued from the shared ready queue and resumed by shared_ready_queue running on any participating thread.

All instances of shared_ready_queue share one global concurrent queue, used as ready queue. This mechanism shares all worker fibers between all instances of shared_ready_queue, thus between all participating threads.

Setup of threads and fibers

In `main()` the fiber-scheduler is installed and the worker fibers and the threads are launched.

    boost::fibers::use_scheduling_algorithm< boost::fibers::algo::shared_work >();

    for ( char c : std::string("abcdefghijklmnopqrstuvwxyz")) {
        boost::fibers::fiber([c](){ whatevah( c); }).detach();
        ++fiber_count;
    }
    barrier b( 4);
    std::thread threads[] = {
        std::thread( thread, & b),
        std::thread( thread, & b),
        std::thread( thread, & b)
    };
    b.wait();
    {
        lock_type lk( mtx_count);
        cnd_count.wait( lk, [](){ return 0 == fiber_count; } );
    }
    BOOST_ASSERT( 0 == fiber_count);
    for ( std::thread & t : threads) {
        t.join();
    }

Install the scheduling algorithm `boost::fibers::algo::shared_work` in the main thread too, so each new fiber gets launched into the shared pool.

Launch a number of worker fibers; each worker fiber picks up a character that is passed as parameter to fiber-function `whatevah`. Each worker fiber gets detached.

Increment fiber counter for each new fiber.

Launch a couple of threads that join the work sharing.

sync with other threads: allow them to start processing

`lock_type` is typedef'ed as `std::unique_lock`< `std::mutex` >

Suspend main fiber and resume worker fibers in the meanwhile. Main fiber gets resumed (e.g returns from `condition_variable_any::wait()`) if all worker fibers are complete.

Releasing lock of mtx_count is required before joining the threads, otherwise the other threads would be blocked inside condition_variable::wait() and would never return (deadlock).

wait for threads to terminate

The start of the threads is synchronized with a barrier. The main fiber of each thread (including main thread) is suspended until all worker fibers are complete. When the main fiber returns from `condition_variable::wait()`, the thread terminates: the main thread joins all other threads.

    void thread( barrier * b) {
        std::ostringstream buffer;
        buffer << "thread started " << std::this_thread::get_id() << std::endl;
        std::cout << buffer.str() << std::flush;
        boost::fibers::use_scheduling_algorithm< boost::fibers::algo::shared_work >();
        b->wait();
        lock_type lk( mtx_count);
        cnd_count.wait( lk, [](){ return 0 == fiber_count; } );
        BOOST_ASSERT( 0 == fiber_count);
    }

Install the scheduling algorithm `boost::fibers::algo::shared_work` in order to join the work sharing.

sync with other threads: allow them to start processing

Suspend main fiber and resume worker fibers in the meanwhile. Main fiber gets resumed (e.g returns from `condition_variable_any::wait()`) if all worker fibers are complete.

Each worker fiber executes function `whatevah()` with character `me` as parameter. The fiber yields in a loop and prints out a message if it was migrated to another thread.

    void whatevah( char me) {
        try {
            std::thread::id my_thread = std::this_thread::get_id();
            {
                std::ostringstream buffer;
                buffer << "fiber " << me << " started on thread " << my_thread << '\n';
                std::cout << buffer.str() << std::flush;
            }
            for ( unsigned i = 0; i < 10; ++i) {
                boost::this_fiber::yield();
                std::thread::id new_thread = std::this_thread::get_id();
                if ( new_thread != my_thread) {
                    my_thread = new_thread;
                    std::ostringstream buffer;
                    buffer << "fiber " << me << " switched to thread " << my_thread << '\n';
                    std::cout << buffer.str() << std::flush;
                }
            }
        } catch ( ... ) {
        }
        lock_type lk( mtx_count);
        if ( 0 == --fiber_count) {
            lk.unlock();
            cnd_count.notify_all();
        }
    }

get ID of initial thread

loop ten times

yield to other fibers

get ID of current thread

test if fiber was migrated to another thread

Decrement fiber counter for each completed fiber.

Notify all fibers waiting on `cnd_count`.

Scheduling fibers

The fiber scheduler `shared_ready_queue` is like `round_robin`, except that it shares a common ready queue among all participating threads. A thread participates in this pool by executing `use_scheduling_algorithm()` before any other Boost.Fiber operation.

The important point about the ready queue is that it’s a class static, common to all instances of shared_ready_queue. Fibers that are enqueued via `algorithm::awakened()` (fibers that are ready to be resumed) are thus available to all threads. It is required to reserve a separate, scheduler-specific queue for the thread’s main fiber and dispatcher fibers: these may not be shared between threads! When we’re passed either of these fibers, push it there instead of in the shared queue: it would be Bad News for thread B to retrieve and attempt to execute thread A’s main fiber.

[awakened_ws]

When `algorithm::pick_next()` gets called inside one thread, a fiber is dequeued from rqueue_ and will be resumed in that thread.

[pick_next_ws]

The source code above is found in work_sharing.cpp.

# Integrating Fibers with Asynchronous Callbacks

## Overview

One of the primary benefits of Boost.Fiber is the ability to use asynchronous operations for efficiency, while at the same time structuring the calling code as if the operations were synchronous. Asynchronous operations provide completion notification in a variety of ways, but most involve a callback function of some kind. This section discusses tactics for interfacing Boost.Fiber with an arbitrary async operation.

For purposes of illustration, consider the following hypothetical API:

    class AsyncAPI {
    public:
        // constructor acquires some resource that can be read and written
        AsyncAPI();

        // callbacks accept an int error code; 0 == success
        typedef int errorcode;

        // write callback only needs to indicate success or failure
        template< typename Fn >
        void init_write( std::string const& data, Fn && callback);

        // read callback needs to accept both errorcode and data
        template< typename Fn >
        void init_read( Fn && callback);

        // ... other operations ...
    };

The significant points about each of `init_write()` and `init_read()` are:

- The `AsyncAPI` method only initiates the operation. It returns immediately, while the requested operation is still pending.

- The method accepts a callback. When the operation completes, the callback is called with relevant parameters (error code, data if applicable).

We would like to wrap these asynchronous methods in functions that appear synchronous by blocking the calling fiber until the operation completes. This lets us use the wrapper function’s return value to deliver relevant data.

`promise<>` and `future<>` are your friends here.

## Return Errorcode

The `AsyncAPI::init_write()` callback passes only an `errorcode`. If we simply want the blocking wrapper to return that `errorcode`, this is an extremely straightforward use of `promise<>` and `future<>`:

    AsyncAPI::errorcode write_ec( AsyncAPI & api, std::string const& data) {
        boost::fibers::promise< AsyncAPI::errorcode > promise;
        boost::fibers::future< AsyncAPI::errorcode > future( promise.get_future() );
        // In general, even though we block waiting for future::get() and therefore
        // won't destroy 'promise' until promise::set_value() has been called, we
        // are advised that with threads it's possible for ~promise() to be
        // entered before promise::set_value() has returned. While that shouldn't
        // happen with fibers::promise, a robust way to deal with the lifespan
        // issue is to bind 'promise' into our lambda. Since promise is move-only,
        // use initialization capture.
    #if ! defined(BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES)
        api.init_write(
            data,
            [promise=std::move( promise)]( AsyncAPI::errorcode ec) mutable {
                                promise.set_value( ec);
                      });

    #else // defined(BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES)
        api.init_write(
            data,
            std::bind([](boost::fibers::promise< AsyncAPI::errorcode > & promise,
                         AsyncAPI::errorcode ec) {
                promise.set_value( ec);
            },
            std::move( promise),
            std::placeholders::_1) );
    #endif // BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES

        return future.get();
    }

All we have to do is:

1. Instantiate a `promise<>` of correct type.

2. Obtain its `future<>`.

3. Arrange for the callback to call `promise::set_value()`.

4. Block on `future::get()`.

This tactic for resuming a pending fiber works even if the callback is called on a different thread than the one on which the initiating fiber is running. In fact, the example program’s dummy `AsyncAPI` implementation illustrates that: it simulates async I/O by launching a new thread that sleeps briefly and then calls the relevant callback.

## Success or Exception

A wrapper more aligned with modern C++ practice would use an exception, rather than an `errorcode`, to communicate failure to its caller. This is straightforward to code in terms of `write_ec()`:

    void write( AsyncAPI & api, std::string const& data) {
        AsyncAPI::errorcode ec = write_ec( api, data);
        if ( ec) {
            throw make_exception("write", ec);
        }
    }

The point is that since each fiber has its own stack, you need not repeat messy boilerplate: normal encapsulation works.

## Return Errorcode or Data

Things get a bit more interesting when the async operation’s callback passes multiple data items of interest. One approach would be to use `std::pair<>` to capture both:

    std::pair< AsyncAPI::errorcode, std::string > read_ec( AsyncAPI & api) {
        typedef std::pair< AsyncAPI::errorcode, std::string > result_pair;
        boost::fibers::promise< result_pair > promise;
        boost::fibers::future< result_pair > future( promise.get_future() );
        // We promise that both 'promise' and 'future' will survive until our
        // lambda has been called.
    #if ! defined(BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES)
        api.init_read([promise=std::move( promise)]( AsyncAPI::errorcode ec, std::string const& data) mutable {
                                promise.set_value( result_pair( ec, data) );
                      });
    #else // defined(BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES)
        api.init_read(
                std::bind([]( boost::fibers::promise< result_pair > & promise,
                              AsyncAPI::errorcode ec, std::string const& data) mutable {
                                promise.set_value( result_pair( ec, data) );
                      },
                      std::move( promise),
                      std::placeholders::_1,
                      std::placeholders::_2) );
    #endif // BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES
        return future.get();
    }

Once you bundle the interesting data in `std::pair<>`, the code is effectively identical to `write_ec()`. You can call it like this:

    std::tie( ec, data) = read_ec( api);

## Data or Exception

But a more natural API for a function that obtains data is to return only the data on success, throwing an exception on error.

As with `write()` above, it’s certainly possible to code a `read()` wrapper in terms of `read_ec()`. But since a given application is unlikely to need both, let’s code `read()` from scratch, leveraging `promise::set_exception()`:

    std::string read( AsyncAPI & api) {
        boost::fibers::promise< std::string > promise;
        boost::fibers::future< std::string > future( promise.get_future() );
        // Both 'promise' and 'future' will survive until our lambda has been
        // called.
    #if ! defined(BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES)
        api.init_read([&promise]( AsyncAPI::errorcode ec, std::string const& data) mutable {
                               if ( ! ec) {
                                   promise.set_value( data);
                               } else {
                                   promise.set_exception(
                                           std::make_exception_ptr(
                                               make_exception("read", ec) ) );
                               }
                      });
    #else // defined(BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES)
        api.init_read(
                std::bind([]( boost::fibers::promise< std::string > & promise,
                              AsyncAPI::errorcode ec, std::string const& data) mutable {
                               if ( ! ec) {
                                   promise.set_value( data);
                               } else {
                                   promise.set_exception(
                                           std::make_exception_ptr(
                                               make_exception("read", ec) ) );
                               }
                      },
                      std::move( promise),
                      std::placeholders::_1,
                      std::placeholders::_2) );
    #endif // BOOST_NO_CXX14_INITIALIZED_LAMBDA_CAPTURES
        return future.get();
    }

`future::get()` will do the right thing, either returning `std::string` or throwing an exception.

## Success/Error Virtual Methods

One classic approach to completion notification is to define an abstract base class with `success()` and `error()` methods. Code wishing to perform async I/O must derive a subclass, override each of these methods and pass the async operation a pointer to a subclass instance. The abstract base class might look like this:

    // every async operation receives a subclass instance of this abstract base
    // class through which to communicate its result
    struct Response {
        typedef std::shared_ptr< Response > ptr;

        // called if the operation succeeds
        virtual void success( std::string const& data) = 0;

        // called if the operation fails
        virtual void error( AsyncAPIBase::errorcode ec) = 0;
    };

Now the `AsyncAPI` operation might look more like this:

    // derive Response subclass, instantiate, pass Response::ptr
    void init_read( Response::ptr);

We can address this by writing a one-size-fits-all `PromiseResponse`:

    class PromiseResponse: public Response {
    public:
        // called if the operation succeeds
        virtual void success( std::string const& data) {
            promise_.set_value( data);
        }

        // called if the operation fails
        virtual void error( AsyncAPIBase::errorcode ec) {
            promise_.set_exception(
                    std::make_exception_ptr(
                        make_exception("read", ec) ) );
        }

        boost::fibers::future< std::string > get_future() {
            return promise_.get_future();
        }

    private:
        boost::fibers::promise< std::string >   promise_;
    };

Now we can simply obtain the `future<>` from that `PromiseResponse` and wait on its `get()`:

    std::string read( AsyncAPI & api) {
        // Because init_read() requires a shared_ptr, we must allocate our
        // ResponsePromise on the heap, even though we know its lifespan.
        auto promisep( std::make_shared< PromiseResponse >() );
        boost::fibers::future< std::string > future( promisep->get_future() );
        // Both 'promisep' and 'future' will survive until our lambda has been
        // called.
        api.init_read( promisep);
        return future.get();
    }

The source code above is found in adapt_callbacks.cpp and adapt_method_calls.cpp.

## Then There’s Boost.Asio

Since the simplest form of Boost.Asio asynchronous operation completion token is a callback function, we could apply the same tactics for Asio as for our hypothetical `AsyncAPI` asynchronous operations.

Fortunately we need not. Boost.Asio incorporates a mechanism This mechanism has been proposed as a conventional way to allow the caller of an arbitrary async function to specify completion handling: N4045. by which the caller can customize the notification behavior of any async operation. Therefore we can construct a completion token which, when passed to a Boost.Asio async operation, requests blocking for the calling fiber.

A typical Asio async function might look something like this: per N4045

    template < ..., class CompletionToken >
    deduced_return_type
    async_something( ... , CompletionToken&& token)
    {
        // construct handler_type instance from CompletionToken
        handler_type<CompletionToken, ...>::type handler(token);
        // construct async_result instance from handler_type
        async_result<decltype(handler)> result(handler);

        // ... arrange to call handler on completion ...
        // ... initiate actual I/O operation ...

        return result.get();
    }

We will engage that mechanism, which is based on specializing Asio’s `handler_type<>` template for the `CompletionToken` type and the signature of the specific callback. The remainder of this discussion will refer back to `async_something()` as the Asio async function under consideration.

The implementation described below uses lower-level facilities than `promise` and `future` because the `promise` mechanism interacts badly with `io_service::stop()`. It produces `broken_promise` exceptions.

`boost::fibers::asio::yield` is a completion token of this kind. `yield` is an instance of `yield_t`:

    class yield_t {
    public:
        yield_t() = default;

        /**
         * @code
         * static yield_t yield;
         * boost::system::error_code myec;
         * func(yield[myec]);
         * @endcode
         * @c yield[myec] returns an instance of @c yield_t whose @c ec_ points
         * to @c myec. The expression @c yield[myec] "binds" @c myec to that
         * (anonymous) @c yield_t instance, instructing @c func() to store any
         * @c error_code it might produce into @c myec rather than throwing @c
         * boost::system::system_error.
         */
        yield_t operator[]( boost::system::error_code & ec) const {
            yield_t tmp;
            tmp.ec_ = & ec;
            return tmp;
        }

    //private:
        // ptr to bound error_code instance if any
        boost::system::error_code   *   ec_{ nullptr };
    };

`yield_t` is in fact only a placeholder, a way to trigger Boost.Asio customization. It can bind a `boost::system::error_code` for use by the actual handler.

`yield` is declared as:

    // canonical instance
    thread_local yield_t yield{};

Asio customization is engaged by specializing `boost::asio::handler_type<>` for `yield_t`:

    // Handler type specialisation for fibers::asio::yield.
    // When 'yield' is passed as a completion handler which accepts only
    // error_code, use yield_handler<void>. yield_handler will take care of the
    // error_code one way or another.
    template< typename ReturnType >
    struct handler_type< fibers::asio::yield_t, ReturnType( boost::system::error_code) >
    { typedef fibers::asio::detail::yield_handler< void >    type; };

(There are actually four different specializations in detail/yield.hpp, one for each of the four Asio async callback signatures we expect.)

The above directs Asio to use `yield_handler` as the actual handler for an async operation to which `yield` is passed. There’s a generic `yield_handler<T>` implementation and a `yield_handler<void>` specialization. Let’s start with the `<void>` specialization:

    // yield_handler<void> is like yield_handler<T> without value_. In fact it's
    // just like yield_handler_base.
    template<>
    class yield_handler< void >: public yield_handler_base {
    public:
        explicit yield_handler( yield_t const& y) :
            yield_handler_base{ y } {
        }

        // nullary completion callback
        void operator()() {
            ( * this)( boost::system::error_code() );
        }

        // inherit operator()(error_code) overload from base class
        using yield_handler_base::operator();
    };

`async_something()`, having consulted the `handler_type<>` traits specialization, instantiates a `yield_handler<void>` to be passed as the actual callback for the async operation. `yield_handler`’s constructor accepts the `yield_t` instance (the `yield` object passed to the async function) and passes it along to `yield_handler_base`:

    // This class encapsulates common elements between yield_handler<T> (capturing
    // a value to return from asio async function) and yield_handler<void> (no
    // such value). See yield_handler<T> and its <void> specialization below. Both
    // yield_handler<T> and yield_handler<void> are passed by value through
    // various layers of asio functions. In other words, they're potentially
    // copied multiple times. So key data such as the yield_completion instance
    // must be stored in our async_result<yield_handler<>> specialization, which
    // should be instantiated only once.
    class yield_handler_base {
    public:
        yield_handler_base( yield_t const& y) :
            // capture the context* associated with the running fiber
            ctx_{ boost::fibers::context::active() },
            // capture the passed yield_t
            yt_( y ) {
        }

        // completion callback passing only (error_code)
        void operator()( boost::system::error_code const& ec) {
            BOOST_ASSERT_MSG( ycomp_,
                              "Must inject yield_completion* "
                              "before calling yield_handler_base::operator()()");
            BOOST_ASSERT_MSG( yt_.ec_,
                              "Must inject boost::system::error_code* "
                              "before calling yield_handler_base::operator()()");
            // If originating fiber is busy testing state_ flag, wait until it
            // has observed (completed != state_).
            yield_completion::lock_t lk{ ycomp_->mtx_ };
            yield_completion::state_t state = ycomp_->state_;
            // Notify a subsequent yield_completion::wait() call that it need not
            // suspend.
            ycomp_->state_ = yield_completion::complete;
            // set the error_code bound by yield_t
            * yt_.ec_ = ec;
            // unlock the lock that protects state_
            lk.unlock();
            // If ctx_ is still active, e.g. because the async operation
            // immediately called its callback (this method!) before the asio
            // async function called async_result_base::get(), we must not set it
            // ready.
            if ( yield_completion::waiting == state) {
                // wake the fiber
                fibers::context::active()->schedule( ctx_);
            }
        }

    //private:
        boost::fibers::context      *   ctx_;
        yield_t                         yt_;
        // We depend on this pointer to yield_completion, which will be injected
        // by async_result.
        yield_completion::ptr_t         ycomp_{};
    };

`yield_handler_base` stores a copy of the `yield_t` instance — which, as shown above, contains only an `error_code*`. It also captures the `context`* for the currently-running fiber by calling `context::active()`.

You will notice that `yield_handler_base` has one more data member (`ycomp_`) that is initialized to `nullptr` by its constructor — though its `operator()()` method relies on `ycomp_` being non-null. More on this in a moment.

Having constructed the `yield_handler<void>` instance, `async_something()` goes on to construct an `async_result` specialized for the `handler_type<>::type`: in this case, `async_result<yield_handler<void>>`. It passes the `yield_handler<void>` instance to the new `async_result` instance.

    // Without the need to handle a passed value, our yield_handler<void>
    // specialization is just like async_result_base.
    template<>
    class async_result< boost::fibers::asio::detail::yield_handler< void > > :
        public boost::fibers::asio::detail::async_result_base {
    public:
        typedef void type;

        explicit async_result( boost::fibers::asio::detail::yield_handler< void > & h):
            boost::fibers::asio::detail::async_result_base{ h } {
        }
    };

Naturally that leads us straight to `async_result_base`:

    // Factor out commonality between async_result<yield_handler<T>> and
    // async_result<yield_handler<void>>
    class async_result_base {
    public:
        explicit async_result_base( yield_handler_base & h) :
                ycomp_{ new yield_completion{} } {
            // Inject ptr to our yield_completion instance into this
            // yield_handler<>.
            h.ycomp_ = this->ycomp_;
            // if yield_t didn't bind an error_code, make yield_handler_base's
            // error_code* point to an error_code local to this object so
            // yield_handler_base::operator() can unconditionally store through
            // its error_code*
            if ( ! h.yt_.ec_) {
                h.yt_.ec_ = & ec_;
            }
        }

        void get() {
            // Unless yield_handler_base::operator() has already been called,
            // suspend the calling fiber until that call.
            ycomp_->wait();
            // The only way our own ec_ member could have a non-default value is
            // if our yield_handler did not have a bound error_code AND the
            // completion callback passed a non-default error_code.
            if ( ec_) {
                throw_exception( boost::system::system_error{ ec_ } );
            }
        }

    private:
        // If yield_t does not bind an error_code instance, store into here.
        boost::system::error_code       ec_{};
        yield_completion::ptr_t         ycomp_;
    };

This is how `yield_handler_base::ycomp_` becomes non-null: `async_result_base`’s constructor injects a pointer back to its own `yield_completion` member.

Recall that the canonical `yield_t` instance `yield` initializes its `error_code*` member `ec_` to `nullptr`. If this instance is passed to `async_something()` (`ec_` is still `nullptr`), the copy stored in `yield_handler_base` will likewise have null `ec_`. `async_result_base`’s constructor sets `yield_handler_base`’s `yield_t`’s `ec_` member to point to its own `error_code` member.

The stage is now set. `async_something()` initiates the actual async operation, arranging to call its `yield_handler<void>` instance on completion. Let’s say, for the sake of argument, that the actual async operation’s callback has signature `void(error_code)`.

But since it’s an async operation, control returns at once to `async_something()`. `async_something()` calls `async_result<yield_handler<void>>::get()`, and will return its return value.

`async_result<yield_handler<void>>::get()` inherits `async_result_base::get()`.

`async_result_base::get()` immediately calls `yield_completion::wait()`.

    // Bundle a completion bool flag with a spinlock to protect it.
    struct yield_completion {
        enum state_t {
            init,
            waiting,
            complete
        };

        typedef fibers::detail::spinlock                    mutex_t;
        typedef std::unique_lock< mutex_t >                 lock_t;
        typedef boost::intrusive_ptr< yield_completion >    ptr_t;

        std::atomic< std::size_t >  use_count_{ 0 };
        mutex_t                     mtx_{};
        state_t                     state_{ init };

        void wait() {
            // yield_handler_base::operator()() will set state_ `complete` and
            // attempt to wake a suspended fiber. It would be Bad if that call
            // happened between our detecting (complete != state_) and suspending.
            lock_t lk{ mtx_ };
            // If state_ is already set, we're done here: don't suspend.
            if ( complete != state_) {
                state_ = waiting;
                // suspend(unique_lock<spinlock>) unlocks the lock in the act of
                // resuming another fiber
                fibers::context::active()->suspend( lk);
            }
        }

        friend void intrusive_ptr_add_ref( yield_completion * yc) noexcept {
            BOOST_ASSERT( nullptr != yc);
            yc->use_count_.fetch_add( 1, std::memory_order_relaxed);
        }

        friend void intrusive_ptr_release( yield_completion * yc) noexcept {
            BOOST_ASSERT( nullptr != yc);
            if ( 1 == yc->use_count_.fetch_sub( 1, std::memory_order_release) ) {
                std::atomic_thread_fence( std::memory_order_acquire);
                delete yc;
            }
        }
    };

Supposing that the pending async operation has not yet completed, `yield_completion::completed_` will still be `false`, and `wait()` will call `context::suspend()` on the currently-running fiber.

Other fibers will now have a chance to run.

Some time later, the async operation completes. It calls `yield_handler<void>::operator()(error_code const&)` with an `error_code` indicating either success or failure. We’ll consider both cases.

`yield_handler<void>` explicitly inherits `operator()(error_code const&)` from `yield_handler_base`.

`yield_handler_base::operator()(error_code const&)` first sets `yield_completion::completed_` `true`. This way, if `async_something()`’s async operation completes immediately — if `yield_handler_base::operator()` is called even before `async_result_base::get()` — the calling fiber will not suspend.

The actual `error_code` produced by the async operation is then stored through the stored `yield_t::ec_` pointer. If `async_something()`’s caller used (e.g.) `yield[my_ec]` to bind a local `error_code` instance, the actual `error_code` value is stored into the caller’s variable. Otherwise, it is stored into `async_result_base::ec_`.

If the stored fiber context `yield_handler_base::ctx_` is not already running, it is marked as ready to run by passing it to `context::set_ready()`. Control then returns from `yield_handler_base::operator()`: the callback is done.

In due course, that fiber is resumed. Control returns from `context::suspend()` to `yield_completion::wait()`, which returns to `async_result_base::get()`.

- If the original caller passed `yield[my_ec]` to `async_something()` to bind a local `error_code` instance, then `yield_handler_base::operator()` stored its `error_code` to the caller’s `my_ec` instance, leaving `async_result_base::ec_` initialized to success.

- If the original caller passed `yield` to `async_something()` without binding a local `error_code` variable, then `yield_handler_base::operator()` stored its `error_code` into `async_result_base::ec_`. If in fact that `error_code` is success, then all is well.

- Otherwise — the original caller did not bind a local `error_code` and `yield_handler_base::operator()` was called with an `error_code` indicating error — `async_result_base::get()` throws `system_error` with that `error_code`.

The case in which `async_something()`’s completion callback has signature `void()` is similar. `yield_handler<void>::operator()()` invokes the machinery above with a success `error_code`.

A completion callback with signature `void(error_code, T)` (that is: in addition to `error_code`, callback receives some data item) is handled somewhat differently. For this kind of signature, `handler_type<>::type` specifies `yield_handler<T>` (for `T` other than `void`).

A `yield_handler<T>` reserves a `value_` pointer to a value of type `T`:

    // asio uses handler_type<completion token type, signature>::type to decide
    // what to instantiate as the actual handler. Below, we specialize
    // handler_type< yield_t, ... > to indicate yield_handler<>. So when you pass
    // an instance of yield_t as an asio completion token, asio selects
    // yield_handler<> as the actual handler class.
    template< typename T >
    class yield_handler: public yield_handler_base {
    public:
        // asio passes the completion token to the handler constructor
        explicit yield_handler( yield_t const& y) :
            yield_handler_base{ y } {
        }

        // completion callback passing only value (T)
        void operator()( T t) {
            // just like callback passing success error_code
            (*this)( boost::system::error_code(), std::move(t) );
        }

        // completion callback passing (error_code, T)
        void operator()( boost::system::error_code const& ec, T t) {
            BOOST_ASSERT_MSG( value_,
                              "Must inject value ptr "
                              "before caling yield_handler<T>::operator()()");
            // move the value to async_result<> instance BEFORE waking up a
            // suspended fiber
            * value_ = std::move( t);
            // forward the call to base-class completion handler
            yield_handler_base::operator()( ec);
        }

    //private:
        // pointer to destination for eventual value
        // this must be injected by async_result before operator()() is called
        T   *   value_{ nullptr };
    };

This pointer is initialized to `nullptr`.

When `async_something()` instantiates `async_result<yield_handler<T>>`:

    // asio constructs an async_result<> instance from the yield_handler specified
    // by handler_type<>::type. A particular asio async method constructs the
    // yield_handler, constructs this async_result specialization from it, then
    // returns the result of calling its get() method.
    template< typename T >
    class async_result< boost::fibers::asio::detail::yield_handler< T > > :
        public boost::fibers::asio::detail::async_result_base {
    public:
        // type returned by get()
        typedef T type;

        explicit async_result( boost::fibers::asio::detail::yield_handler< T > & h) :
            boost::fibers::asio::detail::async_result_base{ h } {
            // Inject ptr to our value_ member into yield_handler<>: result will
            // be stored here.
            h.value_ = & value_;
        }

        // asio async method returns result of calling get()
        type get() {
            boost::fibers::asio::detail::async_result_base::get();
            return std::move( value_);
        }

    private:
        type                            value_{};
    };

this `async_result<>` specialization reserves a member of type `T` to receive the passed data item, and sets `yield_handler<T>::value_` to point to its own data member.

`async_result<yield_handler<T>>` overrides `get()`. The override calls `async_result_base::get()`, so the calling fiber suspends as described above.

`yield_handler<T>::operator()(error_code, T)` stores its passed `T` value into `async_result<yield_handler<T>>::value_`.

Then it passes control to `yield_handler_base::operator()(error_code)` to deal with waking the original fiber as described above.

When `async_result<yield_handler<T>>::get()` resumes, it returns the stored `value_` to `async_something()` and ultimately to `async_something()`’s caller.

The case of a callback signature `void(T)` is handled by having `yield_handler<T>::operator()(T)` engage the `void(error_code, T)` machinery, passing a success `error_code`.

The source code above is found in yield.hpp and detail/yield.hpp.

# Integrating Fibers with Nonblocking I/O

Overview

Nonblocking I/O is distinct from asynchronous I/O. A true async I/O operation promises to initiate the operation and notify the caller on completion, usually via some sort of callback (as described in Integrating Fibers with Asynchronous Callbacks).

In contrast, a nonblocking I/O operation refuses to start at all if it would be necessary to block, returning an error code such as `EWOULDBLOCK`. The operation is performed only when it can complete immediately. In effect, the caller must repeatedly retry the operation until it stops returning `EWOULDBLOCK`.

In a classic event-driven program, it can be something of a headache to use nonblocking I/O. At the point where the nonblocking I/O is attempted, a return value of `EWOULDBLOCK` requires the caller to pass control back to the main event loop, arranging to retry again on the next iteration.

Worse, a nonblocking I/O operation might partially succeed. That means that the relevant business logic must continue receiving control on every main loop iteration until all required data have been processed: a doubly-nested loop, implemented as a callback-driven state machine.

Boost.Fiber can simplify this problem immensely. Once you have integrated with the application's main loop as described in Sharing a Thread with Another Main Loop, waiting for the next main-loop iteration is as simple as calling `this_fiber::yield()`.

Example Nonblocking API

For purposes of illustration, consider this API:

    class NonblockingAPI {
    public:
        NonblockingAPI();

        // nonblocking operation: may return EWOULDBLOCK
        int read( std::string & data, std::size_t desired);

        ...
    };

Polling for Completion

We can build a low-level wrapper around `NonblockingAPI::read()` that shields its caller from ever having to deal with `EWOULDBLOCK`:

    // guaranteed not to return EWOULDBLOCK
    int read_chunk( NonblockingAPI & api, std::string & data, std::size_t desired) {
        int error;
        while ( EWOULDBLOCK == ( error = api.read( data, desired) ) ) {
            // not ready yet -- try again on the next iteration of the
            // application's main loop
            boost::this_fiber::yield();
        }
        return error;
    }

Filling All Desired Data

Given `read_chunk()`, we can straightforwardly iterate until we have all desired data:

    // keep reading until desired length, EOF or error
    // may return both partial data and nonzero error
    int read_desired( NonblockingAPI & api, std::string & data, std::size_t desired) {
        // we're going to accumulate results into 'data'
        data.clear();
        std::string chunk;
        int error = 0;
        while ( data.length() < desired &&
               ( error = read_chunk( api, chunk, desired - data.length() ) ) == 0) {
            data.append( chunk);
        }
        return error;
    }

(Of course there are more efficient ways to accumulate string data. That's not the point of this example.)

Wrapping it Up

Finally, we can define a relevant exception:

    // exception class augmented with both partially-read data and errorcode
    class IncompleteRead : public std::runtime_error {
    public:
        IncompleteRead( std::string const& what, std::string const& partial, int ec) :
            std::runtime_error( what),
            partial_( partial),
            ec_( ec) {
        }

        std::string get_partial() const {
            return partial_;
        }

        int get_errorcode() const {
            return ec_;
        }

    private:
        std::string partial_;
        int         ec_;
    };

and write a simple `read()` function that either returns all desired data or throws `IncompleteRead`:

    // read all desired data or throw IncompleteRead
    std::string read( NonblockingAPI & api, std::size_t desired) {
        std::string data;
        int ec( read_desired( api, data, desired) );

        // for present purposes, EOF isn't a failure
        if ( 0 == ec || EOF == ec) {
            return data;
        }

        // oh oh, partial read
        std::ostringstream msg;
        msg << "NonblockingAPI::read() error " << ec << " after "
            << data.length() << " of " << desired << " characters";
        throw IncompleteRead( msg.str(), data, ec);
    }

Once we can transparently wait for the next main-loop iteration using `this_fiber::yield()`, ordinary encapsulation Just Works.

The source code above is found in adapt_nonblocking.cpp.

# when_any / when_all functionality

Overview

A bit of wisdom from the early days of computing still holds true today: prefer to model program state using the instruction pointer rather than with Boolean flags. In other words, if the program must do something and then do something almost the same, but with minor changes... perhaps parts of that something should be broken out as smaller separate functions, rather than introducing flags to alter the internal behavior of a monolithic function.

To that we would add: prefer to describe control flow using C++ native constructs such as function calls, `if`, `while`, `for`, `do` et al. rather than as chains of callbacks.

One of the great strengths of Boost.Fiber is the flexibility it confers on the coder to restructure an application from chains of callbacks to straightforward C++ statement sequence, even when code in that fiber is in fact interleaved with code running in other fibers.

There has been much recent discussion about the benefits of when_any and when_all functionality. When dealing with asynchronous and possibly unreliable services, these are valuable idioms. But of course when_any and when_all are closely tied to the use of chains of callbacks.

This section presents recipes for achieving the same ends, in the context of a fiber that wants to do something when one or more other independent activities have completed. Accordingly, these are `wait_something()` functions rather than `when_something()` functions. The expectation is that the calling fiber asks to launch those independent activities, then waits for them, then sequentially proceeds with whatever processing depends on those results.

The function names shown (e.g. `wait_first_simple()`) are for illustrative purposes only, because all these functions have been bundled into a single source file. Presumably, if (say) `wait_first_success()` best suits your application needs, you could introduce that variant with the name `wait_any()`.

The functions presented in this section accept variadic argument lists of task functions. Corresponding `wait_something()` functions accepting a container of task functions are left as an exercise for the interested reader. Those should actually be simpler. Most of the complexity would arise from overloading the same name for both purposes.

All the source code for this section is found in wait_stuff.cpp.

Example Task Function

We found it convenient to model an asynchronous task using this function:

    template< typename T >
    T sleeper_impl( T item, int ms, bool thrw = false) {
        std::ostringstream descb, funcb;
        descb << item;
        std::string desc( descb.str() );
        funcb << "  sleeper(" << item << ")";
        Verbose v( funcb.str() );

        boost::this_fiber::sleep_for( std::chrono::milliseconds( ms) );
        if ( thrw) {
            throw std::runtime_error( desc);
        }
        return item;
    }

with type-specific `sleeper()` front ends for `std::string`, `double` and `int`.

`Verbose` simply prints a message to `std::cout` on construction and destruction.

Basically:

1. `sleeper()` prints a start message;

2. sleeps for the specified number of milliseconds;

3. if `thrw` is passed as `true`, throws a string description of the passed `item`;

4. else returns the passed `item`.

5. On the way out, `sleeper()` produces a stop message.

This function will feature in the example calls to the various functions presented below.

## when_any

### when_any, simple completion

The simplest case is when you only need to know that the first of a set of asynchronous tasks has completed — but you don't need to obtain a return value, and you're confident that they will not throw exceptions.

For this we introduce a `Done` class to wrap a `bool` variable with a `condition_variable` and a `mutex`:

    // Wrap canonical pattern for condition_variable + bool flag
    struct Done {
    private:
        boost::fibers::condition_variable   cond;
        boost::fibers::mutex                mutex;
        bool                                ready = false;

    public:
        typedef std::shared_ptr< Done >     ptr;

        void wait() {
            std::unique_lock< boost::fibers::mutex > lock( mutex);
            cond.wait( lock, [this](){ return ready; });
        }

        void notify() {
            {
                std::unique_lock< boost::fibers::mutex > lock( mutex);
                ready = true;
            } // release mutex
            cond.notify_one();
        }
    };

The pattern we follow throughout this section is to pass a `std::shared_ptr<>` to the relevant synchronization object to the various tasks' fiber functions. This eliminates nagging questions about the lifespan of the synchronization object relative to the last of the fibers.

`wait_first_simple()` uses that tactic for `Done`:

    template< typename ... Fns >
    void wait_first_simple( Fns && ... functions) {
        // Use shared_ptr because each function's fiber will bind it separately,
        // and we're going to return before the last of them completes.
        auto done( std::make_shared< Done >() );
        wait_first_simple_impl( done, std::forward< Fns >( functions) ... );
        done->wait();
    }

`wait_first_simple_impl()` is an ordinary recursion over the argument pack, capturing `Done::ptr` for each new fiber:

    // Degenerate case: when there are no functions to wait for, return
    // immediately.
    void wait_first_simple_impl( Done::ptr) {
    }

    // When there's at least one function to wait for, launch it and recur to
    // process the rest.
    template< typename Fn, typename ... Fns >
    void wait_first_simple_impl( Done::ptr done, Fn && function, Fns && ... functions) {
        boost::fibers::fiber( [done, function](){
                                  function();
                                  done->notify();
                              }).detach();
        wait_first_simple_impl( done, std::forward< Fns >( functions) ... );
    }

The body of the fiber's lambda is extremely simple, as promised: call the function, notify `Done` when it returns. The first fiber to do so allows `wait_first_simple()` to return — which is why it's useful to have `std::shared_ptr<Done>` manage the lifespan of our `Done` object rather than declaring it as a stack variable in `wait_first_simple()`.

This is how you might call it:

    wait_first_simple(
            [](){ sleeper("wfs_long",   150); },
            [](){ sleeper("wfs_medium", 100); },
            [](){ sleeper("wfs_short",   50); });

In this example, control resumes after `wait_first_simple()` when `sleeper("wfs_short", 50)` completes — even though the other two `sleeper()` fibers are still running.

### when_any, return value

It seems more useful to add the ability to capture the return value from the first of the task functions to complete. Again, we assume that none will throw an exception.

One tactic would be to adapt our `Done` class to store the first of the return values, rather than a simple `bool`. However, we choose instead to use a `buffered_channel<>`. We'll only need to enqueue the first value, so we'll `buffered_channel::close()` it once we've retrieved that value. Subsequent `push()` calls will return `closed`.

    // Assume that all passed functions have the same return type. The return type
    // of wait_first_value() is the return type of the first passed function. It is
    // simply invalid to pass NO functions.
    template< typename Fn, typename ... Fns >
    typename std::result_of< Fn() >::type
    wait_first_value( Fn && function, Fns && ... functions) {
        typedef typename std::result_of< Fn() >::type return_t;
        typedef boost::fibers::buffered_channel< return_t > channel_t;
        auto chanp( std::make_shared< channel_t >( 64) );
        // launch all the relevant fibers
        wait_first_value_impl< return_t >( chanp,
                                           std::forward< Fn >( function),
                                           std::forward< Fns >( functions) ... );
        // retrieve the first value
        return_t value( chanp->value_pop() );
        // close the channel: no subsequent push() has to succeed
        chanp->close();
        return value;
    }

The meat of the `wait_first_value_impl()` function is as you might expect:

    template< typename T, typename Fn >
    void wait_first_value_impl( std::shared_ptr< boost::fibers::buffered_channel< T > > chan,
                                Fn && function) {
        boost::fibers::fiber( [chan, function](){
                                  // Ignore channel_op_status returned by push():
                                  // might be closed; we simply don't care.
                                  chan->push( function() );
                              }).detach();
    }

It calls the passed function, pushes its return value and ignores the `push()` result. You might call it like this:

    std::string result = wait_first_value(
            [](){ return sleeper("wfv_third",  150); },
            [](){ return sleeper("wfv_second", 100); },
            [](){ return sleeper("wfv_first",   50); });
    std::cout << "wait_first_value() => " << result << std::endl;
    assert(result == "wfv_first");

### when_any, produce first outcome, whether result or exception

We may not be running in an environment in which we can guarantee no exception will be thrown by any of our task functions. In that case, the above implementations of `wait_first_something()` would be naïve: as mentioned in the section on Fiber Management, an uncaught exception in one of our task fibers would cause `std::terminate()` to be called.

Let's at least ensure that such an exception would propagate to the fiber awaiting the first result. We can use `future<>` to transport either a return value or an exception. Therefore, we will change `wait_first_value()`'s `buffered_channel<>` to hold `future< T >` items instead of simply `T`.

Once we have a `future<>` in hand, all we need do is call `future::get()`, which will either return the value or rethrow the exception.

    template< typename Fn, typename ... Fns >
    typename std::result_of< Fn() >::type
    wait_first_outcome( Fn && function, Fns && ... functions) {
        // In this case, the value we pass through the channel is actually a
        // future -- which is already ready. future can carry either a value or an
        // exception.
        typedef typename std::result_of< Fn() >::type return_t;
        typedef boost::fibers::future< return_t > future_t;
        typedef boost::fibers::buffered_channel< future_t > channel_t;
        auto chanp(std::make_shared< channel_t >( 64) );
        // launch all the relevant fibers
        wait_first_outcome_impl< return_t >( chanp,
                                             std::forward< Fn >( function),
                                             std::forward< Fns >( functions) ... );
        // retrieve the first future
        future_t future( chanp->value_pop() );
        // close the channel: no subsequent push() has to succeed
        chanp->close();
        // either return value or throw exception
        return future.get();
    }

So far so good — but there's a timing issue. How should we obtain the `future<>` to `buffered_channel::push()` on the queue?

We could call `fibers::async()`. That would certainly produce a `future<>` for the task function. The trouble is that it would return too quickly! We only want `future<>` items for completed tasks on our `queue<>`. In fact, we only want the `future<>` for the one that completes first. If each fiber launched by `wait_first_outcome()` were to `push()` the result of calling `async()`, the queue would only ever report the result of the leftmost task item — not the one that completes most quickly.

Calling `future::get()` on the future returned by `async()` wouldn't be right. You can only call `get()` once per `future<>` instance! And if there were an exception, it would be rethrown inside the helper fiber at the producer end of the queue, rather than propagated to the consumer end.

We could call `future::wait()`. That would block the helper fiber until the `future<>` became ready, at which point we could `push()` it to be retrieved by `wait_first_outcome()`.

That would work — but there's a simpler tactic that avoids creating an extra fiber. We can wrap the task function in a `packaged_task<>`. While one naturally thinks of passing a `packaged_task<>` to a new fiber — that is, in fact, what `async()` does — in this case, we're already running in the helper fiber at the producer end of the queue! We can simply call the `packaged_task<>`. On return from that call, the task function has completed, meaning that the `future<>` obtained from the `packaged_task<>` is certain to be ready. At that point we can simply `push()` it to the queue.

    template< typename T, typename CHANP, typename Fn >
    void wait_first_outcome_impl( CHANP chan, Fn && function) {
        boost::fibers::fiber(
            // Use std::bind() here for C++11 compatibility. C++11 lambda capture
            // can't move a move-only Fn type, but bind() can. Let bind() move the
            // channel pointer and the function into the bound object, passing
            // references into the lambda.
            std::bind(
                []( CHANP & chan,
                    typename std::decay< Fn >::type & function) {
                    // Instantiate a packaged_task to capture any exception thrown by
                    // function.
                    boost::fibers::packaged_task< T() > task( function);
                    // Immediately run this packaged_task on same fiber. We want
                    // function() to have completed BEFORE we push the future.
                    task();
                    // Pass the corresponding future to consumer. Ignore
                    // channel_op_status returned by push(): might be closed; we
                    // simply don't care.
                    chan->push( task.get_future() );
                },
                chan,
                std::forward< Fn >( function)
            )).detach();
    }

Calling it might look like this:

    std::string result = wait_first_outcome(
            [](){ return sleeper("wfos_first",   50); },
            [](){ return sleeper("wfos_second", 100); },
            [](){ return sleeper("wfos_third",  150); });
    std::cout << "wait_first_outcome(success) => " << result << std::endl;
    assert(result == "wfos_first");

    std::string thrown;
    try {
        result = wait_first_outcome(
                [](){ return sleeper("wfof_first",   50, true); },
                [](){ return sleeper("wfof_second", 100); },
                [](){ return sleeper("wfof_third",  150); });
    } catch ( std::exception const& e) {
        thrown = e.what();
    }
    std::cout << "wait_first_outcome(fail) threw '" << thrown
              << "'" << std::endl;
    assert(thrown == "wfof_first");

### when_any, produce first success

One scenario for when_any functionality is when we're redundantly contacting some number of possibly-unreliable web services. Not only might they be slow — any one of them might produce a failure rather than the desired result.

In such a case, `wait_first_outcome()` isn't the right approach. If one of the services produces an error quickly, while another follows up with a real answer, we don't want to prefer the error just because it arrived first!

Given the `queue< future< T > >` we already constructed for `wait_first_outcome()`, though, we can readily recast the interface function to deliver the first successful result.

That does beg the question: what if all the task functions throw an exception? In that case we'd probably better know about it.

The C++ Parallelism Draft Technical Specification proposes a `std::exception_list` exception capable of delivering a collection of `std::exception_ptr`s. Until that becomes universally available, let's fake up an `exception_list` of our own:

    class exception_list : public std::runtime_error {
    public:
        exception_list( std::string const& what) :
            std::runtime_error( what) {
        }

        typedef std::vector< std::exception_ptr >   bundle_t;

        // N4407 proposed std::exception_list API
        typedef bundle_t::const_iterator iterator;

        std::size_t size() const noexcept {
            return bundle_.size();
        }

        iterator begin() const noexcept {
            return bundle_.begin();
        }

        iterator end() const noexcept {
            return bundle_.end();
        }

        // extension to populate
        void add( std::exception_ptr ep) {
            bundle_.push_back( ep);
        }

    private:
        bundle_t bundle_;
    };

Now we can build `wait_first_success()`, using `wait_first_outcome_impl()`.

Instead of retrieving only the first `future<>` from the queue, we must now loop over `future<>` items. Of course we must limit that iteration! If we launch only `count` producer fibers, the `(count+1)`st `buffered_channel::pop()` call would block forever.

Given a ready `future<>`, we can distinguish failure by calling `future::get_exception_ptr()`. If the `future<>` in fact contains a result rather than an exception, `get_exception_ptr()` returns `nullptr`. In that case, we can confidently call `future::get()` to return that result to our caller.

If the `std::exception_ptr` is not `nullptr`, though, we collect it into our pending `exception_list` and loop back for the next `future<>` from the queue.

If we fall out of the loop — if every single task fiber threw an exception — we throw the `exception_list` exception into which we've been collecting those `std::exception_ptr`s.

    template< typename Fn, typename ... Fns >
    typename std::result_of< Fn() >::type
    wait_first_success( Fn && function, Fns && ... functions) {
        std::size_t count( 1 + sizeof ... ( functions) );
        // In this case, the value we pass through the channel is actually a
        // future -- which is already ready. future can carry either a value or an
        // exception.
        typedef typename std::result_of< typename std::decay< Fn >::type() >::type return_t;
        typedef boost::fibers::future< return_t > future_t;
        typedef boost::fibers::buffered_channel< future_t > channel_t;
        auto chanp( std::make_shared< channel_t >( 64) );
        // launch all the relevant fibers
        wait_first_outcome_impl< return_t >( chanp,
                                             std::forward< Fn >( function),
                                             std::forward< Fns >( functions) ... );
        // instantiate exception_list, just in case
        exception_list exceptions("wait_first_success() produced only errors");
        // retrieve up to 'count' results -- but stop there!
        for ( std::size_t i = 0; i < count; ++i) {
            // retrieve the next future
            future_t future( chanp->value_pop() );
            // retrieve exception_ptr if any
            std::exception_ptr error( future.get_exception_ptr() );
            // if no error, then yay, return value
            if ( ! error) {
                // close the channel: no subsequent push() has to succeed
                chanp->close();
                // show caller the value we got
                return future.get();
            }

            // error is non-null: collect
            exceptions.add( error);
        }
        // We only arrive here when every passed function threw an exception.
        // Throw our collection to inform caller.
        throw exceptions;
    }

A call might look like this:

    std::string result = wait_first_success(
                [](){ return sleeper("wfss_first",   50, true); },
                [](){ return sleeper("wfss_second", 100); },
                [](){ return sleeper("wfss_third",  150); });
    std::cout << "wait_first_success(success) => " << result << std::endl;
    assert(result == "wfss_second");

### when_any, heterogeneous types

We would be remiss to ignore the case in which the various task functions have distinct return types. That means that the value returned by the first of them might have any one of those types. We can express that with Boost.Variant.

To keep the example simple, we'll revert to pretending that none of them can throw an exception. That makes `wait_first_value_het()` strongly resemble `wait_first_value()`. We can actually reuse `wait_first_value_impl()`, merely passing `boost::variant<T0, T1, ...>` as the queue's value type rather than the common `T`!

Naturally this could be extended to use `wait_first_success()` semantics instead.

    // No need to break out the first Fn for interface function: let the compiler
    // complain if empty.
    // Our functions have different return types, and we might have to return any
    // of them. Use a variant, expanding std::result_of<Fn()>::type for each Fn in
    // parameter pack.
    template< typename ... Fns >
    boost::variant< typename std::result_of< Fns() >::type ... >
    wait_first_value_het( Fns && ... functions) {
        // Use buffered_channel<boost::variant<T1, T2, ...>>; see remarks above.
        typedef boost::variant< typename std::result_of< Fns() >::type ... > return_t;
        typedef boost::fibers::buffered_channel< return_t > channel_t;
        auto chanp( std::make_shared< channel_t >( 64) );
        // launch all the relevant fibers
        wait_first_value_impl< return_t >( chanp,
                                           std::forward< Fns >( functions) ... );
        // retrieve the first value
        return_t value( chanp->value_pop() );
        // close the channel: no subsequent push() has to succeed
        chanp->close();
        return value;
    }

It might be called like this:

    boost::variant< std::string, double, int > result =
        wait_first_value_het(
                [](){ return sleeper("wfvh_third",  150); },
                [](){ return sleeper(3.14,          100); },
                [](){ return sleeper(17,             50); });
    std::cout << "wait_first_value_het() => " << result << std::endl;
    assert(boost::get< int >( result) == 17);

### when_any, a dubious alternative

Certain topics in C++ can arouse strong passions, and exceptions are no exception. We cannot resist mentioning — for purely informational purposes — that when you need only the first result from some number of concurrently-running fibers, it would be possible to pass a `shared_ptr<`promise<>`>` to the participating fibers, then cause the initiating fiber to call `future::get()` on its `future<>`. The first fiber to call `promise::set_value()` on that shared `promise` will succeed; subsequent `set_value()` calls on the same `promise` instance will throw `future_error`.

Use this information at your own discretion. Beware the dark side.

## when_all functionality

### when_all, simple completion

For the case in which we must wait for all task functions to complete — but we don't need results (or expect exceptions) from any of them — we can write `wait_all_simple()` that looks remarkably like `wait_first_simple()`. The difference is that instead of our `Done` class, we instantiate a `barrier` and call its `barrier::wait()`.

We initialize the `barrier` with `(count+1)` because we are launching `count` fibers, plus the `wait()` call within `wait_all_simple()` itself.

    template< typename ... Fns >
    void wait_all_simple( Fns && ... functions) {
        std::size_t count( sizeof ... ( functions) );
        // Initialize a barrier(count+1) because we'll immediately wait on it. We
        // don't want to wake up until 'count' more fibers wait on it. Even though
        // we'll stick around until the last of them completes, use shared_ptr
        // anyway because it's easier to be confident about lifespan issues.
        auto barrier( std::make_shared< boost::fibers::barrier >( count + 1) );
        wait_all_simple_impl( barrier, std::forward< Fns >( functions) ... );
        barrier->wait();
    }

As stated above, the only difference between `wait_all_simple_impl()` and `wait_first_simple_impl()` is that the former calls `barrier::wait()` rather than `Done::notify()`:

    template< typename Fn, typename ... Fns >
    void wait_all_simple_impl( std::shared_ptr< boost::fibers::barrier > barrier,
                               Fn && function, Fns && ... functions) {
        boost::fibers::fiber(
                std::bind(
                    []( std::shared_ptr< boost::fibers::barrier > & barrier,
                        typename std::decay< Fn >::type & function) mutable {
                            function();
                            barrier->wait();
                    },
                    barrier,
                    std::forward< Fn >( function)
                )).detach();
        wait_all_simple_impl( barrier, std::forward< Fns >( functions) ... );
    }

You might call it like this:

    wait_all_simple(
            [](){ sleeper("was_long",   150); },
            [](){ sleeper("was_medium", 100); },
            [](){ sleeper("was_short",   50); });

Control will not return from the `wait_all_simple()` call until the last of its task functions has completed.

### when_all, return values

As soon as we want to collect return values from all the task functions, we can see right away how to reuse `wait_first_value()`'s queue<T> for the purpose. All we have to do is avoid closing it after the first value!

But in fact, collecting multiple values raises an interesting question: do we really want to wait until the slowest of them has arrived? Wouldn't we rather process each result as soon as it becomes available?

Fortunately we can present both APIs. Let's define `wait_all_values_source()` to return `shared_ptr<buffered_channel<T>>`.

Given `wait_all_values_source()`, it's straightforward to implement `wait_all_values()`:

    template< typename Fn, typename ... Fns >
    std::vector< typename std::result_of< Fn() >::type >
    wait_all_values( Fn && function, Fns && ... functions) {
        std::size_t count( 1 + sizeof ... ( functions) );
        typedef typename std::result_of< Fn() >::type return_t;
        typedef std::vector< return_t > vector_t;
        vector_t results;
        results.reserve( count);

        // get channel
        std::shared_ptr< boost::fibers::buffered_channel< return_t > > chan =
            wait_all_values_source( std::forward< Fn >( function),
                                    std::forward< Fns >( functions) ... );
        // fill results vector
        return_t value;
        while ( boost::fibers::channel_op_status::success == chan->pop(value) ) {
            results.push_back( value);
        }
        // return vector to caller
        return results;
    }

It might be called like this:

    std::vector< std::string > values =
        wait_all_values(
                [](){ return sleeper("wav_late",   150); },
                [](){ return sleeper("wav_middle", 100); },
                [](){ return sleeper("wav_early",   50); });

As you can see from the loop in `wait_all_values()`, instead of requiring its caller to count values, we define `wait_all_values_source()` to `buffered_channel::close()` the queue when done. But how do we do that? Each producer fiber is independent. It has no idea whether it is the last one to `buffered_channel::push()` a value.

We can address that problem with a counting façade for the `queue<>`. In fact, our façade need only support the producer end of the queue.

[wait_nqueue]

Armed with `nqueue<>`, we can implement `wait_all_values_source()`. It starts just like `wait_first_value()`. The difference is that we wrap the `queue<T>` with an `nqueue<T>` to pass to the producer fibers.

Then, of course, instead of popping the first value, closing the queue and returning it, we simply return the `shared_ptr<queue<T>>`.

    // Return a shared_ptr<buffered_channel<T>> from which the caller can
    // retrieve each new result as it arrives, until 'closed'.
    template< typename Fn, typename ... Fns >
    std::shared_ptr< boost::fibers::buffered_channel< typename std::result_of< Fn() >::type > >
    wait_all_values_source( Fn && function, Fns && ... functions) {
        std::size_t count( 1 + sizeof ... ( functions) );
        typedef typename std::result_of< Fn() >::type return_t;
        typedef boost::fibers::buffered_channel< return_t > channel_t;
        // make the channel
        auto chanp( std::make_shared< channel_t >( 64) );
        // and make an nchannel facade to close it after 'count' items
        auto ncp( std::make_shared< nchannel< return_t > >( chanp, count) );
        // pass that nchannel facade to all the relevant fibers
        wait_all_values_impl< return_t >( ncp,
                                          std::forward< Fn >( function),
                                          std::forward< Fns >( functions) ... );
        // then return the channel for consumer
        return chanp;
    }

For example:

    std::shared_ptr< boost::fibers::buffered_channel< std::string > > chan =
        wait_all_values_source(
                [](){ return sleeper("wavs_third",  150); },
                [](){ return sleeper("wavs_second", 100); },
                [](){ return sleeper("wavs_first",   50); });
    std::string value;
    while ( boost::fibers::channel_op_status::success == chan->pop(value) ) {
        std::cout << "wait_all_values_source() => '" << value
                  << "'" << std::endl;
    }

`wait_all_values_impl()` really is just like `wait_first_value_impl()` except for the use of `nqueue<T>` rather than `queue<T>`:

    template< typename T, typename Fn >
    void wait_all_values_impl( std::shared_ptr< nchannel< T > > chan,
                               Fn && function) {
        boost::fibers::fiber( [chan, function](){
                                  chan->push(function());
                              }).detach();
    }

### when_all until first exception

Naturally, just as with `wait_first_outcome()`, we can elaborate `wait_all_values()` and `wait_all_values_source()` by passing `future< T >` instead of plain `T`.

`wait_all_until_error()` pops that `future< T >` and calls its `future::get()`:

    template< typename Fn, typename ... Fns >
    std::vector< typename std::result_of< Fn() >::type >
    wait_all_until_error( Fn && function, Fns && ... functions) {
        std::size_t count( 1 + sizeof ... ( functions) );
        typedef typename std::result_of< Fn() >::type return_t;
        typedef typename boost::fibers::future< return_t > future_t;
        typedef std::vector< return_t > vector_t;
        vector_t results;
        results.reserve( count);

        // get channel
        std::shared_ptr<
            boost::fibers::buffered_channel< future_t > > chan(
                wait_all_until_error_source( std::forward< Fn >( function),
                                             std::forward< Fns >( functions) ... ) );
        // fill results vector
        future_t future;
        while ( boost::fibers::channel_op_status::success == chan->pop( future) ) {
            results.push_back( future.get() );
        }
        // return vector to caller
        return results;
    }

For example:

    std::string thrown;
    try {
        std::vector< std::string > values = wait_all_until_error(
                [](){ return sleeper("waue_late",   150); },
                [](){ return sleeper("waue_middle", 100, true); },
                [](){ return sleeper("waue_early",   50); });
    } catch ( std::exception const& e) {
        thrown = e.what();
    }
    std::cout << "wait_all_until_error(fail) threw '" << thrown
              << "'" << std::endl;

Naturally this complicates the API for `wait_all_until_error_source()`. The caller must both retrieve a `future< T >` and call its `get()` method. It would, of course, be possible to return a façade over the consumer end of the queue that would implicitly perform the `get()` and return a simple `T` (or throw).

The implementation is just as you would expect. Notice, however, that we can reuse `wait_first_outcome_impl()`, passing the `nqueue<T>` rather than `queue<T>`.

    // Return a shared_ptr<buffered_channel<future<T>>> from which the caller can
    // get() each new result as it arrives, until 'closed'.
    template< typename Fn, typename ... Fns >
    std::shared_ptr<
        boost::fibers::buffered_channel<
            boost::fibers::future<
                typename std::result_of< Fn() >::type > > >
    wait_all_until_error_source( Fn && function, Fns && ... functions) {
        std::size_t count( 1 + sizeof ... ( functions) );
        typedef typename std::result_of< Fn() >::type return_t;
        typedef boost::fibers::future< return_t > future_t;
        typedef boost::fibers::buffered_channel< future_t > channel_t;
        // make the channel
        auto chanp( std::make_shared< channel_t >( 64) );
        // and make an nchannel facade to close it after 'count' items
        auto ncp( std::make_shared< nchannel< future_t > >( chanp, count) );
        // pass that nchannel facade to all the relevant fibers
        wait_first_outcome_impl< return_t >( ncp,
                                             std::forward< Fn >( function),
                                             std::forward< Fns >( functions) ... );
        // then return the channel for consumer
        return chanp;
    }

For example:

    typedef boost::fibers::future< std::string > future_t;
    std::shared_ptr< boost::fibers::buffered_channel< future_t > > chan =
        wait_all_until_error_source(
                [](){ return sleeper("wauess_third",  150); },
                [](){ return sleeper("wauess_second", 100); },
                [](){ return sleeper("wauess_first",   50); });
    future_t future;
    while ( boost::fibers::channel_op_status::success == chan->pop( future) ) {
        std::string value( future.get() );
        std::cout << "wait_all_until_error_source(success) => '" << value
                  << "'" << std::endl;
    }

### wait_all, collecting all exceptions

Given `wait_all_until_error_source()`, it might be more reasonable to make a `wait_all_...()` that collects all errors instead of presenting only the first:

    template< typename Fn, typename ... Fns >
    std::vector< typename std::result_of< Fn() >::type >
    wait_all_collect_errors( Fn && function, Fns && ... functions) {
        std::size_t count( 1 + sizeof ... ( functions) );
        typedef typename std::result_of< Fn() >::type return_t;
        typedef typename boost::fibers::future< return_t > future_t;
        typedef std::vector< return_t > vector_t;
        vector_t results;
        results.reserve( count);
        exception_list exceptions("wait_all_collect_errors() exceptions");

        // get channel
        std::shared_ptr<
            boost::fibers::buffered_channel< future_t > > chan(
                wait_all_until_error_source( std::forward< Fn >( function),
                                             std::forward< Fns >( functions) ... ) );
        // fill results and/or exceptions vectors
        future_t future;
        while ( boost::fibers::channel_op_status::success == chan->pop( future) ) {
            std::exception_ptr exp = future.get_exception_ptr();
            if ( ! exp) {
                results.push_back( future.get() );
            } else {
                exceptions.add( exp);
            }
        }
        // if there were any exceptions, throw
        if ( exceptions.size() ) {
            throw exceptions;
        }
        // no exceptions: return vector to caller
        return results;
    }

The implementation is a simple variation on `wait_first_success()`, using the same `exception_list` exception class.

### when_all, heterogeneous types

But what about the case when we must wait for all results of different types?

We can present an API that is frankly quite cool. Consider a sample struct:

    struct Data {
        std::string str;
        double      inexact;
        int         exact;

        friend std::ostream& operator<<( std::ostream& out, Data const& data);
        ...
    };

Let's fill its members from task functions all running concurrently:

    Data data = wait_all_members< Data >(
            [](){ return sleeper("wams_left", 100); },
            [](){ return sleeper(3.14,        150); },
            [](){ return sleeper(17,          50); });
    std::cout << "wait_all_members<Data>(success) => " << data << std::endl;

Note that for this case, we abandon the notion of capturing the earliest result first, and so on: we must fill exactly the passed struct in left-to-right order.

That permits a beautifully simple implementation:

    // Explicitly pass Result. This can be any type capable of being initialized
    // from the results of the passed functions, such as a struct.
    template< typename Result, typename ... Fns >
    Result wait_all_members( Fns && ... functions) {
        // Run each of the passed functions on a separate fiber, passing all their
        // futures to helper function for processing.
        return wait_all_members_get< Result >(
                boost::fibers::async( std::forward< Fns >( functions) ) ... );
    }

    template< typename Result, typename ... Futures >
    Result wait_all_members_get( Futures && ... futures) {
        // Fetch the results from the passed futures into Result's initializer
        // list. It's true that the get() calls here will block the implicit
        // iteration over futures -- but that doesn't matter because we won't be
        // done until the slowest of them finishes anyway. As results are
        // processed in argument-list order rather than order of completion, the
        // leftmost get() to throw an exception will cause that exception to
        // propagate to the caller.
        return Result{ futures.get() ... };
    }

It is tempting to try to implement `wait_all_members()` as a one-liner like this:

    return Result{ boost::fibers::async(functions).get()... };

The trouble with this tactic is that it would serialize all the task functions. The runtime makes a single pass through `functions`, calling `fibers::async()` for each and then immediately calling `future::get()` on its returned `future<>`. That blocks the implicit loop. The above is almost equivalent to writing:

    return Result{ functions()... };

in which, of course, there is no concurrency at all.

Passing the argument pack through a function-call boundary (`wait_all_members_get()`) forces the runtime to make two passes: one in `wait_all_members()` to collect the `future<>`s from all the `async()` calls, the second in `wait_all_members_get()` to fetch each of the results.

As noted in comments, within the `wait_all_members_get()` parameter pack expansion pass, the blocking behavior of `get()` becomes irrelevant. Along the way, we will hit the `get()` for the slowest task function; after that every subsequent `get()` will complete in trivial time.

By the way, we could also use this same API to fill a vector or other collection:

    // If we don't care about obtaining results as soon as they arrive, and we
    // prefer a result vector in passed argument order rather than completion
    // order, wait_all_members() is another possible implementation of
    // wait_all_until_error().
    auto strings = wait_all_members< std::vector< std::string > >(
            [](){ return sleeper("wamv_left",   150); },
            [](){ return sleeper("wamv_middle", 100); },
            [](){ return sleeper("wamv_right",   50); });
    std::cout << "wait_all_members<vector>() =>";
    for ( std::string const& str : strings) {
        std::cout << " '" << str << "'";
    }
    std::cout << std::endl;

# Sharing a Thread with Another Main Loop

## Overview

As always with cooperative concurrency, it is important not to let any one fiber monopolize the processor too long: that could starve other ready fibers. This section discusses a couple of solutions.

## Event-Driven Program

Consider a classic event-driven program, organized around a main loop that fetches and dispatches incoming I/O events. You are introducing Boost.Fiber because certain asynchronous I/O sequences are logically sequential, and for those you want to write and maintain code that looks and acts sequential.

You are launching fibers on the application’s main thread because certain of their actions will affect its user interface, and the application’s UI framework permits UI operations only on the main thread. Or perhaps those fibers need access to main-thread data, and it would be too expensive in runtime (or development time) to robustly defend every such data item with thread synchronization primitives.

You must ensure that the application’s main loop itself doesn’t monopolize the processor: that the fibers it launches will get the CPU cycles they need.

The solution is the same as for any fiber that might claim the CPU for an extended time: introduce calls to `this_fiber::yield()`. The most straightforward approach is to call `yield()` on every iteration of your existing main loop. In effect, this unifies the application’s main loop with Boost.Fiber’s internal main loop. `yield()` allows the fiber manager to run any fibers that have become ready since the previous iteration of the application’s main loop. When these fibers have had a turn, control passes to the thread’s main fiber, which returns from `yield()` and resumes the application’s main loop.

## Embedded Main Loop

More challenging is when the application’s main loop is embedded in some other library or framework. Such an application will typically, after performing all necessary setup, pass control to some form of `run()` function from which control does not return until application shutdown.

A Boost.Asio program might call `io_service::run()` in this way.

In general, the trick is to arrange to pass control to `this_fiber::yield()` frequently. You could use an Asio timer for that purpose. You could instantiate the timer, arranging to call a handler function when the timer expires. The handler function could call `yield()`, then reset the timer and arrange to wake up again on its next expiration.

Since, in this thought experiment, we always pass control to the fiber manager via `yield()`, the calling fiber is never blocked. Therefore there is always at least one ready fiber. Therefore the fiber manager never calls `algorithm::suspend_until()`.

Using `io_service::post()` instead of setting a timer for some nonzero interval would be unfriendly to other threads. When all I/O is pending and all fibers are blocked, the io_service and the fiber manager would simply spin the CPU, passing control back and forth to each other. Using a timer allows tuning the responsiveness of this thread relative to others.

## Deeper Dive into Boost.Asio

By now the alert reader is thinking: but surely, with Asio in particular, we ought to be able to do much better than periodic polling pings!

This turns out to be surprisingly tricky. We present a possible approach in `examples/asio/round_robin.hpp`.

One consequence of using Boost.Asio is that you must always let Asio suspend the running thread. Since Asio is aware of pending I/O requests, it can arrange to suspend the thread in such a way that the OS will wake it on I/O completion. No one else has sufficient knowledge.

So the fiber scheduler must depend on Asio for suspension and resumption. It requires Asio handler calls to wake it.

One dismaying implication is that we cannot support multiple threads calling `io_service::run()` on the same `io_service` instance. The reason is that Asio provides no way to constrain a particular handler to be called only on a specified thread. A fiber scheduler instance is locked to a particular thread: that instance cannot manage any other thread’s fibers. Yet if we allow multiple threads to call `io_service::run()` on the same `io_service` instance, a fiber scheduler which needs to sleep can have no guarantee that it will reawaken in a timely manner. It can set an Asio timer, as described above — but that timer’s handler may well execute on a different thread!

Another implication is that since an Asio-aware fiber scheduler (not to mention `boost::fibers::asio::yield`) depends on handler calls from the `io_service`, it is the application’s responsibility to ensure that `io_service::stop()` is not called until every fiber has terminated.

It is easier to reason about the behavior of the presented `asio::round_robin` scheduler if we require that after initial setup, the thread’s main fiber is the fiber that calls `io_service::run()`, so let’s impose that requirement.

Naturally, the first thing we must do on each thread using a custom fiber scheduler is call `use_scheduling_algorithm()`. However, since `asio::round_robin` requires an `io_service` instance, we must first declare that.

    std::shared_ptr< boost::asio::io_service > io_svc = std::make_shared< boost::asio::io_service >();
    boost::fibers::use_scheduling_algorithm< boost::fibers::asio::round_robin >( io_svc);

`use_scheduling_algorithm()` instantiates `asio::round_robin`, which naturally calls its constructor:

    round_robin( std::shared_ptr< boost::asio::io_service > const& io_svc) :
        io_svc_( io_svc),
        suspend_timer_( * io_svc_) {
        // We use add_service() very deliberately. This will throw
        // service_already_exists if you pass the same io_service instance to
        // more than one round_robin instance.
        boost::asio::add_service( * io_svc_, new service( * io_svc_) );
        io_svc_->post([this]() mutable {

`asio::round_robin` binds the passed `io_service` pointer and initializes a `boost::asio::steady_timer`:

    std::shared_ptr< boost::asio::io_service >      io_svc_;
    boost::asio::steady_timer                       suspend_timer_;

Then it calls `boost::asio::add_service()` with a nested `service` struct:

    struct service : public boost::asio::io_service::service {
        static boost::asio::io_service::id                  id;

        std::unique_ptr< boost::asio::io_service::work >    work_;

        service( boost::asio::io_service & io_svc) :
            boost::asio::io_service::service( io_svc),
            work_{ new boost::asio::io_service::work( io_svc) } {
        }

        virtual ~service() {}

        service( service const&) = delete;
        service & operator=( service const&) = delete;

        void shutdown_service() override final {
            work_.reset();
        }
    };

... [asio_rr_service_bottom]

The `service` struct has a couple of roles.

Its foremost role is to manage a `std::unique_ptr<`boost::asio::io_service::work`>`. We want the `io_service` instance to continue its main loop even when there is no pending Asio I/O.

But when `boost::asio::io_service::service::shutdown_service()` is called, we discard the `io_service::work` instance so the `io_service` can shut down properly.

Its other purpose is to `post()` a lambda (not yet shown). Let’s walk further through the example program before coming back to explain that lambda.

The `service` constructor returns to `asio::round_robin`’s constructor, which returns to `use_scheduling_algorithm()`, which returns to the application code.

Once it has called `use_scheduling_algorithm()`, the application may now launch some number of fibers:

    // server
    tcp::acceptor a( * io_svc, tcp::endpoint( tcp::v4(), 9999) );
    boost::fibers::fiber( server, io_svc, std::ref( a) ).detach();
    // client
    const unsigned iterations = 2;
    const unsigned clients = 3;
    boost::fibers::barrier b( clients);
    for ( unsigned i = 0; i < clients; ++i) {
        boost::fibers::fiber(
                client, io_svc, std::ref( a), std::ref( b), iterations).detach();
    }

Since we don’t specify a `launch`, these fibers are ready to run, but have not yet been entered.

Having set everything up, the application calls `io_service::run()`:

    io_svc->run();

Now what?

Because this `io_service` instance owns an `io_service::work` instance, `run()` does not immediately return. But — none of the fibers that will perform actual work has even been entered yet!

Without that initial `post()` call in `service`’s constructor, nothing would happen. The application would hang right here.

So, what should the `post()` handler execute? Simply `this_fiber::yield()`?

That would be a promising start. But we have no guarantee that any of the other fibers will initiate any Asio operations to keep the ball rolling. For all we know, every other fiber could reach a similar `this_fiber::yield()` call first. Control would return to the `post()` handler, which would return to Asio, and... the application would hang.

The `post()` handler could `post()` itself again. But as discussed in the previous section, once there are actual I/O operations in flight — once we reach a state in which no fiber is ready — that would cause the thread to spin.

We could, of course, set an Asio timer — again as previously discussed. But in this deeper dive, we’re trying to do a little better.

The key to doing better is that since we’re in a fiber, we can run an actual loop — not just a chain of callbacks. We can wait for something to happen by calling `io_service::run_one()` — or we can execute already-queued Asio handlers by calling `io_service::poll()`.

Here’s the body of the lambda passed to the `post()` call.

     while ( ! io_svc_->stopped() ) {
         if ( has_ready_fibers() ) {
             // run all pending handlers in round_robin
             while ( io_svc_->poll() );
             // block this fiber till all pending (ready) fibers are processed
             // == round_robin::suspend_until() has been called
             std::unique_lock< boost::fibers::mutex > lk( mtx_);
             cnd_.wait( lk);
         } else {
             // run one handler inside io_service
             // if no handler available, block this thread
             if ( ! io_svc_->run_one() ) {
                 break;
             }
         }
    }

We want this loop to exit once the `io_service` instance has been `stopped()`.

As long as there are ready fibers, we interleave running ready Asio handlers with running ready fibers.

If there are no ready fibers, we wait by calling `run_one()`. Once any Asio handler has been called — no matter which — `run_one()` returns. That handler may have transitioned some fiber to ready state, so we loop back to check again.

(We won’t describe `awakened()`, `pick_next()` or `has_ready_fibers()`, as these are just like `round_robin::awakened()`, `round_robin::pick_next()` and `round_robin::has_ready_fibers()`.)

That leaves `suspend_until()` and `notify()`.

Doubtless you have been asking yourself: why are we calling `io_service::run_one()` in the lambda loop? Why not call it in `suspend_until()`, whose very API was designed for just such a purpose?

Under normal circumstances, when the fiber manager finds no ready fibers, it calls `algorithm::suspend_until()`. Why test `has_ready_fibers()` in the lambda loop? Why not leverage the normal mechanism?

The answer is: it matters who’s asking.

Consider the lambda loop shown above. The only Boost.Fiber APIs it engages are `has_ready_fibers()` and `this_fiber::yield()`. `yield()` does not block the calling fiber: the calling fiber does not become unready. It is immediately passed back to `algorithm::awakened()`, to be resumed in its turn when all other ready fibers have had a chance to run. In other words: during a `yield()` call, there is always at least one ready fiber.

As long as this lambda loop is still running, the fiber manager does not call `suspend_until()` because it always has a fiber ready to run.

However, the lambda loop itself can detect the case when no other fibers are ready to run: the running fiber is not ready but running.

That said, `suspend_until()` and `notify()` are in fact called during orderly shutdown processing, so let’s try a plausible implementation.

    void suspend_until( std::chrono::steady_clock::time_point const& abs_time) noexcept {
        // Set a timer so at least one handler will eventually fire, causing
        // run_one() to eventually return.
        if ( (std::chrono::steady_clock::time_point::max)() != abs_time) {
            // Each expires_at(time_point) call cancels any previous pending
            // call. We could inadvertently spin like this:
            // dispatcher calls suspend_until() with earliest wake time
            // suspend_until() sets suspend_timer_
            // lambda loop calls run_one()
            // some other asio handler runs before timer expires
            // run_one() returns to lambda loop
            // lambda loop yields to dispatcher
            // dispatcher finds no ready fibers
            // dispatcher calls suspend_until() with SAME wake time
            // suspend_until() sets suspend_timer_ to same time, canceling
            // previous async_wait()
            // lambda loop calls run_one()
            // asio calls suspend_timer_ handler with operation_aborted
            // run_one() returns to lambda loop... etc. etc.
            // So only actually set the timer when we're passed a DIFFERENT
            // abs_time value.
            suspend_timer_.expires_at( abs_time);
            suspend_timer_.async_wait([](boost::system::error_code const&){
                                        this_fiber::yield();
                                      });
        }
        cnd_.notify_one();
    }

As you might expect, `suspend_until()` sets an `asio::steady_timer` to `expires_at()` the passed `std::chrono::steady_clock::time_point`. Usually.

As indicated in comments, we avoid setting `suspend_timer_` multiple times to the same `time_point` value since every `expires_at()` call cancels any previous `async_wait()` call. There is a chance that we could spin. Reaching `suspend_until()` means the fiber manager intends to yield the processor to Asio. Cancelling the previous `async_wait()` call would fire its handler, causing `run_one()` to return, potentially causing the fiber manager to call `suspend_until()` again with the same `time_point` value...

Given that we suspend the thread by calling `io_service::run_one()`, what’s important is that our `async_wait()` call will cause a handler to run, which will cause `run_one()` to return. It’s not so important specifically what that handler does.

    void notify() noexcept {
        // Something has happened that should wake one or more fibers BEFORE
        // suspend_timer_ expires. Reset the timer to cause it to fire
        // immediately, causing the run_one() call to return. In theory we
        // could use cancel() because we don't care whether suspend_timer_'s
        // handler is called with operation_aborted or success. However --
        // cancel() doesn't change the expiration time, and we use
        // suspend_timer_'s expiration time to decide whether it's already
        // set. If suspend_until() set some specific wake time, then notify()
        // canceled it, then suspend_until() was called again with the same
        // wake time, it would match suspend_timer_'s expiration time and we'd
        // refrain from setting the timer. So instead of simply calling
        // cancel(), reset the timer, which cancels the pending sleep AND sets
        // a new expiration time. This will cause us to spin the loop twice --
        // once for the operation_aborted handler, once for timer expiration
        // -- but that shouldn't be a big problem.
        suspend_timer_.async_wait([](boost::system::error_code const&){
                                    this_fiber::yield();
                                  });
        suspend_timer_.expires_at( std::chrono::steady_clock::now() );
    }

Since an `expires_at()` call cancels any previous `async_wait()` call, we can make `notify()` simply call `steady_timer::expires_at()`. That should cause the `io_service` to call the `async_wait()` handler with `operation_aborted`.

The comments in `notify()` explain why we call `expires_at()` rather than `cancel()`.

This `boost::fibers::asio::round_robin` implementation is used in `examples/asio/autoecho.cpp`.

It seems possible that you could put together a more elegant Fiber / Asio integration. But as noted at the outset: it’s tricky.

# Specualtive execution

Hardware transactional memory

With help of hardware transactional memory multiple logical processors execute a critical region speculatively, e.g. without explicit synchronization. If the transactional execution completes successfully, then all memory operations performed within the transactional region are commited without any inter-thread serialization. When the optimistic execution fails, the processor aborts the transaction and discards all performed modifications. In non-transactional code a single lock serializes the access to a critical region. With a transactional memory, multiple logical processor start a transaction and update the memory (the data) inside the ciritical region. Unless some logical processors try to update the same data, the transactions would always succeed.

Intel Transactional Synchronisation Extensions (TSX)

TSX is Intel's implementation of hardware transactional memory in modern Intel processors intel.com: Intel Transactional Synchronization Extensions . In TSX the hardware keeps track of which cachelines have been read from and which have been written to in a transaction. The cache-line size (64-byte) and the n-way set associative cache determine the maximum size of memory in a transaction. For instance if a transaction modifies 9 cache-lines at a processor with a 8-way set associative cache, the transaction will always be aborted.

TXS is enabled if property `htm=tsx` is specified at b2 command-line and `BOOST_USE_TSX` is applied to the compiler.

A TSX-transaction will be aborted if the floating point state is modified inside a critical region. As a consequence floating point operations, e.g. store/load of floating point related registers during a fiber (context) switch are disabled.

TSX can not be used together with MSVC at this time!

Boost.Fiber uses TSX-enabled spinlocks to protect critical regions (see section Tuning).

# NUMA

Modern micro-processors contain integrated memory controllers that are connected via channels to the memory. Accessing the memory can be organized in two kinds: Uniform Memory Access (UMA) and Non-Uniform Memory Access (NUMA).

In contrast to UMA, that provides a centralized pool of memory (and thus does not scale after a certain number of processors), a NUMA architecture divides the memory into local and remote memory relative to the micro-processor. Local memory is directly attached to the processor's integrated memory controller. Memory connected to the memory controller of another micro-processor (multi-socket systems) is considered as remote memory. If a memory controller access remote memory it has to traverse the interconnect On x86 the interconnection is implemented by Intel's Quick Path Interconnect (QPI) and AMD's HyperTransport. and connect to the remote memory controller. Thus accessing remote memory adds additional latency overhead to local memory access. Because of the different memory locations, a NUMA-system experiences non-uniform memory access time. As a consequence the best performance is achieved by keeping the memory access local.

NUMA

NUMA support in Boost.Fiber

Because only a subset of the NUMA-functionality is exposed by several operating systems, Boost.Fiber provides only a minimalistic NUMA API.

## Supported functionality/operating systems

| AIX | FreeBSD | HP/UX | Linux | Solaris | Windows
--- | --- | --- | --- | --- | --- | ---
pin thread | + | + | + | + | + | +
logical CPUs/NUMA nodes | + | + | + | + | + | + Windows organizes logical cpus in groups of 64; boost.fiber maps {group-id,cpud-id} to a scalar equivalent to cpu ID of Linux (64 * group ID + cpu ID).
NUMA node distance | - | - | - | + | - | -
tested on | AIX 7.2 | FreeBSD 11 | - | Arch Linux (4.10.13) | OpenIndiana HIPSTER | Windows 10

In order to keep the memory access local as possible, the NUMA topology must be evaluated.

    std::vector< boost::fibers::numa::node > topo = boost::fibers::numa::topology();
    for ( auto n : topo) {
        std::cout << "node: " << n.id << " | ";
        std::cout << "cpus: ";
        for ( auto cpu_id : n.logical_cpus) {
            std::cout << cpu_id << " ";
        }
        std::cout << "| distance: ";
        for ( auto d : n.distance) {
            std::cout << d << " ";
        }
        std::cout << std::endl;
    }
    std::cout << "done" << std::endl;

    output:
        node: 0 | cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 | distance: 10 21
        node: 1 | cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 | distance: 21 10
        done

The example shows that the systems consits out of 2 NUMA-nodes, to each NUMA-node belong 16 logical cpus. The distance measures the costs to access the memory of another NUMA-node. A NUMA-node has always a distance `10` to itself (lowest possible value). The position in the array corresponds with the NUMA-node ID.

Some work-loads benefit from pinning threads to a logical cpus. For instance scheduling algorithm `numa::work_stealing` pins the thread that runs the fiber scheduler to a logical cpu. This prevents the operating system scheduler to move the thread to another logical cpu that might run other fiber scheduler(s) or migrating the thread to a logical cpu part of another NUMA-node.

    void thread( std::uint32_t cpu_id, std::uint32_t node_id, std::vector< boost::fibers::numa::node > const& topo) {
        // thread registers itself at work-stealing scheduler
        boost::fibers::use_scheduling_algorithm< boost::fibers::algo::numa::work_stealing >( cpu_id, node_id, topo);
        ...
    }

    // evaluate the NUMA topology
    std::vector< boost::fibers::numa::node > topo = boost::fibers::numa::topology();
    // start-thread runs on NUMA-node `0`
    auto node = topo[0];
    // start-thread is pinnded to first cpu ID in the list of logical cpus of NUMA-node `0`
    auto start_cpu_id = * node.logical_cpus.begin();
    // start-thread registers itself on work-stealing scheduler
    boost::fibers::use_scheduling_algorithm< boost::fibers::algo::numa::work_stealing >( start_cpu_id, node.id, topo);
    std::vector< std::thread > threads;
    for ( auto & node : topo) {
        for ( std::uint32_t cpu_id : node.logical_cpus) {
            // exclude start-thread
            if ( start_cpu_id != cpu_id) {
                // spawn thread
                threads.emplace_back( thread, cpu_id, node.id, std::cref( topo) );
            }
        }
    }
    ...

The example evaluates the NUMA topology with `boost::fibers::numa::topology()` and spawns for each logical cpu a thread. Each spawned thread installs the NUMA-aware work-stealing scheduler. The scheduler pins the thread to the logical cpu that was specified at construction. If the local queue of one thread runs out of ready fibers, the thread tries to steal a ready fiber from another thread running at logical cpu that belong to the same NUMA-node (local memory access). If no fiber could be stolen, the thread tries to steal fibers from logical cpus part of other NUMA-nodes (remote memory access).

Synopsis

    #include <boost/fiber/numa/pin_thread.hpp>
    #include <boost/fiber/numa/topology.hpp>

    namespace boost {
    namespace fibers {
    namespace numa {

    struct node {
        std::uint32_t                   id;
        std::set< std::uint32_t >       logical_cpus;
        std::vector< std::uint32_t >    distance;
    };
    bool operator<( node const&, node const&) noexcept;

    std::vector< node > topology();

    void pin_thread( std::uint32_t);

    }}}

    #include <boost/fiber/algo/numa/work_stealing.hpp>

    namespace boost {
    namespace fibers {
    namespace algo {
    namespace numa {

    class work_stealing;

    }}}

Class `numa::node`

    #include <boost/fiber/numa/topology.hpp>

    namespace boost {
    namespace fibers {
    namespace numa {

    struct node {
        std::uint32_t                   id;
        std::set< std::uint32_t >       logical_cpus;
        std::vector< std::uint32_t >    distance;
    };
    bool operator<( node const&, node const&) noexcept;

    }}}

Data member `id`

    std::uint32_t id;

##

Effects:

  ID of the NUMA-node

Data member `logical_cpus`

    std::set< std::uint32_t > logical_cpus;

##

Effects:

  set of logical cpu IDs belonging to the NUMA-node

Data member `distance`

    std::vector< std::uint32_t > distance;

##

Effects:

  The distance between NUMA-nodes describe the cots of accessing the remote memory.

Note:

  A NUMA-node has a distance of `10` to itself, remote NUMA-nodes have a distance > `10`. The index in the array corresponds to the ID `id` of the NUMA-node. At the moment only Linux returns the correct distances, for all other operating systems remote NUMA-nodes get a default value of `20`.

Member function `operator<`()

    bool operator<( node const& lhs, node const& rhs) const noexcept;

##

Returns:

  `true` if `lhs != rhs` is true and the implementation-defined total order of `node::id` values places `lhs` before `rhs`, false otherwise.

Throws:

  Nothing.

Non-member function `numa::topology()`

    #include <boost/fiber/numa/topology.hpp>

    namespace boost {
    namespace fibers {
    namespace numa {

    std::vector< node > topology();

    }}}

##

Effects:

  Evaluates the NUMA topology.

Returns:

  a vector of NUMA-nodes describing the NUMA architecture of the system (each element represents a NUMA-node).

Throws:

  `system_error`

Non-member function `numa::pin_thread()`

    #include <boost/fiber/numa/pin_thread.hpp>

    namespace boost {
    namespace fibers {
    namespace numa {

    void pin_thread( std::uint32_t);

    }}}

##

Effects:

  Pins `this thread` to the logical cpu with ID `cpu_id`, e.g. the operating system scheduler will not migrate the thread to another logical cpu.

Throws:

  `system_error`

Class `numa::work_stealing`

This class implements `algorithm`; the thread running this scheduler is pinned to the given logical cpu. If the local ready-queue runs out of ready fibers, ready fibers are stolen from other schedulers that run on logical cpus that belong to the same NUMA-node (local memory access). If no ready fibers can be stolen from the local NUMA-node, the algorithm selects schedulers running on other NUMA-nodes (remote memory access). The victim scheduler (from which a ready fiber is stolen) is selected at random.

    #include <boost/fiber/algo/numa/work_stealing.hpp>

    namespace boost {
    namespace fibers {
    namespace algo {
    namespace numa {

    class work_stealing : public algorithm {
    public:
        work_stealing( std::uint32_t cpu_id,
                       std::uint32_t node_id,
                       std::vector< boost::fibers::numa::node > const& topo,
                       bool suspend = false);

        work_stealing( work_stealing const&) = delete;
        work_stealing( work_stealing &&) = delete;

        work_stealing & operator=( work_stealing const&) = delete;
        work_stealing & operator=( work_stealing &&) = delete;

        virtual void awakened( context *) noexcept;

        virtual context * pick_next() noexcept;

        virtual bool has_ready_fibers() const noexcept;

        virtual void suspend_until( std::chrono::steady_clock::time_point const&) noexcept;

        virtual void notify() noexcept;
    };

    }}}}

Constructor

    work_stealing( std::uint32_t cpu_id, std::uint32_t node_id,
                   std::vector< boost::fibers::numa::node > const& topo,
                   bool suspend = false);

##

Effects:

  Constructs work-stealing scheduling algorithm. The thread is pinned to logical cpu with ID `cpu_id`. If local ready-queue runs out of ready fibers, ready fibers are stolen from other schedulers using `topology` (represents the NUMA-topology of the system).

Throws:

  `system_error`

Note:

  If `suspend` is set to `true`, then the scheduler suspends if no ready fiber could be stolen. The scheduler will by woken up if a sleeping fiber times out or it was notified from remote (other thread or fiber scheduler).

Member function `awakened`()

    virtual void awakened( context * f) noexcept;

##

Effects:

  Enqueues fiber `f` onto the shared ready queue.

Throws:

  Nothing.

Member function `pick_next`()

    virtual context * pick_next() noexcept;

##

Returns:

  the fiber at the head of the ready queue, or `nullptr` if the queue is empty.

Throws:

  Nothing.

Note:

  Placing ready fibers onto the tail of the sahred queue, and returning them from the head of that queue, shares the thread between ready fibers in round-robin fashion.

Member function `has_ready_fibers`()

    virtual bool has_ready_fibers() const noexcept;

##

Returns:

  `true` if scheduler has fibers ready to run.

Throws:

  Nothing.

Member function `suspend_until`()

    virtual void suspend_until( std::chrono::steady_clock::time_point const& abs_time) noexcept;

##

Effects:

  Informs `work_stealing` that no ready fiber will be available until time-point `abs_time`. This implementation blocks in `std::condition_variable::wait_until()`.

Throws:

  Nothing.

Member function `notify`()

    virtual void notify() noexcept = 0;

##

Effects:

  Wake up a pending call to `work_stealing::suspend_until()`, some fibers might be ready. This implementation wakes `suspend_until()` via `std::condition_variable::notify_all()`.

Throws:

  Nothing.

# GPU computing

## CUDA

CUDA (Compute Unified Device Architecture) is a platform for parallel computing on NVIDIA GPUs. The application programming interface of CUDA gives access to GPU's instruction set and computation resources (Execution of compute kernels).

Waiting for CUDA streams

CUDA operation such as compute kernels or memory transfer (between host and device) can be grouped/queued by CUDA streams. are executed on the GPUs. Boost.Fiber enables a fiber to sleep (suspend) till a CUDA stream has completed its operations. This enables applications to run other fibers on the CPU without the need to spawn an additional OS-threads. And resume the fiber when the CUDA streams has finished.

    __global__
    void kernel( int size, int * a, int * b, int * c) {
        int idx = threadIdx.x + blockIdx.x * blockDim.x;
        if ( idx < size) {
            int idx1 = (idx + 1) % 256;
            int idx2 = (idx + 2) % 256;
            float as = (a[idx] + a[idx1] + a[idx2]) / 3.0f;
            float bs = (b[idx] + b[idx1] + b[idx2]) / 3.0f;
            c[idx] = (as + bs) / 2;
        }
    }

    boost::fibers::fiber f([&done]{
        cudaStream_t stream;
        cudaStreamCreate( & stream);
        int size = 1024 * 1024;
        int full_size = 20 * size;
        int * host_a, * host_b, * host_c;
        cudaHostAlloc( & host_a, full_size * sizeof( int), cudaHostAllocDefault);
        cudaHostAlloc( & host_b, full_size * sizeof( int), cudaHostAllocDefault);
        cudaHostAlloc( & host_c, full_size * sizeof( int), cudaHostAllocDefault);
        int * dev_a, * dev_b, * dev_c;
        cudaMalloc( & dev_a, size * sizeof( int) );
        cudaMalloc( & dev_b, size * sizeof( int) );
        cudaMalloc( & dev_c, size * sizeof( int) );
        std::minstd_rand generator;
        std::uniform_int_distribution<> distribution(1, 6);
        for ( int i = 0; i < full_size; ++i) {
            host_a[i] = distribution( generator);
            host_b[i] = distribution( generator);
        }
        for ( int i = 0; i < full_size; i += size) {
            cudaMemcpyAsync( dev_a, host_a + i, size * sizeof( int), cudaMemcpyHostToDevice, stream);
            cudaMemcpyAsync( dev_b, host_b + i, size * sizeof( int), cudaMemcpyHostToDevice, stream);
            kernel<<< size / 256, 256, 0, stream >>>( size, dev_a, dev_b, dev_c);
            cudaMemcpyAsync( host_c + i, dev_c, size * sizeof( int), cudaMemcpyDeviceToHost, stream);
        }
        auto result = boost::fibers::cuda::waitfor_all( stream); // suspend fiber till CUDA stream has finished
        BOOST_ASSERT( stream == std::get< 0 >( result) );
        BOOST_ASSERT( cudaSuccess == std::get< 1 >( result) );
        std::cout << "f1: GPU computation finished" << std::endl;
        cudaFreeHost( host_a);
        cudaFreeHost( host_b);
        cudaFreeHost( host_c);
        cudaFree( dev_a);
        cudaFree( dev_b);
        cudaFree( dev_c);
        cudaStreamDestroy( stream);
    });
    f.join();

Synopsis

    #include <boost/fiber/cuda/waitfor.hpp>

    namespace boost {
    namespace fibers {
    namespace cuda {

    std::tuple< cudaStream_t, cudaError_t > waitfor_all( cudaStream_t st);
    std::vector< std::tuple< cudaStream_t, cudaError_t > > waitfor_all( cudaStream_t ... st);

    }}}

Non-member function `cuda::waitfor()`

    #include <boost/fiber/cuda/waitfor.hpp>

    namespace boost {
    namespace fibers {
    namespace cuda {

    std::tuple< cudaStream_t, cudaError_t > waitfor_all( cudaStream_t st);
    std::vector< std::tuple< cudaStream_t, cudaError_t > > waitfor_all( cudaStream_t ... st);

    }}}

###

Effects:

  Suspends active fiber till CUDA stream has finished its operations.

Returns:

  tuple of stream reference and the CUDA stream status

## HIP

HIP is part of the ROC (Radeon Open Compute) platform for parallel computing on AMD and NVIDIA GPUs. The application programming interface of HIP gives access to GPU's instruction set and computation resources (Execution of compute kernels).

Waiting for HIP streams

HIP operation such as compute kernels or memory transfer (between host and device) can be grouped/queued by HIP streams. are executed on the GPUs. Boost.Fiber enables a fiber to sleep (suspend) till a HIP stream has completed its operations. This enables applications to run other fibers on the CPU without the need to spawn an additional OS-threads. And resume the fiber when the HIP streams has finished.

    __global__
    void kernel( int size, int * a, int * b, int * c) {
        int idx = threadIdx.x + blockIdx.x * blockDim.x;
        if ( idx < size) {
            int idx1 = (idx + 1) % 256;
            int idx2 = (idx + 2) % 256;
            float as = (a[idx] + a[idx1] + a[idx2]) / 3.0f;
            float bs = (b[idx] + b[idx1] + b[idx2]) / 3.0f;
            c[idx] = (as + bs) / 2;
        }
    }

    boost::fibers::fiber f([&done]{
        hipStream_t stream;
        hipStreamCreate( & stream);
        int size = 1024 * 1024;
        int full_size = 20 * size;
        int * host_a, * host_b, * host_c;
        hipHostMalloc( & host_a, full_size * sizeof( int), hipHostMallocDefault);
        hipHostMalloc( & host_b, full_size * sizeof( int), hipHostMallocDefault);
        hipHostMalloc( & host_c, full_size * sizeof( int), hipHostMallocDefault);
        int * dev_a, * dev_b, * dev_c;
        hipMalloc( & dev_a, size * sizeof( int) );
        hipMalloc( & dev_b, size * sizeof( int) );
        hipMalloc( & dev_c, size * sizeof( int) );
        std::minstd_rand generator;
        std::uniform_int_distribution<> distribution(1, 6);
        for ( int i = 0; i < full_size; ++i) {
            host_a[i] = distribution( generator);
            host_b[i] = distribution( generator);
        }
        for ( int i = 0; i < full_size; i += size) {
            hipMemcpyAsync( dev_a, host_a + i, size * sizeof( int), hipMemcpyHostToDevice, stream);
            hipMemcpyAsync( dev_b, host_b + i, size * sizeof( int), hipMemcpyHostToDevice, stream);
            hipLaunchKernel(kernel, dim3(size / 256), dim3(256), 0, stream, size, dev_a, dev_b, dev_c);
            hipMemcpyAsync( host_c + i, dev_c, size * sizeof( int), hipMemcpyDeviceToHost, stream);
        }
        auto result = boost::fibers::hip::waitfor_all( stream); // suspend fiber till HIP stream has finished
        BOOST_ASSERT( stream == std::get< 0 >( result) );
        BOOST_ASSERT( hipSuccess == std::get< 1 >( result) );
        std::cout << "f1: GPU computation finished" << std::endl;
        hipHostFree( host_a);
        hipHostFree( host_b);
        hipHostFree( host_c);
        hipFree( dev_a);
        hipFree( dev_b);
        hipFree( dev_c);
        hipStreamDestroy( stream);
    });
    f.join();

Synopsis

    #include <boost/fiber/hip/waitfor.hpp>

    namespace boost {
    namespace fibers {
    namespace hip {

    std::tuple< hipStream_t, hipError_t > waitfor_all( hipStream_t st);
    std::vector< std::tuple< hipStream_t, hipError_t > > waitfor_all( hipStream_t ... st);

    }}}

Non-member function `hip::waitfor()`

    #include <boost/fiber/hip/waitfor.hpp>

    namespace boost {
    namespace fibers {
    namespace hip {

    std::tuple< hipStream_t, hipError_t > waitfor_all( hipStream_t st);
    std::vector< std::tuple< hipStream_t, hipError_t > > waitfor_all( hipStream_t ... st);

    }}}

###

Effects:

  Suspends active fiber till HIP stream has finished its operations.

Returns:

  tuple of stream reference and the HIP stream status

# Running with worker threads

Keep workers running

If a worker thread is used but no fiber is created or parts of the framework (like `this_fiber::yield()`) are touched, then no fiber scheduler is instantiated.

    auto worker = std::thread(
            []{
                // fiber scheduler not instantiated
            });
    worker.join();

If use_scheduling_algorithm<>() is invoked, the fiber scheduler is created. If the worker thread simply returns, destroys the scheduler and terminates.

    auto worker = std::thread(
            []{
                // fiber scheduler created
                boost::fibers::use_scheduling_algorithm<my_fiber_scheduler>();
            });
    worker.join();

In order to keep the worker thread running, the fiber associated with the thread stack (which is called main fiber) is blocked. For instance the main fiber might wait on a `condition_variable`. For a gracefully shutdown `condition_variable` is signalled and the main fiber returns. The scheduler gets destructed if all fibers of the worker thread have been terminated.

    boost::fibers::mutex mtx;
    boost::fibers::condition_variable_any cv;
    auto worker = std::thread(
            [&mtx,&cv]{
                mtx.lock();
                // suspend till signalled
                cv.wait(mtx);
                mtx.unlock();
            });
    // signal termination
    cv.notify_all();
    worker.join();

Processing tasks

Tasks can be transferred via channels. The worker thread runs a pool of fibers that dequeue and executed tasks from the channel. The termination is signalled via closing the channel.

    using task = std::function<void()>;
    boost::fibers::buffered_channel<task> ch{1024};
    auto worker = std::thread(
            [&ch]{
                // create pool of fibers
                for (int i=0; i<10; ++i) {
                    boost::fibers::fiber{
                        [&ch]{
                            task tsk;
                            // dequeue and process tasks
                            while (boost::fibers::channel_op_status::closed!=ch.pop(tsk)){
                                tsk();
                            }
                        }}.detach();
                }
                task tsk;
                // dequeue and process tasks
                while (boost::fibers::channel_op_status::closed!=ch.pop(tsk)){
                    tsk();
                }
            });
    // feed channel with tasks
    ch.push([]{ ... });
    ...
    // signal termination
    ch.close();
    worker.join();

An alternative is to use a work-stealing scheduler. This kind of scheduling algorithm a worker thread steals fibers from the ready-queue of other worker threads if its own ready-queue is empty.

Wait till all worker threads have registered the work-stealing scheduling algorithm.

    boost::fibers::barrier b{2};
    boost::fibers::use_scheduling_algorithm<boost::fibers::algo::work_stealing>(2);
    boost::fibers::mutex mtx;
    boost::fibers::condition_variable_any cv;
    auto worker = std::thread(
            [&mtx,&cv,&b]{
                boost::fibers::use_scheduling_algorithm<boost::fibers::algo::work_stealing>(2);
                // wait till all threads joining the work stealing have been registered
                b.wait();
                mtx.lock();
                // suspend main-fiber from the worker thread
                cv.wait(mtx);
                mtx.unlock();
            });
    // wait till all threads have been registered the scheduling algorithm
    b.wait();
    // create fibers with tasks
    boost::fibers::fiber f{[]{ ... }};
    ...
    // signal termination
    cv.notify_all();
    worker.join();

Because the TIB (thread information block on Windows) is not fully described in the MSDN, it might be possible that not all required TIB-parts are swapped. Using WinFiber implementation might be an alternative (see documentation about implementations fcontext_t, ucontext_t and WinFiber of boost.context).

# Performance

Performance measurements were taken using `std::chrono::highresolution_clock`, with overhead corrections. The code was compiled with gcc-6.3.1, using build options: variant = release, optimization = speed. Tests were executed on dual Intel XEON E5 2620v4 2.2GHz, 16C/32T, 64GB RAM, running Linux (x86_64).

Measurements headed 1C/1T were run in a single-threaded process.

The microbenchmark syknet from Alexander Temerev was ported and used for performance measurements. At the root the test spawns 10 threads-of-execution (ToE), e.g. actor/goroutine/fiber etc.. Each spawned ToE spawns additional 10 ToEs ... until 1,000,000 ToEs are created. ToEs return back their ordinal numbers (0 ... 999,999), which are summed on the previous level and sent back upstream, until reaching the root. The test was run 10-20 times, producing a range of values for each measurement.

## time per actor/erlang process/goroutine (other languages) (average over 1,000,000)

Haskell | stack-1.4.0/ghc-8.0.1 | Go | go1.8.1 | Erlang | erts-8.3
--- | --- | ---
0.05 µs - 0.06 µs | 0.42 µs - 0.49 µs | 0.63 µs - 0.73 µs

Pthreads are created with a stack size of 8kB while `std::thread`'s use the system default (1MB - 2MB). The microbenchmark could not be run with 1,000,000 threads because of resource exhaustion (pthread and std::thread). Instead the test runs only at 10,000 threads.

## time per thread (average over 10,000 - unable to spawn 1,000,000 threads)

pthread | `std::thread` | `std::async`
--- | --- | ---
54 µs - 73 µs | 52 µs - 73 µs | 106 µs - 122 µs

The test utilizes 16 cores with Symmetric MultiThreading enabled (32 logical CPUs). The fiber stacks are allocated by `fixedsize_stack`.

As the benchmark shows, the memory allocation algorithm is significant for performance in a multithreaded environment. The tests use glibc’s memory allocation algorithm (based on ptmalloc2) as well as Google’s TCmalloc (via linkflags="-ltcmalloc"). Tais B. Ferreira, Rivalino Matias, Autran Macedo, Lucio B. Araujo An Experimental Study on Memory Allocators in Multicore and Multithreaded Applications, PDCAT ’11 Proceedings of the 2011 12th International Conference on Parallel and Distributed Computing, Applications and Technologies, pages 92-98

In the `work_stealing` scheduling algorithm, each thread has its own local queue. Fibers that are ready to run are pushed to and popped from the local queue. If the queue runs out of ready fibers, fibers are stolen from the local queues of other participating threads.

## time per fiber (average over 1.000.000)

fiber (16C/32T, work stealing, tcmalloc) | fiber (1C/1T, round robin, tcmalloc)
--- | ---
0.05 µs - 0.09 µs | 1.69 µs - 1.79 µs

# Tuning

Disable synchronization

With `BOOST_FIBERS_NO_ATOMICS` defined at the compiler’s command line, synchronization between fibers (in different threads) is disabled. This is acceptable if the application is single threaded and/or fibers are not synchronized between threads.

Memory allocation

Memory allocation algorithm is significant for performance in a multithreaded environment, especially for Boost.Fiber where fiber stacks are allocated on the heap. The default user-level memory allocator (UMA) of glibc is ptmalloc2 but it can be replaced by another UMA that fit better for the concret work-load For instance Google’s TCmalloc enables a better performance at the skynet microbenchmark than glibc’s default memory allocator.

Scheduling strategies

The fibers in a thread are coordinated by a fiber manager. Fibers trade control cooperatively, rather than preemptively. Depending on the work-load several strategies of scheduling the fibers are possible 1024cores.net: Task Scheduling Strategies that can be implmented on behalf of `algorithm`.

- work-stealing: ready fibers are hold in a local queue, when the fiber-scheduler's local queue runs out of ready fibers, it randomly selects another fiber-scheduler and tries to steal a ready fiber from the victim (implemented in `work_stealing` and `numa::work_stealing`)

- work-requesting: ready fibers are hold in a local queue, when the fiber-scheduler's local queue runs out of ready fibers, it randomly selects another fiber-scheduler and requests for a ready fibers, the victim fiber-scheduler sends a ready-fiber back

- work-sharing: ready fibers are hold in a global queue, fiber-scheduler concurrently push and pop ready fibers to/from the global queue (implemented in `shared_work`)

- work-distribution: fibers that became ready are proactivly distributed to idle fiber-schedulers or fiber-schedulers with low load

- work-balancing: a dedicated (helper) fiber-scheduler periodically collects informations about all fiber-scheduler running in other threads and re-distributes ready fibers among them

TTAS locks

Boost.Fiber uses internally spinlocks to protect critical regions if fibers running on different threads interact. Spinlocks are implemented as TTAS (test-test-and-set) locks, i.e. the spinlock tests the lock before calling an atomic exchange. This strategy helps to reduce the cache line invalidations triggered by acquiring/releasing the lock.

Spin-wait loop

A lock is considered under contention, if a thread repeatedly fails to acquire the lock because some other thread was faster. Waiting for a short time lets other threads finish before trying to enter the critical section again. While busy waiting on the lock, relaxing the CPU (via pause/yield mnemonic) gives the CPU a hint that the code is in a spin-wait loop.

- prevents expensive pipeline flushes (speculatively executed load and compare instructions are not pushed to pipeline)

- another hardware thread (simultaneous multithreading) can get time slice

- it does delay a few CPU cycles, but this is necessary to prevent starvation

It is obvious that this strategy is useless on single core systems because the lock can only released if the thread gives up its time slice in order to let other threads run. The macro BOOST_FIBERS_SPIN_SINGLE_CORE replaces the CPU hints (pause/yield mnemonic) by informing the operating system (via `std::this_thread_yield()`) that the thread gives up its time slice and the operating system switches to another thread.

Exponential back-off

The macro BOOST_FIBERS_RETRY_THRESHOLD determines how many times the CPU iterates in the spin-wait loop before yielding the thread or blocking in futex-wait. The spinlock tracks how many times the thread failed to acquire the lock. The higher the contention, the longer the thread should back-off. A Binary Exponential Backoff algorithm together with a randomized contention window is utilized for this purpose. BOOST_FIBERS_CONTENTION_WINDOW_THRESHOLD determines the upper limit of the contention window (expressed as the exponent for basis of two).

Speculative execution (hardware transactional memory)

Boost.Fiber uses spinlocks to protect critical regions that can be used together with transactional memory (see section Speculative execution).

TXS is enabled if property `htm=tsx` is specified at b2 command-line and `BOOST_USE_TSX` is applied to the compiler.

A TSX-transaction will be aborted if the floating point state is modified inside a critical region. As a consequence floating point operations, e.g. tore/load of floating point related registers during a fiber (context) switch are disabled.

NUMA systems

Modern multi-socket systems are usually designed as NUMA systems. A suitable fiber scheduler like `numa::work_stealing` reduces remote memory access (latence).

Parameters

## Parameters that migh be defiend at compiler's command line

Parameter | Default value | Effect on Boost.Fiber
--- | --- | ---
BOOST_FIBERS_NO_ATOMICS | - | no multithreading support, all atomics removed, no synchronization between fibers running in different threads
BOOST_FIBERS_SPINLOCK_STD_MUTEX | - | `std::mutex` used inside spinlock
BOOST_FIBERS_SPINLOCK_TTAS | + | spinlock with test-test-and-swap on shared variable
BOOST_FIBERS_SPINLOCK_TTAS_ADAPTIVE | - | spinlock with test-test-and-swap on shared variable, adaptive retries while busy waiting
BOOST_FIBERS_SPINLOCK_TTAS_FUTEX | - | spinlock with test-test-and-swap on shared variable, suspend on futex after certain number of retries
BOOST_FIBERS_SPINLOCK_TTAS_ADAPTIVE_FUTEX | - | spinlock with test-test-and-swap on shared variable, while busy waiting adaptive retries, suspend on futex certain amount of retries
BOOST_FIBERS_SPINLOCK_TTAS + BOOST_USE_TSX | - | spinlock with test-test-and-swap and speculative execution (Intel TSX required)
BOOST_FIBERS_SPINLOCK_TTAS_ADAPTIVE + BOOST_USE_TSX | - | spinlock with test-test-and-swap on shared variable, adaptive retries while busy waiting and speculative execution (Intel TSX required)
BOOST_FIBERS_SPINLOCK_TTAS_FUTEX + BOOST_USE_TSX | - | spinlock with test-test-and-swap on shared variable, suspend on futex after certain number of retries and speculative execution (Intel TSX required)
BOOST_FIBERS_SPINLOCK_TTAS_ADAPTIVE_FUTEX + BOOST_USE_TSX | - | spinlock with test-test-and-swap on shared variable, while busy waiting adaptive retries, suspend on futex certain amount of retries and speculative execution (Intel TSX required)
BOOST_FIBERS_SPIN_SINGLE_CORE | - | on single core machines with multiple threads, yield thread (`std::this_thread::yield()`) after collisions
BOOST_FIBERS_RETRY_THRESHOLD | 64 | max number of retries while busy spinning, the use fallback
BOOST_FIBERS_CONTENTION_WINDOW_THRESHOLD | 16 | max size of collisions window, expressed as exponent for the basis of two
BOOST_FIBERS_SPIN_BEFORE_SLEEP0 | 32 | max number of retries that relax the processor before the thread sleeps for 0s
BOOST_FIBERS_SPIN_BEFORE_YIELD | 64 | max number of retries where the thread sleeps for 0s before yield thread (`std::this_thread::yield()`)

# Customization

Overview

As noted in the Scheduling section, by default Boost.Fiber uses its own `round_robin` scheduler for each thread. To control the way Boost.Fiber schedules ready fibers on a particular thread, in general you must follow several steps. This section discusses those steps, whereas Scheduling serves as a reference for the classes involved.

The library's fiber manager keeps track of suspended (blocked) fibers. Only when a fiber becomes ready to run is it passed to the scheduler. Of course, if there are fewer than two ready fibers, the scheduler's job is trivial. Only when there are two or more ready fibers does the particular scheduler implementation start to influence the overall sequence of fiber execution.

In this section we illustrate a simple custom scheduler that honors an integer fiber priority. We will implement it such that a fiber with higher priority is preferred over a fiber with lower priority. Any fibers with equal priority values are serviced on a round-robin basis.

The full source code for the examples below is found in priority.cpp.

Custom Property Class

The first essential point is that we must associate an integer priority with each fiber. A previous version of the Fiber library implicitly tracked an int priority for each fiber, even though the default scheduler ignored it. This has been dropped, since the library now supports arbitrary scheduler-specific fiber properties.

One might suggest deriving a custom `fiber` subclass to store such properties. There are a couple of reasons for the present mechanism.

1. Boost.Fiber provides a number of different ways to launch a fiber. (Consider `fibers::async()`.) Higher-level libraries might introduce additional such wrapper functions. A custom scheduler must associate its custom properties with every fiber in the thread, not only the ones explicitly launched by instantiating a custom `fiber` subclass.

2. Consider a large existing program that launches fibers in many different places in the code. We discover a need to introduce a custom scheduler for a particular thread. If supporting that scheduler's custom properties required a particular `fiber` subclass, we would have to hunt down and modify every place that launches a fiber on that thread.

3. The `fiber` class is actually just a handle to internal `context` data. A subclass of `fiber` would not add data to `context`.

The present mechanism allows you to drop in a custom scheduler with its attendant custom properties without altering the rest of your application.

Instead of deriving a custom scheduler fiber properties subclass from `fiber`, you must instead derive it from `fiber_properties`.

    class priority_props : public boost::fibers::fiber_properties {
    public:
        priority_props( boost::fibers::context * ctx):
            fiber_properties( ctx),
            priority_( 0) {
        }

        int get_priority() const {
            return priority_;
        }

        // Call this method to alter priority, because we must notify
        // priority_scheduler of any change.
        void set_priority( int p) {
            // Of course, it's only worth reshuffling the queue and all if we're
            // actually changing the priority.
            if ( p != priority_) {
                priority_ = p;
                notify();
            }
        }

        // The fiber name of course is solely for purposes of this example
        // program; it has nothing to do with implementing scheduler priority.
        // This is a public data member -- not requiring set/get access methods --
        // because we need not inform the scheduler of any change.
        std::string name;
    private:
        int priority_;
    };

Your subclass constructor must accept a ``context`*` and pass it to the `fiber_properties` constructor.

Provide read access methods at your own discretion.

It's important to call `notify()` on any change in a property that can affect the scheduler's behavior. Therefore, such modifications should only be performed through an access method.

A property that does not affect the scheduler does not need access methods.

Custom Scheduler Class

Now we can derive a custom scheduler from `algorithm_with_properties<>`, specifying our custom property class `priority_props` as the template parameter.

    class priority_scheduler :
        public boost::fibers::algo::algorithm_with_properties< priority_props > {
    private:
        typedef boost::fibers::scheduler::ready_queue_type   rqueue_t;

        rqueue_t                                rqueue_;
        std::mutex                  mtx_{};
        std::condition_variable     cnd_{};
        bool                        flag_{ false };

    public:
        priority_scheduler() :
            rqueue_() {
        }

        // For a subclass of algorithm_with_properties<>, it's important to
        // override the correct awakened() overload.
        virtual void awakened( boost::fibers::context * ctx, priority_props & props) noexcept {
            int ctx_priority = props.get_priority();
            // With this scheduler, fibers with higher priority values are
            // preferred over fibers with lower priority values. But fibers with
            // equal priority values are processed in round-robin fashion. So when
            // we're handed a new context*, put it at the end of the fibers
            // with that same priority. In other words: search for the first fiber
            // in the queue with LOWER priority, and insert before that one.
            rqueue_t::iterator i( std::find_if( rqueue_.begin(), rqueue_.end(),
                [ctx_priority,this]( boost::fibers::context & c)
                { return properties( &c ).get_priority() < ctx_priority; }));
            // Now, whether or not we found a fiber with lower priority,
            // insert this new fiber here.
            rqueue_.insert( i, * ctx);
        }

        virtual boost::fibers::context * pick_next() noexcept {
            // if ready queue is empty, just tell caller
            if ( rqueue_.empty() ) {
                return nullptr;
            }
            boost::fibers::context * ctx( & rqueue_.front() );
            rqueue_.pop_front();
            return ctx;
        }

        virtual bool has_ready_fibers() const noexcept {
            return ! rqueue_.empty();
        }

        virtual void property_change( boost::fibers::context * ctx, priority_props & props) noexcept {
            // Although our priority_props class defines multiple properties, only
            // one of them (priority) actually calls notify() when changed. The
            // point of a property_change() override is to reshuffle the ready
            // queue according to the updated priority value.

            // 'ctx' might not be in our queue at all, if caller is changing the
            // priority of (say) the running fiber. If it's not there, no need to
            // move it: we'll handle it next time it hits awakened().
            if ( ! ctx->ready_is_linked()) {
                return;
            }

            // Found ctx: unlink it
            ctx->ready_unlink();

            // Here we know that ctx was in our ready queue, but we've unlinked
            // it. We happen to have a method that will (re-)add a context* to the
            // right place in the ready queue.
            awakened( ctx, props);
        }

        void suspend_until( std::chrono::steady_clock::time_point const& time_point) noexcept {
            if ( (std::chrono::steady_clock::time_point::max)() == time_point) {
                std::unique_lock< std::mutex > lk( mtx_);
                cnd_.wait( lk, [this](){ return flag_; });
                flag_ = false;
            } else {
                std::unique_lock< std::mutex > lk( mtx_);
                cnd_.wait_until( lk, time_point, [this](){ return flag_; });
                flag_ = false;
            }
        }

        void notify() noexcept {
            std::unique_lock< std::mutex > lk( mtx_);
            flag_ = true;
            lk.unlock();
            cnd_.notify_all();
        }
    };

See ready_queue_t.

You must override the `algorithm_with_properties::awakened()` method. This is how your scheduler receives notification of a fiber that has become ready to run.

`props` is the instance of priority_props associated with the passed fiber `ctx`.

You must override the `algorithm_with_properties::pick_next()` method. This is how your scheduler actually advises the fiber manager of the next fiber to run.

You must override `algorithm_with_properties::has_ready_fibers()` to inform the fiber manager of the state of your ready queue.

Overriding `algorithm_with_properties::property_change()` is optional. This override handles the case in which the running fiber changes the priority of another ready fiber: a fiber already in our queue. In that case, move the updated fiber within the queue.

Your `property_change()` override must be able to handle the case in which the passed `ctx` is not in your ready queue. It might be running, or it might be blocked.

Our example `priority_scheduler` doesn't override `algorithm_with_properties::new_properties()`: we're content with allocating `priority_props` instances on the heap.

Replace Default Scheduler

You must call `use_scheduling_algorithm()` at the start of each thread on which you want Boost.Fiber to use your custom scheduler rather than its own default `round_robin`. Specifically, you must call `use_scheduling_algorithm()` before performing any other Boost.Fiber operations on that thread.

    int main( int argc, char *argv[]) {
        // make sure we use our priority_scheduler rather than default round_robin
        boost::fibers::use_scheduling_algorithm< priority_scheduler >();
        ...
    }

Use Properties

The running fiber can access its own `fiber_properties` subclass instance by calling `this_fiber::properties()`. Although `properties<>()` is a nullary function, you must pass, as a template parameter, the `fiber_properties` subclass.

    boost::this_fiber::properties< priority_props >().name = "main";

Given a `fiber` instance still connected with a running fiber (that is, not `fiber::detach()`ed), you may access that fiber's properties using `fiber::properties()`. As with `this_fiber::properties<>()`, you must pass your `fiber_properties` subclass as the template parameter.

    template< typename Fn >
    boost::fibers::fiber launch( Fn && func, std::string const& name, int priority) {
        boost::fibers::fiber fiber( func);
        priority_props & props( fiber.properties< priority_props >() );
        props.name = name;
        props.set_priority( priority);
        return fiber;
    }

Launching a new fiber schedules that fiber as ready, but does not immediately enter its fiber-function. The current fiber retains control until it blocks (or yields, or terminates) for some other reason. As shown in the `launch()` function above, it is reasonable to launch a fiber and immediately set relevant properties -- such as, for instance, its priority. Your custom scheduler can then make use of this information next time the fiber manager calls `algorithm_with_properties::pick_next()`.

# Rationale

preprocessor defines

## preopcessor defines

BOOST_FIBERS_NO_ATOMICS | no `std::atomic<>` used, inter-thread synchronization disabled
BOOST_FIBERS_SPINLOCK_STD_MUTEX | use `std::mutex` as spinlock instead of default `XCHG`-sinlock with backoff
BOOST_FIBERS_SPIN_BACKOFF | limit determines when to used `std::this_thread::yield()` instead of mnemonic `pause/yield` during busy wait (apllies on to `XCHG`-spinlock)
BOOST_FIBERS_SINGLE_CORE | allways call `std::this_thread::yield()` without backoff during busy wait (apllies on to `XCHG`-spinlock)

distinction between coroutines and fibers

The fiber library extends the coroutine library by adding a scheduler and synchronization mechanisms.

- a coroutine yields

- a fiber blocks

When a coroutine yields, it passes control directly to its caller (or, in the case of symmetric coroutines, a designated other coroutine). When a fiber blocks, it implicitly passes control to the fiber scheduler. Coroutines have no scheduler because they need no scheduler. 'N4024: Distinguishing coroutines and fibers' .

what about transactional memory

GCC supports transactional memory since version 4.7. Unfortunately tests show that transactional memory is slower (ca. 4x) than spinlocks using atomics. Once transactional memory is improved (supporting hybrid tm), spinlocks will be replaced by __transaction_atomic{} statements surrounding the critical sections.

synchronization between fibers running in different threads

Synchronization classes from Boost.Thread block the entire thread. In contrast, the synchronization classes from Boost.Fiber block only specific fibers, so that the scheduler can still keep the thread busy running other fibers in the meantime. The synchronization classes from Boost.Fiber are designed to be thread-safe, i.e. it is possible to synchronize fibers running in different threads as well as fibers running in the same thread. (However, there is a build option to disable cross-thread fiber synchronization support; see this description.)

spurious wakeup

Spurious wakeup can happen when using `std::condition_variable`: the condition variable appears to be have been signaled while the awaited condition may still be false. Spurious wakeup can happen repeatedly and is caused on some multiprocessor systems where making `std::condition_variable` wakeup completely predictable would slow down all `std::condition_variable` operations. David R. Butenhof Programming with POSIX Threads

`condition_variable` is not subject to spurious wakeup. Nonetheless it is prudent to test the business-logic condition in a `wait()` loop — or, equivalently, use one of the `wait( lock, predicate )` overloads.

See also No Spurious Wakeups.

migrating fibers between threads

Support for migrating fibers between threads has been integrated. The user-defined scheduler must call `context::detach()` on a fiber-context on the source thread and `context::attach()` on the destination thread, passing the fiber-context to migrate. (For more information about custom schedulers, see Customization.) Examples `work_sharing` and `work_stealing` in directory `examples` might be used as a blueprint.

See also Migrating fibers between threads.

support for Boost.Asio

Support for Boost.Asio’s async-result is not part of the official API. However, to integrate with a `boost::asio::io_service`, see Sharing a Thread with Another Main Loop. To interface smoothly with an arbitrary Asio async I/O operation, see Then There’s Boost.Asio.

tested compilers

The library was tested with GCC-5.1.1, Clang-3.6.0 and MSVC-14.0 in c++11-mode.

supported architectures

Boost.Fiber depends on Boost.Context - the list of supported architectures can be found here.

# Acknowledgments

I'd like to thank Agustín Bergé, Eugene Yakubovich, Giovanni Piero Deretta and especially Nat Goodspeed.

# Appendix: Installing and Running Tests

Installing the Fiber library

As Fiber is not yet officially part of Boost, it is necessary to embed it in an existing Boost source tree.

The downloaded Fiber library can be placed into an existing Boost source tree by moving the top-level Fiber directory to `libs/fiber` under the top-level Boost directory, then further moving `libs/fiber/include/boost/fiber` (in other words, the Fiber library's `include/boost/fiber` directory) to `boost/fiber` under the top-level Boost directory.

On a Posix system such as Linux or OS X, you may use symlinks instead.

Create a symlink from the Boost directory's `libs/fiber` to the top-level Fiber directory, e.g.:

    cd ~/boost_1_61_0
    ln -s ~/boost-fiber-master libs/fiber

Then create a symlink from the Boost directory's `boost/fiber` to the Fiber library's `include/boost/fiber` directory:

    cd boost
    ln -s ../libs/fiber/include/boost/fiber fiber

For some versions of the Boost.Build system, it was important to use a relative symlink of that form for `boost/fiber`.

Running Tests

Once the Fiber library has been overlaid (or symlinked) into the Boost source tree this way, the Boost.Build system can build it like any other Boost library. In particular:

    cd ~/boost_1_61_0
    ./bootstrap.sh
    ./b2 libs/fiber/test

On Windows, the commands would look more like:

    cd /D %HOMEDRIVE%%HOMEPATH%\boost_1_61_0
    bootstrap
    b2 libs\fiber\test