crew 1.3.1

Threaded dispatcher updates

• Deprecate the dispatchers() method of crew_monitor_local() objects because the dispatcher is a thread in mirai > 2.6.1.
• Remove mirai_status() internal error handling for dispatcher queries. Now that the mirai dispatcher is a thread, crew can trust direct calls to mirai::info() because it is a direct memory lookup rather than a request to an external process.
• Add a synced() method for controllers and controller groups to check that task counts from mirai::info() are up to date with tasks pushed through the controller.
• Use synced() as a guardrail in wait() to prevent a race condition where mirai::info() briefly undercounts newly pushed tasks.
• Guard pop() and collect() against a sub-millisecond race where mirai::info() reports a task as resolved before the task handle itself is ready. If counts say a task should be available but the resolved queue is empty, briefly retry the handle scan before returning NULL.

Other updates

• Use later::global_loop() instead of later::current_loop() in controller$autoscale().
• Test crew_worker() with autoexit = FALSE (https://github.com/r-lib/mirai/issues/535).
• Simplify controller$pop_backlog() and controller$saturated().
• Fix intermittent test failures in crash detection tests by retrying scale() while waiting for worker connections.
• Remove the compatibility layer of the deprecated internal async class for worker launches in plugins.

crew 1.3.0

• Reinstate checks that were temporarily suppressed to help plugins transition to #217.
• Support reset_globals, reset_packages, reset_options, and garbage_collection in crew_controller_sequential() (#217).
• Change argument tls to tlscert in the call to mirai::daemon() (#227, @shikokuchuo).
• Use collections queues instead of custom queues (#229).
• Use collections dictionaries instead of hash environments to track tasks (#229).
• Remove the long-deprecated promise() method of the controller.
• Improve interoperability of crew’s workers by avoiding nanonext::cv_value() for condition variables (#225). Consequences:
  • wait(mode = "all") no longer guarantees that a task is available for pop(). It just consumes a condition variable signal. pop() should always be checked for NULL return values.
  • Until mirai gains a threaded dispatcher, saturated() needs to avoid the overhead of calling status() to get task counts. So the definition of “saturated” has changed: a controller is saturated if the number of uncollected tasks is greater than or equal to the maximum number of workers. Previously, it was the number of unresolved tasks.
  • Internal counters .pushed and .popped are safely removed.
  • Active bindings pushed and popped are removed.
• Controllers no longer have a unpopped() method because it is unnecessary to export and it assumes only the controller submits tasks to the compute profile (affecting interoperability).
• Allow a custom later loop in autoscale().
• Deprecate the pids() methods in controllers and clients and stop babysitting the dispatcher process (#236).
• Deprecate crew_clean() in favor of crew_monitor_local() (#236).
• Trim down the argument list of launch_worker() to just include arguments call and name.
• Simplify the auto-scaling algorithm to support greater interoperability and the eventual threaded dispatcher in mirai (#232).
• Deprecate processes and async in crew launchers in favor of job arrays (#218, #237).
• Drop the unified controller group throttle. Let individual controllers handle their own throttling based on their own auto-scaling settings.
• When tasks_mask is finite, set seconds_min to seconds_interval in the throttle to give tasks enough time to accumulate.
• Use the counters in mirai::info() instead of mirai::status()$events (#232, @shikokuchuo).
• Allow custom compute profiles in crew_client(). This is mainly to support the use of the default compute profile to make crew more compatible with non-crew mirai-based workflows. It’s a quick solution for single controllers but does not work for controller groups.
• crew_client()$start() now errors if a mirai compute profile of the same name is already active.
• Shorten the length of strings from crew_random_name().

crew 1.2.1

• CRAN patch (address MKL “Additional issues”).

crew 1.2.0

• 22.7x speedup in controller$walk() with 100000 tasks: pre-compute the task list and defer auto-scaling until all tasks are pushed. Pushing a million tasks in walk() now takes just under 70 seconds on a local M2 Macbook.
• Use nanonext::ip_addr() instead of getip::getip() to get the default IP address of the host (#216, @shikokuchuo).
• Add a progress bar to walk() report progress when pushing tasks.
• Move arguments reset_globals, reset_packages, reset_options, and garbage_collection from the launcher to the controller (#217). crew_eval() now handles cleanup directly, and workflows running mirai 2.0.0 should mostly run faster because not all cleanup steps are automatically enabled.
• Avoid mirai::call_mirai_() in tests (#222, @shikokuchuo).

crew 1.1.2

• Speedup for targets: avoid superfluous processing in controller$pop().
• Speedup queue methods, especially queue$push(), by making the queue non-portable and using direct access and direct assignment in R6.
• Reduce the frequency of calls to mirai::status() in workloads with a large number of quick tasks. Instead of resetting the throttle every time there is task or worker activity, only reset if there is worker activity. This was achieved by giving relay objects their own throttles instead of having them use the launcher throttles.
• Revert to a logical value for the dispatcher argument of mirai::daemons() (#215, @shikokuchuo).

crew 1.1.1

• Fix incorrect messages in documentation about deprecating the name and workers arguments of controllers. They are not deprecated at the controller level. They are only deprecated in crew_client().

crew 1.1.0

• Change prefix in name_worker() from crew- to crew-worker-.
• Use call_mirai() instead of the now deprecated call_mirai_() (@shikokuchuo, #198).
• Add crew_controller_sequential() (#205).
• Add a serialization argument to crew_client() (#204).

crew 1.0.0

• Avoid partial match in a call to crew_terminate_process().
• Migrate to mirai 2.0.0 with the pair 1 poly protocol (#193). Most notably, all workers now share the same URL instead of having different web sockets for different workers. This gets rid of crew’s previous “slotted” model for workers.
• Prohibit duplicated task names in push(). (The task must be popped first before another task of the same name can be submitted.) This allows controllers to use hash tables to track tasks, which makes push() and pop() orders of magnitude faster. It is also needed for the new retry mechanism based on backup controllers and controller groups.
• Use exponential backoff in throttles.
• Share throttles between scale() and wait() to increase responsiveness.
• Use a unified throttle for controller groups, which requires refactoring wait(mode = "all") for controller groups.
• Name every task and record every command.
• Switch URLs from ws:// (and wss://) to tcp:// (and tls+tcp://) (#193).
• Change R-level error status code to -1 (previously 1).
• Add a queue class for resolved tasks to make pop() and collect() more efficient. pop() is 64 times faster.
• For clarity, rename the "launcher" column of the results to "controller".
• Log crashes and cancellations separately from ordinary errors in summary().

crew 0.10.2

• Eliminate spurious launch_max error from underutilized workers (#189).
• Deprecate launch_max in favor of crashes_error (#189).
• Look for crashes of all workers in rotate() instead of looking for crashes of a specific worker in launch() (#189).
• Add a crashes() launcher method to allow plugins to detect and respond to crashes more easily.
• Change default seconds_idle to 300.
• Move autometric to Suggests:.

crew 0.10.1

• Instrument crew_eval() with autometric::log_phase_set() and autometric::log_phase_reset().

crew 0.10.0

• Give full worker name in the worker launch error message (@multimeric).
• Start the controller automatically in more methods.
• Add an r_arguments argument to supply command line arguments to R for workers (#175, @rpruim).
• Add a cancel() controller method to cancel one or more tasks.
• Call cancel(all = TRUE) from terminate() as a safeguard.
• Make client$dispatcher a ps::ps_handle() handle and add a new client$client handle for the current process.
• To passively log memory usage when log_resources is given, the controller now calls log() as a side effect in most controller methods, with throttling to preserve speed.
• Return a status and status code from pop() etc.
• New internal function as_monad() makes error reporting more consistent.
• Use path.expand() on local log files.
• Switch to Rscript.
• Print session info from crew_worker() before attempting to dial into mirai.
• Add asyncdial = FALSE back to mirai::daemon() call.
• Deprecate local_log_directory and local_log_join in favor of crew_options_local() and the options_local argument.
• Add crew_options_metrics() and the options_metrics argument for recording resource metrics (#178).
• Add a new logging.Rmd vignette to explain best practices for logging and resource usage metrics (#178).

crew 0.9.5

• CRAN patch.

crew 0.9.4

• Do not use extended tasks in Shiny vignette.
• Add a new retry_tasks argument with default TRUE (#170).
• Avoid Base64 encoding functions from nanonext

crew 0.9.3

• Deprecate native crew promises in favor of native event-driven promises in mirai (#162).
• Update the shiny.Rmd and promises.Rmd vignette to recommend native event-driven mirai promises (#162).

crew 0.9.2

• Use .args rather than ... in mirai::mirai() to make sure arguments continue to be passed as local variables in mirai >= 0.13.1.9012.
• Add new controller methods autoscale(), descale(), and started() to facilitate different kinds of Shiny apps.
• Deprecate the scale and throttle methods of controller$promise(). promise() now always calls autoscale() to make sure one and only one auto-scaling loop is running asynchronously. Auto-scaling thus continues even after the promise resolves.
• Add a second example vignette that simulates coin flips.
• Add a new error argument to collect() (#166).

crew 0.9.1

• Rewrite the async Shiny vignette with crew promises and Shiny extended tasks (#157, @jcheng5).
• Clarify the intent of controller$promise(mode = "one") in the vignette on promises (@jcheng5).
• Implement an error argument in pop() which may help with integration with ExtendedTask (@jcheng5).
• Handle task errors in the Shiny vignette (@jcheng5).

crew 0.9.0

• Require nanonext >= 0.12.0 and mirai >= 0.12.0.
• Return to always re-launching backlogged inactive workers (#79, https://github.com/r-lib/mirai/discussions/95).
• Implement push_backlog() and pop_backlog() to manage cases when it is not desirable to push to saturated controllers (https://github.com/ropensci/targets/issues/1220).
• Invisibly return the mirai object of a task pushed to the controller. This allows users to interact with the task directly, e.g. to create a promise object with promises::as.promise() (#146, @jcheng5).
• Add a new walk() method for controllers and controller groups to submit tasks in batch and return control immediately without waiting for any task to complete (#148, @jcheng5).
• Revive the collect() method for popping multiple tasks at once (#148, @jcheng5).
• Add controller group methods nonempty(), resolved(), unresolved(), and unpopped() to help with #148.
• Make the mirai dispatcher error message extremely verbose.

crew 0.8.0

• Configure workers to send themselves a termination signal if the connection to the dispatcher is broken (#141, @psychelzh). Huge thanks to @shikokuchuo for the support through https://github.com/r-lib/mirai/issues/87, https://github.com/r-lib/mirai/pull/88, and https://github.com/r-lib/nanonext/pull/25! The signal itself is platform-dependent and determined by the new function crew_terminate_signal().
• Implement crew_monitor_local() to help users monitor and terminate local R processes created by crew and mirai.
• Implement new utility function crew_terminate_process() to terminate a process manually without resorting to SIGKILL on Windows.
• Throw a warning from controller$map() if at least one task threw one. warnings = FALSE suppresses this behavior.
• Set output = TRUE in daemon() so stdout and stderr streams print.
• Add new arguments local_log_directory and local_log_join to write to local log files.

crew 0.7.0

• Migrate from asyncdial to autoexit.
• Use Sys.info()[["user"]] to get the user in crew_clean() (#135, @luwidmer).
• Use condition variables to wait for tasks more efficiently (#108).
• Because of #108, controller$map() can no longer be used if there are tasks waiting to be popped.
• Use a cli progress bar in map().
• Encapsulate non-function R6 members in the private list and add active bindings where interfaces are necessary (#137). Ad hoc tests that absolutely need to modify private objects can use object$.__enclos_env__$private trick, but these tests should be skipped on CRAN in case there is a change to R6 that breaks this.
• Drop .signal from mirai::mirai() since all mirai tasks signal as of mirai version 0.11.2.9025.
• Implement crew_throttle(), a decoupled mechanism for throttling that can be applied to scaling.
• Bring back the throttle argument, powered by crew_throttle().
• Retry mirai::status() again in daemons_info() and make it configurable using seconds_interval and seconds_timeout in both the client and the launcher (#128).

crew 0.6.0

• Migrate checks to enforce features in version 0.5.0 for reverse dependencies.
• Drop check of backlogged workers, c.f. https://github.com/r-lib/mirai/discussions/63#discussioncomment-7051889 (#79, #124, @shikokuchuo).
• Deprecate seconds_exit because exitlinger in mirai is now obsolete (#125, @shikokuchuo).
• Use mirai::nextget("cv") to count unresolved tasks instead of looping through all the task objects (#131).
• Remove throttling and collect() in auto-scaling. Simplifies much of the code. Made possible by the efficiency gains in #131.
• Simplify wait().
• seconds_interval in map() no longer defaults to controller$client$seconds_interval.
• launcher$terminate_workers() terminates one or more workers, and launcher$terminate terminates the whole launcher.
• Add infrastructure to let custom launcher plugins launch and terminate workers asynchronously (#133). Launchers can set a positive number in the processes field to set the number of local mirai daemons for asynchronous requests to launch and terminate the serious workers. Then, launch_worker() and terminate_worker() can optionally make use of launcher$async$eval() to send these asynchronous calls.

crew 0.5.0

• Suppress interactive browser on Windows which launched on each worker previously (@psychelzh).
• Migrate to the new host/daemon nomenclature in mirai 0.9.1 (#96).
• Suppress status() retries (@shikokuchuo, #100).
• Implement launch_max to error out if workers repeatedly launch without completing any tasks (#101, @shikokuchuo, @multimeric).
• Detect discovered workers more robustly in launcher$done().
• Add a new algorithm argument to the push(), shove(), and map() methods of controllers and controller groups, as well as arguments/fields in crew_eval() and its return value (#113, @shikokuchuo).
• As a default for pseudo-random number generation, leverage widely-spaced L’Ecuyer streams as supported by mirai::nextstream() (#115, @shikokuchuo).
• Move README documentation to vignettes.
• Add a new crew_tls() function for TLS configuration (#120).
• Deprecate the tls_enable and tls_config arguments of crew_client() etc. in favor of a tls argument which accepts crew_tls() objects (#120).

crew 0.4.0

Highlights

• Use the TLS capabilities of nanonext and mirai.
• Add a map() method to the controller class for functional programming (#93).
• Fix bug in scale() which launched too many workers when active - demand was less than 0.
• Add crew_clean() to help clean up any lingering dispatchers and workers from previous sessions.

Other improvements

• Remove temporary controller summary workaround that ensured CRAN compatibility with targets 1.1.3.
• Add guardrails to make sure the controller is started for methods that require this. Part of this involved improving the error messages.
• Retry daemons() in daemons_info().

crew 0.3.0

• Track warnings properly (#81, @brendanf).
• Optimize heavily (#81, #83, @shikokuchuo, @brendanf).
• Use CRAN nanonext 0.9.0.
• Delegate the task pushing and collection logic to a new R6 schedule class (#84).
• Delegate auto-scaling to the launcher (#84).
• Drastically simplify internal logic (#84).
• Rename “router” to “client” and change the function signature of launch_worker() (#84). Unfortunately neither of these changes is back-compatible for third-party launcher plugins. However, the controller interface is still back-compatible, so targets remains unaffected.
• Terminate “lost” workers on websocket rotation.
• Allow NULL result in pop()$result[[1L]].
• Delegate summaries among the controller, schedule, launcher, and client.
• Default to the loopback address for the local controller.
• Use already cumulative assigned and complete stats from mirai 0.8.7.9029.
• Reorganize pkgdown reference and Rd families to clarify which parts are for end-users and which parts are for launcher plugin developers.
• Add controller$shove() and schedule$list() to support fast extensions for functional programming.
• Depend on mirai 0.9.0 or greater.

crew 0.2.1

• Use packageStartupMessage().

crew 0.2.0

• Use .args instead of ... in mirai::mirai().
• Use set.seed() instead of withr::local_seed() (#67, @shikokuchuo).
• Add a vignette to showcase integration with Shiny, co-authored with @dwoodie and with helpful suggestions by @shikokuchuo.
• Drop rlang::duplicate() (#72, @shikokuchuo).
• Do not poll dispatcher availability when starting router objects (@shikokuchuo).
• Implement a saturated() controller method to support targets.
• Add a worker_index column to controller summaries.
• When relaunching workers, prioritize those with unresolved tasks stuck at the NNG level (#75, @shikokuchuo).
• Always relaunch backlogged inactive workers (#79).
• Deprecate the auto_scale argument/field of controllers in favor of the scale argument of push(), pop(), and wait().
• Throttle auto-scaling with interval self$router$seconds_interval (#76).
• Remove clean() and instead terminate lost workers on launch.
• Fix examples.
• Add a collect argument to pop().

crew 0.1.1

• Ensure checks pass on CRAN.

crew 0.1.0

• Remove asyncdial from call to mirai::daemons().
• Explain vision in the launcher plugin vignette.
• Remove the superfluous clean() controller method.
• Clean up logic for scale() and launch() controller methods.
• Add a new inactive() launcher method to help with the above.
• Eliminate superfluous worker launches: clean up the controller logic, and check worker discoveries before worker connections (#51).
• Simplify the launcher plugin interface (#54).
• Commit to using the fast nanonext::.unresolved() (#49).
• Rename the callr launcher to the local process launcher (#56).
• Switch from SHA224 to SHA1 as the hash for random tokens and names (#59, @shikokuchuo).
• Open and confirm bus socket listeners asynchronously in the launcher abstract class (#58).
• Use condition variables to test the connection status of worker instances (#57, @shikokuchuo).
• Allow the dispatcher process to not exist so that the router will keep working if mirai implements the dispatcher another way (#60).
• Add a substitute argument to push() (#63).
• Remove dependency on custom NNG bus sockets, which significantly simplifies the internals, improves robustness, and smooths the integration with mirai (#61).
• Use processx instead of callr for the local process launcher (#64).
• Speed up crew_eval() by e.g. relying on nanonext::mclock() for the time.
• Add an empty() method to check if the controller or controller group is empty.
• Rely on mirai for globals cleanup, garbage collection, and global option resets (#65).
• Expose the package unloading capabilities of mirai (#65).
• Make monads classed environments and leave tibbles to the last minute in pop().
• Rename crew_wait() to crew_retry().
• Export crew_assert() and crew_random_name() to help launcher plugins.
• Rename the listen() method to start() in the router class.
• The poll() method now retries mirai::daemons() for robustness.
• crew_retry() gains a max_tries argument.
• Bug fix: stop setting router$daemons to NULL in poll() when the connection has a momentary problem.
• Simplify listening().

crew 0.0.5

• Write a call() method to make it easier to write custom launcher plugins.
• Add robust documentation and tests for custom launcher plugins.
• Use as.character() to make sure the list argument supplied to rm() is safe. Fixes a test that was failing on R-devel with crew version 0.0.4.
• Remove the seconds_poll_high and seconds_poll_low fields and arguments for compatibility with mirai development and release versions.
• Hard-code the async_dial field in the router for compatibility with mirai development and release versions.

crew 0.0.4

• Adjust references and tests as requested by CRAN.

crew 0.0.3

• First version.