Reinit empty project.

LICENSE.md

@@ -1,6 +1,6 @@
 # The MIT License (MIT)
 
-Copyright (c) 2015-2019 Roberto Ostinelli and Neato Robotics, Inc.
+Copyright (c) 2015-2021 Roberto Ostinelli and Neato Robotics, Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,566 +1,7 @@
 [![Build Status](https://travis-ci.com/ostinelli/syn.svg?branch=master)](https://travis-ci.com/ostinelli/syn) [![Hex pm](https://img.shields.io/hexpm/v/syn.svg)](https://hex.pm/packages/syn)
 
-# Syn (v2)
-**Syn** (short for _synonym_) is a global Process Registry and Process Group manager for Erlang and Elixir. Syn automatically manages addition / removal of nodes from the cluster, and is also able to recover from net splits.
+# Syn (v3)
+**Syn** (short for _synonym_) is a global Process Registry and Process Group manager for Erlang and Elixir.
+Syn automatically manages addition / removal of nodes from the cluster, and is also able to recover from net splits.
 
-## Introduction
-
-##### What is a Process Registry?
-A global Process Registry allows registering a process on all the nodes of a cluster with a single Key. Consider this the process equivalent of a DNS server: in the same way you can retrieve an IP address from a domain name, you can retrieve a process from its Key.
-
-Typical Use Case: registering on a system a process that handles a physical device (using its serial number).
-
-##### What is a Process Group?
-A global Process Group is a named group which contains many processes, possibly running on different nodes. With the group Name, you can retrieve on any cluster node the list of these processes, or publish a message to all of them. This mechanism allows for Publish / Subscribe patterns.
-
-Typical Use Case: a chatroom.
-
-##### What is Syn?
-Syn is a Process Registry and Process Group manager that has the following features:
-
- * Global Process Registry (i.e. a process is uniquely identified with a Key across all the nodes of a cluster).
- * Global Process Group manager (i.e. a group is uniquely identified with a Name across all the nodes of a cluster).
- * Any term can be used as Key and Name.
- * A message can be published to all members of a Process Group (PubSub mechanism).
- * Fast writes.
- * Automatically handles conflict resolution (such as net splits).
- * Configurable callbacks.
- * Processes are automatically monitored and removed from the Process Registry and Process Groups if they die.
-
-## Notes
-In any distributed system you are faced with a consistency challenge, which is often resolved by having one master arbiter performing all write operations (chosen with a mechanism of [leader election](http://en.wikipedia.org/wiki/Leader_election)), or through [atomic transactions](http://en.wikipedia.org/wiki/Atomicity_(database_systems)).
-
-Syn was born for applications of the [IoT](http://en.wikipedia.org/wiki/Internet_of_Things) field. In this context, Keys used to identify a process are often the physical object's unique identifier (for instance, its serial or MAC address), and are therefore already defined and unique _before_ hitting the system.  The consistency challenge is less of a problem in this case, since the likelihood of concurrent incoming requests that would register processes with the same Key is extremely low and, in most cases, acceptable.
-
-In addition, write speeds were a determining factor in the architecture of Syn.
-
-Therefore, Availability has been chosen over Consistency and Syn is [eventually consistent](http://en.wikipedia.org/wiki/Eventual_consistency).
-
-## Setup
-
-### For Elixir
-Add it to your deps:
-
-```elixir
-defp deps do
-  [{:syn, "~> 2.1"}]
-end
-```
-
-### For Erlang
-If you're using [rebar3](https://github.com/erlang/rebar3), add `syn` as a dependency in your project's `rebar.config` file:
-
-```erlang
-{deps, [
-  {syn, {git, "git://github.com/ostinelli/syn.git", {tag, "2.1.4"}}}
-]}.
-```
-Or, if you're using [Hex.pm](https://hex.pm/) as package manager (with the [rebar3_hex](https://github.com/hexpm/rebar3_hex) plugin):
-
-```erlang
-{deps, [
-  {syn, "2.1.4"}
-]}.
-```
-
-Ensure that `syn` is started with your application, for example by adding it in your `.app` file to the list of `applications`:
-
-```erlang
-{application, my_app, [
-    %% ...
-    {applications, [
-        kernel,
-        stdlib,
-        sasl,
-        syn,
-        %% ...
-    ]},
-    %% ...
-]}.
-```
-
-## API
-
-Example code here below is in Erlang. Thanks to Elixir interoperability with Erlang, the equivalent code in Elixir is straightforward.
-
-### Process Registry
-
-To register a process:
-
-```erlang
-syn:register(Name, Pid) ->
-    syn:register(Name, Pid, undefined).
-```
-
-```erlang
-syn:register(Name, Pid, Meta) -> ok | {error, Error}.
-
-Types:
-    Name = any()
-    Pid = pid()
-    Meta = any()
-    Error = taken
-```
-
-| ERROR | DESC
-|-----|-----
-| taken | The Name is already taken by another process.
-
-> You may re-register a process multiple times, for example if you need to update its metadata. When a process gets registered, Syn will automatically monitor it. You may also register the same process with different names.
-
-Processes can also be registered as `gen_server` names, by usage of via-tuples.
-This way, you can use the `gen_server` API with these tuples without referring to the Pid directly.
-
-```erlang
-Tuple = {via, syn, <<"your process name">>}.
-gen_server:start_link(Tuple, your_module, []).
-gen_server:call(Tuple, your_message).
-```
-
-To retrieve a Pid from a Name:
-
-```erlang
-syn:whereis(Name) -> Pid | undefined.
-
-Types:
-    Key = any()
-    Pid = pid()
-```
-
-To retrieve a Pid from a Name with its metadata:
-
-```erlang
-syn:whereis(Key, with_meta) -> {Pid, Meta} | undefined.
-
-Types:
-    Key = any()
-    Pid = pid()
-    Meta = any()
-```
-
-To unregister a previously registered Name:
-
-```erlang
-syn:unregister(Name) -> ok | {error, Error}.
-
-Types:
-    Key = any()
-    Error = undefined
-```
-
-> You don't need to unregister names of processes that are about to die, since they are monitored by Syn and they will be removed automatically. If you manually unregister a process just before it dies, the callback on process exit (see here below) might not get called.
-
-To register a previously registered Name with a _different_ Pid:
-
-```erlang
-syn:unregister_and_register(Name, Pid) ->
-    syn:unregister_and_register(Name, Pid, undefined).
-```
-```erlang
-syn:unregister_and_register(Name, Pid, Meta) -> ok.
-
-Types:
-    Name = any()
-    Pid = pid()
-    Meta = any()
-```
-
-> Due to Syn being eventually consistent, if you were to sequentially `unregister/1` a name and `register/2,3` a process you might experience a `{error, taken}` response to the latter, since the unregistration may not have yet properly propagated when the registration call is made. This call ensures that the registration succeeds and propagates properly.
->
->Note that the previously registered process will not be killed and will be demonitored, so that the `on_process_exit/4` callback will _not_ be called (even if implemented) when the process dies.
-
-To retrieve the total count of registered names:
-
-```erlang
-syn:registry_count() -> non_neg_integer().
-```
-
-To retrieve the total count of registered names of processes running on a specific node:
-
-```erlang
-syn:registry_count(Node) -> non_neg_integer().
-
-Types:
-    Node = atom()
-```
-
-> This is a non-optimized call, use for debugging / monitoring purposes only.
-
-### Process Groups
-
-> There's no need to manually create / delete Process Groups, Syn will take care of managing those for you.
-
-To add a process to a group:
-
-```erlang
-syn:join(GroupName, Pid) ->
-    syn:join(GroupName, Pid, undefined).
-```
-
-```erlang
-syn:join(GroupName, Pid, Meta) -> ok.
-
-Types:
-    GroupName = any()
-    Pid = pid()
-    meta = any()
-```
-
-> A process can join multiple groups. When a process joins a group, Syn will automatically monitor it. A process may join the same group multiple times, for example if you need to update its metadata, though it will still be listed only once in it.
-
-To remove a process from a group:
-
-```erlang
-syn:leave(GroupName, Pid) -> ok | {error, Error}.
-
-Types:
-    GroupName = any()
-    Pid = pid()
-    Error = not_in_group
-```
-
-> You don't need to remove processes that are about to die, since they are monitored by Syn and they will be removed automatically from their groups.
-
-To get a list of the members of a group:
-
-```erlang
-syn:get_members(GroupName) -> [pid()].
-
-Types:
-    GroupName = any()
-```
-
-To get a list of the members of a group with their metadata:
-
-```erlang
-syn:get_members(GroupName, with_meta) ->
-    [{pid(), Meta}].
-
-Types:
-    GroupName = any()
-    Meta = any()
-```
-
-> The order of member pids in the returned array is guaranteed to be the same on every node, however it is not guaranteed to match the order of joins.
-
-To know if a process is a member of a group:
-
-```erlang
-syn:member(Pid, GroupName) -> boolean().
-
-Types:
-    Pid = pid()
-    GroupName = any()
-```
-
-To get a list of the existing groups:
-
-```erlang
-syn:get_group_names() -> [GroupName].
-
-Types:
-    GroupName = any()
-```
-
-To publish a message to all group members:
-
-```erlang
-syn:publish(GroupName, Message) -> {ok, RecipientCount}.
-
-Types:
-    GroupName = any()
-    Message = any()
-    RecipientCount = non_neg_integer()
-```
-
-> `RecipientCount` is the count of the _intended_ recipients.
-
-To call all group members and get their replies:
-
-```erlang
-syn:multi_call(GroupName, Message) ->
-    syn:multi_call(GroupName, Message, 5000).
-```
-
-```erlang
-syn:multi_call(GroupName, Message, Timeout) -> {Replies, BadPids}.
-
-Types:
-    GroupName = any()
-    Message = any()
-    Timeout = non_neg_integer()
-    Replies = [{MemberPid, Reply}]
-    BadPids = [MemberPid]
-      MemberPid = pid()
-      Reply = any()
-```
-
-> Syn will wait up to the value specified in `Timeout` to receive all replies from the members. The members that do not reply in time or that crash before sending a reply will be added to the `BadPids` list.
-
-When this call is issued, all members will receive a tuple in the format:
-
-```erlang
-{syn_multi_call, CallerPid, Message}
-
-Types:
-    CallerPid = pid()
-    Message = any()
-```
-
-To reply, every member must use the method:
-
-```erlang
-syn:multi_call_reply(CallerPid, Reply) -> ok.
-
-Types:
-    CallerPid = pid()
-    Reply = any()
-```
-
-To get a list of the local members of a group (= running on the node):
-
-```erlang
-syn:get_local_members(GroupName) -> [pid()].
-
-Types:
-    GroupName = any()
-```
-
-To get a list of the local members of a group with their metadata:
-
-```erlang
-syn:get_local_members(GroupName, with_meta) ->
-    [{pid(), Meta}].
-
-Types:
-    GroupName = any()
-    Meta = any()
-```
-
-> The order of member pids in the returned array is guaranteed to be the same on every node, however it is not guaranteed to match the order of joins.
-
-To know if a process is a local member of a group:
-
-```erlang
-syn:local_member(Pid, GroupName) -> boolean().
-
-Types:
-    Pid = pid()
-    GroupName = any()
-```
-
-To publish a message to all local group members:
-
-```erlang
-syn:publish_to_local(GroupName, Message) -> {ok, RecipientCount}.
-
-Types:
-    GroupName = any()
-    Message = any()
-    RecipientCount = non_neg_integer()
-```
-
-> `RecipientCount` is the count of the intended recipients.
-
-To retrieve the total count of groups in the cluster:
-
-```erlang
-syn:groups_count() -> non_neg_integer().
-```
-
-> This is a non-optimized call, use for debugging / monitoring purposes only.
-
-To retrieve the count of groups that have at least 1 process running on a specific node:
-
-```erlang
-syn:groups_count(Node) -> non_neg_integer().
-
-Types:
-    Node = atom()
-```
-
-> This is a non-optimized call, use for debugging / monitoring purposes only.
-
-## Callbacks
-In Syn you can specify a custom callback module if you want to:
-
-  * Receive and handle the event of a registered / joined process' exit.
-  * Customize the method to resolve registry naming conflict in case of net splits or race conditions.
-
-### Setup
-The callback module can be set in the environment variable `syn`, with the `event_handler` key. You're probably best off using an application configuration file.
-
-#### Elixir
-In `config.exs` you can specify your callback module:
-
-```elixir
-config :syn,
-  event_handler: MyCustomEventHandler
-```
-
-In your module you then need to specify the behavior and the callbacks. All callbacks are _optional_, so you just need to define the ones you need.
-
-```elixir
-defmodule MyCustomEventHandler do
-  @behaviour :syn_event_handler
-
-  @impl true
-  @spec on_process_exit(
-    name :: any(),
-    pid :: pid(),
-    meta :: any(),
-    reason :: any()
-  ) :: any()
-  def on_process_exit(name, pid, meta, reason) do
-  end
-
-  @impl true
-  @spec on_group_process_exit(
-    group_name :: any(),
-    pid :: pid(),
-    meta :: any(),
-    reason :: any()
-  ) :: any()
-  def on_group_process_exit(group_name, pid, meta, reason) do
-  end
-
-  @impl true
-  @spec resolve_registry_conflict(
-    name :: any(),
-    {pid1 :: pid(), meta1 :: any()},
-    {Pid2 :: pid(), meta2 :: any()}
-  ) -> pid_to_keep :: pid()
-  def resolve_registry_conflict(name, {pid1, meta1}, {pid2, meta2})
-    pid1
-  end
-end
-```
-See details about the callback methods here below.
-
-#### Erlang
-In `sys.config` you can specify your callback module:
-
-```erlang
-{syn, [
-    {event_handler, my_custom_event_handler}
-]}
-```
-
-In your module you then need to specify the behavior and the callbacks. All callbacks are _optional_, so you just need to define the ones you need.
-
-```erlang
--module(my_custom_event_handler).
--behaviour(syn_event_handler).
-
--export([on_process_exit/4]).
--export([on_group_process_exit/4]).
--export([resolve_registry_conflict/3]).
-
--spec on_process_exit(
-    Name :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Reason :: any()
-) -> any().
-on_process_exit(Name, Pid, Meta, Reason) ->
-    ok.
-
--spec on_group_process_exit(
-    GroupName :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Reason :: any()
-) -> any().
-on_group_process_exit(GroupName, Pid, Meta, Reason) ->
-    ok.
-
--spec resolve_registry_conflict(
-    Name :: any(),
-    {Pid1 :: pid(), Meta1 :: any()},
-    {Pid2 :: pid(), Meta2 :: any()}
-) -> PidToKeep :: pid().
-resolve_registry_conflict(Name, {Pid1, Meta1}, {Pid2, Meta2}) ->
-    Pid1.
-```
-
-See details about the callback methods here below.
-
-### Callback methods
-
-#### `on_process_exit/4`
-Called when a registered process exits. It will be called only on the node where the process was running. If a process was registered under _n_ names, this callback will be called _n_ times (1 per registered name).
-
-#### `on_group_process_exit/4`
-Called when a process in a group exits. It will be called only on the node where the process was running. If a process was part of _n_ groups, this callback will be called _n_ times (1 per joined group).
-
-#### `resolve_registry_conflict/3`
-In case of net splits or race conditions, a specific Name might get registered simultaneously on two different nodes. In this case, the cluster experiences a registry naming conflict.
-
-When this happens, Syn will resolve this Process Registry conflict by choosing a single process. By default, Syn keeps track of the time a registration takes place with [`erlang:system_time/0`](http://erlang.org/doc/man/erlang.html#system_time-0), compares values between conflicting processes and:
-
-  * Keeps the one with the higher value (the process that was registered more recently).
-  * Kills the other process by sending a `kill` signal with `exit(Pid, {syn_resolve_kill, Name, Meta})`.
-
-This is a very simple mechanism that can be imprecise, as system clocks are not perfectly aligned in a cluster. If something more elaborate is desired (such as vector clocks), or if you do not want the discarded process to be killed (i.e. to perform a graceful shutdown), you MAY specify a custom handler that implements this `resolve_registry_conflict/3` callback. To this effect, you may also store additional data to resolve conflicts in the `Meta` field, since it will be passed into the callback for both of the conflicting processes.
-
-If implemented, this method MUST return the `pid()` of the process that you wish to keep. The other process will be _not_ be killed, so you will have to decide what to do with it.
-
-> Important Note: the conflict resolution method SHOULD be defined in the same way across all nodes of the cluster. Having different conflict resolution options on different nodes can have unexpected results.
-
-## Anti-Entropy
-Anti-entropy is a mechanism to force alignment between nodes. It isn't needed with Syn in most cases. However, despite Syn's best efforts and under rare conditions, depending on your cluster topology and other factors, it _might_ be possible that registry and groups get misaligned.
-
-> Anti-entropy in Syn is an **experimental** feature. As with every anti-entropy feature, it comes with a cost: during the forced syncing, the local tables will be rebuilt with data that gets sent from other nodes. This takes time, due to the sending of data across nodes and subsequent storage. As an example, a node that handles 1,000,000 local registry / groups processes will have to send data for ~80MB (depending on your metadata's size) to other nodes. During the syncing, Syn might time out some calls. Your mileage may vary, so it is recommended that you benchmark your use case.
-
-### Setup
-To activate anti-entropy you need to set in the environment variable `syn` the key `anti_entropy`. You're probably best off using an application configuration file. If you do not specify the `anti_entropy` key, the anti-entropy mechanism will be disabled by default.
-
-#### Elixir
-In `config.exs` you can specify your anti-entropy settings:
-
-```elixir
-config :syn,
-  anti_entropy: [
-    registry: [interval: 300, interval_max_deviation: 60],
-    groups: [interval: 300, interval_max_deviation: 60]
-  ]
-```
-
-#### Erlang
-In `sys.config` you can specify your anti-entropy settings:
-
-```erlang
-{syn, [
-    {anti_entropy, [
-        {registry, [{interval, 300}, {interval_max_deviation, 60}],
-        {groups, [{interval, 300}, {interval_max_deviation, 60}]
-    ]}
-]}
-```
-
-`interval` specifies in seconds the interval between every anti-entropy syncing, while `interval_max_deviation` the max deviation in seconds from the `interval`. For instance, with an `interval` of 300 seconds and an `interval_max_deviation` of 60, anti-entropy will be called with an interval range of 240 to 360 seconds.
-
-### Manual sync
-You can force an anti-entropy sync on the whole cluster by calling the following:
-
-```erlang
-syn:force_cluster_sync(registry | groups) -> ok.
-```
-
-> As per the notes above, in normal conditions Syn doesn't need to be manually synced. This function will force a full mesh sync on all of the cluster and is **experimental**. Use it as a last resort.
-
-## Internals
-As of v2.1, Syn uses ETS for memory storage and doesn't have any external dependency. Syn has its own replication and naming conflict resolution mechanisms.
-
-## Contributing
-So you want to contribute? That's great! Please follow the guidelines below. It will make it easier to get merged in.
-
-Before implementing a new feature, please submit a ticket to discuss what you intend to do. Your feature might already be in the works, or an alternative implementation might have already been discussed.
-
-Do not commit to master in your fork. Provide a clean branch without merge commits. Every pull request should have its own topic branch. In this way, every additional adjustments to the original pull request might be done easily, and squashed with `git rebase -i`. The updated branch will be visible in the same pull request, so there will be no need to open new pull requests when there are changes to be applied.
-
-Ensure that proper testing is included. To run Syn tests you simply have to be in the project's root directory and run:
-
-```bash
-$ make test
-```
+V3 is a WIP.

src/syn.app.src

@@ -1,7 +1,7 @@
 {application, syn,
     [
         {description, "A global Process Registry and Process Group manager."},
-        {vsn, "2.1.4"},
+        {vsn, "3.0.0"},
         {registered, [
             syn_backbone,
             syn_groups,
@@ -17,8 +17,7 @@
 
         {licenses, ["MIT"]},
         {links, [
-            {"Github", "https://github.com/ostinelli/syn"},
-            {"ostinelli|net", "http://www.ostinelli.net/a-journey-to-syn-v2/"}
+            {"Github", "https://github.com/ostinelli/syn"}
         ]}
     ]
 }.

src/syn.erl

@@ -3,7 +3,7 @@
 %%
 %% The MIT License (MIT)
 %%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
+%% Copyright (c) 2015-2021 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
 %%
 %% Permission is hereby granted, free of charge, to any person obtaining a copy
 %% of this software and associated documentation files (the "Software"), to deal
@@ -27,26 +27,6 @@
 
 %% API
 -export([start/0, stop/0]).
--export([register/2, register/3]).
--export([unregister_and_register/2, unregister_and_register/3]).
--export([unregister/1]).
--export([whereis/1, whereis/2]).
--export([registry_count/0, registry_count/1]).
--export([join/2, join/3]).
--export([leave/2]).
--export([get_members/1, get_members/2]).
--export([get_group_names/0]).
--export([member/2]).
--export([get_local_members/1, get_local_members/2]).
--export([local_member/2]).
--export([groups_count/0, groups_count/1]).
--export([publish/2]).
--export([publish_to_local/2]).
--export([multi_call/2, multi_call/3, multi_call_reply/2]).
--export([force_cluster_sync/1]).
-
-%% gen_server via interface
--export([register_name/2, unregister_name/1, whereis_name/1, send/2]).
 
 %% ===================================================================
 %% API
@@ -59,147 +39,3 @@ start() ->
 -spec stop() -> ok | {error, Reason :: any()}.
 stop() ->
     application:stop(syn).
-
-%% ----- \/ registry -------------------------------------------------
--spec register(Name :: any(), Pid :: pid()) -> ok | {error, Reason :: any()}.
-register(Name, Pid) ->
-    syn_registry:register(Name, Pid).
-
--spec register(Name :: any(), Pid :: pid(), Meta :: any()) -> ok | {error, Reason :: any()}.
-register(Name, Pid, Meta) ->
-    syn_registry:register(Name, Pid, Meta).
-
--spec unregister_and_register(Name :: any(), Pid :: pid()) -> ok | {error, Reason :: any()}.
-unregister_and_register(Name, Pid) ->
-    syn_registry:unregister_and_register(Name, Pid).
-
--spec unregister_and_register(Name :: any(), Pid :: pid(), Meta :: any()) -> ok | {error, Reason :: any()}.
-unregister_and_register(Name, Pid, Meta) ->
-    syn_registry:unregister_and_register(Name, Pid, Meta).
-
--spec unregister(Name :: any()) -> ok | {error, Reason :: any()}.
-unregister(Name) ->
-    syn_registry:unregister(Name).
-
--spec whereis(Name :: any()) -> pid() | undefined.
-whereis(Name) ->
-    syn_registry:whereis(Name).
-
--spec whereis(Name :: any(), with_meta) -> {pid(), Meta :: any()} | undefined.
-whereis(Name, with_meta) ->
-    syn_registry:whereis(Name, with_meta).
-
--spec registry_count() -> non_neg_integer().
-registry_count() ->
-    syn_registry:count().
-
--spec registry_count(Node :: atom()) -> non_neg_integer().
-registry_count(Node) ->
-    syn_registry:count(Node).
-
-%% ----- \/ gen_server via module interface --------------------------
--spec register_name(Name :: any(), Pid :: pid()) -> yes | no.
-register_name(Name, Pid) ->
-    case syn_registry:register(Name, Pid) of
-        ok -> yes;
-        _ -> no
-    end.
-
--spec unregister_name(Name :: any()) -> any().
-unregister_name(Name) ->
-    case syn_registry:unregister(Name) of
-        ok -> Name;
-        _ -> nil
-    end.
-
--spec whereis_name(Name :: any()) -> pid() | undefined.
-whereis_name(Name) ->
-    syn_registry:whereis(Name).
-
--spec send(Name :: any(), Message :: any()) -> pid().
-send(Name, Message) ->
-    case whereis_name(Name) of
-        undefined ->
-            {badarg, {Name, Message}};
-        Pid ->
-            Pid ! Message,
-            Pid
-    end.
-
-%% ----- \/ groups ---------------------------------------------------
--spec join(GroupName :: any(), Pid :: pid()) -> ok.
-join(GroupName, Pid) ->
-    syn_groups:join(GroupName, Pid).
-
--spec join(GroupName :: any(), Pid :: pid(), Meta :: any()) -> ok.
-join(GroupName, Pid, Meta) ->
-    syn_groups:join(GroupName, Pid, Meta).
-
--spec leave(GroupName :: any(), Pid :: pid()) -> ok | {error, Reason :: any()}.
-leave(GroupName, Pid) ->
-    syn_groups:leave(GroupName, Pid).
-
--spec get_members(GroupName :: any()) -> [pid()].
-get_members(GroupName) ->
-    syn_groups:get_members(GroupName).
-
--spec get_members(GroupName :: any(), with_meta) -> [{pid(), Meta :: any()}].
-get_members(GroupName, with_meta) ->
-    syn_groups:get_members(GroupName, with_meta).
-
--spec get_group_names() -> [GroupName :: any()].
-get_group_names() ->
-    syn_groups:get_group_names().
-
--spec member(Pid :: pid(), GroupName :: any()) -> boolean().
-member(Pid, GroupName) ->
-    syn_groups:member(Pid, GroupName).
-
--spec get_local_members(GroupName :: any()) -> [pid()].
-get_local_members(GroupName) ->
-    syn_groups:get_local_members(GroupName).
-
--spec get_local_members(GroupName :: any(), with_meta) -> [{pid(), Meta :: any()}].
-get_local_members(GroupName, with_meta) ->
-    syn_groups:get_local_members(GroupName, with_meta).
-
--spec local_member(Pid :: pid(), GroupName :: any()) -> boolean().
-local_member(Pid, GroupName) ->
-    syn_groups:local_member(Pid, GroupName).
-
--spec groups_count() -> non_neg_integer().
-groups_count() ->
-    syn_groups:count().
-
--spec groups_count(Node :: atom()) -> non_neg_integer().
-groups_count(Node) ->
-    syn_groups:count(Node).
-
--spec publish(GroupName :: any(), Message :: any()) -> {ok, RecipientCount :: non_neg_integer()}.
-publish(GroupName, Message) ->
-    syn_groups:publish(GroupName, Message).
-
--spec publish_to_local(GroupName :: any(), Message :: any()) -> {ok, RecipientCount :: non_neg_integer()}.
-publish_to_local(GroupName, Message) ->
-    syn_groups:publish_to_local(GroupName, Message).
-
--spec multi_call(GroupName :: any(), Message :: any()) ->
-    {[{pid(), Reply :: any()}], [BadPid :: pid()]}.
-multi_call(GroupName, Message) ->
-    syn_groups:multi_call(GroupName, Message).
-
--spec multi_call(GroupName :: any(), Message :: any(), Timeout :: non_neg_integer()) ->
-    {[{pid(), Reply :: any()}], [BadPid :: pid()]}.
-multi_call(GroupName, Message, Timeout) ->
-    syn_groups:multi_call(GroupName, Message, Timeout).
-
--spec multi_call_reply(CallerPid :: pid(), Reply :: any()) -> {syn_multi_call_reply, pid(), Reply :: any()}.
-multi_call_reply(CallerPid, Reply) ->
-    syn_groups:multi_call_reply(CallerPid, Reply).
-
-%% ----- \/ anti entropy ---------------------------------------------
--spec force_cluster_sync(registry | groups) -> ok.
-force_cluster_sync(registry) ->
-    syn_registry:force_cluster_sync();
-force_cluster_sync(groups) ->
-    syn_groups:force_cluster_sync().

src/syn.hrl

@@ -3,7 +3,7 @@
 %%
 %% The MIT License (MIT)
 %%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
+%% Copyright (c) 2015-2021 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
 %%
 %% Permission is hereby granted, free of charge, to any person obtaining a copy
 %% of this software and associated documentation files (the "Software"), to deal

src/syn_app.erl

@@ -3,7 +3,7 @@
 %%
 %% The MIT License (MIT)
 %%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
+%% Copyright (c) 2015-2021 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
 %%
 %% Permission is hereby granted, free of charge, to any person obtaining a copy
 %% of this software and associated documentation files (the "Software"), to deal

src/syn_backbone.erl

@@ -3,7 +3,7 @@
 %%
 %% The MIT License (MIT)
 %%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
+%% Copyright (c) 2015-2021 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
 %%
 %% Permission is hereby granted, free of charge, to any person obtaining a copy
 %% of this software and associated documentation files (the "Software"), to deal
@@ -28,16 +28,10 @@
 
 %% API
 -export([start_link/0]).
--export([get_event_handler_module/0]).
--export([get_anti_entropy_settings/1]).
 
 %% gen_server callbacks
 -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).
 
-%% macros
--define(DEFAULT_EVENT_HANDLER_MODULE, syn_event_handler).
--define(DEFAULT_ANTI_ENTROPY_MAX_DEVIATION_MS, 60000).
-
 %% records
 -record(state, {}).
 
@@ -52,45 +46,6 @@ start_link() ->
     Options = [],
     gen_server:start_link({local, ?MODULE}, ?MODULE, [], Options).
 
--spec get_event_handler_module() -> module().
-get_event_handler_module() ->
-    %% get handler
-    CustomEventHandler = application:get_env(syn, event_handler, ?DEFAULT_EVENT_HANDLER_MODULE),
-    %% ensure that is it loaded (not using code:ensure_loaded/1 to support embedded mode)
-    catch CustomEventHandler:module_info(exports),
-    %% return
-    CustomEventHandler.
-
--spec get_anti_entropy_settings(Module :: registry | groups) ->
-    {IntervalMs :: non_neg_integer() | undefined, IntervalMaxDeviationMs :: non_neg_integer() | undefined}.
-get_anti_entropy_settings(Module) ->
-    case application:get_env(syn, anti_entropy, undefined) of
-        undefined ->
-            {undefined, undefined};
-
-        AntiEntropySettings ->
-            case proplists:get_value(Module, AntiEntropySettings) of
-                undefined ->
-                    {undefined, undefined};
-
-                ModSettings ->
-                    case proplists:get_value(interval, ModSettings) of
-                        undefined ->
-                            {undefined, undefined};
-
-                        I ->
-                            IntervalMs = I * 1000,
-                            IntervalMaxDeviationMs = proplists:get_value(
-                                interval_max_deviation,
-                                ModSettings,
-                                ?DEFAULT_ANTI_ENTROPY_MAX_DEVIATION_MS
-                            ) * 1000,
-                            %% return
-                            {IntervalMs, IntervalMaxDeviationMs}
-                    end
-            end
-    end.
-
 %% ===================================================================
 %% Callbacks
 %% ===================================================================

src/syn_event_handler.erl

@@ -1,128 +0,0 @@
-%% ==========================================================================================================
-%% Syn - A global Process Registry and Process Group manager.
-%%
-%% The MIT License (MIT)
-%%
-%% Copyright (c) 2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
-%%
-%% Portions of code from Ulf Wiger's unsplit server module:
-%% <https://github.com/uwiger/unsplit/blob/master/src/unsplit_server.erl>
-%%
-%% Permission is hereby granted, free of charge, to any person obtaining a copy
-%% of this software and associated documentation files (the "Software"), to deal
-%% in the Software without restriction, including without limitation the rights
-%% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-%% copies of the Software, and to permit persons to whom the Software is
-%% furnished to do so, subject to the following conditions:
-%%
-%% The above copyright notice and this permission notice shall be included in
-%% all copies or substantial portions of the Software.
-%%
-%% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-%% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-%% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-%% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-%% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-%% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-%% THE SOFTWARE.
-%% ==========================================================================================================
--module(syn_event_handler).
-
--export([do_on_process_exit/5]).
--export([do_on_group_process_exit/5]).
--export([do_resolve_registry_conflict/4]).
-
--callback on_process_exit(
-    Name :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Reason :: any()
-) -> any().
-
--callback on_group_process_exit(
-    GroupName :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Reason :: any()
-) -> any().
-
--callback resolve_registry_conflict(
-    Name :: any(),
-    {Pid1 :: pid(), Meta1 :: any()},
-    {Pid2 :: pid(), Meta2 :: any()}
-) -> PidToKeep :: pid() | undefined.
-
--optional_callbacks([on_process_exit/4, on_group_process_exit/4, resolve_registry_conflict/3]).
-
-%% ===================================================================
-%% API
-%% ===================================================================
--spec do_on_process_exit(
-    Name :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Reason :: any(),
-    CustomEventHandler :: module()
-) -> any().
-do_on_process_exit(Name, Pid, Meta, Reason, CustomEventHandler) ->
-    case erlang:function_exported(CustomEventHandler, on_process_exit, 4) of
-        true ->
-            spawn(fun() ->
-                CustomEventHandler:on_process_exit(Name, Pid, Meta, Reason)
-            end);
-        _ ->
-            ok
-    end.
-
--spec do_on_group_process_exit(
-    GroupName :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Reason :: any(),
-    CustomEventHandler :: module()
-) -> any().
-do_on_group_process_exit(GroupName, Pid, Meta, Reason, CustomEventHandler) ->
-
-    case erlang:function_exported(CustomEventHandler, on_group_process_exit, 4) of
-        true ->
-            spawn(fun() ->
-                CustomEventHandler:on_group_process_exit(GroupName, Pid, Meta, Reason)
-            end);
-        _ ->
-            ok
-    end.
-
--spec do_resolve_registry_conflict(
-    Name :: any(),
-    {Pid1 :: pid(), Meta1 :: any(), Time1 :: integer()},
-    {Pid2 :: pid(), Meta2 :: any(), Time2 :: integer()},
-    CustomEventHandler :: module()
-) -> {PidToKeep :: pid() | undefined, KillOtherPid :: boolean() | undefined}.
-do_resolve_registry_conflict(Name, {Pid1, Meta1, Time1}, {Pid2, Meta2, Time2}, CustomEventHandler) ->
-    case erlang:function_exported(CustomEventHandler, resolve_registry_conflict, 3) of
-        true ->
-            try CustomEventHandler:resolve_registry_conflict(Name, {Pid1, Meta1}, {Pid2, Meta2}) of
-                PidToKeep when is_pid(PidToKeep) ->
-                    {PidToKeep, false};
-
-                _ ->
-                    {undefined, false}
-
-            catch Exception:Reason ->
-                error_logger:error_msg(
-                    "Syn(~p): Error ~p in custom handler resolve_registry_conflict: ~p~n",
-                    [node(), Exception, Reason]
-                ),
-                {undefined, false}
-            end;
-
-        _ ->
-            %% by default, keep pid registered more recently
-            %% this is a simple mechanism that can be imprecise, as system clocks are not perfectly aligned in a cluster
-            %% if something more elaborate is desired (such as vector clocks) use Meta to store data and a custom event handler
-            PidToKeep = case Time1 > Time2 of
-                true -> Pid1;
-                _ -> Pid2
-            end,
-            {PidToKeep, true}
-    end.

src/syn_groups.erl

@@ -3,7 +3,7 @@
 %%
 %% The MIT License (MIT)
 %%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
+%% Copyright (c) 2015-2021 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
 %%
 %% Permission is hereby granted, free of charge, to any person obtaining a copy
 %% of this software and associated documentation files (the "Software"), to deal
@@ -28,48 +28,12 @@
 
 %% API
 -export([start_link/0]).
--export([join/2, join/3]).
--export([leave/2]).
--export([get_members/1, get_members/2]).
--export([get_group_names/0]).
--export([member/2]).
--export([get_local_members/1, get_local_members/2]).
--export([local_member/2]).
--export([count/0, count/1]).
--export([publish/2]).
--export([publish_to_local/2]).
--export([multi_call/2, multi_call/3, multi_call_reply/2]).
-
-%% sync API
--export([sync_join/4, sync_leave/3]).
--export([sync_get_local_groups_tuples/1]).
--export([force_cluster_sync/0]).
--export([remove_from_local_table/2]).
-
-%% internal
--export([multicast_loop/0]).
-
-%% tests
--ifdef(TEST).
--export([add_to_local_table/4]).
--endif.
 
 %% gen_server callbacks
 -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).
 
-%% internal
--export([multi_call_and_receive/4]).
-
 %% records
--record(state, {
-    custom_event_handler :: undefined | module(),
-    anti_entropy_interval_ms :: undefined | non_neg_integer(),
-    anti_entropy_interval_max_deviation_ms :: undefined | non_neg_integer(),
-    multicast_pid :: undefined | pid()
-}).
-
-%% macros
--define(DEFAULT_MULTI_CALL_TIMEOUT_MS, 5000).
+-record(state, {}).
 
 %% includes
 -include("syn.hrl").
@@ -82,162 +46,6 @@ start_link() ->
     Options = [],
     gen_server:start_link({local, ?MODULE}, ?MODULE, [], Options).
 
--spec join(GroupName :: any(), Pid :: pid()) -> ok.
-join(GroupName, Pid) ->
-    join(GroupName, Pid, undefined).
-
--spec join(GroupName :: any(), Pid :: pid(), Meta :: any()) -> ok.
-join(GroupName, Pid, Meta) when is_pid(Pid) ->
-    Node = node(Pid),
-    gen_server:call({?MODULE, Node}, {join_on_node, GroupName, Pid, Meta}).
-
--spec leave(GroupName :: any(), Pid :: pid()) -> ok | {error, Reason :: any()}.
-leave(GroupName, Pid) ->
-    case find_groups_entry_by_name_and_pid(GroupName, Pid) of
-        undefined ->
-            {error, not_in_group};
-        _ ->
-            Node = node(Pid),
-            gen_server:call({?MODULE, Node}, {leave_on_node, GroupName, Pid})
-    end.
-
--spec get_members(Name :: any()) -> [pid()].
-get_members(GroupName) ->
-    ets:select(syn_groups_by_name, [{
-        {{GroupName, '$2'}, '_', '_', '_'},
-        [],
-        ['$2']
-    }]).
-
--spec get_members(GroupName :: any(), with_meta) -> [{pid(), Meta :: any()}].
-get_members(GroupName, with_meta) ->
-    ets:select(syn_groups_by_name, [{
-        {{GroupName, '$2'}, '$3', '_', '_'},
-        [],
-        [{{'$2', '$3'}}]
-    }]).
-
--spec get_group_names() -> [GroupName :: any()].
-get_group_names() ->
-    Groups = ets:select(syn_groups_by_name, [{
-      {{'$1', '_'}, '_', '_', '_'},
-      [],
-      ['$1']
-    }]),
-    Set = sets:from_list(Groups),
-    sets:to_list(Set).
-
--spec member(Pid :: pid(), GroupName :: any()) -> boolean().
-member(Pid, GroupName) ->
-    case find_groups_entry_by_name_and_pid(GroupName, Pid) of
-        undefined -> false;
-        _ -> true
-    end.
-
--spec get_local_members(Name :: any()) -> [pid()].
-get_local_members(GroupName) ->
-    Node = node(),
-    ets:select(syn_groups_by_name, [{
-        {{GroupName, '$2'}, '_', '_', Node},
-        [],
-        ['$2']
-    }]).
-
--spec get_local_members(GroupName :: any(), with_meta) -> [{pid(), Meta :: any()}].
-get_local_members(GroupName, with_meta) ->
-    Node = node(),
-    ets:select(syn_groups_by_name, [{
-        {{GroupName, '$2'}, '$3', '_', Node},
-        [],
-        [{{'$2', '$3'}}]
-    }]).
-
--spec local_member(Pid :: pid(), GroupName :: any()) -> boolean().
-local_member(Pid, GroupName) ->
-    case find_groups_entry_by_name_and_pid(GroupName, Pid) of
-        {GroupName, Pid, _Meta, _MonitorRef, Node} when Node =:= node() ->
-            true;
-
-        _ ->
-            false
-    end.
-
--spec count() -> non_neg_integer().
-count() ->
-    Entries = ets:select(syn_groups_by_name, [{
-        {{'$1', '_'}, '_', '_',  '_'},
-        [],
-        ['$1']
-    }]),
-    Set = sets:from_list(Entries),
-    sets:size(Set).
-
--spec count(Node :: node()) -> non_neg_integer().
-count(Node) ->
-    Entries = ets:select(syn_groups_by_name, [{
-        {{'$1', '_'}, '_', '_',  Node},
-        [],
-        ['$1']
-    }]),
-    Set = sets:from_list(Entries),
-    sets:size(Set).
-
--spec publish(GroupName :: any(), Message :: any()) -> {ok, RecipientCount :: non_neg_integer()}.
-publish(GroupName, Message) ->
-    MemberPids = get_members(GroupName),
-    FSend = fun(Pid) ->
-        Pid ! Message
-    end,
-    lists:foreach(FSend, MemberPids),
-    {ok, length(MemberPids)}.
-
--spec publish_to_local(GroupName :: any(), Message :: any()) -> {ok, RecipientCount :: non_neg_integer()}.
-publish_to_local(GroupName, Message) ->
-    MemberPids = get_local_members(GroupName),
-    FSend = fun(Pid) ->
-        Pid ! Message
-    end,
-    lists:foreach(FSend, MemberPids),
-    {ok, length(MemberPids)}.
-
--spec multi_call(GroupName :: any(), Message :: any()) -> {[{pid(), Reply :: any()}], [BadPid :: pid()]}.
-multi_call(GroupName, Message) ->
-    multi_call(GroupName, Message, ?DEFAULT_MULTI_CALL_TIMEOUT_MS).
-
--spec multi_call(GroupName :: any(), Message :: any(), Timeout :: non_neg_integer()) ->
-    {[{pid(), Reply :: any()}], [BadPid :: pid()]}.
-multi_call(GroupName, Message, Timeout) ->
-    Self = self(),
-    MemberPids = get_members(GroupName),
-    FSend = fun(Pid) ->
-        spawn_link(?MODULE, multi_call_and_receive, [Self, Pid, Message, Timeout])
-    end,
-    lists:foreach(FSend, MemberPids),
-    collect_replies(MemberPids).
-
--spec multi_call_reply(CallerPid :: pid(), Reply :: any()) -> {syn_multi_call_reply, pid(), Reply :: any()}.
-multi_call_reply(CallerPid, Reply) ->
-    CallerPid ! {syn_multi_call_reply, self(), Reply}.
-
--spec sync_join(RemoteNode :: node(), GroupName :: any(), Pid :: pid(), Meta :: any()) -> ok.
-sync_join(RemoteNode, GroupName, Pid, Meta) ->
-    gen_server:cast({?MODULE, RemoteNode}, {sync_join, GroupName, Pid, Meta}).
-
--spec sync_leave(RemoteNode :: node(), GroupName :: any(), Pid :: pid()) -> ok.
-sync_leave(RemoteNode, GroupName, Pid) ->
-    gen_server:cast({?MODULE, RemoteNode}, {sync_leave, GroupName, Pid}).
-
--spec sync_get_local_groups_tuples(FromNode :: node()) -> list(syn_groups_tuple()).
-sync_get_local_groups_tuples(FromNode) ->
-    error_logger:info_msg("Syn(~p): Received request of local group tuples from remote node: ~p~n", [node(), FromNode]),
-    get_groups_tuples_for_node(node()).
-
--spec force_cluster_sync() -> ok.
-force_cluster_sync() ->
-    lists:foreach(fun(RemoteNode) ->
-        gen_server:cast({?MODULE, RemoteNode}, force_cluster_sync)
-    end, [node() | nodes()]).
-
 %% ===================================================================
 %% Callbacks
 %% ===================================================================
@@ -251,27 +59,8 @@ force_cluster_sync() ->
     ignore |
     {stop, Reason :: any()}.
 init([]) ->
-    %% monitor nodes
-    ok = net_kernel:monitor_nodes(true),
-    %% rebuild
-    rebuild_monitors(),
-    %% start multicast process
-    MulticastPid = spawn_link(?MODULE, multicast_loop, []),
-    %% get handler
-    CustomEventHandler = syn_backbone:get_event_handler_module(),
-    %% get anti-entropy interval
-    {AntiEntropyIntervalMs, AntiEntropyIntervalMaxDeviationMs} = syn_backbone:get_anti_entropy_settings(groups),
     %% build state
-    State = #state{
-        custom_event_handler = CustomEventHandler,
-        anti_entropy_interval_ms = AntiEntropyIntervalMs,
-        anti_entropy_interval_max_deviation_ms = AntiEntropyIntervalMaxDeviationMs,
-        multicast_pid = MulticastPid
-    },
-    %% send message to initiate full cluster sync
-    timer:send_after(0, self(), sync_from_full_cluster),
-    %% start anti-entropy
-    set_timer_for_anti_entropy(State),
+    State = #state{},
     %% init
     {ok, State}.
 
@@ -285,32 +74,6 @@ init([]) ->
     {noreply, #state{}, Timeout :: non_neg_integer()} |
     {stop, Reason :: any(), Reply :: any(), #state{}} |
     {stop, Reason :: any(), #state{}}.
-
-handle_call({join_on_node, GroupName, Pid, Meta}, _From, State) ->
-    %% check if pid is alive
-    case is_process_alive(Pid) of
-        true ->
-            join_on_node(GroupName, Pid, Meta),
-            %% multicast
-            multicast_join(GroupName, Pid, Meta, State),
-            %% return
-            {reply, ok, State};
-        _ ->
-            {reply, {error, not_alive}, State}
-    end;
-
-handle_call({leave_on_node, GroupName, Pid}, _From, State) ->
-    case leave_on_node(GroupName, Pid) of
-        ok ->
-            %% multicast
-            multicast_leave(GroupName, Pid, State),
-            %% return
-            {reply, ok, State};
-        {error, Reason} ->
-            %% return
-            {reply, {error, Reason}, State}
-    end;
-
 handle_call(Request, From, State) ->
     error_logger:warning_msg("Syn(~p): Received from ~p an unknown call message: ~p~n", [node(), Request, From]),
     {reply, undefined, State}.
@@ -322,24 +85,6 @@ handle_call(Request, From, State) ->
     {noreply, #state{}} |
     {noreply, #state{}, Timeout :: non_neg_integer()} |
     {stop, Reason :: any(), #state{}}.
-
-handle_cast({sync_join, GroupName, Pid, Meta}, State) ->
-    %% add to table
-    add_to_local_table(GroupName, Pid, Meta, undefined),
-    %% return
-    {noreply, State};
-
-handle_cast({sync_leave, GroupName, Pid}, State) ->
-    %% remove entry
-    remove_from_local_table(GroupName, Pid),
-    %% return
-    {noreply, State};
-
-handle_cast(force_cluster_sync, State) ->
-    error_logger:info_msg("Syn(~p): Initiating full cluster groups FORCED sync for nodes: ~p~n", [node(), nodes()]),
-    do_sync_from_full_cluster(),
-    {noreply, State};
-
 handle_cast(Msg, State) ->
     error_logger:warning_msg("Syn(~p): Received an unknown cast message: ~p~n", [node(), Msg]),
     {noreply, State}.
@@ -352,58 +97,6 @@ handle_cast(Msg, State) ->
     {noreply, #state{}, Timeout :: non_neg_integer()} |
     {stop, Reason :: any(), #state{}}.
 
-handle_info({'DOWN', _MonitorRef, process, Pid, Reason}, State) ->
-    case find_groups_tuples_by_pid(Pid) of
-        [] ->
-            %% handle
-            handle_process_down(undefined, Pid, undefined, Reason, State);
-
-        GroupTuples ->
-            lists:foreach(fun({GroupName, _Pid, Meta}) ->
-                %% remove from table
-                remove_from_local_table(GroupName, Pid),
-                %% handle
-                handle_process_down(GroupName, Pid, Meta, Reason, State),
-                %% multicast
-                multicast_leave(GroupName, Pid, State)
-            end, GroupTuples)
-    end,
-    %% return
-    {noreply, State};
-
-handle_info({nodeup, RemoteNode}, State) ->
-    error_logger:info_msg("Syn(~p): Node ~p has joined the cluster~n", [node(), RemoteNode]),
-    groups_automerge(RemoteNode),
-    %% resume
-    {noreply, State};
-
-handle_info({nodedown, RemoteNode}, State) ->
-    error_logger:warning_msg("Syn(~p): Node ~p has left the cluster, removing group entries on local~n", [node(), RemoteNode]),
-    raw_purge_group_entries_for_node(RemoteNode),
-    {noreply, State};
-
-handle_info(sync_from_full_cluster, State) ->
-    error_logger:info_msg("Syn(~p): Initiating full cluster groups sync for nodes: ~p~n", [node(), nodes()]),
-    do_sync_from_full_cluster(),
-    {noreply, State};
-
-handle_info(sync_anti_entropy, State) ->
-    %% sync
-    RemoteNodes = nodes(),
-    case length(RemoteNodes) > 0 of
-        true ->
-            RandomRemoteNode = lists:nth(rand:uniform(length(RemoteNodes)), RemoteNodes),
-            error_logger:info_msg("Syn(~p): Initiating anti-entropy sync for node ~p~n", [node(), RandomRemoteNode]),
-            groups_automerge(RandomRemoteNode);
-
-        _ ->
-            ok
-    end,
-    %% set timer
-    set_timer_for_anti_entropy(State),
-    %% return
-    {noreply, State};
-
 handle_info(Info, State) ->
     error_logger:warning_msg("Syn(~p): Received an unknown info message: ~p~n", [node(), Info]),
     {noreply, State}.
@@ -412,11 +105,8 @@ handle_info(Info, State) ->
 %% Terminate
 %% ----------------------------------------------------------------------------------------------------------
 -spec terminate(Reason :: any(), #state{}) -> terminated.
-terminate(Reason, #state{
-    multicast_pid = MulticastPid
-}) ->
+terminate(Reason, _State) ->
     error_logger:info_msg("Syn(~p): Terminating with reason: ~p~n", [node(), Reason]),
-    MulticastPid ! terminate,
     terminated.
 
 %% ----------------------------------------------------------------------------------------------------------
@@ -429,256 +119,3 @@ code_change(_OldVsn, State, _Extra) ->
 %% ===================================================================
 %% Internal
 %% ===================================================================
--spec multicast_join(GroupName :: any(), Pid :: pid(), Meta :: any(), #state{}) -> any().
-multicast_join(GroupName, Pid, Meta, #state{
-    multicast_pid = MulticastPid
-}) ->
-    MulticastPid ! {multicast_join, GroupName, Pid, Meta}.
-
--spec multicast_leave(GroupName :: any(), Pid :: pid(), #state{}) -> any().
-multicast_leave(GroupName, Pid, #state{
-    multicast_pid = MulticastPid
-}) ->
-    MulticastPid ! {multicast_leave, GroupName, Pid}.
-
--spec join_on_node(GroupName :: any(), Pid :: pid(), Meta :: any()) -> ok.
-join_on_node(GroupName, Pid, Meta) ->
-    MonitorRef = case find_monitor_for_pid(Pid) of
-        undefined ->
-            %% process is not monitored yet, add
-            erlang:monitor(process, Pid);
-
-        MRef ->
-            MRef
-    end,
-    %% add to table
-    add_to_local_table(GroupName, Pid, Meta, MonitorRef).
-
--spec leave_on_node(GroupName :: any(), Pid :: pid()) -> ok | {error, Reason :: any()}.
-leave_on_node(GroupName, Pid) ->
-    case find_groups_entry_by_name_and_pid(GroupName, Pid) of
-        undefined ->
-            {error, not_in_group};
-
-        {GroupName, Pid, _Meta, MonitorRef, _Node} when MonitorRef =/= undefined ->
-            %% is this the last group process is in?
-            case find_groups_tuples_by_pid(Pid) of
-                [_GroupTuple] ->
-                    %% only one left (the one we're about to delete), demonitor
-                    erlang:demonitor(MonitorRef, [flush]);
-
-                _ ->
-                    ok
-            end,
-            %% remove from table
-            remove_from_local_table(GroupName, Pid);
-
-        {GroupName, Pid, _Meta, _MonitorRef, Node} = GroupsEntry when Node =:= node() ->
-            error_logger:error_msg(
-                "Syn(~p): INTERNAL ERROR | Group entry ~p has no monitor but it's running on node~n",
-                [node(), GroupsEntry]
-            ),
-            %% remove from table
-            remove_from_local_table(GroupName, Pid);
-
-        _ ->
-            %% race condition: leave request but entry in table is not a local pid (has no monitor)
-            %% ignore it, sync messages will take care of it
-            ok
-    end.
-
--spec add_to_local_table(GroupName :: any(), Pid :: pid(), Meta :: any(), MonitorRef :: undefined | reference()) -> ok.
-add_to_local_table(GroupName, Pid, Meta, MonitorRef) ->
-    ets:insert(syn_groups_by_name, {{GroupName, Pid}, Meta, MonitorRef, node(Pid)}),
-    ets:insert(syn_groups_by_pid, {{Pid, GroupName}, Meta, MonitorRef, node(Pid)}),
-    ok.
-
--spec remove_from_local_table(GroupName :: any(), Pid :: pid()) -> ok | {error, Reason :: any()}.
-remove_from_local_table(GroupName, Pid) ->
-    case ets:lookup(syn_groups_by_name, {GroupName, Pid}) of
-        [] ->
-            {error, not_in_group};
-
-        _ ->
-            ets:match_delete(syn_groups_by_name, {{GroupName, Pid}, '_', '_', '_'}),
-            ets:match_delete(syn_groups_by_pid, {{Pid, GroupName}, '_', '_', '_'}),
-            ok
-    end.
-
--spec find_groups_tuples_by_pid(Pid :: pid()) -> GroupTuples :: list(syn_groups_tuple()).
-find_groups_tuples_by_pid(Pid) when is_pid(Pid) ->
-    ets:select(syn_groups_by_pid, [{
-        {{Pid, '$2'}, '$3', '_', '_'},
-        [],
-        [{{'$2', Pid, '$3'}}]
-    }]).
-
--spec find_groups_entry_by_name_and_pid(GroupName :: any(), Pid :: pid()) -> Entry :: syn_groups_entry() | undefined.
-find_groups_entry_by_name_and_pid(GroupName, Pid) ->
-    case ets:select(syn_groups_by_name, [{
-        {{GroupName, Pid}, '$3', '$4', '$5'},
-        [],
-        [{{{const, GroupName}, Pid, '$3', '$4', '$5'}}]
-    }]) of
-        [RegistryTuple] -> RegistryTuple;
-        _ -> undefined
-    end.
-
--spec find_monitor_for_pid(Pid :: pid()) -> reference() | undefined.
-find_monitor_for_pid(Pid) when is_pid(Pid) ->
-    case ets:select(syn_groups_by_pid, [{
-        {{Pid, '_'}, '_', '$4', '_'},
-        [],
-        ['$4']
-    }], 1) of
-        {[MonitorRef], _} -> MonitorRef;
-        _ -> undefined
-    end.
-
--spec get_groups_tuples_for_node(Node :: node()) -> [syn_groups_tuple()].
-get_groups_tuples_for_node(Node) ->
-    ets:select(syn_groups_by_name, [{
-        {{'$1', '$2'}, '$3', '_', Node},
-        [],
-        [{{'$1', '$2', '$3'}}]
-    }]).
-
--spec handle_process_down(GroupName :: any(), Pid :: pid(), Meta :: any(), Reason :: any(), #state{}) -> ok.
-handle_process_down(GroupName, Pid, Meta, Reason, #state{
-    custom_event_handler = CustomEventHandler
-}) ->
-    case GroupName of
-        undefined ->
-            error_logger:warning_msg(
-                "Syn(~p): Received a DOWN message from an unjoined group process ~p with reason: ~p~n",
-                [node(), Pid, Reason]
-            );
-        _ ->
-            syn_event_handler:do_on_group_process_exit(GroupName, Pid, Meta, Reason, CustomEventHandler)
-    end.
-
--spec groups_automerge(RemoteNode :: node()) -> ok.
-groups_automerge(RemoteNode) ->
-    global:trans({{?MODULE, auto_merge_groups}, self()},
-        fun() ->
-            error_logger:info_msg("Syn(~p): GROUPS AUTOMERGE ----> Initiating for remote node ~p~n", [node(), RemoteNode]),
-            %% get group tuples from remote node
-            case rpc:call(RemoteNode, ?MODULE, sync_get_local_groups_tuples, [node()]) of
-                {badrpc, _} ->
-                    error_logger:info_msg("Syn(~p): GROUPS AUTOMERGE <---- Syn not ready on remote node ~p, postponing~n", [node(), RemoteNode]);
-
-                GroupTuples ->
-                    error_logger:info_msg(
-                        "Syn(~p): Received ~p group tuple(s) from remote node ~p~n",
-                        [node(), length(GroupTuples), RemoteNode]
-                    ),
-                    %% ensure that groups doesn't have any joining node's entries
-                    raw_purge_group_entries_for_node(RemoteNode),
-                    %% add
-                    lists:foreach(fun({GroupName, RemotePid, RemoteMeta}) ->
-                        case rpc:call(node(RemotePid), erlang, is_process_alive, [RemotePid]) of
-                            true ->
-                                add_to_local_table(GroupName, RemotePid, RemoteMeta, undefined);
-                            _ ->
-                                ok = rpc:call(RemoteNode, syn_groups, remove_from_local_table, [GroupName, RemotePid])
-                        end
-                    end, GroupTuples),
-                    %% exit
-                    error_logger:info_msg("Syn(~p): GROUPS AUTOMERGE <---- Done for remote node ~p~n", [node(), RemoteNode])
-            end
-        end
-    ).
-
--spec do_sync_from_full_cluster() -> ok.
-do_sync_from_full_cluster() ->
-    lists:foreach(fun(RemoteNode) ->
-        groups_automerge(RemoteNode)
-    end, nodes()).
-
--spec raw_purge_group_entries_for_node(Node :: atom()) -> ok.
-raw_purge_group_entries_for_node(Node) ->
-    %% NB: no demonitoring is done, this is why it's raw
-    ets:match_delete(syn_groups_by_name, {{'_', '_'}, '_', '_', Node}),
-    ets:match_delete(syn_groups_by_pid, {{'_', '_'}, '_', '_', Node}),
-    ok.
-
--spec multi_call_and_receive(
-    CollectorPid :: pid(),
-    Pid :: pid(),
-    Message :: any(),
-    Timeout :: non_neg_integer()
-) -> any().
-
-multi_call_and_receive(CollectorPid, Pid, Message, Timeout) ->
-    MonitorRef = monitor(process, Pid),
-    Pid ! {syn_multi_call, self(), Message},
-
-    receive
-        {syn_multi_call_reply, Pid, Reply} ->
-            CollectorPid ! {reply, Pid, Reply};
-        {'DOWN', MonitorRef, _, _, _} ->
-            CollectorPid ! {bad_pid, Pid}
-    after Timeout ->
-        CollectorPid ! {bad_pid, Pid}
-    end.
-
--spec collect_replies(MemberPids :: [pid()]) -> {[{pid(), Reply :: any()}], [BadPid :: pid()]}.
-collect_replies(MemberPids) ->
-    collect_replies(MemberPids, [], []).
-
--spec collect_replies(MemberPids :: [pid()], [{pid(), Reply :: any()}], [pid()]) ->
-    {[{pid(), Reply :: any()}], [BadPid :: pid()]}.
-collect_replies([], Replies, BadPids) -> {Replies, BadPids};
-collect_replies(MemberPids, Replies, BadPids) ->
-    receive
-        {reply, Pid, Reply} ->
-            MemberPids1 = lists:delete(Pid, MemberPids),
-            collect_replies(MemberPids1, [{Pid, Reply} | Replies], BadPids);
-        {bad_pid, Pid} ->
-            MemberPids1 = lists:delete(Pid, MemberPids),
-            collect_replies(MemberPids1, Replies, [Pid | BadPids])
-    end.
-
--spec rebuild_monitors() -> ok.
-rebuild_monitors() ->
-    GroupTuples = get_groups_tuples_for_node(node()),
-    %% ensure that groups doesn't have any joining node's entries
-    raw_purge_group_entries_for_node(node()),
-    %% add
-    lists:foreach(fun({GroupName, Pid, Meta}) ->
-        case erlang:is_process_alive(Pid) of
-            true ->
-                join_on_node(GroupName, Pid, Meta);
-            _ ->
-                remove_from_local_table(GroupName, Pid)
-        end
-    end, GroupTuples).
-
--spec set_timer_for_anti_entropy(#state{}) -> ok.
-set_timer_for_anti_entropy(#state{anti_entropy_interval_ms = undefined}) -> ok;
-set_timer_for_anti_entropy(#state{
-    anti_entropy_interval_ms = AntiEntropyIntervalMs,
-    anti_entropy_interval_max_deviation_ms = AntiEntropyIntervalMaxDeviationMs
-}) ->
-    IntervalMs = round(AntiEntropyIntervalMs + rand:uniform() * AntiEntropyIntervalMaxDeviationMs),
-    {ok, _} = timer:send_after(IntervalMs, self(), sync_anti_entropy),
-    ok.
-
--spec multicast_loop() -> terminated.
-multicast_loop() ->
-    receive
-        {multicast_join, GroupName, Pid, Meta} ->
-            lists:foreach(fun(RemoteNode) ->
-                sync_join(RemoteNode, GroupName, Pid, Meta)
-            end, nodes()),
-            multicast_loop();
-
-        {multicast_leave, GroupName, Pid} ->
-            lists:foreach(fun(RemoteNode) ->
-                sync_leave(RemoteNode, GroupName, Pid)
-            end, nodes()),
-            multicast_loop();
-
-        terminate ->
-            terminated
-    end.

src/syn_registry.erl

@@ -3,7 +3,7 @@
 %%
 %% The MIT License (MIT)
 %%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
+%% Copyright (c) 2015-2021 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
 %%
 %% Permission is hereby granted, free of charge, to any person obtaining a copy
 %% of this software and associated documentation files (the "Software"), to deal
@@ -28,33 +28,12 @@
 
 %% API
 -export([start_link/0]).
--export([register/2, register/3]).
--export([unregister_and_register/2, unregister_and_register/3]).
--export([unregister/1]).
--export([whereis/1, whereis/2]).
--export([count/0, count/1]).
-
-%% sync API
--export([sync_register/6, sync_unregister/3]).
--export([sync_demonitor_and_kill_on_node/5]).
--export([sync_get_local_registry_tuples/1]).
--export([force_cluster_sync/0]).
--export([add_to_local_table/5, remove_from_local_table/2]).
--export([find_monitor_for_pid/1]).
-
-%% internal
--export([multicast_loop/0]).
 
 %% gen_server callbacks
 -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).
 
 %% records
--record(state, {
-    custom_event_handler :: undefined | module(),
-    anti_entropy_interval_ms :: undefined | non_neg_integer(),
-    anti_entropy_interval_max_deviation_ms :: undefined | non_neg_integer(),
-    multicast_pid :: undefined | pid()
-}).
+-record(state, {}).
 
 %% includes
 -include("syn.hrl").
@@ -67,99 +46,6 @@ start_link() ->
     Options = [],
     gen_server:start_link({local, ?MODULE}, ?MODULE, [], Options).
 
--spec register(Name :: any(), Pid :: pid()) -> ok | {error, Reason :: any()}.
-register(Name, Pid) ->
-    register(Name, Pid, undefined).
-
--spec register(Name :: any(), Pid :: pid(), Meta :: any()) -> ok | {error, Reason :: any()}.
-register(Name, Pid, Meta) when is_pid(Pid) ->
-    Node = node(Pid),
-    gen_server:call({?MODULE, Node}, {register_on_node, Name, Pid, Meta, false}).
-
--spec unregister_and_register(Name :: any(), Pid :: pid()) -> ok | {error, Reason :: any()}.
-unregister_and_register(Name, Pid) ->
-    unregister_and_register(Name, Pid, undefined).
-
--spec unregister_and_register(Name :: any(), Pid :: pid(), Meta :: any()) -> ok | {error, Reason :: any()}.
-unregister_and_register(Name, Pid, Meta) when is_pid(Pid) ->
-    Node = node(Pid),
-    gen_server:call({?MODULE, Node}, {register_on_node, Name, Pid, Meta, true}).
-
--spec unregister(Name :: any()) -> ok | {error, Reason :: any()}.
-unregister(Name) ->
-    % get process' node
-    case find_registry_tuple_by_name(Name) of
-        undefined ->
-            {error, undefined};
-        {Name, Pid, _, _} ->
-            Node = node(Pid),
-            gen_server:call({?MODULE, Node}, {unregister_on_node, Name})
-    end.
-
--spec whereis(Name :: any()) -> pid() | undefined.
-whereis(Name) ->
-    case find_registry_tuple_by_name(Name) of
-        undefined -> undefined;
-        {Name, Pid, _, _} -> Pid
-    end.
-
--spec whereis(Name :: any(), with_meta) -> {pid(), Meta :: any()} | undefined.
-whereis(Name, with_meta) ->
-    case find_registry_tuple_by_name(Name) of
-        undefined -> undefined;
-        {Name, Pid, Meta, _} -> {Pid, Meta}
-    end.
-
--spec count() -> non_neg_integer().
-count() ->
-    ets:info(syn_registry_by_name, size).
-
--spec count(Node :: node()) -> non_neg_integer().
-count(Node) ->
-    ets:select_count(syn_registry_by_name, [{
-        {{'_', '_'}, '_', '_', '_', Node},
-        [],
-        [true]
-    }]).
-
--spec sync_register(
-    RemoteNode :: node(),
-    Name :: any(),
-    RemotePid :: pid(),
-    RemoteMeta :: any(),
-    RemoteTime :: integer(),
-    Force :: boolean()
-) ->
-    ok.
-sync_register(RemoteNode, Name, RemotePid, RemoteMeta, RemoteTime, Force) ->
-    gen_server:cast({?MODULE, RemoteNode}, {sync_register, Name, RemotePid, RemoteMeta, RemoteTime, Force}).
-
--spec sync_unregister(RemoteNode :: node(), Name :: any(), Pid :: pid()) -> ok.
-sync_unregister(RemoteNode, Name, Pid) ->
-    gen_server:cast({?MODULE, RemoteNode}, {sync_unregister, Name, Pid}).
-
--spec sync_demonitor_and_kill_on_node(
-    Name :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    MonitorRef :: reference(),
-    Kill :: boolean()
-) -> ok.
-sync_demonitor_and_kill_on_node(Name, Pid, Meta, MonitorRef, Kill) ->
-    RemoteNode = node(Pid),
-    gen_server:cast({?MODULE, RemoteNode}, {sync_demonitor_and_kill_on_node, Name, Pid, Meta, MonitorRef, Kill}).
-
--spec sync_get_local_registry_tuples(FromNode :: node()) -> [syn_registry_tuple()].
-sync_get_local_registry_tuples(FromNode) ->
-    error_logger:info_msg("Syn(~p): Received request of local registry tuples from remote node ~p~n", [node(), FromNode]),
-    get_registry_tuples_for_node(node()).
-
--spec force_cluster_sync() -> ok.
-force_cluster_sync() ->
-    lists:foreach(fun(RemoteNode) ->
-        gen_server:cast({?MODULE, RemoteNode}, force_cluster_sync)
-    end, [node() | nodes()]).
-
 %% ===================================================================
 %% Callbacks
 %% ===================================================================
@@ -173,27 +59,8 @@ force_cluster_sync() ->
     ignore |
     {stop, Reason :: any()}.
 init([]) ->
-    %% monitor nodes
-    ok = net_kernel:monitor_nodes(true),
-    %% rebuild monitors (if coming after a crash)
-    rebuild_monitors(),
-    %% start multicast process
-    MulticastPid = spawn_link(?MODULE, multicast_loop, []),
-    %% get handler
-    CustomEventHandler = syn_backbone:get_event_handler_module(),
-    %% get anti-entropy interval
-    {AntiEntropyIntervalMs, AntiEntropyIntervalMaxDeviationMs} = syn_backbone:get_anti_entropy_settings(registry),
     %% build state
-    State = #state{
-        custom_event_handler = CustomEventHandler,
-        anti_entropy_interval_ms = AntiEntropyIntervalMs,
-        anti_entropy_interval_max_deviation_ms = AntiEntropyIntervalMaxDeviationMs,
-        multicast_pid = MulticastPid
-    },
-    %% send message to initiate full cluster sync
-    timer:send_after(0, self(), sync_from_full_cluster),
-    %% start anti-entropy
-    set_timer_for_anti_entropy(State),
+    State = #state{},
     %% init
     {ok, State}.
 
@@ -207,60 +74,6 @@ init([]) ->
     {noreply, #state{}, Timeout :: non_neg_integer()} |
     {stop, Reason :: any(), Reply :: any(), #state{}} |
     {stop, Reason :: any(), #state{}}.
-
-handle_call({register_on_node, Name, Pid, Meta, Force}, _From, State) ->
-    %% check if pid is alive
-    case is_process_alive(Pid) of
-        true ->
-            %% check if name available
-            case find_registry_tuple_by_name(Name) of
-                undefined ->
-                    %% available
-                    {ok, Time} = register_on_node(Name, Pid, Meta),
-                    %% multicast
-                    multicast_register(Name, Pid, Meta, Time, false, State),
-                    %% return
-                    {reply, ok, State};
-
-                {Name, Pid, _, _} ->
-                    % same pid, overwrite
-                    {ok, Time} = register_on_node(Name, Pid, Meta),
-                    %% multicast
-                    multicast_register(Name, Pid, Meta, Time, false, State),
-                    %% return
-                    {reply, ok, State};
-
-                {Name, TablePid, _, _} ->
-                    %% same name, different pid
-                    case Force of
-                        true ->
-                            demonitor_if_local(TablePid),
-                            %% force register
-                            {ok, Time} = register_on_node(Name, Pid, Meta),
-                            %% multicast
-                            multicast_register(Name, Pid, Meta, Time, true, State),
-                            %% return
-                            {reply, ok, State};
-
-                        _ ->
-                            {reply, {error, taken}, State}
-                    end
-            end;
-        _ ->
-            {reply, {error, not_alive}, State}
-    end;
-
-handle_call({unregister_on_node, Name}, _From, State) ->
-    case unregister_on_node(Name) of
-        {ok, RemovedPid} ->
-            multicast_unregister(Name, RemovedPid, State),
-            %% return
-            {reply, ok, State};
-        {error, Reason} ->
-            %% return
-            {reply, {error, Reason}, State}
-    end;
-
 handle_call(Request, From, State) ->
     error_logger:warning_msg("Syn(~p): Received from ~p an unknown call message: ~p~n", [node(), Request, From]),
     {reply, undefined, State}.
@@ -272,104 +85,6 @@ handle_call(Request, From, State) ->
     {noreply, #state{}} |
     {noreply, #state{}, Timeout :: non_neg_integer()} |
     {stop, Reason :: any(), #state{}}.
-
-handle_cast({sync_register, Name, RemotePid, RemoteMeta, RemoteTime, Force}, State) ->
-    %% check for conflicts
-    case find_registry_tuple_by_name(Name) of
-        undefined ->
-            %% no conflict
-            add_to_local_table(Name, RemotePid, RemoteMeta, RemoteTime, undefined);
-
-        {Name, RemotePid, _, _} ->
-            %% same process, no conflict, overwrite
-            add_to_local_table(Name, RemotePid, RemoteMeta, RemoteTime, undefined);
-
-        {Name, TablePid, _, _} when Force =:= true ->
-            demonitor_if_local(TablePid),
-            %% overwrite
-            add_to_local_table(Name, RemotePid, RemoteMeta, RemoteTime, undefined);
-
-        {Name, TablePid, TableMeta, TableTime} when Force =:= false ->
-            %% different pid, we have a conflict
-            global:trans({{?MODULE, {inconsistent_name, Name}}, self()},
-                fun() ->
-                    error_logger:warning_msg(
-                        "Syn(~p): REGISTRY INCONSISTENCY (name: ~p for ~p and ~p) ----> Received from remote node ~p~n",
-                        [node(), Name, {TablePid, TableMeta, TableTime}, {RemotePid, RemoteMeta, RemoteTime}, node(RemotePid)]
-                    ),
-
-                    case resolve_conflict(Name, {TablePid, TableMeta, TableTime}, {RemotePid, RemoteMeta, RemoteTime}, State) of
-                        {TablePid, KillOtherPid} ->
-                            %% keep table
-                            %% demonitor
-                            MonitorRef = rpc:call(node(RemotePid), syn_registry, find_monitor_for_pid, [RemotePid]),
-                            sync_demonitor_and_kill_on_node(Name, RemotePid, RemoteMeta, MonitorRef, KillOtherPid),
-                            %% overwrite local data to all remote nodes, except TablePid's node
-                            NodesExceptLocalAndTablePidNode = nodes() -- [node(TablePid)],
-                            lists:foreach(fun(RNode) ->
-                                ok = rpc:call(RNode,
-                                    syn_registry, add_to_local_table,
-                                    [Name, TablePid, TableMeta, TableTime, undefined]
-                                )
-                            end, NodesExceptLocalAndTablePidNode);
-
-                        {RemotePid, KillOtherPid} ->
-                            %% keep remote
-                            %% demonitor
-                            MonitorRef = rpc:call(node(TablePid), syn_registry, find_monitor_for_pid, [TablePid]),
-                            sync_demonitor_and_kill_on_node(Name, TablePid, TableMeta, MonitorRef, KillOtherPid),
-                            %% overwrite remote data to all other nodes (including local), except RemotePid's node
-                            NodesExceptRemoteNode = [node() | nodes()] -- [node(RemotePid)],
-                            lists:foreach(fun(RNode) ->
-                                ok = rpc:call(RNode,
-                                    syn_registry, add_to_local_table,
-                                    [Name, RemotePid, RemoteMeta, RemoteTime, undefined]
-                                )
-                            end, NodesExceptRemoteNode);
-
-                        undefined ->
-                            AllNodes = [node() | nodes()],
-                            %% both are dead, remove from all nodes
-                            lists:foreach(fun(RNode) ->
-                                ok = rpc:call(RNode, syn_registry, remove_from_local_table, [Name, RemotePid])
-                            end, AllNodes)
-                    end,
-
-                    error_logger:info_msg(
-                        "Syn(~p): REGISTRY INCONSISTENCY (name: ~p)  <---- Done on all cluster~n",
-                        [node(), Name]
-                    )
-                end
-            )
-    end,
-    %% return
-    {noreply, State};
-
-handle_cast({sync_unregister, Name, Pid}, State) ->
-    %% remove
-    remove_from_local_table(Name, Pid),
-    %% return
-    {noreply, State};
-
-handle_cast(force_cluster_sync, State) ->
-    error_logger:info_msg("Syn(~p): Initiating full cluster FORCED registry sync for nodes: ~p~n", [node(), nodes()]),
-    do_sync_from_full_cluster(State),
-    {noreply, State};
-
-handle_cast({sync_demonitor_and_kill_on_node, Name, Pid, Meta, MonitorRef, Kill}, State) ->
-    error_logger:info_msg("Syn(~p): Sync demonitoring pid ~p~n", [node(), Pid]),
-    %% demonitor
-    catch erlang:demonitor(MonitorRef, [flush]),
-    %% kill
-    case Kill of
-        true ->
-            exit(Pid, {syn_resolve_kill, Name, Meta});
-
-        _ ->
-            ok
-    end,
-    {noreply, State};
-
 handle_cast(Msg, State) ->
     error_logger:warning_msg("Syn(~p): Received an unknown cast message: ~p~n", [node(), Msg]),
     {noreply, State}.
@@ -381,59 +96,6 @@ handle_cast(Msg, State) ->
     {noreply, #state{}} |
     {noreply, #state{}, Timeout :: non_neg_integer()} |
     {stop, Reason :: any(), #state{}}.
-
-handle_info({'DOWN', _MonitorRef, process, Pid, Reason}, State) ->
-    case find_registry_tuples_by_pid(Pid) of
-        [] ->
-            %% handle
-            handle_process_down(undefined, Pid, undefined, Reason, State);
-
-        Entries ->
-            lists:foreach(fun({Name, _Pid, Meta, _Time}) ->
-                %% handle
-                handle_process_down(Name, Pid, Meta, Reason, State),
-                %% remove from table
-                remove_from_local_table(Name, Pid),
-                %% multicast
-                multicast_unregister(Name, Pid, State)
-            end, Entries)
-    end,
-    %% return
-    {noreply, State};
-
-handle_info({nodeup, RemoteNode}, State) ->
-    error_logger:info_msg("Syn(~p): Node ~p has joined the cluster~n", [node(), RemoteNode]),
-    registry_automerge(RemoteNode, State),
-    %% resume
-    {noreply, State};
-
-handle_info({nodedown, RemoteNode}, State) ->
-    error_logger:warning_msg("Syn(~p): Node ~p has left the cluster, removing registry entries on local~n", [node(), RemoteNode]),
-    raw_purge_registry_entries_for_remote_node(RemoteNode),
-    {noreply, State};
-
-handle_info(sync_from_full_cluster, State) ->
-    error_logger:info_msg("Syn(~p): Initiating full cluster registry sync for nodes: ~p~n", [node(), nodes()]),
-    do_sync_from_full_cluster(State),
-    {noreply, State};
-
-handle_info(sync_anti_entropy, State) ->
-    %% sync
-    RemoteNodes = nodes(),
-    case length(RemoteNodes) > 0 of
-        true ->
-            RandomRemoteNode = lists:nth(rand:uniform(length(RemoteNodes)), RemoteNodes),
-            error_logger:info_msg("Syn(~p): Initiating anti-entropy sync for node ~p~n", [node(), RandomRemoteNode]),
-            registry_automerge(RandomRemoteNode, State);
-
-        _ ->
-            ok
-    end,
-    %% set timer
-    set_timer_for_anti_entropy(State),
-    %% return
-    {noreply, State};
-
 handle_info(Info, State) ->
     error_logger:warning_msg("Syn(~p): Received an unknown info message: ~p~n", [node(), Info]),
     {noreply, State}.
@@ -442,11 +104,8 @@ handle_info(Info, State) ->
 %% Terminate
 %% ----------------------------------------------------------------------------------------------------------
 -spec terminate(Reason :: any(), #state{}) -> terminated.
-terminate(Reason, #state{
-    multicast_pid = MulticastPid
-}) ->
+terminate(Reason, _State) ->
     error_logger:info_msg("Syn(~p): Terminating with reason: ~p~n", [node(), Reason]),
-    MulticastPid ! terminate,
     terminated.
 
 %% ----------------------------------------------------------------------------------------------------------
@@ -459,403 +118,3 @@ code_change(_OldVsn, State, _Extra) ->
 %% ===================================================================
 %% Internal
 %% ===================================================================
--spec multicast_register(
-    Name :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Time :: integer(),
-    Force :: boolean(),
-    #state{}
-) -> any().
-multicast_register(Name, Pid, Meta, Time, Force, #state{
-    multicast_pid = MulticastPid
-}) ->
-    MulticastPid ! {multicast_register, Name, Pid, Meta, Time, Force}.
-
--spec multicast_unregister(Name :: any(), Pid :: pid(), #state{}) -> any().
-multicast_unregister(Name, Pid, #state{
-    multicast_pid = MulticastPid
-}) ->
-    MulticastPid ! {multicast_unregister, Name, Pid}.
-
--spec register_on_node(Name :: any(), Pid :: pid(), Meta :: any()) -> {ok, Time :: integer()}.
-register_on_node(Name, Pid, Meta) ->
-    MonitorRef = case find_monitor_for_pid(Pid) of
-        undefined ->
-            %% process is not monitored yet, add
-            erlang:monitor(process, Pid);
-
-        MRef ->
-            MRef
-    end,
-    %% add to table
-    Time = erlang:system_time(),
-    add_to_local_table(Name, Pid, Meta, Time, MonitorRef),
-    {ok, Time}.
-
--spec unregister_on_node(Name :: any()) -> {ok, RemovedPid :: pid()} | {error, Reason :: any()}.
-unregister_on_node(Name) ->
-    case find_registry_entry_by_name(Name) of
-        undefined ->
-            {error, undefined};
-
-        {{Name, Pid}, _Meta, _Clock, MonitorRef, _Node} when MonitorRef =/= undefined ->
-            %% demonitor if the process is not registered under other names
-            maybe_demonitor(Pid),
-            %% remove from table
-            remove_from_local_table(Name, Pid),
-            %% return
-            {ok, Pid};
-
-        {{Name, Pid}, _Meta, _Clock, _MonitorRef, Node} = RegistryEntry when Node =:= node() ->
-            error_logger:error_msg(
-                "Syn(~p): INTERNAL ERROR | Registry entry ~p has no monitor but it's running on node~n",
-                [node(), RegistryEntry]
-            ),
-            %% remove from table
-            remove_from_local_table(Name, Pid),
-            %% return
-            {ok, Pid};
-
-        RegistryEntry ->
-            %% race condition: un-registration request but entry in table is not a local pid (has no monitor)
-            %% sync messages will take care of it
-            error_logger:info_msg(
-                "Syn(~p): Registry entry ~p is not monitored and it's not running on node~n",
-                [node(), RegistryEntry]
-            ),
-            {error, remote_pid}
-    end.
-
--spec maybe_demonitor(Pid :: pid()) -> ok.
-maybe_demonitor(Pid) ->
-    %% try to retrieve 2 items
-    %% if only 1 is returned it means that no other aliases exist for the Pid
-    case ets:select(syn_registry_by_pid, [{
-        {{Pid, '_'}, '_', '_', '$5', '_'},
-        [],
-        ['$5']
-    }], 2) of
-        {[MonitorRef], _} ->
-            %% no other aliases, demonitor
-            erlang:demonitor(MonitorRef, [flush]),
-            ok;
-        _ ->
-            ok
-    end.
-
--spec add_to_local_table(
-    Name :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Time :: integer(),
-    MonitorRef :: undefined | reference()
-) -> ok.
-add_to_local_table(Name, Pid, Meta, Time, MonitorRef) ->
-    %% remove entry if previous exists
-    case find_registry_tuple_by_name(Name) of
-        undefined ->
-            undefined;
-
-        {Name, PreviousPid, _, _} ->
-            remove_from_local_table(Name, PreviousPid)
-    end,
-    %% overwrite & add
-    ets:insert(syn_registry_by_name, {{Name, Pid}, Meta, Time, MonitorRef, node(Pid)}),
-    ets:insert(syn_registry_by_pid, {{Pid, Name}, Meta, Time, MonitorRef, node(Pid)}),
-    ok.
-
--spec remove_from_local_table(Name :: any(), Pid :: pid()) -> ok.
-remove_from_local_table(Name, Pid) ->
-    ets:delete(syn_registry_by_name, {Name, Pid}),
-    ets:delete(syn_registry_by_pid, {Pid, Name}),
-    ok.
-
--spec find_registry_tuple_by_name(Name :: any()) -> RegistryTuple :: syn_registry_tuple() | undefined.
-find_registry_tuple_by_name(Name) ->
-    case ets:select(syn_registry_by_name, [{
-        {{Name, '$2'}, '$3', '$4', '_', '_'},
-        [],
-        [{{{const, Name}, '$2', '$3', '$4'}}]
-    }]) of
-        [RegistryTuple] -> RegistryTuple;
-        _ -> undefined
-    end.
-
--spec find_registry_entry_by_name(Name :: any()) -> Entry :: syn_registry_entry() | undefined.
-find_registry_entry_by_name(Name) ->
-    case ets:select(syn_registry_by_name, [{
-        {{Name, '$2'}, '$3', '_', '_', '_'},
-        [],
-        ['$_']
-    }]) of
-        [RegistryTuple] -> RegistryTuple;
-        _ -> undefined
-    end.
-
--spec find_monitor_for_pid(Pid :: pid()) -> reference() | undefined.
-find_monitor_for_pid(Pid) when is_pid(Pid) ->
-    case ets:select(syn_registry_by_pid, [{
-        {{Pid, '_'}, '_', '_', '$5', '_'},
-        [],
-        ['$5']
-    }], 1) of
-        {[MonitorRef], _} -> MonitorRef;
-        _ -> undefined
-    end.
-
--spec find_registry_tuples_by_pid(Pid :: pid()) -> RegistryTuples :: [syn_registry_tuple()].
-find_registry_tuples_by_pid(Pid) when is_pid(Pid) ->
-    ets:select(syn_registry_by_pid, [{
-        {{Pid, '$2'}, '$3', '$4', '_', '_'},
-        [],
-        [{{'$2', Pid, '$3', '$4'}}]
-    }]).
-
--spec get_registry_tuples_for_node(Node :: node()) -> [syn_registry_tuple()].
-get_registry_tuples_for_node(Node) ->
-    ets:select(syn_registry_by_name, [{
-        {{'$1', '$2'}, '$3', '$4', '_', Node},
-        [],
-        [{{'$1', '$2', '$3', '$4'}}]
-    }]).
-
--spec handle_process_down(Name :: any(), Pid :: pid(), Meta :: any(), Reason :: any(), #state{}) -> ok.
-handle_process_down(Name, Pid, Meta, Reason, #state{
-    custom_event_handler = CustomEventHandler
-}) ->
-    case Name of
-        undefined ->
-            case Reason of
-                {syn_resolve_kill, KillName, KillMeta} ->
-                    syn_event_handler:do_on_process_exit(KillName, Pid, KillMeta, syn_resolve_kill, CustomEventHandler);
-
-                _ ->
-                    error_logger:warning_msg(
-                        "Syn(~p): Received a DOWN message from an unregistered process ~p with reason: ~p~n",
-                        [node(), Pid, Reason]
-                    )
-            end;
-
-        _ ->
-            syn_event_handler:do_on_process_exit(Name, Pid, Meta, Reason, CustomEventHandler)
-    end.
-
--spec registry_automerge(RemoteNode :: node(), #state{}) -> ok.
-registry_automerge(RemoteNode, State) ->
-    global:trans({{?MODULE, auto_merge_registry}, self()},
-        fun() ->
-            error_logger:info_msg("Syn(~p): REGISTRY AUTOMERGE ----> Initiating for remote node ~p~n", [node(), RemoteNode]),
-            %% get registry tuples from remote node
-            case rpc:call(RemoteNode, ?MODULE, sync_get_local_registry_tuples, [node()]) of
-                {badrpc, _} ->
-                    error_logger:info_msg(
-                        "Syn(~p): REGISTRY AUTOMERGE <---- Syn not ready on remote node ~p, postponing~n",
-                        [node(), RemoteNode]
-                    );
-
-                RegistryTuples ->
-                    error_logger:info_msg(
-                        "Syn(~p): Received ~p registry tuple(s) from remote node ~p~n",
-                        [node(), length(RegistryTuples), RemoteNode]
-                    ),
-                    %% ensure that registry doesn't have any joining node's entries (here again for race conditions)
-                    raw_purge_registry_entries_for_remote_node(RemoteNode),
-                    %% loop
-                    F = fun({Name, RemotePid, RemoteMeta, RemoteTime}) ->
-                        resolve_tuple_in_automerge(Name, RemotePid, RemoteMeta, RemoteTime, State)
-                    end,
-                    %% add to table
-                    lists:foreach(F, RegistryTuples),
-                    %% exit
-                    error_logger:info_msg("Syn(~p): REGISTRY AUTOMERGE <---- Done for remote node ~p~n", [node(), RemoteNode])
-            end
-        end
-    ).
-
--spec resolve_tuple_in_automerge(
-    Name :: any(),
-    RemotePid :: pid(),
-    RemoteMeta :: any(),
-    RemoteTime :: integer(),
-    #state{}
-) -> any().
-resolve_tuple_in_automerge(Name, RemotePid, RemoteMeta, RemoteTime, State) ->
-    %% check if same name is registered
-    case find_registry_tuple_by_name(Name) of
-        undefined ->
-            %% no conflict
-            add_to_local_table(Name, RemotePid, RemoteMeta, RemoteTime, undefined);
-
-        {Name, TablePid, TableMeta, TableTime} ->
-            error_logger:warning_msg(
-                "Syn(~p): Conflicting name in auto merge for: ~p, processes are ~p, ~p~n",
-                [node(), Name, {TablePid, TableMeta, TableTime}, {RemotePid, RemoteMeta, RemoteTime}]
-            ),
-
-            case resolve_conflict(Name, {TablePid, TableMeta, TableTime}, {RemotePid, RemoteMeta, RemoteTime}, State) of
-                {TablePid, KillOtherPid} ->
-                    %% keep local
-                    %% demonitor
-                    MonitorRef = rpc:call(node(RemotePid), syn_registry, find_monitor_for_pid, [RemotePid]),
-                    sync_demonitor_and_kill_on_node(Name, RemotePid, RemoteMeta, MonitorRef, KillOtherPid),
-                    %% remote data still on remote node, remove there
-                    ok = rpc:call(node(RemotePid), syn_registry, remove_from_local_table, [Name, RemotePid]);
-
-                {RemotePid, KillOtherPid} ->
-                    %% keep remote
-                    %% demonitor
-                    MonitorRef = rpc:call(node(TablePid), syn_registry, find_monitor_for_pid, [TablePid]),
-                    sync_demonitor_and_kill_on_node(Name, TablePid, TableMeta, MonitorRef, KillOtherPid),
-                    %% overwrite remote data to local
-                    add_to_local_table(Name, RemotePid, RemoteMeta, RemoteTime, undefined);
-
-                undefined ->
-                    %% both are dead, remove from local & remote
-                    remove_from_local_table(Name, TablePid),
-                    ok = rpc:call(node(RemotePid), syn_registry, remove_from_local_table, [Name, RemotePid])
-            end
-    end.
-
--spec resolve_conflict(
-    Name :: any(),
-    {TablePid :: pid(), TableMeta :: any(), TableTime :: integer()},
-    {RemotePid :: pid(), RemoteMeta :: any(), RemoteTime :: integer()},
-    #state{}
-) -> {PidToKeep :: pid(), KillOtherPid :: boolean() | undefined} | undefined.
-resolve_conflict(
-    Name,
-    {TablePid, TableMeta, TableTime},
-    {RemotePid, RemoteMeta, RemoteTime},
-    #state{custom_event_handler = CustomEventHandler}
-) ->
-    TablePidAlive = rpc:call(node(TablePid), erlang, is_process_alive, [TablePid]),
-    RemotePidAlive = rpc:call(node(RemotePid), erlang, is_process_alive, [RemotePid]),
-
-    %% check if pids are alive (race conditions if pid dies during resolution)
-    {PidToKeep, KillOtherPid} = case {TablePidAlive, RemotePidAlive} of
-        {true, true} ->
-            %% call conflict resolution
-            syn_event_handler:do_resolve_registry_conflict(
-                Name,
-                {TablePid, TableMeta, TableTime},
-                {RemotePid, RemoteMeta, RemoteTime},
-                CustomEventHandler
-            );
-
-        {true, false} ->
-            %% keep only alive process
-            {TablePid, false};
-
-        {false, true} ->
-            %% keep only alive process
-            {RemotePid, false};
-
-        {false, false} ->
-            %% remove both
-            {undefined, false}
-    end,
-
-    %% keep chosen one
-    case PidToKeep of
-        TablePid ->
-            %% keep local
-            error_logger:info_msg(
-                "Syn(~p): Keeping process in table ~p over remote process ~p~n",
-                [node(), TablePid, RemotePid]
-            ),
-            {TablePid, KillOtherPid};
-
-        RemotePid ->
-            %% keep remote
-            error_logger:info_msg(
-                "Syn(~p): Keeping remote process ~p over process in table ~p~n",
-                [node(), RemotePid, TablePid]
-            ),
-            {RemotePid, KillOtherPid};
-
-        undefined ->
-            error_logger:info_msg(
-                "Syn(~p): Removing both processes' ~p and ~p data from local and remote tables~n",
-                [node(), RemotePid, TablePid]
-            ),
-            undefined;
-
-        Other ->
-            error_logger:error_msg(
-                "Syn(~p): Custom handler returned ~p, valid options were ~p and ~p, removing both~n",
-                [node(), Other, TablePid, RemotePid]
-            ),
-            undefined
-    end.
-
--spec do_sync_from_full_cluster(#state{}) -> ok.
-do_sync_from_full_cluster(State) ->
-    lists:foreach(fun(RemoteNode) ->
-        registry_automerge(RemoteNode, State)
-    end, nodes()).
-
--spec raw_purge_registry_entries_for_remote_node(Node :: atom()) -> ok.
-raw_purge_registry_entries_for_remote_node(Node) when Node =/= node() ->
-    %% NB: no demonitoring is done, this is why it's raw
-    ets:match_delete(syn_registry_by_name, {{'_', '_'}, '_', '_', '_', Node}),
-    ets:match_delete(syn_registry_by_pid, {{'_', '_'}, '_', '_', '_', Node}),
-    ok;
-raw_purge_registry_entries_for_remote_node(_Node) ->
-    ok.
-
--spec rebuild_monitors() -> ok.
-rebuild_monitors() ->
-    RegistryTuples = get_registry_tuples_for_node(node()),
-    lists:foreach(fun({Name, Pid, Meta, Time}) ->
-        case is_process_alive(Pid) of
-            true ->
-                MonitorRef = erlang:monitor(process, Pid),
-                %% overwrite
-                add_to_local_table(Name, Pid, Meta, Time, MonitorRef);
-            _ ->
-                remove_from_local_table(Name, Pid)
-        end
-    end, RegistryTuples).
-
--spec set_timer_for_anti_entropy(#state{}) -> ok.
-set_timer_for_anti_entropy(#state{anti_entropy_interval_ms = undefined}) -> ok;
-set_timer_for_anti_entropy(#state{
-    anti_entropy_interval_ms = AntiEntropyIntervalMs,
-    anti_entropy_interval_max_deviation_ms = AntiEntropyIntervalMaxDeviationMs
-}) ->
-    IntervalMs = round(AntiEntropyIntervalMs + rand:uniform() * AntiEntropyIntervalMaxDeviationMs),
-    {ok, _} = timer:send_after(IntervalMs, self(), sync_anti_entropy),
-    ok.
-
--spec demonitor_if_local(pid()) -> ok.
-demonitor_if_local(Pid) ->
-    case node(Pid) =:= node() of
-        true ->
-            %% demonitor
-            MonitorRef = syn_registry:find_monitor_for_pid(Pid),
-            catch erlang:demonitor(MonitorRef, [flush]),
-            ok;
-
-        _ ->
-            ok
-    end.
-
--spec multicast_loop() -> terminated.
-multicast_loop() ->
-    receive
-        {multicast_register, Name, Pid, Meta, Time, Force} ->
-            lists:foreach(fun(RemoteNode) ->
-                sync_register(RemoteNode, Name, Pid, Meta, Time, Force)
-            end, nodes()),
-            multicast_loop();
-
-        {multicast_unregister, Name, Pid} ->
-            lists:foreach(fun(RemoteNode) ->
-                sync_unregister(RemoteNode, Name, Pid)
-            end, nodes()),
-            multicast_loop();
-
-        terminate ->
-            terminated
-    end.

src/syn_sup.erl

@@ -3,7 +3,7 @@
 %%
 %% The MIT License (MIT)
 %%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
+%% Copyright (c) 2015-2021 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
 %%
 %% Permission is hereby granted, free of charge, to any person obtaining a copy
 %% of this software and associated documentation files (the "Software"), to deal

test/syn_groups_SUITE.erl

@@ -1,1285 +0,0 @@
-%% ==========================================================================================================
-%% Syn - A global Process Registry and Process Group manager.
-%%
-%% The MIT License (MIT)
-%%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
-%%
-%% Permission is hereby granted, free of charge, to any person obtaining a copy
-%% of this software and associated documentation files (the "Software"), to deal
-%% in the Software without restriction, including without limitation the rights
-%% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-%% copies of the Software, and to permit persons to whom the Software is
-%% furnished to do so, subject to the following conditions:
-%%
-%% The above copyright notice and this permission notice shall be included in
-%% all copies or substantial portions of the Software.
-%%
-%% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-%% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-%% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-%% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-%% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-%% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-%% THE SOFTWARE.
-%% ==========================================================================================================
--module(syn_groups_SUITE).
-
-%% callbacks
--export([all/0]).
--export([init_per_suite/1, end_per_suite/1]).
--export([groups/0, init_per_group/2, end_per_group/2]).
--export([init_per_testcase/2, end_per_testcase/2]).
-
-%% tests
--export([
-    single_node_join_and_monitor/1,
-    single_node_join_and_leave/1,
-    single_node_join_errors/1,
-    single_node_groups_count/1,
-    single_node_group_names/1,
-    single_node_publish/1,
-    single_node_multicall/1,
-    single_node_multicall_with_custom_timeout/1,
-    single_node_callback_on_process_exit/1,
-    single_node_monitor_after_group_crash/1
-]).
--export([
-    two_nodes_join_monitor_and_unregister/1,
-    two_nodes_local_members/1,
-    two_nodes_groups_count/1,
-    two_nodes_group_names/1,
-    two_nodes_publish/1,
-    two_nodes_local_publish/1,
-    two_nodes_multicall/1,
-    two_nodes_groups_full_cluster_sync_on_boot_node_added_later/1,
-    two_nodes_groups_full_cluster_sync_on_boot_syn_started_later/1,
-    three_nodes_anti_entropy/1,
-    three_nodes_anti_entropy_manual/1
-]).
--export([
-    three_nodes_partial_netsplit_consistency/1,
-    three_nodes_full_netsplit_consistency/1
-]).
-
-%% include
--include_lib("common_test/include/ct.hrl").
-
-%% ===================================================================
-%% Callbacks
-%% ===================================================================
-
-%% -------------------------------------------------------------------
-%% Function: all() -> GroupsAndTestCases | {skip,Reason}
-%% GroupsAndTestCases = [{group,GroupName} | TestCase]
-%% GroupName = atom()
-%% TestCase = atom()
-%% Reason = any()
-%% -------------------------------------------------------------------
-all() ->
-    [
-        {group, single_node_groups},
-        {group, two_nodes_groups},
-        {group, three_nodes_groups}
-    ].
-
-%% -------------------------------------------------------------------
-%% Function: groups() -> [Group]
-%% Group = {GroupName,Properties,GroupsAndTestCases}
-%% GroupName = atom()
-%% Properties = [parallel | sequence | Shuffle | {RepeatType,N}]
-%% GroupsAndTestCases = [Group | {group,GroupName} | TestCase]
-%% TestCase = atom()
-%% Shuffle = shuffle | {shuffle,{integer(),integer(),integer()}}
-%% RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail |
-%%			   repeat_until_any_ok | repeat_until_any_fail
-%% N = integer() | forever
-%% -------------------------------------------------------------------
-groups() ->
-    [
-        {single_node_groups, [shuffle], [
-            single_node_join_and_monitor,
-            single_node_join_and_leave,
-            single_node_join_errors,
-            single_node_groups_count,
-            single_node_group_names,
-            single_node_publish,
-            single_node_multicall,
-            single_node_multicall_with_custom_timeout,
-            single_node_callback_on_process_exit,
-            single_node_monitor_after_group_crash
-        ]},
-        {two_nodes_groups, [shuffle], [
-            two_nodes_join_monitor_and_unregister,
-            two_nodes_local_members,
-            two_nodes_groups_count,
-            two_nodes_group_names,
-            two_nodes_publish,
-            two_nodes_local_publish,
-            two_nodes_multicall,
-            two_nodes_groups_full_cluster_sync_on_boot_node_added_later,
-            two_nodes_groups_full_cluster_sync_on_boot_syn_started_later
-        ]},
-        {three_nodes_groups, [shuffle], [
-            three_nodes_partial_netsplit_consistency,
-            three_nodes_full_netsplit_consistency,
-            three_nodes_anti_entropy,
-            three_nodes_anti_entropy_manual
-        ]}
-    ].
-%% -------------------------------------------------------------------
-%% Function: init_per_suite(Config0) ->
-%%				Config1 | {skip,Reason} |
-%%              {skip_and_save,Reason,Config1}
-%% Config0 = Config1 = [tuple()]
-%% Reason = any()
-%% -------------------------------------------------------------------
-init_per_suite(Config) ->
-    Config.
-
-%% -------------------------------------------------------------------
-%% Function: end_per_suite(Config0) -> void() | {save_config,Config1}
-%% Config0 = Config1 = [tuple()]
-%% -------------------------------------------------------------------
-end_per_suite(_Config) ->
-    ok.
-
-%% -------------------------------------------------------------------
-%% Function: init_per_group(GroupName, Config0) ->
-%%				Config1 | {skip,Reason} |
-%%              {skip_and_save,Reason,Config1}
-%% GroupName = atom()
-%% Config0 = Config1 = [tuple()]
-%% Reason = any()
-%% -------------------------------------------------------------------
-init_per_group(two_nodes_groups, Config) ->
-    %% start slave
-    {ok, SlaveNode} = syn_test_suite_helper:start_slave(syn_slave),
-    %% config
-    [{slave_node, SlaveNode} | Config];
-init_per_group(three_nodes_groups, Config) ->
-    %% start slave
-    {ok, SlaveNode1} = syn_test_suite_helper:start_slave(syn_slave_1),
-    {ok, SlaveNode2} = syn_test_suite_helper:start_slave(syn_slave_2),
-    %% config
-    [{slave_node_1, SlaveNode1}, {slave_node_2, SlaveNode2} | Config];
-init_per_group(_GroupName, Config) ->
-    Config.
-
-%% -------------------------------------------------------------------
-%% Function: end_per_group(GroupName, Config0) ->
-%%				void() | {save_config,Config1}
-%% GroupName = atom()
-%% Config0 = Config1 = [tuple()]
-%% -------------------------------------------------------------------
-end_per_group(two_nodes_groups, Config) ->
-    SlaveNode = proplists:get_value(slave_node, Config),
-    syn_test_suite_helper:connect_node(SlaveNode),
-    syn_test_suite_helper:clean_after_test(),
-    syn_test_suite_helper:stop_slave(syn_slave),
-    timer:sleep(1000);
-end_per_group(three_nodes_groups, Config) ->
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    syn_test_suite_helper:connect_node(SlaveNode1),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    syn_test_suite_helper:connect_node(SlaveNode2),
-    syn_test_suite_helper:clean_after_test(),
-    syn_test_suite_helper:stop_slave(syn_slave_1),
-    syn_test_suite_helper:stop_slave(syn_slave_2),
-    timer:sleep(1000);
-end_per_group(_GroupName, _Config) ->
-    syn_test_suite_helper:clean_after_test().
-
-%% -------------------------------------------------------------------
-%% Function: init_per_testcase(TestCase, Config0) ->
-%%				Config1 | {skip,Reason} | {skip_and_save,Reason,Config1}
-%% TestCase = atom()
-%% Config0 = Config1 = [tuple()]
-%% Reason = any()
-%% -------------------------------------------------------------------
-init_per_testcase(TestCase, Config) ->
-    ct:pal("Starting test: ~p", [TestCase]),
-    Config.
-
-%% -------------------------------------------------------------------
-%% Function: end_per_testcase(TestCase, Config0) ->
-%%				void() | {save_config,Config1} | {fail,Reason}
-%% TestCase = atom()
-%% Config0 = Config1 = [tuple()]
-%% Reason = any()
-%% -------------------------------------------------------------------
-end_per_testcase(_, _Config) ->
-    syn_test_suite_helper:clean_after_test().
-
-%% ===================================================================
-%% Tests
-%% ===================================================================
-single_node_join_and_monitor(_Config) ->
-    GroupName = "my group",
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    PidWithMeta = syn_test_suite_helper:start_process(),
-    PidOther = syn_test_suite_helper:start_process(),
-    %% retrieve
-    [] = syn:get_members(GroupName),
-    [] = syn:get_members(GroupName, with_meta),
-    false = syn:member(Pid, GroupName),
-    false = syn:member(PidWithMeta, GroupName),
-    false = syn:member(PidOther, GroupName),
-    %% join
-    ok = syn:join(GroupName, Pid),
-    ok = syn:join(GroupName, PidWithMeta, {with, meta}),
-    ok = syn:join("other-group", PidOther),
-    %% retrieve
-    true = syn:member(Pid, GroupName),
-    true = syn:member(PidWithMeta, GroupName),
-    false = syn:member(PidOther, GroupName),
-    true = lists:sort([Pid, PidWithMeta]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([{Pid, undefined}, {PidWithMeta, {with, meta}}])
-        =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    %% re-join
-    ok = syn:join(GroupName, PidWithMeta, {with2, meta2}),
-    true = lists:sort([{Pid, undefined}, {PidWithMeta, {with2, meta2}}])
-        =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    %% kill process
-    syn_test_suite_helper:kill_process(Pid),
-    syn_test_suite_helper:kill_process(PidWithMeta),
-    syn_test_suite_helper:kill_process(PidOther),
-    timer:sleep(100),
-    %% retrieve
-    [] = syn:get_members(GroupName),
-    [] = syn:get_members(GroupName, with_meta),
-    false = syn:member(Pid, GroupName),
-    false = syn:member(PidWithMeta, GroupName).
-
-single_node_join_and_leave(_Config) ->
-    GroupName = "my group",
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    PidWithMeta = syn_test_suite_helper:start_process(),
-    %% retrieve
-    [] = syn:get_members(GroupName),
-    [] = syn:get_members(GroupName, with_meta),
-    false = syn:member(Pid, GroupName),
-    false = syn:member(PidWithMeta, GroupName),
-    %% join
-    ok = syn:join(GroupName, Pid),
-    ok = syn:join(GroupName, PidWithMeta, {with, meta}),
-    %% retrieve
-    true = syn:member(Pid, GroupName),
-    true = syn:member(PidWithMeta, GroupName),
-    true = lists:sort([Pid, PidWithMeta]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([{Pid, undefined}, {PidWithMeta, {with, meta}}])
-        =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    %% leave
-    ok = syn:leave(GroupName, Pid),
-    ok = syn:leave(GroupName, PidWithMeta),
-    timer:sleep(100),
-    %% retrieve
-    [] = syn:get_members(GroupName),
-    [] = syn:get_members(GroupName, with_meta),
-    false = syn:member(Pid, GroupName),
-    false = syn:member(PidWithMeta, GroupName).
-
-single_node_join_errors(_Config) ->
-    GroupName = "my group",
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    Pid2 = syn_test_suite_helper:start_process(),
-    %% join
-    ok = syn:join(GroupName, Pid),
-    ok = syn:join(GroupName, Pid2),
-    true = syn:member(Pid, GroupName),
-    true = syn:member(Pid2, GroupName),
-    %% leave
-    ok = syn:leave(GroupName, Pid),
-    {error, not_in_group} = syn:leave(GroupName, Pid),
-    %% kill
-    syn_test_suite_helper:kill_process(Pid2),
-    timer:sleep(200),
-    {error, not_in_group} = syn:leave(GroupName, Pid2),
-    {error, not_alive} = syn:join(GroupName, Pid2).
-
-single_node_groups_count(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    Pid2 = syn_test_suite_helper:start_process(),
-    Pid3 = syn_test_suite_helper:start_process(),
-    %% join
-    ok = syn:join({"group-1"}, Pid),
-    ok = syn:join({"group-1"}, Pid2),
-    ok = syn:join({"group-1"}, Pid3),
-    ok = syn:join({"group-2"}, Pid2),
-    %% count
-    2 = syn:groups_count(),
-    2 = syn:groups_count(node()),
-    %% kill & unregister
-    ok = syn:leave({"group-1"}, Pid),
-    syn_test_suite_helper:kill_process(Pid2),
-    syn_test_suite_helper:kill_process(Pid3),
-    timer:sleep(100),
-    %% count
-    0 = syn:groups_count(),
-    0 = syn:groups_count(node()).
-
-single_node_group_names(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    Pid2 = syn_test_suite_helper:start_process(),
-    Pid3 = syn_test_suite_helper:start_process(),
-    %% join
-    ok = syn:join({"group-1"}, Pid),
-    ok = syn:join({"group-1"}, Pid2),
-    ok = syn:join({"group-1"}, Pid3),
-    ok = syn:join({"group-2"}, Pid2),
-    %% names
-    GroupNames = syn:get_group_names(),
-    2 = length(GroupNames),
-    true = lists:member({"group-1"}, GroupNames),
-    true = lists:member({"group-2"}, GroupNames),
-    %% kill & unregister
-    ok = syn:leave({"group-1"}, Pid),
-    syn_test_suite_helper:kill_process(Pid2),
-    syn_test_suite_helper:kill_process(Pid3),
-    timer:sleep(100),
-    %% count
-    [] = syn:get_group_names().
-
-single_node_publish(_Config) ->
-    GroupName = "my group",
-    Message = {test, message},
-    %% start
-    ok = syn:start(),
-    %% start processes
-    ResultPid = self(),
-    F = fun() ->
-        receive
-            Message -> ResultPid ! {received, self(), Message}
-        end
-    end,
-    Pid = syn_test_suite_helper:start_process(F),
-    Pid2 = syn_test_suite_helper:start_process(F),
-    _OtherPid = syn_test_suite_helper:start_process(F),
-    %% join
-    ok = syn:join(GroupName, Pid),
-    ok = syn:join(GroupName, Pid2),
-    true = syn:member(Pid, GroupName),
-    true = syn:member(Pid2, GroupName),
-    %% send
-    {ok, 2} = syn:publish(GroupName, Message),
-    %% check
-    receive
-        {received, Pid, Message} -> ok
-    after 2000 ->
-        ok = published_message_was_not_received_by_pid_1
-    end,
-    receive
-        {received, Pid2, Message} -> ok
-    after 2000 ->
-        ok = published_message_was_not_received_by_pid_2
-    end.
-
-single_node_multicall(_Config) ->
-    GroupName = <<"my group">>,
-    %% start
-    ok = syn:start(),
-    %% start processes
-    F = fun() ->
-        receive
-            {syn_multi_call, RequestorPid, get_pid_name} ->
-                syn:multi_call_reply(RequestorPid, {pong, self()})
-        end
-    end,
-    Pid1 = syn_test_suite_helper:start_process(F),
-    Pid2 = syn_test_suite_helper:start_process(F),
-    PidUnresponsive = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:join(GroupName, Pid1),
-    ok = syn:join(GroupName, Pid2),
-    ok = syn:join(GroupName, PidUnresponsive),
-    %% call
-    {Replies, BadPids} = syn:multi_call(GroupName, get_pid_name),
-    %% check responses
-    true = lists:sort([
-        {Pid1, {pong, Pid1}},
-        {Pid2, {pong, Pid2}}
-    ]) =:= lists:sort(Replies),
-    [PidUnresponsive] = BadPids.
-
-single_node_multicall_with_custom_timeout(_Config) ->
-    GroupName = <<"my group">>,
-    %% start
-    ok = syn:start(),
-    %% start processes
-    F1 = fun() ->
-        receive
-            {syn_multi_call, RequestorPid, get_pid_name} ->
-                syn:multi_call_reply(RequestorPid, {pong, self()})
-        end
-    end,
-    Pid1 = syn_test_suite_helper:start_process(F1),
-    F2 = fun() ->
-        receive
-            {syn_multi_call, RequestorPid, get_pid_name} ->
-                timer:sleep(5000),
-                syn:multi_call_reply(RequestorPid, {pong, self()})
-        end
-    end,
-    PidTakesLong = syn_test_suite_helper:start_process(F2),
-    PidUnresponsive = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:join(GroupName, Pid1),
-    ok = syn:join(GroupName, PidTakesLong),
-    ok = syn:join(GroupName, PidUnresponsive),
-    %% call
-    {Replies, BadPids} = syn:multi_call(GroupName, get_pid_name, 2000),
-    %% check responses
-    [{Pid1, {pong, Pid1}}] = Replies,
-    true = lists:sort([PidTakesLong, PidUnresponsive]) =:= lists:sort(BadPids).
-
-single_node_callback_on_process_exit(_Config) ->
-    %% use custom handler
-    syn_test_suite_helper:use_custom_handler(),
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    Pid2 = syn_test_suite_helper:start_process(),
-    %% join
-    TestPid = self(),
-    ok = syn:join(group_1, Pid, {pid_group_1, TestPid}),
-    ok = syn:join(group_2, Pid, {pid_group_2, TestPid}),
-    ok = syn:join(group_1, Pid2, {pid2, TestPid}),
-    %% kill 1
-    syn_test_suite_helper:kill_process(Pid),
-    receive
-        {received_event_on, pid_group_1} ->
-            ok;
-        {received_event_on, pid2} ->
-            ok = callback_on_process_exit_was_received_by_pid2
-    after 1000 ->
-        ok = callback_on_process_exit_was_not_received_by_pid
-    end,
-    receive
-        {received_event_on, pid_group_2} ->
-            ok;
-        {received_event_on, pid2} ->
-            ok = callback_on_process_exit_was_received_by_pid2
-    after 1000 ->
-        ok = callback_on_process_exit_was_not_received_by_pid
-    end,
-    %% unregister & kill 2
-    ok = syn:leave(group_1, Pid2),
-    syn_test_suite_helper:kill_process(Pid2),
-    receive
-        {received_event_on, pid2} ->
-            ok = callback_on_process_exit_was_received_by_pid2
-    after 1000 ->
-        ok
-    end.
-
-single_node_monitor_after_group_crash(_Config) ->
-    GroupName = "my group",
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    %% join
-    ok = syn:join(GroupName, Pid),
-    %% kill groups
-    exit(whereis(syn_groups), kill),
-    timer:sleep(200),
-    %% retrieve
-    true = syn:member(Pid, GroupName),
-    [Pid] = syn:get_members(GroupName),
-    %% kill process
-    syn_test_suite_helper:kill_process(Pid),
-    timer:sleep(200),
-    %% retrieve
-    false = syn:member(Pid, GroupName),
-    [] = syn:get_members(GroupName).
-
-two_nodes_join_monitor_and_unregister(Config) ->
-    GroupName = "my group",
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    LocalPid = syn_test_suite_helper:start_process(),
-    RemotePid = syn_test_suite_helper:start_process(SlaveNode),
-    RemotePidJoinRemote = syn_test_suite_helper:start_process(SlaveNode),
-    OtherPid = syn_test_suite_helper:start_process(),
-    %% retrieve
-    [] = syn:get_members("group-1"),
-    [] = syn:get_members(GroupName),
-    [] = syn:get_members(GroupName, with_meta),
-    false = syn:member(LocalPid, GroupName),
-    false = syn:member(RemotePid, GroupName),
-    false = syn:member(RemotePidJoinRemote, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    [] = rpc:call(SlaveNode, syn, get_members, [GroupName]),
-    [] = rpc:call(SlaveNode, syn, get_members, [GroupName, with_meta]),
-    false = rpc:call(SlaveNode, syn, member, [LocalPid, GroupName]),
-    false = rpc:call(SlaveNode, syn, member, [RemotePid, GroupName]),
-    false = rpc:call(SlaveNode, syn, member, [RemotePidJoinRemote, GroupName]),
-    false = rpc:call(SlaveNode, syn, member, [OtherPid, GroupName]),
-    %% join
-    ok = syn:join(GroupName, LocalPid),
-    ok = syn:join(GroupName, RemotePid, {with_meta}),
-    ok = rpc:call(SlaveNode, syn, join, [GroupName, RemotePidJoinRemote]),
-    ok = syn:join("other-group", OtherPid),
-    timer:sleep(200),
-    %% retrieve local
-    true = lists:sort([LocalPid, RemotePid, RemotePidJoinRemote]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([{LocalPid, undefined}, {RemotePid, {with_meta}}, {RemotePidJoinRemote, undefined}])
-        =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    true = syn:member(LocalPid, GroupName),
-    true = syn:member(RemotePid, GroupName),
-    true = syn:member(RemotePidJoinRemote, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    %% retrieve remote
-    true = lists:sort([LocalPid, RemotePid, RemotePidJoinRemote])
-        =:= lists:sort(rpc:call(SlaveNode, syn, get_members, [GroupName])),
-    true = lists:sort([{LocalPid, undefined}, {RemotePid, {with_meta}}, {RemotePidJoinRemote, undefined}])
-        =:= lists:sort(rpc:call(SlaveNode, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode, syn, member, [LocalPid, GroupName]),
-    true = rpc:call(SlaveNode, syn, member, [RemotePid, GroupName]),
-    true = rpc:call(SlaveNode, syn, member, [RemotePidJoinRemote, GroupName]),
-    false = rpc:call(SlaveNode, syn, member, [OtherPid, GroupName]),
-    %% leave & kill
-    ok = rpc:call(SlaveNode, syn, leave, [GroupName, LocalPid]),
-    ok = syn:leave(GroupName, RemotePid),
-    syn_test_suite_helper:kill_process(RemotePidJoinRemote),
-    syn_test_suite_helper:kill_process(OtherPid),
-    timer:sleep(200),
-    %% retrieve
-    [] = syn:get_members("group-1"),
-    [] = syn:get_members(GroupName),
-    [] = syn:get_members(GroupName, with_meta),
-    false = syn:member(LocalPid, GroupName),
-    false = syn:member(RemotePid, GroupName),
-    false = syn:member(RemotePidJoinRemote, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    [] = rpc:call(SlaveNode, syn, get_members, [GroupName]),
-    [] = rpc:call(SlaveNode, syn, get_members, [GroupName, with_meta]),
-    false = rpc:call(SlaveNode, syn, member, [LocalPid, GroupName]),
-    false = rpc:call(SlaveNode, syn, member, [RemotePid, GroupName]),
-    false = rpc:call(SlaveNode, syn, member, [RemotePidJoinRemote, GroupName]),
-    false = rpc:call(SlaveNode, syn, member, [OtherPid, GroupName]).
-
-two_nodes_local_members(Config) ->
-    GroupName = "my group",
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    LocalPid = syn_test_suite_helper:start_process(),
-    RemotePid = syn_test_suite_helper:start_process(SlaveNode),
-    RemotePidJoinRemote = syn_test_suite_helper:start_process(SlaveNode),
-    OtherPid = syn_test_suite_helper:start_process(),
-    %% local members
-    [] = syn:get_local_members(GroupName),
-    [] = syn:get_local_members(GroupName, with_meta),
-    false = syn:local_member(LocalPid, GroupName),
-    false = syn:local_member(RemotePid, GroupName),
-    false = syn:local_member(RemotePidJoinRemote, GroupName),
-    false = syn:local_member(OtherPid, GroupName),
-    %% remote members
-    [] = rpc:call(SlaveNode, syn, get_local_members, [GroupName]),
-    [] = rpc:call(SlaveNode, syn, get_local_members, [GroupName, with_meta]),
-    false = rpc:call(SlaveNode, syn, local_member, [LocalPid, GroupName]),
-    false = rpc:call(SlaveNode, syn, local_member, [RemotePid, GroupName]),
-    false = rpc:call(SlaveNode, syn, local_member, [RemotePidJoinRemote, GroupName]),
-    false = rpc:call(SlaveNode, syn, local_member, [OtherPid, GroupName]),
-    %% join
-    ok = syn:join(GroupName, LocalPid),
-    ok = syn:join(GroupName, RemotePid, {meta, 2}),
-    ok = rpc:call(SlaveNode, syn, join, [GroupName, RemotePidJoinRemote]),
-    ok = syn:join({"other-group"}, OtherPid),
-    timer:sleep(200),
-    %% local members
-    [LocalPid] = syn:get_local_members(GroupName),
-    [{LocalPid, undefined}] = syn:get_local_members(GroupName, with_meta),
-    [OtherPid] = syn:get_local_members({"other-group"}),
-    true = syn:local_member(LocalPid, GroupName),
-    false = syn:local_member(RemotePid, GroupName),
-    false = syn:local_member(RemotePidJoinRemote, GroupName),
-    false = syn:local_member(OtherPid, GroupName),
-    true = syn:local_member(OtherPid, {"other-group"}),
-    %% remote members
-    true = lists:sort([RemotePid, RemotePidJoinRemote])
-        =:= lists:sort(rpc:call(SlaveNode, syn, get_local_members, [GroupName])),
-    true = lists:sort([{RemotePid, {meta, 2}}, {RemotePidJoinRemote, undefined}])
-        =:= lists:sort(rpc:call(SlaveNode, syn, get_local_members, [GroupName, with_meta])),
-    false = rpc:call(SlaveNode, syn, local_member, [LocalPid, GroupName]),
-    true = rpc:call(SlaveNode, syn, local_member, [RemotePid, GroupName]),
-    true = rpc:call(SlaveNode, syn, local_member, [RemotePidJoinRemote, GroupName]),
-    false = rpc:call(SlaveNode, syn, local_member, [OtherPid, GroupName]),
-    %% leave & kill
-    ok = rpc:call(SlaveNode, syn, leave, [GroupName, LocalPid]),
-    ok = syn:leave(GroupName, RemotePid),
-    syn_test_suite_helper:kill_process(RemotePidJoinRemote),
-    syn_test_suite_helper:kill_process(OtherPid),
-    timer:sleep(200),
-    %% local members
-    [] = syn:get_local_members(GroupName),
-    [] = syn:get_local_members(GroupName, with_meta),
-    %% remote members
-    [] = rpc:call(SlaveNode, syn, get_local_members, [GroupName]),
-    [] = rpc:call(SlaveNode, syn, get_local_members, [GroupName, with_meta]).
-
-two_nodes_groups_count(Config) ->
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    LocalPid = syn_test_suite_helper:start_process(),
-    RemotePid = syn_test_suite_helper:start_process(SlaveNode),
-    RemotePidRegRemote = syn_test_suite_helper:start_process(SlaveNode),
-    _PidUnjoined = syn_test_suite_helper:start_process(),
-    %% join
-    ok = syn:join(<<"local group">>, LocalPid),
-    ok = syn:join(<<"remote group">>, RemotePid),
-    ok = rpc:call(SlaveNode, syn, join, [<<"remote group join_remote">>, RemotePidRegRemote]),
-    timer:sleep(500),
-    %% count
-    3 = syn:groups_count(),
-    1 = syn:groups_count(node()),
-    2 = syn:groups_count(SlaveNode),
-    %% kill & unregister processes
-    syn_test_suite_helper:kill_process(LocalPid),
-    ok = syn:leave(<<"remote group">>, RemotePid),
-    syn_test_suite_helper:kill_process(RemotePidRegRemote),
-    timer:sleep(100),
-    %% count
-    0 = syn:groups_count(),
-    0 = syn:groups_count(node()),
-    0 = syn:groups_count(SlaveNode).
-
-two_nodes_group_names(Config) ->
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    LocalPid = syn_test_suite_helper:start_process(),
-    RemotePid = syn_test_suite_helper:start_process(SlaveNode),
-    RemotePidRegRemote = syn_test_suite_helper:start_process(SlaveNode),
-    _PidUnjoined = syn_test_suite_helper:start_process(),
-    %% join
-    ok = syn:join(<<"local group">>, LocalPid),
-    ok = syn:join(<<"remote group">>, RemotePid),
-    ok = rpc:call(SlaveNode, syn, join, [<<"remote group join_remote">>, RemotePidRegRemote]),
-    timer:sleep(500),
-    %% names
-    GroupNames = syn:get_group_names(),
-    3 = length(GroupNames),
-    true = lists:member(<<"local group">>, GroupNames),
-    true = lists:member(<<"remote group">>, GroupNames),
-    true = lists:member(<<"remote group join_remote">>, GroupNames),
-    %% kill & unregister processes
-    syn_test_suite_helper:kill_process(LocalPid),
-    ok = syn:leave(<<"remote group">>, RemotePid),
-    syn_test_suite_helper:kill_process(RemotePidRegRemote),
-    timer:sleep(100),
-    %% names
-    [] = syn:get_group_names().
-
-two_nodes_publish(Config) ->
-    GroupName = "my group",
-    Message = {test, message},
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    ResultPid = self(),
-    F = fun() ->
-        receive
-            Message -> ResultPid ! {received, self(), Message}
-        end
-    end,
-    LocalPid = syn_test_suite_helper:start_process(F),
-    LocalPid2 = syn_test_suite_helper:start_process(F),
-    RemotePid = syn_test_suite_helper:start_process(SlaveNode, F),
-    RemotePid2 = syn_test_suite_helper:start_process(SlaveNode, F),
-    OtherPid = syn_test_suite_helper:start_process(F),
-    %% join
-    ok = syn:join(GroupName, LocalPid),
-    ok = syn:join(GroupName, LocalPid2),
-    ok = syn:join(GroupName, RemotePid),
-    ok = syn:join(GroupName, RemotePid2),
-    timer:sleep(200),
-    %% send
-    {ok, 4} = syn:publish(GroupName, Message),
-    %% check
-    receive
-        {received, LocalPid, Message} -> ok
-    after 2000 ->
-        ok = published_message_was_not_received_by_local_pid
-    end,
-    receive
-        {received, LocalPid2, Message} -> ok
-    after 2000 ->
-        ok = published_message_was_not_received_by_local_pid_2
-    end,
-    receive
-        {received, RemotePid, Message} -> ok
-    after 2000 ->
-        ok = published_message_was_not_received_by_remote_pid
-    end,
-    receive
-        {received, RemotePid2, Message} -> ok
-    after 2000 ->
-        ok = published_message_was_not_received_by_remote_pid_2
-    end,
-    receive
-        {received, OtherPid, Message} ->
-            ok = published_message_was_received_by_other_pid
-    after 250 ->
-        ok
-    end.
-
-two_nodes_local_publish(Config) ->
-    GroupName = "my group",
-    Message = {test, message},
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    ResultPid = self(),
-    F = fun() ->
-        receive
-            Message -> ResultPid ! {received, self(), Message}
-        end
-    end,
-    LocalPid = syn_test_suite_helper:start_process(F),
-    LocalPid2 = syn_test_suite_helper:start_process(F),
-    RemotePid = syn_test_suite_helper:start_process(SlaveNode, F),
-    RemotePid2 = syn_test_suite_helper:start_process(SlaveNode, F),
-    OtherPid = syn_test_suite_helper:start_process(F),
-    %% join
-    ok = syn:join(GroupName, LocalPid),
-    ok = syn:join(GroupName, LocalPid2),
-    ok = syn:join(GroupName, RemotePid),
-    ok = syn:join(GroupName, RemotePid2),
-    timer:sleep(200),
-    %% send
-    {ok, 2} = syn:publish_to_local(GroupName, Message),
-    %% check
-    receive
-        {received, LocalPid, Message} -> ok
-    after 2000 ->
-        ok = published_message_was_not_received_by_local_pid
-    end,
-    receive
-        {received, LocalPid2, Message} -> ok
-    after 2000 ->
-        ok = published_message_was_not_received_by_local_pid_2
-    end,
-    receive
-        {received, RemotePid, Message} ->
-            ok = published_message_was_received_by_remote_pid
-    after 250 ->
-        ok
-    end,
-    receive
-        {received, RemotePid, Message} ->
-            ok = published_message_was_received_by_remote_pid_2
-    after 250 ->
-        ok
-    end,
-    receive
-        {received, OtherPid, Message} ->
-            ok = published_message_was_received_by_other_pid
-    after 250 ->
-        ok
-    end.
-
-two_nodes_multicall(Config) ->
-    GroupName = <<"my group">>,
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    F = fun() ->
-        receive
-            {syn_multi_call, RequestorPid, get_pid_name} ->
-                syn:multi_call_reply(RequestorPid, {pong, self()})
-        end
-    end,
-    Pid1 = syn_test_suite_helper:start_process(F),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode, F),
-    PidUnresponsive = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:join(GroupName, Pid1),
-    ok = syn:join(GroupName, Pid2),
-    ok = syn:join(GroupName, PidUnresponsive),
-    timer:sleep(500),
-    %% call
-    {Replies, BadPids} = syn:multi_call(GroupName, get_pid_name),
-    %% check responses
-    true = lists:sort([
-        {Pid1, {pong, Pid1}},
-        {Pid2, {pong, Pid2}}
-    ]) =:= lists:sort(Replies),
-    [PidUnresponsive] = BadPids.
-
-two_nodes_groups_full_cluster_sync_on_boot_node_added_later(_Config) ->
-    %% stop slave
-    syn_test_suite_helper:stop_slave(syn_slave),
-    %% start syn on local node
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:join(<<"group">>, Pid),
-    %% start remote node and syn
-    {ok, SlaveNode} = syn_test_suite_helper:start_slave(syn_slave),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(1000),
-    %% check
-    [Pid] = syn:get_members(<<"group">>),
-    [Pid] = rpc:call(SlaveNode, syn, get_members, [<<"group">>]).
-
-two_nodes_groups_full_cluster_sync_on_boot_syn_started_later(Config) ->
-    %% get slaves
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start syn on local node
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:join(<<"group">>, Pid),
-    %% start ib remote syn
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(500),
-    %% check
-    [Pid] = syn:get_members(<<"group">>),
-    [Pid] = rpc:call(SlaveNode, syn, get_members, [<<"group">>]).
-
-three_nodes_partial_netsplit_consistency(Config) ->
-    GroupName = "my group",
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid0Changed = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    OtherPid = syn_test_suite_helper:start_process(),
-    timer:sleep(100),
-    %% retrieve local
-    [] = syn:get_members("group-1"),
-    [] = syn:get_members(GroupName),
-    [] = syn:get_members(GroupName, with_meta),
-    false = syn:member(Pid0, GroupName),
-    false = syn:member(Pid0Changed, GroupName),
-    false = syn:member(Pid1, GroupName),
-    false = syn:member(Pid2, GroupName),
-    %% retrieve slave 1
-    [] = rpc:call(SlaveNode1, syn, get_members, [GroupName]),
-    [] = rpc:call(SlaveNode1, syn, get_members, [GroupName, with_meta]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid0, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid0Changed, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid1, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid2, GroupName]),
-    %% retrieve slave 2
-    [] = rpc:call(SlaveNode2, syn, get_members, [GroupName]),
-    [] = rpc:call(SlaveNode2, syn, get_members, [GroupName, with_meta]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid0, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid0Changed, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid1, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid2, GroupName]),
-    %% join
-    ok = syn:join(GroupName, Pid0),
-    ok = syn:join(GroupName, Pid0Changed, {meta, changed}),
-    ok = rpc:call(SlaveNode1, syn, join, [GroupName, Pid1]),
-    ok = rpc:call(SlaveNode2, syn, join, [GroupName, Pid2, {meta, 2}]),
-    ok = syn:join("other-group", OtherPid),
-    timer:sleep(200),
-    %% retrieve local
-    true = lists:sort([Pid0, Pid0Changed, Pid1, Pid2]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid0Changed, {meta, changed}},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    true = syn:member(Pid0, GroupName),
-    true = syn:member(Pid0Changed, GroupName),
-    true = syn:member(Pid1, GroupName),
-    true = syn:member(Pid2, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    %% retrieve slave 1
-    true = lists:sort([Pid0, Pid0Changed, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid0Changed, {meta, changed}},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode1, syn, member, [Pid0, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [OtherPid, GroupName]),
-    %% retrieve slave 2
-    true = lists:sort([Pid0, Pid0Changed, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode2, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid0Changed, {meta, changed}},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode2, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode2, syn, member, [Pid0, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [OtherPid, GroupName]),
-    %% disconnect slave 2 from main (slave 1 can still see slave 2)
-    syn_test_suite_helper:disconnect_node(SlaveNode2),
-    timer:sleep(500),
-    %% retrieve local
-    true = lists:sort([Pid0, Pid0Changed, Pid1]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid0Changed, {meta, changed}},
-        {Pid1, undefined}
-    ]) =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    true = syn:member(Pid0, GroupName),
-    true = syn:member(Pid0Changed, GroupName),
-    true = syn:member(Pid1, GroupName),
-    false = syn:member(Pid2, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    %% retrieve slave 1
-    true = lists:sort([Pid0, Pid0Changed, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid0Changed, {meta, changed}},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode1, syn, member, [Pid0, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [OtherPid, GroupName]),
-    %% disconnect slave 1
-    syn_test_suite_helper:disconnect_node(SlaveNode1),
-    timer:sleep(500),
-    %% leave 0Changed
-    ok = syn:leave(GroupName, Pid0Changed),
-    %% retrieve local
-    true = lists:sort([Pid0]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([
-        {Pid0, undefined}
-    ]) =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    true = syn:member(Pid0, GroupName),
-    false = syn:member(Pid0Changed, GroupName),
-    false = syn:member(Pid1, GroupName),
-    false = syn:member(Pid2, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    %% reconnect all
-    syn_test_suite_helper:connect_node(SlaveNode1),
-    syn_test_suite_helper:connect_node(SlaveNode2),
-    timer:sleep(5000),
-    %% retrieve local
-    true = lists:sort([Pid0, Pid1, Pid2]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    true = syn:member(Pid0, GroupName),
-    false = syn:member(Pid0Changed, GroupName),
-    true = syn:member(Pid1, GroupName),
-    true = syn:member(Pid2, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    %% retrieve slave 1
-    true = lists:sort([Pid0, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode1, syn, member, [Pid0, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [OtherPid, GroupName]),
-    %% retrieve slave 2
-    true = lists:sort([Pid0, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode2, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode2, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode2, syn, member, [Pid0, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [OtherPid, GroupName]).
-
-three_nodes_full_netsplit_consistency(Config) ->
-    GroupName = "my group",
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid0Changed = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    OtherPid = syn_test_suite_helper:start_process(),
-    timer:sleep(100),
-    %% retrieve local
-    [] = syn:get_members("group-1"),
-    [] = syn:get_members(GroupName),
-    [] = syn:get_members(GroupName, with_meta),
-    false = syn:member(Pid0, GroupName),
-    false = syn:member(Pid0Changed, GroupName),
-    false = syn:member(Pid1, GroupName),
-    false = syn:member(Pid2, GroupName),
-    %% retrieve slave 1
-    [] = rpc:call(SlaveNode1, syn, get_members, [GroupName]),
-    [] = rpc:call(SlaveNode1, syn, get_members, [GroupName, with_meta]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid0, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid0Changed, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid1, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid2, GroupName]),
-    %% retrieve slave 2
-    [] = rpc:call(SlaveNode2, syn, get_members, [GroupName]),
-    [] = rpc:call(SlaveNode2, syn, get_members, [GroupName, with_meta]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid0, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid0Changed, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid1, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid2, GroupName]),
-    %% join
-    ok = syn:join(GroupName, Pid0),
-    ok = syn:join(GroupName, Pid0Changed, {meta, changed}),
-    ok = rpc:call(SlaveNode1, syn, join, [GroupName, Pid1]),
-    ok = rpc:call(SlaveNode2, syn, join, [GroupName, Pid2, {meta, 2}]),
-    ok = syn:join("other-group", OtherPid),
-    timer:sleep(200),
-    %% retrieve local
-    true = lists:sort([Pid0, Pid0Changed, Pid1, Pid2]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid0Changed, {meta, changed}},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    true = syn:member(Pid0, GroupName),
-    true = syn:member(Pid0Changed, GroupName),
-    true = syn:member(Pid1, GroupName),
-    true = syn:member(Pid2, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    %% retrieve slave 1
-    true = lists:sort([Pid0, Pid0Changed, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid0Changed, {meta, changed}},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode1, syn, member, [Pid0, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [OtherPid, GroupName]),
-    %% retrieve slave 2
-    true = lists:sort([Pid0, Pid0Changed, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode2, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid0Changed, {meta, changed}},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode2, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode2, syn, member, [Pid0, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [OtherPid, GroupName]),
-    %% disconnect everyone
-    rpc:call(SlaveNode1, syn_test_suite_helper, disconnect_node, [SlaveNode2]),
-    syn_test_suite_helper:disconnect_node(SlaveNode1),
-    syn_test_suite_helper:disconnect_node(SlaveNode2),
-    timer:sleep(2000),
-    %% leave 0Changed
-    ok = syn:leave(GroupName, Pid0Changed),
-    timer:sleep(500),
-    %% retrieve local
-    true = lists:sort([Pid0]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([
-        {Pid0, undefined}
-    ]) =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    true = syn:member(Pid0, GroupName),
-    false = syn:member(Pid0Changed, GroupName),
-    false = syn:member(Pid1, GroupName),
-    false = syn:member(Pid2, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    %% reconnect all
-    syn_test_suite_helper:connect_node(SlaveNode1),
-    syn_test_suite_helper:connect_node(SlaveNode2),
-    rpc:call(SlaveNode1, syn_test_suite_helper, connect_node, [SlaveNode2]),
-    timer:sleep(1500),
-    %% retrieve local
-    true = lists:sort([Pid0, Pid1, Pid2]) =:= lists:sort(syn:get_members(GroupName)),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(syn:get_members(GroupName, with_meta)),
-    true = syn:member(Pid0, GroupName),
-    false = syn:member(Pid0Changed, GroupName),
-    true = syn:member(Pid1, GroupName),
-    true = syn:member(Pid2, GroupName),
-    false = syn:member(OtherPid, GroupName),
-    %% retrieve slave 1
-    true = lists:sort([Pid0, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode1, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode1, syn, member, [Pid0, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode1, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode1, syn, member, [OtherPid, GroupName]),
-    %% retrieve slave 2
-    true = lists:sort([Pid0, Pid1, Pid2])
-        =:= lists:sort(rpc:call(SlaveNode2, syn, get_members, [GroupName])),
-    true = lists:sort([
-        {Pid0, undefined},
-        {Pid1, undefined},
-        {Pid2, {meta, 2}}
-    ]) =:= lists:sort(rpc:call(SlaveNode2, syn, get_members, [GroupName, with_meta])),
-    true = rpc:call(SlaveNode2, syn, member, [Pid0, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [Pid0Changed, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid1, GroupName]),
-    true = rpc:call(SlaveNode2, syn, member, [Pid2, GroupName]),
-    false = rpc:call(SlaveNode2, syn, member, [OtherPid, GroupName]).
-
-three_nodes_anti_entropy(Config) ->
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% set anti-entropy with a very low interval (0.25 second)
-    syn_test_suite_helper:use_anti_entropy(groups, 0.25),
-    rpc:call(SlaveNode1, syn_test_suite_helper, use_anti_entropy, [groups, 0.25]),
-    rpc:call(SlaveNode2, syn_test_suite_helper, use_anti_entropy, [groups, 0.25]),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    Pid2Isolated = syn_test_suite_helper:start_process(SlaveNode2),
-    timer:sleep(100),
-    %% inject data to simulate latent conflicts
-    ok = syn_groups:add_to_local_table("my-group", Pid0, node(), undefined),
-    ok = rpc:call(SlaveNode1, syn_groups, add_to_local_table, ["my-group", Pid1, SlaveNode1, undefined]),
-    ok = rpc:call(SlaveNode2, syn_groups, add_to_local_table, ["my-group", Pid2, SlaveNode2, undefined]),
-    ok = rpc:call(SlaveNode2, syn_groups, add_to_local_table, ["my-group-isolated", Pid2Isolated, SlaveNode2, undefined]),
-    timer:sleep(5000),
-    %% check
-    Members = lists:sort([
-        {Pid0, node()},
-        {Pid1, SlaveNode1},
-        {Pid2, SlaveNode2}
-    ]),
-    Members = syn:get_members("my-group", with_meta),
-    Members = rpc:call(SlaveNode1, syn, get_members, ["my-group", with_meta]),
-    Members = rpc:call(SlaveNode2, syn, get_members, ["my-group", with_meta]),
-    [{Pid2Isolated, SlaveNode2}] = syn:get_members("my-group-isolated", with_meta),
-    [{Pid2Isolated, SlaveNode2}] = rpc:call(SlaveNode1, syn, get_members, ["my-group-isolated", with_meta]),
-    [{Pid2Isolated, SlaveNode2}] = rpc:call(SlaveNode2, syn, get_members, ["my-group-isolated", with_meta]).
-
-three_nodes_anti_entropy_manual(Config) ->
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    Pid2Isolated = syn_test_suite_helper:start_process(SlaveNode2),
-    timer:sleep(100),
-    %% inject data to simulate latent conflicts
-    ok = syn_groups:add_to_local_table("my-group", Pid0, node(), undefined),
-    ok = rpc:call(SlaveNode1, syn_groups, add_to_local_table, ["my-group", Pid1, SlaveNode1, undefined]),
-    ok = rpc:call(SlaveNode2, syn_groups, add_to_local_table, ["my-group", Pid2, SlaveNode2, undefined]),
-    ok = rpc:call(SlaveNode2, syn_groups, add_to_local_table, ["my-group-isolated", Pid2Isolated, SlaveNode2, undefined]),
-    %% call anti entropy
-    ok = syn:force_cluster_sync(groups),
-    timer:sleep(5000),
-    %% check
-    Members = lists:sort([
-        {Pid0, node()},
-        {Pid1, SlaveNode1},
-        {Pid2, SlaveNode2}
-    ]),
-    Members = syn:get_members("my-group", with_meta),
-    Members = rpc:call(SlaveNode1, syn, get_members, ["my-group", with_meta]),
-    Members = rpc:call(SlaveNode2, syn, get_members, ["my-group", with_meta]),
-    [{Pid2Isolated, SlaveNode2}] = syn:get_members("my-group-isolated", with_meta),
-    [{Pid2Isolated, SlaveNode2}] = rpc:call(SlaveNode1, syn, get_members, ["my-group-isolated", with_meta]),
-    [{Pid2Isolated, SlaveNode2}] = rpc:call(SlaveNode2, syn, get_members, ["my-group-isolated", with_meta]).

test/syn_registry_SUITE.erl

@@ -1,1242 +0,0 @@
-%% ==========================================================================================================
-%% Syn - A global Process Registry and Process Group manager.
-%%
-%% The MIT License (MIT)
-%%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
-%%
-%% Permission is hereby granted, free of charge, to any person obtaining a copy
-%% of this software and associated documentation files (the "Software"), to deal
-%% in the Software without restriction, including without limitation the rights
-%% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-%% copies of the Software, and to permit persons to whom the Software is
-%% furnished to do so, subject to the following conditions:
-%%
-%% The above copyright notice and this permission notice shall be included in
-%% all copies or substantial portions of the Software.
-%%
-%% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-%% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-%% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-%% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-%% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-%% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-%% THE SOFTWARE.
-%% ==========================================================================================================
--module(syn_registry_SUITE).
-
-%% callbacks
--export([all/0]).
--export([init_per_suite/1, end_per_suite/1]).
--export([groups/0, init_per_group/2, end_per_group/2]).
--export([init_per_testcase/2, end_per_testcase/2]).
-
-%% tests
--export([
-    single_node_register_and_monitor/1,
-    single_node_register_and_unregister/1,
-    single_node_register_unregister_and_kill_process/1,
-    single_node_registration_errors/1,
-    single_node_registry_count/1,
-    single_node_register_gen_server/1,
-    single_node_callback_on_process_exit/1,
-    single_node_ensure_callback_process_exit_is_called_if_process_killed/1,
-    single_node_monitor_after_registry_crash/1
-]).
--export([
-    two_nodes_register_monitor_and_unregister/1,
-    two_nodes_registry_count/1,
-    two_nodes_registration_race_condition_conflict_resolution_keep_more_recent_remote/1,
-    two_nodes_registration_race_condition_conflict_resolution_keep_more_recent_local/1,
-    two_nodes_registration_race_condition_conflict_resolution_keep_remote_with_custom_handler/1,
-    two_nodes_registration_race_condition_conflict_resolution_keep_local_with_custom_handler/1,
-    two_nodes_registration_race_condition_conflict_resolution_when_process_died/1,
-    two_nodes_registry_full_cluster_sync_on_boot_node_added_later/1,
-    two_nodes_registry_full_cluster_sync_on_boot_syn_started_later/1,
-    two_nodes_unregister_and_register/1
-]).
--export([
-    three_nodes_partial_netsplit_consistency/1,
-    three_nodes_full_netsplit_consistency/1,
-    three_nodes_start_syn_before_connecting_cluster_with_conflict_keep_more_recent/1,
-    three_nodes_start_syn_before_connecting_cluster_with_custom_conflict_resolution_keep_remote/1,
-    three_nodes_registration_race_condition_custom_conflict_resolution/1,
-    three_nodes_anti_entropy/1,
-    three_nodes_anti_entropy_manual/1,
-    three_nodes_concurrent_registration_unregistration/1,
-    three_nodes_resolve_conflict_on_all_nodes/1
-]).
-
-%% support
--export([
-    start_syn_delayed_and_register_local_process/3,
-    start_syn_delayed_with_custom_handler_register_local_process/4,
-    seq_unregister_register/3
-]).
-
-%% include
--include_lib("common_test/include/ct.hrl").
--include_lib("../src/syn.hrl").
-
-%% ===================================================================
-%% Callbacks
-%% ===================================================================
-
-%% -------------------------------------------------------------------
-%% Function: all() -> GroupsAndTestCases | {skip,Reason}
-%% GroupsAndTestCases = [{group,GroupName} | TestCase]
-%% GroupName = atom()
-%% TestCase = atom()
-%% Reason = any()
-%% -------------------------------------------------------------------
-all() ->
-    [
-        {group, single_node_process_registration},
-        {group, two_nodes_process_registration},
-        {group, three_nodes_process_registration}
-    ].
-
-%% -------------------------------------------------------------------
-%% Function: groups() -> [Group]
-%% Group = {GroupName,Properties,GroupsAndTestCases}
-%% GroupName =  atom()
-%% Properties = [parallel | sequence | Shuffle | {RepeatType,N}]
-%% GroupsAndTestCases = [Group | {group,GroupName} | TestCase]
-%% TestCase = atom()
-%% Shuffle = shuffle | {shuffle,{integer(),integer(),integer()}}
-%% RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail |
-%%			   repeat_until_any_ok | repeat_until_any_fail
-%% N = integer() | forever
-%% -------------------------------------------------------------------
-groups() ->
-    [
-        {single_node_process_registration, [shuffle], [
-            single_node_register_and_monitor,
-            single_node_register_and_unregister,
-            single_node_register_unregister_and_kill_process,
-            single_node_registration_errors,
-            single_node_registry_count,
-            single_node_register_gen_server,
-            single_node_callback_on_process_exit,
-            single_node_ensure_callback_process_exit_is_called_if_process_killed,
-            single_node_monitor_after_registry_crash
-        ]},
-        {two_nodes_process_registration, [shuffle], [
-            two_nodes_register_monitor_and_unregister,
-            two_nodes_registry_count,
-            two_nodes_registration_race_condition_conflict_resolution_keep_more_recent_remote,
-            two_nodes_registration_race_condition_conflict_resolution_keep_more_recent_local,
-            two_nodes_registration_race_condition_conflict_resolution_keep_remote_with_custom_handler,
-            two_nodes_registration_race_condition_conflict_resolution_keep_local_with_custom_handler,
-            two_nodes_registration_race_condition_conflict_resolution_when_process_died,
-            two_nodes_registry_full_cluster_sync_on_boot_node_added_later,
-            two_nodes_registry_full_cluster_sync_on_boot_syn_started_later,
-            two_nodes_unregister_and_register
-        ]},
-        {three_nodes_process_registration, [shuffle], [
-            three_nodes_partial_netsplit_consistency,
-            three_nodes_full_netsplit_consistency,
-            three_nodes_start_syn_before_connecting_cluster_with_conflict_keep_more_recent,
-            three_nodes_start_syn_before_connecting_cluster_with_custom_conflict_resolution_keep_remote,
-            three_nodes_registration_race_condition_custom_conflict_resolution,
-            three_nodes_anti_entropy,
-            three_nodes_anti_entropy_manual,
-            three_nodes_concurrent_registration_unregistration,
-            three_nodes_resolve_conflict_on_all_nodes
-        ]}
-    ].
-%% -------------------------------------------------------------------
-%% Function: init_per_suite(Config0) ->
-%%				Config1 | {skip,Reason} |
-%%              {skip_and_save,Reason,Config1}
-%% Config0 = Config1 = [tuple()]
-%% Reason = any()
-%% -------------------------------------------------------------------
-init_per_suite(Config) ->
-    Config.
-
-%% -------------------------------------------------------------------
-%% Function: end_per_suite(Config0) -> void() | {save_config,Config1}
-%% Config0 = Config1 = [tuple()]
-%% -------------------------------------------------------------------
-end_per_suite(_Config) ->
-    ok.
-
-%% -------------------------------------------------------------------
-%% Function: init_per_group(GroupName, Config0) ->
-%%				Config1 | {skip,Reason} |
-%%              {skip_and_save,Reason,Config1}
-%% GroupName = atom()
-%% Config0 = Config1 = [tuple()]
-%% Reason = any()
-%% -------------------------------------------------------------------
-init_per_group(two_nodes_process_registration, Config) ->
-    %% start slave
-    {ok, SlaveNode} = syn_test_suite_helper:start_slave(syn_slave),
-    %% config
-    [{slave_node, SlaveNode} | Config];
-init_per_group(three_nodes_process_registration, Config) ->
-    %% start slave
-    {ok, SlaveNode1} = syn_test_suite_helper:start_slave(syn_slave_1),
-    {ok, SlaveNode2} = syn_test_suite_helper:start_slave(syn_slave_2),
-    %% config
-    [{slave_node_1, SlaveNode1}, {slave_node_2, SlaveNode2} | Config];
-init_per_group(_GroupName, Config) ->
-    Config.
-
-%% -------------------------------------------------------------------
-%% Function: end_per_group(GroupName, Config0) ->
-%%				void() | {save_config,Config1}
-%% GroupName = atom()
-%% Config0 = Config1 = [tuple()]
-%% -------------------------------------------------------------------
-end_per_group(two_nodes_process_registration, Config) ->
-    SlaveNode = proplists:get_value(slave_node, Config),
-    syn_test_suite_helper:connect_node(SlaveNode),
-    syn_test_suite_helper:clean_after_test(),
-    syn_test_suite_helper:stop_slave(syn_slave),
-    timer:sleep(1000);
-end_per_group(three_nodes_process_registration, Config) ->
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    syn_test_suite_helper:connect_node(SlaveNode1),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    syn_test_suite_helper:connect_node(SlaveNode2),
-    syn_test_suite_helper:clean_after_test(),
-    syn_test_suite_helper:stop_slave(syn_slave_1),
-    syn_test_suite_helper:stop_slave(syn_slave_2),
-    timer:sleep(1000);
-end_per_group(_GroupName, _Config) ->
-    syn_test_suite_helper:clean_after_test().
-
-%% -------------------------------------------------------------------
-%% Function: init_per_testcase(TestCase, Config0) ->
-%%				Config1 | {skip,Reason} | {skip_and_save,Reason,Config1}
-%% TestCase = atom()
-%% Config0 = Config1 = [tuple()]
-%% Reason = any()
-%% -------------------------------------------------------------------
-init_per_testcase(TestCase, Config) ->
-    ct:pal("Starting test: ~p", [TestCase]),
-    Config.
-
-%% -------------------------------------------------------------------
-%% Function: end_per_testcase(TestCase, Config0) ->
-%%				void() | {save_config,Config1} | {fail,Reason}
-%% TestCase = atom()
-%% Config0 = Config1 = [tuple()]
-%% Reason = any()
-%% -------------------------------------------------------------------
-end_per_testcase(_, _Config) ->
-    syn_test_suite_helper:clean_after_test().
-
-%% ===================================================================
-%% Tests
-%% ===================================================================
-single_node_register_and_monitor(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    PidWithMeta = syn_test_suite_helper:start_process(),
-    %% retrieve
-    undefined = syn:whereis(<<"my proc">>),
-    %% register
-    ok = syn:register(<<"my proc">>, Pid),
-    ok = syn:register({"my proc 2"}, Pid),
-    ok = syn:register(<<"my proc with meta">>, PidWithMeta, {meta, <<"meta">>}),
-    %% retrieve
-    Pid = syn:whereis(<<"my proc">>),
-    Pid = syn:whereis({"my proc 2"}),
-    {PidWithMeta, {meta, <<"meta">>}} = syn:whereis(<<"my proc with meta">>, with_meta),
-    %% re-register
-    ok = syn:register(<<"my proc with meta">>, PidWithMeta, {meta2, <<"meta2">>}),
-    {PidWithMeta, {meta2, <<"meta2">>}} = syn:whereis(<<"my proc with meta">>, with_meta),
-    %% kill process
-    syn_test_suite_helper:kill_process(Pid),
-    syn_test_suite_helper:kill_process(PidWithMeta),
-    timer:sleep(100),
-    %% retrieve
-    undefined = syn:whereis(<<"my proc">>),
-    undefined = syn:whereis({"my proc 2"}),
-    undefined = syn:whereis(<<"my proc with meta">>).
-
-single_node_register_and_unregister(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    %% retrieve
-    undefined = syn:whereis(<<"my proc">>),
-    %% register
-    ok = syn:register(<<"my proc">>, Pid),
-    ok = syn:register(<<"my proc 2">>, Pid),
-    %% retrieve
-    Pid = syn:whereis(<<"my proc">>),
-    Pid = syn:whereis(<<"my proc 2">>),
-    %% unregister 1
-    ok = syn:unregister(<<"my proc">>),
-    %% retrieve
-    undefined = syn:whereis(<<"my proc">>),
-    Pid = syn:whereis(<<"my proc 2">>),
-    %% unregister 2
-    ok = syn:unregister(<<"my proc 2">>),
-    {error, undefined} = syn:unregister(<<"my proc 2">>),
-    %% retrieve
-    undefined = syn:whereis(<<"my proc">>),
-    undefined = syn:whereis(<<"my proc 2">>).
-
-single_node_register_unregister_and_kill_process(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:register(<<"synonym_to_unregister">>, Pid),
-    ok = syn:register(<<"synonym_to_keep">>, Pid),
-    %% unregister
-    ok = syn:unregister(<<"synonym_to_unregister">>),
-    %% retrieve
-    undefined = syn:whereis(<<"synonym_to_unregister">>),
-    Pid = syn:whereis(<<"synonym_to_keep">>),
-    %% kill process
-    true = syn_test_suite_helper:kill_process(Pid),
-    timer:sleep(100),
-    %% retrieve
-    undefined = syn:whereis(<<"synonym_to_keep">>).
-
-single_node_registration_errors(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    Pid2 = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:register(<<"my proc">>, Pid),
-    {error, taken} = syn:register(<<"my proc">>, Pid2),
-    %% kill processes
-    syn_test_suite_helper:kill_process(Pid),
-    syn_test_suite_helper:kill_process(Pid2),
-    timer:sleep(100),
-    %% retrieve
-    undefined = syn:whereis(<<"my proc">>),
-    %% try registering a dead pid
-    {error, not_alive} = syn:register(<<"my proc">>, Pid).
-
-single_node_registry_count(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    Pid2 = syn_test_suite_helper:start_process(),
-    PidUnregistered = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:register(<<"my proc">>, Pid),
-    ok = syn:register(<<"my proc 2">>, Pid2),
-    %% count
-    2 = syn:registry_count(),
-    2 = syn:registry_count(node()),
-    %% kill & unregister
-    syn_test_suite_helper:kill_process(Pid),
-    ok = syn:unregister(<<"my proc 2">>),
-    syn_test_suite_helper:kill_process(PidUnregistered),
-    timer:sleep(100),
-    %% count
-    0 = syn:registry_count(),
-    0 = syn:registry_count(node()).
-
-single_node_register_gen_server(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start gen server via syn
-    {ok, Pid} = syn_test_gen_server:start_link(),
-    %% retrieve
-    Pid = syn:whereis(syn_test_gen_server),
-    %% call
-    pong = syn_test_gen_server:ping(),
-    %% send via syn
-    syn:send(syn_test_gen_server, {self(), send_ping}),
-    receive
-        send_pong -> ok
-    after 1000 ->
-        ok = did_not_receive_gen_server_pong
-    end,
-    %% stop server
-    syn_test_gen_server:stop(),
-    timer:sleep(200),
-    %% retrieve
-    undefined = syn:whereis(syn_test_gen_server),
-    %% send via syn
-    {badarg, {syn_test_gen_server, anything}} = (catch syn:send(syn_test_gen_server, anything)).
-
-single_node_callback_on_process_exit(_Config) ->
-    %% use custom handler
-    syn_test_suite_helper:use_custom_handler(),
-    %% start
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    Pid2 = syn_test_suite_helper:start_process(),
-    %% register
-    TestPid = self(),
-    ok = syn:register(<<"my proc">>, Pid, {pid, TestPid}),
-    ok = syn:register(<<"my proc - alternate">>, Pid, {pid_alternate, TestPid}),
-    ok = syn:register(<<"my proc 2">>, Pid2, {pid2, TestPid}),
-    %% kill 1
-    syn_test_suite_helper:kill_process(Pid),
-    receive
-        {received_event_on, pid} ->
-            ok;
-        {received_event_on, pid2} ->
-            ok = callback_on_process_exit_was_received_by_pid2
-    after 1000 ->
-        ok = callback_on_process_exit_was_not_received_by_pid
-    end,
-    receive
-        {received_event_on, pid_alternate} ->
-            ok;
-        {received_event_on, pid2} ->
-            ok = callback_on_process_exit_was_received_by_pid2
-    after 1000 ->
-        ok = callback_on_process_exit_was_not_received_by_pid
-    end,
-    %% unregister & kill 2
-    ok = syn:unregister(<<"my proc 2">>),
-    syn_test_suite_helper:kill_process(Pid2),
-    receive
-        {received_event_on, pid2} ->
-            ok = callback_on_process_exit_was_received_by_pid2
-    after 1000 ->
-        ok
-    end.
-
-single_node_ensure_callback_process_exit_is_called_if_process_killed(_Config) ->
-    Name = <<"my proc">>,
-    %% use custom handler
-    syn_test_suite_helper:use_custom_handler(),
-    %% start
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    %% register
-    TestPid = self(),
-    ok = syn:register(Name, Pid, {some_meta, TestPid}),
-    %% remove from table to simulate conflict resolution
-    syn_registry:remove_from_local_table(Name, TestPid),
-    %% kill
-    exit(Pid, {syn_resolve_kill, Name, {some_meta, TestPid}}),
-    receive
-        {received_event_on, some_meta} ->
-            ok
-    after 1000 ->
-        ok = callback_on_process_exit_was_not_received_by_pid
-    end.
-
-single_node_monitor_after_registry_crash(_Config) ->
-    %% start
-    ok = syn:start(),
-    %% start processes
-    Pid = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:register(<<"my proc">>, Pid),
-    %% kill registry
-    exit(whereis(syn_registry), kill),
-    timer:sleep(200),
-    %% retrieve
-    Pid = syn:whereis(<<"my proc">>),
-    %% kill process
-    syn_test_suite_helper:kill_process(Pid),
-    timer:sleep(200),
-    %% retrieve
-    undefined = syn:whereis(<<"my proc 2">>).
-
-two_nodes_register_monitor_and_unregister(Config) ->
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    LocalPid = syn_test_suite_helper:start_process(),
-    RemotePid = syn_test_suite_helper:start_process(SlaveNode),
-    RemotePidRegRemote = syn_test_suite_helper:start_process(SlaveNode),
-    %% retrieve
-    undefined = syn:whereis(<<"local proc">>),
-    undefined = syn:whereis(<<"remote proc">>),
-    undefined = syn:whereis(<<"remote proc reg_remote">>),
-    undefined = rpc:call(SlaveNode, syn, whereis, [<<"local proc">>]),
-    undefined = rpc:call(SlaveNode, syn, whereis, [<<"remote proc">>]),
-    undefined = rpc:call(SlaveNode, syn, whereis, [<<"remote proc reg_remote">>]),
-    %% register
-    ok = syn:register(<<"local proc">>, LocalPid),
-    ok = syn:register(<<"remote proc">>, RemotePid),
-    ok = rpc:call(SlaveNode, syn, register, [<<"remote proc reg_remote">>, RemotePidRegRemote]),
-    timer:sleep(500),
-    %% retrieve
-    LocalPid = syn:whereis(<<"local proc">>),
-    RemotePid = syn:whereis(<<"remote proc">>),
-    RemotePidRegRemote = syn:whereis(<<"remote proc reg_remote">>),
-    LocalPid = rpc:call(SlaveNode, syn, whereis, [<<"local proc">>]),
-    RemotePid = rpc:call(SlaveNode, syn, whereis, [<<"remote proc">>]),
-    RemotePidRegRemote = rpc:call(SlaveNode, syn, whereis, [<<"remote proc reg_remote">>]),
-    %% kill & unregister processes
-    syn_test_suite_helper:kill_process(LocalPid),
-    ok = syn:unregister(<<"remote proc">>),
-    syn_test_suite_helper:kill_process(RemotePidRegRemote),
-    timer:sleep(100),
-    %% retrieve
-    undefined = syn:whereis(<<"local proc">>),
-    undefined = syn:whereis(<<"remote proc">>),
-    undefined = syn:whereis(<<"remote proc reg_remote">>),
-    undefined = rpc:call(SlaveNode, syn, whereis, [<<"local proc">>]),
-    undefined = rpc:call(SlaveNode, syn, whereis, [<<"remote proc">>]),
-    undefined = rpc:call(SlaveNode, syn, whereis, [<<"remote proc reg_remote">>]).
-
-two_nodes_registry_count(Config) ->
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    LocalPid = syn_test_suite_helper:start_process(),
-    RemotePid = syn_test_suite_helper:start_process(SlaveNode),
-    RemotePidRegRemote = syn_test_suite_helper:start_process(SlaveNode),
-    _PidUnregistered = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:register(<<"local proc">>, LocalPid),
-    ok = syn:register(<<"remote proc">>, RemotePid),
-    ok = rpc:call(SlaveNode, syn, register, [<<"remote proc reg_remote">>, RemotePidRegRemote]),
-    timer:sleep(500),
-    %% count
-    3 = syn:registry_count(),
-    1 = syn:registry_count(node()),
-    2 = syn:registry_count(SlaveNode),
-    %% kill & unregister processes
-    syn_test_suite_helper:kill_process(LocalPid),
-    ok = syn:unregister(<<"remote proc">>),
-    syn_test_suite_helper:kill_process(RemotePidRegRemote),
-    timer:sleep(100),
-    %% count
-    0 = syn:registry_count(),
-    0 = syn:registry_count(node()),
-    0 = syn:registry_count(SlaveNode).
-
-two_nodes_registration_race_condition_conflict_resolution_keep_more_recent_remote(Config) ->
-    ConflictingName = "COMMON",
-    %% get slaves
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(1000),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode),
-    %% inject into syn to simulate concurrent registration
-    ok = syn_registry:add_to_local_table(ConflictingName, Pid0, node(), erlang:system_time() - 1000000000, undefined),
-    %% register on slave node to trigger conflict resolution on master node
-    ok = rpc:call(SlaveNode, syn, register, [ConflictingName, Pid1, SlaveNode]),
-    timer:sleep(1000),
-    %% check metadata, resolution happens on master node
-    {Pid1, SlaveNode} = syn:whereis(ConflictingName, with_meta),
-    {Pid1, SlaveNode} = rpc:call(SlaveNode, syn, whereis, [ConflictingName, with_meta]),
-    %% check that other processes are not alive because syn killed them
-    false = is_process_alive(Pid0),
-    true = rpc:call(SlaveNode, erlang, is_process_alive, [Pid1]).
-
-two_nodes_registration_race_condition_conflict_resolution_keep_more_recent_local(Config) ->
-    ConflictingName = "COMMON",
-    %% get slaves
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(1000),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode),
-    %% inject into syn to simulate concurrent registration
-    ok = syn_registry:add_to_local_table(ConflictingName, Pid0, node(), erlang:system_time() + 1000000000, undefined),
-    %% register on slave node to trigger conflict resolution on master node
-    ok = rpc:call(SlaveNode, syn, register, [ConflictingName, Pid1, SlaveNode]),
-    timer:sleep(1000),
-    %% check metadata, resolution happens on master node
-    Node = node(),
-    {Pid0, Node} = syn:whereis(ConflictingName, with_meta),
-    {Pid0, Node} = rpc:call(SlaveNode, syn, whereis, [ConflictingName, with_meta]),
-    %% check that other processes are not alive because syn killed them
-    true = is_process_alive(Pid0),
-    false = rpc:call(SlaveNode, erlang, is_process_alive, [Pid1]).
-
-two_nodes_registration_race_condition_conflict_resolution_keep_remote_with_custom_handler(Config) ->
-    ConflictingName = "COMMON",
-    %% get slaves
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% use customer handler
-    syn_test_suite_helper:use_custom_handler(),
-    rpc:call(SlaveNode, syn_test_suite_helper, use_custom_handler, []),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(1000),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode),
-    %% register
-    ok = syn:register(ConflictingName, Pid0, node()),
-    %% trigger conflict resolution on master node with something less recent (which would be discarded without a custom handler)
-    ok = syn_registry:sync_register(node(), ConflictingName, Pid1, keep_this_one, erlang:system_time() - 1000000000, false),
-    timer:sleep(1000),
-    %% check metadata, resolution happens on master node
-    {Pid1, keep_this_one} = syn:whereis(ConflictingName, with_meta),
-    %% check that other processes are not alive because syn killed them
-    true = is_process_alive(Pid0),
-    true = rpc:call(SlaveNode, erlang, is_process_alive, [Pid1]),
-    %% check that discarded process is not monitored
-    {monitored_by, []} = erlang:process_info(Pid0, monitored_by).
-
-two_nodes_registration_race_condition_conflict_resolution_keep_local_with_custom_handler(Config) ->
-    ConflictingName = "COMMON",
-    %% get slaves
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% use customer handler
-    syn_test_suite_helper:use_custom_handler(),
-    rpc:call(SlaveNode, syn_test_suite_helper, use_custom_handler, []),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(1000),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode),
-    %% inject into syn to simulate concurrent registration with something more recent (which would be picked without a custom handler)
-    ok = syn_registry:add_to_local_table(ConflictingName, Pid0, keep_this_one, undefined, erlang:system_time() + 1000000000),
-    %% register on slave node to trigger conflict resolution on master node
-    ok = rpc:call(SlaveNode, syn, register, [ConflictingName, Pid1, SlaveNode]),
-    timer:sleep(1000),
-    %% check metadata, resolution happens on master node
-    {Pid0, keep_this_one} = syn:whereis(ConflictingName, with_meta),
-    {Pid0, keep_this_one} = rpc:call(SlaveNode, syn, whereis, [ConflictingName, with_meta]),
-    %% check that other processes are not alive because syn killed them
-    true = is_process_alive(Pid0),
-    true = rpc:call(SlaveNode, erlang, is_process_alive, [Pid1]),
-    %% check that discarded process is not monitored
-    {monitored_by, []} = rpc:call(SlaveNode, erlang, process_info, [Pid1, monitored_by]).
-
-two_nodes_registration_race_condition_conflict_resolution_when_process_died(Config) ->
-    ConflictingName = "COMMON",
-    %% get slaves
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% use customer handler
-    syn_test_suite_helper:use_custom_handler(),
-    rpc:call(SlaveNode, syn_test_suite_helper, use_custom_handler, []),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(1000),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode),
-    %% inject into syn to simulate concurrent registration
-    syn_registry:add_to_local_table(ConflictingName, Pid0, keep_this_one, 0, undefined),
-    timer:sleep(250),
-    %% kill process
-    syn_test_suite_helper:kill_process(Pid0),
-    %% register to trigger conflict resolution
-    timer:sleep(250),
-    ok = rpc:call(SlaveNode, syn, register, [ConflictingName, Pid1, SlaveNode]),
-    timer:sleep(250),
-    %% check
-    {Pid1, SlaveNode} = syn:whereis(ConflictingName, with_meta),
-    {Pid1, SlaveNode} = rpc:call(SlaveNode, syn, whereis, [ConflictingName, with_meta]),
-    %% check that process is alive
-    true = rpc:call(SlaveNode, erlang, is_process_alive, [Pid1]).
-
-two_nodes_registry_full_cluster_sync_on_boot_node_added_later(_Config) ->
-    %% stop slave
-    syn_test_suite_helper:stop_slave(syn_slave),
-    %% start syn on local node
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:register(<<"proc">>, Pid),
-    %% start remote node and syn
-    {ok, SlaveNode} = syn_test_suite_helper:start_slave(syn_slave),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(1000),
-    %% check
-    Pid = syn:whereis(<<"proc">>),
-    Pid = rpc:call(SlaveNode, syn, whereis, [<<"proc">>]).
-
-two_nodes_registry_full_cluster_sync_on_boot_syn_started_later(Config) ->
-    %% get slaves
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% start syn on local node
-    ok = syn:start(),
-    %% start process
-    Pid = syn_test_suite_helper:start_process(),
-    %% register
-    ok = syn:register(<<"proc">>, Pid),
-    %% start ib remote syn
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(500),
-    %% check
-    Pid = syn:whereis(<<"proc">>),
-    Pid = rpc:call(SlaveNode, syn, whereis, [<<"proc">>]).
-
-two_nodes_unregister_and_register(Config) ->
-    Name = "common name",
-    %% get slave
-    SlaveNode = proplists:get_value(slave_node, Config),
-    %% use custom handler
-    syn_test_suite_helper:use_custom_handler(),
-    rpc:call(SlaveNode, syn_test_suite_helper, use_custom_handler, []),
-    %% start
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    PidLocal = syn_test_suite_helper:start_process(),
-    PidLocal2 = syn_test_suite_helper:start_process(),
-    PidRemote = syn_test_suite_helper:start_process(SlaveNode),
-    %% register
-    ok = rpc:call(SlaveNode, syn, register, [Name, PidRemote, SlaveNode]),
-    timer:sleep(250),
-    {PidRemote, SlaveNode} = syn:whereis(Name, with_meta),
-    {PidRemote, SlaveNode} = rpc:call(SlaveNode, syn, whereis, [Name, with_meta]),
-    %% un/register local over remote
-    Node = node(),
-    ok = syn:unregister_and_register(Name, PidLocal, Node),
-    timer:sleep(1000),
-    {PidLocal, Node} = syn:whereis(Name, with_meta),
-    {PidLocal, Node} = rpc:call(SlaveNode, syn, whereis, [Name, with_meta]),
-    ok = rpc:call(SlaveNode, syn, unregister_and_register, [Name, PidRemote, {SlaveNode, 2}]),
-    timer:sleep(1000),
-    {PidRemote, {SlaveNode, 2}} = syn:whereis(Name, with_meta),
-    {PidRemote, {SlaveNode, 2}} = rpc:call(SlaveNode, syn, whereis, [Name, with_meta]),
-    %% check that overwritten process is not monitored
-    {monitored_by, []} = erlang:process_info(PidLocal, monitored_by),
-    %% register local
-    ok = syn:unregister_and_register(Name, PidLocal, Node),
-    %% check a monitor exists
-    {monitored_by, [_MonitoringPid]} = erlang:process_info(PidLocal, monitored_by),
-    %% un/register local over local
-    ok = syn:unregister_and_register(Name, PidLocal2, {Node, 2}),
-    timer:sleep(1000),
-    {PidLocal2, {Node, 2}} = syn:whereis(Name, with_meta),
-    {PidLocal2, {Node, 2}} = rpc:call(SlaveNode, syn, whereis, [Name, with_meta]),
-    %% check that overwritten process is not monitored
-    {monitored_by, []} = erlang:process_info(PidLocal, monitored_by).
-
-three_nodes_partial_netsplit_consistency(Config) ->
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid0Changed = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    timer:sleep(100),
-    %% retrieve
-    undefined = syn:whereis(<<"proc0">>),
-    undefined = syn:whereis(<<"proc0-changed">>),
-    undefined = syn:whereis(<<"proc1">>),
-    undefined = syn:whereis(<<"proc2">>),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc0">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc0-changed">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc1">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc2">>]),
-    %% register (mix nodes)
-    ok = rpc:call(SlaveNode2, syn, register, [<<"proc0">>, Pid0]),
-    ok = syn:register(<<"proc1">>, Pid1),
-    ok = rpc:call(SlaveNode1, syn, register, [<<"proc2">>, Pid2]),
-    ok = rpc:call(SlaveNode1, syn, register, [<<"proc0-changed">>, Pid0Changed]),
-    timer:sleep(200),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    Pid0Changed = syn:whereis(<<"proc0-changed">>),
-    Pid1 = syn:whereis(<<"proc1">>),
-    Pid2 = syn:whereis(<<"proc2">>),
-    Pid0 = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    Pid0Changed = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]),
-    Pid0 = rpc:call(SlaveNode2, syn, whereis, [<<"proc0">>]),
-    Pid0Changed = rpc:call(SlaveNode2, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode2, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode2, syn, whereis, [<<"proc2">>]),
-    %% disconnect slave 2 from main (slave 1 can still see slave 2)
-    syn_test_suite_helper:disconnect_node(SlaveNode2),
-    timer:sleep(1000),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    Pid0Changed = syn:whereis(<<"proc0-changed">>),
-    Pid1 = syn:whereis(<<"proc1">>),
-    undefined = syn:whereis(<<"proc2">>), %% main has lost slave 2 so 'proc2' is removed
-    Pid0 = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    Pid0Changed = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]), %% slave 1 still has slave 2 so 'proc2' is still there
-    %% disconnect slave 1
-    syn_test_suite_helper:disconnect_node(SlaveNode1),
-    timer:sleep(500),
-    %% unregister proc0-changed
-    ok = syn:unregister(<<"proc0-changed">>),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    undefined = syn:whereis(<<"proc0-changed">>),
-    undefined = syn:whereis(<<"proc1">>),
-    undefined = syn:whereis(<<"proc2">>),
-    %% reconnect all
-    syn_test_suite_helper:connect_node(SlaveNode1),
-    syn_test_suite_helper:connect_node(SlaveNode2),
-    timer:sleep(5000),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    undefined = syn:whereis(<<"proc0-changed">>),
-    Pid1 = syn:whereis(<<"proc1">>),
-    Pid2 = syn:whereis(<<"proc2">>),
-    Pid0 = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]),
-    Pid0 = rpc:call(SlaveNode2, syn, whereis, [<<"proc0">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode2, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode2, syn, whereis, [<<"proc2">>]).
-
-three_nodes_full_netsplit_consistency(Config) ->
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid0Changed = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    timer:sleep(100),
-    %% retrieve
-    undefined = syn:whereis(<<"proc0">>),
-    undefined = syn:whereis(<<"proc0-changed">>),
-    undefined = syn:whereis(<<"proc1">>),
-    undefined = syn:whereis(<<"proc2">>),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc0">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc0-changed">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc1">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc2">>]),
-    %% register (mix nodes)
-    ok = rpc:call(SlaveNode2, syn, register, [<<"proc0">>, Pid0]),
-    ok = rpc:call(SlaveNode2, syn, register, [<<"proc0-changed">>, Pid0Changed]),
-    ok = syn:register(<<"proc1">>, Pid1),
-    ok = rpc:call(SlaveNode1, syn, register, [<<"proc2">>, Pid2]),
-    timer:sleep(200),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    Pid0Changed = syn:whereis(<<"proc0-changed">>),
-    Pid1 = syn:whereis(<<"proc1">>),
-    Pid2 = syn:whereis(<<"proc2">>),
-    Pid0 = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    Pid0Changed = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]),
-    Pid0 = rpc:call(SlaveNode2, syn, whereis, [<<"proc0">>]),
-    Pid0Changed = rpc:call(SlaveNode2, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode2, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode2, syn, whereis, [<<"proc2">>]),
-    %% disconnect slave 2 from main (slave 1 can still see slave 2)
-    syn_test_suite_helper:disconnect_node(SlaveNode2),
-    timer:sleep(1000),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    Pid0Changed = syn:whereis(<<"proc0-changed">>),
-    Pid1 = syn:whereis(<<"proc1">>),
-    undefined = syn:whereis(<<"proc2">>), %% main has lost slave 2 so 'proc2' is removed
-    Pid0 = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    Pid0Changed = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]), %% slave 1 still has slave 2 so 'proc2' is still there
-    %% disconnect slave 2 from slave 1
-    rpc:call(SlaveNode1, syn_test_suite_helper, disconnect_node, [SlaveNode2]),
-    timer:sleep(500),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    Pid0Changed = syn:whereis(<<"proc0-changed">>),
-    Pid1 = syn:whereis(<<"proc1">>),
-    undefined = syn:whereis(<<"proc2">>), %% main has lost slave 2 so 'proc2' is removed
-    undefined = syn:whereis(<<"proc2">>, with_meta),
-    Pid0 = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    Pid0Changed = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]),
-    %% disconnect slave 1
-    syn_test_suite_helper:disconnect_node(SlaveNode1),
-    timer:sleep(500),
-    %% unregister
-    ok = syn:unregister(<<"proc0-changed">>),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    undefined = syn:whereis(<<"proc0-changed">>),
-    undefined = syn:whereis(<<"proc1">>),
-    undefined = syn:whereis(<<"proc2">>),
-    %% reconnect all
-    syn_test_suite_helper:connect_node(SlaveNode1),
-    syn_test_suite_helper:connect_node(SlaveNode2),
-    rpc:call(SlaveNode1, syn_test_suite_helper, connect_node, [SlaveNode2]),
-    timer:sleep(1500),
-    %% retrieve
-    Pid0 = syn:whereis(<<"proc0">>),
-    undefined = syn:whereis(<<"proc0-changed">>),
-    Pid1 = syn:whereis(<<"proc1">>),
-    Pid2 = syn:whereis(<<"proc2">>),
-    Pid0 = rpc:call(SlaveNode1, syn, whereis, [<<"proc0">>]),
-    undefined = rpc:call(SlaveNode1, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode1, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode1, syn, whereis, [<<"proc2">>]),
-    Pid0 = rpc:call(SlaveNode2, syn, whereis, [<<"proc0">>]),
-    undefined = rpc:call(SlaveNode2, syn, whereis, [<<"proc0-changed">>]),
-    Pid1 = rpc:call(SlaveNode2, syn, whereis, [<<"proc1">>]),
-    Pid2 = rpc:call(SlaveNode2, syn, whereis, [<<"proc2">>]).
-
-three_nodes_start_syn_before_connecting_cluster_with_conflict_keep_more_recent(Config) ->
-    ConflictingName = "COMMON",
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    %% start delayed
-    start_syn_delayed_and_register_local_process(ConflictingName, Pid0, 2000),
-    timer:sleep(250),
-    rpc:cast(SlaveNode1, ?MODULE, start_syn_delayed_and_register_local_process, [ConflictingName, Pid1, 2000]),
-    timer:sleep(250),
-    rpc:cast(SlaveNode2, ?MODULE, start_syn_delayed_and_register_local_process, [ConflictingName, Pid2, 2000]),
-    timer:sleep(1000),
-    %% disconnect all
-    rpc:call(SlaveNode1, syn_test_suite_helper, disconnect_node, [SlaveNode2]),
-    syn_test_suite_helper:disconnect_node(SlaveNode1),
-    syn_test_suite_helper:disconnect_node(SlaveNode2),
-    timer:sleep(3000),
-    [] = nodes(),
-    %% reconnect all
-    syn_test_suite_helper:connect_node(SlaveNode1),
-    syn_test_suite_helper:connect_node(SlaveNode2),
-    rpc:call(SlaveNode1, syn_test_suite_helper, connect_node, [SlaveNode2]),
-    timer:sleep(2500),
-    %% count
-    1 = syn:registry_count(),
-    1 = rpc:call(SlaveNode1, syn, registry_count, []),
-    1 = rpc:call(SlaveNode2, syn, registry_count, []),
-    %% retrieve
-    true = lists:member(syn:whereis(ConflictingName), [Pid0, Pid1, Pid2]),
-    true = lists:member(rpc:call(SlaveNode1, syn, whereis, [ConflictingName]), [Pid0, Pid1, Pid2]),
-    true = lists:member(rpc:call(SlaveNode2, syn, whereis, [ConflictingName]), [Pid0, Pid1, Pid2]),
-    %% check metadata
-    {Pid2, SlaveNode2} = syn:whereis(ConflictingName, with_meta),
-    {Pid2, SlaveNode2} = rpc:call(SlaveNode1, syn, whereis, [ConflictingName, with_meta]),
-    {Pid2, SlaveNode2} = rpc:call(SlaveNode2, syn, whereis, [ConflictingName, with_meta]),
-    %% check that other processes are not alive because syn killed them
-    false = is_process_alive(Pid0),
-    false = rpc:call(SlaveNode1, erlang, is_process_alive, [Pid1]),
-    true = rpc:call(SlaveNode2, erlang, is_process_alive, [Pid2]).
-
-three_nodes_start_syn_before_connecting_cluster_with_custom_conflict_resolution_keep_remote(Config) ->
-    ConflictingName = "COMMON",
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    %% start delayed
-    start_syn_delayed_with_custom_handler_register_local_process(ConflictingName, Pid0, {node, node()}, 1500),
-    rpc:cast(
-        SlaveNode1,
-        ?MODULE,
-        start_syn_delayed_with_custom_handler_register_local_process,
-        [ConflictingName, Pid1, keep_this_one, 1500])
-    ,
-    rpc:cast(
-        SlaveNode2,
-        ?MODULE,
-        start_syn_delayed_with_custom_handler_register_local_process,
-        [ConflictingName, Pid2, {node, SlaveNode2}, 1500]
-    ),
-    timer:sleep(500),
-    %% disconnect all
-    rpc:call(SlaveNode1, syn_test_suite_helper, disconnect_node, [SlaveNode2]),
-    syn_test_suite_helper:disconnect_node(SlaveNode1),
-    syn_test_suite_helper:disconnect_node(SlaveNode2),
-    timer:sleep(1500),
-    [] = nodes(),
-    %% reconnect all
-    syn_test_suite_helper:connect_node(SlaveNode1),
-    syn_test_suite_helper:connect_node(SlaveNode2),
-    rpc:call(SlaveNode1, syn_test_suite_helper, connect_node, [SlaveNode2]),
-    timer:sleep(5000),
-    %% count
-    1 = syn:registry_count(),
-    1 = rpc:call(SlaveNode1, syn, registry_count, []),
-    1 = rpc:call(SlaveNode2, syn, registry_count, []),
-    %% retrieve
-    true = lists:member(syn:whereis(ConflictingName), [Pid0, Pid1, Pid2]),
-    true = lists:member(rpc:call(SlaveNode1, syn, whereis, [ConflictingName]), [Pid0, Pid1, Pid2]),
-    true = lists:member(rpc:call(SlaveNode2, syn, whereis, [ConflictingName]), [Pid0, Pid1, Pid2]),
-    %% check metadata that we kept the correct process on all nodes
-    {Pid1, keep_this_one} = syn:whereis(ConflictingName, with_meta),
-    {Pid1, keep_this_one} = rpc:call(SlaveNode1, syn, whereis, [ConflictingName, with_meta]),
-    {Pid1, keep_this_one} = rpc:call(SlaveNode1, syn, whereis, [ConflictingName, with_meta]),
-    %% check that other processes are still alive because we didn't kill them
-    true = is_process_alive(Pid0),
-    true = rpc:call(SlaveNode1, erlang, is_process_alive, [Pid1]),
-    true = rpc:call(SlaveNode2, erlang, is_process_alive, [Pid2]),
-    %% check that discarded processes are not monitored
-    {monitored_by, []} = erlang:process_info(Pid0, monitored_by),
-    {monitored_by, []} = rpc:call(SlaveNode2, erlang, process_info, [Pid2, monitored_by]).
-
-three_nodes_registration_race_condition_custom_conflict_resolution(Config) ->
-    ConflictingName = "COMMON",
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% use customer handler
-    syn_test_suite_helper:use_custom_handler(),
-    rpc:call(SlaveNode1, syn_test_suite_helper, use_custom_handler, []),
-    rpc:call(SlaveNode2, syn_test_suite_helper, use_custom_handler, []),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(500),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    %% inject into syn to simulate concurrent registration with something less recent (which would be discarded without a custom handler)
-    ok = rpc:call(SlaveNode1, syn_registry, add_to_local_table, [ConflictingName, Pid1, keep_this_one, erlang:system_time() - 1000000000, undefined]),
-    %% register on master node to trigger conflict resolution
-    ok = syn:register(ConflictingName, Pid0, node()),
-    timer:sleep(1000),
-    %% retrieve
-    true = lists:member(syn:whereis(ConflictingName), [Pid0, Pid1, Pid2]),
-    true = lists:member(rpc:call(SlaveNode1, syn, whereis, [ConflictingName]), [Pid0, Pid1, Pid2]),
-    true = lists:member(rpc:call(SlaveNode2, syn, whereis, [ConflictingName]), [Pid0, Pid1, Pid2]),
-    %% check metadata that we kept the correct process on all nodes
-    {Pid1, keep_this_one} = syn:whereis(ConflictingName, with_meta),
-    {Pid1, keep_this_one} = rpc:call(SlaveNode1, syn, whereis, [ConflictingName, with_meta]),
-    {Pid1, keep_this_one} = rpc:call(SlaveNode1, syn, whereis, [ConflictingName, with_meta]),
-    %% check that other processes are still alive because we didn't kill them
-    true = is_process_alive(Pid0),
-    true = rpc:call(SlaveNode1, erlang, is_process_alive, [Pid1]),
-    true = rpc:call(SlaveNode2, erlang, is_process_alive, [Pid2]).
-
-three_nodes_anti_entropy(Config) ->
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% use customer handler
-    syn_test_suite_helper:use_custom_handler(),
-    rpc:call(SlaveNode1, syn_test_suite_helper, use_custom_handler, []),
-    rpc:call(SlaveNode2, syn_test_suite_helper, use_custom_handler, []),
-    %% set anti-entropy with a very low interval (0.25 second)
-    syn_test_suite_helper:use_anti_entropy(registry, 0.25),
-    rpc:call(SlaveNode1, syn_test_suite_helper, use_anti_entropy, [registry, 0.25]),
-    rpc:call(SlaveNode2, syn_test_suite_helper, use_anti_entropy, [registry, 0.25]),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    Pid0Conflict = syn_test_suite_helper:start_process(),
-    Pid1Conflict = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2Conflict = syn_test_suite_helper:start_process(SlaveNode2),
-    timer:sleep(100),
-    %% inject data to simulate latent conflicts
-    ok = syn_registry:add_to_local_table("pid0", Pid0, node(), 0, undefined),
-    ok = rpc:call(SlaveNode1, syn_registry, add_to_local_table, ["pid1", Pid1, SlaveNode1, 0, undefined]),
-    ok = rpc:call(SlaveNode2, syn_registry, add_to_local_table, ["pid2", Pid2, SlaveNode2, 0, undefined]),
-    ok = syn_registry:add_to_local_table("conflict", Pid0Conflict, node(), erlang:system_time() + 1000000000, undefined),
-    ok = rpc:call(SlaveNode1, syn_registry, add_to_local_table, ["conflict", Pid1Conflict, keep_this_one, erlang:system_time(), undefined]),
-    ok = rpc:call(SlaveNode2, syn_registry, add_to_local_table, ["conflict", Pid2Conflict, SlaveNode2, erlang:system_time() + 1000000000, undefined]),
-    %% wait to let anti-entropy settle
-    timer:sleep(5000),
-    %% check
-    Node = node(),
-    {Pid0, Node} = syn:whereis("pid0", with_meta),
-    {Pid1, SlaveNode1} = syn:whereis("pid1", with_meta),
-    {Pid2, SlaveNode2} = syn:whereis("pid2", with_meta),
-    {Pid1Conflict, keep_this_one} = syn:whereis("conflict", with_meta),
-    {Pid0, Node} = rpc:call(SlaveNode1, syn, whereis, ["pid0", with_meta]),
-    {Pid1, SlaveNode1} = rpc:call(SlaveNode1, syn, whereis, ["pid1", with_meta]),
-    {Pid2, SlaveNode2} = rpc:call(SlaveNode1, syn, whereis, ["pid2", with_meta]),
-    {Pid1Conflict, keep_this_one} = rpc:call(SlaveNode1, syn, whereis, ["conflict", with_meta]),
-    {Pid0, Node} = rpc:call(SlaveNode2, syn, whereis, ["pid0", with_meta]),
-    {Pid1, SlaveNode1} = rpc:call(SlaveNode2, syn, whereis, ["pid1", with_meta]),
-    {Pid2, SlaveNode2} = rpc:call(SlaveNode2, syn, whereis, ["pid2", with_meta]),
-    {Pid1Conflict, keep_this_one} = rpc:call(SlaveNode2, syn, whereis, ["conflict", with_meta]).
-
-three_nodes_anti_entropy_manual(Config) ->
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% use customer handler
-    syn_test_suite_helper:use_custom_handler(),
-    rpc:call(SlaveNode1, syn_test_suite_helper, use_custom_handler, []),
-    rpc:call(SlaveNode2, syn_test_suite_helper, use_custom_handler, []),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    Pid0Conflict = syn_test_suite_helper:start_process(),
-    Pid1Conflict = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2Conflict = syn_test_suite_helper:start_process(SlaveNode2),
-    timer:sleep(100),
-    %% inject data to simulate latent conflicts
-    ok = syn_registry:add_to_local_table("pid0", Pid0, node(), 0, undefined),
-    ok = rpc:call(SlaveNode1, syn_registry, add_to_local_table, ["pid1", Pid1, SlaveNode1, 0, undefined]),
-    ok = rpc:call(SlaveNode2, syn_registry, add_to_local_table, ["pid2", Pid2, SlaveNode2, 0, undefined]),
-    ok = syn_registry:add_to_local_table("conflict", Pid0Conflict, node(), erlang:system_time() + 1000000000, undefined),
-    ok = rpc:call(SlaveNode1, syn_registry, add_to_local_table, ["conflict", Pid1Conflict, keep_this_one, erlang:system_time(), undefined]),
-    ok = rpc:call(SlaveNode2, syn_registry, add_to_local_table, ["conflict", Pid2Conflict, SlaveNode2, erlang:system_time() + 1000000000, undefined]),
-    %% call anti entropy
-    ok = syn:force_cluster_sync(registry),
-    timer:sleep(1000),
-    %% check
-    Node = node(),
-    {Pid0, Node} = syn:whereis("pid0", with_meta),
-    {Pid1, SlaveNode1} = syn:whereis("pid1", with_meta),
-    {Pid2, SlaveNode2} = syn:whereis("pid2", with_meta),
-    {Pid1Conflict, keep_this_one} = syn:whereis("conflict", with_meta),
-    {Pid0, Node} = rpc:call(SlaveNode1, syn, whereis, ["pid0", with_meta]),
-    {Pid1, SlaveNode1} = rpc:call(SlaveNode1, syn, whereis, ["pid1", with_meta]),
-    {Pid2, SlaveNode2} = rpc:call(SlaveNode1, syn, whereis, ["pid2", with_meta]),
-    {Pid1Conflict, keep_this_one} = rpc:call(SlaveNode1, syn, whereis, ["conflict", with_meta]),
-    {Pid0, Node} = rpc:call(SlaveNode2, syn, whereis, ["pid0", with_meta]),
-    {Pid1, SlaveNode1} = rpc:call(SlaveNode2, syn, whereis, ["pid1", with_meta]),
-    {Pid2, SlaveNode2} = rpc:call(SlaveNode2, syn, whereis, ["pid2", with_meta]),
-    {Pid1Conflict, keep_this_one} = rpc:call(SlaveNode2, syn, whereis, ["conflict", with_meta]).
-
-three_nodes_concurrent_registration_unregistration(Config) ->
-    CommonName = "common-name",
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid0 = syn_test_suite_helper:start_process(),
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    timer:sleep(100),
-    %% register on 0
-    ok = syn:register(CommonName, Pid0, node()),
-    timer:sleep(250),
-    %% check
-    Node = node(),
-    {Pid0, Node} = syn:whereis(CommonName, with_meta),
-    {Pid0, Node} = rpc:call(SlaveNode1, syn, whereis, [CommonName, with_meta]),
-    {Pid0, Node} = rpc:call(SlaveNode2, syn, whereis, [CommonName, with_meta]),
-    %% simulate unregistration with inconsistent data
-    syn_registry:sync_unregister(SlaveNode1, Pid1, CommonName),
-    timer:sleep(250),
-    %% check
-    Node = node(),
-    {Pid0, Node} = syn:whereis(CommonName, with_meta),
-    {Pid0, Node} = rpc:call(SlaveNode1, syn, whereis, [CommonName, with_meta]),
-    {Pid0, Node} = rpc:call(SlaveNode2, syn, whereis, [CommonName, with_meta]).
-
-three_nodes_resolve_conflict_on_all_nodes(Config) ->
-    CommonName = "common-name",
-    %% get slaves
-    SlaveNode1 = proplists:get_value(slave_node_1, Config),
-    SlaveNode2 = proplists:get_value(slave_node_2, Config),
-    %% start syn on nodes
-    ok = syn:start(),
-    ok = rpc:call(SlaveNode1, syn, start, []),
-    ok = rpc:call(SlaveNode2, syn, start, []),
-    timer:sleep(100),
-    %% start processes
-    Pid1 = syn_test_suite_helper:start_process(SlaveNode1),
-    Pid2 = syn_test_suite_helper:start_process(SlaveNode2),
-    timer:sleep(100),
-    %% register on slave 1
-    ok = rpc:call(SlaveNode1, syn, register, [CommonName, Pid1, SlaveNode1]),
-    timer:sleep(500),
-    %% check
-    {Pid1, SlaveNode1} = syn:whereis(CommonName, with_meta),
-    {Pid1, SlaveNode1} = rpc:call(SlaveNode1, syn, whereis, [CommonName, with_meta]),
-    {Pid1, SlaveNode1} = rpc:call(SlaveNode2, syn, whereis, [CommonName, with_meta]),
-    %% force a sync registration conflict on master node from slave 2
-    syn_registry:sync_register(node(), CommonName, Pid2, SlaveNode2, erlang:system_time() + 1000000000, false),
-    timer:sleep(1000),
-    %% check
-    {Pid2, SlaveNode2} = syn:whereis(CommonName, with_meta),
-    {Pid2, SlaveNode2} = rpc:call(SlaveNode1, syn, whereis, [CommonName, with_meta]).
-
-%% ===================================================================
-%% Internal
-%% ===================================================================
-start_syn_delayed_and_register_local_process(Name, Pid, Ms) ->
-    spawn(fun() ->
-        lists:foreach(fun(Node) ->
-            syn_test_suite_helper:disconnect_node(Node)
-        end, nodes()),
-        timer:sleep(Ms),
-        [] = nodes(),
-        %%
-        syn:start(),
-        ok = syn:register(Name, Pid, node())
-    end).
-
-start_syn_delayed_with_custom_handler_register_local_process(Name, Pid, Meta, Ms) ->
-    spawn(fun() ->
-        lists:foreach(fun(Node) ->
-            syn_test_suite_helper:disconnect_node(Node)
-        end, nodes()),
-        timer:sleep(Ms),
-        [] = nodes(),
-        %% use customer handler
-        syn_test_suite_helper:use_custom_handler(),
-        %%
-        syn:start(),
-        ok = syn:register(Name, Pid, Meta)
-    end).
-
-seq_unregister_register(Name, Pid, Meta) ->
-    syn:unregister(Name),
-    syn:register(Name, Pid, Meta).

test/syn_test_event_handler.erl

@@ -1,71 +0,0 @@
-%% ==========================================================================================================
-%% Syn - A global Process Registry and Process Group manager.
-%%
-%% The MIT License (MIT)
-%%
-%% Copyright (c) 2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
-%%
-%% Permission is hereby granted, free of charge, to any person obtaining a copy
-%% of this software and associated documentation files (the "Software"), to deal
-%% in the Software without restriction, including without limitation the rights
-%% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-%% copies of the Software, and to permit persons to whom the Software is
-%% furnished to do so, subject to the following conditions:
-%%
-%% The above copyright notice and this permission notice shall be included in
-%% all copies or substantial portions of the Software.
-%%
-%% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-%% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-%% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-%% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-%% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-%% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-%% THE SOFTWARE.
-%% ==========================================================================================================
--module(syn_test_event_handler).
--behaviour(syn_event_handler).
-
-%% API
--export([on_process_exit/4]).
--export([on_group_process_exit/4]).
--export([resolve_registry_conflict/3]).
-
-%% ===================================================================
-%% Syn Callbacks
-%% ===================================================================
--spec on_process_exit(
-    Name :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Reason :: any()
-) -> any().
-on_process_exit(_Name, _Pid, {PidId, TestPid}, _Reason) when is_pid(TestPid) ->
-    TestPid ! {received_event_on, PidId};
-on_process_exit(_Name, _Pid, {Meta, TestPid}, _Reason) when is_pid(TestPid) ->
-    TestPid ! {received_event_on, Meta};
-on_process_exit(_Name, _Pid, _Meta, _Reason) ->
-    ok.
-
--spec on_group_process_exit(
-    GroupName :: any(),
-    Pid :: pid(),
-    Meta :: any(),
-    Reason :: any()
-) -> any().
-on_group_process_exit(_GroupName, _Pid, {PidId, TestPid}, _Reason) when is_pid(TestPid) ->
-    TestPid ! {received_event_on, PidId};
-on_group_process_exit(_GroupName, _Pid, _Meta, _Reason) ->
-    ok.
-
--spec resolve_registry_conflict(
-    Name :: any(),
-    {LocalPid :: pid(), LocalMeta :: any()},
-    {RemotePid :: pid(), RemoteMeta :: any()}
-) -> PidToKeep :: pid().
-resolve_registry_conflict(_Name, {LocalPid, keep_this_one}, {_RemotePid, _RemoteMeta}) ->
-    LocalPid;
-resolve_registry_conflict(_Name, {_LocalPid, _LocalMeta}, {RemotePid, keep_this_one}) ->
-    RemotePid;
-resolve_registry_conflict(_Name, {LocalPid, _LocalMeta}, {_RemotePid, _RemoteMeta}) ->
-    LocalPid.

test/syn_test_gen_server.erl

@@ -1,66 +0,0 @@
-%% ==========================================================================================================
-%% Syn - A global Process Registry and Process Group manager.
-%%
-%% The MIT License (MIT)
-%%
-%% Copyright (c) 2015-2019 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
-%%
-%% Permission is hereby granted, free of charge, to any person obtaining a copy
-%% of this software and associated documentation files (the "Software"), to deal
-%% in the Software without restriction, including without limitation the rights
-%% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-%% copies of the Software, and to permit persons to whom the Software is
-%% furnished to do so, subject to the following conditions:
-%%
-%% The above copyright notice and this permission notice shall be included in
-%% all copies or substantial portions of the Software.
-%%
-%% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-%% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-%% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-%% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-%% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-%% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-%% THE SOFTWARE.
-%% ==========================================================================================================
--module(syn_test_gen_server).
--behaviour(gen_server).
-
-%% API
--export([start_link/0]).
--export([ping/0]).
--export([stop/0]).
-
-%% gen_server callbacks
--export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).
-
-start_link() ->
-    gen_server:start_link({via, syn, ?MODULE}, ?MODULE, [], []).
-
-ping() ->
-    gen_server:call({via, syn, ?MODULE}, ping).
-
-stop() ->
-    gen_server:cast({via, syn, ?MODULE}, stop).
-
-init(State) ->
-    {ok, State}.
-
-handle_call(ping, _From, State) ->
-    {reply, pong, State}.
-
-handle_cast(stop, State) ->
-    {stop, normal, State};
-
-handle_cast(_Msg, State) ->
-    {noreply, State}.
-
-handle_info({SenderPid, send_ping}, State) ->
-    SenderPid ! send_pong,
-    {noreply, State}.
-
-terminate(_Reason, _State) ->
-    ok.
-
-code_change(_OldVsn, State, _Extra) ->
-    {ok, State}.

test/syn_test_suite_helper.erl

@@ -1,125 +0,0 @@
-%% ==========================================================================================================
-%% Syn - A global Process Registry and Process Group manager.
-%%
-%% The MIT License (MIT)
-%%
-%% Copyright (c) 2015 Roberto Ostinelli <roberto@ostinelli.net> and Neato Robotics, Inc.
-%%
-%% Permission is hereby granted, free of charge, to any person obtaining a copy
-%% of this software and associated documentation files (the "Software"), to deal
-%% in the Software without restriction, including without limitation the rights
-%% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-%% copies of the Software, and to permit persons to whom the Software is
-%% furnished to do so, subject to the following conditions:
-%%
-%% The above copyright notice and this permission notice shall be included in
-%% all copies or substantial portions of the Software.
-%%
-%% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-%% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-%% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-%% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-%% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-%% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-%% THE SOFTWARE.
-%% ==========================================================================================================
--module(syn_test_suite_helper).
-
-%% API
--export([start_slave/1, start_slave/4]).
--export([stop_slave/1, stop_slave/2]).
--export([connect_node/1, disconnect_node/1]).
--export([clean_after_test/0]).
--export([start_process/0, start_process/1, start_process/2]).
--export([kill_process/1]).
--export([use_custom_handler/0]).
--export([use_anti_entropy/2]).
--export([send_error_logger_to_disk/0]).
-
-%% internal
--export([process_main/0]).
-
-%% ===================================================================
-%% API
-%% ===================================================================
-start_slave(NodeShortName) ->
-    {ok, Node} = ct_slave:start(NodeShortName, [{boot_timeout, 10}]),
-    CodePath = code:get_path(),
-    true = rpc:call(Node, code, set_path, [CodePath]),
-    {ok, Node}.
-start_slave(NodeShortName, Host, Username, Password) ->
-    {ok, Node} = ct_slave:start(Host, NodeShortName, [
-        {boot_timeout, 10},
-        {username, Username},
-        {password, Password}
-    ]),
-    CodePath = code:get_path(),
-    true = rpc:call(Node, code, set_path, [CodePath]),
-    {ok, Node}.
-
-stop_slave(NodeShortName) ->
-    {ok, _} = ct_slave:stop(NodeShortName).
-stop_slave(Host, NodeShortName) ->
-    {ok, _} = ct_slave:stop(Host, NodeShortName).
-
-connect_node(Node) ->
-    net_kernel:connect_node(Node).
-
-disconnect_node(Node) ->
-    erlang:disconnect_node(Node).
-
-clean_after_test() ->
-    Nodes = [node() | nodes()],
-    %% shutdown
-    lists:foreach(fun(Node) ->
-        %% close syn
-        rpc:call(Node, application, stop, [syn]),
-        %% clean env
-        rpc:call(Node, application, unset_env, [syn, event_handler]),
-        rpc:call(Node, application, unset_env, [syn, anti_entropy])
-    end, Nodes).
-
-start_process() ->
-    Pid = spawn(fun process_main/0),
-    Pid.
-start_process(Node) when is_atom(Node) ->
-    Pid = spawn(Node, fun process_main/0),
-    Pid;
-start_process(Loop) when is_function(Loop) ->
-    Pid = spawn(Loop),
-    Pid.
-start_process(Node, Loop) ->
-    Pid = spawn(Node, Loop),
-    Pid.
-
-kill_process(Pid) ->
-    exit(Pid, kill).
-
-use_custom_handler() ->
-    application:set_env(syn, event_handler, syn_test_event_handler).
-
-use_anti_entropy(registry, Interval) ->
-    application:set_env(syn, anti_entropy, [
-        {registry, [
-            {interval, Interval},
-            {interval_max_deviation, 0.1}
-        ]}
-    ]);
-use_anti_entropy(groups, Interval) ->
-    application:set_env(syn, anti_entropy, [
-        {groups, [
-            {interval, Interval},
-            {interval_max_deviation, 0.1}
-        ]}
-    ]).
-
-send_error_logger_to_disk() ->
-    error_logger:logfile({open, atom_to_list(node())}).
-
-%% ===================================================================
-%% Internal
-%% ===================================================================
-process_main() ->
-    receive
-        _ -> process_main()
-    end.