Part 3.4: Peer-to-peer with Infinispan

In the peer-to-peer overview, we showed that
there are two fundamentally different types of peer-to-peer architectures:

full replication, as implemented by Ehcache
distributed hash tables, as implemented by Hazelcast

Infinispan’s peer-to-peer set-up implements both, full replication and
distributed hash tables. Moreover, each of them can be configured in
synchronous mode as well as in asynchronous mode.

This page describes Infinispan’s peer-to-peer set-up.

When Infinispan is configured in async mode, the cache will become inconsistent
when the same key is modified by two peers at the same time.
Therefore, like Ehcache, Infinispan in async mode can only be used
in two scenarios:

Scenario 1: Each key is only written once, like in a WORM (write-once-read-many) application.
Scenario 2: Each key is only updated by a single Tomcat instance, like an HTTP session key when a session-aware load balancer is used.

However, unlike Ehcache, Infinispan’s peer-to-peer implementation also
provides a synchronous configuration supporting distributed atomic operations.

Initialization

In order to set up the cluster, we need to configure the two instances with
two different configuration files: One defines the node as node1, while
the other defines the node as node2.

The name of the configuration is passed to the initialization code via
the system property infinispan.config.filename, see section “How to Run”
below:

@Override
public void contextInitialized(ServletContextEvent servletContextEvent) {
    try {
        String cfg = System.getProperty("infinispan.config.filename", "infinispan1.xml");
        EmbeddedCacheManager cacheManager = new DefaultCacheManager(cfg);
        Cache<String, UserEventList> cache = cacheManager.getCache();
        ServletContext context = servletContextEvent.getServletContext();
        context.setAttribute(CACHE_MANAGER, cacheManager);
        context.setAttribute(CACHE, cache);
    } catch (IOException e) {
        throw new WebApplicationException(e);
    }
}

The following example of infinispan1.xml configures Infinispan in fully
replicated mode with asynchronous replication.

<infinispan xsi:schemaLocation="urn:infinispan:config:5.2 infinispan-config-5.2.xsd"
            xmlns="urn:infinispan:config:5.2"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <global>
        <transport nodeName="node2" clusterName="cluster"/>
    </global>
    <default>
        <clustering mode="replication">
            <async/>
        </clustering>
        <eviction maxEntries="1000" strategy="LRU"/>
    </default>
</infinispan>

In order to switch to distributed hash table mode, replication needs
to be replaced with dist:

<clustering mode="dist">
    <async/>
</clustering>

Both, replication and dist mode are also available in synchronous
configuration. In that case, <async/> needs to be replaced with
<sync/>.

As the Infinispan cache implements ConcurrentMap, the initialization of
the REST interface stays unchanged:

// ...
@Context
private ServletContext context;
private ConcurrentMap<String, UserEventList> map;

@PostConstruct
@SuppressWarnings("unchecked")
public void init() {
    map = (ConcurrentMap) context.getAttribute(CACHE);
}

Shutdown

@Override
public void contextDestroyed(ServletContextEvent servletContextEvent) {
    ServletContext context = servletContextEvent.getServletContext();
    EmbeddedCacheManager cacheManager = (EmbeddedCacheManager) context.getAttribute(CACHE_MANAGER);
    cacheManager.stop();
}

Write

As the Infinispan cache implements ConcurrentMap, the write implementation
is the same as with the Map-based reference implementation:

@POST
@Path("{user}")
@Consumes(MediaType.APPLICATION_JSON)
public void appendEvent(@PathParam("user") String user, String msg) {
    boolean success;
    map.putIfAbsent(user, UserEventList.emptyList());
    do {
        UserEventList oldMsgList = map.get(user);
        UserEventList newMsgList = UserEventList.append(oldMsgList, msg);
        success = map.replace(user, oldMsgList, newMsgList);
    }
    while ( ! success );
}

However, when Infinispan is configured in <async/> mode,
the cache may become inconsistent when
map.replace(K, K, V)
is called.

While Ehcache throws an exception when
create-and-swap operations are called in a configuration that cannot provide
consistency guarantees, the Infinispan calls to putIfAbsent()
and replace()
do not fail, but may leave the cache in an inconsistent state.

In <sync/> mode consistency is guaranteed, as the operations are only
successful when each peer involved acknowledges successful operation.

Read

@GET
@Path("{user}")
@Produces(MediaType.APPLICATION_JSON)
public List<String> searchByUser(@PathParam("user") String user) {
    UserEventList result = map.get(user);
    if ( result == null ) {
        return new ArrayList<>();
    }
    return result.getMessages();
}

How to Run

Our example code is hosted on GitHub.
The example can be run with maven.

Instance 1:

mvn tomcat7:run-war -pl part03.infinispan -am verify \
    -Dmaven.tomcat.port=8080 -Dinfinispan.config.filename=infinispan1.xml

Instance 2:

mvn tomcat7:run-war -pl part03.infinispan -am verify \
        -Dmaven.tomcat.port=9090 -Dinfinispan.config.filename=infinispan2.xml

In order to switch between replication and dist mode, or between sync
and async mode, the configuration files need to be edited.

The Web interface is then accessible via
http://localhost:8080
and
http://localhost:9090.

This was the last page of the peer-to-peer part of this series.
In the next part, we will introduce client-server deployments as follows:

Initialization

Shutdown

Write

Read

How to Run

Next