[[EJB_Services]] = EJB Services This chapter explains how clustering of EJBs works in WildFly {wildflyVersion}. [[ejb-subsystem]] == EJB Subsystem [[ejb-timer]] == EJB Timer Wildfly now supports clustered database backed timers. For details have a look to the https://docs.jboss.org/author/display/WFLY/EJB3+Clustered+Database+Timers[EJB3 reference section] [[marking-an-ejb-as-clustered]] === Marking an EJB as clustered WildFly {wildflyVersion} allows clustering of stateful session beans. A stateful session bean can be marked with `@org.jboss.ejb3.annotation.Clustered` annotation or be marked as clustered using the jboss-ejb3.xml's `` element. MyStatefulBean [source, java] ---- import org.jboss.ejb3.annotation.Clustered; import javax.ejb.Stateful;   @Stateful @Clustered public class MyStatefulBean { ... } ---- jboss-ejb3.xml [source, java] ----     DDBasedClusteredBean true ---- [[deploying-clustered-ejbs]] === Deploying clustered EJBs Clustering support is available in the HA profiles of WildFly {wildflyVersion}. In this chapter we'll be using the standalone server for explaining the details. However, the same applies to servers in a domain mode. Starting the standalone server with HA capabilities enabled, involves starting it with the standalone-ha.xml (or even standalone-full-ha.xml): [source, java] ---- ./standalone.sh -server-config=standalone-ha.xml ---- This will start a single instance of the server with HA capabilities. Deploying the EJBs to this instance _doesn't_ involve anything special and is the same as explained in the link:Admin_Guide{outfilesuffix}#Application_deployment[application deployment chapter]. Obviously, to be able to see the benefits of clustering, you'll need more than one instance of the server. So let's start another server with HA capabilities. That another instance of the server can either be on the same machine or on some other machine. If it's on the same machine, the two things you have to make sure is that you pass the port offset for the second instance and also make sure that each of the server instances have a unique `jboss.node.name` system property. You can do that by passing the following two system properties to the startup command: [source, java] ---- ./standalone.sh -server-config=standalone-ha.xml -Djboss.socket.binding.port-offset= -Djboss.node.name= ---- Follow whichever approach you feel comfortable with for deploying the EJB deployment to this instance too. [IMPORTANT] Deploying the application on just one node of a standalone instance of a clustered server does *not* mean that it will be automatically deployed to the other clustered instance. You will have to do deploy it explicitly on the other standalone clustered instance too. Or you can start the servers in domain mode so that the deployment can be deployed to all the server within a server group. See the https://docs.jboss.org/author/display/AS71/Admin+Guide[admin guide] for more details on domain setup. Now that you have deployed an application with clustered EJBs on both the instances, the EJBs are now capable of making use of the clustering features. [[failover-for-clustered-ejbs]] === Failover for clustered EJBs Clustered EJBs have failover capability. The state of the @Stateful @Clustered EJBs is replicated across the cluster nodes so that if one of the nodes in the cluster goes down, some other node will be able to take over the invocations. Let's see how it's implemented in WildFly {wildflyVersion}. In the next few sections we'll see how it works for remote (standalone) clients and for clients in another remote WildFly server instance. Although, there isn't a difference in how it works in both these cases, we'll still explain it separately so as to make sure there aren't any unanswered questions. [[remote-standalone-clients]] ==== Remote standalone clients In this section we'll consider a remote standalone client (i.e. a client which runs in a separate JVM and _isn't_ running within another WildFly 8 instance). Let's consider that we have 2 servers, server X and server Y which we started earlier. Each of these servers has the clustered EJB deployment. A standalone remote client can use either the link:Developer_Guide{outfilesuffix}#EJB_invocations_from_a_remote_client_using_JNDI[JNDI approach] or native JBoss EJB client APIs to communicate with the servers. The important thing to note is that when you are invoking clustered EJB deployments, you do *not* have to list all the servers within the cluster (which obviously wouldn't have been feasible due the dynamic nature of cluster node additions within a cluster). The remote client just has to list only one of the servers with the clustering capability. In this case, we can either list server X (in jboss-ejb-client.properties) _or_ server Y. This server will act as the starting point for cluster topology communication between the client and the clustered nodes. Note that you have to configure the _ejb_ cluster in the jboss-ejb-client.properties configuration file, like so: [source, java] ---- remote.clusters=ejb remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false ---- [[cluster-topology-communication]] ==== Cluster topology communication When a client connects to a server, the JBoss EJB client implementation (internally) communicates with the server for cluster topology information, if the server had clustering capability. In our example above, let's assume we listed server X as the initial server to connect to. When the client connects to server X, the server will send back an (asynchronous) cluster topology message to the client. This topology message consists of the cluster name(s) and the information of the nodes that belong to the cluster. The node information includes the node address and port number to connect to (whenever necessary). So in this example, the server X will send back the cluster topology consisting of the other server Y which belongs to the cluster. In case of stateful (clustered) EJBs, a typical invocation flow involves creating of a session for the stateful bean, which happens when you do a JNDI lookup for that bean, and then invoking on the returned proxy. The lookup for stateful bean, internally, triggers a (synchronous) session creation request from the client to the server. In this case, the session creation request goes to server X since that's the initial connection that we have configured in our jboss-ejb-client.properties. Since server X is clustered, it will return back a session id and along with send back an _"affinity"_ of that session. In case of clustered servers, the affinity equals to the name of the cluster to which the stateful bean belongs on the server side. For non-clustered beans, the affinity is just the node name on which the session was created. This _affinity_ will later help the EJB client to route the invocations on the proxy, appropriately to either a node within a cluster (for clustered beans) or to a specific node (for non-clustered beans). While this session creation request is going on, the server X will also send back an asynchronous message which contains the cluster topology. The JBoss EJB client implementation will take note of this topology information and will later use it for connection creation to nodes within the cluster and routing invocations to those nodes, whenever necessary. Now that we know how the cluster topology information is communicated from the server to the client, let see how failover works. Let's continue with the example of server X being our starting point and a client application looking up a stateful bean and invoking on it. During these invocations, the client side will have collected the cluster topology information from the server. Now let's assume for some reason, server X goes down and the client application subsequent invokes on the proxy. The JBoss EJB client implementation, at this stage will be aware of the affinity and in this case it's a cluster affinity. Because of the cluster topology information it has, it knows that the cluster has two nodes server X and server Y. When the invocation now arrives, it sees that the server X is down. So it uses a selector to fetch a suitable node from among the cluster nodes. The selector itself is configurable, but we'll leave it from discussion for now. When the selector returns a node from among the cluster, the JBoss EJB client implementation creates a connection to that node (if not already created earlier) and creates a EJB receiver out of it. Since in our example, the only other node in the cluster is server Y, the selector will return that node and the JBoss EJB client implementation will use it to create a EJB receiver out of it and use that receiver to pass on the invocation on the proxy. Effectively, the invocation has now failed over to a different node within the cluster. [[remote-clients-on-another-instance-of-wildfly-8]] ==== Remote clients on another instance of WildFly {wildflyVersion} So far we discussed remote standalone clients which typically use either the EJB client API or the jboss-ejb-client.properties based approach to configure and communicate with the servers where the clustered beans are deployed. Now let's consider the case where the client is an application deployed another AS7 instance and it wants to invoke on a clustered stateful bean which is deployed on another instance of WildFly {wildflyVersion}. In this example let's consider a case where we have 3 servers involved. Server X and Server Y both belong to a cluster and have clustered EJB deployed on them. Let's consider another server instance Server C (which may or may _not_ have clustering capability) which acts as a client on which there's a deployment which wants to invoke on the clustered beans deployed on server X and Y and achieve failover. The configurations required to achieve this are explained in link:Developer_Guide{outfilesuffix}#EJB_invocations_from_a_remote_server_instance[this chapter]. As you can see the configurations are done in a jboss-ejb-client.xml which points to a remote outbound connection to the other server. This jboss-ejb-client.xml goes in the deployment of server C (since that's our client). As explained eariler, the client configuration need *not* point to all clustered nodes. Instead it just has to point to one of them which will act as a start point for communication. So in this case, we can create a remote outbound connection on server C to server X and use server X as our starting point for communication. Just like in the case of remote standalone clients, when the application on server C (client) looks up a stateful bean, a session creation request will be sent to server X which will send back a session id and the cluster affinity for it. Furthermore, server X asynchronously send back a message to server C (client) containing the cluster topology. This topology information will include the node information of server Y (since that belongs to the cluster along with server X). Subsequent invocations on the proxy will be routed appropriately to the nodes in the cluster. If server X goes down, as explained earlier, a different node from the cluster will be selected and the invocation will be forwarded to that node. As can be seen both remote standalone client and remote clients on another WildFly {wildflyVersion} instance act similar in terms of failover. [[testcases-for-failover-of-stateful-beans]] ==== Testcases for failover of stateful beans We have testcases in WildFly {wildflyVersion} testsuite which test that whatever is explained above works as expected. The https://github.com/wildfly/wildfly/blob/master/testsuite/integration/clust/src/test/java/org/jboss/as/test/clustering/cluster/ejb3/stateful/remote/failover/RemoteEJBClientStatefulBeanFailoverTestCase.java[RemoteEJBClientStatefulBeanFailoverTestCase] tests the case where a stateful EJB uses @Clustered annotation to mark itself as clustered. We also have https://github.com/wildfly/wildfly/blob/master/testsuite/integration/clust/src/test/java/org/jboss/as/test/clustering/cluster/ejb3/stateful/remote/failover/dd/RemoteEJBClientDDBasedSFSBFailoverTestCase.java[RemoteEJBClientDDBasedSFSBFailoverTestCase] which uses jboss-ejb3.xml to mark a stateful EJB as clustered. Both these testcases test that when a node goes down in a cluster, the client invocation is routed to a different node in the cluster.