Published on The O'Reilly Network (http://www.oreillynet.com/)
http://www.onjava.com/pub/a/onjava/2003/08/20/jboss_clustering.html
See this if you're having trouble printing code examples

## J2EE Clustering with JBoss

by Ivelin Ivanov
08/20/2003

In a recent article, Bill Burke and Sacha Labourey explained the key components of JBoss 3 clustering. We will now present several new clustered services recently introduced in JBoss 3.2.2, which was not yet released at the time of the writing of that article. Basic knowledge of JBoss 3 clustering and farming is a prerequisite for this article.

Several design patterns commonly occur in clustered enterprise applications. Though they are not covered by the current J2EE specification, JBoss provides several built-in services to help the development of components that follow these patterns.

### Singleton Service

A clustered singleton service is deployed on multiple nodes in a cluster but runs on only one of the nodes. The node running the singleton service is typically called the master node. When the master fails, another master is selected from the remaining nodes and the service is restarted on the new master.

Figure 1. Clustered singleton service

Many times, an application needs to ensure that a certain task is run exactly once. Thus, only one of the nodes in a cluster should execute the task. The other nodes should knowingly remain passive. Examples of singleton tasks include:

Related Reading Java Enterprise Best Practices By The O'Reilly Java Authors Table of Contents Index Sample Chapter Read Online--Safari Search this book on Safari: Code Fragments only

- Sending email to the system administrator when the system is brought up or taken down. This notification does not take into account how many nodes are in the cluster. As long as at least one node is active, the system is up. When all nodes are down, the system is down.
- Database schema validation upon startup. When a database-driven application is brought up, it is a good practice for the middle tier servers to verify whether the version of the business logic they implement matches the database schema.
- Sending recurring notifications to system users. For example, a calendar application might send out email prior to each instance of a scheduled recurring meeting.
- Load balancing of queued tasks. It's popular to use a single coordinator that distributes tasks among nodes in a cluster.
- Fault tolerance. If an application uses a distributed cache, it is common to designate a single master node responsible for maintaining a current copy of distributed states. The other nodes make requests of the current master node. When the master fails, another node takes over its responsibilities.

While it is fairly easy to implement such singleton tasks in a single VM, the solution will usually not work immediately in a clustered environment. Even in the simple case of a task activated upon startup on one of the nodes in a two-node cluster, several problems must be addressed:

- When the application is started simultaneously on both nodes, which VM should run the singleton task?
- When the application is started on one node and then started later on another, how does the second node know not to run the singleton task again?
- When the node that started the task fails, how does another node know to resume the task?
- When the node that started the task fails, but later recovers, how do we ensure that the task remains running on only one of the nodes?

The logic to solve these problems is unlikely to be included in the design of a single-VM solution. However, a solution can be found to address the case at hand and it can be patched on the startup task. This is an acceptable approach for a few startup tasks and two-node clusters.

As the application grows and becomes more successful, more startup tasks may be necessary. The application may also need to scale to more than two nodes. The clustered singleton problem can quickly become mind-boggling for larger clusters, where the different node startup scenarios are far more difficult to enumerate than in the two-node case. Another complicating factor is communication efficiency. While two nodes can directly connect to each other and negotiate, 10 nodes will have to establish 45 total connections to use the same technique.

This is where JBoss comes in handy. It eliminates most of the complexity and allows application developers to focus on building singleton services regardless of the cluster topology.

We will illustrate how the JBoss clustered singleton facility works with an example. First, we will need a service archive descriptor. Let's use the one that ships with JBoss under server/all/farm/cluster-examples-service.xml. The following is an excerpt:

    <!--    | This MBean is an example of a cluster Singleton    -->
    <mbean code="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
              name="jboss.examples:service=HASingletonMBeanExample">
    </mbean>
    <!- - -->
       <!--    | This is a singleton controller which works similarly to the
               | SchedulerProvider (when a MBean target is used)    -->
       <mbean code="org.jboss.ha.singleton.HASingletonController"
              name="jboss.examples:service=HASingletonMBeanExample-HASingletonController">
          <depends>jboss:service=DefaultPartition</depends>
          <depends>jboss.examples:service=HASingletonMBeanExample</depends>
          <attribute name="TargetName">jboss:service=HASingletonMBeanExample</attribute>
          <attribute name="TargetStartMethod">startSingleton</attribute>
          <attribute name="TargetStopMethod">stopSingleton</attribute>
       </mbean>
    <!- - -->

This file declares two MBeans, `HASingletonMBeanExample` and `HASingletonController`. The first one is a singleton service that contains the custom code. It is a simple JavaBean with the following source code:

    public class HASingletonMBeanExample
      implements HASingletonMBeanExampleMBean {

        private boolean isMasterNode = false;

        public void startSingleton() {
            isMasterNode = true;
        }

        public boolean isMasterNode() {
            return isMasterNode;
        }

        public void stopSingleton() {
            isMasterNode = false;
        }
    }

All of the custom logic for this particular singleton service is contained within this class. Our example is not too useful; it simply indicates, via the `isMasterNode` member variable, whether the master node is running the singleton. This value will be true only on the one node in the cluster where it is deployed.

`HASingletonMBeanExampleMBean` exposes this variable as an MBean attribute. It also exposes `startSingleton()` and `stopSingleton()` as managed MBean operations. These methods control the lifecycle of the singleton service. JBoss invokes them automatically when a new master node is elected.

How does JBoss control the singleton lifecycle throughout the cluster? The answer to this question is in the MBean declarations. Notice that the `HASingletonMBeanExample-HASingletonController` MBean also takes the name of the sample singleton MBean and its start and stop methods.

On each node in the cluster where these MBeans are deployed, the controller will work with all of the other controllers with the same MBean name deployed in the same cluster partition to oversee the lifecycle of the singleton. The controllers are responsible for tracking the cluster topology. Their job is to elect the master node of the singleton upon startup, as well as to elect a new master should the current one fail or shut down. In the latter case, when the master node shuts down gracefully, the controllers will wait for the singleton to stop before starting another instance on the new master node.

A singleton service is scoped in a certain cluster partition via its controller. Notice that, in the declaration above, the controller MBean depends on the MBean service `DefaultPartition`. If the partition where the singleton should run is different than the default, its name can be provided to the controller via the MBean attribute `PartitionName`.

Clustered singletons are usually deployed via the JBoss farming service. To test this example, just drop the service file above in the server/all/farm directory. You should be able to see the following in the JBoss JMX web console:

Figure 2. Controller MBean view. The `MasterNode` attribute will have value `True` on only one of the nodes.

Figure 3. Sample singleton MBean view. The `MasterNode` attribute will have the same value as the `MasterNode` attribute on the controller MBean.

### Scheduler Service

The JBoss 3 Scheduler service is covered in detail by the JBoss 3: Administration and Development book. It has three interdependent components:

- `ScheduleManager`, which serves as a centralized registry for registering and executing schedules.
- `ScheduleProvider`, which abstracts the custom logic that creates schedules.
- `Schedulable`, which represents a task that is executed at scheduled times.

Often, applications need to schedule tasks that have to be executed once in the scope of the application, whether it is running standalone or in a cluster of multiple nodes. Examples of such tasks include regular database cleanup, email notifications, and scheduled reports.

All JBoss `ScheduleProvider` services accept an MBean attribute that enables them to schedule tasks on only one node in the cluster (of one or more nodes). The attribute name is `HASingleton` and its value is a Boolean type. When set to `True`, all `ScheduleProvider` MBeans registered with the same name and deployed in the same cluster partition will coordinate and make sure that the schedule is only provided to the schedule manager on one of the nodes. When set to `False`, each of the schedule providers will act independently and as a result, they will all schedule tasks with their local scheduler managers. The default value of the attribute is `True`, which allows transparent transition of standalone schedule providers to a clustered environment.

The name of the partition where a singleton schedule provider service will be deployed can be set via the `PartitionName` attribute. By default, the value assumes the default JBoss partition name.

Here is an excerpt from the example service archive descriptor cluster-examples-service.xml, located in the server/all/farm directory in the JBoss installation.

    <!--
     | This MBean is an example of an HA Schedule Target
     | which is identical to a regular Schedule Target
     | (the example class is the same, just the MBean has different names)
     - ->
    <mbean code="org.jboss.varia.scheduler.example.SchedulableMBeanExample"
           name="jboss.examples:service=HASchedulableMBeanExample">
    </mbean>
    <!- - -->

    <!--
     | The Schedule Manager has to be started whenever
     | schedules are needed.
     |
     | Uncomment only if not started by
     | another service (e.g. schedule-manager-service.xml)
     - ->
    <mbean code="org.jboss.varia.scheduler.ScheduleManager"
           name="jboss:service=ScheduleManager">
       <attribute name="StartAtStartup">true</attribute>
    </mbean>
    <!- - -->

    <!--
     | This is a single schedule Provider which works like the
     | one in schedule-manager-service.xml
     |
     | The key difference is the explicit use of the HASingleton MBean attribute
     | to make the provider a clustered singleton.
     | When HASingleton is set to true the MBean will usually declare dependency
     | on a cluster partition. In this case it is the DefaultPartition.
     | When not explicitly set the attribute defaults to true.
     |
     | The same attribute can also be used for the other schedule providers as well:
     | DBScheduleProvider and XMLScheduleProvider
     |
     |
     - ->
    <mbean code="org.jboss.varia.scheduler.SingleScheduleProvider"
           name="jboss:service=HASingleScheduleProvider">
       <depends>jboss:service=DefaultPartition</depends>
       <depends>jboss:service=ScheduleManager</depends>
       <depends>jboss.examples:service=HASchedulableMBeanExample</depends>
       <attribute name="HASingleton">true</attribute>
       <attribute name="ScheduleManagerName">jboss:service=ScheduleManager</attribute>
       <attribute name="TargetName">jboss:service=HASchedulableMBeanExample</attribute>
       <attribute name="TargetMethod">
            ;hit( NOTIFICATION, DATE, REPETITIONS, SCHEDULER_NAME, java.lang.String )
       </attribute>
       <attribute name="DateFormat"></attribute>
       <attribute name="StartDate">NOW</attribute>
       <attribute name="Period">10000</attribute>
       <attribute name="Repetitions">10</attribute>
    </mbean>
    <!- - -->

The deployment descriptor above is almost identical to the one provided by schedule-manager-service.xml but is intended to be deployed via farming. Although not necessary, since the default value is `True`, the descriptor specifies the `HASingleton` attribute of the schedule provider for illustration of its usage.

### Notification Service

When it comes to reliable, mission-critical, and sophisticated J2EE messaging, one API stands out: JMS. The JBossMQ service is a robust JMS implementation that meets the high standards commanded by the API for distributed, transactional, and secure messaging. Then why another notification service?

Most typical clustered applications are deployed in a symmetric, homogeneous environment where components need to notify each other about various lifecycle or domain-change events, much like AWT and Swing widgets exchange events. This is where the JBoss cluster notifications come in. They fill the need for reliable, quick, and lightweight events. Cluster notifications do not require new skills from developers familiar with JMX, because they comply with the JMX notifications API.

Figure 4. Clustered notification service

If you have used JMX notifications before, the clustering extension should be a breeze. There are two common ways to employ clustered notifications: by extending `HAServiceMBeanSupport` or by delegating the work to instances of this class.

Let's look at an example that comes with the JBoss distribution. Here is an excerpt from the familiar descriptor cluster-examples-service.xml, located in server/all/farm.

    <!--
     | This MBean is an example showing how to extend a cluster notification broadcaster
     | Use the sendNotiication() operation to trigger new clustered notifications.
     | Observe the status of each instance of this mbean in the participating cluster partition nodes.
     - ->
    <mbean code="org.jboss.ha.jmx.examples.HANotificationBroadcasterExample"
    	name="jboss.examples:service=HANotificationBroadcasterExample">
    	<depends>jboss:service=DefaultPartition</depends>
    </mbean>
    <!- - -->

    <!--
     | This MBean is an example that shows how to delegate notification services
     |    to a HANotificationBroadcaster.
     | Use the sendNotiication() operation to trigger new clustered notifications.
     | Observe the status of each instance of this mbean in the participating cluster partition nodes.
     - ->
    <mbean code="org.jboss.ha.jmx.examples.HANotificationBroadcasterClientExample"
    	name="jboss.examples:service=HANotificationBroadcasterClientExample">
    	<depends>jboss.examples:service=HANotificationBroadcasterExample</depends>
    	<attribute name="HANotificationBroadcasterName">
                  jboss.examples:service=HANotificationBroadcasterExample
        </attribute>
    </mbean>
    <!- - -->

The MBean implemented by the `HANotificationBroadcasterExample` class provides common clustering services like shared distributed state, replication management, and notifications. It is a convenience class that can be extended and customized or used as is. Its most important attribute is `PartitionName`, which fixates the partition of nodes where all MBeans with the same name will work together.

The other MBean in this example is implemented by the `HANotificationBroadcasterClientExample` class. It uses the former MBean's services to broadcast notifications to the local listeners, as well as remote cluster listeners that have subscribed to the `HANotificationBroadcasterExample` MBean. At the same time, the client MBean will also receive all notifications sent through any instance of the broadcaster. Clear as mud?

Maybe a picture will help. Here is a screenshot of the client MBean attributes:

Figure 5. `HANotification Client Attributes`

The first attribute points to the name of the cluster notification broadcaster to which we will subscribe. The next attribute of interest is `ReceivedNotifications`. When a message is sent to any deployed instance of the broadcaster MBean on any of the partition nodes, it will be received and listed in the `Value` column for this attribute. In the screenshot above, there are two received notifications, `iivanov2` and `mau`. Even though it is not obvious from the picture, one of these messages was received from a remote host.

Now, a screenshot of the MBean operation of interest:

Figure 6. `HANotification` client operation

The name is not very friendly, but it is distinct and will do for this example. When this operation is invoked with a text message as its argument, the value will appear in the `ReceivedNotification` attribute on each of the client MBeans deployed in the same partition.

Let's look at the code of the client MBean to see that it is actually straightforward. I made an attempt to annotate the code and keep it simple, so that there is no need for additional analysis.

    public class HANotificationBroadcasterClientExample
      extends ServiceMBeanSupport
      implements HANotificationBroadcasterClientExampleMBean, NotificationListener
    {

      /**
       *
       * On service start, subscribes to notification sent by this broadcaster or
       * its remote peers.
       *
       */
      protected void startService() throws Exception
      {
        super.startService();
        addHANotificationListener(this);
      }

      /**
       *
       * On service stop, unsubscribes to notification sent by this broadcaster or
       * its remote peers.
       *
       */
      protected void stopService() throws Exception
      {
        removeHANotificationListener(this);
        super.stopService();
      }

      /**
       * Broadcasts a notification to the cluster partition.
       *
       * This example does not ensure that a notification sequence number
       * is unique throughout the partition.
       *
       */
      public void sendTextMessageViaHANBExample(String message)
        throws InstanceNotFoundException, MBeanException, ReflectionException
      {
        long now = System.currentTimeMillis();
        Notification notification =
          new Notification(
            "hanotification.example.counter",
            super.getServiceName(),
            now,
            now,
            message);
        server.invoke(
            broadcasterName_,
            "sendNotification",
            new Object[] { notification },
            new String[] { Notification.class.getName() }
            );
      }

      /**
       * Lists the notifications received on the cluster partition
       */
      public Collection getReceivedNotifications()
      {
        return messages_;
      }

      /**
       * @return the name of the broadcaster MBean
       */
      public String getHANotificationBroadcasterName()
      {
        return broadcasterName_ == null ? null : broadcasterName_.toString();
      }

      /**
       *
       * Sets the name of the broadcaster MBean.
       *
       * @param
       */
      public void setHANotificationBroadcasterName(String newBroadcasterName)
        throws InvalidParameterException
      {
        if (newBroadcasterName == null)
        {
          throw new InvalidParameterException("Broadcaster MBean must be specified");
        }
        try
        {
          broadcasterName_ = new ObjectName(newBroadcasterName);
        }
        catch (MalformedObjectNameException mone)
        {
          log.error("Broadcaster MBean Object Name is malformed", mone);
          throw new InvalidParameterException("Broadcaster MBean is not correctly formatted");
        }
      }

      protected void addHANotificationListener(NotificationListener listener)
                                             throws InstanceNotFoundException
      {
        server.addNotificationListener(broadcasterName_, listener,
          /* no need for filter */ null,
          /* no handback object */ null);
      }

      protected void removeHANotificationListener(NotificationListener listener)
                     throws InstanceNotFoundException, ListenerNotFoundException
      {
        server.removeNotificationListener(broadcasterName_, listener);
      }

      public void handleNotification(
        Notification notification,
        java.lang.Object handback)
      {
        messages_.add(notification.getMessage());
      }

      // Attributes ----------------------------------------------

      Collection messages_ = new LinkedList();

      /**
       * The broadcaster MBean that this class listens to and delegates HA notifications to
       */
      ObjectName broadcasterName_ = null;

    }

The key points of the code are highlighted. Notice how subscription to the cluster broadcaster is accomplished via the MBean server API.

It is also important to note that the subscription is local. Both the client and the MBean server reside in the same VM. The `sendNotification()` invocation is also local to the VM. The cluster broadcaster hides the implementation details of working together with its remote peers to deliver notifications throughout all nodes.

If you aren't too concerned with dependencies on JBoss-specific classes, you can directly extend the cluster broadcaster class, in which case all method calls will be direct instead of proxied through the MBean Server. Here is the code for an extended broadcaster:

    public class HANotificationBroadcasterExample
      extends HAServiceMBeanSupport
      implements HANotificationBroadcasterExampleMBean
    {

      /**
       *
       * On service start, subscribes to notification sent by this broadcaster or
       * its remote peers.
       *
       */
      protected void startService() throws Exception
      {
        super.startService();
        addNotificationListener(listener_, /* no need for filter */ null, /* no handback object */ null);
      }

      /**
       *
       * On service stop, unsubscribes to notification sent by this broadcaster or
       * its remote peers.
       *
       */
      protected void stopService() throws Exception
      {
        removeNotificationListener(listener_);
        super.stopService();
      }

      /**
       * Broadcasts a notification to the cluster partition.
       *
       * This example does not ensure that a notification sequence number
       * is unique throughout the partition.
       *
       */
      public void sendTextMessage(String message)
      {
        long now = System.currentTimeMillis();
        Notification notification =
          new Notification("hanotification.example.counter", super.getServiceName(), now, now, message);
        sendNotification(notification);
      }

      /**
       * Lists the notifications received on the cluster partition
       */
      public Collection getReceivedNotifications()
      {
        return messages_;
      }

      Collection messages_ = new LinkedList();

      NotificationListener listener_ = new NotificationListener()
      {
        public void handleNotification(Notification notification,
                                       java.lang.Object handback)
         {
           messages_.add( notification.getMessage() );
         }
      };
    }

This class behaves similarly to the previous one. If you examine its MBean View in the JMX console, you will notice that it has the same `ReceivedNotifications` attribute. The messaging operation is called `sendTextMessage()`.

### Conclusion

This text shows more examples where clustering is both powerful and complex. Fortunately, JBoss eliminates much of the complexity, allowing developers to enjoy the benefits of high availability. Writing singleton services in a distributed environment with JBoss is almost as easy as writing single VM singletons. If you previously invested in the JBoss Scheduler for a standalone server environment, you can keep your code and make it automatically valid in a clustered environment. Finally, lightweight notifications are no longer the prerogative of standalone applications. JBoss cluster notifications are yet another step towards practical horizontal scalability.

Ivelin Ivanov needs a better biography.

Return to ONJava.com.

oreillynet.com Copyright © 2003 O'Reilly & Associates, Inc.