Under the hood of GlassFish v2 Clustering

GlassFish v2 is a production-ready Java EE 5 application server currently in beta. There was a lot of interest about GlassFish this year during JavaOne. I had a chance to talk to many customers interested in GlassFish. Some were already using Sun’s distribution and dealing with very large deployments. In this blog, I will share few tricks and tips that may come in handy while dealing with GlassFish v2.

First, a quick re-cap. GlassFish v2 has support for profiles (OnePager, blog – [1], [2]) where the server installation is configured for different usage. Three profiles are available by default: developer, cluster, and enterprise. Developers would typically work with a single Java EE server instance and use the developer profile. Cluster and enterprise profiles are targeted for deployments.

A typical GlassFish v2 installation [in cluster or enterprise profile] consists of one or more administration domain. A domain provides a common authentication and administration point for a collection of Java EE server instances. Domain Administration Server (DAS) handles all the administrative operations in a domain.

Each remote machine [in cluster or enterprise profile] has a light weight process called Node Agent. DAS interacts with the Node Agent to manage the life-cycle operations (create, delete, start, stop) in a remote machine. Node Agents and remote Java EE server instances synchronizes with DAS during startup to get the latest configuration from central repository managed by DAS.

GlassFish v2 supports the concept of cluster where multiple Java EE server instances can be grouped for horizontal scalability. GlassFish v2 also supports in-memory replication feature where state of a deployed application is replicated to a peer in the cluster and thus enables the application to be highly available. Each Java EE server instance in a cluster shares the same set of applications and configuration information.

To summarize, GlassFish v2 offers enterprise features such as clustering (screencast, blog), high availability (screencast), and load balancing (GF page, blog) to support highly scalable, volume enterprise deployments for service-oriented architecture (SOA) and Web 2.0 applications. Refer to GlassFish v2 architecture wiki and JavaOne session TS-4436: Technical Overview of GlassFish v2 (abstract, blog) for more details.

Q: Does GlassFish v2 support hot deployment to a cluster?

Yes, GlassFish v2 offers multi-phase deployment to cluster and minimizes deployment related failures to distributed machines. Use the –target option in asadmin deploy command (example, %asadmin deploy –target cluster1 foo.war) or the Administration Console. Deployment is dynamic; i.e., deployed application bits are transferred automatically to the remote server instances. You don’t need to restart the cluster after deploying an application. v2 also offers other deployment features such as auto-deploy, directory deploy, ant integration (blog), NetBeans Integration, etc. Refer to deployment documentation for more details.

Q: Is it possible to deploy the same application to multiple clusters?

Yes. Use asadmin create-application-ref (example, %asadmin create-application-ref –target cluster2 myApp
) and asadmin delete-application-ref commands with –target option in CLI or manage target for the deployed application in Administration Console.

Q: Is there a way to start a remote server instance when DAS is not running?

Yes, you may use the startserv/stopserv scripts under the bin directory (blog) or re-start the node agent. asadmin start-node-agent command will automatically start the remote server instances without synchronizing with DAS.

Q: How do I force a full synchronization of a remote server instance?

A hidden file named .com_sun_appserv_timestamp keeps track of the last successful synchronization time of DAS. If you remove these files manually (not an official interface) under the remote server instance(s), a full synchronization will happen during the next server re-start. If synchronization fails for any reason, these files are automatically removed to force a full synchronization. In GlassFish v2, significant improvement was made in synchronization performance (~40% – your mileage may vary).

The configuration under remote instances are treated as cache (for example, all files under nodeagents/na1/server1). In extreme cases, if you remove all files of a remote server instance and restart the node agent, the remote server instance (for example, server1) will be recreated and all necessary files will be synchronized. Refer to synchronization documentation for more details.

Q: Is there a way to backup/clone DAS so that it is not a single point of failure for management?

DAS has no role in application runtime functionality. So, when DAS is un-available, the cluster will continue to run. You may use %asadmin backup-domain command to create a backup of DAS. After that, please backup the zip from backup-domain command in a highly available storage. When /if DAS‘s machine crashes, you may use %asadmin restore-domain command with the previously backed-up zip to restore the domain in another machine. During DAS‘s startup, it will ping all the node agents to notify the new co-ordinates of DAS. Solaris [Sun] Cluster has support for Sun Java System Application Server. You may use it to provide automatic failover. Refer to this documentation for more details.

Q: Is there a way to specify machine specific JVM options for a server instance in a cluster?

Yes. You may use tokens (${token}) in the config element for a cluster and define the token property with different values for different server instance. Refer to administration reference guide for more details. Here is a quick example where we are setting max heap size for a server instance that is part of a cluster.

[domain ...]
   [applications]...[/applications]
   [resources]...[/resources]

   [configs]
     [config ... name="cluster1-config"]
     ...
     [java-config ...]
         [jvm-options]${MAX_HEAP_SIZE}[/jvm-options]
      [/java-config]
     [/config]
   [/configs]

  [servers]
    [server ... config-ref="cluster1-config" name="server1"]
      [system-property name="MAX_HEAP_SIZE" value="-Xmx512m"/]
    [/server]
  [/servers]

  [clusters]...[/clusters]
  [node-agents]...[/node-agents]

  [lb-configs]...[/lb-configs]
  [load-balancers]...[/load-balancers]

[/domain]

Q: Is it possible to specify JNDI name at cluster scope? For example, is it possible to use two different connection pools for an application that is deployed to two clusters?

Yes. You may use the same tokenization strategy described above. Create a token for JDBC resource pool name and then specify the token property with desired (different) values for the two clusters. When application code looks up the JNDI name, it will end up connecting to two different database pools for the two different clusters. For example,

   [jdbc-resource ... jndi-name="jdbc/db" pool-name="${POOL_NAME}"]
   [cluster ... name="cluster1"]
     [system-property name="POOL_NAME" value="DerbyPool"/]
   [/cluster]
   [cluster ... name="cluster2"]
     [system-property name="POOL_NAME" value="MySQLPool"/]
   [/cluster]

Q: Say, I am dealing with a large deployment involving many clusters with multiple server instances (> 30). Is there a way to optimize DAS’s foot-print?

During the cluster creation, instead of cloning the config for each cluster, use a reference to an existing config (example, cluster1-config). If you need to customize a value, use tokens and define them in each cluster. This will reduce the overall memory foot-print. Also, avoid deploying the applications to DAS (on Administration Console, target named server) unnecessarily. You may consider creating new domains (%asadmin create-domain) when appropriate.

Q: Is there a way share a library across a cluster?

Yes. If you copy the library jar under the cluster’s config/lib directory (example, domains/domain1/config/cluster1-config/lib), the library will be available cluster wide. You will need to update the classpath-prefix or classpath-suffix. Refer to Sivakumar’s blog for more classloader related issues. According to Siva’s blog, library jars can be shared at domain and application scope also in GlassFish v2.

• domains/domain1/lib – domain wide scope, common classloader add the jars automatically
• domains/domain1/config/cluster1-config/lib – config wide, update classpath-prefix or classpath-suffix
• domains/domain1/lib/applibs – application scope, added to application class loader automatically
• domains/domain1/config/cluster1-config/lib/ext – adds to java.ext.dirs automatically

Q: Do you need special packaging of applications to use them in cluster?

No. Application code level changes are not needed. GlassFish v2 offers in-memory replication for high-availability with almost zero configuration also.

Q: Is it possible to install different versions on the same machine?

For GlassFish, this is not an issue. For SJS AS 9.1, you may also use the filebased installer. For native package based installation, SJS AS 8.x will be upgraded to SJS AS 9.1 on the same machine.

Q: Do you have support for monitoring?

GlassFish v2 offers a wide range of monitoring statistics. Statistics on server log, server runtime, deployed applications, resources and transactions are available. Refer to the following blogs ([1], [2]) for more details. Call Flow (tech tip) allows you to find out how a particular application response is being processed across different containers of the Application Server. Web services monitoring is also available (article) along with new improvements in CLI (blog) for mornitoring.

Q: Do you have any support for large data center deployments?

Yes, you may use N1 Service Provisioning System with SJS AS. We are working on a new plugin for SJS AS 9.1 (Sun’s distribution of GlassFish v2) also.

JDBC Update:
Refer to these great blogs ([1], [2], [3], [4]) for JDBC connection pool related issues.