Configuring remote JMX access for Weblogic Server

Posted on December 2nd, 2009 by roland

In our series of articles about configuring remote JMX access for the jmx4perl proxy mode, this article tackles how to enable JMX remoting for Weblogic Server 9 and 10. It is not specific to jmx4perl and explains several different setups and possible problems.

But before we start, kudos to this excellent blog post and Weblogic’s own documentation which helped me quite a lot during my journey through the depth of Weblogic JMX export.

As worked out in a previous post JBoss is a bit of a mess when it to comes to export the JDK’s PlatfomMBeanServer. As it turns out for Weblogic things are not much easier (but there are ‘workarounds’). Weblogic even doubles the number of available MBeanServers to four.

MBean Servers in Weblogic

Weblogic comes with three own MBeanServers, which are exported via RMI/IIOP as JSR-160 connectors. They can be looked up via a certain JNDI name as shown in the table below. Additionally, there is the ubiquitous PlatformMBeanServer which can be exported the usual way as described later in this post.

MBean Server	JNDI Name
Domain Runtime MBean Server	`weblogic.management.mbeanservers.domainruntime`
Runtime MBean Server	`weblogic.management.mbeanservers.runtime`
Edit MBean Server	`weblogic.management.mbeanservers.edit`
PlatformMBeanServer	—

The Runtime MBean Server specifies an individual application server, whereas the Domain Runtime MBean Server exposes the MBeans for all servers in a cluster. The Edit MBean Server is used for accessing and modifying the domain configuration. The Weblogic documentation contains further details.

When using jmx4perl’s proxy mode you have to choose the MBeanServer in advance (in contrast to the agent mode, where MBeanServers are merged to one virtual MBeanServer).

Two access modes

There are two ways how the MBeanServers mentioned above can be exported for remote access:

Via RMI/IIOP exported by Weblogic. This way, the three Weblogic MBeanServers (those with an JNDI name) can be exported (but not the PlatformMBeanServer directly). Advantage of this export is that it can be enabled with the admin console and that it includes the complete Weblogic security stack.
Via RMI/JRMP exported by the JVM. This allows for the PlatformMBeanServer to be exported (but not the other, Weblogic specific MBeanServers). It gets enabled as usual by setting certain java defines as startup options and is secured the JDK way. A forthcoming blog will clarify how to setup security for JDK exported JSR-160 connectors.

Both methods are explained in detail in the following sections.

RMI/IIOP exported by Weblogic

First of all, some configuration items must be set to enable IIOP exported MBeans. In the admin console, check that following properties are set:

First, IIOP must be enabled: Domain (‘wl_server’) / Environment / Servers -> Server (‘examplesServer’) -> Protocols -> IIOP -> Enable IIOP
Allow for anonymous read access, if you want to monitor without sending credentials: Domain (‘wl_server’) -> Security -> General -> Anonymous Admin Lookup Enabled
If you want to secure IIOP access (or want to have write access), set the name and password of the default user: Domain (‘wl_server’) / Environment / Servers -> Server (‘examplesServer’) -> Protocols -> IIOP -> (Advanced) -> “Default IIOP Username” and “Default IIOP Password”

You need to restart the server if you change one of the options above. The JMX service URL for accessing Weblogic JMX connectors via IIOP looks like

service:jmx:rmi:///jndi/iiop://<server address>:<port>/<jndi name>

where the JNDI name is one of the MBeanServer’s JNDI name as described above. For example:

 service:jmx:iiop:///jndi/iiop://bhut:7001/weblogic.management.mbeanservers.runtime

Now you are ready to access you Weblogic MBeans via jmx4perl:

jmx4perl http://proxy:8888/j4p  \ 
      --target \ 
service:jmx:iiop:///jndi/iiop://target:7001/weblogic.management.mbeanservers.runtime \ 
      --target-user weblogic \
      --target-password weblogic \
      list

Here proxy:8888 specifies the server, where j4p.war is running as a proxy servlet and target is the remote JMX enabled Weblogic server. Please note, that the usual info command of jmx4perl won’t work here, since we don’t have access to any Java 5 MXBeans (like the MemoryMBean). But see below for how to make this work, too.

There are some issues, though.

Same Java version for j4p proxy and Weblogic

There is one important point you should take care of if you connect to the target platform via IIOP. As described in this bug report, there are issues when Java 5 and Java 6 virtual machines are communicating via IIOP. A typical error happening in such a scenario looks like:

026030 ERROR               STDERR| Nov 28, 2009 1:21:51 PM 
    com.sun.corba.se.impl.io.InputStreamHook  
    throwOptionalDataIncompatibleException
    WARNING: "IOP00800008: (MARSHAL) Not enough space left 
                                     in current chunk"
    org.omg.CORBA.MARSHAL:   vmcid: OMG  minor code: 8  completed: No
  at com.sun.corba.se.impl.logging.
       OMGSystemException.rmiiiopOptionalDataIncompatible2
          (OMGSystemException.java:2709)
.....
  at org.omg.stub.javax.management.
       remote.rmi._RMIConnection_Stub.getMBeanInfo(Unknown Source)
  at javax.management.remote.rmi.RMIConnector
     $RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1052)

The only known workaround for this problem is to run the proxy and the target server with the same JDK version (either both Java 5 or both Java 6). Since you have normally the freedom to choose the JDK for at least the proxy, this shouldn’t be a big problem.

ClassNotFoundException

As with every RMI based connection, you need to be sure that type definitions are available on both sides of the connection. As Weblogic export Weblogic specific data types via JMX you will encounter ClassCastExceptions when you access these remotely from a generic client (without Weblogic specific classes in the classpath):

  Caused by: java.lang.ClassNotFoundException: 
     weblogic.wsee.ws.dispatch.server.OneWayHandler 
        (no security manager: RMI class loader disabled)
     [exec]     at sun.rmi.server.LoaderHandler.loadClass
  ...

I.e. you won’t be able to access the MBeanServer via RMI/IIOP with jconsole due to missing local classes.

RMI/JRMP exported by JVM

As an alternative the usual way for exporting MBeans via RMI/JRMP can be used. This includes to set some java defines as startup options (probably within setDomainEnv.sh) like:

JAVA_OPTIONS="$JAVA_OPTIONS \ 
    -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.port=9999 \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.authenticate=false"

In this example we switched of security complete for sake of demonstration. You can now connect with jmx4perl or with your favorite JMX console as usual:

jmx4perl http://proxy:8888/j4p  \
   --target service:jmx:rmi:///jndi/rmi://target:9999/jmxrmi \ 
   read java.lang:type=Memory HeapMemoryUsage
 
jconsole service:jmx:rmi:///jndi/rmi://target:9999/jmxrmi

Unfortunately, by default, only the PlatformMBeanServer gets exported without any Weblogic specific MBean. However, there is a way to get to the Weblogic MBeans as described in the next section.

Using PlatformMBeanServer for Weblogic MBeans

For monitoring purposes having Weblogic runtime MBeans and JVM MXBeans in different MBeanServers which are exportable in different ways is quite annoying. However, for Weblogic there is a solution by configuring WLS to use the PlatformMBeanServer as it’s MBeanServer. With this configuration, your are able to access Weblogic MBeans from RMI/JRMP Service URLs (as it is used with ‘normal’ JMX clients like jconsole) and Java MXBeans from the RMI/IIOP connector used by Weblogic.

For this to work, you need to set Domain (‘wl_server’) -> Configuration -> General -> (Advanced) -> Platform MBeanServer enabled in the admin console for both WLS 9 and 10. Afterwards a server restart is required.

For WLS 10 an additional configuration parameter has to be set in order to let WLS use the PlatformMBeanServer for its runtime MBeans. The attribute PlatformMBeanServerUsed needs to be set to true on the JMXMBean (it is false by default). Unfortunately I didn’t find a way to set this attribute via the admin console, but only via the Weblogic scripting environment WLST. Assuming that your current working directory is you WLS 10 root directory, you should use wlst.sh to fire up the WLST in interactive mode and when the server is not running:

$ common/bin/wlst.sh
.....
wls:/offline> readDomain('samples/domains/wl_server')
wls:/offline/wl_server>cd('JMX')
wls:/offline/wl_server/JMX>ls()
drw-   NO_NAME_0
wls:/offline/wl_server/JMX>cd('NO_NAME_0')
wls:/offline/wl_server/JMX/NO_NAME_0>ls()
-rw-   CompatibilityMBeanServerEnabled               true
-rw-   DomainMBeanServerEnabled                      true
-rw-   EditMBeanServerEnabled                        true
-rw-   InvocationTimeoutSeconds                      0
-rw-   ManagementEJBEnabled                          true
-rw-   Name                                          null
-rw-   Notes                                         null
-rw-   PlatformMBeanServerEnabled                    false
-rw-   PlatformMBeanServerUsed                       false
-rw-   RuntimeMBeanServerEnabled                     true
wls:/offline/wl_server/JMX/NO_NAME_0>set('PlatformMBeanServerUsed','true')
wls:/offline/wl_server/JMX/NO_NAME_0>set('PlatformMBeanServerEnabled','true')
wls:/offline/wl_server/JMX/NO_NAME_0>updateDomain()
wls:/offline/wl_server/JMX/NO_NAME_0>closeDomain()
wls:/offline>exit()

The path to the domain (samples/domains/wl_server), an the name of the JMX-Bean (NO_NAME_0) might differ at your side, but I think, the idea is clear. As shown above you also can set PlatformMBeanServerEnabled via WLST as well.

Fire up the server and you should be ready for accessing WLS MBeans and JDK MXMBeans from within the same MBeanServer on Weblogic 10.

BTW, setting PlatformMBeanServerUsed via jmx4perl itself doesn’t work, because Weblogic needs some extra boilerplate (which I didn’t dive into) before configuration can be changed via JMX:

jmx4perl http://proxy:8888/j4p \ 
  --target \
     service:jmx:iiop:///jndi/iiop://target:7001/weblogic.management.mbeanservers.edit \
  --target-user weblogic \ 
  --target-password weblogic \
  write com.bea:Name=wl_server,Type=JMX PlatformMBeanServerUsed true
 
ERROR: org.omg.CORBA.UNKNOWN:   vmcid: 0x0  minor code: 0 completed: Maybe

and on the WLS console:

<Warning> <RMI> <BEA-080003> RuntimeException thrown by rmi server: 
  javax.management.remote.rmi.RMIConnectionImpl.setAttribute
  weblogic.management.NoAccessRuntimeException: 
    Operation can not be performed as caller has not started an edit session.
    at weblogic.management.mbeanservers.edit.internal.EditLockInterceptor.checkEditLock
    ...

Summary

For setting up Weblogic for remote JMX access quite some configuration is needed. There are traps, but at least for normal monitoring needs everything should work fine at the end. Nevertheless, if you have the chance to operate jmx4perl in agent mode (without the need for remote JSR-160 connectors), I still recommend to connect directly to the server via the j4p.war running as an agent and not as a proxy ;-).

Tags: J2EE, jmx, Jmx4Perl, jsr-160, Nagios
Filed under Jmx4Perl |

14 Responses to “Configuring remote JMX access for Weblogic Server”

Dwayne Says:
April 20th, 2010 at 7:06
Excellent article. I’m still having issues leveraging the proxy mode to a remote Weblogic server. I’m trying to do this via ‘iiop’ service calls, but I keep getting the following exception when trying to pull back an attribute from domainruntime:

ERROR: javax.management.InstanceNotFoundException: Unable to contact MBeanServer for com.bea:Name=MyServer,Location=MyServer,Type=ServerRuntime

IIOP is enabled and default user and pass have been set for the protocol.

Any insights?

Thanks!

[Reply]
Dwayne Reply:
April 22nd, 2010 at 23:31
Ok. . .In investigating further, I’ve observed that the ‘–target’ arguments appear to be ignored by the j4p.war proxying the request. When using a list operation, it responds only with those objects inside the container that hosting the war file. Also running a packet sniffer on the same server, filtering on the remote host and port for the targeted weblogic instance. . .nothing registers.

[Reply]
roland Reply:
June 7th, 2010 at 20:16
@Dwayne, thanks for your investigations. In fact, I didn’t tested the proxy mode with Weblogic hosting the agent servlet, only the ‘lighweight’ ones (tomcat, jetty). When times permit, I will fix this.

Glad to know that it works for you with tomcat ;-) (and sorry for the late answer, somehow I didn’t receive notifications on new comments from wordpress)

[Reply]
Dwayne Reply:
July 21st, 2010 at 20:24
@roland, Thanks for the response. . .

I’ve been tasked with getting this to work under Weblogic as the proxy server for several reasons. I tried again recently and noticed some additional behavior not mentioned previously. One, it appears that Weblogic is taking the credentials passed in for the remote Weblogic instance and trying authenticate with that (–target-user & –target-pass). This is under 11G. I’m not sure if check_jmx4perl is reversing the creds passed in. . .haven’t had a chance to inspect the Weblogic.pm file. Also, Tomcat has been exhausting it’s threads when the remote server is down for a reason. Tried passing in a timeout value via the check script but it appears to be ignored. . .it always waits 3 mins.

Thirdly, I’m wondering what the support will be for IIOP over SSL (proto iiops). Haven’t tested that yet, but it will be more important as more instances are only exposed over encrypted channels. Our current work around is to deploy the agent directly to the Admin server of the instance we want to get at and have it proxy requests to the managed nodes (SSL).

My lead has some concerns in terms of features that he would like to see added – is there a published roadmap for jmx4perl/j4p?

I’m really trying to keep this going in our team as it really is a comprehensive addon for Nagios – the best in it’s class as far as I’m concerned.

Hopefully, we continue our development around it.

No worries in regards to the late response as I’m well aware of how busy we all are.

Thanks again for contributing a really well through out solution for J2EE/Nagios integration.

-Dwayne

[Reply]
roland Reply:
July 21st, 2010 at 22:01
@Dwayne, if I understand the first issue right, it is that a Weblogic setup as proxy server user –target-user and –target-password for authenticating against the proxy (which should be –user and –password). Is this correct ? BTW, Weblogic.pm is not related with any proxying stuff, it is only used for resolving aliases. Proxying is done the same way for all app-servers.

For the timeout issue, this is probably a bug (frankly, I didn’t have any tests for the timeout option). I will investigate this soon. The thread exhaustion is probably due of too many threads hanging for this timeout.

Concerning the IIOP over SSL stuff for connecting to the target server, there is no much magic. The code for opening an remote JMX connection can be found in Jsr160RequestDispatcher:

String urlS = targetConfig.getUrl(); JMXServiceURL url = new JMXServiceURL(urlS); Map<String,Object> env = prepareEnv(targetConfig.getEnv()); JMXConnector ret = JMXConnectorFactory.newJMXConnector(url,env); ret.connect();

So, AFAIK it depends on the setup of the proxy and the target URL. I never tested it with IIOP/SSL, though. In case you try it, I would be keen on your results ;-)

For the roadmap: I just released version 0.70, which was quite a big release (although the version jump was quite minor). Especially the check_jmx4perl Nagios Plugin was in the focus, and IMO the most powerful feature in the new release are parameterized checks, provided via a configuration file (which is in some sense the successor for simple aliases).

For the next release, the major focus is on extracting the Java part in a new project (jolokia), since I realized that the agents are a story on their own and useful in a pure Java world on their own. Otherwise, I’m always keen on user feedback and the development is really demand driven. If the proxy mode needs to be polished and tested (frankly, although I have tons of integration tests for the agents themselves, the proxy mode was never high on the task list because of lack of feedback) and people want it, I will do it.

If you really want to get sure, that your stuff gets in, I’m in the lucky position, that I can offer commercial support via my employer. The detailed conditions would have to be worked out, though, but I’m confident that we can offer something reasonable with SLAs etc. (we already had the first consulting projects for jmx4perl). So for the future, the is a strong commitment to jmx4perl, it’s by far more than a simple hobby project.

[Reply]
roland Reply:
July 21st, 2010 at 22:12
I forgot this code snippet for setting up the environment of the connection:

private Map<String,Object> prepareEnv(Map<String, Object> pEnv) { if (pEnv == null || pEnv.size() == 0) { return pEnv; } Map<String,Object> ret = new HashMap<String, Object>(pEnv); String user = (String) ret.remove("user"); String password = (String) ret.remove("password"); if (user != null && password != null) { ret.put(Context.SECURITY_PRINCIPAL, user); ret.put(Context.SECURITY_CREDENTIALS, password); ret.put("jmx.remote.credentials",new String[] { user, password }); } return ret; }

Dwayne Reply:
July 21st, 2010 at 22:53
@roland, Thanks for the quick response!

In regards to the first issue, the only credentials I pass to the proxy are those for the remote instance via the –target-user and –target-password options. There is no special authentication setup where the war file is deployed. So for instance, I have two j4p instances running on the same server in my test environment – one is hosted on Tomcat – the other on Weblogic. When I fire a call via jmx4perl utility script to Tomcat as a proxied request, it’s successful. If I take the same command and change only the port number to hit the Weblogic instance, I get the following exception:

Caused by: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User weblogic javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied

I verified that the proxy server is where it stopped (tcpdump on the other server registered no hits). Here, the user ‘weblogic’ was passed in via –target-user. So I don’t understand why the war file would take that at try to auth against the local instance as opposed to the remote host. I’ll continue to investigate. . .was working with Oracle earlier this morning. . .

Looks like a lotta love went into the new release. I’m reviewing the deltas now. I’ll definitely review everything mentioned here with my lead.

In the meantime, I’ll continue reviewing the new release and share feedback with you once I have it.

-Dwayne
Dwayne Says:
April 27th, 2010 at 19:21
Got this working under Tomcat. . .originally had the war file deployed under Weblogic/jrocket. Might go and try Weblogic again some day under Sun’s JDK, but for now, I can work with this under Tomcat.

[Reply]
Robert Says:
May 7th, 2010 at 16:54
Yes, definitely good article. I’m trying to use WLST to access the PlatformMBeanServer. Is this possible? I have configured it like you suggest. Access should be possible given what you describe. However I can’t find the hook. It’s not under serverRuntime(), the JVMRuntime MBean only shows very basic Heap attributes.

[Reply]
Robert Says:
May 7th, 2010 at 17:12
found it :) It’s under custom Cheers, Robert

[Reply]
Herry Reply:
March 24th, 2011 at 11:28
@Robert, I’ve configured it as Roland suggested, but i still can’t find the PlatformMBeanServer from custom() or serverRuntime().

[Reply]
Srikanth Says:
April 14th, 2011 at 16:18
Awesome blog roland. Keep rocking….

[Reply]
Amit Jain Says:
September 13th, 2011 at 15:57
Hi, We are getting below exceptionCaused by: javax.naming.NamingException: Couldn’t connect to any host [Root exception is org.omg.CORBA.NO_PERMISSION: vmcid: 0x0 minor code: 0 completed: No] at weblogic.corba.j2ee.naming.Utils.wrapNamingException(Utils.java:83) at weblogic.corba.j2ee.naming.ORBHelper.getORBReferenceWithRetry(ORBHelper.java:605) at weblogic.corba.j2ee.naming.ORBHelper.getORBReference(ORBHelper.java:559) at weblogic.corba.j2ee.naming.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:85) at weblogic.corba.j2ee.naming.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:31) at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:46) at javax.naming.spi.NamingManager.getInitialContext(Unknown Source) at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source) at javax.naming.InitialContext.init(Unknown Source) at javax.naming.InitialContext.(Unknown Source)

please suggest

regards, Amit J.

[Reply]
Himanshu Reply:
January 17th, 2012 at 18:23
@Amit Jain,

I am also having same error when I am trying to access MBean registered in one of the manged servers from another managed server. The the Managed server 2 is not able to make any connection to JMX bean server in managed server 1. I can easily connect to managed server JMX Bean server from standalone java class and JConsole.

I am using IP address and port to connect to JMXConnectorFactory.

HK

[Reply]