WebLogic thread monitoring is very important for all slow or hanging application server issues. All WebLogic (middleware) requests are processed by thread pool thus it becomes a very important place to check for the problems. In Oracle DBA terms it’s very similar to checking database active sessions.
Weblogic server architecture
Weblogic server has a centralized AdminServe that allows deployment and configuration of resources like JNDI configuration, JMS (Java Message Services), Data Source (database connection pools). WebLogic server can have multiple managed servers. All managed servers can co-exist individually and they can be clustered as well.
Each WebLogic server processes different classes of work in different queues, it is based on priority and ordering requirements and to avoid deadlocks.
Weblogic server uses a single thread pool, in which all types of work are executed. Weblogic server prioritizes work based on the rules and run-time metrics and it automatically adjusts the number of threads. The queue monitors throughput over time and based on history data it adjusts the number of threads. It can increase and decrease the number of treads of Weblogic managed server.
For each Managed server and AdminServer we can monitor the threads, which is an effective way to monitor the workload on Weblogic servers.
The Admin Console
The most common mechanism to monitor Weblogic server is the Administrative console, which runs on AdminServer. From console we can administer each managed servers of the domain, like startup/shutdown and configuration of various properties.
We can login using the Weblogic username and password. Administrator superuser is usually weblogic user, which is created while creating domain. Using this account we can create other accounts which can be administrator or less privilege users for only monitoring purpose.
After logging in you will see following screen. To check AdminServer or Managed Server click on Servers below Environment. (Or on left hand panel Expand Environment, and click on Servers)
Once you click on Servers you will see following screenshot. Where you can see AdminServer and all the managed serves of domain. Here in this example managed servers are grouped in four different clusters. Managed servers can run on single or different machines. Click on Managed Server you want to monitor.
Once you click on server, to monitor threads click on Monitoring and then Threads tab and you will see following screen.
On above screen, you can monitor further threads by clicking Next (On right hand top corner of the Table: Self-Tuning Thread Pool Threads).
You can customize table to display all of the threads on one single page by clicking Customize this table and select Number of rows displayed per page value from drop down. You also can select or de-select columns to display in table.
Console will remember this setting when login next time for the specific managed server.
To check issues of slow application response or hanging application you should be interested in Hogger and Stuck threads.
Let’s understand about thread state.
Active: The number of active execute threads in the pool (shown in Self Tuning thread pool table first column).
Total: Total number of threads.
Idle: Threads that are ready to pick up new work.
Standby: These threads are activated when more threads are needed, these threads remain in standby pool and activated when needed.
Hogging: Thread held by a request. These can be potential stuck threads after the configured timeout or it will return to pool before that.
Stuck: Thread is stuck working on request for more than configured stuck thread maximum time. You can imagine Stuck thread as long running job. Once the condition causing the stuck threads is cleared they will disappear and back to thread pool.
The Managed server status will show Health as Warning when you have Stuck threads in thread pool.
To identify potential issues or to check Stuck threads, there are various methods:
On Admin console, sort the Thread pool table based on Stuck in descending order, or you can check threads with Stuck column value as True and look at the Current request column and thread ID. Using thread ID, you can able to check what this thread was doing before became STUCK.
You will have complete detail of Stuck thread in the managed server log file on the server node. In the managed server log, look for string ‘BEA-000337’ or ‘STUCK’ word and timestamp. The log entry will show all about the request and potential problem or current problem. Advantage of log file is we can check the historical STUCK thread occurrences as well.
To understand more about STUCK threads, you can take thread dump by clicking button “Dump Thread Stacks” on thread monitoring page and you will have complete dump of all threads in pool. You can then locate all the STUCK threads in dump and you can understand more about STUCK threads.
By monitoring WebLogic servers using Admin Console or by checking log files you can identify potential issues or ongoing problems of slow or hanging application.
To learn more, or for assistance with any Weblogic issues, contact Cintra today!
Written by Dilip Patel, Senior Oracle Apps DBA, Cintra UK – May 2017