Analyzing stuck threads on a native method

Ask Time：2016-06-01T22:26:05 Author：volatile

I'm using tomcat and java 1.7.0_80. The product is rule engine using drools. I'm having a slowdown after loading the system. I took a thread dump and see many threads that look like:

http-bio-9980-exec-48" daemon prio=10 tid=0x00007fa8b43a3000 nid=0x10299             runnable [0x00007fa9522c5000]
java.lang.Thread.State: RUNNABLE
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
    at java.lang.Class.getConstructor0(Class.java:2895)
    at java.lang.Class.newInstance(Class.java:354)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:77)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:46)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
.
.
.

I think these threads are stuck, so I did another thread dump (to see what happens on the native method):

00007faa1dabf6d5      __pthread_cond_wait + 0xc5
0x00007faa1cb638ad      _ZN15JvmtiRawMonitor9raw_enterEP6Thread + 0x25d
0x00007faa1cb3f63d      _ZN8JvmtiEnv15RawMonitorEnterEP15JvmtiRawMonitor + 0xad
0x00007faa1b7089a8      debugMonitorEnter + 0x38
0x00007faa1b6f5205      event_callback + 0xe5
0x00007faa1b6f5e3e      cbClassPrepare + 0x8e
0x00007faa1cb548c4      _ZN11JvmtiExport18post_class_prepareEP10JavaThreadP12klassOopDesc + 0x1b4
0x00007faa1ca1de8e      _ZN13instanceKlass15link_class_implE19instanceKlassHandlebP6Thread + 0x45e
0x00007faa1cae55c7      JVM_GetClassDeclaredConstructors + 0x1b7
0x00007faa180cac39      * java.lang.Class.getDeclaredConstructors0(boolean) bci:0 (Interpreted frame)

Can someone tell me why do I have contention in here?

Author:volatile，eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article：https://stackoverflow.com/questions/37571502/analyzing-stuck-threads-on-a-native-method

Mike Dunlavey :

Well, let me try to answer you. I'm going to make an assumption, that topmost entries are the innermost calls.\n\nIt's in reflect, invoking some method, it's hard to say what, but\nthat requires creating one or more methods and running them, and\nthat requires making an new instance of java.lang.Class, and\nthat requires getting a constructor of the class, and\nthat dips into the JVM, which\nhits an event that drops it into debugMonitor, which\nis in a thread wait, quite possibly\nhanging for user input in the debugger.\n\nSo perhaps you need to look at another thread?\nOr maybe this is the right one, and you just need to ignore the levels past the event callback.\nIn any event, I would look at that reflect to see if it's referring to your code.\nThat's what you need to find.\nIt does no good to go deep into system code that you would presume is working OK.",

2016-06-01T21:25:37

Analyzing stuck threads on a native method