Attaching to a Java Process fails with 'AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded'

Applies To

Java SE JDK and JRE - Version 8 and later

Introduction

While attempting to use any tools that use the Attach API to attach to a running Java process, such as jmap, jstack, jconsole, jinfo, Java Flight Recorder (JFR), etc., the following error is thrown:

<Error attaching VM with PID: <PID> - com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded>

This error occurs even if the tools are run with the same user (UID) and group (GID) as the target JVM.

Cause

The issue occurs when the target JVM’s socket file is missing, most likely because it was deleted by some other process. The Attach API relies on the target JVM having a socket file named .java_pid<PID> in the /tmp directory. The /tmp directory is hard-coded in the JVM and cannot be changed, even if you configure temporary files to use another directory. This /tmp/.java_pid<PID> file is created only once: either when the JVM starts or upon the first time attaching to the JVM. Once created, the JVM relies on that file as its only socket file for allowing any tools that use the Attach API to attach to it. If the file gets deleted by some other process, such as a /tmp sweeper or other means, the tools fail to attach to the JVM with the above error.

Note:

For Red Hat Enterprise Linux (RHEL) 7, please be aware of the change with the default cleaup behavior of the /tmp directory:

• RHEL6 uses tmpwatcher to sweep the /tmp directory. The tmpwatcher does NOT delete socket special files even if they are “aged” (default 10 days).
• RHEL7 uses systemd-tmpfiles-clean.service to sweep the /tmp directory. By default, this service deletes socket special files if they are “aged” (default 10 days).

If the /tmp filesystem is mounted with the noatime option, the access timestamp of the files in /tmp are never updated. Therefore, in RHEL 7, the default file cleanup mechanism will delete /tmp/.java_pid* files after 10 days (by default). With the default mount options, the access timestamp of the files in /tmp are updated every 24 hours.

Solution

The solution is to determine how the /tmp/.java_pid<PID> file gets deleted on your operating system and employ workarounds or solutions to avoid the deletion. Mounting /tmp using default settings will correct the specific problem mentioned for RHEL7, or perhaps changing the default behavior of systemd-tmpfiles-clean.service.

Particularly for the jstack command, you can use the -F option, which bypasses the Attach API by using the ptrace system call to read the thread table. Please note that:

• This will pause the JVM while its internal state is read.
• You need to have system level (root/admin) permissions to run jstack -F (same as ptrace).
• The Java version from which you run jstack -F must be the same version as the target JVM.
• In the case of a StackOverflowError or if the target JVM is hung or in some other bad state, you may not be able to connect to it with jstack - F. A good check is to see if you can connect with ptrace or jcmd. If you can connect with one of these but not jstack -F, then that is likely a bug with jstack.

Last reviewed on Sat Feb 01 2025 00:00:00 GMT+0000 (Coordinated Universal Time)