Poison


  • 首页

  • 归档

  • 标签

  • 搜索
close
Poison

System.nanoTime()

发表于 2022-08-03

最近查了个关于 System.nanoTime() 的问题,起因是业务里面将 System.nanoTime() 返回的数值作为了业务中的唯一值,最后发现了值相同的数据,询问编写这块代码的同事,同事反馈说当时编写的时候以为 System.nanoTime() 的精度很高,不会出现重复的数据。但是从现象来看,出现了重复的数据。

我们可以用一段简单的代码复现该问题:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
public class Test {

private static volatile long a = -1, b = -2;

public static void main(String[] args) {
long max = 1_000_000;
new Thread(() -> {
for (int i = 0; i < max; i++) {
a = System.nanoTime();
}
}).start();
new Thread(() -> {
for (int i = 0; i < max; i++) {
b = System.nanoTime();
}
}).start();
for (int i = 0; i < max; i++) {
if (a == b) {
System.out.println("nanoTime not unique");
}
}
}

}
阅读全文 »
Poison

Shift Operators

发表于 2022-08-01

最近写了些关于 Bit 操作的代码,才发现使用左移操作符时,当右侧操作数(需要移动的位数)大于左侧操作数比特个数时,实际移动位数等同于右侧操作数对左侧操作数比特个数求余。可能说的不是很明白,可以用一个简单的示例来说明,比如以下代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public class BitShiftTest {

private static String toBinaryString(int i) {
return String.format("%32s", Integer.toBinaryString(i)).replace(' ', '0');
}

public static void main(String[] args) {
int i = 0b00000000_00000000_00000000_00000001;
int j = i << 35;
System.out.println(j);
System.out.println(toBinaryString(j));
}

}
阅读全文 »
Poison

DeleteOnExitHook

发表于 2022-07-18

严格来说,DeleteOnExitHook 导致的内存占用不应该归结为内存泄漏,开发者应该恰当地使用 java.io.File#deleteOnExit 方法,如果持续对不同的文件调用该方法,那么长时间运行的 JVM 实例的内存最终将被填满。为了规避这个问题,建议应用程序自行进行临时文件管理,而不是依赖于 JVM 退出来进行相应的文件清理操作。

如果通过 Google 搜索 java.io.DeleteOnExitHook memory leak,可知有不少开源项目存在该问题,主要受影响的为长时间运行的服务,java.io.DeleteOnExitHook#files 占用的内存往往在数百兆以上而且随着服务的持续运行会持续增长。

如 HIVE-11768 中提到的 .pipeout 文件问题,其解决方案为自行编写了 ShutdownHookManager 类,该类相比 JDK 提供的 DeleteOnExitHook 新增了 cancelDeleteOnExit 方法,且应用自行管理这部分临时文件的生命周期,不再使用后删除文件并调用 cancelDeleteOnExit 方法移除这部分文件名,以解决内存占用问题。本次 commit 可参考:HIVE-11768 : java.io.DeleteOnExitHook leaks memory on long running Hi… · apache/hive@1a81c26 · GitHub。

附上 JDK 8 中 DeleteOnExitHook 的源码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
/**
* This class holds a set of filenames to be deleted on VM exit through a shutdown hook.
* A set is used both to prevent double-insertion of the same file as well as offer
* quick removal.
*/
class DeleteOnExitHook {
private static LinkedHashSet<String> files = new LinkedHashSet<>();
static {
// DeleteOnExitHook must be the last shutdown hook to be invoked.
// Application shutdown hooks may add the first file to the
// delete on exit list and cause the DeleteOnExitHook to be
// registered during shutdown in progress. So set the
// registerShutdownInProgress parameter to true.
sun.misc.SharedSecrets.getJavaLangAccess()
.registerShutdownHook(2 /* Shutdown hook invocation order */,
true /* register even if shutdown in progress */,
new Runnable() {
public void run() {
runHooks();
}
}
);
}

private DeleteOnExitHook() {}

static synchronized void add(String file) {
if(files == null) {
// DeleteOnExitHook is running. Too late to add a file
throw new IllegalStateException("Shutdown in progress");
}

files.add(file);
}

static void runHooks() {
LinkedHashSet<String> theFiles;

synchronized (DeleteOnExitHook.class) {
theFiles = files;
files = null;
}

ArrayList<String> toBeDeleted = new ArrayList<>(theFiles);

// reverse the list to maintain previous jdk deletion order.
// Last in first deleted.
Collections.reverse(toBeDeleted);
for (String filename : toBeDeleted) {
(new File(filename)).delete();
}
}
}
References

jdk/DeleteOnExitHook.java at jdk8-b120 · openjdk/jdk · GitHub
Memory Leak on DeleteOnExitHook - Stack Overflow
JDK-4513817 File.deleteOnExit consumes memory - Java Bug System
SPARK-14261 Memory leak in Spark Thrift Server - ASF JIRA

Poison

ReadWriteLock

发表于 2022-06-16

首先看一段存在 bug 的代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
@GuardedBy("isolatedEndpointsSetLock")
private final Set<Endpoints> isolatedEndpointsSet;
private final ReadWriteLock isolatedEndpointsSetLock;

public ProducerImpl(String group) throws ClientException {
super(group);
this.defaultSendCallbackExecutor = new ThreadPoolExecutor(
Runtime.getRuntime().availableProcessors(),
Runtime.getRuntime().availableProcessors(),
60,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>(),
new ThreadFactoryImpl("SendCallbackWorker"));

this.sendingRouteDataCache = new ConcurrentHashMap<String, SendingTopicRouteData>();

this.isolatedEndpointsSet = new HashSet<Endpoints>();
this.isolatedEndpointsSetLock = new ReentrantReadWriteLock();
}

/**
* Check the status of isolated {@link Endpoints}, rejoin it if it is healthy.
*/
@Override
public void doHealthCheck() {
final Set<Endpoints> routeEndpointsSet = getRouteEndpointsSet();
final Set<Endpoints> expired = new HashSet<Endpoints>(Sets.difference(routeEndpointsSet, isolatedEndpointsSet));
// remove all isolated endpoints which is expired.
isolatedEndpointsSetLock.writeLock().lock();
try {
isolatedEndpointsSet.removeAll(expired);
} finally {
isolatedEndpointsSetLock.writeLock().unlock();
}

HealthCheckRequest request = HealthCheckRequest.newBuilder().build();
isolatedEndpointsSetLock.readLock().lock();
try {
for (final Endpoints endpoints : isolatedEndpointsSet) {
Metadata metadata;
try {
metadata = sign();
} catch (Throwable t) {
continue;
}
final ListenableFuture<HealthCheckResponse> future =
clientManager.healthCheck(endpoints, metadata, request, ioTimeoutMillis, TimeUnit.MILLISECONDS);
Futures.addCallback(future, new FutureCallback<HealthCheckResponse>() {
@Override
public void onSuccess(HealthCheckResponse response) {
final Status status = response.getCommon().getStatus();
final Code code = Code.forNumber(status.getCode());
// target endpoints is healthy, rejoin it.
if (Code.OK.equals(code)) {
isolatedEndpointsSetLock.writeLock().lock();
try {
isolatedEndpointsSet.remove(endpoints);
} finally {
isolatedEndpointsSetLock.writeLock().unlock();
}
log.info("Rejoin endpoints which is isolated before, clientId={}, endpoints={}", id,
endpoints);
return;
}
log.warn("Failed to rejoin the endpoints which is isolated before, clientId={}, code={}, "
+ "status message=[{}], endpoints={}", id, code, status.getMessage(), endpoints);
}

@Override
public void onFailure(Throwable t) {
log.error("Failed to do health check, clientId={}, endpoints={}", id, endpoints, t);
}
});
}
} finally {
isolatedEndpointsSetLock.readLock().unlock();
}
}
阅读全文 »
Poison

SPARK-39357

发表于 2022-06-03
References

SPARK-39357 pmCache memory leak caused by IsolatedClassLoader - ASF JIRA
SPARK-39357 Fix pmCache memory leak caused by IsolatedClassLoader by tianshuang · Pull Request #36741 · apache/spark · GitHub
Effect of ThreadLocals and side-by-side classloading - Stack Overflow
How isolated are static variables? - Software Engineering Stack Exchange

1…456…27

131 日志
119 标签
GitHub LeetCode
© 2025 Poison 蜀ICP备16000644号
由 Hexo 强力驱动
主题 - NexT.Mist