Poison

Tomcat #419

在 Tomcat 的官方配置文档 Apache Tomcat 8 Configuration Reference 中,其中对 maxPostSize 的描述如下:

The maximum size in bytes of the POST which will be handled by the container FORM URL parameter parsing. The limit can be disabled by setting this attribute to a value less than zero. If not specified, this attribute is set to 2097152 (2 megabytes). Note that the FailedRequestFilter can be used to reject requests that exceed this limit.

可见 maxPostSize 默认为 2MB,但是我们发现,在最大堆大小配置较低的应用服务器上,若客户端 POST 数百兆的大字符串则会触发 OOM 而未进行 maxPostSize 的检查。查询 Tomcat 处理 Content-Typemultipart/form-data 的 POST 请求处理逻辑可知,数据以流的形式写入服务器本地磁盘,然后再从磁盘中读取文件创建字符串,与该问题相关的代码位于:Request.java at 8.5.66

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
if (part.getSubmittedFileName() == null) {
String name = part.getName();
String value = null;
try {
value = part.getString(charset.name());
} catch (UnsupportedEncodingException uee) {
// Not possible
}
if (maxPostSize >= 0) {
// Have to calculate equivalent size. Not completely
// accurate but close enough.
postSize += name.getBytes(charset).length;
if (value != null) {
// Equals sign
postSize++;
// Value length
postSize += part.getSize();
}
// Value separator
postSize++;
if (postSize > maxPostSize) {
parameters.setParseFailedReason(FailReason.POST_TOO_LARGE);
throw new IllegalStateException(sm.getString(
"coyoteRequest.maxPostSizeExceeded"));
}
}
parameters.addParameter(name, value);
}

其中关键之处在于数据以流写入服务器本地磁盘后,会调用 part.getString 创建字符串,该方法实现位于 DiskFileItem.java at 8.5.66

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
/**
* Returns the contents of the file as an array of bytes. If the
* contents of the file were not yet cached in memory, they will be
* loaded from the disk storage and cached.
*
* @return The contents of the file as an array of bytes
* or {@code null} if the data cannot be read
*/
@Override
public byte[] get() {
if (isInMemory()) {
if (cachedContent == null && dfos != null) {
cachedContent = dfos.getData();
}
return cachedContent;
}

byte[] fileData = new byte[(int) getSize()];
InputStream fis = null;

try {
fis = new FileInputStream(dfos.getFile());
IOUtils.readFully(fis, fileData);
} catch (final IOException e) {
fileData = null;
} finally {
IOUtils.closeQuietly(fis);
}

return fileData;
}

/**
* Returns the contents of the file as a String, using the specified
* encoding. This method uses {@link #get()} to retrieve the
* contents of the file.
*
* @param charset The charset to use.
*
* @return The contents of the file, as a string.
*
* @throws UnsupportedEncodingException if the requested character
* encoding is not available.
*/
@Override
public String getString(final String charset)
throws UnsupportedEncodingException {
return new String(get(), charset);
}

可以看出,先根据本地磁盘文件的大小创建一个字节数组,然后将文件读取至内存中的字节数组,最后用这个字节数组创建 String 实例。那么当我们使用 JDK 8 时,假设暂存至磁盘的文件大小为 200MB,且存储的字符均为 ASCII 字符,易知,内存中首先会创建一个 200MB 的字节数组,然后使用该字节数组去创建一个 String 实例,继续跟随 String 实例创建代码,将会调用至 StringCoding.java at jdk8-b120

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
char[] decode(byte[] ba, int off, int len) {
int en = scale(len, cd.maxCharsPerByte());
char[] ca = new char[en];
if (len == 0)
return ca;
if (cd instanceof ArrayDecoder) {
int clen = ((ArrayDecoder)cd).decode(ba, off, len, ca);
return safeTrim(ca, clen, cs, isTrusted);
} else {
cd.reset();
ByteBuffer bb = ByteBuffer.wrap(ba, off, len);
CharBuffer cb = CharBuffer.wrap(ca);
try {
CoderResult cr = cd.decode(bb, cb, true);
if (!cr.isUnderflow())
cr.throwException();
cr = cd.flush(cb);
if (!cr.isUnderflow())
cr.throwException();
} catch (CharacterCodingException x) {
// Substitution is always enabled,
// so this shouldn't happen
throw new Error(x);
}
return safeTrim(ca, cb.position(), cs, isTrusted);
}
}

即再创建一个元素个数为 200 * 1024 * 1024 的字符数组,且因为单个字符占用两个字节,即此处的字符数组占用 400MB 内存,创建完成后,才会进行 maxPostSize 大小的比较。也就是说,在刚才的过程中,POST 大字符串将会导致内存占用增加 600MB,如果最大堆大小为 512MB,那么将触发 OOM。同理我们可以计算出上面的处理逻辑在 POST 一个文件大小超过 最大堆大小/3 的字符串时,将会触发 OOM。

我们先用一段示例代码验证下内存占用,示例工程位于:GitHub - tianshuang/tomcat-oom: Emulate Tomcat OOM。我们使用 TomcatOomApplication.java 启动该 Spring Boot 应用,并设置参数 -Xmx1g,即先设置最大堆大小为 1G,此时使用 TomcatOomApplicationTests.java POST 一个 文件大小 为 200MB 的字符串,内存变化如下:


可以看出,内存先增加了 200MB,然后又增加了 400MB。注意,我在创建完字节数组时暂停了一下,所以图上才能体现出 200 多兆时的转折点。即内存由 20MB 增长至 231MB 再增长至 632MB,与代码中的逻辑一致。因为我们此时设置的最大堆大小为 1G,所以不会触发 OOM,此时触发的异常为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
2022-03-04 23:24:35.464 ERROR 4126 --- [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.multipart.MultipartException: Failed to parse multipart servlet request; nested exception is java.lang.IllegalStateException: The multi-part request contained parameter data (excluding uploaded files) that exceeded the limit for maxPostSize set on the associated connector] with root cause

java.lang.IllegalStateException: The multi-part request contained parameter data (excluding uploaded files) that exceeded the limit for maxPostSize set on the associated connector
at org.apache.catalina.connector.Request.parseParts(Request.java:2990) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.Request.parseParameters(Request.java:3294) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.Request.getParameter(Request.java:1169) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:381) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:75) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:201) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1629) [tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.66.jar:8.5.66]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_311]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_311]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.66.jar:8.5.66]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_311]

我们重启该 Spring Boot 应用并设置 -Xmx512m,此时,再次 POST 一个仅包含 ASCII 字符的文件大小为 200MB 的字符串,可以观察到内存变化如下:


即增长至 229MB 就不再增长了,为什么呢?因为去申请内存占用为 400MB 的字符数组时,就触发 OOM 了,异常栈帧如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
2022-03-04 23:31:15.213 ERROR 12328 --- [nio-8080-exec-1] o.a.coyote.http11.Http11NioProtocol      : Failed to complete processing of a request

java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149) ~[na:1.8.0_311]
at java.lang.StringCoding.decode(StringCoding.java:193) ~[na:1.8.0_311]
at java.lang.String.<init>(String.java:426) ~[na:1.8.0_311]
at java.lang.String.<init>(String.java:491) ~[na:1.8.0_311]
at org.apache.tomcat.util.http.fileupload.disk.DiskFileItem.getString(DiskFileItem.java:327) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.ApplicationPart.getString(ApplicationPart.java:127) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.Request.parseParts(Request.java:2972) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.Request.parseParameters(Request.java:3294) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.Request.getParameter(Request.java:1169) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:381) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:75) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:201) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1629) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-8.5.66.jar:8.5.66]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_311]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_311]

至此清楚原因后,我们很容易就可以修复该 bug,即先判断文件大小是否超过 maxPostSize,如果不超过再进行字符串转换,如果超过则直接抛出 IllegalStateException 以避免构建大字符串,所以我提交了 PR 对该问题进行修复。

Remark

以上演示是在使用 Servlet 默认 MultipartConfig 配置的情况下,因为 maxRequestSize 的默认配置为无上限,可参见:Uploading Files with Java Servlet Technology。但其实在 Spring Boot 的自动配置逻辑中,已经将 maxRequestSize 设置为了 10MB,可参见:MultipartProperties (Spring Boot 2.6.4 API)。所以如果为 Spring Boot 应用且未显式进行配置的情况下,在 POST 超过 10MB 的字符串时,会在 FileItemIteratorImpl.java at 8.5.56 根据 Content-Length 预先判断并抛出异常,根本不会写入磁盘,所以演示时我在 Spring Boot 配置中恢复为了 Servlet 默认配置以复现该问题,配置参考:tomcat-oom/application.properties at main · tianshuang/tomcat-oom · GitHub。在非 Spring Boot 的应用中,大多数使用的默认配置,则容易触发上述的异常。

以上演示使用的 JDK 版本为 8,如果使用的 JDK 为 9 或以上的版本,则可以观察到更低的内存占用,因为引入了字符串压缩,可参考:JEP 254: Compact StringsJDK 9 Release Notes。当使用 9 或更高版本的 JDK 且传输的字符串均为 ASCII 字符时,单个字符仅占用一个字节,而不像 JDK 8 占用两个字节。

Reference

part.getString will cause OOM without checking maxPostSize, checking maxPostSize first will avoid OOM caused by huge string by tianshuang · Pull Request #419 · apache/tomcat · GitHub
Fix #419. Check parameter value size before conversion to String · apache/tomcat@3e9dd49 · GitHub
Apache Tomcat 8 (8.5.76) - Changelog
CPU and memory live charts | IntelliJ IDEA