I recently reached for Go’s
ioutil.ReadAll utility function to read some data from a HTTP request body. I’d read that this function should be used with care because it can lead to large values being read into memory, potentially causing crashes. I was curious to find out how that could happen.
Looking at the source for
ioutil.Readall, we can see that it reads data into a buffer, implemented as a byte slice, with
bytes.Buffer.ReadFrom. Reading the source for
ReadFrom, we see that it will read data up to the buffer’s capacity and then attempt to grow the buffer, calling the private
From the docs for
if the buffer can't grow it will panic with ErrTooLarge. Within
grow itself, the condition that triggers this panic is
c > maxInt-c-n, where
c is the capacity of the current buffer and
n is the minimum slice size passed to a
Read call by
Buffer.ReadFrom. This is because, assuming re-slicing or dropping previously read bytes are not possible, Go will grow the buffer to twice its current capacity, plus
maxInt is defined as
int(^uint(0) >> 1). Unary
^ is the bitwise
NOT operator: it inverts every bit of the unsigned integer
0, yielding the maximum possible unsigned integer.
>>, the right shift operator, here shifts bits right by one position, equivalent to dividing by two. So this expression returns the maximum possible signed integer (the leftmost bit being the sign bit).
On a 64-bit architecture then,
grow will refuse to create a slice longer than
9223372036854775807. Practically speaking, we’re always going to exhaust the available memory before reaching that length, so where else can
ErrTooLarge be thrown? The answer is in
makeSlice, called from
grow, which recovers from a panic thrown by
make to throw
make itself is mapped to a type-specific implementation when compiling, in this case
makeslice, which panics when asked to create a slice larger than the maximum memory allocation.