More Pipe inconsistencies

The number of bytes written to a java.nio.channels.Pipe‘s sink at a time will determine the apparent size of the pipe (i.e. the number of bytes that can be written before the pipe blocks). If one byte is written at a time, the size of the pipe is 33012 for Windows or 1 (yes, 1) for Linux. If two bytes are written at a time the sizes are 33012 and 2. Four bytes gives 33012 and 4. Seeing a trend?

Windows will eventually give you the known value of 32768. With Linux, you need to specify more bytes than the known buffer size (4096) to get the buffer size.

The code used is as follows:

// create a Pipe and retrieve its sink and source
// NOTE:  the sink and source are SelectableChannels
final Pipe pipe =;
final WritableByteChannel sink = pipe.sink();
final ReadableByteChannel source = pipe.source();
// set the sink to non-blocking and create and register a write
// Selector on it.  The Selector is used to determine when the sink
// is "full".
// NOTE:  the cast is required since there is no common super-type
//        for selectable + readable / writable
final Selector writeSelector =;
((SelectableChannel)sink).register(writeSelector, SelectionKey.OP_WRITE);
// continue to write to the sink until it is "full"
// NOTE:  the sanity upper-bound is used to ensure that, in a remote
//        case, a sink is not infinite
final ByteBuffer writeBuffer = ByteBuffer.allocate(BUFFER_SIZE);
for(int i=0; i<BUFFER_SIZE; i++)
writeBuffer.put((byte)(i & 0xFF)); // arbitrary
boolean isInfinite = true;  // set to false if limit found on write
int numberOfBytesWritten = 0;
for(int i=0; i<UPPER_BOUND/*sanity*/; i++)
// ensure that data can be written
// NOTE:  selectNow() is used so that it does not block
if(writeSelector.selectNow() > 0)
// clear the selected keys (required)
// write the data
// NOTE:  the actual data written is arbitrary
numberOfBytesWritten += sink.write(writeBuffer);
} else
// the sink is full.  Flag that a limit was found and break
// out of loop.
isInfinite = false;

UPPER_BOUND is some large number (say 10000) and BUFFER_SIZE has values as described above.

And, no, changing selectNow() to something like select(1000L/*1s*/) doesn’t matter (for those worried about concurrency problems).

I should mention that Pipes javadoc does state platform inconsistencies and that this case falls under that unbrella. It’s one thing to read a javadoc. It’s another to actually see those inconsistencies first hand.

An interesting Linux tidbit: if the following code is added after the code listed above with a BUFFER_SIZE less than 4096 then numberOfBytesWritten will be non-zero.

// attempt to write more to the sink even though we shouldn't be
// able to
writeBuffer.limit(1/*writes one byte*/);
numberOfBytesWritten = sink.write(writeBuffer);

It seems that the Linux write selector is lying to us. This is not the case on Windows.

Link-back to main entry: NIO and SSL.



  1. This might be an edge-triggered vs. level-triggered selection thing. (Does java.nio.channels.spi.SelectorProvider.getSelector() yield an epoll-based selector under Linux?) That’s the behavior I would expect if so – write availability will not be reported again until a write actually fails with EWOULDBLOCK.
    You could alter the test to ignore this difference by removing the selector – you don’t need it. Just continue writing until a write fails; the pipe is then full.
    In fact, you don’t even need to loop. Just write something really big. If less than that is actually written (and the return value of write will tell you), that’s the buffer size.

  2. I certainly appreciate your insights. When I get the opportunity I’ll remove the selector and just watch the number of bytes written.
    I should point out though that the point of this exercise was less about what each platform does and why but the fact that WORA (write once run anywhere) is a fantasy when it comes to pipes.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s