That Ubuntu’s retry command captures stdout

Debian’s retry command allows you to repeatedly run a command until it succeeds. This is useful for running commands that intermittently fail.

Example usage:

retry --until=success --times=3 $FLAKEY_CMD

But we discovered the hard way today that it captures the wrapped command’s stdout until the command exits, which can be a problem in environments where commands are killed after a period of no output. This happens in CircleCI.

To be fair, this is noted in the command’s help text:

Retry captures stdout into memory, and if the command was successful stdout is passed on to stdout as normal, while if the command was repeated stdout is passed to stderr instead. This ensures that output is passed to stdout once and once only.

The solution in our case was to extend Circle’s timeout using the no_output_timeout parameter.