The answer was not that obvious. Even though we are using RestTemplate for executing REST calls, the actual connections are opened by underlying ClientHttpRequestFactory. By default, RestTemplate is created with SimpleClientHttpRequestFactory, which uses standard Java Library's UrlHttpConnection.
Starting from Java ?, UrlHttpConnection supports HTTP 1.1 and keep-alive, if destination server supports it, if certain conditions are satisfied. You can turn it off or specify the maximum number of open connections via system properties, but those are all the options you have. For several reasons, there is no option to specify how long the connection can stay open.
However, we have an option to instantiate RestTemplate with HttpComponentsClientHttpRequestFactory, which uses Apache's HttpComponents library to manage connections. This allows greater control over connection pool settings.
So, the answer to the question in the title: YES, RestTemplate re-uses connections if certain conditions are met.