We maintain all of our server's /etc directories in subversion repositories and just disabled password logins on our subversion server. So that means you actually need ssh-agent to work when using sudo. This can be done by editting the /etc/sudoers file and adding the following line:
Defaults env_keep="SSH_AUTH_SOCK"
Using a jumpbox with subversion Proper access rights with SSH+SVN