Frankly, I don't understand what you are really doing. Describe how to do it in my opinion.
For the SSH key via the ssh client must be the private key and the server to which it connects corresponding public key.
Ansible by default will run under your user (under which you are working on a machine with ansible). That is, you should use the command ssh-copy-id to put the public key of this user on all servers you want to manage. (It can be done manually through the edit in the server file ~/.ssh/authorized_keys)
More can be done more conveniently, in my opinion - that when the ansible is the machine you connect to other servers, use your own key, not user key ansible. On your car you need to include AgentForwarding for the host with ansible:~/.ssh/config: