Responsibilities - Vendor form

Hi, 

I am looking to have employee A and employee B to have edit privileges to  Vendor X, and employee C and employee D to have edit privileges to  Vendor Y. No one else should be able to edit Vendor X and and Vendor Y.

I was trying to achieve this by using the privileges form under vendor, however, despite that I have assigned a responsibility and a worker to the vendor, I still see that the vendor record is open for editing for everyone. 

Are there any other parts of the setup missing?

Thanks