Ah, I see. Thanks for the clarification, Daniel. The script TfsBuildServerPrerequisites.ps1 mentioned in the documentation page does collect the build dependencies for you, but it's only limited to the files listed at http://msdn.microsoft.com/en-us/library/ff622991.aspx. If your project references more assemblies, you will need to either update the script or manually copy them to the build server.
Thanks,
Xiaoying
Thanks,
Xiaoying