Well, as you identify, the embedding of videos on a third-party websites is allowed (with some restrictions) by YouTube.
I think the difficulty here is in the fact that you link to a service which would allow someone to download the videos embedded on your site. Generally speaking, I don't believe there is anything in the YouTube terms of service which allows the creation of a permanent copy of the video, and it would probably be a pretty straightforward case of infringement against someone who actually used the download link on your site to download the video for YouTube. While the Digital Millennium Copyright Act does allow an internet service provider who complies with certain requirements to avoid liability for content posted by users, it's also possible that the court could find some amount of contributory infringement if it can be shown that your website profits from the infringement. (See MGM Studios, Inc. v. Grokster, Ltd., 545 U.S. 913 (2005) for a discussion of some of these issues).
I would also wonder whether or not your posting of a download link would constitute a violation of the anti-circumvention provisions of the Digital Millennium Copyright Act (17 USC 1201 et. seq.) , since the YouTube software does not allow for viewers of its content to make permanent copies of the content. Assuming this inability for the YouTube user to create such a permanent copy within the bounds of the software provided by YouTube could be considered an "effective protection afforded by a technological measure", then your posting of a link to a third-party downloading site could almost certainly be seen as an "offer to the public" of that circumvention measure. The case you present would certainly seem on the surface to be somewhat similar to Realnetworks, Inc. v. Streambox, Inc., 2000 U.S. Dist. LEXIS 1889 (W.D. Wash. Jan. 18, 2000), in which an injunction was entered against Streambox for a fairly similar set of facts to yours. It could certainly be in your interest to examine that case as well as further cases concerning anti-circumvention and the DMCA are concerned.