I just did the same (A4 for 540)...
Traded in my 2004 1.8T A4 (6 spd) for a 2003 540 M-Sport 6 spd just two months ago. I love it! My advice -- look for a 2003 M-Sport. It might take you a little while to find one, but it will be well worth the wait. If you're in the midwest, I have a few leads I can recommend.
Best of luck!