"We both don't know how the examiner interpretes a reference in way which anticipates our claim. The interpretation doesn't seem making sense. I wonder if I can call and talk to the examiner and ask him how he saw the referecen in that way or I have to let the outside counsel to talk to the examiner."
The circumstances described certainly suggest that an interview is warranted, and possibly necessary. But before I expend the time and resources to have an interview, I always try to determine whether the issue is that I don;t understand how the examiner: a) is interpreting the applied reference; or b) has interpreted the claim language. Regardless of the answer, an examiner interview can be valuable. However, the latter case is often easier to figure out (even without an interview), and can often be adequately address in a written response.
"What are the downsides of interviewing an examiner?"
There are a number of possible downsides, all of which are easily mitigated. The most obvious downside to having an examiner interview is the cost and time involved. If you are hiring outside counsel to conduct the interview, they will often bill for the time spent traveling to the PTO, preparing for the interview, and conducting the interview. Thus, without a meaningful cap, a simple 20 minute interview can cost as much as $1000 or more in attorneys fees! So, consider negotiating a flat rate for the interview with your outside counsel. Another potenttial downside is the creation of a "negative" interview summary, or an interview summary that contains unfortunate statements that you don't want to appear in court. That downside is easily addressed by conducting the interview in person, if possible. It is often the case that an attorney who conducts an in person examiner interview can dictate the interview summary to the examiner. In that way, the interview summary says nothing more, or less, than what the attorney wants it to say. The last potential downside I will mention is that it is quite possible to piss off the examiner during an interview. I've posted a couple of times about an interview that I remember from my time at the PTO, some 10 years ago, where the applicant's attorney acted like a complete ass. So, if you do decide to have an interview, be respectiful to the examiner. He/she is a busy person, and they do not (generally) hold any ill will against an applicant or attorney who asks conducts themselves in a professional manner.
"May the examiner comes up with more references against our patent application, if we talk to him? "
An examiner can always dig up more references. Conducting an interview does not, in my experience, increase this risk. Of course, you would not want to say anything damaging to your case during the interview, which would lead the examiner to art that they might not otherwise have considered (bearing in mind your duty to disclose material information to the PTO, of course).
"If I can talk to the examiner, should I make an appointment?"
Absolutely. If you cold call the examiner, he or she will rarely be willing to speak to you about a particular case. As previously mentioned, examiners are busy people. More senior examiners look at 10-20 cases per week, sometimes more, and cannot be expected to remember the nuances of your case instataneously. Thus, an examiner will most often request that the interview take place after he/she has had a period of time to review the applicant in question.