The title battles featured a number of tense moments between the two drivers, including on-track collisions at the 2014 Belgian Grand Prix and the 2016 Spanish Grand Prix.
Upon beating Hamilton to the world title in 2016, Rosberg announced his immediate retirement from F1, prompting Mercedes to sign Valtteri Bottas as his replacement.
Speaking about the competition between Hamilton and Rosberg on Jake Humphrey's High Performance Podcast, Mercedes team boss Wolff dismissed the idea that the friction helped bring out the best in the drivers.
"I'm not sure it gets the best out of both, because that is negativity, and you still have to be a team player," Wolff said.
"If the debriefing room is full of negativity, because the two drivers are hostile with each other, then that will spill over into the energy into the room, and that is not something I will ever allow again."
Wolff explained how he was unable to get to the bottom of the friction between Hamilton and Rosberg, saying there was "historical context" Mercedes was unaware of, and that he may never fully understand.
"I couldn't change it, because the drivers were hired before I came," Wolff said.
"Nobody actually thought what is the dynamic between the two? What is the past between the two? There was a lot of historical context that none of us knew, and will never know.
"That's why it is something that we're looking at, how do the drivers work with each other, what happens in the case of failure of one and the other.
"We accept the annoyance and pain if it goes against one, but we're trying still to keep the positive dynamic in the team."
After some early-season tension in 2014, Hamilton and Rosberg first came to blows on-track at Spa when Rosberg refused to back out of an early on-track battle. He later told the team it was in response to Hamilton ignoring team orders at the previous race in Hungary.
Wolff worked to nevertheless create an environment in which both drivers recognised that nothing was more important that the team result, and the importance of representing the Mercedes brand.
"It was very difficult, because I came into the team as a newcomer in Formula 1, and Nico and Lewis had been in the sport for much longer," Wolff said.
"But still I was able to create an environment where they had to respect the team, sometimes with an iron fist, and they understood that they couldn't let us down, they couldn't let Mercedes down.
"In the events of 2014, I felt there was some selfish behaviour. I said the next time you come close to the other car, your teammate, you think about the Mercedes brand. You think about single individuals in the team.
"You think about Dieter Zetsche, the CEO of Mercedes. That's going to change the way you act. You're not going to put your teammate into the wall.
"I always made clear that if this was going to happen regularly and there was a pattern, I have no fear in making somebody miss races."