In the latest twist in the custody battle between Gossip Girls' Kelly Rutherford and her ex-husband Daniel Giersch, a German businessman, Giersch has accused Rutherford of kidnapping their children.

The accusation was made when Rutherford refused to return Hermes, age 8, and Helena, age 6, to Monaco this past Friday following New York Supreme Court Judge Ellen Frances Gesner's upholding of a court order requiring the children to fly back to their father in Europe.

Rutherford stated, "I hope this decision will end this painful litigation and that my children will be allowed to live in peace in their own country."

It was not to be. Giersch filed a writ of habeas corpus in NYC following her failure to comply with their custody agreement and the court order.

The writ required Rutherford and their two children to appear in court on Tuesday, but on that day, Rutherford came to the hearing alone.

Her lawyer explained that this was to protect the children from "the media circus" but the move backfired when it irked the judge.

Head of the family group in Mischon de Reya, NY Michael Stutman said to PEOPLE, "given Kelly's initial failure to send the children back, showing up without them was probably the nail in the coffin."

The judge ordered the children to be brought to court and they came through a side entrance, only to say goodbye to their mom.

They went straight to the airport from there with their paternal grandmother with whom they are very close.

"The kids didn't seem to look like they missed a beat. They were happy, they kissed their mother goodbye, they were happy to see their grandmother, they had smiles on their [faces]." Ira Garr, Giersch's attorney stated. "The only complaint they had is that they really wanted to get back to playing, it was boring in the courtroom."

Rutherford also put up a brave face, telling the children that they were going to see their papa and that they were going with their Oma.

She reassured them that she would see them soon. "It was done very civilly," Garr commented. There was no comment from Rutherford who went quietly out of the courthouse.

The bitter custody battle has been going on since the couple divorced in 2008. In 2012, the children were ordered to live with their father in Monaco on a temporary basis. The arrangement was that they were to spend their summers with their mom in the States.

Now, however, Rutherford's access to her children could be further restricted.