It seems almost impossible these days to remember a time when the Duggar family was not plagued by scandal.

After it was revealed that Josh Duggar inappropriately touched young girls, two of whom were his younger sisters, Jessa Duggar and Jill Duggar, as a teenager, the pressure seems to be building up to a breaking point for him.

His relatives seem to fear that he could end up hurting himself as a result of the backlash.

The amount of public shame that he has to face, admittedly, may be unbearable.

In addition to feeling disgraced, and dragging his entire family into the spotlight, his parents, Jim Bob and Michelle Duggar, and his 18 brothers and sisters, are concerned that he could be suicidal as well, according to a report from the July 6th issue of Star magazine.

"Josh has seen his world and his family's world crumble," a source told the magazine. "And he knows that it's all his fault. He can never rebound himself from this. He is hiding away leaving his loved ones fearing the worst, [that] he will do something drastic."

During this time, his family is trying to keep a close eye on him in order to make sure that he does not harm himself.

"The longer he stays isolated, the more depressed he gets -- that has his loved ones terrified for him," says the insider. "They fear that shame is too much."

While Josh may be having a hard time forgiving himself, his sisters Jess and Jill have been able to forgive him, and move forward.