Robin Thicke Has Only Seen Son Once Since Judge Granted Paula Patton Temporary Sole Custody: Source

After losing temporary sole custody to ex Paula Patton, Robin Thicke spent time with his son Julian on Friday, two sources tell PEOPLE. But the singer did not see his son the following day — kicking off another round of squabbling between the ex-spouses.

“He did not exercise his right to spend time with his son and the monitor on Saturday, citing a business meeting. However, Paula offered alternate times that Julian could see Robin with the monitor that weekend, but he declined all visitation,” says a source close to the situation. “These are the actions of a father who claimed he wanted to spend time with his son.”

However, a source close to Thicke says the singer is “focused” on his son. “ keeps trying to bury him, and this is a concerted effort to prejudice the public against him. He has taken no such campaign against her,” says the Thicke source. “He’s taking this extremely seriously, and it’s unfortunate and very difficult when you’re on the receiving end of a campaign of negativity.”

Last Thursday, an L.A. judge gave Patton sole legal and physical custody and Thicke supervised visitation with Julian three days a week at a neutral location until the next hearing on Feb. 24. Otherwise, Thicke has to stay 100 yards away from Patton, Julian and Patton’s mother, per the temporary restraining order against Thicke.

Thicke, 39, and Patton, 41, separated in 2014, finalized their divorce in 2015 and seemingly enjoyed an amicable coparenting arrangement until recent months. Last week, the actress alleged in shocking court documents that he physically and verbally abused her, cheated on her throughout their marriage and struggles with drug and alcohol addiction.

“Paula very much wanted to keep this within the family, as she believes that was in Julian’s best interest,” the source close to the situation previously told PEOPLE. “However, she has to fight back to protect her son. As a mother, it’s one thing to stay quiet if you’re the victim of abuse, but once that extends to your child, you have to speak up.”

A rep for Thicke declined to comment on the allegations, but his attorney Angela Pierce di Donato told PEOPLE in a statement: “Paula never reported any domestic violence until she was in a position of contempt by violating the custody orders.” However, the source close to the situation says Patton “was never in contempt of court” and that “allegations to the contrary are patently false.”

As for allegations of drug use, Pierce di Donato claims “DCFS asked both Paula and Robin to drug test. Robin agreed and submitted to his test, but Paula refused to test. Infidelity has nothing to do with custody. She is attempting to throw anything at him to hurt him, but Robin’s focus is their son.”

However, the source close to the situation continues to deny the drug test allegation. “There is no mandatory requirement that Paula be drug-tested, but she was happy to do one with proper protocol and a proper lab setting,” the source tells PEOPLE. “Robin’s drug test is a complete red herring because it was an unobserved drug test in his home. Paula did not take the drug test based on legal advice that she should not validate an improper test.”

The exes’ custody battle first came to light earlier this month. On Jan. 12, the actress filed court papers in L.A., saying she had “become concerned about ’s drinking and drug use, as well as the forms of punishment he is using to discipline Julian,” citing spankings she alleges left the boy “scared.” (DCFS is investigating the claims.)

Thicke adamantly denied the abuse at the time, maintaining he has employed “light spanking” as a “last resort” but that he “never exceeded age-appropriate discipline.”