AITA for refusing to honor request made by my niece's father to limit my interactions with his daughter?

My niece through marriage father has asked me to limit my interactions with his daughter since she has been starting to call me uncle daddy, since my wife and sister are very close and she spends a great deal of time with her aunt my wife, and I can work remotely when I choose so I am also around.

Her father does work long hours and has had to take on extra shifts cause now they live separately, and are co-parenting and my wife's sister is staying with us since we have a multi family home. So in essence I see his daughter more then he does. He wants my wife's sister to get back on her feet, ans while I agree I do not think as a family we should let her suffer and struggle or live with a man she does not want to be with so we took her in.

He feels like she is starting to view me more as a father, she has recently started to call him daddy [insert real name] and things she use to have him do she comes to me because he is no longer present. Like bed time stories as one example.

I do understand where he is coming from but at the sametime it is not my place to try and mend their relationship and living situation. If ny wife's sister was comfortable sharing a space I would be all for him staying in the second family home with her. Not my intent to split them, but she does not. I also understand he has to work long hours and his only free time is on the weekend. He does spend that time with her at his own place and she spends the weekends with him. Problem is during the week I am the only male figure she really sees, and starting next month I will also be picking her up from daycare. He is not a fan of this, but the daycare hours are only from 8 to 4:30, and he works and would never make it in time.

He feels like I am an asshole for not honoring his request and feels like I am trying to take over his role as father. Which I personally understand where he is coming from but do not see it. I love my niece sure no blood ties but I have been with my wife since HS, and I have known her sister ever since. I do view her as a sister so I see nothing wrong with me being close to her or her kid. The uncle daddy thing is weird and I have tried to correct it but she is 2, and when we try to correct it she says no you are uncle daddy.

Am I in the wrong for not limiting my interactions for the sake to save that daddy dynamic with her father. I do not have kids, nor did I have a father so I will not pretend to understand that dynamic. That being said I tols him i refuse to ignore or tell his tell my niece no or limit my time with her cause he is upset she sees me more. Not my fault my wifes sister refuses to work things out with him and does not feel comfortable around him.

(Hope I did the post right this time.)

Ages are Me(29M) Wife(28F) Wife's sister(22F) Nieces father(24M)