Which phase are you at? It goes like this:
0 (intro): Talk to girl at night, automatic transition to upper floor. Includes introductions and the likes. Have to take over the first dude on the top row. He fingers you and masturbates. Nothing more.
1: Have to take over bottom right man, the dude with the brown hair and mustache. No penetration from him.
2: Have to take care of the third guy from the left, bottom row. Old man with a bald spot. The bath has black haired dude and brown haired dude (manager) first time, then only brown haired dude. Has penetration for bald man.
3 (after army cutscene has appeared): Have to take care of the bald man from phase two, as well as the dude diagonally up to the left from him, or the second dude from the left of the top row. Both of them has penetration, and can be repeated infinitely. Also includes sex with the manager in the bath, as well as by the entrance in the morning. Also has sex with the red haired dude in the morning, as well as a repeatable scene immediately after.
For me it was at first stuck on the second phase, since the army cutscene refused to appear. It wasn't fixable for me, so I had to load a previous save, from before I even started the brothel scenes. I did the beach scenes (all of them: two peep scenes, spoke to blond guy both times, two scenes with red haired beach boy, two scenes with the couple that gets peeped on). The second scene with the beach boy had to be repeated twice, since it never turned to night (because of the army). When I then spoke to the old man (real owner of the brothel) and left the brothel again, the army cutscene happened. So it actually happened before I ever even started the brothel storyline, and still counted toward the third phase.