I leave the sides and back alone as well and the shrimplets seem to come out onto that a lot and IF there are new borns that is the only place where I get to see them.
When you do a water change you need to add in the new water very slowly also, most use some sort of dripper setup.
It may just be that the setup wasn't established enough, that is what I put my fungus down too anyway. Mine wasn't really anywhere that I could remove it but it cleared in less than a week, but if you can remove some manually I would do that.I imagine just adding the shrimp may cause a small/temporary imbalance as it then has to adjust again, if that makes sense?
Hopefully you won't loose any more shrimps, and I wouldn't worry too much about the fungus unless it just keeps getting worse, as I have had this happen a few times with newish tanks and it has always just cleared up in about a week.
Simon