As already mentioned find out about your local areas laws, you may need a permit for fishing even if you technically are not doing the traditional fishing rod fishing. Also try to find out any species that are strictly no take. In England I believe that newts could be a no take species (by the way if you are in England I personally would like to see some newts) and non-native crays would be kill on site while native crays would be release, in America I believe in certain states its sunfish that are not allowed to be taken.
I take a couple of hand nets that I got from a fishing shop, they are actually designed for netting bait fish out of bait wells on boats and have short handles but I find them good to scoop through the water to net fish from the front and back similantiously. Long handle nets can also be good to reach into deeper areas that you don't want to try and wade into. For possible nets I would check out pet shops for things like fish nets and butterfly nets as well as fishing tackle shops. You will want a net with good durable mesh/ fabric for scooping though weedlife and possibly over rocks. The smallish nets I have that I mentioned are light weight and the frame is made of hollow aluminium.
I always take at least one clear sided bucket with a sealable lid. In the lid I have drilled a hole that allows me to thread through an airline attached to a battery airpump. You could also get a small 20L fish tank to take happy snaps of your captures before releasing them again.
Take with you a hat to keep off the sun, possibly sunscreen to protect your face and arms and some polorised sunglasses to help cut through the glare on the water and see the critters getting about in the water. Take along some food and water ,the bug can be addictive and you may end up spending longer than expected from food and resfreshment options. Also take a small first aid kit with you, you may never need it but better to be safe than sorry. Keep an eye on the weather any sudden changes or storms and get the hell out of the water, especially if your in a potential flash flooding area. Get some durable good grip shoes like reef shoes, sneakers/ joggers are fine but its not nice squelching about for the rest of the day in soaked shoes and socks. Also reef shoes will offer your feet some protection from broken bottles and sharp rocks. Failing that you could go the whole hog and get anglers waders.
If you want to be really scientific you could take with you a TDS pen, pH pen and thermometer. A note book to record species and other observations could also be good. I often go out firefly hunting/ watching and I keep a journal of the weather conditions, moon light, firefly activity levels etc. It helps me later in deciding if its the right conditions again to get optimal sightings in certain areas or if the fireflies are even likely to be active.
If using hand nets scoop through water weeds and any build up of leaf debris this is where you can find some interesting critters. Check out on and around any submerged wood. If the creek/ stream is sandy on the bottom look for tracks like snail tracks. If your primarily after small water critters try back waters and sheltered areas out of strong flow. Looking under rocks can also be worthwhile and a snorkel google can help with visability if you didn't want to totally submerge your head/ face.
To bait or attract the local inhabitants fresh or stale bread almost never fails, but I often also take along a selection of my fish foods that are pellets or wafers, you could also try fish burley from your local fishing tackle shop. Vegemite I would say works as an attractant because of the yeast, garlic might also work since it is supposed to encourage a fishes appetite.
As for funny looks by the general public..... just act totally natural like they what you are doing is totally normal. If you act like a fugitive it will get the public suspicious, but calmly go about what your doing and ignore them and they may merely approach you out of curiosity. If they talk to you, you may even infect another person with the bug, then you could have an expedition companion, to share your finds and fun with.
And most of all have fun and share your findings