Is it legal for a private pilot to fly below 500 feet agl?

Is it legal for a private pilot to fly below 500 feet AGL? I've rode with many pilots, and they would often go below 500 feet AGL. Is this legal? When you squawk VFR, do surrounding towers still watch you on radar?