The experts say yes. Frequent stopping and starting is called short-cycling and adds quite a strain to your system.
More importantly, you want your "air conditioning" to run and condition the air (lower the humidity) so that you will feel cooler at a warmer temperature. Thus you get the full efficency from your unit and save money.
For more information:
Also check out Harvey Sach's "9 Ways to Make Your Home More Energy Efficient" for more savvy tips.