Does it really make a big difference with summer vs all-season tires?

Good summer UHP tires typically offer better wet traction than all-season tires. If you do not use the tires in snow there is no reason to compromise with an all-season tire.