They try to annoy you so that they can expel you from the land. Had they been successful, no one would have been left behind except a few.