Technically you would not have to send an invite if the people who would participate in the conference call knew when the call would be and the conference number for the call.
Having said the above the host and the participants would have to have their respective Host/Participant codes to join the conference. This enforces security so that someone who dials a wrong number or the like doesn't end up in the middle of your conference call.
To recap, if John, Joe and Sue all know that a conference call is scheduled every first Monday of the month at 10am and the know the conference call number is 999-555-1212, they would not need any invites. However John (being the host) would need his Host code of 999-999 (made up) and Joe and Sue would need their Participants code of 333-333.