Without the ability to adjust the difficulty, assuming that Bitcoin had value and miners were joining the network, blocks would come in way too fast, expediting the issuance to the point where Bitcoin wouldn't be good as a money.