The reason why the world's oldest rocks are found in Australia is that Australia is geologically a very stable continent. Mountains are normally built on continental plate boundaries, either when two plates collide (e.g. Himalayas), or when volcanoes erupt (e.g. Iceland and New Zealand).
But Australia is away from plate boundaries, so there has been no new mountain building in Australia for a long time. (The last one in eastern Australia was the uplift of the Great Dividing Range for 80 million years ago while in the rest of Australia it has been even longer).
This stability has given erosion a lot of time to work on Australian landscape, which is consequently very flat compared to all other continents in the world. The erosion has also exposed some of the most amazing rock formations in the world, like Uluru (Ayers Rock) and Devil's Marbles in Northern Territory, Bungle Bungles and Wave Rock in Western Australia, Three Sisters in the Blue Mountains in New South Wales, Twelve Apostles in Victoria, Wilpena Pound and Murphy's Haystacks in South Australia, Cradle Mountain in Tasmania, and many many more.
NOTE: This website is written in British English, which is the English we use in Australia. You will find words like "traveller", "harbour" and "realise", and they are all correct in the language used in Australia.