Growing up at Hampton Court, young Prince George was utterly miserable. When his father died, he had become the charge of his grandfather, George II – a man of moderate intelligence, minimal charm, and maximum temper.

One day, flying into a violent rage for reasons unrecorded, he beat his teenage grandson so brutally the youngster swore he would never live at the sprawling red-brick palace.

In 1761, a year after becoming King George III, he bought Buckingham House, in central London, and set to work turning it into the palace that endures to this day.

From then on, Hampton Court, just 12 miles from central London, ceased to be used by the Royal Family as a home. Instead, its splendid gardens were opened to the public and its empty rooms subdivided into apartments which George III gave to retired servants and royal cousins fleeing revolutions on the Continent.

Fortunately for future generations, he ensured conservators preserved the palace’s architectural treasures. But this represented the first quiet phase in Hampton Court’s history for years.

Situated on a bend in the River Thames, it had always been a desirable place and, thanks to its wealthy, titled residents, the stage for centuries of drama.

Excavations have proved there was a villa on the site as far back as Roman times and, in the Middle Ages, the estate belonged to Lady Godiva’s son, Aelfgar.

It then became the property of a group of warrior monks who made a tidy sum renting the small manor they built there to wealthy Londoners wanting summer breaks from the capital when the plague was at its height.

The last person to rent Hampton Court from the warrior monks was Henry VIII’s chief minister, Cardinal Thomas Wolsey, who used his fortune to turn the manor into a palace grand enough to host the Hapsburg emperor when he visited England.

Despite his vow of celibacy as a Catholic priest, Wolsey also brought his two illegitimate children, Dorothy and Thomas, to visit him at Hampton Court. Their mother was his confessor’s sister, which must have made trips to church awkward. However, it was the King’s messy private life, not Wolsey’s, that brought Hampton Court into the royals’ possession.

Henry VIII’s attempts to divorce his first wife, the Spanish princess Katherine of Aragon, turned into a six-year-long diplomatic fiasco that destroyed many careers, including Cardinal Wolsey’s, who died in disgrace after the King blamed him for failing to persuade the Pope to grant the divorce.

Encouraged by the new Protestant religion, Henry VIII broke with the Roman Catholic Church in 1533, the same year his new wife, Anne Boleyn, was crowned queen. Property belonging to the Catholic Church was seized, including Hampton Court, which was taken from its monastic owners and became a royal palace.

Interested in the arts and architecture, Anne had grand plans for Hampton Court, which included nearly doubling it in size and building a new wing for the queen’s apartments. But in 1536, Anne was accused of multiple acts of adultery and beheaded.

Her enemies claimed she had conducted an affair with the Anglo-Welsh landowner William Brereton while they were both at Hampton Court in 1533. Both Anne and her co-accused protested their innocence to no avail. The majority believed them at the time – even many of her enemies.

Five years later, irony or karma came knocking at Hampton Court when Henry’s fifth wife, the 18-year-old Queen Catherine Howard was arrested at Hampton Court.

Servants’ rooms were searched and, in one, a handwritten love letter was found from the Queen to her husband’s attendant, Thomas Culpepper.

Both Thomas and the Queen were later executed and one of the corridors at Hampton Court is now known as “the Haunted Gallery”, as it’s said Catherine’s ghost can still be heard screaming there, begging her husband for mercy.

With its lavish rooms and gardens, Hampton Court remained a favourite royal home for the rest of the 16th century. Henry VIII’s daughter, Queen Mary I, spent her honeymoon there and her sister, Elizabeth I, celebrated surviving smallpox at Hampton Court in 1562.

When Elizabeth I died without children in 1603, her crowns passed to her Scottish godson, King James VI, who became King James I in England, Wales, and Ireland.

He chose Hampton Court to host his family’s first Christmas in England and, safe to say, events were rather raucous. During 12 days of almost non-stop parties at the palace, William Shakespeare premiered some of his plays. The tipsy Spanish ambassador tried to punch the French ambassador in the face after they bumped into each other during a dance in the Great Hall.

Several aristocrats became so spectacularly drunk that they fell over, taking a couple of buffet tables with them.

And King James’s alleged lover – the handsome but stunningly dim Lord Philip Herbert (by his own admission, he knew of nothing except dogs, horses and revelry) – nearly collapsed mid-dance, weighed down by all the jewels sewn into his costume.

A week later, while everybody else trembled and groaned through hangovers of epic proportions, James I got back to work by hosting a church conference at the palace at which he commissioned his famous English translation of the Bible.

It later became one of the most beloved books in the English-speaking world. James I’s grandson Charles II spent his honeymoon at Hampton Court in 1662, but his gentle Portuguese wife, Princess Catherine of Braganza, was horrified to discover he had invited his mistress Barbara Palmer to join them.

The Queen was so upset by her husband’s infidelity that she suffered a nosebleed, but Charles would not back down and gave Barbara her own wing to live in at the palace.

There, Barbara raised their six illegitimate children and feuded with Charles’s other mistresses – including the cross-dressing Italian socialite Hortense Mancini, who ended her own relationship with the King when she slept with his recently-married daughter – and Cockney actress Nell Gwyn, who sent sweets to a rival mistress as a peace offering but failed to mention she had dosed them with a very strong laxative.

Barbara got her own back on Charles II by having several affairs of her own: with a circus acrobat, a London theatre actor, a duel-fighting baron and a future general.

After George III ended royal residency at Hampton Court, his granddaughter Queen Victoria and her husband Prince Albert put the palace to good use by allocating apartments there to widows left destitute by their husbands.

Other rooms later went to prominent public figures such as Lady Baden-Powell, widow of the Scouts’ founder, and the brilliant mathematician Michael Faraday, who had suffered a nervous breakdown.

A century later, Elizabeth II’s coronation ball was held in the palace.

The last Tsar of Russia’s sister, Grand Duchess Xenia, also lived there, protected from potential kidnapping by the Communist KGB.

In 1920, King George V insisted that an oak from the palace grounds be used to make the coffin of the Unknown Warrior, to commemorate all those who had died in the First World War who had no known graves.

The King’s decision was especially moving to Hampton Court staff member Thomas Abnett, a lamp-lighter who had lost two of his sons on the Western Front. The Hampton Court oak coffin now lies at the heart of Westminster Abbey, where the Unknown Warrior was buried, with the King as chief mourner, on Remembrance Day 1920.

With its cocktail of shocking, scandalous and often moving events, Hampton Court has stood at the heart of British history for the last 500 years.

George III may have been miserable there, but modern-day visitors will love strolling through its stunning gardens, creeping along its haunted corridors, and even losing themselves in its famous hedge maze.

• The Palace: From the Tudors to the Windsors, 500 Years of History at Hampton Court by Gareth Russell (HarperCollins, £25) is out now. For free UK P&P, visit expressbookshop.com or call 020 3176 3832