In the shadow of a 353-year-old fortress, a former trading post for gold, ivory and slaves, a pack of young boys play a game of pick-up soccer. The ball thuds against the tall, whitewashed stone wall that surrounds what was, for the very unfortunate, their last stop on African soil before passage to the harsh reality of the New World.

Beyond that wall, through the gate and past the thick wooden door, a cracked cobblestone courtyard opens to the sea. Fifteen original canons still guard against pirate attacks, the unused cannonballs having fused together over the centuries.

Cape Coast Castle squats on a rocky cliff that juts out into the Atlantic Ocean on the seacoast of Ghana, West Africa. To the east, the castle looks over the town centre, a tangled mix of time-weathered colonial buildings, cinder block homes, and shacks capped by corrugated tin. To the west, large dugout canoes wait on sandy shores for muscular young men to finish repairing the pale green fishing nets.

The castle was originally founded by the Swedes in 1653, taken by the Danes and then passed to the Dutch, finally becoming a possession of the English in 1662. It served as a base for colonial activity in the country the British called the Gold Coast. A French bombardment levelled the fort in 1757, but it was re-built in grander scale by the Brits, and this is what stands today as a UNESCO World Heritage site.

The airy rooms of the top floor were home to the well-heeled residents of the castle. Sunlight still creeps through the shuttered windows of the Governor’s residence, the patterned shadows falling on warped mahogany floors. But the light stops there.

Below ground level are the clammy, dirt-floor dungeons where slaves were imprisoned. Small slits high in the walls allow a trickle of air into the cells. It’s suffocating for one person, yet hundreds were crammed into these quarters. And there were many such rooms.

The west coast of Ghana, from the sprawling, bustling capital of Accra to the Cote d’Ivoire border, is home to the densest concentration of European forts and castles on the African continent. Twenty-nine of the original 37 castles are still around, stretched out along the shore like hooks on a fishing line. Some are crumbling, some are well preserved, but all resonate with a difficult and complex history. In between the castles are Fante fishing villages framed by palm trees and turquoise seas.

The journey from Accra to Cape Coast town, population 300,000 including outlying villages, is about 200 kilometres. Depending on the state of the roads, it can feel longer. The route runs within drumbeat distance of the shore, but unfortunately there are only glimpses of the sea from the windows of the well-travelled, state-owned buses. Transport comes in ordinary and luxury, with the former making up in character what it lacks in comfort.

Whenever the bus stops, an eclectic grocery store arrives at your window. Hawkers, mostly women with laden baskets balanced on their heads, rush over to sell water, oranges, plantain chips, pineapples, yams, crackers, handkerchiefs and toilet paper. An occasional young man holds up a dead grasscutter, a large, tasty bush rodent that looks like a beaver on a diet. Hands reach up and reach down, and the women run alongside the bus as it picks up speed.

I arrive in the evening, as the African sun casts a warm hue over this faded colonial centre, the British capital of the Gold Coast until 1876. If you are drawn to the grand narratives of early modern history, Cape Coast is a good read. The stories are written in the castle and the churches, in the old European buildings and the Ghanaian homes, in the food stalls and family shops, and in Ghana’s first university and some of the country’s biggest boarding schools.

A bit frayed at the edges and looking its age, Cape Coast is nonetheless an energetic town with a solid enough infrastructure. There is a range of hotels and guesthouses available, from well-appointed suites costing around $80, to the sparse $8 room I stay in, with its wobbly ceiling fan and a splash not a shower.

I find one of the two restaurants with front-row ocean seats. The full moon is reflected on the crest of the waves as they break for shore, rows of gleaming teeth biting into the sand. I have a Star beer – from Ghana’s first brewery – and a large bowl of groundnut stew, a thick puree of peanuts, spices, and meat of choice, eaten with rice balls, mashed yam, or fermented maize.

After another Star, I walk back to my guesthouse, past a lumpy soccer pitch at Victoria Park. An orphaned statue of the old queen stands alone on the sidelines, watching the kids play football in the dark.

The smell of street food mixes with the scent of kerosene in the humid night air. Music fills the air, streetlamps flicker, and shadows emerge from alleys. Young lovers loiter, the women standing seductively, one hip cocked to the side and the glimpse of an arched back contrasting against the colourful African prints. Vendors, cars, bicycles, goats, chickens, frogs creaking from sewer trenches, preachers with growling voices – life spills in all directions.

After a surprisingly good sleep, I’m at the castle at the opening time of 8:00 am, before other visitors, to better hear the ghosts of the place. From the courtyard, I step down into the dank underground chambers where the slaves were stored – housed being too generous of a word. In one of the rooms, a lone lightbulb hangs from a wire, helping to illuminate the past.

I’m startled by the voice of Kingsley Kofi Yeboah, the long-serving historian and curator of the fort. He gives me a tour of the castle’s museum, which has evocative exhibits on Ghana’s history, slavery, and the lives of Blacks after the diaspora.

He then takes me to see the tiny ‘condemned cell’, into which captives who revolted were locked, up to 50 at a time. They died of suffocation and starvation, a deterrent to the other prisoners. The walls have scratch marks. “It may not be pleasant history,” says Yeboah, “but it’s the history of all of us, of you and me, and it’s what brought us here today.”

During the Trans-Atlantic Slave Trade, somewhere around 20 million people were kidnapped and transplanted to the Americas, the Caribbean and Europe between the 15th and 19th centuries. The labour of these enslaved Africans became the backbone of lucrative economies in sugar cane, tobacco, cotton, cocoa, rice and coffee.

European merchants generally did not travel inland to buy the slaves, but acquired them (three men to each woman) from African middlemen who in turn had bought them from various African slave hunters. The shackled captives were marched hundreds of miles from the countryside to the coast, the mothers carrying babies on their backs.

At the castles, the prisoners were sorted according to age and sex – families were usually broken up – and barter