“They all showed me their ID cards one by one as they came in.”
It was just one of many examples of racism he experienced as a black man working in the City and later for big media companies, including shocking pay disparities and few opportunities for promotion.
Mr Abbey, who has written a book about his experiences, Think Like a White Man, says the lack of black people at the top in business is a big part of the problem, and like others he is calling for change.
Britain’s biggest business lobby group, the CBI, has just unveiled a campaign to get at least one ethnic minority person onto every FTSE 100 board and every FTSE 250 one by 2024.
It says that more than a third of the boards of the biggest listed UK businesses are still all-white, with that rising to two thirds in smaller public companies.
Meanwhile, one of the UK’s biggest investment firms L&G says it will use its vote to put pressure on bosses who don’t make their top teams more diverse by 2022, putting the people responsible for board appointments in the firing line.
“You need people on the board who actually reflect your customers and society, but a lot of boards look like nothing has changed since the 1970s,” says Mr Abbey.
“You think, there are almost two million black people in the country, surely one of those people are good enough to be on your board.”
Boards help direct a company’s affairs, so making them more diverse helps to change attitudes across the whole company, experts say.
It makes business sense too. Those with more ethnically diverse senior leadership teams are on average 36% more profitable, according to research from consultancy McKinsey.
Lord Karan Bilimoria, the CBI’s president and co-founder of the well-known beer brand Cobra, says he’s seen the benefits first hand, both in his own business and in other firms whose boards he has sat on.
“In the early days of building up Cobra our company literally was a mini United Nations. We had Americans, Europeans, people from different parts of South Asia, different religions, races and creeds,” he tells the BBC.
But the British-Indian businessman says too few firms offer such an “amazing melting pot” and change has been “painfully slow”.
That is why he is spearheading the group’s Change the Race Ratio campaign, which also calls for companies to set tough targets to improve diversity and publish them within 12 months.
It won’t be easy. The CBI’s 2021 goal for ethnic diversity on FTSE 100 boards was actually recommended by the government-commissioned Parker Review back in 2016, and earlier this year that report’s author said it would be “challenging” to meet.
But Lord Bilimoria thinks the appetite for change is growing, with this summer’s Black Lives Matter protests lending a new urgency to the cause.
“We had the Davies Review [on improving the gender balance on boards] in 2011, and just last week we heard in the FTSE 350 there is only one company now that does not have a woman on their board,” he says.
“Our aim is to do the same for ethnicity.”
Mr Abbey thinks the CBI is sincere but believes it will take “more muscle” to fix the problem. He prefers L&G’s approach, saying that “things change when money talks”.
For many, however, there won’t be real change until the government steps in – something Theresa May’s government promised to do after the Parker Review.
She launched a consultation on making it mandatory for companies with over 250 staff to report their ethnicity pay gaps – the difference between what white members of staff get in average pay versus those from black, Asian or minority ethnic backgrounds (BAME).
A similar requirement to publish gender pay gaps came into effect in 2017, with the first reports being due in April 2018. But the results of Mrs May’s consultation on ethnicity have never been published.