IF you’re a fan of ITV’s Sunday night costume drama Downton Abbey – and let’s face it, why wouldn’t you be? – you can’t help but have picked up on the fact that it is set right here in North Yorkshire.

The servants talk about going to church in Easingwold. The young middle-class solicitor, Matthew Crawley, who thanks to the sinking of the Titanic suddenly finds himself heir to the great stately pile and to the title Earl of Grantham, talks about becoming a partner in a firm of solicitors in Ripon.

There is a reference to a Thirsk undertaker who will pick up bodies on Sundays. And members of the existing Earl of Grantham’s upper-class family talk about going hunting with the York and Ainsty.

So Downton Abbey is obviously supposed to be here, somewhere. But where?

From the references, you would imagine it must be somewhere in that triangle of broad acres between Ripon, Thirsk and Easingwold. But, due respect to Brafferton, Helperby and Crakehill, there is nothing much there. There is Newby Hall just across the A1 – but you only have to look at the photos of Downton to know instantly that that’s not it. Or at least, it’s not where the programme-makers chose to film the drama.

Cast the net a little further afield, and we have no shortage of stately homes – Allerton Park, Beningbrough Hall, Castle Howard, Hovingham Hall, Sutton Park and Nunnington Hall among them – which could have been pressed into use as a pretty decent Downton Abbey, if the makers of the programme had so wished. But clearly they didn’t, because it isn’t any of them. Look at the photo.

The National Trust doesn’t know where Downton could be. It’s certainly not anywhere in Yorkshire that belongs to them, says Clare Fletcher, the Trust’s learning and communities officer at Beningbrough Hall.

If she’d been asked by the programme makers about good possible locations to film the series, she’d have suggested Allerton Park or Nostell Priory, she says. But she wasn’t, so she didn’t.

Neither was Trevor Mitchell, regional director of English Heritage. They have plenty of fine old properties in Yorkshire – but none of them fit the bill either. And English Heritage doesn’t have a ‘secret’ country house hidden away somewhere that it has never told anyone about, he insists.

Then he has a thought. Downton Abbey is set in 1912. So is there, perhaps, a great country house that, between then and now, has been pulled down – and that perhaps inspired the Downton Abbey of the programme?

He consults a book: Lost Houses Of York And The North Riding, by Edward Waterson and Peter Meadows. There are a couple of possibilities, he says: Thirkleby Hall near Thirsk, the magnificent country seat of the Frankland Family, was one. The family didn’t survive the First World War, the estate was sold in 1919, and the great house fell into disrepair. It was bought by an American in 1927, who took the splendid Italianate façade to the United States. The rest of the house was demolished.

Another possible candidate, Trevor says, might have been Wood End near Thornton-le-Street, one time home of the Earl of Cathcart. It was eventually sold, in 1921, to a group of “farmers, timber merchants and antique dealers” known collectively as the “forty thieves” because they gradually dismantled the house.

Another vanished stately home, if a little further afield, is Warter Priory at, well, Warter. Once the home of Lord Nunburnholme, it fell into disrepair and was, sadly, demolished after the last of its contents were sold at auction in May, 1972. An old photograph of the Priory reveals a large, square-built building with corner spires: very much in the character of the Downton Abbey we see on-screen.

Could this have been the inspiration for the house in the series?

The drama was written for TV by the actor and screenwriter Julian Fellowes – a man who clearly knows something about the lives of the upper classes. Sadly, he couldn’t be contacted at the time of writing.

But recent media interviews he has given indicate that he had one stately home in mind all the time while writing – and it isn’t in Yorkshire at all.

It is Highclere Castle in Berkshire, home of the Earl and Countess of Carnarvon.

The Downton Abbey production team apparently spent six months visiting country houses that would be possible locations – including a number in Yorkshire, a spokeswoman assured The Press – but in the end Fellowes insisted on it being Highclere after all.

“I love Highclere and wanted Gosford” – his Oscar-winning film Gosford Park – “to be at Highclere,” the writer and actor said in one interview. “To me, Highclere is a unique architectural statement and tells us much about the confidence of the late Victorians and the confidence of high Empire.”

So that’s one mystery solved. But another still remains: which stately home inspired Fellowes’ Downton? The great house is clearly supposed to be in North Yorkshire. So maybe the writer was, after all, inspired it by memories one of the region’s lost stately homes.

It’s a nice thought. Warter Priory was in the East Riding, not in North Yorkshire: but it’s not that far off. It would still be my best bet. And what a wonderful tribute the series would make to a vanished great home.

• The next episode of Downton Abbey is on ITV1 at 9pm on Sunday.

• Highclere Castle has been home to the Carnarvon family since 1679. Built on an ancient site, the original house was recorded in the Domesday Book.

The present-day castle was designed in 1842 by Sir Charles Barry, the architect also responsible for building the Houses of Parliament in Westminster. It is set in 1,000 acres of grounds designed by Capability Brown.

The castle’s history also includes a connection with ancient Egypt, as the 5th Earl, with Howard Carter, discovered the tomb of Tutankhamen in the Valley of the Kings in 1922. The castle now houses an exhibition that commemorates this historic event, including some rare antiquities from the 5th Earl’s earlier Egyptian excavations.

• Beninbrough Hall may not be Downton Abbey, but it is still the place to go if you want to get an idea of what life was like ‘downstairs’ in a great country house in Victorian and Edwardian times.

The hall’s ‘downstairs tours’ take in the kitchens and laundry, and give you a chance to meet the butler, visit the wine cellar and pop into the room where the housekeeper kept the keys.

Tours are free and take place at 2pm and 3pm every Tuesday afternoon until the end of October.