If you’ve used our website in the last 18 months, you’ll be aware that we’re in the process of replacing our old Catalogue with a brand new shiny one. Discovery, our new catalogue, has been designed to provide a similar experience to the search tools found elsewhere across the web, with features like filters (to refine searches) and tagging (to help other people find records). It has also been designed with large volumes of data in mind – the old Catalogue was creaking under the weight of several million records, and with many more to come in the next few years (especially with the impending shift to 20-year rule, and the expected arrival of even more government records) we decided that the time had come to build a new one.
Screenshot of Discovery
Right from the outset we knew that it would be an enormous undertaking. Our ambition has always been to replace not just the Catalogue, but also to incorporate data from other systems such as DocumentsOnline, the National Register of Archives and the ARCHON directory, in order to provide a ‘one-stop shop’ for anyone wanting to search our collection – and those held by other archives – and to download digital copies of records where available. The practicalities of doing this have been challenging, to say the least – with a number of different technical systems to deal with, along with enormous amounts of feedback from our many different user groups and staff, it has taken time to build a system that fulfils all our users’ needs.
Our approach has been to develop Discovery over time, gradually adding new features and functionality. Discovery is a very different system to the old Catalogue, and works in a different way – the iterative approach has allowed us to test it, help users get used to it and gather feedback that has helped us improve the system as we go. We’ve also been able to fix minor technical issues (bugs) as they are reported, releasing new versions of Discovery on a fairly regular basis.
The last major development came in September, when we launched the new browse feature (and switched off DocumentsOnline). We had a big push to ask for feedback that would help us identify any remaining ‘showstoppers’ (major issues) and decide when we could switch off the old Catalogue, without causing problems to our users. We received a mix of comments, complaints and suggestions – all really useful! We’ve already fixed some of the major sticking points, and are working on others, but we thought it would be helpful if we explained some of these in more detail.
How search results are displayed
The order in which search results are returned is probably the main bugbear of both users and staff. At the moment results are displayed in relevance order, using logic based on the search term used. This is how most search engines work, and is fairly standard across the web. However, this is not how the old Catalogue returns search results (based on catalogue reference logic). We’re trying to improve our understanding of each option, and the implications of each; we’re going to look at users’ motivations for searching, and how results may vary across different record series. We’re not sure how long this process will take, but it’s currently our highest priority – watch this space.
Searching digitised records
Since we moved our DocumentsOnline collections into Discovery, we’ve encountered a few problems. Many users have reported that when using the quick search boxes in our online records pages they’ve encountered odd search results; for example, users searching for the surname ‘Bedford’ in our wills collection have had lots of results for the place name Bedford.
Digitised collections have their own sets of metadata – the information attached to digital files that is used to search and browse – that doesn’t quite match up with the data contained in catalogue descriptions. This is because we have additional metadata which is more detailed and structured than the related catalogue descriptions are. In the digitised collections metadata, Bedford is marked as either a name or a place depending on the usage but this is not how the catalogue descriptions work. In short, the two sets of metadata don’t quite match up, and as a result the search functionality that’s been built into Discovery – to search catalogue metadata – can return unexpected results. Fixing this is a fairly complicated process, but we’re on the case.
Browsing
Browse Discovery
On the whole we’ve received very positive feedback about the new browse feature, which we’re obviously delighted about! Some users (and staff) have asked to see a bit more text from the catalogue description, to help with the browsing process – at the moment only one line of text appears, and you have to click into the full description to see more information. Once again though everybody wants something different, so we’re probably going to run some user testing to make sure that the solution we propose will work for at least the majority of users.
The new release
This week we’ve released a new version of Discovery. This fixes some bugs that had been reported, as well as issues that were causing problems (in one case we were accidentally sending users round in circles, which we’ve hopefully resolved simply by removing a confusing link!).
Keep talking to us!
In all, we’ve already received thousands of comments, compliments, complaints and suggestions about Discovery, which are logged for review and prioritisation. We always intended Discovery to be a work-in-progress, to an extent – it has been built to be flexible, in order to accommodate the changing needs of our users, along with the expansion of our collection over the next few years. That’s why we need you to keep telling us what you think of it, because we can’t fix your problems if we don’t know what they are!
You can provide feedback by email, web form, Twitter (using the hashtag #tnadiscovery), via the comments below, or in person, if you’re visiting our reading rooms or attending our regular user forum.
We’ll be back with another blog in a couple of weeks’ time, with news of the progress we’ve been making on integrating data from other archives (such as National Register of Archives and ARCHON) into Discovery – stay tuned!
Emma,
Thanks for the updates and it is a big but very worthwhile project.
I don’t understand these index terms and how TNA have got to the terms attached to the records. To take two examples, a complaint from The Speaker about precedence at Remembrance Sunday is indexed under ‘Public Disorder’ which is a term for riots or rebellion and the presentation to Parliament of the White Paper on public expenditure which is a Command Paper and therefore a Parliamentary Paper is indexed under ‘Personal and Family Papers’. Since the new version of Discovery was released a number of files are now described as having no description and the description is lower down or you have to click on details to find the description.
The issue of people’s names and places is an issue, Nelson comes to mind and unless they can be seperated it is an issue that is hard to solve. There is the problem that when you click on ‘details’ of some of the entries it sends you up the page and it is difficult to scroll to the next or previous page.
Hello,
Will there be a way to select a particular database only, and conduct all searches within that db, without having to go back to the basic level ‘refine your search’ bit each time you change criteria? My searches are only in PCC wills, and where previously you could select ‘only’ wills, and then refine by surname/place/date as many times as you liked from within that area, now you have to start at the beginning each time, reselecting the dataset, the date-range, etc, which is quite time-consuming.
Also, it would be extremely helpful to be able to search by strict date order, and in smaller periods of time than just by century, which I can’t seem to find how to do thus far….? I am struggling with navigating the site – the refine’ function is just not powerful enough to do precision searches, but I appreciate such things are being worked on and should in time make more sense!
I agree with Helen, at the moment the latest date to search is from 1950 which is wide, is it not possible to have an additional date as it produces a lot of records. Quite often the earliest dates are for series that started hundreds of years earlier and include later records.
I would also like the latest releases to be listed better, for example you can see the records released in the last week or the last six months but nothing in between. In the old days TNA used to have a paper list but it now seems this is not being replaced electronically.
IM TRYING TO LOCATE A WILL OF MY ANCESTOR WHO WAS A LIEUTENANT COLONEL IN THE BRITISH ARMY DURING THE AMERICAN REVOLUTION, HIS NAME WAS CHARLES COOKE, HE WAS A LAWYER