Phil Moir's Blog

Top tip - Genes Reunited blogs

Welcome to the new Genes Reunited blog!

  • We regularly add blogs covering a variety of topics. You can add your own comments at the bottom.
  • The Genes Reunited Team will be writing blogs and keeping you up to date with changes happening on the site.
  • In the future we hope to have guest bloggers that will be able to give you tips and advice as to how to trace your family history.
  • The blogs will have various privacy settings, so that you can choose who you share your blog with.

Electoral Rolls

Looking for living relatives?

Search our UK Electoral Rolls (2002-2013) and find your living relatives today.

Search Electoral Rolls

New electoral roll records

Technical Update - 29th-May-2012


Published on 29 May 2012 20:14 : technical : 28 comments : 7065 views

We have had quite a few releases since I last wrote, and I deeply apologise for this, and for not keeping members updated about progress in other areas of concern. This posting is going to focus on the changes we released this morning that are related to the search. I hope to post a blog later this week that will detail some of the other changes about all the other improvements we have made to the site since mid February and progress on other features.

Search Improvements

In the past, we have deliberately kept our searches simple, so that there is less confusion and to ensure that results are returned in a reasonable amount of time. We have been aware that this simplicity has created its own problems. Previous tinkering's with the search have usually led to a decline in performance, or less than accurate results, but we have listened to the requests to make some change. So the target for this release was to focus on BMD and Census searches, make more options available to search by, make more flexible options, return more accurate results, and to perform well. The whole team on GR has focused on making this change and we have been pleasantly surprised with our achievement, and in fact during development we found that we were able to introduce some options that previously we hadn't intended, but we felt they will be useful to members so kept them in.

Looking at "Search census records" first, the first thing we have added is an option on the Forename and Surname fields that lets you opt for either "Includes any of", "Exact match", "Starts with" or "Similar spelling". These options are replicated in all BMD and Census searches. The BMD search works slightly different and I will refer to these differences later. The "Exact match" means that we will search for the whole name provided and only return results with that name. Our aim is not to confuse these results by trying to predict what you want, but let you get straight to the specific results that matter. With regard to the Forename field, choosing "Exact match" will return those people who have ALL the forenames entered.


Search the census for Includes any of .. Forenames

Running the search above will return 2,648 results, with "Mary Barnes", "Ann Barnes", "Ann Sarah Barnes", etc. That's quite a lot to look through.


Results from the census for Includes any of .. Forenames

Now switch it to "Exact match" on Forename, and the result is quite different. There is 1 result. Also, the results came back in a few seconds or less. Sorting and moving through the pages of results is also significantly faster.


Results for the census for exact match .. Forenames

You can continue to make alterations to the search using the search box at the side of the results page, and for the Census records you can even run the same search by switching between Censuses.


Amend the search

The other options on name fields are "Starts with" and "Similar spelling". These options only check the first forename only. If more than one forename is entered, then it will check for Includes any of instead. So remember to only enter one Forename when using these options. Starts with is self explanatory, but Similar spelling requires a little guidance on expectations. This method uses Soundex which is a well known phonetic algorithm for indexing names by sound, in English. For a deeper understanding of this, read this Wikipedia article. Don't expect perfection with this method, but you may find it does find the ms-transcribed record your after. Also because of the complexity of this search, it does take a bit longer to perform. If we redo the search for "Mary Ann Barnes" with exact match on Forename and similar spelling on Surname, we get 291 matches. And here's a snippet of the results returned. As you can see it picks up amongst others, Byrnes, Burrington, Burnside, Burns, etc.


Similar spelling search

BMD searching is much the same although there are slight variations on the results returned. The BMD database does have a relevancy indicator, so we do try close matching, or return initialed results rather than just full name. Hyphenated surnames only need one part of the surname entered to be found. And the Includes any of forenames is not available as such because of the relevancy factor. Performance was a major concern on BMDs and one of the major contributors to the poor performance was the Place keywords option. We have changed this into a drop down selection by Country and County, which should ensure we can deliver results quickly.

Include other people in search

Across both the Census records and the BMDs we have added the ability to include forenames/surnames where relevant of other people that might be connected to the person searched for. For birth records, this is Mother's maiden name, for marriage records, this is Spouse's name (with variant options) and for census records, we have other forenames in household. These add complexity to the query so they will take a little longer to perform, but we hope this opens up new doors to finding your missing relations.

Lost Records - Found!

While we were testing our new search options we removed the mandatory Surname check, and by accident (or by intrigue), we wondered what would happen if we entered NO search criteria. You can try it here. There were 31,216 matches in the 1841 Census! These records have been transcribed, but because of the poor quality of the original document or image, the surname was not decipherable, so have been reported as ??? or blank or similar. So what we can now show is these hidden records for the first time on Genes. Even better, you can search using all the other options, Forename, Year of birth, Place keywords, etc. to narrow down your search and maybe that record you could never find before.


Lost records - no surname

Reference Details Shown and Searchable

On the Census transcription we now display the Reference Information. For years 1841 to 1901 this includes Folio, Page and Piece detail. For 1911, it includes the same plus 4 additional fields.


Reference information shown on transcription

This information can then be used on a new Search by reference option (Census only) for which a link appear below the standard Census search.


Search by reference link

This form requires that the core three fields be entered, but for 1911 it includes the additional fields also shown on the transcription. This result from this search will be individuals who either occupied the same building or were in similar location/area for which the census was taken.


Search by reference form

Previous Searches are back and better

You may not have noticed but the Recent Search feature was removed from the site for performance reasons earlier in the year. It used to store links to the last three searches you did, and this was done by storing a cookie on your local machine. If you cleared your cookies the information was lost, it also wasn't terribly accurate half the time and it only held three search references across the site. Not that useful. We dropped the use of cookies for this feature, and instead use a caching mechanism on our servers where we will store your searches for up to a year, so if your logged in, you can use different machines, but still see the previous search links. We also store up to 3 searches per recordset we have. i.e. 3 for 1911 census, 3 for 1901 census, etc. The link we store will re-perform the exact search, with all the same selections as you chose previously, not just the name and year. These links appear at the bottom of the relevant search form.


Recent searches

New "Year" validation on searches

Across all our searches and the Search All records feature we have implemented year validation. The intention is to save you time and deliver only relevant results. It is probably easier to show with an example. If I were to choose the following search.


Search year validation

We used to scan all our datasets looking for this person (Harry Turnbull) with this birth year (born 1880). Well obviously, he couldn't exist in the 1871 census or earlier as he wasn't born, so rather than checking and telling you there are no matches, we trim the results down to only those relevant sets of data. We will still tell you when no results have been found for valid searches, but we won't waste time checking or telling you about the others.


Only showing valid results

I hope this explanation will tempt you to try these new search features and help you to locate the records your after. We did have a few teething issues this morning, such as performance on the 1901 search, or searching using place keywords and other forenames, but we fixed these and have been monitoring usage, and all seems to be working well now. Do let us know if you like the new options and if they help you find records you hadn't been able to locate before.

Best regards, Phil Moir
Genes Reunited, Lead Developer

Comments

Profile Picture
Send Message
by NewquayOAP on 30 May 2012 14:38 :
Spotted these changes when I searched this morning. Obviously I haven't given this any real test but I have to say that this has to be a major improvement in the search facilities, particularly the ability to search census records by reference

Profile Picture
Send Message
by Graham on 1 Jun 2012 12:06 :
I think the changes are a vast improvement; especially where you can now enter both peoples names when searching for a marriage record. I always thought this would be a good thing to be able to do. :-)
Profile Picture
Send Message
by Stephanie on 1 Jun 2012 22:59 :
the change is a good idea but unfortunately i have been unable to view the free bmd as the image can not be shown when i click on to view :-(
Profile Picture
Send Message
by David on 2 Jun 2012 00:05 :
Dear Phil Moir
I am pleased to see that new search features are now available.

However for example, when searching for a Birth and adding a Mothers' maiden name, I believe that the ONLY results that should be shown are when the maiden name is matched exactly and not when it is blank. I realise that maiden names were only recorded after 1912 so in this instance I would not expect any results to show pre 1912 births.

The ability to be able to add additional forenames in the household of a Census is a feature that should enable families to be tracked easier from one Census to another but as with my comment regarding searching Births above, the results should only be those that match exactly, the additional names added. Either that or the drop down box regarding exact match etc, should be added to this field.

Also, when searching Census pages I have always thought that it would be nice (essential in cases where the family is split over 2 sheets in the earlier census returns) to be able to view either the next or previous page in the census. There are many occasions where close family lived within a few doors of each other and these could be picked up by being able to view these extra pages. Would it be possible to have a PREVIOUS PAGE & NEXT PAGE button displayed on the page to allow this.

Regards
Profile Picture
Send Message
by philmoir on 2 Jun 2012 09:04 :
Hi All, Thanks for the feedback (good and bad).

Stephanie: The free BMD image viewing problem will be resolved by Wednesday. I had thought this issue to be fixed, but appears something must have been over looked.

David: The BMD searching uses a relevancy factor, and is not designed to deliver "only" exact results. So in the case mentioned the system knows that maiden is not available and will adjust the search to not include it for that period. In some cases that will be desirable, but in others less so.

The Census search is designed slightly differently, and again in the case mentioned, if you put Thomas as an Other forename in household, you will only get those households that have a Thomas. With regard to viewing images, all the census years have next and previous buttons for viewing images EXCEPT 1901. There were technical reasons for this when we applied the option to the others, but it will be available later this year.

Hope these answers help.
Profile Picture
Send Message
by David on 2 Jun 2012 12:05 :
Hi Phil
Thank you very much for your response above.

I had completely missed the Previous/Next buttons on the census pages so apologies for raising that one. (How long have they been there because I have often sent an email to your support colleagues for this information and it has always been emailed back to me - no one has ever said Press the Button - even as recent as yesterday!)

I understand your comment regarding the search on Births when a maiden name is quoted not being designed to deliver only relevant results but surely that is the whole point of being able to enter a maiden name. I tried searching for my Grandmothers children as an example. This is my Dad's Mum. I know she had 2 boys, my Dad and my Uncle. I entered just her Surname, nothing else and get 21,748 results. However, when I add her maiden name as well, I get 47,668 results. This cannot be correct!

With respect to the census and being able to add additional forenames, again the results are not correct. The results returned are census pages where the additional name is on the same page, not in the same household.

In this open forum I have not included specifics to the examples I have quoted but I am more than happy to share the specific information with you by email if you would like to contact me, or maybe I can send an email to support and ask that it is forwarded to you.

Best regards


Profile Picture
Send Message
by Colin on 3 Jun 2012 15:33 :
Hi Phil,
Any improvement to the search facility is always welcome. However, when finding a result, I have been copying and saving the transcripts (these's bieng easier to read quickly than originals). When copied to a Word document, up to now it has saved the formatting. and the info is shown within a table. Over the last day or so this is not happening and though it copies the details they are not laid out in table form. Is this something that has been changed deliberately or by accident and if so is there any chance of reveresing it?
Other than that small niggle, Keep up the good work!
Colin Parry.
Profile Picture
Send Message
by philmoir on 3 Jun 2012 23:57 :
Hi David, With regard to the Census search, I think you may be right in that I suspect we are linking on page reference and not household. We will have to change this.

Hi Colin, We haven't made any changes to the building of the results or transcription pages so I am little confused. It should still work as before. Copy and pasting such information is generally a browser issue. You may have accidently changed a copy option. Try it again, and let me know how if you have any success.
Profile Picture
Send Message
by Colin on 6 Jun 2012 12:30 :
Hi Phil, Thanks for the reply. I tried it again and I am still having the same problem. I even tried it with a record I have already saved and again it saved the info but not in table form whereas it did previously.

I don't think I've changed anything and I've no idea how to change copy settings as I've never needed to.
I don't know if this will help but I am using Xp Pro and Word (office 2002) to save the info to. I have this problem on both my Laptop and desktop.
ANy suggestions woudl be most welcome!
Thanks
Colin
Profile Picture
Send Message
by David on 6 Jun 2012 12:54 :
Hi Phil
Thank you for your response.
I look forward to the (hopefully) slight revision that will be required regarding the census search.
Regarding the Birth search though, I would request that the decision to include all blank maiden names even if a name has been entered in the seach criteria is reviewed. This has the potential to be an extremely powerful feature if only results that match the maiden name EXACTLY are returned. Other than that, maybe add the drop down box with one of the options, to ensure only an exact match is returned. Users would not expect records with blank maiden names to be returned when a maiden name has been deliberately entered as a criteria.
Profile Picture
Send Message
by Strebor on 6 Jun 2012 17:54 :
Hi Phil,
The revised search is great. However in the BMD search is it also possible to put back the "place keywords" as well as England, Wales as this would make life a lot easier rather than trawling through looking for towns and cities in the regions.
Regards

Diana
Profile Picture
Send Message
by David on 7 Jun 2012 13:14 :
Hi Colin
I hope you don't mind me writing this but I saw your post.
I have tried to paste some searched results and pasted them into Word.
It actually worked perfectly, which is what I guess used to be the case for you based upon your posting.
In WORD, under the edit tab, there is an option called Paste Special. Mine is set to HTML, which enables the format to be copied. There is also an option for unformatted.
Is it possible that you have unformatted set now?
Regards
Profile Picture
Send Message
by David on 8 Jun 2012 00:02 :
Hi Phil
I apologise for again raising the topic of using Mother's Maiden name (mmn), when searching for births.
However, I have been carrying out some further checks, with some quite strange (but repeatable) results.

I searched for Births in 1920 +/- 2 years.
I entered the name SQUIRES in the mmn field.
First Name was left blank.
These 3 variables (mmn, First Name & Year of Birth were kept constant)

I tried with 7 different surnames:
- Davis (30 matches of which only 2 have a Davis surname)
- Thompson (29 matches of which only 1 has a Thompson surname)
- Holden (28 matches of which none have a Holden surname)
- Moore (31 matches of which only 3 have a Moore surname)
- Dawson (28 matches of which none have a Dawson surname)
- Robson (28 matches of which none have a Robson surname)
- Edwards (28 matches of which none have a Edwards surname)

In all cases the results returned have the mmn of Squires but in all cases, 28 of those results also have a Surname of Squires, NOT the surname entered, despite Surname being specified with an 'Exact match' criteria.

Incidentally, all the above appears to have something to do with the First name field being left blank, because if it is filled in with a known correct name, the search works perfectly. However, the powerful feature of being able to enter a mmn should ensure complete families (post 1912) can be discovered and of course the individual first names would not be known at that stage.

Regards

Profile Picture
Send Message
by Colin on 8 Jun 2012 16:59 :
Hi David, No, I dont mind you writing, I need all the help I can get! LOL
I have tried what you suggested,
In my versions of Word the "paste special" offers only Paste:- "Unformatted Text" or "Unformatted Unicode Text" and there appears to be no way to turn the Unformatted part off! I was using the ordinary "Paste" function before and that was working fine.
I've even reinstalled Word but to no avail.
It seems to me that the copy function is not copying the table info, I Have pasted into a blank HTML page in Dreamweaver and that does not copy the table either. But if I copy the the table as code from the source code from GR and paste that into the code section of the dreamweaver page it does show the table (albeit smaller and without the coloured backgrounds on alternative lines) I could live with doing it this way if it was not such a chore trawling through all the code to get to the actual info I want.
Thank you for trying to help and if you have any further suggestions please feel free ..........
Kind Regards
Colin
Profile Picture
Send Message
by Pamela on 11 Jun 2012 15:43 :
Hi Phil, there was a comment earlier about not being able to view free BMD images which I think has been sorted, but this is still the case with the Marriage section of Free BMD.Is this likely to be sorted soon as I've been trying for several days to view images.
Pam
Profile Picture
Send Message
by Colin on 12 Jun 2012 00:03 :
Hi Phil, Andy & All,
Regarding the problem I was having copying transcripts to Word. I checked the Microsoft help thingy and came accross an article about this very problem! The cause was the Skype "Click to call" Add On for Internet Explorer. The last version of skype messed up the paste special feature (which is why my version of Word only showed unformatted options)

I have now downloaded the latest version of Skype which fixes this bug I can now happily copy and paste to my little heart's delight and it keeps the info in the table! YIPPEEE!!!! lol

Thanks again for your help and I hope this solves the problem for anyone else suffering the same problem.

Happy hunting!
Colin
Profile Picture
Send Message
by David on 12 Jun 2012 19:15 :
Hi Colin
That's good.
Glad you have sorted it!
Regards
Profile Picture
Send Message
by Wendy on 16 Jun 2012 15:10 :
Hi Phil,

I'm unable to view the free BMD images, have been trying for a while now.

Wendy
Profile Picture
Send Message
by PGL on 16 Jun 2012 16:22 :
Hi Phil

I have the same problem as Wendy. I am looking for the death of Israel Warrington from 1871 to 1881. I can only get all counties instead of individual ones. It tells me there are 14 matches but none of them show.

regards
Pat
Profile Picture
Send Message
by PGL on 16 Jun 2012 16:29 :
Hi Phil

I have the same problem as Wendy. I am looking for the death of Israel Warrington from 1871 to 1881. I can only get all counties instead of individual ones. It tells me there are 14 matches but none of them show.

regards
PGL
Profile Picture
Send Message
by chas3351 on 16 Jun 2012 20:09 :
Hi The changes are bril !!!!
OK there are a few issues in number of records reported.
I think the best one is the ability to search marraiges with spouse forename and surname.
I will be taking a close look at your search by reference.
Well done. As an old software writter of 1980's I know how hard some bugs are to find.
Ken
Profile Picture
Send Message
by Elizabeth on 17 Jun 2012 14:20 :
As a Scot, living in Scotland with 98% of my family in Scotland,I find most websites, including Genes, do not do Scotland. I appriciate that Scotlandspeople is available to me, but it would be great if I could access something on Genes from scotland.
Liz
Profile Picture
Send Message
by philmoir on 23 Jun 2012 01:05 :
Hi, With regard to David's comments on the Birth search, and apparent wrong results. In fact this is not the whole story. In general, a child will bear the father's surname when a birth is registered, and thus you would not normally expect the child to have the mother's maiden name at registration. The 28 cases mentioned by David are in fact those where the mother's maiden name seems to have been used, and therefore we don't discount these as possible matches when looking in this way. There are never likely to be that many (I would suspect), so better that this door is left open for investigation, otherwise there may be no other way to get these records. I hope this helps explain the situation.

Re. Elizabeth and Scottish records. I am also a Scot, and most of my personal family history is Scottish, although I am able to feed my habit by doing my wife's family history (she is English). I know it has been mentioned before but just to be very clear, ScotlandsPeople is owned by the Scottish Government (brightsolid - that own Genes Reunited - only run it on their behalf) and unlike other sources of records, such as the British Newspaper archive, we do not have permission to access these records. We and brightsolid are looking at ways of getting Scottish records available to members on Genes, but don't give up the resources available in trees.

The BMD images issue will be looked into first thing next week. I am terribly sorry, as I believed it had been fixed and there appears to be some miscommunication.

Regarding the places in the BMD search. For those that were familiar with the older version of the search, you may remember it timed out a lot. i.e. It said too often that it could not complete the search. This was primarily down to the search by place keyword. It unfortunately cannot be optimised for that at the moment. Hopefully in the relatively near future we will be introduce something that will support a more open place definition.

Thanks for the supportive comments, and keep us posted on any issues.
Profile Picture
Send Message
by David on 2 Jul 2012 23:18 :
Hi Phil
Thanks for the explanation.
Regards
Profile Picture
Send Message
by John on 5 Jul 2012 04:43 :
Hi Phil
A question on another matter.
I use Firefox as my principal browser but just recently in the last few days the GR site has not been displaying properly as the mini family tree does not come up and the ads at the top of the page do not display (I can live with this!).
Additionally and much worse, the family tree does not display at all. All that happens is that the new tab opens and I get a little twirly circle on a blank screen saying 'please wait'.
I have to use the dreaded IE to use GR now. Is it an incompatibility with a Firefox upgrade or is there a deeper and more sinister reason?
Profile Picture
Send Message
by David on 31 Jul 2012 18:39 :
Hi Phil
Further to my many comments above regarding the entry of Mothers Maiden Name (mmn) when searching Births, I would like to suggest a different solution.
When a relatively short date range is added, it appears that the search does in fact list out the exact matches (ie mmn and Surname) at the top of the list.
Would it be possible to ALWAYS list out the most likely correct matches of both mmn and Surname if the mmn is entered, before all the blank mmn entries. This I believe would give the best of both worlds. It would ensure that after 1912 when the mmn became available that it is possible to identify families relatively easily, whilst also ensuring that the other entries where the mmn is blank are still listed.
Profile Picture
Send Message
by Ian on 7 Aug 2012 22:00 :
New tree is disastrous...Slow and cumbersome.

Admit it and get the old one back ASAP.....

Otherwise you are just ignoring the obvious....
Profile Picture
Send Message
by Robert on 25 Aug 2012 06:27 :
Are the people with Genes big brother FIND MY PAST having these problems?

Bob.