AI-ding History

Colourisation

The machine learning model performs colourisation based on semantic meanings, such as grass is green, the sky is blue, and faces are pink. While we can be certain the photographed subjects had pink faces and hands, clothing colours are impossible to prove, but add to the realism, as Lady Marion demonstrates.

Which do you prefer? Lady Sophia Frances Cust, later Lady Tower (1811-1882), John Hume Egerton, later Viscount Alford, MP (1812-1851) and Captain, The. Hon. Charles Henry Cust (1813-1875) as a boy. The B&W original or in colour with upscaling x 2. Sophia shows one of the defects of AI, a 'zombie' left hand, an unrecognised shape for colourisation.

Synthetic images

AI is proficient in producing impressively photorealistic high-quality photos of faces that don't exist. This is the 1st Baron Brownlow reimagined.

Facial recognition & verification

If we have an unknown photograph or portrait of a person is there a way to identify that individual?

Face detection picks out faces, but without remembering details. In the examples below, a purple box is drawn around that face. 

Face recognition uses that detection to identify a face in a photo or portrait against a pre-existing database of faces. Face verification compares two faces and gives a probability of them being identical. 

Here we use two independent machine learning programmes. Azure's Face API gives a probability between 0 and 1, typically 'yes' to the same person if 0.5 or above, 'no' if <0.5. Amazon Rekognition's face comparison provides a similarity score, typically >85% for the same person.

Below are examples of Belton faces compared to a reference image of Kitty Brownlow in 1952 the year of her death.

Using face verification

Let's take an oil painting bought at the 1984 auction as possibly of Kitty Brownlow and apply AI. Comparing this painting to photographs of Dorothy Power. The results detailed below support that this painting is Lady Dorothy Brownlow (née Power) or her during one of 4 earlier marriages. Peregrine Brownlow married Dottie, as she was called, in 1954 after Kitty's death as his second wife. 

A Silver Locket

Further confusion arises with a silver locket engraved with Leila. Stated is, a photographic portrait of the late Lord Brownlow's second wife. But his second wife was Dorothy above!

This is Leila Reynolds, or during one of two previous marriages as Mrs Player or Lady Manton. The oil painting of Dorothy and the locket photo belong to different people. Confidence is 0.12. The locket image belongs to known photos of Leila as below. She married Peregrine Brownlow as his third wife in 1969.

A Flapper

AI confirms the left-hand photo, below as Peregrine Brownlow's cousin, Barbara, Maud Brownlow's sister's daughter.

Facial recognition and identification is not related to the colour of the image or the medium

Below right is Phillip Mercier's oil on canvas 1725 conversation piece at Belton with Eleanor Brownlow, Viscountess Tyrconnel in her wheelchair. Left is Mercier's preparatory study of Eleanor in red, black and white chalk on watermarked paper. The two faces have a 74% match based on OpenCV Face Recognition.

Mrs Frances Dayrell in the swing and her husband Francis Dayrell have a failed match with Eleanor's pastel (<66%).

A 'new' portrait of 'Young' Sir John Brownlow?

Here arises uncertainty with C17 portraiture. The arbitrary cut-off of 0.5 for 'yes' or 'no' is too crude. The Belton oil painting of an 'unknown' on the left is compared to know portraits of Young Sir John’ Brownlow (YSJB), family members and unrelated faces from the C17. We know that Old Lady Brownlow paid £6 for painting of YSJB for his marriage on March 26, 1676. Could this be the painting of the sixteen-year-old? 

We only have three verified portraits of YSJB, making a statistical approach difficult. We must rely on a graphical representation of the data for inference. And so, facial identity is explored with the two machine learning programmes above and then finally with a third, the VGG Face Descriptor.

Euclidean distance

Since preparing this article, I have updated the face verification with a fourth painting of YSJB from Grimesthorpe Castle (above left, unknown above right), acquired May 2022. 

A third way of determining similarity is to measure the Euclidean distance between the face embedding vectors using the VGG Face Descriptor (Visual Geometry Group, University of Oxford). The closer the faces are related, the shorter the Euclidean distance. Again, the reference is the unknown man versus the known 4 YSJB portraits and 50 C17/C18 male portraits. 

A box & whisker representation of the distances is given below. The boxes indicates the median, lower & upper quartiles, the whiskers the two extremes.

As with the two other methods, the four YSJB portraits have the shortest distance to the unknown.

Statistical Analysis

Student's t-test can be used for normally distributed samples of any size down to N=2. The Lilliefors test of Normality indicates that the 4 YSJB portrait Euclidean distances are from a normal distribution. For the 50-portrait sample, the magnitude of the difference between the sample distribution and the normal distribution is medium, p<0.05. However, The t-test is valid for large samples from non-normal distributions. Applying the t-test to the Euclidean distance of the 4 portraits of YSJB and the unrelated portrait sample, the YSJB oils are highly significantly closer to the unknown portrait. The p-value equals 0.00006675.

The Permutation Test also confirms the difference between the mean Euclidean distances as highly significant.

Hence, both graphically and statistically the Unknown Young Man is highly likely to be another portrait of YSJB.

Wigs & face identification

Identifying the portrait faces is befuddled by the narrower face of bewigged men. Apparent from the rectangular shape of the identifying boxes around YSJB and the squarer boxes around the modern images above. AI programmes are trained on 21st century photographs of real faces. A C17 artist's interpretation of a bewigged head lacks feature details present in modern photographs. The box plot below shows the facial area calculated by a machine learning face detection programme in 10 each of C17 male portraits and C21 male photographs. The bewigged face area is significantly less in size than modern male (Mann-Whitney U test P<0.01). The condition of the painting and the quality of photography will all affect the result. This explains both the difficulty in identification and ageing historical portraits.

Face Clustering

Face clustering is unsupervised learning, different from above - we have only the faces themselves. From there we need to identify and count the number of unique people in a dataset. Each resulting cluster (ideally) represents a unique individual. Once again a face embedding 128-dimension feature vector is extracted from each face then the vectors close together are clustered, graphically. Below is the result for Kitty Brownlow, Dorothy Beatty and YSJB included are the Unknown Young Gentleman and the painting labelled as possibly Kitty.

The faces were submitted to machine learning altogether without any labelling. The result is clustering almost as a human would predict. The Unknown Young Gentleman clusters with YSJB. The exception is the Kitty painting which falls into the unknown category along with two Dorothy photographs. This approach works best with a near full face view rather than profiles.

Text Mining

Analysing large chunks of text as found in the Brownlow archival documents is another target for AI. Text mining transforms the free (unstructured) text in documents into normalized, structured data suitable for analysis. One such role is the extraction of keywords from the Belton Establishment accounts, in this case the voluminous records kept by John Trigg, steward to Alice Brownlow (transcript March to January 1703, Julian calendar). Is there a difference in purchases when at Belton during 1703 compared to when living in rented accommodation at Holland House, London the same year? Having extracted those nouns a word cloud highlights differences based on the top 150 words keeping Trigg's vernacular spelling.

Named Entity Recognition

is another machine learning process, e.g.

“Person”: Mark Zuckerberg, “Company”: Facebook, “Location”: United States

Applied to the 4,799 word letters of Lucy Cust, we isolate,

Allington, Denton, Baker St, Mrs Fane, Batchelor, Holywell, Henry, Kelly, Brownlow, Fulbeck among many others all names and places from her letters.

Sentiment analysis 

reports these letters as negative with 63% probability. The letters reflect her anxiety about losing parliamentary seats in a coming general election.