Rembrandt’s iconic The Night Watch masterpiece, which hangs in the Rijksmuseum in Amsterdam, has been returned to its “original” state thanks to artificial intelligence (AI). The painting had been cut down during the 18th century and the removed pieces subsequently lost. Now, scanning technology and AI. have attempted to recreate the missing fragments using convolutional neural networks (CNN) used by computers to analyze visual imagery. This is the latest experiment that suggests AI could play a fundamental role in art restoration.

AI Restores Damaged Masterpiece 

The monumental painting, from 1642, now measures around 15 feet wide by 13 feet high. It was reduced to these dimensions in 1715 to squeeze into a space between two doors in Amsterdam’s Town Hall. The remodelling removed two feet from the left of the canvas, nine inches from the top, five from the bottom and three from the right side. 

A copy of the painting by Gerrit Lundens made 12 years after the original gave restorers an idea of the content of the missing sections. Three figures had disappeared from the left-hand side meaning the original composition had a different focal point. However, the copy is far reduced in size, different in artistic style, and not entirely accurate in composition. 

Here, AI technology stepped in. The Rijkmuseum’s senior scientist, Robert Erdmann, chose to use computer technology to recreate the lost fragments. He adopted CNN and fed the computer high-resolution scans of Rembrandt’s work. This enabled him to train the computer to reproduce the artist’s style. Using Lundens’ copy as a rough guide, the computer was thus able to recreate the missing sections in Rembrandt’s manner. 

The team then printed these newly created pieces onto canvas and varnished them to resemble a painting. Attached to metal plates, they will hang beside the picture to give visitors an idea of Rembrandt’s original conception. 

AI for Art Restoration and Conservation

This is not the first time AI has helped unveil the hidden secrets of an artwork. In 2019, researchers used AI to analyze the Van Eyck brothers’ Ghent Altarpiece (1432), one of the world’s most renowned paintings. Because many of the altarpiece’s 12 panels are painted on both sides, X-ray images can be difficult to interpret. The research team therefore used a newly developed algorithm to deconstruct the information within the X-rays. This allowed them to uncover hereby unknown details about the double-sided panels of Adam and Eve. 

The researchers used CNN to created one-sided paintings of the two panels. They trained the deep neural network with color photographs of the two sides of the panels as well as the mixed X-ray of the whole panel. AI was then able to develop an algorithm to separate the X-ray into two distinct images of the two sides. 

Thus, the restorers could read the X-rays separately. This helped them identify damage to the painting and areas where varnish or overpainting conceal cracks and structural problems. It also allows researchers to learn more about the artist’s working techniques. For example, they could identify built-up layers of paint or modifications made while working. 

“This approach demonstrates that artificial intelligence-oriented techniques—powered by deep learning—can be used to potentially solve challenges arising in art investigation,” said Dr. Miguel Rodrigues of the Electronic & Electrical Engineering department at the University College London.

Commenting on the Ghent Altarpiece experiments, the website Oxbridge Applications writes, “Students planning to apply for History of Art can reflect on how this type of AI-assisted analysis may potentially prove useful for countless art applications, including material identification, brush-stroke technique analysis and perhaps even visualisation of previously unidentified, hidden designs and underdrawings.”

AI in Art Creation

The technology used for Rembrandt’s painting restoration has also been the subject of experimentation for AI art production. Google’s DeepDream is a neural network that can identify certain objects and shapes and produce “art” in distinct styles after having analyzed millions of images. Using style transfer, the neural network can transform a photograph into a Van Gogh-esque painting, for example. 

In 2018, Christie’s auctioned its first A.I.-generated artwork. Portrait of Edmond Belamy is one of a series of portraits of the fictional Belamy family. Obvious, a Paris-based collective, created the paintings using a method they call “generative adversarial network” (GAN). 

“The algorithm is composed of two parts,” said the collective, speaking to Christie’s. “On one side is the Generator, on the other the Discriminator. We fed the system with a data set of 15,000 portraits painted between the 14th century to the 20th. The Generator makes a new image based on the set, then the Discriminator tries to spot the difference between a human-made image and one created by the Generator. The aim is to fool the Discriminator into thinking that the new images are real-life portraits. Then we have a result.” The AI signed artwork sold for an unexpectedly high $432,500 at auction. 

AI has been used in other ways in the art world, for example bringing old paintings to “life” – including The Night Watch.