Computer Vision and
Via Ferrata 5, 27100 Pavia - ITALY
web-vision@unipv.it
+39 0382 98 5372/5486
Generative AI techniques, emerged in recent years, can automatically create multimedia content of any type. In this specific case, they are applied in the field of digital humanities to create images, that show a possible solution for the virtual reconstruction of damaged, or with missing portions, frescoes of the Castle of Mirabello. Goal is to create immersive and interactive experiences for a virtual visit of the castle, with the integration of real and virtual items into a digital context.
The output image generation can be ex-nihilo, or driven by a series of user inputs, such as prompt, a starting image, and a selection. The most common is the prompt: it is a text sequence containing a few hundred characters (the specific number depends on how the starting neural network was originally trained). The text is created freely by the user, and is used to describe, in as much detail as possible and using a semi-natural language, the desired characteristics of the output image. The prompt contains both positive elements (what the image should contain; how the image should be generated) and negative elements (what the image should not contain, and so on). The prompting is the art or technique of creating text strings to guide the neural network in generating output appropriate to the user’s application and expectations. Note that within the prompt can be summarized, in a machine-understandable language, the historical descriptions of the artwork, or the results of the studies of experts on the object to be preserved (art historians, restorers, etc.).
In the case of reconstruction or extension of existing images, the starting image is the second key input, in addition to the prompt. In particular, when the starting image is deteriorated, damaged or has missing portions, the technique is called in-painting: the neural network attempts to complement the input image by generating the missing content. In-painting contrasts with out-painting, in which the generation occurs beyond the boundaries of the starting image. A reconstruction, in fact, uses both techniques in a complementary way. It is essential to understand that the generation is strongly driven by the input image pixels, and the network will tend to maintain a relatively short "distance" between the content generated and the starting one. Such a distance can be estimated both in terms of low level, such as the gradient of colors, brightness, etc., but also at higher level; thus are preserved pictorial style, represented subjects, defects or imperfections inherent of the original, etc.
The field of action of the generation can be further influenced with appropriate selection of the virtual "canvas", so that AI generates new pixels (using prompt and starting image) only in the parts that have actually been selected.
Equally important is the evaluation of which portions of the starting image is the input, which are more relevant (it is possible to weigh different areas for importance) and which instead should be ignored. Finally, it is important to understand that regardless of good prompts, starting image and selection, the output is always random, and is driven by a number of technical parameters that the expert must carefully adjust. Every neural network is a separate case. Altering the training of the network, even in small part, significantly alters the knowledge base of the network itself, from which the pixels that constitute the output image are synthesized.
Although many networks are publicly available already trained, the quality and scientific reliability of the result can only be ensured by retraining the network, weighing significantly higher a carefully selected dataset. One technique used in the field of digital humanities is that of the collection of focused database that contain, for example, works of art: of the same author; of the same historical period; of similar style; located in geographically close places; etc. The choice of this database requires a strong knowledge of the domain and therefore the action of the AI expert must be guided, also in this case, by art history experts.
In conclusion, the process of reconstruction with inpainting techniques of the frescoes, object of this project, requires high specific skills, both in the technical and humanistic field, and the scientific value of the result, in addition to being validated by the many examples already in the literature, it is ensured by the use of techniques that allow to govern the neural network step by step. In addition, the digital reconstruction of the frescoes with Generative AI techniques can be flanked by the 3D reconstruction of the castle in which these frescoes are housed, in order to provide a "virtual canvas" on which to place the reconstructed frescoes.
Castle's history. A primitive castle, with a tower and a fence, was built between 1325 and 1341 by the Fiamberti family. In 1360 Galeazzo II Visconti bought half of the castle, with its lands, while the remaining part of the complex was expropriated. Gian Galeazzo made the building restored in 1384: located in the Visconti Park, which connected the Castle of Pavia to the Cartosa of Pavia, it was surrounded by walls and towers. From 1425, with the renovation of the park wanted by Filippo Maria Visconti, the castle acquired a deep symbolic value and the fourteenth-century building, which was probably in poor condition, was transformed into a house for the Duke’s residence and the Park Captain’s seat, responsible for the custody and management of the ducal assets. The park, mainly because of its abundance of water, was a source of different materials: cultivated land, vineyards, orchards, woods for timber, pastures, livestock, dovecotes, fish ponds and a large amount of game reserved for hunting, with the addition of production facilities, like the mill and the furnaces. The castle looked like a fortified hunting palace, protected by a wall and a moat with three drawbridges. In 1472 Galeazzo Maria Sforza had the building renovated and enlarged, and it was repeatedly used as a seat of political representation, during the hunts and banquets organized within the park. Probably around 1491, Gian Galeazzo Maria Sforza donated the castle to Galeazzo Sanseverino, who had the building renovated, transforming it into a larger country residence and adapting it to the new canons of noble representation.
The works suffered a setback in 1496, then resumed in 1501 and continued (with a short interval from 1512 to 1515 when Massimiliano Sforza resumed control of the duchy) until 1521. The castle was damaged during the French siege of Pavia in 1522. During the Battle of Pavia in 1525 the castle hosted the King of France Francis I, taken prisoner, and with the death of Galeazzo Sanseverino, who fought in the French ranks and tried to defend King Francis I from capture, the complex returned to the availability of the Ducal Chamber. It was then managed (along with the vast land properties attached) by the various captains of the park that followed each other until 1754, when the Magistrate Camerario of the State of Milan put on sale the large property that was bought by the Milanese aristocrat Antonio Giorgio Clerici. After Clerici’s death, his daughter sold the entire property to the San Matteo hospital in Pavia in 1768. Between 1854 and 1860 the administration of the hospital had demolished about a third of the structures and the remaining parts had various uses: tavern, home of farmers, elementary school and the seat of the town of Mirabello. Since the 1960s, the hospital administration has been renting the apartments to private individuals. In 2002, the castle was purchased by the municipality of Pavia.
The castle today. Purchased in 2005 by the Policlinico San Matteo, the entire area has been subject to redevelopment of the green, with the elimination of climbing and the arrangement of the roof of the eastern wing, at risk of collapse, and the attic. For the inauguration of the reopening of the park, in October 2022, for a whole weekend guided tours were organized and a camp was set up to show the field life, clothing, armor and armaments at the time of the Battle of Pavia, accompanied by the cooking of the zuppa alla pavese (Pavia soup), whose recipe has reached our days. It is said that Francis I, taken prisoner by Spanish soldiers after the defeat of his army, was brought inside the Repentita farmhouse (still existing) to heal the slight wounds he had received in the fight and refresh himself with some food. The local peasant, according to legend, had at that time only meat broth, dry bread and some eggs. She then put the bread in the boiling broth and added the eggs, serving the captive sovereign a simple but tasty dish.
The frescoes. The frescoes that decorated the rooms are still largely hidden under several layers of plaster, both on the ground and first floor. On the ground floor was brought to light a coat of arms engraved and painted by Sanseverino, while some essays highlighted the presence, on the first floor of the building, of a painted band, made with the technique of chiaroscuro and about a meter high. The important cycle of frescoes, which runs under the shutter of the wooden ceiling of the four rooms of the first floor, presents a recurring decorative theme. Numerous essays carried out in the spring of '96 by Dr Paola Zanolini have highlighted a continuous theory of wreaths and winged putti supporting cornucopiae ending with spiral decorations, pottery and candelabra between floral motifs.
The great fresco coat of arms brought to light on this occasion is that of Sanseverino, the noble family of Salerno origin who had occasion to emerge, especially with Galeazzo Sanseverino, at the court of the Visconti and then at the court of France. The coat of arms bears the arms of the Sanseverino (the stars) divided into four sections with those of the Aragona (the vertical bands) and the Visconti snake. The fresco depicts, around the coat of arms, the collar of the Order of Saint Michael, with the image of the Archangel.
Following are some photos of frescoes of Mirabello Castle rooms, as they are now: a cycle of frescoes of the first floor and two photos of a 16th century decoration of the South facade.
Here are the frescoes of the first and second western rooms of the castle and those of rooms, stairs and first eastern floor, left and right.
