And so we venture further and further into this new frontier between fiction and reality. This new technology stands to redefine not just technological capabilities but also the possibility to create new emotional landscapes to explore.
26 April 2024 — Human memory is unreliable. It’s unreliable because we forget details, remember things incorrectly, believe things happened when they didn’t, and have different perspectives. We rely on human memory for everything, especially when it comes to recreating the past.
The past can be recorded and recreated but one company is transforming the past into real images, when none were taken. The MIT Technology Review explains how in article titled “Generative AI Can Turn Your Most Precious Memories Into Photos That Never Existed”. The article might be behind a paywall when you click thru (all the good stuff is always behind a paywall these days) and I cannot get it onto my Slideshare, unfortunately, so I will clip a few paragraphs. The introduction, quoting Pau Garcia, founder of the company:
Maria grew up in Barcelona, Spain, in the 1940s. Her first memories of her father are vivid. As a six-year-old, Maria would visit a neighbor’s apartment in her building when she wanted to see him. From there, she could peer through the railings of a balcony into the prison below and try to catch a glimpse of him through the small window of his cell, where he was locked up for opposing the dictatorship of Francisco Franco.
There is no photo of Maria on that balcony. But she can now hold something like it: a fake photo—or memory-based reconstruction, as the Barcelona, Spain-based design studio Domestic Data Streamers puts it—of the scene that a real photo might have captured. The fake snapshots are blurred and distorted, but they can still rewind a lifetime in an instant.
The image:
The “synthetic memory” image created through AI tools. It recreates Maria’s childhood memory “of seeing my imprisoned father through the bars of a balcony like the one where my neighbor lived”.
Garcia continues:
“It’s very easy to see when you’ve got the memory right, because there is a very visceral reaction,” says Pau Garcia, founder of Domestic Data Streamers. “It happens every time. It’s like, ‘Oh! Yes! It was like that!’”
Dozens of people have now had their memories turned into images in this way via Synthetic Memories, a project run by Domestic Data Streamers. The studio uses generative image models, such as OpenAI’s DALL-E, to bring people’s memories to life. Since 2022, the studio, which has received funding from the UN and Google, has been working with immigrant and refugee communities around the world to create images of scenes that have never been photographed, or to re-create photos that were lost when families left their previous homes. Now Domestic Data Streamers is taking over a building next to the Barcelona Design Museum to record people’s memories of the city using synthetic images.
Synthetic Memories takes memories and uses AI to make them memories real. Garcia founded the studio and got the idea to start Synthetic Memories after speaking with Syrian refugees. An elderly Syrian wanted to capture her memories for her descendants and Garcia graffitied a building to record them. His idea has taken off. He notes:
Dozens of people have now had their memories turned into images in this way via Synthetic Memories.
Anyone is allowed to add a memory to the archive.
In order to recreate a memory, Garcia and his team developed a simple process. An interviewer sits with a subject, who then asks them to recount a memory. A prompt engineer writes a prompt for a model on a laptop to generate an image. Garcia’s team have written an entire glossary of prompting terms. The terms need to be edited to be accurate, so the engineers work with the subjects.
Garcia and his team learned that older subjects connect better with physical copies of their images. Images that are also blurry and warped resonate more with people too, because memories aren’t remembered in crisp detail.
To clarify, the images generated from this project blend techniques from photography, drawing, and watercolors. Faces remain obscured, ensuring the images are never mistaken for actual photographs. This is crucial because vague images have proven more effective in memory recall exercises. A clear image might highlight inconsistencies, while an unfinished image allows our imagination to complete it, often in line with our memories.
As Garcia stresses, it is important to remember the difference between synthetic images and real photography. He says synthetic memories aren’t meant to be factual. He worries (as do many AI analysts) that if a larger company than his use better versions of DALL-E and Midjourney and their progeny they’ll forgo the older “vague” models and just go for photorealism.
Also, the project’s potential reaches beyond nostalgia. It has vast scientific and therapeutic implications. The company is collaborating with social workers, psychologists, and medical experts in dementia and psychoneurology to understand the positive impact that synthetic memories can have on the progression of degenerative diseases such as Alzheimer’s or senile dementia.
Preliminary findings indicate this methodology could significantly enhance Reminiscence Therapy, which uses various sensory stimuli like music, objects, and, yes, images to enhance the cognitive functions of people living with degenerative diseases. A significant majority of patients, 11 out of 16, expressed a desire to increase sessions on synthetic memory recreation after their initial experience.
Yet, those in advanced dementia stages showed limited involvement, emphasizing the need to tailor the therapy to individual cognitive capacities, potentially also hinting at a new way to gauge cognitive decline.
Many patients expressed interest in having their generated memories printed, offering them tangible connections to the past. Researchers also recognized that some memories take time, spanning more than one session, stressing the importance of therapy flexibility. Interestingly, group sessions with synthetic memories, tested over 20 times, cultivated rich interactions among participants.
This enabled them to share cherished memories, boosting their interpersonal bonds, showcasing the benefits of group reminiscence in fostering communication among other dementia patients.
And memory boosting for both dementia patients and others is key. Taking the memory of Maria (the photo image above) as an example, the research team noted:
She vividly recalled her first memory of her father, seeing him from the perch of a rented balcony that faced the “La Modelo” prison where he was imprisoned. “That was the only way I could see my father for four years, through the bars of that balcony and the bars of his prison cell window. I was six at that time.” Maria’s description of the place and the historical context of that moment was used to generate an image that not only resonated with her but also moved us deeply. Upon seeing the image, Maria reported that looking at the image was akin to peering into a part of her past that she no longer had the words to articulate.
And while the Synthetic Memories project offers a promise for personal reminiscence, its potential is far-reaching. I did a little research this morning beyond the MIT article and one research team noted you could document the experiences of marginalized communities and preserve cultural legacies. Additionally, the data collected offers invaluable insights into studies on memory, cognition, and the sociology of aging.
However, the intersection of memory, technology, and art also raises ethical concerns. Ensuring the privacy of participants, maintaining the authenticity of memories, and determining the psychological implications of these synthetic images are just a few issues that need careful consideration. So it is interesting to see the next steps the company is taking that includes forming partnerships with healthcare institutions and mental health organizations to broaden the project’s scope and ensure ethical practices.
To that end, the team is focusing on transparency and reproducibility of the research by creating an accessible knowledge base so experiments can be replicated and verified. That’s great science.
And so we venture further and further into this new frontier between fiction and reality. This new technology stands to redefine not just technological capabilities but also the possibility to create new emotional landscapes to explore.
Fun and terror … at the same time.