In recent times the use of artificial intelligence, machine learning, GAN's and algorithms in artistic applications has been on the rise. There are many AI tools available today to assist creators at various points in the production process, whether it's streamlining normally tedious tasks with precision, or generating completely new stunning pieces of digital art.

Our question:  Can AI generate 360° images formatted for fulldome? (domemasters)

Our answer:  Yes!  (..kind of)

Tools/software: Kodak SP3604K, Runway ML, StyleGAN2 on Google Colab.  

Machine learning is an area of research within the field of artificial intelligence (AI). For this experiment, we used RunwayML, a platform for training your own models. The term model refers to the result of training a machine learning algorithm with a dataset. A machine learning algorithm find patterns within the data and develops its own rules for how to represent—or model—those patterns to perform a specific task. These rules are not designed by humans; they are learned from the data on which the model was trained.

People build models in different programming languages, with different frameworks, and for different purposes, such as to recognize spam email, classify objects, edit images and video, and capture motion. In our case, we want to train a model that generates images of architectural facades that respect the geometric attributes and perspective of the domemaster format.

Here are the results: AI urban environments, domemaster below. 

We rendered 20+ minutes of fully AI-generated, morphing 360° urban environments with some interesting results:

• the geometry works! 
• the perspective works!
• our AI domemaster has 230°+ fov since our dataset was shot with the SP3604K
• full flexibility of latent walk render, speed and frames using StyleGAN2 in Colab (tutorial below)
• limitation: training resolution at 512x512 or 1024x1024 means rendered images must be upscaled before being usable in the dome (topaz AI gigapixel worked well).  This can create unwanted artifacts in the image.

So how is this possible?

First step: We need an image dataset.  We created an image dataset of 3000 fisheye images shot in downtown Montreal which we used to train our AI model. RunwayML makes training very easy: all you need to do is supply your own dataset and determine the number of steps that the algorithm should take to learn the features. During each step, the model learns a little bit more about the features in your dataset.

The time it takes to train a model depends on a combination of many factors, including but not limited to the type of machine learning algorithm (there are algorithms of varying complexity for working with images, words, or sounds) and the number of training steps.

RunwayML greatly reduces the amount of time needed to train your own generative image model by using using a technique called Transfer Learning. With this technique, an existing pre-trained model called StyleGAN is retrained on your dataset. StyleGAN was trained on a massive amount of images (70,000!), and Transfer Learning leverages all of the image features it learned from the original data to speed up the process of learning features in your image dataset. Not only does this greatly reduce training time, but it also reduces the number of images that you need to train your own model. 

Training steps progress animation:

Second step: After a few hours our model has finished training, great!  During training, your model learns to produce new images that look like the ones in your dataset. When training is complete, the model will generate a seemingly infinite number of synthetic images that appear as if they came from the dataset itself. The space of all-new possible images that your model can create is called the latent space, which you can explore when you run your model. RunwayML allows you to run the model directly in its UI (at a cost), or to download the model as a .pkl.  Training already comes at a cost, and we want the flexibility of exploring the latent space ourselves, so we download the model and run it remotely on Google Colab with StyleGan2.  The following tutorial explains the exact workflow with links in the description.

Conclusion: Promising results for artists wishing to explore the possibilities of AI-generated 360° environments in the dome.  

You can check more of our AI domemaster experiments here:

Contact us at info@nestimmersion.ca for questions or info.  

Thanks!

Sean & Pat @ NEST IMMERSION