About

This page is a step by step instruction on how to use the Fisheye Render Setup plugin for Unreal Engine to render frames that can be stitched into a fisheye format using our custom stitcher. The plugin allows you take a Camera Sequence in Unreal and render out modified Camera Angles of it, creating Left, Right, Top and Bottom frames in stereo that can then be stitched into Fisheye frames.

Using Unreal Engine with its real time capabilities to render can speed up production time of movies immensely, as dome movie render times can be reduced from weeks and months to hours and days.

Download the Unreal Engine plugin

• Unreal Engine 5.0 | 2023-10-19
• Unreal Engine 5.1 | 2023-10-19
• Unreal Engine 5.2 | 2023-10-19
• Unreal Engine 5.3 | 2023-10-19
• Unreal Engine 5.3.2 | 2024-02-13 | Major changes to camera setup, deprecates old behavior and makes parts of the guide obsolete

Making a New Project

1. Create a new project. You can use any of the premade templates that suit your project the best, but for this tutorial we will use a blank project.
2. Make sure you have the following settings in your project. It should be the default settings, with the addition of turning on Raytracing

Project Settings

1. If your computer supports it, turn on Use Hardware Ray Tracing when available
2. Set Ray Lighting Mode to Hit Lighting for reflections

Installing the Unreal Engine Plugin

1. Download the plugin matching your Unreal Engine version
2. Add the FisheyeRenderSetup folder to your projects Plugins folder. If you don’t have a Plugins folder you can add one.
3. Start your project and check the Plugins browser to see that the plugin is installed
4. Open the Settings menu in your Content Browser and enable Show Plugin Content
5. You can now find the FisheyeRenderSetup content folder under All > Plugins
6. Right click LevelSequenceAssetAction and select Run Editor Utility Blueprint to ensure that it can be run further down the line

Level Setup

The following steps assumes that you have a blank scene with nothing in it. You can copy the DomeFisheyeTemplateLevel and use it as a starting point if you want to skip the following setup steps in Light Setup, Post Process Volume and Camera Rig. Alternatively, you could copy and paste the relevant actors into another level.

Lighting

1. Go to the Window tab and Open Environment Light Mixer
2. Press every button EXCEPT Atmospheric Light 1
3. In the outliner, select the Skylight and turn on the Real Time Capture checkbox

Post Process Volume

1. Add a Post Process Volume to your scene
2. Check Infinite Extent (Unbound) so that the volume affects the whole scene

3. Change the reflection method to Standalone Ray Traced. This sets the reflection method to Hardware instead of software. Lumen is limited in how good reflections it can produce, especially on highly reflective materials. Lumen cheats by using Screen space reflections, but this doesn't work for fisheye renders as objects visible in one camera won't reflect in the others. With Hardware Ray Tracing we get around this problem, with the trade-off being higher rendering time. You can omit this step if you don't have any issues with reflections.

   1. Tweak Max Roughness to set how reflective a material has to be to get a reflection. Lower values include more objects.
   2. Increase Max Bounces to get a more correct reflection. This is costly, so don't go higher than you need.
   3. Increase Samples Per Pixels if your reflective surfaces seem grainy. Higher values will be more costly, as it will cast a ray through each pixel for each level.

4. You need to turn off some screen space effects as they don't work with fisheye stitching. If you have them turned on you will see borders between the stitched images.

   1. Set Bloom > Intensity to 0.0
   
   2. Set Image Effects > Vignette Intensity to 0.0
   

5. Unreal Engine has Auto Exposure turned on by default (you can also edit this in project settings). We want to turn this off as the exposure should not change between camera angles.

   1. Set Exposure > Metering Mode to Manual
   2. _Exposure > Turn Apply Physical Camera Exposure Off
   3. Tweak Exposure > Exposure Compensation to match the lighting you want in your scene

Camera Rig (Deprecated as of 5.2.3)

The following steps are deprecated as of 5.3.2. Instead, you can add and use a CameraActor as you usually would.

We will now create a camera rig that matches the dome we want to render to. The goal is to create a Camera Transform Actor that we control in the animation and attaching a static camera to it that has a different rotation for each camera angle.

1. Add an actor into the level. Name this actor CameraTransform
2. Add another actor and name it DomeTilt. Attach DomeTilt to CameraTransform and set its relative rotation to (0, 63, 0). Make sure the other transforms are at default values
3. Add a CameraActor to the level and attach it to DomeTilt. If you want you can set the relative rotation to (0, -63, 0) to make the camera face forward for preview purposes. Make sure the other transforms are at default values. Set the cameras Aspect Ratio to 1.0 and Post Process Blend Weight to 0.0

Dome Preview (Partly deprecated as of 5.3.2)

The DomePreviewCamera should now be attached directly to the CameraActor.

If you want you can also add a Dome Preview, to check how the fisheye view would look in a dome

1. Go to All > Plugins > FisheyeRenderSetup Content > DomePreview
2. Add a DomePreviewSphere. You can set the dome to be either 180° or 165°, as well as set the dome tilt angle. At VisC the dome is 165° but the actual fisheye video should be 180°, as it is skewed to fit at a later stage. So for that purpose the preview should be set to 180°.
3. Add a DomePreviewCamera. Attach it to DomeTilt and make sure the transforms are at default values.
4. Select the DomePreviewCamera and press the Update Camera button in the Details menu. The camera view will now be drawn to the DomePreviewSphere
5. You can also check Capture Every Frame to have the camera always update but beware of the performance hit

Creating Sequences (Partly deprecated as of 5.2.3)

As of 5.3.2, step 3 can be ommitted and instead you animate the camera actor directly as you usually would. You are free to keyframe objects as you wish, the plugin is now smart about what objects it modifies. The scaling in step 4 can be done directly on the CameraActor.

To render out a scene, you need to create sequences that you can render out using the Movie Render Queue. The goal is to create a base sequence, and then create duplicates with the adjusted camera angles needed to render out the images we use for stitching together fisheye images.

1. Create a new Level Sequence and open it.
2. Select the CameraActor from your rig and drag it into Sequence. You should see it be added as a new track, along with a Camera Cuts track. You can also manually create a Camera Cuts track and add your camera to it.
3. Drag in the CameraTransform actor and add a Transform track to it. Add keyframes to CameraTransform's transform to animate your camera. You need to add at least one keyframe to keep it in place, even if you plan to have a static camera. (IMPORTANT Do not add keyframes to the CameraActor. It should remain static as we change it for the camera angles). You could alternatively animate the camera and other objects in a separate sequence and add it as a Subsequence.

4. If the camera gets too close to an object, the stereo effect will break as the brain can't parse what's going on. Bascially, if the dome surface is 7 meters away, you get a good stereo effect of an object that is 5 meters away. But if the object is 50 cm away, the stereo will be too intense and it will make you nauseous. If this is the case, you might want to adjust the distance between your eyes instead, effectively scaling the user. If you scale down the viewer to 1/10th, 50 cm becomes 5 meters in the eyes of the viewer. To achive this in Sequencer, you can keyframe the scale of any parent above the CameraActor in hierarchy. This means that the relative location of CameraActor in relation to its parent will be scaled, so the normal "eye distance" of 3.5 cm will be 0.35 at 1/10th scale etc.

Creating Camera Angle Duplicates

Once you have a base Level Sequence, you want to create duplicates for each camera angle, as well as eye of you are rendering for stereo. To do this we use scripted functionality from the plugin.

1. Right click your Level Sequence and select Scripted Asset Action > Make Duplicates with Fisheye Camera Angles

2. Now you should see a new folder appearing, containing the new duplicates. The prefix indicates which eye position and camera angle each Sequence contains. For example, LB_Sequence is Left eye, Bottom camera angle.

3. (Deprecated as of 5.2.3) Before you render you should check that the new camera angles are correct. If something seems off, make sure to check in the base sequence that you have assigned a keyed value to CameraTransform, and that the CameraActor has no keys.
4. If you want to adjust your sequence, make the adjustments in the base Sequence and then redo the duplication step. It should override the old duplicates.

Render with Movie Render Queue

It is now time to render your sequences. This is done with the Movie Render Queue

1. Open Window > Cinematics > Movie Render Queue
2. Drag one of the sequences from the Content Browser into the Movie Render Queue
3. Open the settings and press Load/Save Pre-set. Select the RenderSettings pre-set which is part of the plugin. Press Accept to apply the new settings
4. You can now drag in the rest of the sequences, and RenderSettings should apply to all of them

5. If you want to change something in RenderSettings, you can either change it locally for one sequence in the Movie Render Queue window or change it globally by saving the RenderSettings file in the top right corner, or by editing it directly in the plugin folder and saving it there. If changes have only been made locally, it is marked with a *, e.g. RenderSettings*.

   1. In the Output tab you can control various settings, such as where the frames will be saved and under what name, the resolution of your frames and if you want to render a custom set of frames.

2. If your desired render resolution is higher than what your computer can manage, you can use the settings in the High Resolution tab to set a tile count which will split each frame into smaller tiles. For example, if you set the tile count to 4, it will split a 4096x4096 image into 4x4 1028x1028 images, which can be rendered on a far less powerful computer. However, if you can manage high resolutions right away it is most likely quicker to render.

3. Should you choose to use High Resolution, you will also need to adjust settings in Anti-Aliasing. Set the anti-aliasing method to none, as it is not supported for tiled rendering. Instead, you use Temporal Samples to remove any aliasing issues. This creates a number of subframes based on your TS count, captured with a small time difference, which can be averaged to produce a more clean end frame. Increase the count as you see fit to remove any artifacts that may occur, at the cost of higher render times.

Be aware that the use of Anti-Alising can cause GPU crashes when used together with Hardware Ray Tracing. This behavior has been observed in Unreal Engine 5.1/5.2. The crash is caused by having a Render Warm Up Count bigger then 0. When using this setting, the engine tries to submit n frames to the VRAM, without ticking the engine between each frame. This will cause a crash as the VRAM memory simply runs out. The correct settings is instead:

• Render Warm Up Frames
• Render Warm Up Count = 0
• Render Warm Up Frames = 32 (This seems like a good number but you can change it as needed)

Niagara Systems

As the rendering basically restarts for each camera angle, the scene will simulate again for each render. This means that things like Niagara Particle systems can't have a random factor, as it needs to produce the same result for each render.

1. In your Niagara Emitter/System, make sure that Emitter > Determinism is checked to ensure the same result each render

2. We can control the Niagara System by adding it to our Sequence. Add a Niagara Component track and add a System Life Cycle track to it.

3. Right click the Life Cycle track and set Properties > Age Update Mode to Desired Age. This makes the sequencer control the life span of the particle system, allowing you to scrub though the simulation with a determined state for each frame.

Making Procedural Materials Deterministic

Some Materials changes with time and other variables that might vary between runs. As this might cause issues with rendering multiple times, we need a deterministic solution. Therefore, you might want to use a Global Material Collection to control a material with the sequencer.

1. Create a Materials > Global Parameter Collection and open it up
2. Add any values that you might want to change globally. They can be both scalar and vector parameters
3. Drag your GPC into a material and select your parameter of choice from the dropdown in the Details menu
4. Plug the value in as needed, e.g. as the Time input for a Panner node

5. Drag your GPC into a sequencer and add the parameter you want to control as a track.
6. Set keyframes for the parameter and watch as your material changes when playing the sequence.

Raytraced Refraction

Transparent materials with refraction, such as glass, can cause issues when rendering fisheye frames as the default refraction seems to be view dependent. This is solved by changing the translucency type in the Post Process Volume from Raster to Ray Tracing. You then have to set up the transparent material to utilize ray tracing:

• Blend Mode - Translucent
• Two Sided
• Lighting Mode - Forward Shading
• Refraction Mode - Index Of Refraction

Common issues and bugs

This section is a collection of known issues that may occur based on user error, which can be hard to debug.

Sequence actor placed in level causes fighting over camera control

If your camera render is not behaving as expected, it might be because the sequence that you are trying to render is also placed in the level as an actor. This is intended use of sequences in some instances, as it allows you to keyframe movement of objects and play them on demand in a game. However, if the sequence placed in the level is also the one you are trying to render this will cause issues, since both sequences will be fighting for the control of the camera. Of course, the same issue will occur with two different sequences that are trying to control the same camera actor. To debug this issue, search the level outliner for any LevelSequence that might cause issues.

Camera angles look wierd on duplicates (Deprecated as of 5.3.2)

If your camera animation looks good on the base sequence, but is not behaving as expected in the duplicate sequences, this might be an issue with your keyframes. As mentioned previously, it is important that you DONT keyframe the Camera as it will block the plugin from rotating the camera, and to ALWAYS keyframe the rest of your objects or remove the transform track entirely on that object.