The VES Handbook of Virtual Production

The VES Handbook of Virtual Production is a comprehensive guide to everything about virtual production available today – from pre-production to digital character creation, building a stage, choosing LED panels, setting up Brain Bars, in-camera compositing of live action and CG elements, Virtual Art Departments, Virtual Previs and scouting, best practices and much more. Current and forward-looking, this book covers everything one may need to know to execute a successful virtual production project – including when it is best to use virtual production and when it is not. More than 80 industry leaders in all fields of virtual production share their knowledge, experiences, techniques, and best practices. The text also features, charts, technical drawings, color images, and an extensive glossary of virtual production terms. The VES Handbook of Virtual Production is a vital resource for anyone wishing to gain essential knowledge in all aspects of virtual production. This is a must-have book for both aspiring and veteran professionals. It has been carefully compiled by the editors of The VES Handbook of Visual Effects.

Chapter 1. What is Virtual Production a) Definition of Virtual Production / Types of Virtual Production b) What Types of Scenes Work Well for Virtual Production c) Greenscreen for Virtual Production Chapter 2. Advantages and Disadvantages of Shooting in Virtual Production a) Advantages and Disadvantage of Using LED Volumes for Production Chapter 3. How does the Virtual Production Supervisor and Producer Relate to Other Departments a) Virtual Production Supervisor Relationships b) How the VFX Team on a Virtual Production Show Interfaces with Other Departments c) Production Design for Virtual Production - What Has Changed, What Has Not? Chapter 4. How To Shoot and Edit Animation Using Live-Action Virtual Production a) Introduction: The Creative Story Process b) Performance and Motion Capture for Animation c) Virtual Camera and Director’s Layout d) Creative Editorial e) Summary Chapter 5. Visualization a) What is Storyvis b) Previs/Mocap Pipeline and Flow chart c) VR Scouting d) VCAM - Virtual Camera System Used During Prep e) Techvis and Motion Control Techvis f) Stuntvis g) VP Tools for On-Set Visualization h) Simulcam j) Postvis Chapter 6. What Roles Are Needed for Virtual Production? a) Overview of Virtual Production Staffing and Organization b) Producing Virtual Production Chapter 7. VAD (Virtual Art Department) a) Definition of VAD / Workflow Integration b) Artists for Asset Creation/Sets c) Building Assets in Game Engines/Unreal d) Managing Asset Quality and Performance Needs for Virtual Production e) Virtual Scouting with Production Design - VR System f) Virtual Pre-lighting with the DP g) Multi-user Virtual Location Scouting – Pre-production h) Techvis for Art Departments j) Optimization and Delivery k) Digital Content Creation Software Used in VAD l) Content Library - Digital Backlot of Assets m) Version Control Software Chapter 8. Digital Asset Creation a) Building 3D Assets in a VFX Facility b) Capturing Surfaces with Photogrammetry to Create Assets c) Procedural Digital Asset Creation d) Move AI - Markerless Motion Capture e) Using Miniatures for Virtual Production f) Preparing 2D Plates for Virtual Production Chapter 9. Volumetric Capture for Humans a) Overview of Volumetric Capture b) Volumetric Capture / Considerations for Production - Wardrobe, Hair, Makeup, Blocking c) Volumetric Capture with Actors for Virtual Production Chapter 10. How to Make Characters for Use in Virtual Production a) Smart Simulated Characters (Digital Extras) b) MetaHumans: Creating Digital Humans for Production c) Animating MetaHumans with Motion Capture Chapter 11. How to Capture Environments for LED Walls a) Photogrammetry and LiDAR Sets for VP b) Photogrammetry Optimization c) Plate Photography for Playback on LED Walls Chapter 12. Process Work on LED Stage a) General Understanding of Process Work b) Break it Down Into Techniques Chapter 13. Projects Shot on an LED Stage a) LED Wall ICVFX b) LED Volume - 3D Tracking Full Volume versus 2D Playback on Set Chapter 14. Challenges and Limitations of Shooting in a Volume a) Limitations To Be Aware of When Shooting In a Volume b) Moiré, Viewing Angle, Banding, Latency, Artifacts, and Frame Sync c) Color Gamut and Color Shift d) Sound Dampening e) Post Cleanup of the LED Wall Chapter 15. Setting Up an LED Volume Stage - Permanent versus Pop Up a) Setting up a Full Volume for LED Walls b) Stage Designs c) Mobile LED Stages and Walls d) LED Display Wall - LED Modules, Processors, and Rigging e) Display Considerations Chapter 16. LED Display Technology and Hardware a) Characteristics of LED Displays b) Color Consistency and Predictability c) LED Display Validation Protocols d) What Are the Aspects of Choosing a Panel e) Types of Tools That Connect Cameras, Monitors and LED Displays to Computers f) Key Characteristics of LED Display Processors g) LED, OLED, and Playback Servers Chapter 17. Volume Control - Brain Bar Labor a) Description of Labor types for the Brain Bar b) Rendering and Control Computers Chapter 18. Software for Virtual Production a) Virtual Production Software b) Types of Content for On-Set Virtual Production c) On-Set Applications d) Media Servers for Filmmaking Chapter 19. Cameras for Virtual Production a) Characteristics of Cameras for Virtual Production b) LED Processing for Camera - Brompton Technology - Frame Remapping for ICVFX c) LED Processing for Camera - Megapixel VR’s GhostFrame d) Calibration and Lens Metadata Chapter 20. Camera Tracking for Virtual Production a) Overview of Inside-Out and Outside-In Tracking Systems b) Outside-In Tracking Systems c) Physical Tracker - Inertial / Optical / Hybrid / Encoder Based Solutions Chapter 21. Introduction to Color Management for LED Walls a) The Definition for Matching b) Rendering a Virtual Scene to Produce Linear Light c) Creating a Signal from Linear Light d) LED Processor Conversions e) Photography of LED Panels f) Preview of Camera Look in Real-Time Engine g) Limitations of LED Wall Panels Chapter 22. Art Dept Integration in the Digital and Physical World a) Getting the Environment Ready for the Cinematographer and Lighting in the Volume b) Blending the Physical and Digital World c) Workflow of Shooting on LED Wall d) Real-Time Compositing Between Physical and Digital World Chapter 23. Lighting Types for Virtual Production (LED Volume) a) DPs on Lighting for a LED Volume b) Lighting for the Outer Frustum c) Using the LED Wall as a Direct Reflection Lighting Tool d) External Lighting on Stage versus LED Wall Lighting e) On-Set Image Based Lighting (OS IBL) / Lights with Special Integration for Virtual Production f) Lighting Control Systems, Pixel Mapping and Protocols with Virtual Production Chapter 24. Epic Training - (Unreal Engine) a) Training In-Camera Visual Effects (ICVFX) for Virtual Production Chapter 25. Unity Software - Overview/Aspects of Virtual Production a) Introduction b) World Building in Unity c) Camera Tracking in Unity d) Simulcam with GS Shooting - Live Set Composting e) Ziva Dynamics- Photographic (Realistic) People, Animals, and Creatures Chapter 26. The Future of Virtual Production a) Overview of the Future b) Beyond Flat Screens for On-Set Virtual Production Stages c) Alternative Display Technology: Projection Based Volume Using White Cyc d) High Resolution Depth (HD3D) e) MicroLED - High-End Display and Resolution f) Full Gamut Color- RGBW g) NVIDIA Omniverse and Artificial Intelligence h) What is Next? - Beyond the Volume Acknowledgments and Thank You Appendix A: Charts and Technical Diagrams Appendix B: : The Virtual Production Glossary

Susan Zwerman, VES, is an experienced Visual Effects Producer with a passion for cutting-edge film production. She is highly respected for her expertise in VFX and Virtual Production Budgeting and Scheduling. As chair of the DGA UPM/AD VFX Digital Technology Committee, Susan organizes virtual production seminars to introduce members to this exciting and evolving new technology. Susan received the Frank Capra Achievement Award in recognition of career achievement and service to the industry and the Directors Guild of America in 2013. She is a member of the Academy of Motion Picture Arts and Sciences, the Producers Guild of America, the Directors Guild of America, and a member and Fellow of the VES. Jeffrey A. Okun, VES, is an award-winning Visual Effects Supervisor who is more than conversant with virtual production. He is a member and Fellow of the VES, and a member of The Academy of Motion Pictures, Arts and Sciences, the American Society of Cinematographers (ASC), the Television Academy, and the Editor’s Guild. Okun created visual effects tracking and bidding software in 1992 that is still in wide use within the industry today, as well as the revolutionary visual effects techniques dubbed the "PeriWinkle Effect" and the "Pencil Effect" – a predictive budgeting tool.