Mapping the landscape of VFX and AI: A PRISMA-guided systematic review
List of Authors
  • Afeez Nawfal Mohd Isa , Cheah Ying Yi

Keyword
  • VFX, Visual Effects, CGI, AI, Artificial Intelligence

Abstract
  • Objective: This systematic literature review aims to comprehensively synthesize existing evidence on Visual Effects (VFX) and Artificial Intelligence (AI) using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The primary objective is to map the landscape of VFX and AI, examining the historical development of VFX techniques and practices in the context of AI advancements. Methods: Following PRISMA guidelines, a thorough search of five electronic databases—IEEE Xplore, Scopus, Taylor & Francis Online, ACM Digital Library, and ScienceDirect—was conducted to identify articles published up to April 18, 2024. Inclusion criteria encompassed English-language journal articles. Independent reviewers screened articles for eligibility, resolving discrepancies through consensus. Data extraction and synthesis were performed systematically. Results: A total of 41 studies met the inclusion criteria and were included in the review. These studies covered a wide range of topics related to VFX and AI, with various study designs. The findings are presented in a narrative synthesis. Discussion: The review indicates that AI development in VFX is comprehensive, covering topics such as image processing, inpainting, video processing, 3D modelling, rendering, deepfake technology, style transfer, tracking, animation, simulation, rigging, texture compression, and depth map enhancement. These advancements highlight the transformative role of AI in enhancing creative expression, production efficiency, and the quality of visual effects. Conclusion: This systematic literature review provides a comprehensive overview of the current state of knowledge on AI and VFX. The integration of AI into VFX is profoundly transforming the industry, enhancing creative expression, production efficiency, and the quality of visual effects. Continued advancements in AI technologies promise to further revolutionize VFX practices, offering new possibilities and efficiencies for future productions.

Reference
  • 1. Bau, D., Strobelt, H., Peebles, W., Wulff, J., Zhou, B., Zhu, J., & Torralba, A. (2019). Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics, 38(4), 1–11. https://doi.org/10.1145/3306346.3323023

    2. Bernard, H. R. (2006). Research methods in anthropology: Qualitative and Quantitative Approaches. Altamira Press.

    3. Cao, L., Liu, J., Du, K., Guo, Y., & Wang, T. (2020). Guided Cascaded Super-Resolution Network for Face Image. IEEE Access, 8, 173387–173400. https://doi.org/10.1109/ access.2020.3025972

    4. Chen, G., Ding, S., & Liu, W. (2023). Application of computer image processing technology in visual communication system. Applied Artificial Intelligence, 37(1). https://doi.org/10.1080/08839514.2023.2204264

    5. Chen, Q., Wang, Y., Wang, H., & Yang, X. (2021). Data-driven simulation in fluids animation: A survey. Virtual Reality & Intelligent Hardware/Xuni Xianshi Yu Zhineng Yingjian, 3(2), 87–104. https://doi.org/10.1016/j.vrih.2021.02.002

    6. Dinur, E. (2017). The Filmmaker’s Guide to Visual Effects: The Art and Techniques of VFX for directors, producers, editors and cinematographers.

    7. Egger, M., Smith, G. D., & O'Rourke, K. (2001). Introduction: rationale, potentials, and `promise of systematic reviews. Systematic reviews in health care: meta‐analysis in context, 1-19

    8. Fan, C., Fu, W., & Liu, S. (2022). A high-precision correction method in non-rigid 3D motion poses reconstruction. Connection Science, 34(1), 2845–2859. https://doi.org/10.1080/0 9540091.2022.2151569

    9. Ghasemieh, A., & Kashef, R. (2024). Towards explainable artificial intelligence in deep vision based odometry. Computers & Electrical Engineering, 115, 109127. https://doi.org/10.1016/j.compeleceng.2024.109127

    10. Guo, K., Lincoln, P., Davidson, P., Busch, J., Yu, X., Whalen, M., Harvey, G., Orts-Escolano, S., Pandey, R., Dourgarian, J., Tang, D., Tkach, A., Kowdle, A., Cooper, E. A., Dou, M., Fanello, S., Fyffe, G., Rhemann, C., Taylor, J., ... Izadi, S. (2019). The relightables. ACM Transactions on Graphics, 38(6), 1–19. https://doi.org/10.1145/3355089. 3356571

    11. Hameleers, M., Van Der Meer, T. G., & Dobber, T. (2024). Distorting the truth versus blatant lies: The effects of different degrees of deception in domestic and foreign political deepfakes.Computers in Human Behavior, 152, 108096. https://doi.org/10.1016/j.chb. 2023.108096

    12. Holmes, S., & Going, L. P. (2023). Visual effects for indie filmmakers: A Guide to VFX Integration and Artist Collaboration. Routledge.

    13. Hou, Z., Kim, K., Zhang, G., & Li, P. (2023). A study on the realization of virtual simulation FACE based on artificial intelligence. Journal of Information and Communication Convergence Engineering, 21(2), 152–158. https://doi.org/10.56977/jicce.2023. 21.2.152

    14. Hu, W., Song, H., Zhang, F., Zhao, Y., & Shi, X. (2023). Style transfer of Thangka images highlighting style attributes. IEEE Access, 11, 104817–104829. https://doi.org/10.1109/ access.2023.3318258

    15. Hua, Z., Fan, G., & Li, J. (2020). Iterative residual network for image dehazing. IEEE Access, 8, 167693–167710. https://doi.org/10.1109/access.2020.3023906

    16. Kang, D., Kim, Y., Kwon, S., Kim, H., Kim, J., & Paik, J. (2023a). Video quality assessment system using deep optical flow and fourier property. IEEE Access, 11, 132131–132146. https://doi.org/10.1109/access.2023.3335352

    17. Kang, D., Kim, Y., Kwon, S., Kim, H., Kim, J., & Paik, J. (2023b). Video quality assessment system using deep optical flow and fourier property. IEEE Access, 11, 132131–132146. https://doi.org/10.1109/access.2023.3335352

    18. Kim, B., & Seo, S. (2022). Intelligent Digital Human Agent Service with Deep Learning Based-Face recognition. IEEE Access, 10, 72794–72805. https://doi.org/10.1109/access.2022. 3188852

    19. Kufel, J., Bargieł-Łączek, K., Kocot, S., Koźlik, M., Bartnikowska, W., Janik, M., Czogalik, Ł., Dudek, P., Magiera, M., Lis, A., Paszkiewicz, I., Nawrat, Z., Cebula, M., & Gruszczyńska, K. (2023). What is machine Learning, artificial neural networks and Deep Learning?— Examples of Practical applications in medicine. Diagnostics, 13(15), 2582. https://doi.org/10.3390/diagnostics13152582

    20. Li, S. (2023). Application of artificial intelligence-based style transfer algorithm in animation special effects design. Open Computer Science, 13(1). https://doi.org/10.1515/comp-2022-0255

    21. Li, S., Zhang, B., Zhu, W., & Yang, X. (2020). FMPN: Fusing multiple Progressive CNNs for Depth Map Super-Resolution. IEEE Access, 8, 170754–170768. https://doi.org/10.1109/access.2020.3024650

    22. Li, Z., & Jahng, S. G. (2023). Computer-aided digital image inpainting algorithm and image special effects processing based on deep learning. Computer-Aided Design and Applications, 88–100. https://doi.org/10.14733/cadaps.2024.s1.88-100

    23. Liu, Q., Jia, R., Zhao, C., Li, X., Sun, H., & Zhang, X. (2020). Face Super-Resolution reconstruction based on Self-Attention Residual Network. IEEE Access, 8, 4110–4121. https://doi.org/10.1109/access.2019.2962790

    24. Lu, E., Cole, F., Dekel, T., Xie, W., Zisserman, A., Salesin, D., Freeman, W. T., & Rubinstein, M. (2020). Layered neural rendering for retiming people in video. ACM Transactions on Graphics, 39(6), 1–14. https://doi.org/10.1145/3414685.3417760

    25. Luo, C., Li, B., & Li, F. (2023). Iterative back projection network based on deformable 3D convolution. IEEE Access, 11, 122586–122597. https://doi.org/10.1109/access. 2023.3325659

    26. Luo, X., Huang, J., Szeliski, R., Matzen, K., & Kopf, J. (2020). Consistent video depth estimation. ACM Transactions on Graphics, 39(4). https://doi.org/10.1145/3386569.3392377

    27. Mahapatra, A., Siarohin, A., Lee, H., Tulyakov, S., & Zhu, J. (2023). Text-Guided synthesis of Eulerian cinemagraphs. ACM Transactions on Graphics, 42(6), 1–13. https://doi.org/10.1145/3618326

    28. Malik, A., Kuribayashi, M., Abdullahi, S. M., & Khan, A. N. (2022). DeepFake detection for human face images and Videos: a survey. IEEE Access, 10, 18757–18775. https://doi.org/10.1109/access.2022.3151186

    29. Martel, J., Lindell, D. B., Lin, C. Z., Chan, E. R., Monteiro, M. a. A., & Wetzstein, G. (2021). Acorn. ACM Transactions on Graphics, 40(4), 1–13. https://doi.org/10.1145/3450626. 3459785

    30. Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Elgharib, M., Fua, P., Seidel, H., Rhodin, H., Pons‐Moll, G., & Theobalt, C. (2020). XNect. ACM Transactions on Graphics, 39(4). https://doi.org/10.1145/3386569.3392410

    31. Mo, J., & Zhou, Y. (2020). The image inpainting algorithm used on multi-scale generative adversarial networks and neighbourhood. Automatika, 61(4), 704–713. https://doi.org/10.1080/00051144.2020.1821535

    32. Okun, J. A., Zwerman, S., & Society, V. E. (2010). The VES Handbook of Visual Effects: Industry Standard VFX Practices and Procedures. Taylor & Francis.

    33. Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., ... McKenzie, J. E. (2021). PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, n160. https://doi.org/10.1136/bmj.n160

    34. Pandey, R., Escolano, S. O., LeGendre, C., Häne, C., Bouaziz, S., Rhemann, C., Debevec, P., & Fanello, S. (2021). Total relighting. ACM Transactions on Graphics, 40(4), 1–21. https://doi.org/10.1145/3450626.3459872

    35. Pardeshi, A. S., & Karbhari, V. B. (2019). Recent trends in VFX (Virtual Effects) and SFX (Special Effects). Zenodo (CERN European Organization for Nuclear Research). https://doi.org/10.5281/zenodo.7735881

    36. Qiu, D., Yang, H., Deng, X., & Liu, Y. (2023). Superpixel segmentation based on image density. Systems Science & Control Engineering, 11(1). https://doi.org/10.1080/ 21642583.2023.2185915

    37. Qiu, J., Gao, Y., & Shen, M. (2021). Semantic-SCA: Semantic structure image inpainting with the Spatial-Channel attention. IEEE Access, 9, 12997–13008. https://doi.org/10.1109/access.2021.3051982

    38. Rickitt, R. (2007). Special effects: The History and Technique. Rouse, M. (2024, April 12). Artificial Intelligence (AI). Techopedia. https://www.techopedia.com/definition/190/ artificial-intelligence-ai

    39. Seide, B. (2021). Artificial intelligence in digital visual effects. NTU Singapore. https://hdl.handle.net/10356/151632

    40. Shafie, S. H. M. (2024). Vehicle emissions are polluting Malaysia’s cities. https://doi.org/10.54377/e217-1e97

    41. Special effects: the history and technique. (2001). Choice/Choice Reviews, 38(06), 38–3240. https://doi.org/10.5860/choice.38-3240

    42. Su, Y., Wu, M., & Yan, Y. (2023). Image enhancement and brightness equalization algorithms in low illumination environment based on multiple frame sequences. IEEE Access, 11, 61535–61545. https://doi.org/10.1109/access.2023.3286538

    43. Sun, C., Lai, H., Wang, L., & Jia, Z. (2021). Efficient attention fusion network in Wavelet domain for demoireing. IEEE Access, 9, 53392–53400. https://doi.org/10.1109/access.2021. 3070809

    44. Teotia, K., Mallikarjun, B. R., Pan, X., Kim, H., Garrido, P., Elgharib, M., & Theobalt, C. (2024). HQ3DAVatar: High Quality Implicit 3D Head Avatar. ACM Transactions on Graphics. https://doi.org/10.1145/3649889

    45. The VES Handbook of Visual Effects. (n.d.). Google Books. https://books.google.com.my/books?hl=zhCN&lr=&id=VsXrDwAAQBAJ&oi=fnd&pg=PP1&dq=%22visual+effects%22&ots=Y-hTRGTrSG&sig=yf7XS3rG8G-bM-Uf4k BRt2kxXA&redir_esc=y#v=onepage&q=%22visual%20effects%22&f=false

    46. Tian, F. (2024). Denoising Monte Carlo Rendering Models Combined with JDKPNet in Artistic Design Application Scenarios. Intelligent Systems With Applications, 22, 200338. https://doi.org/10.1016/j.iswa.2024.200338

    47. Vaidyanathan, K., Salvi, M., Wronski, B., Akenine-Möller, T., Ebelin, P., & Lefohn, A. (2023). Random-Access neural compression of material textures. https://doi.org/10.1145/3592407

    48. Veile, J. W., Kiel, D., Müller, J. M., & Voigt, K. (2019). Lessons learned from Industry 4.0 implementation in the German manufacturing industry. Journal of Manufacturing Technology Management, 31(5), 977–997. https://doi.org/10.1108/jmtm-08-2018-0270

    49. Wang, K., Duan, Y., & Yang, Y. (2020). Single image dehazing algorithm based on pyramid mutil-scale transposed convolutional network. Systems Science & Control Engineering, 9(sup1), 150–160. https://doi.org/10.1080/21642583.2020.1833780

    50. Xing, J., Luan, F., Liu, Y., Hu, X., Qian, H., & Xu, K. (2022). Differentiable rendering using RGBXY derivatives and optimal transport. ACM Transactions on Graphics, 41(6), 1–13. https://doi.org/10.1145/3550454.3555479

    51. Ye, W., Chen, H., Zhang, Z., Liu, Y., Weng, S., & Chang, C. (2019). Hybrid scheme of image’s regional colorization using mask R-CNN and poisson editing. IEEE Access, 7, 115901–115913. https://doi.org/10.1109/access.2019.2936258

    52. Zhang, B., Yan, W., Li, G., Fei, J., Zhang, C., & Chen, C. (2019). Image enhancement via indented frame over fusion. IEEE Access, 7, 181092–181099. https://doi.org/10.1109/access. 2019.2956747

    53. Zhang, H., Ye, Y., Shiratori, T., & Komura, T. (2021). ManipNet. ACM Transactions on Graphics, 40(4), 1–14. https://doi.org/10.1145/3450626.3459830

    54. Zhang, Q., Zou, C., Shao, M., & Hong, L. (2023). A Single-Stage unsupervised denoising Low-Illumination enhancement network based on Swin-Transformer. IEEE Access, 11, 75696–75706. https://doi.org/10.1109/access.2023.3297490

    55. Zhang, S., Wang, L., Zhang, J., Gu, L., Jiang, X., Zhai, X., Sha, X., & Chang, S. (2020). Consecutive context perceive generative adversarial networks for serial sections inpainting. IEEE Access, 8, 190417–190430. https://doi.org/10.1109/access. 2020.3031973

    56. Zheng, Z., Zhao, X., Zhang, H., Liu, B., & Liu, Y. (2023). AvatarREX: Real-time Expressive Full-body Avatars. ACM Transactions on Graphics, 42(4), 1–19. https://doi.org/10.1145/3592101

    57. Zhu, D., Zhan, W., Jiang, Y., Xu, X., & Guo, R. (2021). MIFFUse: a Multi-Level feature fusion network for infrared and visible images. IEEE Access, 9, 130778–130792. https://doi.org/10.1109/access.2021.3111905