Mapping the landscape of VFX and AI: A PRISMA-guided systematic review
List of Authors
  • Afeez Nawfal Mohd Isa , Cheah Ying Yi

Keyword
  • VFX, Visual Effects, CGI, AI, Artificial Intelligence

Abstract
  • Objective: This systematic literature review aims to comprehensively synthesize existing evidence on Visual Effects (VFX) and Artificial Intelligence (AI) using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The primary objective is to map the landscape of VFX and AI, examining the historical development of VFX techniques and practices in the context of AI advancements. Methods: Following PRISMA guidelines, a thorough search of five electronic databases—IEEE Xplore, Scopus, Taylor & Francis Online, ACM Digital Library, and ScienceDirect—was conducted to identify articles published up to April 18, 2024. Inclusion criteria encompassed English-language journal articles. Independent reviewers screened articles for eligibility, resolving discrepancies through consensus. Data extraction and synthesis were performed systematically. Results: A total of 41 studies met the inclusion criteria and were included in the review. These studies covered a wide range of topics related to VFX and AI, with various study designs. The findings are presented in a narrative synthesis. Discussion: The review indicates that AI development in VFX is comprehensive, covering topics such as image processing, inpainting, video processing, 3D modelling, rendering, deepfake technology, style transfer, tracking, animation, simulation, rigging, texture compression, and depth map enhancement. These advancements highlight the transformative role of AI in enhancing creative expression, production efficiency, and the quality of visual effects. Conclusion: This systematic literature review provides a comprehensive overview of the current state of knowledge on AI and VFX. The integration of AI into VFX is profoundly transforming the industry, enhancing creative expression, production efficiency, and the quality of visual effects. Continued advancements in AI technologies promise to further revolutionize VFX practices, offering new possibilities and efficiencies for future productions.

Reference
  • 1. Bau, D., Strobelt, H., Peebles, W., Wulff, J., Zhou, B., Zhu, J., & Torralba, A. (2019). Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics, 38(4), 1–11. https://doi.org/10.1145/3306346.3323023
    2. Bernard, H. R. (2006). Research methods in anthropology: Qualitative and Quantitative Approaches. Altamira Press.
    3. Cao, L., Liu, J., Du, K., Guo, Y., & Wang, T. (2020). Guided Cascaded Super-Resolution Network for Face Image. IEEE Access, 8, 173387–173400. https://doi.org/10.1109/ access.2020.3025972
    4. Chen, G., Ding, S., & Liu, W. (2023). Application of computer image processing technology in visual communication system. Applied Artificial Intelligence, 37(1). https://doi.org/10.1080/08839514.2023.2204264
    5. Chen, Q., Wang, Y., Wang, H., & Yang, X. (2021). Data-driven simulation in fluids animation: A survey. Virtual Reality & Intelligent Hardware/Xuni Xianshi Yu Zhineng Yingjian, 3(2), 87–104. https://doi.org/10.1016/j.vrih.2021.02.002
    6. Dinur, E. (2017). The Filmmaker’s Guide to Visual Effects: The Art and Techniques of VFX for directors, producers, editors and cinematographers.
    7. Egger, M., Smith, G. D., & O'Rourke, K. (2001). Introduction: rationale, potentials, and `promise of systematic reviews. Systematic reviews in health care: meta‐analysis in context, 1-19
    8. Fan, C., Fu, W., & Liu, S. (2022). A high-precision correction method in non-rigid 3D motion poses reconstruction. Connection Science, 34(1), 2845–2859. https://doi.org/10.1080/0 9540091.2022.2151569
    9. Ghasemieh, A., & Kashef, R. (2024). Towards explainable artificial intelligence in deep vision based odometry. Computers & Electrical Engineering, 115, 109127. https://doi.org/10.1016/j.compeleceng.2024.109127
    10. Guo, K., Lincoln, P., Davidson, P., Busch, J., Yu, X., Whalen, M., Harvey, G., Orts-Escolano, S., Pandey, R., Dourgarian, J., Tang, D., Tkach, A., Kowdle, A., Cooper, E. A., Dou, M., Fanello, S., Fyffe, G., Rhemann, C., Taylor, J., ... Izadi, S. (2019). The relightables. ACM Transactions on Graphics, 38(6), 1–19. https://doi.org/10.1145/3355089. 3356571
    11. Hameleers, M., Van Der Meer, T. G., & Dobber, T. (2024). Distorting the truth versus blatant lies: The effects of different degrees of deception in domestic and foreign political deepfakes.Computers in Human Behavior, 152, 108096. https://doi.org/10.1016/j.chb. 2023.108096
    12. Holmes, S., & Going, L. P. (2023). Visual effects for indie filmmakers: A Guide to VFX Integration and Artist Collaboration. Routledge.
    13. Hou, Z., Kim, K., Zhang, G., & Li, P. (2023). A study on the realization of virtual simulation FACE based on artificial intelligence. Journal of Information and Communication Convergence Engineering, 21(2), 152–158. https://doi.org/10.56977/jicce.2023. 21.2.152
    14. Hu, W., Song, H., Zhang, F., Zhao, Y., & Shi, X. (2023). Style transfer of Thangka images highlighting style attributes. IEEE Access, 11, 104817–104829. https://doi.org/10.1109/ access.2023.3318258
    15. Hua, Z., Fan, G., & Li, J. (2020). Iterative residual network for image dehazing. IEEE Access, 8, 167693–167710. https://doi.org/10.1109/access.2020.3023906
    16. Kang, D., Kim, Y., Kwon, S., Kim, H., Kim, J., & Paik, J. (2023a). Video quality assessment system using deep optical flow and fourier property. IEEE Access, 11, 132131–132146. https://doi.org/10.1109/access.2023.3335352
    17. Kang, D., Kim, Y., Kwon, S., Kim, H., Kim, J., & Paik, J. (2023b). Video quality assessment system using deep optical flow and fourier property. IEEE Access, 11, 132131–132146. https://doi.org/10.1109/access.2023.3335352
    18. Kim, B., & Seo, S. (2022). Intelligent Digital Human Agent Service with Deep Learning Based-Face recognition. IEEE Access, 10, 72794–72805. https://doi.org/10.1109/access.2022. 3188852
    19. Kufel, J., Bargieł-Łączek, K., Kocot, S., Koźlik, M., Bartnikowska, W., Janik, M., Czogalik, Ł., Dudek, P., Magiera, M., Lis, A., Paszkiewicz, I., Nawrat, Z., Cebula, M., & Gruszczyńska, K. (2023). What is machine Learning, artificial neural networks and Deep Learning?— Examples of Practical applications in medicine. Diagnostics, 13(15), 2582. https://doi.org/10.3390/diagnostics13152582
    20. Li, S. (2023). Application of artificial intelligence-based style transfer algorithm in animation special effects design. Open Computer Science, 13(1). https://doi.org/10.1515/comp-2022-0255
    21. Li, S., Zhang, B., Zhu, W., & Yang, X. (2020). FMPN: Fusing multiple Progressive CNNs for Depth Map Super-Resolution. IEEE Access, 8, 170754–170768. https://doi.org/10.1109/access.2020.3024650
    22. Li, Z., & Jahng, S. G. (2023). Computer-aided digital image inpainting algorithm and image special effects processing based on deep learning. Computer-Aided Design and Applications, 88–100. https://doi.org/10.14733/cadaps.2024.s1.88-100
    23. Liu, Q., Jia, R., Zhao, C., Li, X., Sun, H., & Zhang, X. (2020). Face Super-Resolution reconstruction based on Self-Attention Residual Network. IEEE Access, 8, 4110–4121. https://doi.org/10.1109/access.2019.2962790
    24. Lu, E., Cole, F., Dekel, T., Xie, W., Zisserman, A., Salesin, D., Freeman, W. T., & Rubinstein, M. (2020). Layered neural rendering for retiming people in video. ACM Transactions on Graphics, 39(6), 1–14. https://doi.org/10.1145/3414685.3417760
    25. Luo, C., Li, B., & Li, F. (2023). Iterative back projection network based on deformable 3D convolution. IEEE Access, 11, 122586–122597. https://doi.org/10.1109/access. 2023.3325659
    26. Luo, X., Huang, J., Szeliski, R., Matzen, K., & Kopf, J. (2020). Consistent video depth estimation. ACM Transactions on Graphics, 39(4). https://doi.org/10.1145/3386569.3392377
    27. Mahapatra, A., Siarohin, A., Lee, H., Tulyakov, S., & Zhu, J. (2023). Text-Guided synthesis of Eulerian cinemagraphs. ACM Transactions on Graphics, 42(6), 1–13. https://doi.org/10.1145/3618326
    28. Malik, A., Kuribayashi, M., Abdullahi, S. M., & Khan, A. N. (2022). DeepFake detection for human face images and Videos: a survey. IEEE Access, 10, 18757–18775. https://doi.org/10.1109/access.2022.3151186
    29. Martel, J., Lindell, D. B., Lin, C. Z., Chan, E. R., Monteiro, M. a. A., & Wetzstein, G. (2021). Acorn. ACM Transactions on Graphics, 40(4), 1–13. https://doi.org/10.1145/3450626. 3459785
    30. Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Elgharib, M., Fua, P., Seidel, H., Rhodin, H., Pons‐Moll, G., & Theobalt, C. (2020). XNect. ACM Transactions on Graphics, 39(4). https://doi.org/10.1145/3386569.3392410
    31. Mo, J., & Zhou, Y. (2020). The image inpainting algorithm used on multi-scale generative adversarial networks and neighbourhood. Automatika, 61(4), 704–713. https://doi.org/10.1080/00051144.2020.1821535
    32. Okun, J. A., Zwerman, S., & Society, V. E. (2010). The VES Handbook of Visual Effects: Industry Standard VFX Practices and Procedures. Taylor & Francis.
    33. Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., ... McKenzie, J. E. (2021). PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, n160. https://doi.org/10.1136/bmj.n160
    34. Pandey, R., Escolano, S. O., LeGendre, C., Häne, C., Bouaziz, S., Rhemann, C., Debevec, P., & Fanello, S. (2021). Total relighting. ACM Transactions on Graphics, 40(4), 1–21. https://doi.org/10.1145/3450626.3459872
    35. Pardeshi, A. S., & Karbhari, V. B. (2019). Recent trends in VFX (Virtual Effects) and SFX (Special Effects). Zenodo (CERN European Organization for Nuclear Research). https://doi.org/10.5281/zenodo.7735881
    36. Qiu, D., Yang, H., Deng, X., & Liu, Y. (2023). Superpixel segmentation based on image density. Systems Science & Control Engineering, 11(1). https://doi.org/10.1080/ 21642583.2023.2185915
    37. Qiu, J., Gao, Y., & Shen, M. (2021). Semantic-SCA: Semantic structure image inpainting with the Spatial-Channel attention. IEEE Access, 9, 12997–13008. https://doi.org/10.1109/access.2021.3051982
    38. Rickitt, R. (2007). Special effects: The History and Technique. Rouse, M. (2024, April 12). Artificial Intelligence (AI). Techopedia. https://www.techopedia.com/definition/190/ artificial-intelligence-ai
    39. Seide, B. (2021). Artificial intelligence in digital visual effects. NTU Singapore. https://hdl.handle.net/10356/151632
    40. Shafie, S. H. M. (2024). Vehicle emissions are polluting Malaysia’s cities. https://doi.org/10.54377/e217-1e97
    41. Special effects: the history and technique. (2001). Choice/Choice Reviews, 38(06), 38–3240. https://doi.org/10.5860/choice.38-3240
    42. Su, Y., Wu, M., & Yan, Y. (2023). Image enhancement and brightness equalization algorithms in low illumination environment based on multiple frame sequences. IEEE Access, 11, 61535–61545. https://doi.org/10.1109/access.2023.3286538
    43. Sun, C., Lai, H., Wang, L., & Jia, Z. (2021). Efficient attention fusion network in Wavelet domain for demoireing. IEEE Access, 9, 53392–53400. https://doi.org/10.1109/access.2021. 3070809
    44. Teotia, K., Mallikarjun, B. R., Pan, X., Kim, H., Garrido, P., Elgharib, M., & Theobalt, C. (2024). HQ3DAVatar: High Quality Implicit 3D Head Avatar. ACM Transactions on Graphics. https://doi.org/10.1145/3649889
    45. The VES Handbook of Visual Effects. (n.d.). Google Books. https://books.google.com.my/books?hl=zhCN&lr=&id=VsXrDwAAQBAJ&oi=fnd&pg=PP1&dq=%22visual+effects%22&ots=Y-hTRGTrSG&sig=yf7XS3rG8G-bM-Uf4k BRt2kxXA&redir_esc=y#v=onepage&q=%22visual%20effects%22&f=false
    46. Tian, F. (2024). Denoising Monte Carlo Rendering Models Combined with JDKPNet in Artistic Design Application Scenarios. Intelligent Systems With Applications, 22, 200338. https://doi.org/10.1016/j.iswa.2024.200338
    47. Vaidyanathan, K., Salvi, M., Wronski, B., Akenine-Möller, T., Ebelin, P., & Lefohn, A. (2023). Random-Access neural compression of material textures. https://doi.org/10.1145/3592407
    48. Veile, J. W., Kiel, D., Müller, J. M., & Voigt, K. (2019). Lessons learned from Industry 4.0 implementation in the German manufacturing industry. Journal of Manufacturing Technology Management, 31(5), 977–997. https://doi.org/10.1108/jmtm-08-2018-0270
    49. Wang, K., Duan, Y., & Yang, Y. (2020). Single image dehazing algorithm based on pyramid mutil-scale transposed convolutional network. Systems Science & Control Engineering, 9(sup1), 150–160. https://doi.org/10.1080/21642583.2020.1833780
    50. Xing, J., Luan, F., Liu, Y., Hu, X., Qian, H., & Xu, K. (2022). Differentiable rendering using RGBXY derivatives and optimal transport. ACM Transactions on Graphics, 41(6), 1–13. https://doi.org/10.1145/3550454.3555479
    51. Ye, W., Chen, H., Zhang, Z., Liu, Y., Weng, S., & Chang, C. (2019). Hybrid scheme of image’s regional colorization using mask R-CNN and poisson editing. IEEE Access, 7, 115901–115913. https://doi.org/10.1109/access.2019.2936258
    52. Zhang, B., Yan, W., Li, G., Fei, J., Zhang, C., & Chen, C. (2019). Image enhancement via indented frame over fusion. IEEE Access, 7, 181092–181099. https://doi.org/10.1109/access. 2019.2956747
    53. Zhang, H., Ye, Y., Shiratori, T., & Komura, T. (2021). ManipNet. ACM Transactions on Graphics, 40(4), 1–14. https://doi.org/10.1145/3450626.3459830
    54. Zhang, Q., Zou, C., Shao, M., & Hong, L. (2023). A Single-Stage unsupervised denoising Low-Illumination enhancement network based on Swin-Transformer. IEEE Access, 11, 75696–75706. https://doi.org/10.1109/access.2023.3297490
    55. Zhang, S., Wang, L., Zhang, J., Gu, L., Jiang, X., Zhai, X., Sha, X., & Chang, S. (2020). Consecutive context perceive generative adversarial networks for serial sections inpainting. IEEE Access, 8, 190417–190430. https://doi.org/10.1109/access. 2020.3031973
    56. Zheng, Z., Zhao, X., Zhang, H., Liu, B., & Liu, Y. (2023). AvatarREX: Real-time Expressive Full-body Avatars. ACM Transactions on Graphics, 42(4), 1–19. https://doi.org/10.1145/3592101
    57. Zhu, D., Zhan, W., Jiang, Y., Xu, X., & Guo, R. (2021). MIFFUse: a Multi-Level feature fusion network for infrared and visible images. IEEE Access, 9, 130778–130792. https://doi.org/10.1109/access.2021.3111905