Metamorphic Testing and Serverless Computing: A Basic Architecture

Yakiv Yusyn, Tetiana Zabolotnia

Abstract


Automated testing of complex software systems can be a challenging task, and today there are a large number of methods for its implementation. One such method is metamorphic testing, which effectively solves the problems of usual methods and is gaining popularity. However, performing metamorphic tests can take a long time, so the question arises of their distributed running, including in the cloud. Thus, the authors of this study considered the designing of a cloud serverless architecture of software for metamorphic testing. The serverless architecture for metamorphic testing is proposed, which is based on the composition of the entire system from 5 individual components: models, data generator, software artifact under test, metamorphic relations, and serverless functions. For each of the main possible types of software artifacts, the possibility of using a serverless architecture for metamorphic testing is considered. The developed architecture is presented in the form of component, deployment, and sequence diagrams. The use of the proposed architecture in practice is shown by the example of testing two software artifacts – a class library and a web application. Performance measurements have shown that despite the additional network delay when running one test, the performance of all tests in general in the case of a serverless architecture is closer to local startup and will be faster with increasing complexity and number of tests.

Full Text:

PDF

References


Weyuker E., The Oracle Assumption of Program Testing, Proc. of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980, pp. 44-49.

Barr T., Harman M., McMinn P., Shahbaz M., Yoo S., The Oracle Problem in Software Testing: A Survey, IEEE Transactions on Software Engineering, 41 (5), pp.507–525, 2015. DOI: 10.1109/TSE.2014.2372785.

Chen T., Cheung S., Yiu S., Metamorphic testing: a new approach for generating next test cases, Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, 1998.

Bourque P., Fairley R., Chapter 4: Software Testing, SWEBOK v3.0: Guide to the Software Engineering Body of Knowledge. IEEE. pp. 4–1–4–17, 2014.

Zhou Z., Zhang S., Hagenbuchner M., Tse T., Kuo F.-C., Chen T., Automated functional testing of online search services, Software Testing, Verification and Reliability, 22 (4), pp.221-243, 2012. DOI: 10.1002/stvr.437.

Zhou Z., Tse T., Kuo F.-C., Chen T., Automated functional testing of web search engines in the absence of an oracle, Technical Report TR-2007-06, Department of Computer Science, The University of Hong Kong, Hong Kong, 2007.

Zhou Z., Xiang S., Chen T., Metamorphic testing for software quality assessment: A study of search engines, IEEE Transactions on Software Engineering, 42 (3), pp.264-284, 2016. DOI: 10.1109/TSE.2015.2478001.

Tao Q., Wu W., Zhao C., Shen W., An automatic testing approach for compiler based on metamorphic testing technique, Proc. of the 17th Asia Pacific Software Engineering Conference (APSEC), pp.270-279, 2010. DOI: 10.1109/APSEC.2010.39.

Le V., Afshari M., Su Z., Compiler validation via equivalence modulo inputs, Proc. of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp.216-226, 2014. DOI: 10.1145/2666356.2594334.

Kuo F.-C., Liu S., Chen T., Testing a binary space partitioning algorithm with metamorphic testing, Proc. of the 2011 ACM Symposium on Applied Computing, pp.1482–1489, 2011. DOI: 10.1145/1982185.1982502.

Jameel T., Mengxiang L., Liu C., Test oracles based on metamorphic relations for image processing applications, Proc. of the 16th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), pp.1-6, 2015. DOI: 10.1109/SNPD.2015.7176238.

Pullum L.L., Ozmen O., Early results from metamorphic testing of epidemiological models, Proc. of the ASE/IEEE International Conference on BioMedical Computing (BioMedCom), pp.62-67, 2012. DOI: 10.1109/BioMedCom.2012.17.

Ramanathan A., Steed C.A., Pullum L.L., Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics, Proc. of the ASE/IEEE International Conference on BioMedical Computing (BioMedCom), pp.68-73, 2012. DOI: 10.1109/BioMedCom.2012.18.

Segura S., Fraser G., Sanchez A., Ruiz-Cortes A., A survey on metamorphic testing, IEEE Transactions on Software Engineering, 42 (9), pp.805-824, 2016. DOI: 10.1109/TSE.2016.2532875.

Chen T., Kuo F.-C., Liu H., Poon P.-L., Towey D., Tse T., Zhou Z., Metamorphic testing: A review of challenges and opportunities, ACM Computing Surveys 51 (1), pp.1-27, 2018. DOI: 10.1145/3143561.

Troup M., Yang A., Kamali A., Giannoulatou E., Chen T., Ho J., A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline, Proc. of the 1st International Workshop on Metamorphic Testing (MET ’16). ACM, New York, NY, pp.33–36. DOI: 10.1109/MET.2016.014.

Roberts M., Serverless Architectures, [Online]. Available: https://martinfowler.com/articles/serverless.html (accessed October 1, 2021).

Microsoft, Azure Functions – Serverless Apps and Computing, [Online]. Available: https://azure.microsoft.com/en-us/services/functions/ (accessed October 1, 2021).

Microsoft, Azure Functions Pricing, [Online]. Available: https://azure.microsoft.com/en-us/pricing/details/functions/ (accessed October 1, 2021).

Microsoft, Pricing Calculator, [Online]. Available: https://azure.com/e/243a23dd3327440bb5567813152d3ef0 (accessed October 1, 2021).

Tresness C., Understanding serverless cold start, [Online]. Available: https://azure.microsoft.com/en-us/blog/understanding-serverless-cold-start/ (accessed October 1, 2021).

Yusyn Y., YetAnotherConsoleTables: NuGet Package, [Online]. Available: https://www.nuget.org/packages/YetAnotherConsoleTables/ (accessed October 1, 2021).

Richter J., CLR via C# 4th Edition, Microsoft Press, 2013.

DuckDuckGo, Main Page. [Online]. Available: https://duckduckgo.com/ (accessed October 1, 2021).




DOI: https://doi.org/10.31449/inf.v46i6.4184

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.