After work with Vagrant catalog and after writing tests in Robot Framework it's time to summarize and shift to new level. It is time to organize automated testing of bulk part of products with Openshift. Below I will try to describe causes which led me to it.
Testing at present:
Vagrant - very nice for manual testing. Created catalog with virtual environments which provide uniformity in manual testing (and not only). Once created and documented - all teams use them without any problems. Provision and semi-automated deployments.
Robot Framework - complete framework for testing purpose, good documentation, expandable by modules, built-in report system, documentation generation etc.
Disadvantages of current situation: Vagrant (VirtualBox as hypervisor) is very slow solution. Lack of orchestration and management does not give to deploy complex virtual environments. (if I need two SMTP services, for this purpose I should: have two virtual machines, two unique DNS records, two unique IP-addresses, somehow convey these parameters in tests etc. and all this is very slow and takes a lot of times for supporting).
Core requirements for automated testing:
- Reliable solution.
- Scalable, controllability, autonomy.
- Speed and high density.
Nobody want use product which will stop development in a future. All small projects and startups may to close in any time, but such monster like Red Hat can not afford it. Besides that Red Hat has solid reputation as active participant of Open Source.
Product should be scalable and require minimum maintenance. All resource management (CPU, RAM, IP address, DNS records, data storage etc.) should be as automatic as possible. Rich management tools like command line interface, API, web console. Our teams should submit tasks and get a results on exit.
Huge quantity of products work with full virtualization but it is very slow, even if working in stateless mode and you have many precreated objects. Speed limit - it very important question when it relate to huge amount of tests and required amount of hardware resources.