An experiment-driven approach to software product and service development is getting increasing attention as a way to channel limited resources to the efficient creation of customer value. In this approach, software functionalities are developed incrementally and validated in continuous experiments with stakeholders such as customers and users. The experiments provide factual feedback for guiding subsequent development.
Although case studies on experimentation conventions in the industry exist, an understanding of the state of the practice is incomplete. Furthermore, the obstacles and success factors of continuous experimentation have been little discussed. To these ends, an interview-based qualitative survey was conducted, exploring the experimentation experiences of ten software development companies.
The study found that although the principles of continuous experimentation resonated with industry practitioners, the state of the practice was not mature. In particular, experimentation was rarely systematic and continuous. Key challenges related to changing organizational culture, accelerating development cycle speed, measuring customer value and product success, and securing resources. Success factors included an empowered organizational culture and deep customer and domain knowledge. There was also a good availability of technical tools and competence to support experimentation.
