Biographies & Abstracts

Silvia Almagia

OSM DevOps: combining CI/CD and Interop Testing Best Practices

Open Source MANO (ETSI OSM) is a project developing an open source implementation of the Network Function Virtualization Management and Orchestration stack (NVF MANO) aligned with the ETSI NFV Information Model. OSM has a wide and diverse community of 65+ members with different competences and backgrounds in open source development, IT, Telecom and NFV spaces and counts with the support of ETSI’s Center for Testing and Interoperability. One of the main challenges the group faced was the design and setup of the Continuous Integration/Continuous Development (CI/CD) pipeline allowing to automatically test and validate code contributions to the project. The group developed a comprehensive and extensible CI/CD pipeline allowing to automate a wide range of tests going from simple OSS license compliance to exhaustive interoperability testing.

Silvia Almagia joined ETSI in 2012 as a Technical Expert in the Center for Testing and Interoperability. Her current activities include providing technical management and testing expertise in the areas of open source, interoperability testing and proofs of concept (PoC). Among others, she is in charge of the NFV Plugtest Program, the Open Source MANO project (ETSI OSM) and the NVF PoC Framework. Prior to this, Silvia was Lead Member of the Technical Staff at Ulticom where she was responsible for the design and development of software based carrier grade solutions enabling 3G and 4G services.

Matej Artač, Damian Andrew Tamburri, Vasilis Papanikolaou, Ismael Torres Boigues and Guiliano Casale

Testing Big Data applications at design time and runtime with DICE

An important segment of IoT research and practice explores the role of live data, which is only the beginning of innovative applications that then live in the world of Big Data. The providers of Data Intensive Applications (DIAs) face difficulties such as the question of performance of the application runtime. The challenges include how to design the DIA to uphold a desired quality or how to deploy the application without spending too much time on recurring manual operations. In the Horizon2020 project DICE which focuses on developing Open Source software and performance engineering methods for Big Data, we are addressing these problems. With DICE UML metamodel and profiles, the users design their application first by building UML models at three levels of abstraction: platform independent, technology specific and deployment specific level. Testing of the application design then already occurs at the design stage thanks to formal models and formal analysis tools such as DICE Simulation, DICE Verification and DICE Optimization. The novelty of DICE tools comes from integration of tools and industry good practices into a streamlined workflow. It complements functional quality assurance by providing non-functional feedback on the application.

Matej Artač is a researcher at the XLAB Research, Slovenia. He leads the DICE EU H2020’s workpackage on deployment and configuration. His experience includes work on distributed systems and smart energy data. His interests include improving software quality by automating testing deployment.

Luca Berardinelli and Hong-Linh Truong

Combining Model-driven Engineering and Elastic Execution for Testing Uncertainty in CPS

Cyber Physical Systems incorporate many elements from the Internet of Things to interact with the environment. Simultaneously, they contain elements from cloud computing technologies in order to boost computational, networking, and storage capabilities. Therefore, CPSs have to be capable to adapt its structure and behavior at runtime – they have to be "elastic" (eCPS).
In such eCPS, failures can occur during scaling or during normal system operation. In this context, models can help dealing with complexity and driving the engineering process. Within the community of model-driven engineering (MDE), effort is spent to automate testing based on system models (model-based testing, MBT). In addition, this presentation is focused on uncertainty, a so far neglected aspect of CPS.
This presentation is about reporting experiences and challenges on combining model-driven engineering (MDE) methodologies with elastic execution models to design and test the uncertainty of real-world CPS.

Tamás Bőhm, Tibor Csöndes, György Réthy and Antal Wu-Hen-Chang

A flexible, multi-purpose, open source test platform for IoT testing

The Internet of Things (IoT) will have a profound impact in the future. It will enable the inter-connection of devices in various fields of application, from consumer devices, to utilities based meters to sensors in industries. Testing IoT is an important challenge. The challenges stem mainly from the extreme number of connected devices that are expected, highlighting the importance of non-functional testing. But it is not just the sheer number of devices that makes testing challenging but the wide range of functionalities these devices implement with a diverse set of typically open-source technologies, calling for functional testing. Considering the fierce competition in this field, testing frameworks need to support fast time-to-market through easy adaptation to changing technologies and use cases. In order to tackle these challenges, test automation is a must and Model-Based Testing (MBT) techniques are recommended. The presentation will describe the use case of free, open source TTCN-3-based test solutions allowing both functional and non-functional testing, supporting the client and server sides of a wide assortment of IoT solutions, and fulfilling the above test tool requirements.

Robin Bussenot, Hervé Leblanc and Christian Percebois

Integration testing based on Behavior Driven Development

Generally, integration tests are written in natural language and are executed on a test bench step-by-step manually. The lack of formalism introduces misinterpretations by testers. The step-by-step execution can be fastidious, expensive and error prone. The presented work is focused on the system integration testing phase for avionics systems and aims at introducing agile methods with a new generation of test benches. To manage system integration testing activities, concepts from the Behavior Driven Development approach (BDD) were reused: A new category of Domain Specific Language (DSLs), a Domain Specific Test Language (DSTL), was formalized. Furthermore, based on the DSTL an executable code for a test bench was generated via a pivot language. In the presented approach, the focus is on the description of test scenarios for ATA 21 (air conditioning and pressurization) and ATA 42 (Integrated Modular Avionics).

Michele Cargnani and Anthony Wiles

Using TDL in the development of standardized test specifications for IoT and other technologies

ETSI testing activities cover a wide range of technical domains including 3G and 5G mobile communications, Virtual Network Functions, Multi-access Edge Computing, Internet of Things, Machine-to-Machine and Intelligent Transport Systems. With limited resources and the need for a fast turn-around, ETSI is looking to find the most efficient ways to write these complex, large-scale test specifications in an agile yet consistent manner. The Test Description Language (TDL) has shown promises of simplification and efficiency by abstracting from the complexities of the final, executable test cases. This presentation is a report of ETSI’s experiences on the (early) use of TDL as a key component in the toolchain for the development of test specifications in the ETSI Technical Committees and Industry Specification Groups for NFV, MEC, M2M, ITS etc. both for protocols and API’s.

Bruno Legeard

The Power of visual representation for an efficient test design – feedback from an IT implementation project in an industrial company

STREIT Group is an industrial company specialized in the large-scale manufacturing of mechanical parts for the automotive industry. The Enterprise-Resource-Planning (ERP) implemented at STREIT is a strategic backbone to ensure quality services with clients and suppliers in a context highly regulated by ISO Standards (9001 & 14001). Initiated in 2016 on the key domains of procurement, sales, inventory management, quality and finance with SAGE X3 V7, the company decided to adopt an innovative testing methodology based on a lightweight and visual representation of the ERP business processes and rules. The presentation is a return on experience of this project, illustrating the method adopted to produce and maintain acceptance tests (functional and end-to-end). It shows how business analysts and testers may improve their collaborative work using a common language and how this can make the test process more efficient during the phases of test analysis, test design, test implementation and test execution.

Alexis Despeyroux

From Manual Testing to Cognitive Test Automation

In a recent survey of the French Testing Board in 2017, 43% of the respondents replied that less than 10% of their testing is automated. This tutorial explains how companies can achieve complete automation of the end to end testing process for their Enterprise IT applications. Attendees will learn about the evolution of the software testing process over the past 20 years. The tutorial presents latest technology as applied in industrial test automation across various domains. It is about creating models for test generation by reusing various, already available assets like Gherkin feature files, business process models or manual tests. Furthermore, a sneak preview will be given how Artificial Intelligence (AI) can be effectively used to even further automate testing processes and which magnitude of change in test automation can be expected from AI in the not too distant future.

Berk Dülger and M. Barış Sarıalioğlu

Sustainable Test Automation: Collaborate within Team

In IT area, one of the biggest challenges is to keep legacy and new systems alive at the same time. Each product added to portfolio or feature to an existing product might bring regression effects that break software in production. Test automation takes a step further freeing valuable human resource by transferring repetitive regression test activities to computers. As each level of automation has its own type of benefits, highest level UI Tests provide a vision for end-to-end system behavior. However, test automation has its disadvantages. Even solid architectures require continuous maintenance for a smooth execution. Considering it is not that easy neither to have programming skills nor business know-how at the same time, it is much more suitable to divide automation responsibility into layers and assign to project member. Driver Layer, Implementation Layer and Business Layer have different functions one can benefit from. This architecture also promotes re-usability with Keyword-Driven Test Design. It combines proven test design techniques with automated test ownership.

Felix Elliger

On combining IaaS and configuration management into a test automation framework

Test automation evolves – its requirements change based on the system under test as well as organizational demands. On the one hand, the increasingly agile world of software development strives for continuous testing to provide immediate feedback on code changes. On the other hand, the Internet of Things (IoT) targets the secure interconnection of billions of devices, generating lots of data and a respective load on central IoT components. How can test automation be available on demand while scaling with ever-growing demands on test infrastructure and tooling of IoT? The presented solution will sketch the tasks, based on Infrastructure-as-a-Service (IaaS) and configuration management, creating a company-internal platform for automated testing.

Dr. Omar Elloumi

The role of interoperability in building a profitable IoT: the case of oneM2M

Nearly 40% of IoT-related economic impact requires interoperability between IoT systems, according to a market research study from McKinsey. oneM2M partnership project developed a software framework that allows interoperability within and across IoT systems. It can be considered as an “Android” for the Internet of Things but it sits both on the field devices/gateways and cloud servers, and it is a voluntary standard that is influenced by oneM2M members at large. The keynote highlights the problem statement of interoperability for IoT as well as the value proposition of a standards-based approach to IoT interoperability: avoiding lock-in to vendors or cloud providers. The role of complementary open source approaches and interoperability events in improving the quality of the specifications and achieving time-to-market with interoperable products will also be addressed.

István Forgács

4Test, an Agile MBT Method

Model-based testing (MBT) is a test design method which starts from a model of the planned code on the basis of which test cases are then generated from the model. Models in MBT can be graphical, text-based or mixed. However, none of these methods support agile testing. Another problem is that traditional models are invented for program development, not for testing. 4Test is a new method which addresses these problems. Here, the model is a simple Gherkin-based text which is understandable by all stakeholders. The related tool is easy to learn and use. Starting point is a user/feature specification which is related to a model. Analyzing the feature, the agile team tries to collect the categories finding the preconditions, inputs, outputs and categories which can be further divided into choices. The key element of the method are the constraints which are relations among the choices. As this model extends the original Gherkin syntax for the description of the constraints, some new elements make it possible to reduce the work for model writing. A significant advantage is that models can be arranged in a hierarchy. The final output is a generated excel sheet where the tests with all the steps help manual testing.

Marcus Friman

Automated Active Assurance of Critical SLAs for Dynamic 5G Network Slices

This presentation will look at the DevOps in Service Chains and 5G Network Slices PoC driven by Netrounds in collaboration with Telenor, Intel, RIFT.io and Arctos Labs, featuring technology from Open Source MANO. The PoC was created to address why DevOps and automation will be critical to network operation procedures of service providers. It also addresses questions about the kind of challenges which will occur with NFV in terms of increasing network complexity and subscriber demands on quality. The presentation is centered on a core solution of Netrounds, an automatable active test and service assurance tool, and how it can be completely orchestrated to assure 5G network slice SLAs and service KPIs in dynamic, virtual network environments.

Sven Gröning, Jens Schmutzler and Christian Wietfeld

Conformance Testing of Electric Vehicle Charging Communication based on TTCN-3

Electric Vehicles (EVs) are a key component for green transportation and future smart grids. At the same time, they define a new, scalable but heterogeneous global market. Hence, interoperability of charging communication is a core enabler for user acceptance and economics of scale. However, its complexity increases by the number of use cases, e.g. for association, authentication and accounting used for Plug and Charge (PnC), demand management or bidirectional power flow. Thus, a considerable amount of testing is required in order to enable interoperability between independent implementations. EV charging communication brings together cross-domain vendors from the automotive-, utility- and communication domain. However, only a standardized approach based on a publicly available and formally defined testing language like TTCN-3 holds true for the required testing scope of the charging communication interface and a systematic approach for corresponding automated test procedures. The presentation gives an overview of requirement-based conformance testing of the Combined Charging System (CCS) using TTCN-3 and describes furthermore lessons learned using TTCN-3 as a formal test case description in a multi-domain standardization process and how automated testing based on TTCN-3 can improve interoperability of electric vehicle charging communication.

Patrick Harms and Jens Grabowski

Automating Usability Testing for Prototypes of the Things in the Internet using Augmented and Virtual Reality

The Internet of Things (IoT) is not only facing challenges like technical aspects, but also the increasing complexity of devices. Moreover, users need to be able to configure and use IoT devices as well as to understand and solve occurring problems. To ensure this, developers need to apply usability techniques during the development process. Augmented and Virtual Reality (AR/VR) are possible vehicles to apply usability testing by using virtual prototypes of devices under development. Presenting such a virtual prototype to a user makes it possible to get an impression whether the users can carry out the intended actions and to simulate challenging user situations. Using a Google Cardboard or similar equipment, AR/VR can nowadays be experienced by almost any owner of a smartphone. This presentation describes the overall scenario for applying automated usability testing on virtual device prototypes by using AR/VR.

Jani Haukinen, Jouni Rajala

Levering test automation to verify infrastructure

Large scale test automation solutions have tendency to have slow feedback cycle, unreliable test results and large root cause analysis scope. Due to this, there is considerable potential for improved efficiency by adding infrastructure specific automated tests. On large scale enterprise solutions, one of the challenges is maintaining test automation reliability. Initial infrastructure testing enables faster feedback cycle and considerable reduces the failures reported from application/service level functional tests that are caused by infrastructure problems. This limits the scope of root cause analysis because it is typically known if the failure is application code or in the infrastructure setup. Moreover, this solution has the potential for considerable time and money savings while using modern SDLC models such as Agile, DevOps, SAFe.

Jens Hempel and Dirk Reufsteck

Certification Testing for Communication in Virtual Power Plants

Testing is essential for decision making in the certification process of energy systems. Strict adherence to standards on communication/cyber security is the basis for operation of the critical energy infrastructure (including virtual power plants). The industry association VHPready choses the lean test process as basis for the certification of the components and systems of the related VPP’s. This common approach shall promote a standardized industry solution and hence the spread of this important Smart Grid technology. Through the membership of Fraunhofer FOKUS in the VHPready industry alliance, core elements of the TTCN-3 test automation language are used in the introduced test suite. Besides that, practices of test development for accredited operation were applied. The VHPready approach may inspire other multi-standard certification test developers directly. The introduced process bridges the gap between demand for accredited certification and a lean test process. This presentation provides insight into the requirements of accredited testing based on two independently developed WAN standards used in the energy environment.

Ivan Jovanovikj

Framework for Constructing Context-Specific Migration

Software migration is a proven and often applied technique for software reuse. Software testing is an important activity in software migration as it verifies whether the migrated system still provides the same functionality as the source system. Since there can be thousands of test cases, automated migration of test cases is also desired. Following the idea of model-driven software migration, the reference method extracts models out of existing test cases and reflects the system changes. At the end, in a model-based testing manner, it generates test cases for the migrated system. All in all, a novel approach for constructing context-specific co-evolution enabled migration methods for test cases is presented. This method was applied in an industrial project with the main goal to migrate parts of the well-known Eclipse Modelling Framework (EMF) along with the Object Constraint Language (OCL) from Java to C#. The approach is basically applicable in any migration scenario that includes existing test cases. Depending on the context information, an appropriate migration method gets tailored and it can be applied to migrate the test cases.

Mika Katara

Ten Test Automation Pitfalls to avoid when introducing new tools

A project trying to introduce advanced test automation tools can fail due to non-tool related reasons hampering the adoption of the new, important technologies. These problems are common to all test automation projects introducing new tools. Some points which make the project fail often are unrealistic expectations while making too big promises, automating chaos, lack of manual testing skills as well as a lack of communication skills or too complex verification steps amongst others. Therefore, further discussions in this context should be promoted.

Anne Kramer and Martin Beißer

Strategies for safety-relevant vehicle tests at Continental

Developing safety-relevant components like air bags has never been easy, but the increasing degree of interactions between various components makes it harder than ever. Basically, each internal state of the component/controller has to be combined with each situation which may occur in the environment of the vehicle. The approach of the specific method for testing safety-relevant car functions combines model-based test design, automated test case generation and automated test execution in a simulated environment. While test execution is performed using a virtual test driving tool, model based scenarios are generated automatically using a test case generator. The exact tool chain consists of Enterprise Architect, MBTsuite, PTC Integrity and CarMaker. Because of the high flexibility, test execution is possible at every stage of development (MiL, SiL, HiL). The novelty of the approach lies in the continuous integration of artefacts and tools – from the test model to executable test scenarios in the virtual environment of the simulation system. Applying the method allows to decrease the effort for implementing and maintaining test cases.

Markus Lachenmayr and Florian Krautwurm

Testing a Fault-Tolerant Cyber-Physical System designed for Testability

A dependable cyber-physical system (CPS) responds correctly and timely in its intended environment. This requirement also applies while testing reliability, availability and performance properties. Testing such systems and properties pose various challenges. It includes stimulating selected CPS parts with correct as well as faulty data in critical situations and checking selected CPS parts at selected points in time or during time intervals. Other challenges are the differentiation of the CPS's treatment of temporary and permanent faults as well as the absolute non-disturbance of the tested system's functionality. Meeting these challenges requires to design the CPS architecture for testability. For the presentation, an integration platform of an X-by-wire eCar will be the example of a CPS. After deriving testable requirements, a series of design tactics enabling or improving the system’s testability will be enabled. Therefore, the focus is on test probes – Built-in Monitors and Maintenance Interfaces with particular demands in probe environments.

Philip Makedonski, Gusztáv Adamis, Martti Käärik, Finn Kristoffersen and György Réthy

From TDL to TTCN-3: A Step by Step Tutorial

This tutorial will briefly introduce both TTCN-3 and TDL including their scope and limitations as well as semantic gaps between them. With the ultimate goal of showcasing how executable TTCN-3 code can be derived from a higher level TDL test description, necessary steps and assumptions, as well as common challenges associated with mapping languages at different levels of abstraction, will explained in detail and supported by illustrative examples.

Philip Makedonski, Gusztáv Adamis, Martti Käärik, Finn Kristoffersen and György Réthy

Mapping TDL to TTCN-3

While continuing development of the “Test Description Language” (TDL), the Technical Committee “Methods for Testing and Specification” (TC-MTS) at ETSI is focused on bridging the gap between high level test purposes and test cases. The process of stepwise development of tests from requirement specifications is well established and used in both, standardization and industry. This process is supported by methods and languages, such as TTCN-3 and TDL. TDL bridges the gap between declarative test purpose specifications and imperative test case specifications. The lack of a standardized mapping of TDL to TTCN-3 may result in a proliferation of different and possibly incompatible tool- and user-specific mappings of TDL test descriptions to executable test cases. Facing these challenges and common problems of bridging the gaps between languages and representations, users and tool vendors aiming to do the same with other technologies may benefit. This presentation will showcase experiences and the currently ongoing work on the standardized mapping of TDL to TTCN-3.

Klaus Moessner

IoT based services

Services that involve actuation, or rely on the availability of information and data obtained from sensors or from connected devices in general are often rather volatile. The reliability of connected devices is mostly sub-optimal, and such devices, especially if unsupervised, may have been tempered with. The communication links are mostly over the air and therefore may suffer from the various types of wireless communications problems. IoT based services will need to have built in means to overcome these obstacles; this speech will discuss a range of the issues that lead to problems in Service provision, highlight the need for automated testing measures to test IoT based Services, and will present techniques currently investigated that may help to overcome such problems.

Francesco Oppedisano, Sergio Borghese, Francesco Lamonica and Enrico La Vela

TDL for testing collaboration IT services: the NetResults experience

In the era of Industry 4.0, human labor is becoming “smarter” than ever and, as a consequence, collaboration IT services such as “desktop sharing” evolve to be of mission-critical importance for businesses. The specific action performed on a specific GUI can be separated from the coordination and communication that takes place among the automatic agents performing the test activity. This approach paved the way to a new set of “comparative” testing setups which are not only capable of testing the service itself, but also to benchmark it against other (homologous) services available in the market. Using ETSI TDL, interactions among testing agents can be described and drive the development of a set of mechanisms that are common to any service of this kind. The application specific components were developed using computer-vision frameworks such as Sikuli. The main purpose of the presentation is to share information about impressions and feelings about this TDL-based approach to test an IT service.

Jorge Pascal, Patrick Meuth, Jörg Reiner and Mathias Helminger

Introducing and adopting Model-Based-Testing for development projects

As the automotive industry is facing the confluence of different new technologies, the implementation of highly integrated software functions in embedded systems is required. The effects of these changes on testing processes and methods reflect the need of higher software quality standards. Model-based Testing (MBT) addresses these challenges by reducing the time to market while maximizing quality standards such as test coverage. A further advantage of MBT is the possibility to derive test cases for subsets of a given model. Test suites can be derived for the different variants of the same version out of one single-source-model. Although MBT refers usually to all testing activities through the entire model-based-development process, this approach focuses on the Hardware-in-the-Loop (HiL) testing phase. The adoption of MBT allows to enhance the testing quality, to reduce the testing cycle time, to develop new testing strategies and to have accurate test metrics such as test coverage.

Patrick Meuth obtained his Bachelor of engineering in Renewable Energies at the TU Ingolstadt (Germany). He works as test development Engineer at TKI Automotive with special focus on Model-based testing, an Audi AG and IPETRONIK GmbH joint venture founded in 2012.

Mathias Helminger is a systems engineer at Assystem Germany GmbH since 2014. In 2014 he obtained his MSc. in computer science at TU München in the field of testing in real-time contexts. During his studies he was working for BMW, re-designing a test automation framework. Since 2015 he is the lead developer of MODICA (model-based test design) but is also involved in broad activities around Assystem´s testing ecosystem and topics around IT-security.

Rafael Schirru and Anke Abromeit

Configuring a TTCN-3 Test System in a Complex Multi-System Environment

The telematics infrastructure (TI) is a complex environment including decentral products such as smart cards and card terminals as well as central products, i.e. servers in the TI network. It is the aim of Gematik to connect all parties involved in the German public health sector by building and operating the telematics infrastructure. Approval tests comprise product tests, performance tests, integration tests as well as end-to-end tests. In some product tests, each system with interfaces to the system under test is mocked using parallel test components (PTCs) in TTCN-3 or simulated outside of TTCN-3. When executing test cases, the test case configuration has to be adapted according to different criteria. ConfiqStore allows gematik to adapt a basic test configuration for different test cases, products and workplaces. The configuration data is held in XML files and organized in different layers. Eventually, the XML data is converted to TTCN-3 records and provided to the test scripts. Currently, ConfigStore cannot be used outside of gematik. This presentation aims at offering room for discussion about the topic of configuration of complex test systems for test designers using TTCN-3.

Martin Schneider and Marc-Florian Wendland

Testing Non-Functional Quality Characteristics of Cyber-Physical Systems

Cyber-physical systems have to deal with several challenges in contrast to traditional systems that are mainly related to their complex interaction with the cyber and their physical environment. These leads to challenges when testing those systems, in particular with respect to quality aspects such as reliability, security and safety aspects. Some of the key questions are: Which scenarios may have a negative impact on these quality characteristics? How to model those situations adequately and cost-efficiently? The presentation will discuss the abovementioned challenges and propose a novel solution based on uncertainty and search-based testing of cyber-physical systems. The integration of the proposed approach in a tool for model-based testing (Fokus!MBT) will be shown. The audience will get a new view on modeling and testing of non-functional quality characteristics for cyber-physical systems that will become essential for new, autonomous applications of cyber-physical systems such as autonomous cars or industrial robots interacting with humans.

Harry Sneed

Testing Micro Services

Testing Micro Services is important because of an increasing number of applications are now being built with Micro Services. Converting legacy systems into micro services is a favored approach in software migration. The biggest cost factor is the testing of the services. The novelty of this approach is the technique of extracting test cases from the WSDL or REST interface, converting them into a test script to generate requests and to validate the responses. The outcome of this work is a tool supported process for modal testing of micro-services against a semi-formal service specification in natural language which is converted to an assertion script. The requests are generated from the pre-assertions, the responses are validated against the post assertions. In the end, the quality of the service is evaluated and presented in a tester’s dashboard. The key takeaway is the knowledge of how to approach the problem of defining, specifying and testing micro services.

Markku Suominen and Tommi Pernila

Automated adaptive quality and security monitoring in 5G networks

As modern networks are becoming more complex with 5G, it has become quite impossible to fully understand them. In realistic, large scale installations there are a multitude on small errors such as occasional timeouts and unnecessary packet replications, which are not critical on their own. When combined, these small errors can form much larger issues over a time period and are almost impossible to spot by human eye beforehand. This approach enables automated testing and monitoring of security within networks and simulation of potential incidents due to security related anomalies or cumulative amount of minor bugs passed through earlier testing stages. As a part of a research program, an automated solution (sensor) has been developed. This solution consists of machine learning capabilities, neutral networks and other state of the art technologies that can automatically spot anomalies in complex environments. This solution has been developed for spotting security anomalies in telecommunication and enterprise networks.

Marcin Szczukiewicz and Łukasz Wałach

Automated testing model for complex and highly configurable software systems in globally distributed organization

Nokia’s radio base station facilitates wireless communication between user terminal as IoT and a network. It is a complex system consisting of several components, either hardware or software, and can be configured for different operation settings. It is obvious that testing of large systems manually is an outdated technique, however traditional test automation within an international corporation causes creation of autonomous units formed. It leads to a situation in which the number of independently created test beds and their configuration variants increase at a high rate. To reduce time and manpower cost, it is necessary to think outside the box by treating testbed just as another software component. This presentation is about the process of reducing the number of manual tests to a minimum, of shortening the time needed to deliver the software to the customer. Furthermore, it will be shown how to reduce costs and to achieve Continuous Delivery Deployment Pipeline with automated acceptance tests in the cloud.

Madis Taimre and Õnne Mets

Model-Based Testing of 3D video games

Currently, almost all game testing is done manually. Some automation is done using bots, capture and replay tools, using custom scripting framework or model-based testing. The video game industry is one of the fastest growing segments with compound annual growth rate 4.8% from $71.3 billion in 2015 to $90.1 billion in 2020. Every day, more than 500 new games are submitted to the iOS App Store. Despite the heavy development, there are very little options for automated system testing of games and there is definitely a desperate need for it. The presentation is based on a case study provided by Virtual Heritage where model-based testing was applied on a virtual reality game VR Quiz. The presentation proposes a model-based approach for automated system testing using a framework which was created as part of the case study for the most popular game development platform, Unity. It can be attached to different test automation platforms.

Marc-Florian Wendland and Ina Schieferdecker

Suitability of UTP and TDL for model-based testing – Checking for compliance with ES 202 951

The main objective of MBT is to improve the efficiency by automating the test design activities. The efforts require to create maintain and update a test model are thus success-critical for MBT. These efforts are partly influenced by tooling, but mainly depend on the adequacy and applicability of (test) modelling languages. ETSI has developed a standard (ES 202 951) on requirements a language for MBT should support in order to be adequate for MBT approaches. ES 202 951 was written to effectively specify requirements that support automated test design in the first place. Although ETSI ES 202 951 is not a new standard, it attracted little attention from the industry so far. In the presentation, we demonstrate an application of ES 202 951 to two standardized test modelling languages (i.e., UTP and TDL) and a systematic evaluation of the conformance of these languages to ES 202 951.

Hans-Werner Wiesbrock and Sadegh Sadeghipour

Reactive test of embedded systems using models

The focus in this presentation is on testing embedded systems with a high variability of interaction with the environment. Consequently, their cost-effective quality assurance is a crucial requirement of a successful development process. The solution presented here is a model-based testing approach enabling an automated test execution and evaluation. The novelty of the presented approach is using a Domain Specific Language (DSL) within the environment model as well as a parallel simulation/execution of the environment model and the test object. There are different time scales of embedded systems affecting their test process - very short time scale, short time scale and low time scale. This presentation is focused on cases of low time scale and recommends using a kind of reactive test specification. The approach presented has been implemented using the test framework ContinoProva, which performs automated test execution and evaluation across various tools deployed in the test environment. Reactive tests, specified in the UML tool Enterprise Architect using UML Testing Profile and the DSL extension mentioned above, were integrated into ContinoProva using the interface mentioned.

Jens Grabowski, Tomas Urban, Kristóf Szabados, György Réthy, Julien Deltour and Jacob Wieland

Modernizing TTCN-3

The TTCN-3 testing language has intensively been developed by ETSI during the last decade and, by today, it consists of 14 ETSI standards. The language is also endorsed by ITU-T as the Z.16x and Z.17x Recommendation series. By now TTCN-3 is used exceptionally as the formal specification language of standardized test suites and has also become an important testing technology at various ETSI member companies and in several industrial domains and standards organizations. On evolving the language for more than a decade, ETSI TC MTS has recognized the need for the harmonization of language concepts as well as adding sophisticated features for application and test framework development including object oriented features. For doing this, ETSI TC MTS initiated the Specialist Task Force (STF) 533 on “TTCN-3 Evolution 2017”. This presentation will present and discuss the new object oriented features of TTCN-3 and report on the harmonization of TTCN-3 language concepts.

Man Zhang, Shaukat Ali, Tao Yue and Phu H. Nguyen

Uncertainty-wise and model-based Testing of Industrial Cyber-Physical Systems

The industrial applications of cyber-physical systems (CPS) are visible in many domains like in robotics, aerospace or healthcare. Given the critical nature of their applications, it is important that a CPS performs its function normally even in an uncertain environment. In this poster, uncertainty-wise and model-based testing frameworks will be presented. The frameworks are specifically developed for testing CPS under uncertainty. They utilize model-based testing as well as search-based testing and include one uncertainty-wise prioritization solution to prioritize the sequence to execute test cases in a cost-effective way with the multi-objective search. To promote the application of these uncertainty-wise and model-based testing frameworks in industry, some have already been implemented in the Certifylt tool – a commercial model-based testing tool.