UCAAT2016 Accepted Abstracts
Zoltan Elzer, Patrik Paksy and Benjamin Teke. Big Data interpretation and
challenges in mobile network testing
Abstract: The
complexity of mobile networks is continuously increasing, meshing whole
countries, including several network functions and interfaces between them.
Ericsson is one of the biggest manufacturers of network functions like radio
base stations, user databases and application servers. All these functions run a
software logic that handles signalling protocols, maintaining user and service
information and communicates with peer functions in a standardized way. Despite
the well-defined functional roles and signalling protocols, testing every node
and every interface between them individually does not necessarily imply that
the network as a whole behaves as requested. Therefore, network level testing of
signalling sequences are necessary and useful.
Testing of telecommunication networks are performed on multiple levels. The
complexity of testing and amount of data need to be maintained and processed are
exponentially increasing when we go from single software module tests to
end-to-end network level acceptance test of a whole network. Monitoring the
network and troubleshooting problems is essential part of testing flow.
Automation of such a testing requires the support of multiple protocols and
smart techniques to correlate signalling messages captured on different network
interfaces. Based on the sheer size of a network and its data traffic, it also
needs a fast, efficient and scalable way of performing this correlation, which
can be solved by using Big Data processing on a computer cluster. These
technologies are not only giving us a way of managing this amount of data
without the need for a supercomputer, but with the right tools and
implementation, online, near real-time network analysis is also possible.
In addition to the new data handling
approach automatic recognition of the network and network elements, categorizing
signalling flow, and automatic highlight of faulty scenarios can also quicken
the whole testing flow. With help of this, testers are able to focus only on the
most important data from huge amount of available information, and later as an
evolution, it would be possible to apply automatic procedures for well-known
situation.
Andras Naszrai. Contract testing in the cloud in GE Healthcare
Abstract: We
have created a data drive contract test framework to lower the integration costs
of our cloud based software and we put this framework into the cloud to use it
as a testing as a service solution. We use the same framework to automatically
simulate the surroundings of our software components to further decrease the
cost of our contract testing efforts by creating on-demand isolated
environments.
Andrew Pollner. Test Automation
Engineering
Abstract: As with testing itself, test automation does not have a
formal pedigree of education or industry standard support. While this is all
starting to change, the application of test automation for a given project is
still not a guaranteed success, given the technical and other challenges.
By creating a body of knowledge for test automation engineering, the ISTQB is
helping to define standard concepts that anybody working with test automation
should know and be able to apply to their test automation efforts. This
presentation will be a discussion of the ISTQB Advanced Test Automation –
Engineering Syllabus, a published document currently in beta.
Harry Sneed. Assuring the Quality of Test Cases
Abstract: IT
users are now making a significant investment in testing their applications. A
major part of that investment is devoted to the creation and maintenance of test
cases. Many organizations have already several thousand test cases. The quality
of test cases is however not enough. The test cases must also be of sufficient
quality, meaning that they not only cover the requirements but also that they
are consistent and complete in addition to being formally correct. The users
must know that they can rely on their test cases. For that reason the quality of
the test cases should be checked. This presentation describes an approach to
ensuring the quality of test cases by checking them against the requirement
specification and the design as well as against the formal rules of defining
test cases. The approach has been implemented in a tool and applied in several
IT projects. The result indicates that this approach can be highly useful in
assessing the quality of test cases.
Abstract:
ETSI UCAAT 2016 Presentation Proposal
October 26-28 2016, Budapest, Hungary page 2 of 3
activities in different phase We take about half a year to move all test cases
(>10,000) to cloud seamlessly. Time-to- faultfinding is reduced by 30%, product
quality is intuitive and demo on office TV every day, release candidate become
available always. The way how we do clouding test and the learning we gain when
moving test on cloud can help others who face the similar difficulty and
especially for the one who work for multi-components product. And also I hope I
can have this chance to communicate with other test experts about clouding test.
Branka Rakic. Continuous testing starts with the smart architectural decisions
Abstract: The
quality of software is based on synergy of all the parties involved in the
effort: Development, Testing and IT Operations. From that perspective, the
software testing process will start from the moment when the first line of code
is created and submitted to the code repository, to ensure that success criteria
and quality requirements will be met and will be compliant with the agreed
Definition of Done.
The efficient test strategy is based on
a proactive approach to build, maintain and regularly check the code quality
throughout the entire software lifecycle. It builds on current best practices,
aiming to extend and refine them as much as possible in the direction of test
automation. Tests are automated and integrated into the build process as early
as possible. A continuous testing platform runs in the background, automatically
executing the tests and ensuring that issues are identified almost immediately.
Thus reducing the time-to-release and closing the circle to ensure a successful
Continuous Delivery process.
In this presentation, the case study
about a Public Transport Software will be demonstrated. The system consists of
Android tablets and different sensors located in vehicles which are sending
messages to the back-end. The back-end processes the messages and forwards them
again to vehicles.
The implementation of the system is in
C# and the continuous testing is implementing using NUnit framework. In order to
support fully automation, the tools like a message generator and data player
have been implemented. Furthermore, publish–subscribe pattern provides
exchanging messages between system components and test hooks at the same time.
The other challenges that have been faced are simulating real scenarios like
sending messages from a fleet where the messages are produced by multiple
vehicles with multiple IPs.
In order to have high stability of the
tests, a clean test environment is provisioned using PowerShell DSC scripts for every
execution as a part of the continuous pipeline. Also, every test is independent
- have own data set. NDBunit enables creating required data set which is removed
in the same test enabling clean database for the other tests.
Sebastian Dengler and Patrick Wunner. Automated System Testing towards
Continuous Integration in Embedded Environments (Automotive Industry)
Abstract: Continuous
Integration, already implemented for a large range of software development
domains, is also becoming more and more important in embedded environments like
the development of electronic control units in the automotive industry.
Recently, we introduced a concept to
integrate automated Hardware-in-the-Loop testing of automotive electronic
control units into an emerging continuous integration environment. The
continuous integration environment considered by our concept is based on a
Jenkins server and the test framework used for automated testing and reporting
is iTestStudio, which is a proprietary development.
In this presentation, the intention is
to share our recent experience with implementing this concept and its
application to a real life customer project.
The talk consists of three parts.
In the first part, the importance of
continuous integration for the development of embedded systems, especially
automotive electronic control units, will be highlighted and possible use cases
with focus on testing activities will be introduced.
In the second part, our conceptual
approach for integrating automated Hardware-in-the-Loop testing into commonly
used continuous integration environments will be presented.
The third part is supposed to exchange
our experience while implementing and applying the approach to the test process
of a real life customer project.
The talk concludes with best practice
recommendations and lessons learnt.
Gabor Megyaszai. Streamlining
performance verification through automation and design
Abstract: While
addressing the change necessary to achieve continuous delivery capability and
DevOps practices as one of the first obstacles we had to face was the overhead
put on the delivery time by the traditional way of performance verification. The
creation of environments and long test executions – mostly derived from
insufficient practices – heavily limited the timeframe and even the content of
each release. Problems originated in the low level of automation of SUT creation
and configuration and the automation difficulty in test case execution,
especially in result analysis.
To establish baseline we collected data
from the past two years about number of environment bring-ups, bring-up time,
number of test case executions and test results being in “Not analysed” state.
To tackle this complex problem we took
two parallel path. On one hand, we started redefining how we are doing
performance tests, by reducing the test and SUT complexity to the smallest
possible increment. On the other hand, we get heavy automation work going to
cover configuration and execution.
We created a scalable dynamic deployment
framework (AvED – Automated virtual Environment Deployment), with which we can
carry out on demand SUT and non-SUT deployment in a fraction of the original
configuration time. This Python based application carries out the deployment
through REST API and triggers the configuration system. The configuration can be
read from three different sources: predefined XML, described by test
requirement, selected manually (for special circumstances). We joined AvED with
Jenkins effectively fitting performance verification into our SCM/CI pipeline.
With the utilization of AvED and scalable systems we are currently able to
deploy 48 parallel test environments to our available cloud capacity both on
VMware and Openstack infrastructure.
To test the performance of different
possible cloud variants we created a Python based measurement system to
establish the performance baseline on reference configuration and developed a
Python, Power CLI and Bash based performance predictor tool which utilizes IPSL
and Gatling. The results of performance prediction can be further refined and
expanded by the use of Clover (Cloud Verification) tool. With the use of
predictor we can already specify the VNF performance on a never before tested
infrastructure without having to actually deploy the VNF itself.
In order to automate the test result
analysis we are developing a SUT behaviour and data discrepancy recognition
frame along with selective log analytics.
With our renewed test cases and
increased automation, we significantly decreased performance verification
turnover time from months to weeks or days and we are able to provide load/mass
traffic testing feedback to development even in the desired 2 hours long CI
cycle. With the reduction of testing cycle times we are able to introduce new
types of tests into our delivery process, such as new ways of chaos and
robustness tests with Pumba or Faulty Cat, which ultimately leads to higher
coverage and quality affecting all stakeholders throughout our VNF delivery.
In my presentation, I would provide a
general overview on how we redesigned our test cases in order to meet automation
possibilities, and would introduce our dynamic deployment framework and its
connections towards the SCM/CI pipeline and automated test execution frame.
I would demonstrate the key benefits of
simplification in automation by which we were able to create a modular and
generic system which can be applied companywide or even outside Nokia.
Stefan Dorsch and Andreas Ulrich. Embracing Non-Determinism in Testing
Abstract: Software-Engineers
shy away from non-deterministic behaviour in their software systems they develop
for good reasons. They fear that non-determinism causes unintended and unknown
system states, from which the system cannot recover and continue performing its
anticipated task. Similar arguments are given by software testers. Their fear is
that test runs cannot be reproduced and determining the reason of a discovered
fault becomes a highly costly, if not unmanageable task. While these arguments
cannot be ignored, there is a merit for non-determinism in testing. The major
benefit is that non-deterministic tests enable the detection of system faults
that are hard to discover using conventional means of test design such as tests
derived from user stories or testing based on code-coverage criteria.
Istvan Turai, Tamas Cser and Ray Grieselhuber. How
Testing Automation is the Perfect Domain to Apply Machine Learning
Abstract: Software
testing, today, has been outpaced by modern agile development methodologies. The
tools and techniques, while evolving, still rely on manual operation or coding
of automation and these solutions result in software projects that lack
intelligence. This leads to an enormous bottleneck in the software development
process, as new features and updates are frequently delayed by the amount of
time it takes to test them. To be sure, many positive advances in the domain of
behaviour-driven development have contributed to better automation solutions.
Testing at the functional layer remains a problem. Advances in browser
automation technology, especially open-source tools such as Selenium, have
helped greatly. Many problems, however, persist.
Fortunately, we are at a time where AI
and machine-learning technologies are now viable for small teams. We have spent
the last 2.5 years working on adapting these technologies to the problem of
automated testing. This experience has confirmed our hypothesis that the future
of automated testing is in intelligent systems learning from the websites and
applications where it is applied.
In this session, we will share what we
have learned, the specific challenges we encountered, and how we solved them on
a technical level.
Florian Spiteller. Automated Testing for Autonomous Cars? Challenges, Solutions,
Opportunities
Abstract: Automated
testing, mostly using the Hardware-in-the-Loop approach, has been the industry
standard to help ensure the correct behavior of safety relevant software in
automotive ECUs. These ECUs are commonly connected to miscellaneous systems,
sensors and actors inside the car. New developments in the field of Vehicle-2-X
communication and autonomous driving raise the need to also test systems which
are wireless-connected to the environment (infrastructure, other vehicles)
surrounding the car.
The talk will show that this cannot be
done by simply using the proven test methods and will point out the challenges,
e.g. time synchronization of distributed but connected events. This is done with
the help of a realistic example: A connected car (1st time-base) approaches a
junction equipped with intelligent infrastructure (2nd time-base), the
interaction is verified with external test equipment (3rd time-base). As the
mentioned new technologies are about to enter the market soon the industry is,
also based on the authors own industry experiences, in urgent need for new and
adapted test methods.
The main part of the talk is focused on
presenting an approach to test connected systems using existing toolchains in a
real world scenario. With the help of a self-developed, deterministic wireless
bridge the fault injection method can be utilized also with connected systems
allowing an automatization of the test. To better explain this test method, the
above mentioned example is extended to demonstrate an automated test of an
autonomous car. This also shows the high relevance of the described test method
as in the near future more and more systems for connected and autonomous cars
have to be tested. With the help of the described scenario the automated test
could not only be used for testing itself, but also for machine learning
purpose.
The author will finally explain the
advantages, disadvantages and faced challenges of the shown concept. To help
adapt the approach also to other industry fields the main development steps will
be outlined. The talk concludes with a summary pointing out the key-takeaways
and lessons learnt.
Pekka Aho and Matias Suarez. Automated regression analysis through graphical
user interface
Abstract: A
large part of software testing through graphical user interface (GUI) is still
performed manually. Although capture and replay (C&R) tools are commonly used to
automate the most critical test cases but maintaining a large test suite with
C&R tools usually requires too much manual effort as the GUI changes often
during the development. Model-based GUI testing (MBGT) has been proposed to
reduce the maintenance effort but most MBGT tools require specific modelling
expertise and significant effort in creating the models.
We would like to introduce the concept
of automated regression analysis through GUI and present our experiences in
using it for more than a year in automating the regression analysis and testing
during the development of commercial end-user GUI software products. The concept
of automated regression analysis through GUI is quite new, and the authors are
the only ones having academic publications on the topic.
The idea in automated regression
analysis through GUI is to use dynamic analysis to automatically extract a new
behavioural model of each version of the GUI and automatically find the changes
in the GUI by comparing the extracted models of consequent versions. The method
has very high level of automation, the manual work remaining in going through
the detected changes and deciding if the change was intentional or a regression
bug.
Murphy is an open source tool for
automated regression analysis and testing through GUI. Originally Murphy was
developed as an internal tool for F-Secure Ltd but it has been open sourced.
The experiences from F-Secure Ltd have
been very positive. Continuous integration tool Jenkins triggered Murphy tool to
extract a new model of the latest version of the GUI application three times a
day, comparing the new model with the previous version, and reporting changes as
web links attached to email. Although a large part of the changes were
intentional or “false positives”, it did not require too much work to visually
inspect the results graphically presented as screen shots 3 times a day.
Compared to the earlier automated test scripts, especially the maintenance work
was significantly reduced.
Ish Kumar and Stephan Schulz. Industrial deployment of MBT-based
test automation in a large enterprise IT program
Abstract: Many
people talk about MBT in an industrial approach – and possibly report about
piloting it. We have deployed MBT based solution which encompasses tools from
multiple vendors for multiple years now in a large enterprise IT testing
program. As part of our next steps we are extending our approach to also include
and integrate behavioural driven development techniques used in our DevOps
environments.
The work presented in this presentation
demonstrates full test automation of the testing process in a real world
industrial setting. The solution presented completely automates test design,
test execution, as well as the interface towards requirement and test
management. MBT based solution also helps identify the impact of change in
requirements on testing scenarios.
The solution has been applied
successfully for testing of more than 20 mission critical applications and more
than 40 end-2-end services within a large industrial Enterprise IT program in
the telecommunication sector. The goal of the presentation is to share
information how this success was achieved and naysayers within the organization
were overcome using this test service provider as an example.
Karl Ambrus. Modelling
of Complex Distributed Test Scenarios
Abstract: Aircraft
are long living products (40 years) with a lot of upgrade programs and
enhancements during their lifetime. To handle this it is essential to guaranty a
continuity of the test environment and to cover the growth of functionality,
complexity and variants of the aircraft systems.
For the integration environment the main
focus is the enhancement of the toolset and methodology for system integration
by extending the capabilities with new methods like test case generation,
parallel virtual testing and highly automated regression testing.
The classic way of testing is based on
S/W testing, subsystem and system testing each using a different toolset and
test procedures. Testing of virtual equipment using simulation environments and
system integration on test benches are completely independent worlds. This
causes a lot of effort by rewriting test procedures when switching between real
and virtual integration tools.
The investigations done at AD&S show
that virtualization of the test environment and distribution of test services
combined with automatic test case generation is a promising starting point
towards highly parallel testing of complex test scenarios using distributed
virtual test environments. Goal is the portability of test procedures between
compatible real and virtual integration platforms.
The method for an automated test process
for testing a distributed complex environment starting with a test model
implementing a test variant management, the test case and test script generation
and the test execution is shown in the presentation.
Similar challenges for shorter, faster
and better testing strategies and test coverage also show up in a lot of
different industrial sectors like cars, trains and other complex systems.
Fully automation of the system
integration process based on modelling of test scenarios and use cases,
automated test case generation, virtualization of the test process and
distribution of the test execution to test services on distributed virtual test
platforms is the key enabler for handling of more and more complex test
scenarios and system variants.
Jürgen Großmann and Dorian Knoblauch. Fuzz Testing ITS
Abstract: Intelligent
transport systems (ITS) can make an important and innovative contribution to
efficient, cleaner and safer mobility. However, safety and efficiency are
directly related to the quality and well-functioning of the underlying technical
infrastructure. European standardization organizations like ETSI have taken
major steps to support the take of ITS through standardization and testing
activities. While currently ETSI provides support for conformance and
interoperability testing, a systematic robustness and security testing approach
is missing. To overcome this issue, we have integrated our Fuzz Testing Library
Fuzzino in the ETSI ITS Conformance Test Suite provided by Spirent.
Fuzz testing is a well-known, effective
and widely accepted approach to identify and locate robustness and security
related weaknesses and vulnerabilities in software-based system. Fuzz testing is
about systematically injecting invalid or unexpected input data to a system
under test. That way, security-relevant vulnerabilities may be detected when the
system under test processes such data instead of rejecting it. Fuzzino
(https://github.com/fraunhoferfokus/Fuzzino) is a library that supports the
generation of test data for fuzz testing. It provides a set of data generation
heuristics that target known weaknesses (e.g. integer or buffer overflows) and
allows for finding new weaknesses by randomly modifying test data.
Building our approach on top of the
existing ITS Conformance Test Suite serves us with the required data type
definitions, adapters and codecs and thus allows us to apply fuzz tests on all
layers of the ETSI ITS communication stack. Test data generation is done on
basis of the existing ASN.1 specifications included in the ETSI standards and
the ITS Conformance Test Suite. The PDU data templates that are used for the
conformance tests are taken as seeds for the fuzz data generation. Due to a
previous model extraction the fuzz data generator is fully aware of the
specifications and type boundaries of each designated field of the PDU. This
Information is used during the mutational fuzzing process. For test
specification and execution, we made use of Spirent’s TTworkbench and thus rely
on an industry grade, standardized and highly flexible TTCN-3 based test
automation system. Our fuzz testing approach covers communication scenarios at
all layers of the ITS architecture and aims to systematically generate
communication data in such a way that they are still accepted by the SUT but
stress and violate the boundaries of specification.
We have successfully evaluated our
approach with different ITS devices from different vendors. We have been able to
find vulnerabilities that crash the communication stack at least at on of the
devices and applied a number of optimization strategies to minimize the number
of generated test data and thus test cases. In our presentation we will outline
our fuzz testing approach, we will describe how we have integrated the approach
in the ETSI ITS Conformance Test Suite, and provide an overview on different
fuzz testing strategies and their implication with respect to test suite size,
test execution time and their capabilities to find vulnerabilities. Finally, we
will provide an outlook on how our ITS Robustness and Security Test Suite can be
used as part of industry grade security assessment processes and during ETSI
events like the ETSI ITS Plug Test series.
Wojciech Tanski and Tomasz Lewczyk. Testing untestable – best practices in
developing testable application.
Abstract: The
presentation is based on a real case from one project where testabiliy of the
product was not a top priority for developers a couple of years ago. Now, with
growing complexity of application, manual testing is not enough to ensure good
quality. Forgetting about testability years ago resulted in many issues during
test automation now and increased its cost significantly. In addition to higher
cost of test automation it causes also lower stability what can be avoided when
certain rules are followed. Regarding novelty – you can easily find in internet
articles about creating software that is testable from unit tests perspective
but as for now no one talks about what makes application testable from
acceptance tests perspective. We will show some lessons learned in regards to
removing obstacles by using some workarounds in test automation area where
application was written in a way it was hard to write automated acceptance
tests. Anyone will be able to introduce those best practices in his/her
organization. Presentation will be based mainly on testing web applications
using Selenium with some examples on desktop applications and a OCR application
framework - Sikulix - as a “last chance” tool that supports desktop and web
applications testing. Every attendee will be able to apply those best practices
in his/her projects. Any organization can benefit from our approach as it is
universal method for showing how to properly create software so it is more
efficient for automated testers. The key-takeaway is the knowledge on how to
better collaborate between testers and developers to achieve faster and cheaper
high quality software. Last but not least our motto: “not taking into
consideration the testability of the product does not mean it cannot be tested
in an automatic way, but the cost of it will be high so this topic should not be
neglected since the very beginning of the project”.
Patrick Harms and Jens Grabowski. Experiences
with Automated Field Usability Testing Using Generated Task
Models
Abstract: Web
portals are the key communication channels for most businesses today. They can
range from simple representation of a company, via online shops, to integrated
platforms as a service. The latter ones are usually hosted by the company for
its customers to provide certain functionality, such as an issue tracking
system. As business changes daily, such web portals need to adapt flexibly. This
ranges from smaller changes of a certain aspect up to a full relaunch of a
website.
In this required dynamicity, website
managers seldom have sufficient time for applying usability engineering
methodologies, such as user testing or expert-oriented evaluation. Instead,
changes are done and rolled out as fast as possible and directly to users. In
case a change causes usability problems, these may not show up directly. Instead
only in the long run they may represent as decreased conversion rates,
disappointed users, or more work for the help desk. In such situations, it is
hard or even impossible to determine, which of the previous changes may have
caused the issues.
In our work, we developed a methodology
for model-based, automated usability engineering of software including websites
and desktop applications. Herein, we record the actions that users perform on
the level of mouse clicks, text entries, and even individual key presses. From
the recorded data, we generate a model representing the typical tasks users
perform with the software. In addition to our analyses, these models may serve
as input for usage-based test case generation. Afterwards, we analyse the
recorded data and the model for known usability smells. These are patterns of
user behaviour that indicate a potential usability issue. The smells have a
direct reference to the involved parts and elements of a website, a description
of the potential usability issue, and a proposal for its removal. We validated
our approach in three case studies and showed that it is capable of providing
helpful results.
In the presentation, we plan to briefly
describe our approach and show some example results. In addition, we will
describe the intended usage of our approach for a company’s web portal so that a
continuous measurement and assessment of the portal’s usability is done.
Depending on the average number of users per day, first representative analysis
results can be available in short term in each iteration cycle of the website.
In addition, we will present our work in progress focussing on a cloud platform
as a service solution. This platform allows users of our approach and our
tooling to use a preinstalled and preconfigured infrastructure to perform
analyses with just one click. We will also show how easily a recording of a
website can be configured using our tooling with state of the art content
management systems. We hope to get into fruitful discussion with potential users
of our approach resulting in valuable feedback.
Martin Gijsen. Forget silver bullets and be context-driven
Abstract: While
situations may look the same, they rarely are. So when considering how to
approach test automation, it is not useful to look for a silver bullet solution
or even accept one when offered. An approach that will result in long lasting
test automation success depends on the context, the situation. The questions to
ask relate to the PuPPET areas (People, Processes, Policies, Environment
and Technologies).
Real life examples from personal
experience will show how the answers and the resulting test automation approach
can (and normally should!) differ from one project to the next.
Gaspar Nagy. Property
Based BDD Examples
Abstract: BDD
(Behavior-driven development) is a software development process focusing on
active collaboration, which illustrates and automates the requirements using key
examples of the problem domain. In BDD the formalized examples use a natural
language-based DSL driven by the Given/When/Then keywords.
At the same time, property-based testing
(PBT) uses abstract (mathematical) formulas to declare expectations for the
output values given some constraints on the input. The PBT tools try to disproof
that the application fulfills these requirements by taking samples from the
valid input value space.
The experience shows that for
understanding and properly implementing the requirements, the team has to
understand the requirements as a set of abstract rules, where collecting key
examples can help a lot.
BDD is strong at managing and automating
these key examples and PBT is strong at defining and automating the rules, so
the question should arise: can these two methods somehow support each-other? Can
we increase the value of BDD by formalizing and testing the rules? Can we
increase the value of PBT by a better communication of the rules and constraints
we have?
This session shows an experiment of
combining BDD and PBT. It provides examples of how the specification could look
like in this combined world and what it could be used for. And there is a
working, open-source prototype too!
Abbas Ahmad, Elizabeta Fourneret, Bruno Legeard, Franck Le Gall, Naum Spaseski, Elemer Lelik and György Réthy. MBT to
TTCN-3 tool chain: The oneM2M experience
Abstract: The
Internet of Things (IoT) has increased its footprint becoming globally a 'must
have' for today's most innovative companies. Applications extend to multitude of
domains, such as smart cities, healthcare, logistics, manufacturing, etc.
Gartner Group estimates an increase up to 21 billion connected things by 2020.
More and more platforms are being developed to respond to this fast growing
demand. To manage the heterogeneity of the things and the data streams over
large scale and secured deployments, IoT and data platforms are becoming a
central part of the IoT systems. To respond to this fast growing demand we see
more and more platforms being developed, requiring systematic security and
functional testing. Our solution proposes a Model-Based testing approach to
generate and execute TTCN-3 code.
Teemu Kanstrén, Jussi Liikka and Jukka Mäkelä. Testing IoT services and devices
in a 5G test network
Abstract: In
UCAAT 2015 we presented the overall architecture of our 5G test network and the
testing technologies deployed on top of it. In this presentation we describe the
first concrete test cases executed in this environment, the new techniques we
developed to make these possible, and the lessons learned from these. We also
briefly address some of the unanswered questions by participants from UCAAT 2015
such as the scope of our virtualized EPC (evolved packet core) and how we use
different (virtualized) monitoring tools to capture overall monitoring data at
global network level.
Ting Miao, Nak-Myoung Sung,
Jaeyoung Hwang, Naum Spaseski, György Réthy, Elemer Lelik, Jaeseung Song and
Jaeho Kim. Development of an Open-Sourced Conformance Testing Tool – oneM2M IoT
Tester
Abstract: IoT
server platforms are required to be implemented to support communications with
diverse devices supporting different type of protocols and undertake a huge
amount of data storing, processing and retrieving tasks in real-time manner,
which in turn exposes new challenges for testing IoT server platforms in terms
of supporting multiple protocols as well as testing complexity, speed,
efficiency and cost etc. In addition, when the conformance testing is taken into
consideration for the IoT platforms, it would be a huge work and takes long time
if all the required functionalities of a standard are manually tested. To
overcomes these testing challenges, we propose to apply the TTCN-3 test language
to enable the automation testing, reuse the open-sourced Eclipse Titan
environment, and design a system adapter associated with a codec for HTTP
protocol to enable the communications between the test system and the tested IoT
platform. In practice, we extended the Eclipse Titan by implemented a system and
codec for oneM2M HTTP binding protocol and it turns out to be a conformance
testing tool, named oneM2MTester. The performance of the oneM2MTester has been
evaluated in the 2nd oneM2M Interoperability Event held in May, 2016.
Ksenia Vecherinina and Martti Käärik. Web test automation using TTCN-3 and
MBT framework
Abstract: Automated
testing of web sites is by no means an unknown topic for anyone and many
languages have been made available for that purpose. However, the simplicity of
most commonly used testing languages such as Cucumber and Robot quickly becomes
a hindrance when you need to work with large data structures. This is where
TTCN-3 come in.
This presentation introduces a solution
that enables the use of TTCN-3 language and tools for testing web applications.
The implementation has been in use for several years in the industry and a case
study is presented together with the technical solution.
Based on the experience, the
presentation points out the advantages of using TTCN-3 and MBT in the testing
and development process. It also mentions the aspects of web testing that still
need improvement.
Emmanuel Gaudin and Mihal Brumbulli. Test
cases to find the best architecture in terms of performance
Abstract: When
performance requirements are defined, they are always related to a set of
scenarios that serve as a basis for analysis. The scenarios are derived from the
high level requirements of the system and are actually validation test cases.
That means validation test cases are useful to help defining the architecture of
a system before the development has started.
The presentation will show how to
estimate the best architecture based on an abstract executable model containing
the performance estimations, a set of possible architectures, and a set of
typical test cases.
Abderrazek Boufahja, Eric Poiseau and Alain Ribault. Model Based Testing and
Coverage of XML Requirements
Abstract: The
use of XML based standards is growing nowadays, especially with the adoption of
RESTful and SOAP communication layers. The conformity of messages gains in
criticality, especially for complex transactions and critical domains
(healthcare exchange, ISOBUS XML transactions, security, etc). The
interoperability between systems highly depends on the conformity to standards
of the received messages. The validation of XML documents based on standards
specifications is a complex task especially for complex XML structures. Some
validation tools exist, like schematrons validation, however, based on the IHE
Europe experience with these tools, they have many drawbacks: complex to use and
to implement, difficult maintainability, and a slow time of execution. Under the
IHE Europe and Kereval health lab, we developed for several years a new
model-based methodology to validate XML requirements: Gazelle ObjectsChecker.
This methodology is a combination of multiple technologies from MBT and
requirements analysis: OCL, UML, DresdenOCL, Topcased, TAML, and Acceleo
templates technology. The aim of this methodology is to simplify XML
requirements checking, to facilitate the maintenance of the tools created, and
to deal with the weakness of schematrons.
The inputs for Gazelle ObjectsChecker
are UML class models defined using specific stereotypes, and containing OCL
constraints. These constraints describe the requirements coming from XML based
standards. The output of this tool is a set of generated JAVA classes allowing
easily validating XML documents. We generate also documentation and a testing
suite for all the constraints.
This new methodology has reached now a
high level of stability and usefulness. It has been deployed in dozens of
environments and healthcare infrastructures, more than 100 validators were
created using this methodology, and they are especially used by IHE Europe to
organize European interoperability testing sessions (or what we call Connec, and
accredited testing sessions. Currently, dozens of thousands of XML documents
were validated based on this methodology. The use of this methodology has proved
its strengthen regarding requirements coverage, but also it proved that this
methodology is easy to maintain and faster than other validation tools.
Ceren Şahin Gebizli, Hasan Sozer and Ali Ozer Ercan. Successive
Refinement of Models for Model-Based Testing to
Increase System Test Effectiveness
Abstract: Model-based
testing is used for automatically generating test cases based on models of the
system under test. The effectiveness of system tests depends on the contents of
these models. Therefore, we introduce a novel three-step model refinement
approach. We represent system models in the form of Markov Chains. First, we
update state transition probabilities in these models based on usage profile.
Second, we update the resulting models based on fault likelihood that is
estimated with a static analysis of the source code. Third, we update these
models based on error likelihood that is estimated with dynamic analysis. We
generate and execute test cases after each refinement step. We applied our
approach in the context of an industrial case study for model-based testing of a
Smart TV system. We observed promising results, in which new faults were
revealed after each refinement.
Lajos Cseppentő and Zoltán Micskei. Evaluating
code-based test input generator tools
Abstract: In
recent years several tools have been developed to automatically select relevant
test inputs from the source code of the system under test. However, each of
these tools has different advantages, and there is little detailed feedback
available on the actual capabilities of the various tools. In order to evaluate
test input generators we collected a representative set of programming language
concepts that should be handled by the tools, and mapped these core concepts and
challenging features like handling the environment or multi-threading to 300+63
code snippets respectively. These snippets would serve as inputs for the tools.
We created an automated framework to execute and evaluate these snippets, and
performed experiments on five Java and one .NET-based tools using symbolic
execution, search-based and random techniques. The test suites’ coverage, size,
generation time and mutation score were compared. The results highlight the
strengths and weaknesses of each tool and approach, and identify hard code parts
that are difficult to tackle for most of the tools. We hope that our research
could serve as actionable feedback to tool developers and help practitioners
assess the readiness of test input generation.
Test automation is intended to increase speed and accuracy of information about
the SUT partly by allowing engineers to improve the speed and usefulness of
their communications. The best possible interfaces and user experience for the
person testing are required to support this otherwise the use of automation will
decrease rather than increase velocity of projects. If the speed and accuracy of
test information provided to teams is lowered, with poor test reporting and
inaccurate decision making, engineers and managers will become frustrated. It
may even lead to disaster.
Good automation tools will help us make good decisions about the SUT and
maximise the value of the limited time we have to deliver software products to
market. Poor automation tools will delay decision making, increase the
likelihood of errors of judgement, and they frustrate both engineers and
managers.
Current practices (agile, build pipelines, devops) arise from a need to address
delivery speed and accuracy as well as engineering quality. But automating the
tests and then forcing people to spend time inaccurately and slowly interpreting
the outcomes simply is not cost effective or helpful. There are many examples of
poor interfaces and tools contributing to, or even forcing, humans to make bad,
even fatal decisions. Examples such as the London Ambulance Dispatch system
(http://bit.ly/1tr2TJZ), and the EU Farm Payments online application system
(http://bbc.in/1xEoUE3) show us that poor interfaces can be time wasting,
expensive and dangerous.
This matters for test automation because, although automation tools are written
and serviced by engineers, the people who use the automation can be
non-technical for example user acceptance representative, product owners,
business sponsors, managers or end users. Does the information they get from the
automation and provide to engineers improve in speed, accuracy and usefulness as
a result of the automation or not? What will maximise our ability to get the
most from the test automation? What will maximise the accuracy and usefulness of
the information provided to engineers, managers and others?
There is a need for TX for Test Automation: that is, the Test Experience for
those people who will request, design, and review the results of the automated
tests and monitoring. This requires information design and delivery (arguably
the purpose of our industry). Attention to detail “front of house” for the UX
for customers and end users can be extended behind the scenes to TX for the
engineers, benefiting all. TX can be improved, by consideration of the UX for
the automation tool and the tests so that methods and lessons from User
Experience Design (UXD) and User Experience Testing (UXT) may be applied to test
automation.
Isabel will consider human factors (as the engineers are human too) as well as
the support of improved decision making around quality, speed and accuracy of
responses to issues.
Three key points:
- Test automation requires consideration of the UX for the tool and the
tests;
- People who use automation might not always be technical but they are
always human;
- UXD and UXT for test automation supports improved decision making and
quality.
TUTORIAL Elizabeta Fourneret and Bruno Legeard. Security Testing using Models
and Test Patterns
Abstract: The tutorial will present an approach and best-practices on
model-based security testing that have reached maturity during the last 6 years.
The approach is based on test patterns and has been integrated with an
industry-strength model-based testing (MBT) tool-chain.
The tutorial will further present examples on the usage of models and test
patterns for security testing in the context of security components or
middleware software, such as Hardware Security Modules (HSM) or IoT platforms,
respectively. In addition, our experience has resulted in a positive impact on
test automation and fault detection effectiveness. Thus, we will further discuss
the lessons learnt and the key messages on applying MBT using test patterns in
the industry.
TUTORIAL Philip Makedonski, Andreas Ulrich, Martti Käärik, Gusztáv Adamis, Finn
Kristoffersen and Xavier Zeitoun. Testing and domain-specific modelling with
TDL
Abstract: The
tutorial provides an introduction to the ETSI test description language TDL. TDL
fills the gap between high-level test purpose descriptions, which are often
given in a natural language, and executable, concrete test cases. While concrete
test cases realize the behaviour of the tester for a given test purpose, TDL
provides the user with a language to specify scenarios of interactions between
tester and SUT, which detail a test purpose description sufficiently to enable
partially automated translation to concrete test cases.
TUTORIAL Isabel Evans. Human factors
for test automation and industrialisation
Abstract: Although
this conference is about automation, people are at the heart of what is to be
achieved by that push to industrialisation and tooling. People in teams are
making the change from manual to automated testing and therefore factors of
attitude to change, teamwork, motivation and communication are going to be very
important. If automation projects are to succeed, we also need to consider human
factors required for success. Faced with industrialisation, people exhibit fear,
disbelief and denial. Evidence from other disciplines show us two types of issue
arising from human interaction with industrialisation: reluctance to lose skills
and over-reliance on the automation. Drawing on practical experience and
research, this tutorial provides a much wider view of the human aspects of
automation than is usual for the industry, and novel combinations of ideas such
as TX for the UXD of automation, and adoption of studies from non-technical
disciplines to help us understand how the human and the automation interact.
Delegates will be provided with an opportunity to identify and discuss problems
and potential solutions to human factor problems around implementation of
industrialised automation, and a number of practical ways to address teamwork
and human problems in projects.
The methods presented are applicable to
people in all forms of endeavour where change and specifically a move to
automation/industrialisation is intended.
key points:
1. Implementing automation and
industrialization involves human factors of teamwork and beyond teamwork;
2. Other disciplines and industries have
lessons we should apply to the industrialisation of our own industry;
3. Models exist to help us understand
how to work with rather than against people in and affected by our projects.
TUTORIAL
Julian Harty. Using Mobile Analytics to Improve Testing and Development Practice
Abstract: Current
practices for testing mobile apps are limited and flawed. Recent research
indicates mobile analytics can augment and enhance both human-oriented and
automated testing of mobile apps. This tutorial will help researchers learn
about current state of the art and state of practice in industry and help
practitioners discover practical ways to augment, adapt and refine their testing
of mobile apps to make their work more fulfilling, with less waste, and also
ultimately deliver improved releases of the apps they are responsible for.