The Role of Trust in Testing
A conversation with Manas Menon, Lead Systems Engineer.
What role does trust play in the testing process? Why is testing important in building trust?
Testing builds confidence that not only do we say something will work, we have shown that it will work, and we’ve shown that again and again in a specified set of scenarios. Testing provides that ground truth information.
How do you build trust in the testing process itself?
Our goal is to constantly bring the testing process closer to reality. It’s important to communicate its teachings and its limitations constantly to developers and the product team and to be clear about gaps we’ve identified and how we are working to close them, now and in the future. This helps the rest of the company understand which areas of performance are understood and which will require additional effort going forward.
When we think of testing, we think about the results we get from it and what we learn. What role does trust play in learning?
When learning we essentially trust that the tested scenarios are representative of the use case of the actual system. We trust that the failures, mitigation strategies, and environmental properties are sufficiently accurate to be of value.
What is the role of testing systems in helping us learn?
We need systems around testing to learn at efficient rate. Without systems in place you’re essentially testing at random. If you test at random, you’ll eventually find useful pieces of information, but testing targeted scenarios lets you find that information orders of magnitude more quickly than you would otherwise. And build upon that learning to create a product that has maximum impact on mission.
Stay tuned for the next post on the Role of Trust.
In the context of robotics, coordinated exploration refers to the deployment of multi-robot systems in order to explore an environment. These multi-robot systems collectively develop a structural model of that environment by moving through and navigating around it.
A robotic system will acquire more experience the more it operates. This experience translates into learned information about how the actions the robot takes will impact its ability to interact with its environment.
In terms of multi-robot systems, the ability to communicate enables the robots to make coordinated decisions of who should go where and when, and to assess the reason why a particular robot is better suited for the task.
For a robotic system, exploration involves the system making decisions of where it should go in order to acquire information and learn about its environment. Multi-robot exploration applies to systems in which multiple robots are working together.
As a robot explores, intelligence starts to manifest through the system’s ongoing assessment of its observations. Specifically, the robot will assess each observation’s informative value and information gain associated with it.
The concept of resiliency is rooted in psychology. When applied to robotic systems, resilient intelligence refers to the ability of the system to cope with unexpected challenges and adapt to overcome them.
Artificial intelligence is a broad field that encompasses many different aspects of what it means to create an intelligent system. Learning is a subset of intelligence. For robotic systems, learning typically refers to identifying relationships.
At Shield AI, we engineer robotic systems that are able to adapt their behavior in order to augment, extend, mitigate, and support the user. This enables anyone to engage with our systems, having never worked with them, and perform as if they were experts.
Two things are required to build trust in robotic systems: Human understanding of how the system is expected to operate and reliable system performance adhering to expectation.
For a robotic system, trust is about the system engaging as expected, in a consistent manner, time and time again. This concept is not unique to robotics systems; it is true for any engineered system.