Bayesian statistical model checking with application to Simulink/Stateflow verification

We address the problem of model checking stochastic systems, i.e. checking whether a stochastic system satisfies a certain temporal property with a probability greater (or smaller) than a fixed threshold. In particular, we present a novel Statistical Model Checking (SMC) approach based on Bayesian statistics. We show that our approach is feasible for hybrid systems with stochastic transitions, a generalization of Simulink/Stateflow models. Standard approaches to stochastic (discrete) systems require numerical solutions for large optimization problems and quickly become infeasible with larger state spaces. Generalizations of these techniques to hybrid systems with stochastic effects are even more challenging. The SMC approach was pioneered by Younes and Simmons in the discrete and non-Bayesian case. It solves the verification problem by combining randomized sampling of system traces (which is very efficient for Simulink/Stateflow) with hypothesis testing or estimation. We believe SMC is essential for scaling up to large Stateflow/Simulink models. While the answer to the verification problem is not guaranteed to be correct, we prove that Bayesian SMC can make the probability of giving a wrong answer arbitrarily small. The advantage is that answers can usually be obtained much faster than with standard, exhaustive model checking techniques. We apply our Bayesian SMC approach to a representative example of stochastic discrete-time hybrid system models in Stateflow/Simulink: a fuel control system featuring hybrid behavior and fault tolerance. We show that our technique enables faster verification than state-of-the-art statistical techniques, while retaining the same error bounds. We emphasize that Bayesian SMC is by no means restricted to Stateflow/Simulink models: we have in fact successfully applied it to very large stochastic models from Systems Biology.

@INPROCEEDINGS{DBLP:conf/hybrid/ZulianiPC10,
	pdf = {pub/bayesmcest.pdf},
	slides = {pub/bayesmcest-slides.pdf},
	TR = {DBLP:conf/hybrid/ZulianiPC10:TR},

  author    = {Paolo Zuliani and
               Andr{\'e} Platzer and
               Edmund M. Clarke},
  title     = {Bayesian Statistical Model Checking with
               Application to {Simulink/Stateflow}
               Verification},
  booktitle = {HSCC},
  year      = {2010},
  pages     = {243-252},
  doi       = {10.1145/1755952.1755987},
  editor    = {Karl Henrik Johansson and
               Wang Yi},
  longbooktitle = {Proceedings of the 13th ACM
               International Conference on
               Hybrid Systems: Computation and Control,
               HSCC 2010, Stockholm,
               Sweden, April 12-15, 2010},
  publisher = {ACM},
  isbn      = {978-1-60558-955-8},
  abstract  = {
    We address the problem of model checking stochastic
    systems, i.e. checking whether a stochastic system
    satisfies a certain temporal property with a
    probability greater (or smaller) than a fixed
    threshold. In particular, we present a novel
    Statistical Model Checking (SMC) approach based on
    Bayesian statistics. We show that our approach is
    feasible for hybrid systems with stochastic
    transitions, a generalization of Simulink/Stateflow
    models. Standard approaches to stochastic (discrete)
    systems require numerical solutions for large
    optimization problems and quickly become infeasible
    with larger state spaces. Generalizations of these
    techniques to hybrid systems with stochastic effects
    are even more challenging. The SMC approach was
    pioneered by Younes and Simmons in the discrete and
    non-Bayesian case. It solves the verification problem
    by combining randomized sampling of system traces
    (which is very efficient for Simulink/Stateflow) with
    hypothesis testing or estimation. We believe SMC is
    essential for scaling up to large Stateflow/Simulink
    models. While the answer to the verification problem is
    not guaranteed to be correct, we prove that Bayesian
    SMC can make the probability of giving a wrong answer
    arbitrarily small. The advantage is that answers can
    usually be obtained much faster than with standard,
    exhaustive model checking techniques. We apply our
    Bayesian SMC approach to a representative example of
    stochastic discrete-time hybrid system models in
    Stateflow/Simulink: a fuel control system featuring
    hybrid behavior and fault tolerance. We show that our
    technique enables faster verification than
    state-of-the-art statistical techniques, while
    retaining the same error bounds. We emphasize that
    Bayesian SMC is by no means restricted to
    Stateflow/Simulink models: we have in fact successfully
    applied it to very large stochastic models from Systems
    Biology.}
}```