SPSToolbox - User Manual Sándor Kolumbán May 15, 2013

Contents 1

Introduction

2

2

Short description of the theory

2

3

Using the toolbox 3.1 GenerateSPSSetup . . . . . . . . . . . . . 3.2 IsModelPartOfSPSConfidenceSet . . . . . 3.3 IsRegressionParamPartOfSPSConfidenceSet 3.4 CalculateBJGradient . . . . . . . . . . . . 3.5 CalculateBJNoiseRealization . . . . . . . . 3.6 DetermineRank . . . . . . . . . . . . . . . 3.7 GenerateTestRegion . . . . . . . . . . . . . 3.8 SimulateBJSample . . . . . . . . . . . . . 3.9 SimulateSystem . . . . . . . . . . . . . . .

4

5

Examples 4.1 Linear regression . . . . . . . . . 4.1.1 testLinRegMembership.m 4.1.2 showConfidenceRegion.m 4.1.3 testLinRegConfidence.m . 4.2 LTI systems . . . . . . . . . . . . 4.2.1 testLTIMembership.m . . 4.2.2 showConfidenceRegion.m 4.2.3 testLTIConfidence.m . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

Deployment

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

4 4 4 5 5 6 6 6 7 7

. . . . . . . .

7 7 7 7 8 9 9 9 9 11

Versions Date 30. 04. 2013

. . . . . . . . .

Author Kolumbán

Contribution Initial version

1

1

Introduction

This paper briefly introduces a hypothesis testing procedure called sign perturbed sums (SPS) and describes the use of the Matlab Toolbox that implements it. The method was developed in a series of papers [1, 2, 3] and it allows hypothesis testing of parameters in case of linear regression problems and in case of LTI Box Jenkins models ([4]). The important features of the method are that the only assumption about the noise or error values is that they have symmetric distribution about zero and that no asymptotic theory is used. This means that the confidence of the hypothesis test is rigorous. For instruction on installation and licensing see Section 5. The structure of the manual is as follows. Section 2 illustrates the principles of the algorithm without going into details on a toy example. More details can be found in the cited references. The methods provided by the toolbox and their usage is presented in Section 3 and examples are given in Section 4. If you use the toolbox in published work, please cite as @INPROCEEDINGS{kolumban2013aacs, author = {S\’andor Kolumb\’an}, title = {SPS Toolbox for Matlab}, booktitle = {Proceedings of the Automation and Applied Computer Science Workshop 2013 (AACS’13)}, year = {2013}, month = jun, isbn = {978-963-313-078-0} }

2

Short description of the theory

In order to introduce the terminology, this section briefly presents the SPS method on a toy example. We show the steps of the method here and illustrate them on the simple one parameter problem yk = θ0 + nk

k = 1, . . . , N

Some of these steps are redundant in case of this toy example but in case of linear dynamical systems parameter estimation they are needed. We want to create a confidence test for a model parameter θ on confidence level 1 − q/m. The steps of the algorithm are the following. 1. m − 1 random sign sequences of length N and a random permutation of length m should be generated. α(i, k) = 2 round(RAND) − 1

i = 1, . . . , m − 1

k = 1, . . . , N

α(0, k) = 1 is the first sign sequence and it is fixed. ˆk (θ) corresponding to the data and the tested model. 2. Compute the noise sequence N ˆk (θ) = yk − θ N ˆk (θ) 3. Create perturbed noise sequences based on N ˆ (i) (θ) = α(i, k)N ˆk (θ) = α(i, k)(yk − θ) N k 2

4. Using the perturbed noise sequences, create perturbed versions of the measurements (i) ˆk (θ) = θ + α(i, k)(yk − θ) Yˆk (θ) = θ + N

Quadratic cost function is used for the estimation of the unknown parameter. The estimation is given by the solution of the normal equation to this. For each noise realization there is a different cost function and a different normal equation. It is important that from this point on there are m independent identification problems that are (i) considered. These are indexed with i = 0, . . . , m − 1 and the corresponding measurements Yˆk (θ) are considered as constant given values. Their dependence on θ is made explicit because their value depends on it but in calculating the cost function values J (i) they are thought of as constants. J (i) =

N X

(i)

(Yˆk (θ) − θ)2

k=1

5. Calculate the gradient of the prediction errors. In this simple example these are (i)

ψk = −1 6. Compute the perturbed covariance estimate Ψ(i) =

N 1 X (i) h (i) iT =1 ψk ψk N k=1

7. Calculate the normal equation of the cost function N

  X ∂J (i) (i) = −2 Yˆk (θ) − θ = 0 ∂θ k=1

8. The Si values are the gradient vectors of the cost function at θ in the different identification scenarios. Si = 2N θ −

N X k=1

(i) 2Yˆk (θ) = 2N θ −

N X

2 [θ + α(i, k)(yk − θ)] = −

N X

2α(i, k)(yk − θ)

k=1

k=1

9. Create the different Zi values which measure the length of the gradient weighted with the inverse of the covariance matrix h i−1 Zi = SiT Ψ(i) Si = Si2 10. Order the different Zi values in descending order. In case of ties use the random permutation generated at the beginning the make the ordering explicit. 11. If Z0 is between the m largest Z value, then the model θ is no part of the confidence set. Since the LS estimate of the original problem will make Z0 = 0, it will always be inside the set. The basic underlying idea is that m different identification problems are created. From the perspective of θ0 these are identically distributed as the original problem and they are also independent from each other (showing this is a bit more technical). We calculate the gradient of the objective function of the considered model θ on each identification problem (Si ) and measure their length Zi . We say that the selected model is pretty good if it is closer to solving the original problem than the other ones. As we go to infinity with θ it is easy to see in this simple case that Z0 will outgrow every other Z value as the coefficient of θ is the largest here in absolute value. 3

3

Using the toolbox

This section describes the functionality of methods belonging to the toolbox. The main functions of the toolbox are • GenerateSPSSetup: generates the random quantities required by the method, containing the random signs and the random permutation to resolve ties. • IsModelPartOfSPSConfidenceSet: examines if a given Box-Jenkins model belongs to the confidence set defined by the given SPS setup and the measurement data. • IsRegressionParamPartOfSPSConfidenceSet: examines if a given parameter belongs to the confidence set defined by the given SPS setup and the measurement data in the linear regression case.

3.1

GenerateSPSSetup

A confidence set determined by the SPS method is characterized by the given measurement data, the random signs and the random permutation to resolve ties. The random signs and permutations are encapsulated in a Matlab structure that is generated by the function GenerateSPSSetup. [sps] = GenerateSPSSetup(q, m, N) Input parameters: • q, m: positive integers for a confidence level 1 − q/m. • N: the number of sample points. Output parameters: • sps: A structure containing an N × m matrix with the random sign sequences (sps.Signs). The first column is the all-one column. The random permutation for tie resolution (sps.TieOrder) and the values related to the confidence level are also stored (sps.q, sps.m, sps.Confidence).

3.2

IsModelPartOfSPSConfidenceSet

This routine checks if a given Box-Jenkins model is part of a confidence set defined by an SPS setup and the data or not. [inconf] = IsModelPartOfSPSConfidenceSet(model, sps, Y, U) Input parameters: • model: a Matlab structure describing a Box-Jenkins model of the form y[k] =

C(q −1 ) B(q −1 ) u[k] + e[k] A(q −1 )F (q −1 ) A(q −1 )D(q −1 )

The coefficients of the polynomials are put as row vectors into a Matlab structure. The vectors should start with their first non-zero elements. A, F , D and C should be monic (first element 1), deg(AF ) ≥ deg(B), deg(AD) = deg(C) and C should be a stable polynomial. • sps: an SPS setup generated by GenerateSPSSetup. 4

• Y, U: column vectors of the same length containing the measured outputs and inputs respectively. Output parameters: • inconf: Boolean value showing whether the given model is inside the confidence set or not. • Z: The vector containing the Zi values that was used to determine the rank of Z0 .

3.3

IsRegressionParamPartOfSPSConfidenceSet

This routine checks if a given parameter vector is part of a confidence set defined by an SPS setup and the data or not. It is assumed that the data is generated from the model Y = XT θ + E [inconf] = IsRegressionParamPartOfSPSConfidenceSet(theta, sps, Y, X) Input parameters: • theta: a column vector of length nθ containing the coefficients of the linear model • sps: an SPS setup generated by GenerateSPSSetup. • Y: column vector of length N containing the measured outputs. • X: an nθ -by-N matrix containing the regressor vectors as columns. Output parameters: • inconf: Boolean value showing whether the given model is inside the confidence set or not. • Z: The vector containing the Zi values that was used to determine the rank of Z0 .

3.4

CalculateBJGradient

This routine calculates the gradient of the prediction errors for a given model and data. [Psi] = CalculateBJGradient(model, Y, U) Input parameters: • model: just as in the case of IsModelPartOfSPSConfidenceSet • Y, U: column vectors of the same length containing the measured outputs and inputs respectively. Output parameters: • Psi: an nθ × N matrix containing the gradients. Each column k is defined as

5

∂e[k] ∂θ .

3.5

CalculateBJNoiseRealization

This routine calculates the noise sequence needed to generate the given measurements using the given model. [N] = CalculateBJNoiseRealization(model, Y, U) Input parameters: • model: just as in the case of IsModelPartOfSPSConfidenceSet • Y, U: column vectors of the same length containing the measured outputs and inputs respectively. Output parameters: • N: column vector with the same length as the input or the output.

3.6

DetermineRank

This routine is used to order count the number of values in a vector Z that are greater that the first item. [r] = DetermineRank(Z, perm) Input parameters: • Z: row vector of length m. • perm: a permutation of numbers 1, . . . , m Output parameters: • r: the number of values in Z that are greater than Z(1). If two values are equal in the vector, then their order is determined by the corresponding items in the given permutation.

3.7

GenerateTestRegion

In order to visualize a two dimensional confidence region, a point grid needs to be defined. This routine generates points inside a convex polygon if its vertices are given. [TR] = GenerateTestRegion(upperPoints, lowerPoints, stepsize) Input parameters: • upperPoints: coordinates of the vertices of the upper edges of the polygon as columns of a matrix. • lowerPoints: coordinates of the vertices of the lower edges of the polygon as columns of a matrix. • stepsize: column vectors of length two containing the resolution of the sample grid along the respective dimensions. Output parameters: • TR: a matrix with two rows. Each column corresponding to a point inside the given polygon.

6

3.8

SimulateBJSample

This routine calculates the output of a Box-Jenkins model with a given noise realization and input. [Y] = SimulateBJSample(model, U, N) Input parameters: • model: just as in the case of IsModelPartOfSPSConfidenceSet • U, N: column vectors of the same length containing the inputs and the noise samples respectively. Output parameters: • Y: column vector containing the output of the system.

3.9

SimulateSystem

This routine calculates the output of a system in the form

B(q −1 ) A(q −1 ) .

[Y] = SimulateSystem(B, A, U) Input parameters: • B, A: row vectors containing the coefficients of the polynomials B and A. They should have the same length. Pad with zeros if needed. • U: column vector containing the input values. Output parameters: • Y: column vector containing the output of the system.

4

Examples

Along with the code of the toolbox, some example scripts are also attached to illustrate the usage of the routines. These can be found in the "Samples" folder. Samples for the linear regression hypothesis testing are in the "Linear regression" subfolder, whereas samples related to Box-Jenkins models are in the "LTI systems" subfolder.

4.1

Linear regression

4.1.1

testLinRegMembership.m

Least squares estimates always belong to the SPS confidence set as in their case Z0 = 0. This script illustrates this by calculating the LS estimate and checking its membership. 4.1.2

showConfidenceRegion.m

The structure of the confidence sets corresponding to a given SPS setup and data can be complex. This example defines a fine grained grid around the LS estimate of a selected two parameter model and evaluates the membership on every grid point. The sample code demonstrates how to generate and plot confidence regions in two dimensions, the result is given in Figure 1. 7

Figure 1: The 0.85 confidence region for a two parameter linear regression problem. 4.1.3

testLinRegConfidence.m

This script is written to demonstrate that the SPS method indeed generates confidence regions with the required confidence level. A high number of membership queries are carried out for the nominal model on different confidence levels using different random noise values and SPS setups in each query. The membership answers are recorded and saved to the disk. The second part of the script visualizes the probability with which the nominal model was accepted to be in the confidence region or not. After 15000 queries on each confidence level, the results are given in Figure 2.

Figure 2: The relative frequency of accepting the nominal model at given confidence levels (red) and the prescribed values for this (blue).

8

4.2

LTI systems

4.2.1

testLTIMembership.m

Prediction error estimates always belong to the SPS confidence set as in their case Z0 = 0. This script illustrates this by calculating the PEM estimate and checking its membership. The PEM estimate is calculated using the pem method in the Matlab System Identification Toolbox. 4.2.2

showConfidenceRegion.m

The structure of the confidence sets corresponding to a given SPS setup and data can be complex. This examples defines a fine grained grid around the PEM estimate of a selected two parameter model and evaluates the membership on every grid point. The sample code demonstrates how to generate and plot parts of the confidence regions in two dimensions, the result is given on Figure 3.

Figure 3: Part of the 0.85 confidence region for a first order nominal system and output error noise model.

4.2.3

testLTIConfidence.m

This script is written to demonstrate that the SPS method indeed generates confidence regions with the required confidence level. A high number of membership queries are carried out for the nominal model on different confidence levels using different random noise values and SPS setups in each query. The membership answers are recorded and saved to the disk. The second part of the script visualizes the probability with which the nominal model was accepted to be in the confidence region or not. After 15000 queries on each confidence level, the results are given in Figure 4.

9

Figure 4: The relative frequency of accepting the nominal model at given confidence levels (red) and the prescribed values for this (blue).

10

5

Deployment

SPSToolbox is distributed under the terms of the GNU General Public License. For different licensing options, contact the developers. The code is available on GitHub using the following links: • GitHub repo history: https://github.com/kolixx/SPSToolbox • link for cloning the repo: git://github.com/kolixx/SPSToolbox.git The different releases are contained in the root of the repo as archives containing the date of the release in the file name. Add the folder SPSToolbox to the Matlab path collection using the menu "File/Set Path...". Mind that Matlab should be running with administrative privileges during this operation.

References [1] B. Csáji, M. Campi, and E. Weyer, “Non-asymptotic confidence regions for the least-squares estimate,” in Proceedings of the 16th IFAC Symposium on System Identification (SYSID 2012), pp. 227–232, 2012. [2] B. Csáji, M. Campi, and E. Weyer, “Sign-perturbed sums (sps): A method for constructing exact finite-sample confidence regions for general linear systems,” in Decision and Control (CDC), 2012 IEEE 51st Annual Conference on, pp. 7321–7326, 2012. [3] S. Kolumbán and I. Vajk, “Exploring confidence sets constructed using sign-perturbed sums,” in Decision and Control (CDC), 2013 IEEE 52st Annual Conference on, 2013. [4] L. Ljung, System Identification - Theory for the User, 2nd edition, vol. 2 of PTR Prentice Hall Information and System Sciences Series. 1999.

11

SPSToolbox - User Manual - GitHub

May 15, 2013 - Contents. 1 Introduction .... booktitle = {Proceedings of the Automation and Applied Computer Science Workshop ..... System Sciences Series.

303KB Sizes 8 Downloads 524 Views

Recommend Documents

User Manual - GitHub
Page 1. User Manual. Project Odin. Kyle Erwin. Joshua Cilliers. Jason van Hattum. Dimpho Mahoko. Keegan Ferrett. Page 2. Contents. Contents. Section1 .

GWR4.09 User Manual - GitHub
Starting the program, Exiting the program, and Tab design ...................... 5 ..... GWR4 runs on Windows Vista, Windows 7, 8 and 10 environments with the .

OCEMR: User Manual - GitHub
In order to access the program, OCEMR, find the Firefox tab located to the left of the screen. .... click on the view option next to the patient record with the most ..... entered, the report will appear as a download at the bottom of the popup scree

The User Manual - GitHub
Defined Wireless Networking Experiments 2017 ..... 1.7.3 Encryption. Mininet-WiFi supports all the common wireless security protocols, such as WEP (Wired Equivalent. Privacy), WPA (Wi-Fi Protected Access) and WPA2. ..... mac80211_hwsim practical exam

VFS User Manual - GitHub
wind turbines considering the non-linear effects of the free surface with a two-phase .... the nodes of the computational domain are classified depending on their location with ...... bcs.dat need to be set to any non-defined value such as 100.

User Manual - GitHub
IIS-1. 0x01C2 2000---0x01C2 23FF. 1K. IIS-0. 0x01C2 2400---0x01C2 27FF. 1K ..... IIS 1 CLOCK REGISTER ...... NFC can monitor the status of R/B# signal line.

ZotPad User Manual - GitHub
Settings. 19. Troubleshooting and getting help. 24. Technical information. 27 .... These will be replaced with document previews for PDF files after the files have ...

Sequencer64 User Manual 0.94.4 - GitHub
Feb 3, 2018 - Doxygen output (including a PDF file) provides a developer's reference manual. – Debian packaging was ..... can play music. The items that appear in this tab depend on four things: • What MIDI devices are connected to the computer.

RFBee User Manual v1.1 - GitHub
Aug 27, 2010 - 2. 1.2. Specifications. ... 2. 1.3. Electrical Characteristics . ..... >>Datasheet: http://www.atmel.com/dyn/resources/prod_documents/doc2545.pdf. >>Arduino ... >>Datasheet: http://focus.ti.com/docs/prod/folders/print/cc1101.html.

Design module user manual - GitHub
In the design module objects like buildings can be selected. For each case, measures ... Figure 3, parts of the web application for control and visualizing data. 1.

DuinoCube User Manual by Simon Que - GitHub
delve into the finer details of some system components, such as the FAT file ... importantly, Arduino shields can be designed to stack on top of one another, .... but can be run on Linux using the Wine emulator. http://notepadplusplus.org/ .... Conne

Clam AntiVirus 0.99.1 User Manual - GitHub
1 Introduction. 4 .... 1 Introduction. 6. – HTML. – RTF. – PDF. – Files encrypted with CryptFF and ...... Dynamic Network Services, Inc (http://www.dyndns.org/).

user guide - GitHub
TOOLS AnD EVA ITEMS CAn BE FOUnD In A nEW TAB UnDER SCIEnCE CATEGORy. .... But THE greatest thing above all is KSP community. ... Of course, we still need hard work to improve our mods and we have many other ideas as.

User Guide - GitHub
Requires the query, phrase, or word on its right hand side to not be in the document. [ATTRIBUTE]:. Requires the value of the document attribute describe between the brackets [ ] to equal the value to the right of the colon. Multiword phrases are exp

User Guide - GitHub
2.2 Download and Installation via App Manager . .... Cytoscape/GEXF “app” that allows network diagrams described using the GEXF file format to be imported ...

WDS User Archetypes MASTER.indd - GitHub
government and early in their career. WHY THEY'RE USING THE STANDARDS ... technology stacks and work-flows. USER ARCHETYPE. The reviewer.

Hedgehog User Guide 2.4.0 - GitHub
Hedgehog can be installed to run on a single server with all the components local to that server. .... They are documented separately in the Hedgehog Tools PDF. ... RSSACD. In order to collect the zone-size and load-time statistics a dedicated.

User Manual - EEVblog
written notice of any defect prior to the end of the applicable warranty period. ATL's obligation ... ATL's systems contain ATL's proprietary software in machine—readable form. This ..... To display pre-defined or custom (user-defined) annotation.

User Manual - EEVblog
ATL's systems contain ATL's proprietary software in machine—readable form. This ..... To display pre-defined or custom (user-defined) annotation with the Text A ...

User Manual -
low-tech digital kiosk that is preloaded with digital content and software which is licensed under free and open license. Associated Documentation. The following ...

Sweave User Manual - CiteSeerX
data change and documents the code to reproduce the analysis in the same file ... Many S users are also LATEX users, hence no new software or syntax has to ...

Reference Manual - GitHub
for the simulation of the electron cloud buildup in particle accelerators. 1 Input files .... points of the longitudinal beam profile of sec- ondary beams.

Scop3D – Manual - GitHub
for linux http://genesis.ugent.be/downloadredirect.php?toolname=scop3D-linux for mac http://genesis.ugent.be/downloadredirect.php?toolname=scop3D-mac.

Haxe 3 Manual - GitHub
of programs in Haxe. Each Haxe class has an explicit name, an implied path and zero or more class fields. Here we will focus on the general structure of classes and their relations, while leaving the details of class fields for Class Fields (Chapter