REVIEWING SMALL BUSINESS ENVIRONMENTAL MANAGEMENT TRAINING PROGRAMS Janice Redmond Edith Cowan University [email protected] Elizabeth Walker Edith Cowan University [email protected] Alan Coetzer Edith Cowan University [email protected] Summary This paper outlines a study that investigates a gap in academic and industry knowledge; the impact on small businesses of environmental training programs designed specifically for this business cohort. Considerable government and private funds are spent on all types of training in the developed world each year yet little thorough evaluation of this expenditure occurs, particularly in the specific area of environmental management training (Carleton-Hug & Hug, 2010). Small business engagement in environmental management is critical to achieving the Australian government’s environmental targets (e.g. carbon emissions), yet the business’s expertise and knowledge on environmental management issues is lacking (Department of Climate Change, 2010). There has been no systematic review of environmental training programs outside the United States their impact elsewhere is unknown (Carleton-Hug & Hug, 2010). This paper discusses the first stage and as such is restricted to as small number of Western Australian reports.

1

INTRODUCTION This paper outlines a study that sought to investigate a gap in academic and industry knowledge; the impact on small businesses of environmental training programs designed specifically for this business cohort. Considerable government and private funds are spent on all types of training in the developed world each year yet little thorough evaluation of this expenditure occurs, particularly in the specific area of environmental management training (Carleton-Hug & Hug, 2010). With a growing sense of urgency for environmental management to combat climate change, it is important that effective training designs (i.e. those that bring about short and long term environmental improvements) are identified, particularly those delivered to small business. This is because expenditure on small enterprise training in Australia is estimated to be in the region of $40 million per year. Further, small businesses in Australia account for 96% of all businesses, almost half of all employment and one-third of industry value-added1 (Department of Industry Innovation Science and Research, 2011). Small business engagement in environmental management is critical to achieving the Australian government’s environmental targets (e.g. carbon emissions, water usage/protection), yet the business’s expertise and knowledge on environmental management issues is lacking (Department of Climate Change, 2010). As there has been no systematic review of the various environmental training programs outside the United States their impact elsewhere is unknown (Carleton-Hug & Hug, 2010). This study sought to redress this situation in Australia in three consecutive stages by: (1) reviewing final reports available from Western Australia; (2) reviewing other Australian state/territory reports; and finally (3) interviewing program managers about the environmental management training programs for small businesses that they have been involved in over the past 5 years. In particular it seeks to examine the training design and evaluation methods used in the programs so that a clearer understanding of their individual and collective impact can be determined. This paper discusses the first stage and as such is restricted to as small number of Western Australian reports. BACKGROUND TO THE STUDY Training and evaluation Throughout the world governments are contributing billions of dollars to provide assistance and development programs to the business community and in particular the small business community. Despite the high level of expenditure there are a very limited number of evaluations being undertaken into the effectiveness and impact of publically funded programs and whether the money invested does create best value. The decision to spend considerable amounts of money with no rigorous evaluation about either the impact the training has had on measureable objectives such as small business growth and profitability or income tax revenue, may be injudicious, given the ever-competing demand for government funds. Moreover, when programs are subject to evaluation, the criteria used are most often based on immediate ‘success’ via happy sheets rather than long term growth and development, and provide only a snapshot or point-in-time information (OECD, 2002).

1

Industry value added is the measure of the contribution by businesses in each industry to gross domestic product.

2

A brief review of the literature on small business training found there to be a lack of longitudinal studies and the evaluations which had been undertaken were generally program specific around the two broad areas of business start-up and management training development. The review confirmed that there was a lack of comprehensive evaluations of government funded small business development programs (see, for example, the Victorian Auditor-General’s Office (2011) (VAGO) evaluation on the Time to Thrive (TTT) program) which was critical of the evaluation process. Further no evaluations of environmental training programs specifically for small businesses could be found that related to Australia. There was however literature outlining the difficulties in conducting evaluations and these difficulties offer some explanation for the lack of existing, thorough evaluations. Undertaking an effective program evaluation has several difficulties. Firstly, information sought by the different parties involved in program delivery is not always for the same purpose. Funding bodies require evaluations to justify expenditure, whereas businesses require evaluations for cost-benefit purposes (Dionne, 1996). In these situations inconsistency in, or lack of, evaluation can result if no clear guidelines are provided for those undertaking the evaluations, i.e. in knowing how and what to evaluate (Mann & Robertson, 1996). Secondly, it can be easier to conduct evaluations at the completion of program delivery rather than arrange follow up evaluations which can be time consuming and potentially costly (Bramley, 1999; Dionne, 1996). These completion based evaluations are often designed to monitor the program’s delivery and usefulness at their conclusion and are merely individual satisfaction ratings, commonly referred to as ‘happy sheets’. It is also possible that participants report finding the training useful and enjoyable but never put it into action, thus making any impact difficult to measure in any tangible way, (Massey, Tweed, & Lewis, 2003; OECD, 2002), although it is acknowledged that there are many intangible benefits that can be derived from attending training courses. Participant reactions and instant feedback can provide important information for decision making regarding the continuation of training programs and should be included in any evaluation (Mann & Robertson, 1996; Redmond & Walker, 2008), however they are not reliable indicators of other potential outcomes of training, such as participant learning, behaviour change and improvement in business performance (Kirkpatrick, 1996; Steensma & Groeneveld, 2010). Thirdly, with publically funded programs, follow-up evaluations or ongoing support is often not provided, and program impacts on business performance often experience a lag time (Billington, Neeson, & Barrett, 2009). The difficulties these issues create in the evaluation process and the minimal time program delivery agents have available for reporting to funding bodies means that most evaluations will be only a measure of the operational aspects of training delivery, such as program take-up by participants and cost effectiveness as per the original budget, at that particular point in time. According to Henry et. al. (2004) evaluating participant satisfaction should be for this operational purposes only and not used as a substitute for a more thorough evaluation that could include outcome based measures such as growth in sales, staff turnover, and other key performance indicators. Despite the difficulties noted above, evaluations do take place and much has been written about how thorough procedures can be set in place to ensure their outcomes are valid. Two of the key authors in this area are Kirkpatrick (1996) and Storey (2000). Evaluating programs involves two main measures – the satisfaction of participants and the impact on the business (OECD, 2002). Greene (2009) asserts that a distinction needs to be made between what is 3

merely monitoring and what is actually an evaluation of the program. Kirkpatrick’s (1996) model of evaluation is a widely used framework for evaluating management training and development (Bramley, 1999; McMullan, Chrisman, & Vesper, 2001) due to its simplicity and practicality (Kirkpatrick, 1996). However it is recognised that the last two levels of Kirkpatrick’s (1996) model (behaviour evaluation and results evaluation) are rarely used (Hearn, 1988; OECD, 2002), often due to resource constraints, particularly amongst the small business sector, and the difficulty in isolating the impact training had on business performance (Storey & Westhead, 1994). The first three steps of Storey’s (2000) model involve measuring cost effectiveness, participant opinions on delivery and access, and participant views on what difference the training has made. These “happy sheets” often provide the funding body with positive reports about the program. Moving beyond the monitoring of programs to the real evaluation steps involves determining the impact on firm performance compared with firms who have not participated in the training (Greene, 2009; Storey, 2000). However, the obvious time consuming and difficult nature of establishing this type of comparison prior to the program commencing, potentially creates problems when funding is provided for defined periods of time. Ultimately, the best method for program evaluation is relating program outcomes to the stated objectives (McMullan, et al., 2001; Storey, 2000), yet often these objectives are broadly defined making any outcome based evaluation difficult. They also require the evaluator to choose to either make inferences about objectives or not report on whether they were met (McMullan, et al., 2001; Storey, 2000). Storey (2000) goes further to say “a characteristic of governments in all developed countries seem to be, at best, being opaque about the objectives of small business policy” (p. 177). This vagueness by governments in the specification of objectives means that any evaluation which reports on objectives versus outcomes makes it difficult to find a definite link between training and the performance of small business. There is evidence to suggest that training programs are beneficial in the short term, however there is little empirical evidence that it encourages participants to engage in lifelong learning or quantifies the extent to which it transfers to improved business performance (Massey, et al., 2003; Storey & Westhead, 1994). This therefore raises the question about what constitutes improved performance. The measure of business performance covers a range of different measures including profit/sales growth, survival, maximising income, minimising tax burden (OECD, 2002) and for the purposes of this study it is assumed that performance will involve a positive change in environmental behaviour by small businesses. The study will focus on training to improve environmental performance in small business as this is a relatively new area of required competence in small businesses (Condon, 2004; Hillary, 2004; Revell & Blackburn, 2007) but one that is vital to get right. There has been limited evaluation of environmental management training programs to date (Carleton-Hug & Hug, 2010; Keene & Blumstein, 2010) despite considerable and growing government expenditure, so this research is very timely. To address this research gap, this study will seek to answer the question: How effective are the designs and evaluations of recent environmental management training programs delivered to Australian small business?

4

CONCEPTUAL FRAMEWORK FOR THE STUDY The conceptual framework that will help to focus and bound data collection in this study is adapted from Grossman and Salas (2011) and shows the direct link between training inputs and the conditions of transfer of that learning into the workplace. We have amended the model to depict the small business owner and their firm as part of the process and highlighted the three key steps in the process: training design, learning and retention, and generalisation and maintenance. The first two steps have been highlighted in bold on our conceptual framework diagram, as we will focus on how the training was designed and how this assisted learning and retention by the small business owner-managers both immediately (preworkplace evaluation) and when they went back into their work environment (post workplace evaluation). Grossman and Salas (2011) discuss the importance of the training design on conditions of transfer to the workplace and point to the use of behaviour modelling as a method of increasing transfer. Therefore, the study will seek to determine whether elements of behaviour modelling were used within the training design. The third step (generalisation and maintenance) has dotted lines on our conceptual framework diagram as it is our assertion that because most program reports will not include what happened in the workplace after the training (or not in any great detail) due to the training design or evaluation method employed, it will not always be possible to determine the conditions of transfer or in fact whether the training had transferred to the work environment at all. In addition, we have added the work of Kirkpatrick (1996) and Storey (2000) to the amended model as we will use their work as guides in reviewing how thoroughly the environmental training program evaluations were conducted (i.e. thoroughness will be assessed based on evidence of the level/steps taken in the evaluation process). The four levels of evaluation in Kirkpatrick’s (1996) model are outlined below: (1) Reaction evaluation describes how the participants felt about the training experience. (2) Learning evaluation is a measure of an increase in knowledge determined by pre and post training evaluations. (3) Behaviour evaluation determines the extent to which the learning is implemented into work practices. (4) Results evaluation relates to the effect on the business and environment as a result of the training. Kirkpatrick’s model is similar to Storey’s (2000) model which outlines six steps to effectively monitor and evaluate a program’s impact. Storey’s (2000) model is outlined as follows: MONITORING Step1 Take up of schemes Step 2 Recipients opinions Step 3 Recipients views of the difference made by the Assistance EVALUATION Step 4 Comparison of the performance of ‘Assisted’ with ‘Typical’ firms Step 5 Comparison with ‘Match’ firms

5

Step 6

Taking account of selection bias

This model was adopted by the OECD (2007) as a means of establishing a baseline method for evaluating programs delivered to SME’s.

METHODS The study is to be completed in three stages: an analysis of written reports from Western Australia (findings of the analysis are reported here), an analysis of reports from other states/territories, and semi-structured interviews with the environmental management training program managers. As noted previously, a conceptual framework was designed to guide the study and incorporates the work of Grossman and Salas (2011); Storey (2003) and Kirkpatrick (1996) regarding the transfer of training to the workplace and evaluation methodology. The written reports reviewed include funded environmental management training programs for Australian small businesses over the past five years and was limited to those that involve the use and efficiency of water and energy as these are both the most common targets for environmental protection and the two practices that are most relevant in small businesses. The analysis of the documents incorporated the use of a content analytic approach as recommended by Patton (1990). To obtain reports a total of 35 organisations were contacted including 3 Federal and 7 State government departments, 11 local governments; 5 regional councils, 6 Development Commissions and 4 other organisations of mixed origin. From this sample only 4 written reports were available. The reasons given for a lack of reports were that the organisation focussed on schools or householder action rather than small business, or the reports were not 6

for general release and would only available if a successful freedom of information application was submitted. Given the resources required to try and ascertain these additional documents was extensive, the research team decided to not pursue this avenue of enquiry.

FINDINGS Our findings are based on an analysis of four reports. We drew four main conclusions from our experiences of trying to obtain reports. First, that little small business focussed environmental training is occurring in Western Australia. Second, that programs that do occur are often not reported on and may be short term and ad hoc in nature. Third, representatives of several organisations suggested that other organisations were more involved in environmental management and/or small business environmental management than they were, however, when follow up calls were conducted with the ‘other’ organisations this was found not to be the case. Finally, it became clear that environmental officers and environmental program funding sources keep changing and institutional knowledge is not being maintained. The anecdotal evidence suggests that staff at some organisations were unaware that programs had been conducted in the past by their department and of staff that were aware, there was a general reluctance to willingly hand over documents as they believed little had been achieved. Other anecdotal comments voiced a sense of frustration that good programs were not given sufficient time to run and that although much time had gone into the planning of the programs, the implementation was of too short a duration. In addition, feedback from people in these organisations suggested there was a lower priority for environmental programs within the newer funding body. These issues will not be explored further as they are outside the scope of this paper, however they have relevance to the outcomes that are achieved in engaging small business in environmental management. We now move on to evaluating the four reports which are outlined in Table 1. Table 1: Comparison of Reports Funding Duration Objectives

Participants Intervention Design Evaluation data

Report 1

Report 2

Report 3

Report 4

unknown 15 months Energy & Land fill reduction

$AU222,000 21 months Not stated

91 SMEs  Facilitated

150 SMEs  Free energy audits 

$AU40,000 24 months Overall awareness raising 404 SMEs  Facilitated

$AU45,000 24 months Overall awareness raising 120 SMEs  Facilitated







Report 1. Final report Climate actions July 2008

7

Organisations involved: South Metropolitan Regional Council in collaboration with the Cities of Canning, Cockburn, Fremantle and Rockingham and Towns of East Fremantle and Kwinana. Funding support: Federal Government Department of Environment, Water heritage and the Arts. Funding amount: Not reported. Delivery time frame: February 2007 – May 2008. Objectives: This program resulted from a pilot program completed in 2005-6 engaging 36 small businesses and sought to engage 80-100 local centre businesses and provide individualised support to these businesses to identify and implement actions in energy and waste-to-landfill reduction. In addition, the program sought to engage 1,500 surrounding households in energy, water, waste reduction, support the cross-pollination of greenhouse gas abatement ideas and lessons learnt among participants to motivate businesses and residents in taking actions, and assess the effectiveness of the project model with participating businesses and households through verbal and written feedback and measurement where possible. Business Participants: 91 small to medium sized businesses Design: This program tested the Enterprise facilitationTM model. In this model the central role is of a Green Enterprise Facilitator. With the small businesses the facilitator’s role being to engage in frequent communication on site and by telephone, making available energy efficient products, arranging site visits and quotations from service providers to enable businesses to take immediate action to improve energy efficiency. Businesses with outstanding results were invited to an award ceremony and given a framed Certificate of Excellence and other prizes. As the project was also designed to engage residents through an indirect approach of postal invitations that allowed the participating businesses to act as information hub for the households (over 830 engaged) the residents were mailed information sheets including stories about the progress of the participating businesses. Intervention: Free, confidential service to improve energy efficiency via a Green Enterprise Facilitator. Evaluation data: 93% participation and 76% actions response rate. Areas of specific action included: refrigeration (35%), lighting (15%), and recycling (12%). Estimated savings achieved by businesses were: annual saving of almost 540,000 kWh of electricity; annual saving of GHG emissions of 500 tCO2e; total annual financial saving of around $93, 700; and average annual saving of over 6000 kWh in electricity and 5.7 tCO2e in GHG emissions per business. The resident response rate to the mail out was 7% and was considered to be less effective than the 10% response rate in 2005. An estimated outcome of 427 tCO2e in GHG emissions abated per year for participating households was reported. Overall the design of the program was considered effective by the reporting body and this is evidenced in their statement that the program “achieved great success in abating greenhouse emissions in small businesses” (p. x).

8

Report 2. Switched on Business and Industry. Organisations involved: The Cities of Gosnells and Armadale and Serpentine Jarrahdale Shire (as South East Regional Energy Group (SEREG). Launched switch your thinking! Syt! In 2002 and under this brand delivered the Switched on Business and Industry (SOBI) project. Partners were Perth Region NRM. Funding support: Federal government AusIndustry program grant Funding amount: $222,000 Delivery time frame: February 2008- December 2009. Objectives: Not stated in the report. Business Participants: Small and medium enterprises that operated in retail (30%), health and community services (16%), personal services (14%), hospitality (12%), manufacturing (10%), property services (7%), culture and recreation (4%), construction (2.5%), wholesale (2.5%) and transport (0.75%). Design: Use of social marketing techniques to engage small and medium enterprises. To deliver environmental audits and action plans, provide ongoing one-on-one mentoring and opportunities to access sustainable business management skills training and advanced resource efficiency workshops. In addition, businesses were offered a follow-up audit at 12 months. Intervention: 150 free environmental audits including energy, water, waste and transport efficiency and provided individually tailored to reports to the business. Evaluation data: Combined financial estimates on energy and water showed an average saving per business of $952.11. Follow up audits found that the program collectively saved the following: $68,000 p.a. on reduced energy, water, waste and transport. Saved 460,245 kWh of electricity and natural gas (in the first 12 months); Saved 66,350 litres of potable water (in the first 12 months); mitigated over 440 tonnes of GHG emissions (in the first 12 months) and reduced waste to landfill per year by over 118,800 litres (in the first 12 months).

Report 3 Final report Light Industry Precinct Improvement Project (Cities of Gosnells and Cockburn)

Organisations involved: Cities of Gosnells and Cockburn and Edith Cowan University Funding support: Cities of Gosnells and Cockburn and Edith Cowan University Funding amount: $40,000 Delivery time frame: 2007-2009

9

Objectives: General awareness raising of the issues related to environmental management and to engage small businesses so as to assist them to make organisational changes that would benefit their overall bottom line. Business Participants: 2009 data collection included a smaller number of businesses overall (Maddington-Kenwick n=83; Cockburn n=51) than the number that participated in 2007 (Maddington-Kenwick n=205; Cockburn n=199). The profile of the actual businesses in both locations was dominated by manufacturing businesses (in 2009 this ranged from 45% - 49%), although there were also a substantial number of engineering (in 2009 ranged from 9-13%) and motor vehicles servicing businesses (in 2009 this ranged from 15-19%) in these locations. In Cockburn the sample had substantially more retail businesses whereas Maddington-Kenwick has substantially less retail than was the case in 2007. Specific data on the demographics of the owner-manager of each participating business were provided. Design: The research design incorporated survey data collection and an intervention program in both locations to try to create interest and raise awareness of environmental issues among small businesses. The survey data was collected directly with the small business ownermanager or other appropriate staff member at 2 points in time over 18 months and the intervention conducted in the interim period. Intervention: The intervention program included the following: Individual waste management advice for business by PRNRM staff; Developing links between business operators and recycling contractors; Promotion of local businesses that offered recycling services and new recycling facilities that became available during the project to the local business community; Provision of recycling bins and communal paper and cardboard bins via the local government authorities and recyclers into the industrial areas; Legal compliance advice by PRNRM staff to the local business community; Energy and water reduction strategies to the local business community; The creation of a stakeholder partnership between PRNRM staff, local government authorities, recycling contractors and the local business community; and The development and distribution of a regular newsletter to the local business community.

Evaluation data: The initial aim of the project was to create interest and raise awareness of environmental issues among small businesses in the two light industrial area and in this the project was very successful with the report indicating that 70% of the businesses had an increased level of interest in this topic. Although other topics were covered in this project the focus in this paper is water and energy and therefore we look only at these two issues here. In regard to water in this project the main finding was on wastewater where 27% of businesses did not treat their wastewater before disposal and little information was available to educate business owners and staff in this regard. There was between 10% and 30% improvement in awareness of, and use of, water efficiency measures between 2007 and 2009 in the two locations. Energy costs were checked by most businesses (over 70%) however, little effort to monitor or implement reduction strategies was evident. About 10% more businesses in both locations indicated at the end of the project that had implemented energy efficiency measures in their business.

10

The other area where the owner-managers were asked to provide feedback was in areas where they were consciously implementing environmental behaviour in their businesses. Of the eleven conscious environmental behaviours that were explored, in both locations ‘using energy efficient lighting’ was reported to be among the most practiced behaviour (M=4.0 and 4.33) whereas ‘purchasing green energy’ was the least frequent behaviour (M=1.8 and 1.2).

Report 4 Final Report City of Stirling SMEs & Environmental Management

Organisations involved: The City of Stirling and Edith Cowan University Funded: Jointly by City of Stirling and Edith Cowan University Funding amount: $45,000 Delivery time frame: 2009-2010 Objectives: General awareness raising of the issues related to environmental management and to engage small businesses so as to assist them to make organisational changes that would benefit their overall bottom line. Business Participants: 120 small businesses from across ANZIC codes with largest representation from retail (23%), Transport, postal and warehousing (14%), Construction (12%) and wholesaling (10%). Other category representation is reported. Specific data on the demographics of the owner-manager of each participating business were provided. Design: Incorporated survey data collection and an intervention program. The survey data was collected both face-to-face and by mail from the business owner-manager at two points in time over eight months. Intervention: Between the data collection periods an intervention program was implemented, including conducting environmental audits and the dissemination of relevant education material. The audits were carried out by specially trained staff and the education material was delivered by post or in person. The interventions included the following: Advice on waste reduction strategies directly to the individual businesses; Distribution of fact sheets on environmental issues and practices; Establishment of links between businesses and with waste recycling contractors; Environmental energy and water audits and reporting. Evaluation data: Less than a third of the initial 120 respondents were interested in an audit being conducted on their business and at the end of the project only 24 businesses had actually had an audit completed and these were focussed on energy rather than water as many businesses had little water use within their operations. The mean rating by the participants in terms of the audit usefulness was 4.6 out of 6 with nearly 75% finding them of good or very good use.

11

The annual expenditure by the small businesses on energy ranged from $2,585 to $43,250. Estimated annual energy savings were identified and ranged from 21% to 49% for the businesses. These savings when realised would equate to reductions in GHG abated tonnes of between 5 and 70 tonnes per business. The post-intervention survey asked the businesses what changes they had made in their businesses in terms of energy, transport, was and water use reduction and results from the analysis of this data showed that more attention was paid to changes in energy (41%), waste (25%), and transport (15%), than water (12%). In addition, it was reported that only 17% of the business indicated that they had prior business-specific environmental management training and this was recommended as a potential area for intervention. A sample audit report was provided to show the level of detailed feedback given to the businesses that took up the offer of a free audit. Significant factors that were highlighted in the report were: (a) the difficulty in engaging small businesses in environmental projects; (b) the value of having a repeat presence by stakeholders to improve engagement and develop trust so as to encourage information sharing and (c) the identified environmental training needs of small businesses. Positive findings included the overall finding that the engaged businesses were improving their environmental performance and were seeking more information. An area where the greatest gap in knowledge and action was found was in water efficiency and management. While many of the businesses were not large water uses, there was a considerable amount of untreated wastewater being directed into the main water systems. Linking the information in the reports back to our conceptual framework, as anticipated none on the interventions went beyond either the first two Kirkpatrick stages or the first three Storey stages. Table 2 Comparison of evaluation stages (Storey, 2000) Evaluation stage Report 1 Report 2 Report 3 Step 1 Take up of schemes Step 2 Recipients opinions Step 3 Recipients views of the difference made by the Assistance Step 4 Comparison of the performance of ‘Assisted’ with ‘Typical’ firms Step 5 Comparison with ‘Match’ firms Step 6 Taking account of selection bias

Report 4

























x

x

x

x

x

x

x

x

x

x

x

x

12

DISCUSSION Training inputs In regard to funding as a training input, in one case the funding was a joint effort between Federal government and a partner organisations (including local government) that were delivering the project on their behalf. The other three projects involved local government and partner organisations that were delivering the project on their behalf. The amount of expenditure varied considerably. Based on the number of participants in the projects the expenditure per business can be loosely calculated in three cases. In report 2, $222,000 was provided to audit 150 businesses. This represents an expenditure of $1480 per business. The savings estimated were $952 per business. In report 3, $40,000 was provided to create awareness and increase interest in environmental management. With 404 participants involved in this study an expenditure of $99 per business was made. In report 4, $45,000 was provided to survey and audit as many local businesses as could be encouraged to participate. Initial interest of 120 businesses was achieved however only 24 participated in the audits. If the value is calculated to include all 120 businesses it represents an expenditure of $375 per business. Regarding design as a training input, the designs appear to be appropriate to small businesses in 11 critical ways: 1. Went to the business 2. Focussed on practical issues 3. Repeat engagement strategies were included 4. Specific feedback provided to the business 5. Limited or no cost to the business to participate 6. Not one off but over time engagement 7. Covered a range of issues to allow business choice/inclusion 8. Provided individualised support and mentoring, including assistance to solve specific environmental issues 9. Experts involved in key components of the program 10. Sought to provide economic case for businesses to implement audit recommendations. 11. Allowed businesses of all industries to become involved. Only one program provided a reward for the participants for their engagement in the program, however the usefulness of this as motivation cannot be evaluated as no information is given in the report. This program also tried to link local businesses and the broader residential community by having the business participants mentor the householder. This was a novel idea which has merit. Householders were given information sheets to inform them what the businesses were doing about environmental issues. This strategy could easily have had benefits for both the business and the community. Unfortunately, no details about the level of interaction that took place or the outcomes this may have achieved for either the business or the householder were provided so we are unable to evaluate this strategy. The report that advises that its design included opportunities to access further sustainable business management training and advanced resource efficiency workshops did not provide any details of the training or workshops content or indeed how many businesses asked or attended this training.

13

In addition to funding and design, a range of other factors that could be considered training inputs were also addressed in the reports. Duration: All had at least a 12 months engagement period. This timeframe is understandable however at this point in time longer term durations would be recommended as most small businesses are still in the ‘low hanging fruit’ stage and want more information. Business characteristics including target size, industry: In general they targeted only the small and medium businesses but the design of one intervention also sought to engage the businesses as mentors for the wider community. Industry data showed a range of industry interest but in general the programs were dominated by participants from the retail and manufacturing sectors. Sample (business owner) characteristics: Only two reports provided demographic data on the business owner-manager. Government or other environmental specialist: Only two provided a clear statement of expert involvement, however, all appeared to have expert input. Instructor characteristics: These were either not outlined at all or in limited depth. Venue: – All went to the business. Monitoring/ support point in time versus ongoing: All seemed to have encompassed an ongoing approach for the duration of the project. Training outputs There were very few learning outcomes discussed in the reports and this may be due to the fact that the interventions were not formal training programs. One report did provide a percentage of increased environmental awareness among the program participants. In most cases reports focussed on the practical outcomes and often centred round energy rather than water. In one case percentage improvements in environmental practices as a consequence of participation were also provided. There was little other evaluation data about outcomes of the interventions provided, with only one report giving the mean rating that participants felt about the usefulness of the audits that were conducted. In at least one report it was difficult to ascertain if the estimates of outcomes (e.g. financial savings, GHG emission reductions) were actually made or an estimate of what could potentially be achieved if businesses followed through on audit suggestions. Two other reports made it clear that they were estimates of what could be achieved by the participating businesses if they followed the recommendations of the audit reports. One or two point in time evaluations were generally used and these did not appear to look at return on investment outcomes for the funding body or the business. In one report it was established that a 10% increase in implementation of business energy efficiency measures was achieved. An examination of where the projects sit within Storey’s six steps, showed that only projects 3 and 4 included pre-post evaluations and none of the projects incorporated comparisons with typical or matched firms. In regard to feedback/recommendations for future program design as training outputs, the reports did acknowledge where improvements for future projects could be made and in the two University reports specific issues that the funding body could address were provided. Only one report specifically stated that it would disseminate the results to stakeholders. Since none of the projects involved traditional classroom based instruction, issues relating to transfer of training, and maintenance and generalisation of learning, were understandably not addressed in any of the reports.

14

CONCLUSION AND IMPLICATIONS For environmental management training and development programs to be improved it is important that individual program reports detail the program design and provide thorough evaluation data to show how the training and development has had an impact, including the transfer of the learning into the workplace. However, the results of this study show that while impacts in terms of practical outcomes are shown in some detail, the specific design changes required and the impact of the program in the business are not well studied or reported. In many cases this is due to the short term nature of the programs (and their funding) which only allow a one or two point in time research connections with the small business ownermanager. This then only provides point in time assessment of the environmental management issues impacting on the business operations. While some progress in training and development design seems to have occurred over the period 2007 – 2010 with program becoming more focussed and holistic, in the majority, the program reports lacked sufficient detail to assist program design development. These reports generally discussed outcomes based on participant numbers and estimates of savings generated at a specific point in time similar to what is outlined in the first two of Kirkpatrick’s (1996) evaluation model and Storey’s (2000) three evaluation monitoring phases rather than reporting on higher level outcomes. The implementation of phase two of the study provided more insight into the impact of these environmental management programs. The short term nature of the programs and lack of follow up, impacts on the ability of the organisations running the programs to collect more meaningful data. In addition, it seems clear that the timeframes of these programs would not allow the subject matter experts to assist the business owner-managers to work towards a continual improvement process within their businesses even when they and the stakeholder were keen to develop their knowledge or skills further so that they could continue to improve the environmental performance of their businesses. The findings of the study are suggestive that the evaluation of environmental management training and development programs in Australia, and potentially elsewhere must become more rigorous so that it is clear to all stakeholders what impact they have had on small business environmental performance. Moreover, programs that provide an opportunity for longer engagement with small business owner-managers would be more valuable. Government and training and development providers need to ensure that future programs are developed and funded based on sound design principles.

REFERENCES Billington, L., Neeson, R., & Barrett, R. (2009). The effectiveness of workshops as managerial learning opportunities. Education and Training, 51(8/9), 733-746. Bramley, P. (1999). Evaluating effective management learning. Journal of European Industrial Training, 23(3), 145-153. Carleton-Hug, A., & Hug, J. W. (2010). Challenges and opportunites for evaluating environmental education programs. Evaluation and Program Planning, 33, 159-164. Condon, L. (2004). Sustainability and Small to Medium Sized Enterprises - How to Engage Them. Australian Journal of Environmental Education, 20(1), 57-67. 15

Department of Climate Change and Energy Efficiency. (2010). Report of the Prime Minister's task froce on energy efficiency. Retrieved 3 March 2011 from www.climatechange.gov.au. Department of Industry Innovation Science and Research. (2011). Key Statistics: Australian Small Business. Canberra: Retrieved from www.innovation.gov.au. Dionne, P. (1996). The Evaluation of Training Activities: A Complex Issue Involving Different Stakes. Human Resource Development Quarterly, 7(3), 279-286. Greene, F. J. (2009). Assessing the impact of policy interventions: the influence of evaluation methodology. Environment and Planning C: Government and Policy, 27, 216-229. Griffin, R. P. (2010). Means and ends: effective training evaluation. Industrial and Commercial Training, 42(4), 220-225. Grossman, R. & Salas, E. (2011). the transfer of training: what really matters. International Journal of Training and Development, 15 (2), 103-120. Hearn, W. M. (1988). Beyond training: A process model for transfer, evaluation and institutionalisation. Journal of Management Development, 7(3), 22-28. Henry, C., Hill, F., & Leitch, C. (2004). The Effectiveness of Training for New Business Creation. International Small Busines Journal, 22(3), 249-271. Hillary, R. (2004). Environmental management systems and the smaller enterprise. Journal for Cleaner Production, 12, 561-569. Keene, M., & Blumstein, D., T. (2010). Environmental education: A time of change, a time for change. Evaluation and Program Planning 33, 201-201. Kirkpatrick, D. (1996). Great ideas revisited: Revisiting Kirpatricks's four-level model. Training and Development, 50(1), 54-57. Mann, S., & Robertson, I. T. (1996). What should training evaluations evaluate? Journal of European Industrial Training, 20(9), 14-20. Massey, C., Tweed, D., & Lewis, K. (2003). New Zealand's BIZ training programme: service provider perspectives. Education and Training, 45(8/9), 439-448. McMullan, E., Chrisman, J., & Vesper, K. (2001). Some Problems in Using Subjective Measures of Effectiveness to Evaluate Entrepreneneurial Assistance Programs. Entrepreneurship Theory and Practice, 26(1), 37-54. OECD. (2002). Management Training in SME's. In OECD (Ed.). Paris. OECD. (2007). OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes. Paris. OECD. (2010, 17-18 November). Issue Paper 1: Innovative SMEs and Entrepreneurship for Job creation and Growth. Paper presented at the OECD Working Party on SMEs and Entrepreneurship (WPSMEE) "Bologna +10" High-Level Meeting on "Lessons from the global crisis and the way forward to job creation and growth", Paris. Patton, M.Q. (1990). Qualitative evaluation and research methods. (2nd ed.). Beverly Hills: Sage Publications. Patton, M.Q. (2003). Qualitative evaluation checklist. Retrieved 7.02.12 from www.wmich.edu/evalctr/checklists. Redmond, J., & Walker, E. A. (2008). A new approach to small business training: community based education. Education and Training, 50(8/9), 697-712. Redmond, J., & Walker, B. (2009). Final report: Light industry precinct improvement project (Cities of Gosnells and Cockburn). Available from www.ecu.edu.au. Redmond, J., & Walker, B. (2010). Final report: City of Stirling Small and medium enterprise and environmental management. Available from www.ecu.edu.au. Revell, A., & Blackburn, R. (2007). The Business Case for Sustainability? An Examination of Small Firms in the UK’s Construction and Restaurant Sectors. Business Strategy and the Environment, 16, 404-420. 16

South Metrolpolitan Regional Council (SMRC). (2008). Final report: Climate actions at home at work and in the community. Final report to The Department of Environment, Water, Heritage and the Arts. Available from SMRC Perth, Western Australia. Steensma, H., & Groeneveld, K. (2010). Evaluating a training using the "four levels model". Journal of Workplace Learning, 22(5), 319-331. Storey, D. (2000). Six Steps to Heaven: Evaluating the Impact of Public Policies to Support Small Businesses in Developed Economies. In D. Sexton, L & H. Landström (Eds.), The Blackwell Handbook of Entrepreneurship. Oxford, UK: Blackwell Publishers. Storey, D., & Westhead, P. (1994). Management training and small firm performance: A critical review. Working Paper No. 18 CSME Working Papers: Warwick Business School's Small and Medium Sized Enterprise Centre. South East Regional Energy Group (SEREG). (n.d.). Switched on Business and Industry. The Cities of Gosnells and Armadale and Serpentine Jarrahdale Shire (as South East Regional Energy Group (SEREG). Available from SEREG, Western Australia. Victorian Auditor-General's Office. (March 2011). Effectiveness of Small Business Victoria's Support Programs. Melbourne: Victorian Government Print Retrieved from www.audit.vic.gov.au.

17