Feasibility Study Department of Transportation Federal Aviation Administration July 12, 2002
|
Grant Thornton |
Table of Contents
Appendices:
Appendix A: Activity Task
Listing.......................................A-1
Appendices not
yet available online. |
|
Section 1. Introduction
The Federal Aviation Administration (FAA) proposes to conduct a competitive sourcing study of the Flight Service Station 0FSS)/Automated Flight Service Station (AFSS) function (from here forward referred to as FSS) in accordance with Office of Management and Budget (OMB) Circular A-76 policy. A competitive sourcing study will determine whether it is more efficient and cost effective to have FSS functions performed by FAA employees or by a contractor. The Competitive Sourcing Initiative in the President's Management Agenda encourages all federal agencies to subject commercial functions, currently performed by government personnel, to competition from the private sector to drive improved performance and cost efficiency. This initiative requires that competitive sourcing studies be completed for 15% of the Fiscal Year (FY) 2000 FAA commercial activity inventory by the end of FY 2003. The FSS function was proposed for competitive sourcing primarily as a result of conclusions reached in reports issued by the Office of Inspector General for Aviation. The Report on Automated Flight Service Stations: Significant Consolidation Benefits with OASIS, December 2001 stated that the FAA has an opportunity to significantly reduce costs if the current 61 FSS sites were consolidated in conjunction with the deployment of the Operational and Supportability Implementation System (OASIS). Another report, Management Advisory Memorandum on Acquisitions for Automated Flight Services, December 1996, recommended that the FAA explore innovative acquisition strategies for providing flight services. These reports suggest that current and future demand for flight services may be met with fewer FSS sites without compromising aviation safety. In order for the FAA to realize maximum savings, it is important that a complete evaluation of FSS functions be conducted. One of the best methods of accomplishing this is by subjecting the function and infrastructure to competition, enabling market forces to drive efficiencies. This approach was deemed to be fairer since it allows for the Government workforce to compete for retaining the work in-house rather than direct conversion to a contract service provider. Consequently, it was determined that the FSS function was a strong candidate for competitive sourcing. 1.2. Purpose of Feasibility Study Once FSS was proposed as a candidate
for competitive sourcing, the next step was to determine the feasibility of
successfully competing FSS activities with the private sector under an OMB
A-76 cost comparison. Grant Thornton LLP was tasked to conduct a Feasibility
Study, to help the FAA determine whether to proceed with competitively
sourcing the FSS function. This report presents the methodology, feasibility
criteria, and associated findings of Grant Thornton's analysis. The report
also includes recommendations on which FSS activities should and should not
be subjected to competitive sourcing. In addition, associated rationale to
support such recommendations as well as other considerations that may be
pertinent to or worthy of continued analysis are presented. |
|
Section 2. Methodology This section describes the methodology Grant Thornton applied to conduct the Feasibility Study. As depicted in Figure 1, a five-step approach to performing the study was applied Details for each step are outlined throughout this section of tile report. Analysis performed and the associated findings for each step are provided in Section 3. |
|
Figure 1. Approach
to Performing Feasibility Study |
|
Step 1:
Identify Activities to Consider for Competition - This step is comprised
of creating a list of FSS activities using the FY 2002 FAIR Act Inventory
and validating the list of primary activities with FAA subject matter
experts. The end result is a list of primary FSS activities to be considered
for inclusion in the competitive sourcing study. Step 2: Establish Criteria to Evaluate Feasibility - This step includes identifying themes of criteria to be used as evaluation factors for testing the feasibility of each primary activity identified in Step l. For each criteria theme, a set of key questions is developed to capture the appropriate information necessary to perform the analysis. In addition, a weighted tier structure is created and each criteria question is weighted and assigned to a tier based on the level of impact it could have on feasibility. This step lays the foundation for how the analysis is performed. Step 3: Test Criteria and Conduct Market Research - In this step, two forms of data collection are performed simultaneously. First, Grant Thornton works with FSS subject matter experts to respond to the criteria questions identified is Step 2 for each primary activity. Second, targeted commercial service providers are surveyed to assess market capability, interest, and experience. The end result of each activity provides the information necessary to perform the analysis described in Step 4. Step 4: Assess Feasibility of Competing Activities - The criteria performance of each primary activity is scored using the methodology and guidelines set forth in the previous steps. An analysis is then performed to quantify the impact each criterion will have on the activities under study. The result is a weighted average score for each primary activity, which is measured against a scale to determine the feasibility of competing the activity. Step 5: Recommend Activities to
Compete - Using the analysis results of Step 4, a recommended list of
primary activities that can be competed is derived in this step. In
addition, recommendations on next steps are provided. |
|
Section 3. Analysis and Findings This section describes tile analysis Grant Thornton performed to conduct the Feasibility Study. Details on the analysis and findings for each step are outlined in this section of the report. 3.1. Step 1: Identify Activities to Consider for Competition In this step, Grant Thornton created a list of the primary FSS activities to be considered for inclusion in the competitive sourcing study. We began this step by developing a list of FSS activities using the FY 2002 FAIR Act inventory that was currently under development. The list was then validated and referred through several facilitated workshop sessions with FSS subject matter experts from the FAA. The primary activities identified, and thus evaluated in this feasibility study include:
For each primary activity identified, we also defined specific tasks performed within each activity even though the feasibility analysis was performed at the primary activity level. This level of detail was necessary to aid in determining the feasibility of the function, because it was entirely possible that a specific task that was determined to be infeasible for study could have caused its entire primary activity to be deemed as infeasible for study depending on the importance of the specific task. It was through this identification process that we decided to separate NOTAM L and D for the purposes of this study. The specific tasks identified for each primary activity will also be useful for other phases of the competitive sourcing study. Appendix A provides a complete listing of the specific tasks. 3.2. Step 2: Establish Criteria to Evaluate Feasibility After the primary activities were
defined, specific criteria were established to evaluate the feasibility of
subjecting the activities to competitive sourcing. The feasibility criteria
are divided into four important themes: 1) Mission Accomplishment, 2)
Industry Availability, 3) Security, and 4) Personnel. Table 1 depicts
the criteria themes and describes the intended purpose for selecting them as
evaluation factors for testing feasibility. |
CRITERIA THEME | DESCRIPTION |
Mission Accomplishment | The Mission Accomplishment criteria theme addresses the impact a commercial service provider will have on the FAA mission. It determines if the function mandates government performance and identifies other concerns that may impact the feasibility of a commercial source performing the activities (e.g. liability issues, certification requirements, etc.). |
Industry Availability | The Industry Availability criteria theme is used to determine the capability, interest, and experience of commercial sources to perform FSS functions. |
Security | The Security criteria theme attempts to identify any security risks associated with the FSS function being performed by a commercial source. It addresses homeland security, security risks to government property, and potential conflict of providing services to military operations. |
Personnel | The Personnel criteria theme is used to determine if there are potential human resource issues that may impact the feasibility of competing specific activities. |
For each criteria theme,
we then developed a set of key questions that would allow us to elicit the
information necessary to evaluate the feasibility for each primary activity.
The basic question is what aspects of a function or activity must be
considered in determining whether or not it could be competed. The
roundtable discussions we facilitated with FAA personnel and subsequent
interviews with subject matter experts helped us refine the criteria themes
and associated questions. Table 2 depicts the questions within each
criteria theme that resulted from our roundtable discussions. When answered,
these questions will allow us to determine the feasibility for each primary
activity identified in the first step of the study. |
CRITERIA THEME | QUESTIONS FOR EVALUATION |
Mission Accomplishment |
|
Industry Availability |
|
Security |
|
Personnel |
|
While Table 2 represents the final set of criteria we used to evaluate overall feasibility, there were other criteria we evaluated that were purposely omitted. The primary reason for exclusion was that some of these criteria were evaluated to support the Business Case Analysis more than the Feasibility Assessment. We also dropped a few other criteria after undergoing a criteria-ranking exercise, which were deemed unfit to determine feasibility. 3.2.2. Tier Structure and Weightings We recognized that not all of the criteria questions have the same weight or impact on the feasibility of competing a particular activity. For this reason, we designed a weighting scheme that enabled us to rank the criteria in order of importance. The "tier structure" we built is as follows: Tier 1 - Criteria in this level can determine feasibility without consideration of other criteria in Tiers 2 and 3. In other words, if an activity does not pass these criteria, there is no need to further evaluate the feasibility because the activity will automatically be deemed not feasible to be competed. Tier 2 - Criteria in this level significantly impact feasibility but must be evaluated with the consideration of other criteria. Failing to pass one question in this tier will not prevent an activity from being feasible, but it does warrant further evaluation and consideration based on the other criteria in Tiers 2 and 3. Tier 3 - Criteria in this level have the lowest impacts on feasibility but are important to evaluate. These criteria cannot alone determine the feasibility but may have impacts based on the results from Tier 2 and other Tier 3 questions. Using these definitions as guidelines, we then set the overall weightings for each tier and assigned the criteria to appropriate tiers. As Tier 1 definition suggests, those criteria questions that were considered "showstoppers" were assigned to this level. By applying a weighting of 50% to Tier 1, we assured that any primary activity that does not pass the criteria question in Tier 1 would in fact be a "No Go" even without taking into account any of the other criteria. We also ranked the criteria in order of importance and set the weightings within each tier, as indicated by the percentages under the criteria. By design, all of the criteria within each tier sum to 100%, as do the three weightings of Tiers l, 2, and 3. Figure 2 below depicts the tier structure and weightings, and outlines the specific criteria contained within each level. Figure 2: Tier Structure and Weighting 3.3. Step 3: Test Criteria and Conduct Market Research In this phase, we collected relevant data to help test the feasibility of competing each primary activity defined. We accomplished this by conducting a workshop with subject matter experts from the FAA to address important issues, debate critical points, and obtain answers to the criteria questions defined in Section 3.2.1. All criteria were evaluated with direct input from FAA personnel with the exception of those in the industry Availability theme. The criteria in Industry Availability were assessed using the results of the Market Research, which is discussed in the next section. As Table 3 illustrates, the end product of this effort was a populated matrix, designed to capture whether a primary activity "passes" or "fails" the criteria test. This matrix was used as a primary, source, along with the market research results, for the feasibility analysis described in Section 3.4. Appendix B contains the detailed responses by FSS subject matter experts to each criteria question for all primary activities evaluated. Table 3: Criteria Response Matrix 3.3.2. Conduct Market Research Industry Availability is a key factor
in determining the feasibility of competing FSS activities. For this reason,
we conducted market research to evaluate the criteria within this theme. The
instrument we used was a high-level
The survey details services currently performed by FSS personnel, such as pre-flight briefings and services to aircraft in flight, and captures whether the commercial source currently has the capability to provide each service. Other questions addressed in the survey, which will be used in other parts of this competitive sourcing study (e.g. Business Case Analysis), include:
A total of 12 companies were targeted for market research and selected based on three different criteria:
The companies selected and pre-approved by the FAA were DynCorp, EDS, Harris, Lockheed Martin, Midwest Air Traffic Control Services, NAV Canada, Northrop Grumman, Raytheon, Robinson/Van Buren, SAIC, SERCO, and Signal Corporation. The survey was conducted by initially contacting the selected companies to determine participation interest and establish an appropriate point of contact. The point of contact was then sent a copy of the survey to prepare for a follow-up phone interview, during which we captured actual survey results and discussed other related topics. As the market research remits in this section will describe, the information and data collected from these companies were key inputs into our analysis and valuable in shaping the final remits of this feasibility study.
Of the 12 commercial sources we targeted for the market survey, three companies agreed to participate. These companies are Midwest Air Traffic Control Services, SAIC, and SERCO. Before the results of the market study are presented, a brief description of each company's background is provided. Midwest Air Traffic Control Services, Inc. - has been providing air traffic control, weather observation and reporting, NAVAID reporting, and control tower equipment maintenance services since 1978. Midwest currently provides support and controls over three million annual aircraft movements at over sixty airports across the United States. Midwest currently has employees who are licensed and certified by the National Weather Service (NWS), Federal Communications Commission (FCC) and the FAA. Its key customers include the US Air Force, Air National Guard, the NW Arkansas Regional Airport, Salisbury-Wicomico County Airport and the Friedman Memorial Airport Air Traffic Control Tower. SAIC - is the largest employee-owned research and engineering firm in the nation. SAIC and its subsidiaries have more than 40,000 employees with offices in over 150 cities worldwide. They are primarily an reformation systems and technology company, but provide various other services to clients in the health care, energy, and transportation industries. They currently assist the FAA in the development and modernization of civil aviation and the national airspace. They also provide air traffic control operations, system engineering and integration services, physical and information security, and information technology services to major clients such as the NASA, FAA, major airports, and major airliners. SERCO - is a non-government
provider of air traffic services, It operates on a global basis and handles
about 6 million aircraft movements each year at 90 airports. It provides
primary air traffic services including air traffic control to Civil Aviation
Authority (CAA), FAA, Transport Canada and international Civil Aviation
Organization (ICAO), air traffic engineering, meteorological services,
aeronautical information services, aviation systems development and air traffic
control training. SERCO is a publicly held corporation in the London Stock 3.3.2.2. Market Survey Results The primary objective of the market research was to obtain answers to criteria questions within the Industry Availability criteria theme discussed in previous sections. Specifically, the criteria questions were:
Overall, our market research findings support the feasibility of competing
FSS activities based on these three
criteria. Midwest, SAIC, and SERCO all indicated that they have the
capabilities to provide and are currently
providing, if not all, most of the primary activities identified in the FSS
function. They also demonstrated strong
interest in delivering FSS services to the FAA. For NOTAM D, SERCO indicated
that it is currently providing
this service to customers outside the United States. Table 4 provides the
criteria evaluation results based on the
survey responses received. |
Is there a commercial source available to provide the services? | Is there industry interest in submitting a proposal for the functions being competed? | Are commercial sources currently performing these services? | |
Primary Activity Pre-flight pilot briefing services Services to aircraft in flight Search and rescue services Notices to Airmen-Local (NOTAM L) Notices to Airmen-Distant (NOTAM D) Aviation weather and information broadcasting Navigation aid monitoring Weather observations En route flight advisory services (Flight Watch) Other FSS activities |
Response Y Y Y Y Y Y Y Y Y Y |
Response Y Y Y Y Y Y Y Y Y N/A |
Response Y Y Y Y Y Y Y Y Y N/A |
|
In addition to the survey results that directly support the criteria
evaluation analysis, other relevant information
and data were gathered through the market research, which will be important
in shaping the Business Case
Analysis and supporting competitive sourcing recommendations that will be
provided in the future. These
findings include:
See Appendix C for individual company survey results, which provide additional details on each company's capabilities, experience, and background. 3.4. Step 4: Assess Feasibility of Competing Activities In this step, an analysis was performed to quantify the impact each criterion will have on the activity under study. The following sections describe the analytical methodology Grant Thornton applied and the results of this analysis. Using the tier structure and weightings, we scored the answer to each criteria question (i.e., results of Steps 3.3.1 and 3.3.2.2) for each primary activity. Each criteria test was given a score of either a "1" or a "0", where:
Consequently, if the answer to a criteria question supported the feasibility to compete an activity, it received a score of 1. Conversely, if it did not support the feasibility to subject an activity to a competitive sourcing study, it received a score of 0. For example, if the question was asking whether a commercial source is available to provide FSS services, and the answer was "yes", it received a score of "1". If the answer to the question, "Does the function mandate performance only by the Government?" was "no" it still received a score of "1" because the end result is that it is feasible to study the activity in question based on this criterion. As these examples attempt to show, an answer "yes" did not always score a "l" and an answer "no" did not always score a "0". 3.4.2. Weighted Average Score and Scale For each primary activity within the FSS function, a final score was calculated to determine "Go" or "No Go" to whether the activity should be subjected to competitive sourcing. This number is a weighted average figure of all the scores the primary activity received in the criteria tests, using the weight percentages defined in Section 3.2.2. All final scores range from a scale of 0 to 1, where: Figure 3: Feasibility Scale We set 0.75 as the threshold for determining "Go" or "No Go". The rationale behind this is that a primary activity must "pass" the Tier 1 criteria, which is weighted at 50%, and achieve at least 25% "passing" rate within Tiers 2 and 3. By strategically assigning the percentages to each tier, the scoring system contains a built in mechanism to ensure a primary activity passes both the question in Tier 1 and at least half of the questions in Tier 2 and 3 to be considered feasible. Consequently, if a primary activity does not pass the question in Tier 1, the maximum score it can return is 0.50 even if it passes every other criterion in the other two tiers, and thus would be a "No Go". Conversely, if a primary activity does pass the question in Tier 1 but does not pass any of the criteria in Tiers 2 and 3, it also would return a final score of 0.50 and would still be a "No Go". The list of primary activities we identified as "Go" and "No Go" using this methodology is provided in Section 3. 3.4.3. Criteria Scoring Results In the previous sections we described the scoring system, tier structure, assignment of criteria questions to appropriate tiers, and criteria ranking and weighting. This section brings all of these steps together including the results from the interviews with FAA personnel, the market research, and the scoring process. As discussed previously, the criteria questions were answered from multiple sources including market research and interviews with FAA subject matter experts. See Appendix B for FAA's detailed responses to criteria questions for all primary, activities evaluated. Once we had the answers for each criteria question, we converted the results into scores using the "1" and "0' system explained in Section 3.4.1. in some cases, the criteria question was deemed to be "Not Applicable" for a particular primary activity. For example, for the question "If function requires certification, is certification available to contractor personnel?" the answer was "N/A" if there is no certification required for that primary activity. In those cases, a response of "N/A" reflects a positive response and thus was given a score of "l" during the conversion process. In the case where "N/A" meant the criteria question could not be scored for that particular primary activity, the weighting was distributed among the other questions in the tier. The following table portrays the scoring result for each primary activity and the weighted average score for each tier. Table 5: Criteria Scores In answering the criteria questions and developing the scores, we made the following set of assumptions:
3.5. Step 5: Recommend Activities to Compete In this last step of the study, we developed a list of the primary FSS activities that can be included in the competitive sourcing study and those that cannot, based on the outcomes of the criteria evaluation described above. These specific results are provided in section 4. This section provides the final results of the feasibility study Grant Thornton conducted, using the methodology and analysis outlined in Sections 2 and 3 above. It illustrates the final criteria evaluation scores and summarizes Grant Thornton's recommendations on activities that are and are not feasible for competitive sourcing. This section also discusses other issues and considerations that did not affect the outcome of the feasibility study but warrant further consideration in the Business Case Analysis or other phases of the competitive sourcing study. 4.1. Overall Feasibility Results Once we had the weighted average score for each primary activity by tier, the next step was to calculate the Total Weighted Average Score for each activity across all tiers. Table 6 summarizes the scores for each primary activity across all the tiers and calculates the Total Weighted Average Score for each primary activity, taking into account the weights applied to each tier. Based on the Total Weighted Average Score, it also displays the results for each primary activity on whether it is a "Go" or a "No Go" depending on the 0.75 threshold. |
According to the results of the weighted average scoring system, the issuance of Notices to Airmen-Distant (NOTAM-Ds) is the only primary, activity performed by FSS personnel that is not feasible for further study. NOTAM D failed to pass the first tier because the nature of the work performed (e.g. obtaining, classifying, and formatting NOTAM Ds) in this area impacts the status of National Airspace System. For that reason, the activity mandates performance by the Government only. All other primary activities however scored high enough to be deemed feasible for competition, in some cases, there is a potential risk inherent in having the services provided by a commercial source, but those risks were not substantial enough to warrant excluding them from further study. For example, within a primary activity there may be a single task that could represent a risk to homeland security but overall, the primary activity is not inherently governmental. The following paragraphs provide detailed descriptions of the final results and supporting arguments for each primary activity. Pre-flight Pilot Briefing Services Services to Aircraft in Flight Search and Rescue Services Notices to Airmen - Local (NOTAM L) Notices to Airmen - Distant (NOTAM D) Aviation Weather and Information Broadcasting Navigation Aid Monitoring Weather Observations En Route Flight Advisory Services (Flight Watch) Other FSS Activities As we analyzed the feasibility of each primary activity above, we identified themes or issues that will need to be addressed as the FAA moves into the next stages of the competitive sourcing process. The following is a list of these themes and a brief description of theh' current or future impacts on the FSS function and the FAA. Security Liability Certifications After Hours Military Operations Support International Liaison Facilities and Equipment Maintenance Section 5. Recommendations and Next Steps Based on the analysis and results of tiffs feasibility assessment, the following actions are recommended for next steps:
|
Appendices Will be added as soon as I can Scan them. |