July 20, 2017

New Report Suggests FMCSA’s Safety Measurement System is Sound

New Report Suggests FMCSA's Safety Measurement System is Sound

NASEM recently released a prepublication copy of the report entitled “Improving Motor Carrier Safety Measurement.” The researchers reviewed FMCSA’s current CSA program by looking into SMS’s public usage, measure calculation, and discrimination ability of measures.

Findings reveal the SMS to be reasonably structured. Its methodology for identifying high-risk motor carriers was also found to be defendable.

However, empirical validation of some aspects of the SMS is still insufficient and requires improvements, as the report suggests. A large degree of the things currently being done is still based on subject-matter expertise. The researchers, therefore, recommended FMCSA to adopt a more statistically-founded approach to safety measurement.

Following is a list of acronyms that are going to be used in this article.

  • NASEM — National Academies of Science, Engineering, and Medicine
  • CSA — Compliance, Safety, and Accountability
  • FMCSA — Federal Motor Carrier Safety Administration
  • SMS — Safety Measurement System
  • CMV — Commercial Motor Vehicle
  • BASIC — Behavioral Analysis and Safety Improvement Categories
  • FAST — Fixing America’s Surface Transportation
  • MCMIS — Motor Carrier Management Information System
  • CVSA — Commercial Vehicle Safety Alliance
  • IRT — Item Response Theory
  • VMT — Vehicle Miles Travelled
  • APU — Average Number of Power Units (Number of CMVs owned or leased by carriers)
  • MMUCC — Model Minimum Uniform Crash Criteria
  • ELD — Electronic Logging Device

Now that we are familiar with all the terms we are going to use, let’s dive into the report and see what it suggests.

On the CSA Program

FMCSA’s CSA program aims to ensure that CMV carriers keep high safety standards in freight and passenger transportation.

The program utilizes its safety measurement system as a prioritization tool that ranks motor companies based on safety ratings. The agency then targets low ranking carriers for intervention programs to reduce the possibility of future crashes.

The rating method of the safety measurement system is based on violation class and frequency assessed mainly over roadside inspections over the past two years.

The violations are divided into six groups that each have its respective weight and scoring metrics. The Safety Measurement System (SMS) integrates these six measures with another weighted measure — crash frequency. These seven categories are known as the BASICs.

Carriers with many crashes, inspections, and lots of violation data have their seven BASICs computed. The resulting calculation ranks them among peer groups. Greater values mean that carriers have a higher crash frequency or violation count compared to the rest.

Carriers that rank above thresholds set by FMCSA are given a range of interventions. Warning letters, investigations, fines, and even operation suspension are some of the interventive actions FMCSA can take against these carriers.

CVSA trained officers inspect over 3 million CMVs each year. These safety enforcers check and determine whether any CMVs are operating in violation of mandated safety regulations.

Data on CMV crashes, inspections, and violations are logged into the MCMIS. The SMS retrieves its data input from the MCMIS for calculations.

Review on FMCSA’s Safety Measurement System

Stakeholders, outside reviewers, and other groups and individuals criticized the safety measurement system for multiple reasons.

Some of those points are:

  • Utilizing assessments that are highly variable
  • Absence of fault consideration in the weighted crash count
  • Grouping carriers in same peer groups despite different tasks and operations
  • Utilizing measures sensitive to varying effects from different states
  • Employing metrics that do not predict the future crash frequency of carriers
  • Using measures that do not reflect the safety performance improvement efforts of carriers over time.

Because of the value at risk, FMCSA recognized the need to examine whether the criticisms are valid. The agency also wanted to analyze the performance of the SMS and note down improvements needed to be made.

The FAST Act of 2015 previously recommended the FMCSA to fund a study by NASEM to evaluate the safety measurement system.

Two units in NASEM collaborated — the Transportation Research Board and the Committee on National Statistics. These groups assembled to form the review panel of FMCSA’s CSA Program and began the study in March 2016.

The team was tasked to analyze the ability of SMS metrics to identify high-risk from low-risk carriers. The team was also charged to assess SMS’ public usage and review the data and processes used in calculating measures.

With the findings, the team was then finally tasked to provide advice on data collection and safety measurement methods.

System Assessment

The panel assessed the following about the current safety measurement system being employed:

  • Many factors that contribute to crashes are not present in the MCMIS.
  • MCMIS data as input for creating a crash prediction model that relies on carrier-level behavior is counter-productive.
  • FMCSA’s approach with the SMS focuses more on crash prevention than predictions.
  • SMS identifies carriers that place little priority on safety performance practices.
  • Intervention is aimed at behavior modification that hopes to reduce possible future crash.
  • The panel believes that SMS’ general approach is sound.
  • In response to the criticisms, researchers found the current SMS implementation as defendable, fair, and not overtly biased.
  • The panel believes some implementation details lack the support of empirical studies and are ad hoc.
  • The researchers argue that replacing the current algorithm with a statistical model would address implementation details.

The panel concluded that the SMS is currently structured reasonably well and can be defended from criticisms. The researchers, however, also acknowledge that there is room for improvement by adopting an approach that is more statistically principled compared to SMS.

Statistical Model

The researchers noted similarities in the approach used by SMS with IRT models.

IRT, in a nutshell, shows the relationship between a latent trait and the probability of giving a particular response. The latent trait is assumed to influence giving that particular response directly.

IRT systems have been applied in other contexts and have shown to be both fruitful and efficient.

Because of theoretical advantages in using IRT, the researchers recommended a model to be developed over the next two years. If the system performs well in identifying high-risk carriers for alert, FMCSA should replace the SMS with it.

According to the panel, the model has the following benefits over the SMS.

  • System scoring utilizes current observed data instead of dated empirical information
  • Calculation transparency is enhanced
  • Variances of scores and ranks can be directly estimated
  • The likelihood of being chosen for inspection is accounted for
  • A basis on the impact of data insufficiency on carrier safety ratings can be established
  • It justifies evaluating the current structure of the BASICs — including which violations belong to which basic
  • A natural way of subdividing the system into more layers can be naturally explored with IRT models
  • Safety as a multidimensional construct can be investigated — thereby determining the number of BASICs needed from the factors
  • Time can be accounted for, and proper time weights can be integrated
  • New safety measures can be incorporated when available without requiring a rework of the whole system
  • Ranges to better grasp ranking overlaps can be created
  • Changes in safety can be adapted over time

Improved Data

The researchers noted areas in the MCMIS data that could be improved.

  • Because an update on carriers’ VMT and APU is only needed every two years, non-recency of figures is possible. Flawed or out-of-date data affects carrier percentile ranks in the SMS.
  • Crash data collection procedures are not standardized across states. A substantial chunk of crash data is also missing from the MCMIS.
  • A good deal of information written in police reports is not depicted in the MCMIS.

The panel recommended collecting VMT and APU data by state and by month. The researchers also advised the MMUCC to be universally adopted by the FMCSA to standardize crash data collection.

The panel also believes that carrier processes and practices may predict future crash rates. MCMIS, however, lacks information on company operations.

The researchers noted that the SMS believes that carrier operating procedures partly contribute to a significant amount of crashes.

The panel, therefore, recommends that the following information is obtained.

  • Turnover rate of company employees
  • Type of cargo being handled
  • Level and method of compensation
  • Better data on exposure — such as high-quality VMT from ELDs.

The researchers noted that ELDs would be mandatory for most carriers by the end of 2017. With the possibility of ELD results being reported to FMCSA, VMT data can be conveniently produced and definitive.

What’s Next?

The findings of the report point to increased data collection in the following years.

With the ELD Mandate coming into effect this December, more information can be obtained for safety measures analysis.

Some ELDs can accurately measure and monitor vehicle performance and anticipate possible service needs. Because of their superior assessment ability, FMCSA may increase production standards for these devices soon.

KeepTruckin ELDs are already ahead of the game with our advanced measurement features.

Our ELDs not only capture vehicle performance but also track the driving style of individual truckers. It measures speeding, excessive acceleration, and hard cornering of drivers — data that may be needed by FMCSA in the future.

Want to know more about our KeepTruckin ELD Solution? You can contact us a call at 855-434-ELOG or email us support@keeptruckin.com for more information.

Click the blue button below to request a free demo.

Request Demo