The
Relationship Between Reliability (Rxx) and Validity (Rxy)
Coefficients[1]
By
Billy J.
McCroskey, Ph.D.
Given two independent
predictors of the same criterion, the predictor with the higher reliability
will also have the higher validity coefficient. Reliability coefficients tend
to be greater than validity coefficients because reliability places an upper
limit on validity. The rule for maximum possible validity due to the
upper limit placed on it by reliability is: The correlation between the predictor and independent criterion can
never be higher than the square root of the product of the reliability of the
predictor and the reliability of the criterion. For purposes of
understanding and comparing Reliability (Rxx) and Validity
(Rxy) coefficients, the table below best illustrates their
relationship in terms of equivalent ranges and magnitude.[2]
RELIABILITY VALIDITY VERBAL SCALE EQUIVALENTS
0.97-0.99+ 0.70-0.99+
Extremely High
0.94-0.96 0.60-0.69 Very High
0.90-0.93 0.50-0.59 High
0.80-0.89 0.40-0.49 Moderate
0.70-0.79 0.30-0.39 Low
0.00-0.69 0.00-0.29 Does not meet the
Scientific Standard[3]
Maximum Possible Validity =
SQRT[Rxx(Predictor) times Rxx(Criterion)]
Table 2: Correlational Analysis of Software Estimates of Transferability of Skills Relative to N=115 IARP Vocational Expert Participants' Estimates of Transferability[4]
Spearman’s Rho
Software
Package N
Predictive Validity Coefficient p-Value
MVQSTM MTSP 2003TM 56
0.56
0.00
VolcanoTM 4.1 MTS-R[5] 56
0.48
0.00
VolcanoTM 4.1 MTS-O[6] 56
0.39
0.00
SkillTranTM 63 0.27
0.03
Control
31 0.00
0.00
OASYSTM 62
-0.15
0.26
LifeStepTM
26
-0.21
0.31
The
Dunn, et. al. TSA study was based on an Injured Worker with
typical back restrictions in Bexar County, Texas.
In
a second TSA study based on 10 Non-Injured Workers[7]
completed at the same Fall 2003 IARP Conference, Grimley, Lowe, Williams and
Hahn (2004) found:
Table
3: Pearson R Predictive Validity Coefficients
|
|
|||
|
Marked correlations are significant at p < .05000 |
|
|||
|
Sampling Distribution: N=50 Job TSAs rated by 79 Vocational Experts
(VE) |
||||
|
|
TSA System
|
Predictive Validity Coefficients |
|
|
|
|
MVQS MTSP 2003-R[8] |
0.96 |
|
|
|
|
Volcano 4.1 MTS-R |
0.91 |
|
|
|
|
Volcano 4.1 MTS-O |
0.89 |
|
|
|
|
Volcano 4.1
SSTS-R |
0.76 |
|
|
|
|
Volcano 4.1
SSTS-O |
0.73 |
|
|
|
|
Volcano 4.1 VQ-R |
0.50 |
|
|
|
|
Control |
0.04 |
|
|
|
Note:
Job #'s Served as the Control Group
and IARP Vocational |
||||
|
Expert (VE) Job TSA Ratings were
the Criterion. |
||||
Summary
and Conclusions
In
the two IARP TSA Validy Studies (See Tables 2 & 3 above), MVQS Job-Person
Matching TSA Levels from MTSP 2003 and Volcano 4.1 were found to be valid
relative to Vocational Expert Ratings at the Fall 2003 IARP Conference in San
Antonio, TX. These scientific findings support the High Reliability and High to
Extremely High Face, Content, Predictive and Construct Validity of MVQS 2003
and Volcano 4.1 products for use with both injured and non-injured workers.
Results generalize to MVQS2005.02 and Volcano 5.02 programs.
MVQS
2005.02 and Volcano 5.02 Product Costs (Separately & Bundled Together)[9]
MVQS 2005.02 (McDOT, McPLOT and MTSP) Program Package $2,680.00
Volcano
5.02 Program Package $1,415.00
Four
30-hour CEU Training Courses Package $ 660.00
All
Program Packages Bundled together (Save $1,530.00) $3,225.00
Contact: `
Dr. Billy J. McCroskey
8209 Halifax Court North
Brooklyn Park, MN 55443-2538
Ph: 763-569-0680
When using the MVQS2005 Programs, General Labor Market Access (GLMA) should not be included, when considering, calculating and determining a reasonable opinion with respect to % of Vocational Disability. Instead, it is recommended that Specific Labor Market Access (SLMA) relative to specific post-injury job type matches and their associated wage estimates, be included when considering, calculating and determining a reasonable opinion with respect to % of Vocational Disability.
In terms of MVQS2005 Programs, % of Vocational Disability and Loss of Earning Capacity are assumed to be synonymous. The recommended formula for calculating the relative % of Vocational Disability for a given injured worker undergoing MVQS Vocational Analysis is:
(PreInjEarningCapacityEstimate
- PostInjEarningCapacityEstimate) / PreInjEarningCapacityEstimate
In MVQS Programs, General Labor Market Access (GLMA) is a reasonably controlled variable. It is controlled through the use of "frequently-hired-for" area job bank databases covering c. 92% of the available Specific Job Types in the Relevant Labor Market of Interest (RLMI). MVQS Volcano area job bank databases account for c. 92% of the available job openings for Specific Job Types available in the RLMI. As such, GLMA estimates reported in selected MVQS Vocational Analysis Reports are assumed to be reasonable at the Specific Job Type Level. Therefore, GLMA estimates are not moderator variables, and should not be considered part of any % of Vocational Disability calculation(s) stemming from the use of MVQS Volcano 5.01 Pre/Post-Injury wage estimates. Instead, one should consider and focus on Specific Labor Market Access (SMLA) in terms of Specific Job-Person Job Type Matches, as moderator variables, which do impact on % of Vocational Disability (Loss of Earning Capacity) considerations.
Theoretically, in lieu of a set of better, more reasonable, indicators, the single best, most reasonable, % of Vocational Disability Estimate would be the average, or mid-point, across all possible sets of such Pre/Post-Injury Earning Capacity Estimates, within known SEE error limits. For known SEE error limits associated with MVQS 2005 earning capacity estimates across specific job types, see McCroskey, Hahn & Dennis [Journal of Forensic Vocationology (JOFV), Vol. 6(1), Fall, 2000, pp. 107-134; See also, the MVQS2001Resources.doc, pp. 73-99; and the JOFV (7)1, 2002, pp. 128-157 (Linear Regression) and pp. 158-178 (Curvilinear Regression) VQ-Wage Studies].
Things to Consider:
Support for the MVQS2005 Pre-Injury Earning Capacity Estimate you may select, typically, but not always (depending on the specific facts in the case), comes from calculated annual hourly earnings (based on the latest pre-injury full year's worth of wages divided by 2080 hours in a standard work year).
Support for the MVQS2005 Post-Injury Earning Capacity Estimate(s) you may select, typically, but not always (depending on the specific facts in the case), comes from estimated annual hourly earnings. Such earnings should be based on an assumed post-injury full year's worth of wages associated with specifically selected reasonable job types from the client's post-injury available job match options divided by 2080 hours in a standard work year). Support for specifically selected job types and their associated wage estimates, typically comes from local labor market surveys designed to confirm the existence of, and identify typical wages associated with, such selected job types in the relevant labor market(s) of interest. (See Occupational Density estimates in Volcano 5.01).
The existence of available residual Transferable Skills for specific post-injury job type options, along with their level of job difficulty as measured by the Vocational Quotient (VQ) and the availability and length of any Specific Vocational Preparation (SVP; either formal, in-formal, or on-the-job training), necessary to bridge the gap between where the client currently is and where (s)he would need to be (in order to gain access to selected potential post-injury job options), would be moderator variables, which would most certainly impact on access to such specifically selected jobs and ultimately, access to the post-injury earning capacity data point(s) selected.
In cases where there is an absence of residual Transferable Skills, demonstrated client Values Agreement relative to specific post-injury job type options, along with their level of job difficulty as measured by the Vocational Quotient (VQ) level of Overall Job Difficulty and the availability and length of any Specific Vocational Preparation (SVP; either formal, in-formal, or on-the-job training), necessary to bridge the gap between where the client currently is, and where (s)he would need to be (in order to gain access to the selected potential post-injury job options), are moderator variables which would most certainly impact on access to such specifically selected jobs and ultimately, the post-injury earning capacity data point(s) selected.
Volcano
5.02 Percent of Vocational Disability based on Future Diminished Earning
Capacity Los Angeles, CA Labor Market Area Year 2005 Example
Date of Birth: 01/01/1970.
Client Work History: 1) Automobile
Mechanic (Primary), and 2) Automobile-Repair-Service Estimator
(Secondary).
Work-related Injury: Low Back Injury. Auto Mechanic no longer Matches; Service Esitmator still a TSA Job Match.
Pre-Injury Earning Capacity (EC): 50th %ile Earning
Capacity (EC) for Auto Mechanic in Los Angeles, CA in 2005
Post-Injury EC Assumption: 25th %ile EC for Automobile-Repair-Service
Estimator in Los Angeles, CA in 2005
1)
Locate Automobile Mechanic – Add to Work History
2)
Left-click 3)Select & Choose Pre-Injury Earning Capacity at 50th $ile
3)
Locate Automobile-Repair-Service Estimator – Add to Work History
4)
Left-click 3)Select & Choose Post-Injury Earning Capacity at 25th
$ile
5)
Analyze VDARE High Across Work History
6) Inpact
Medical Disability on VDARE High Across Worker Traits Profile
7)
Left-click Part 2: Button #6)Calculate % of Vocational Disability
8)
Type in Client Name and Date Of Birth, and
9)
Left Click Date of Analysis, Date To Begin Loss, Compute and Pres Value of
Total Loss / Percent of Vocational Disability, then ZReport1 to Print Report.
10)
Print ZReport1.rtf: % of Vocational
Disability Report
Client Name: John Doe
ZReport1.rtf:
FUTURE DIMINISHED EARNING CAPACITY (DEC)
FileName: C:\ZReport1.rtf)
Analysis Date: 07/20/2005
Date of Birth: 01/01/1970
Age: 35.55
Occupation (Pre): 620261010 Automobile
Mechanic
Yearly Earning Capacity (EC): $ 53206.40 Hourly: $ 25.58
Pre Standard Error Estimate (SEe): 1.32 Dollars
Pre County: Los Angeles
Pre State: California
Pre GeoLoc ECLR: 1.1768
Occupation(Post): 620261018
Automobile-Repair-Service Estimator
Yearly Earning Capacity (EC): $ 34964.80 Hourly: $ 16.81
Post Standard Error Estimate (SEe): 1.10 Dollars
Post County: Los Angeles
Post State: California
Post GeoLoc ECLR: 1.1768
Date to Begin Loss (DTBL): 07/20/2005
Age: 35.55
Date for Present Value: 07/20/2005
Age: 35.55
PastLoss (Exclusive of
Benefits): 0.00
PastLoss+Fringe Benefits (For Immediate
Payout): 0.00
Fringe Benefits (FB) to Pay Ratio: 27.53%, 2005
Wage Growth Year:
2005
Real Wage Growth (WG) Rate: 0.98 Percent
Real Interest (RI) Rate: 2.83 Percent
Net Discount Rate: 1.85 Percent
Age to begin Loss: 35.55
Future Work Life Expectancy: 30.45
Hours Per Year: 2080
Assumed Retirement Age: 66
Future Diminished Earning Capacity
(DEC+WG+FB): 825128.06
Pres Valu of Future Diminished Earning
Capacity: 543697.11
Past Loss with
Benefits: 0.00
Discount Adjustment for Date To Begin
Loss: 0.00
Pres Valu Past & Future Loss
(PL+FB)+(DEC+WG+FB):
543697.11
PERCENT OF VOCATIONAL
DISABILITY: 34.28
(c) 1984-2005 by Billy J. McCroskey, Ph.D.
All Rights Reserved
Volcano 5.02 Attachments
Report1: High Across Past Relevant Work History
(VDARE MLD)
BASIS: Vocational Diagnosis and Assessment
of Residual Employability
(VDARE) Maximum Least Demonstrated
(MLD) Worker Trait Profile
Primary GeoLocation: California
Secondary GeoLocation: Los Angeles
Year: 2005
Wage Inflation: 1.0689
GeoLocation ECLR: 1.1768
Openings Cutoff: 1
File Name: C:\JobsDOT.rtf Worker Traits Requirements S
PD1-6 EC1-7 V
DOT Code
DOT Title (OccDiversity)
RMLSPQKFMEC1234561234567 P
----------------------------------------------------------------------
620261010 Automobile Mechanic 433432334223001011000101 7
620261018 Automobile-Repair-Service Estimat
433323223212001111000000 7
----------------------------------------------------------------------
Maximum Least Demonstrated Worker
Traits:-> 433433334223001111000101 7
LMA R1: LABOR MARKET ACCESS Occupational
Diversity & Density Measures
BASIS: Vocational Diagnosis and Assessment
of Residual Employability
(VDARE) Maximum Least Demonstrated
(MLD) Worker Trait Profile
Primary GeoLocation: California
Secondary GeoLocation: Los Angeles
Year: 2005
Openings Cutoff: 1
1) VDARE MLD LABOR MARKET ACCESS to DOT Title
Matches
Available DOT Titles : 12972
Accessible DOT Titles : 2084
* Percent Relative Standard Error of DOT
Title Matches : 2.66%
Accessible Titles as a Percent of Available
DOT Titles : 16.07
2) VDARE MLD LABOR MARKET ACCESS to Expected
Yearly Openings
Available Yearly Openings : 968536
Accessible Yearly Openings : 287084
* Percent Relative Standard Error of
Yearly Openings : 2.26%
Accessible Openings as a Percent of Available
Openings : 29.64
3) VDARE MLD LABOR MARKET ACCESS to Expected
Current Employment
Available Current Employment : 2983898
Accessible Current Employment : 884458
* Percent Relative Standard Error of
Current Employment : 2.26%
Accessible Employment as a Percent of
Available Employment : 29.64
Report2: DOT Title Matches by MVQS Transferable
Skills % (MTS) Sort
BASIS: Vocational Diagnosis and Assessment
of Residual Employability
(VDARE) Residual Employability
Profile (REP) Evaluative Data
Primary GeoLocation: California
Secondary GeoLocation: Los Angeles
Year: 2005
Wage Inflation: 1.0689
GeoLocation ECLR: 1.1768
Openings Cutoff: 1
File Name: C:\JobsDOT.rtf OccDensity
Transferable
Expectancies
Skills
Percent Yearly Current
DOT Code
DOT Title (OccDiversity)
MTS SSTS SVP Openings Employed
=====================================XXX==============================
620261018 Automobile-Repair-Service
100 100 7
168 517
620684018 Brake Adjuster 77 67 2
9 29
825381026 Electrical Inspector 50
67 7 541 1667
559381014 Rubber Tester 47 50 6 3 8
825381014 Automatic-Window-Seat-and 47
67 4 4 13
822261026 Testing-and-Regulating Te 46
67 7 10 31
706361014 Assembly Inspector 45 50 5 6 19
822361030 Trouble Locator, Test Des 45
67 6 4 11
729381010 Electrical-Equipment Test 44 67 6 8 25
701261010 Quality-Control Inspector 44
33 6 14 44
807267010 Shop Estimator 44 83 6 5 15
619364010 Inspector I 42 50 6 5 14
609684010 Inspector, General 40 50 4 326 1003
=====================================XXX==============================
MVQS
2005.02 and Volcano 5.02 Product Costs (Separately & Bundled Together)[10]
MVQS 2005.02 (McDOT, McPLOT and MTSP) Program Package $2,680.00
Volcano
5.02 Program Package $1,415.00
Four
30-hour CEU Training Courses Package $ 660.00
All
Program Packages Bundled together (Save $1,530.00) $3,225.00
Contact:
Dr. Billy J. McCroskey
8209 Halifax Court North
Brooklyn Park, MN 55443-2538
Ph: 763-569-0680