Home methodology

Methods And Audit Trail

This page lists the public data path for the Evidence dashboard. Longer codebooks and release notes live in the checked-in docs.

Public Reproducibility

The public release is built from:

  • Public USAJOBS parquet files and released NLx aggregate tables.
  • Public crosswalk snapshots.
  • Evidence source SQL under sources/usajobs and sources/nlx.
  • Checked-in ETL and audit code.

Agreement And Audit Trail links to data downloads, versioned software archives, review procedures and reproduction checks. See What Postings And Taxonomies Can And Cannot Say for limitations of taxonomy coverage.

Table 1. Public corpus size and task coverage
No Results

Monthly Active Jobs

USAJOBS monthly charts use active-month denominators. A posting contributes to each month between its start and end date when that date range is valid.

Table 2. USAJOBS active-month construction
No Results

Table 3. Monthly source windows by corpus

Terms. The NLx corpus represents approximately 162 million source postings before monthly expansion. Dashboard files contain aggregate evidence only; raw NLx posting text is not public. Monthly active jobs = summed active posting-month rows. A posting can contribute to more than one month when its active date range spans months. For more information, see NLx Research Hub.
No Results

Revealed Comparative Advantage (RCA)

RCA compares an entity's share of a task or Work Activity with that same item's share in the full corpus. Values above 1 indicate over-representation in the selected occupation, agency, or industry.

RCA = (entity item mentions / all entity item mentions) / (corpus item mentions / all corpus item mentions)

Job-Ad-Derived Occupation-Task Pair Two-Model Agreement Procedure

Distinct occupation-task pairs are reviewed with the publication two-model agreement rule to separate likely cross-occupational task signal from extraction noise. This is the job-ad-derived occupation-task pair two-model agreement procedure, one of several LLM agreement procedures in the broader project. Canonical O*NET-listed pairs are retained automatically. In summaries of reviewed pairs, an unlisted pair passes review only when Claude Sonnet 4.6 and GPT-4o both mark it plausible. This is distinct from filtering the full profiles, which retain eligible unreviewed links as explained below. Gemini v2 outputs are diagnostic only and are excluded from the publication rule.

The review now includes three batches with the same prompt and judge panel:

Sample Purpose Main reported result
Top RCA pairs Tests whether the most distinctive occupation-task pairs include real cross-occupational task signal Pass rate by O*NET migration distance; retained vs. excluded pairs
High-volume pairs Tests whether the most frequently mentioned task pairs show the same agreement pattern Pass rate and exclusion rate compared with the RCA sample
Random pairs Estimates the baseline plausibility rate outside top-ranked dashboard records Overall pass rate and false-positive/false-negative boundary checks

The samples remain separate in reporting. The RCA sample supports the Finding 1 headline; the high-volume and random samples are robustness checks and denominator context.

Table 4. LLM agreement inclusion summary by review batch
No Results

Table 5. Two-model reliability by review batch
No Results

Applying review exclusions to profiles

This dashboard uses an earlier review snapshot. Version dates and the newer exclusion release distinguish it from the current manuscript companion.

The database excludes occupation-task pairs listed in the review blocklist for the relevant corpus. It retains unreviewed pairs that are not blocklisted. Thus, the legacy table name JaatTasks_Validated identifies a filtered table, not a guarantee that every retained link received two positive reviews. Agreement tables above describe the reviewed sample. The reproduction guide explains input requirements; a rebuild without the blocklist does not apply these exclusions.

Data Quality Checks

Table 6. Wage, date and green-task source checks

Terms. Wage and date rows are USAJOBS source-quality diagnostics. Green task rows are the raw O*NET green-task crosswalk size, not the LLM-validated green signal.
No Results

O*NET AI Term Coverage
No Results

AI Label Provenance

Public AI code labels are taken from crosswalks/ai_a6_5_redacted_final2.csv. The dashboard does not use older AI label snapshots for public display.

Gated Material

Sentence-level AIMatch examples are held until a public-safe sentence artifact is available.

Release Documentation

Reference files in the public repository:

  • docs/reproducibility-bundle.md
  • docs/data-audit/public-dashboard-audit.md
  • docs/data-audit/public-artifact-manifest.md
  • docs/data-audit/public-codebook.md
  • docs/findings-ledger.md

The repository guide identifies external inputs, generated files and the commands needed to rebuild the dashboard. Local file links and build caches are not research inputs.