Agreement And Audit Trail
Use this page to inspect the data sources, model-review rules and code behind the dashboard. Project materials are organized through the Job Ad Research at QSB-LUC GitHub organization and Loyola Data Mining Hugging Face collections.
Version of the review exclusions
This Evidence dashboard preserves an earlier analysis snapshot. The saved local database is dated 6 June 2026 (its recorded last-save date); the public dashboard-source release is dated 16 June 2026 (Git commit bbce0111f422afd20d5a3022a0f75294fa6d8f90). These dates describe the database snapshot and source release, not the last deployment of the website. The presentation and documentation were revised on 17 October 2026 without rebuilding its task profiles.
A newer occupation-task review release is available. The current manuscript companion documents the expanded review and filtering rules. Download the 19,682 excluded pairs and their provenance and status counts. The exclusions come from a selected 56,856-row ledger; they do not imply every retained link was reviewed. The expanded exclusions have not been applied to this historical Evidence dashboard’s profiles.
How To Inspect The Dashboard
Most figures and tables on the dashboard are backed by visible Evidence SQL. To inspect a page, use the menu in the top-right corner of the dashboard and turn on query display; the SQL blocks used to build the visible charts and tables will appear on the page.
The dashboard audit repository is available at Job-Ad-Research-at-QSB-LUC/jaat-lmi-dashboard-audit. The repository contains the dashboard source, source SQL, audit documentation, public input manifests, and rebuild scripts. See the repository README for the current rebuild path from public input files to the static dashboard.
Public Project Links
Core public project resources:
The data deposit and the repository snapshots are separate releases. USAJOBS records are distributed through the Hugging Face datasets above; raw NLx posting text requires authorized access.
Table 1. Public project links and repositories
Reproducibility Package
Table 2. Public release bundle components
Required Checks
Table 3. Public release agreement checks
Job-Ad-Derived Occupation-Task Pair Agreement Procedure
This section covers the job-ad-derived novel occupation-task pair LLM agreement procedure. It is one LLM agreement procedure in the broader project, not the only agreement activity.
Table 4. Job-ad-derived occupation-task pair agreement outputs
LLM Judge Prompt Templates
The prompts below are the literal templates used for the agreement procedures. Runtime calls fill the bracketed fields with the occupation, task, and definition text shown in each template; model outputs are parsed only as YES or NO.
Occupation-task pair agreement prompt
Occupation: {soc_title} ({occ8})
Occupation description: {soc_description}
Task: {task_text}
Does this task plausibly belong to this occupation? Answer with one word: YES or NO.Green task signal prompt
DEFINITION. The green economy is "the economic activity related to reducing
the use of fossil fuels, decreasing pollution and greenhouse gas emissions,
increasing the efficiency of energy usage, recycling materials, and
developing and adopting renewable sources of energy."
[Source: O*NET Green Occupations report, https://www.onetcenter.org/dl_files/Green.pdf]
A task is "substantively green work" if its TEXT directly advances one of
these green-economy activities. Generic operations, generic maintenance, and
administrative tasks are NOT substantively green UNLESS the task text itself
specifies an environmental, energy-efficiency, conservation, pollution-
reduction, recycling, or renewable-energy objective.
Examples:
- YES: "Develop strategies to address energy use, recycling, pollution reduction."
- YES: "Monitor emission control devices for compliance with regulations."
- YES: "Compute design specifications for implementation of conservation practices."
- YES: "Maintain wind turbines or solar arrays to ensure peak energy production."
- NO: "Assist engineering personnel to solve operating problems."
- NO: "Maintain equipment, making repairs as necessary."
- NO: "Schedule new employee orientations."
Task: {task_text}
Is this task substantively green work? Answer with one word: YES or NO.STEM task signal prompt
DEFINITION. STEM occupations "require education in science, technology,
engineering, and mathematics (STEM) disciplines."
[Source: O*NET STEM page, https://www.onetonline.org/find/stem]
A task is "substantively STEM work" if performing it requires applying
specific scientific, technological, engineering, or mathematical knowledge.
Routine skilled trades, installation, repair, and maintenance are NOT
substantively STEM UNLESS the task text requires technical analysis,
engineering design, scientific measurement, mathematical modeling,
programming, diagnostics, instrumentation, or interpretation of technical
data.
Examples:
- YES: "Conduct statistical analyses of biological or environmental data."
- YES: "Design experiments to test theoretical hypotheses."
- YES: "Develop algorithms to solve scientific or mathematical problems."
- YES: "Calibrate avionics instrumentation using interferometry."
- YES: "Interpret CAD/CAM models to verify part tolerances against engineering specs."
- NO: "Weld metal parts together using portable gas welding equipment."
- NO: "Maintain, adjust, and clean equipment, perform minor repairs."
- NO: "Lay out pipe routes following blueprints."
Task: {task_text}
Is this task substantively STEM work? Answer with one word: YES or NO.Downloadable Two-Model Agreement Workbooks
The workbooks below contain the results of the Sonnet + GPT-4o two-model agreement procedures. Each workbook consolidates the reviewed rows, LLM-validated rows, disagreed/blocklisted rows, agreement statistics, and metadata for one run. Legacy llm_as_judge_* CSV filenames remain in crosswalks/ only as compatibility inputs for existing SQL and warehouse rebuilds.
| Procedure | Workbook |
|---|---|
| Occupation-task pairs, top RCA batch | XLSX |
| Occupation-task pairs, high-volume batch | XLSX |
| Occupation-task pairs, random postings-derived batch | XLSX |
| Occupation-task pairs, independent null baseline | XLSX |
| Green task bundle | XLSX |
| STEM task bundle | XLSX |
Download agreement workbook manifest CSV
Green and STEM Task Agreement Procedures
The green and STEM task reviewers are separate LLM agreement procedures used for task-domain time series. They use prompt v2 and require a two-model unanimous YES for inclusion; Gemini was quota-blocked and is diagnostic only in this run.
Table 5. Green and STEM task agreement outputs
The green and STEM task workbook links above supersede the older per-CSV public downloads. Their workbook tabs are metadata, reviewed, agreement_retained, disagreed, agreement_stats, and discovery_rejected.
Public Inputs And Inspection Paths
Table 6. Public input and provenance inventory
Rebuild Path
bin/reproduce.sh is the executable path from public inputs to the dashboard. With --skip-etl, it starts from the staged DuckDB warehouse, rebuilds derived artifacts, rematerializes Evidence source parquets, and builds the static site. With --public-usajobs-dir, --nlx-dir, and --crosswalk-dir, it rebuilds the warehouse from public USAJOBS parquets, public aggregate files from the NLx corpus, and frozen crosswalk snapshots before rebuilding the dashboard outputs.