Workshop 6: Emergent Behaviors

Using LLMs for complex extraction tasks

What are emergent behaviors?

LLMs weren't explicitly trained to extract structured data, but they can. These "emergent" abilities let us automate tasks that used to require custom ML models and weeks of work.

Today you'll extract multiple fields at once and return structured JSON.

The Tool

Open Job Posting Analyzer
This tool lets you process the dataset line by line with your prompt. The data is loaded automatically for you, you just need to write the prompt and run it.

Exercise: Job Posting Analysis

Extract structured data in JSON format from each job posting.

Fields to extract

title — the job title
level — intern, junior, mid, senior, staff, lead, or executive
type — full-time, part-time, contract, internship, or freelance
remote — remote, hybrid, or on-site
skills_required — list of mandatory skills
skills_preferred — list of nice-to-have skills
years_experience — 0, 1-2, 3-5, 5+, or 10+
salary — min, max, currency, and period (year or hour), or null if not specified

Example

Input:

Looking for a junior frontend dev to join our startup! React experience a plus but not required — we'll train the right person. Must be willing to work on-site in Austin. $60-75k.

Output:

{
  "title": "Frontend Developer",
  "level": "junior",
  "type": "full-time",
  "remote": "on-site",
  "skills_required": [],
  "skills_preferred": ["React"],
  "years_experience": "0",
  "salary": {
    "min": 60000,
    "max": 75000,
    "currency": "USD",
    "period": "year"
  }
}

Your task

Write a prompt that extracts all fields at once
Process each posting and review the results
Compare outputs — where do you disagree with the model?

Edge cases to watch for: "Salary DOE", "unpaid", "top-of-market comp", "equity-heavy", required vs. preferred skills, vague experience requirements.

Questions to Consider

How did you structure your prompt to get valid JSON every time?
Did the model ever hallucinate information not in the text?
How would you handle cases where the model returns invalid JSON?
What would you do differently if processing 10,000 rows?
When is this "good enough" vs. when do you need a custom model?