hiring-agent

Hiring Agent

Resume-to-Score pipeline that extracts structured data from PDFs, enriches with GitHub signals, and outputs a fair, explainable evaluation.

---

Contents

• Context and intent
• Coverage
• Overview
• Architecture
• Installation and Setup
  • Prerequisites
  • Quick setup with pip
  • Ollama models
• Configuration
• How it works
• CLI usage
• Directory layout
• Provider details
• Contributing
• License

---

Context and intent

This project got a lot of attention recently, and some of the discussion surfaced misconceptions worth addressing directly.

What this is not:

• Not an ATS (Applicant Tracking System)
• Not used to screen HackerRank’s open roles
• Not a product available to HackerRank customers

What it actually is:

Every year HackerRank receives 50,000–60,000 intern applications. No human can read that many resumes well. This tool was built to rank them — helping decide which resumes to read first. Resumes scoring below the cutoff are filtered out, but the cutoff is intentionally set very low so only candidates at the very bottom of the distribution are removed. The vast majority pass through to human review, where the real decisions are made.

Since this was built, HackerRank has also shipped AI Interviewer (Chakra) to automate the first round of interviews — so candidates are no longer assessed on their resume alone.

On the default model:

The repo ships with gemma4:latest as the default because it runs locally on most laptops without any cloud API key. Actual intern resumes at HackerRank are evaluated using a top-tier Gemini model. The repo ships with a demo config, not the production one.

---

Coverage

Articles and discussions that have shaped how we think about improving this project:

Article	Key takeaway
HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74/100. No — 88/100. Actually 83/100. — Dan Kinsky	Deep statistical analysis of score variance across 100 runs of the same resume. Isolates which categories are stable (technical skills) vs. noisy (project quality judgments). Points to LLM non-determinism as the root cause.
The Score Depends on the Roll of the Dice — Pinggy Blog	Reproduces the variance findings and surfaces a security issue: invisible text embedded in PDFs can inflate scores significantly.
The Hiring Rubric Inside — ByteIota	Breaks down the scoring weights and argues that a GitHub-centric rubric disadvantages engineers whose work is in private enterprise repos. Also notes the signal degradation risk as candidates optimize for the now-public rubric.
Analyzing resume scoring consistency — Mariano Gobea Alcoba, DEV Community	Proposes concrete fixes: standardized data formats, versioned evaluation models, ensemble scoring, and explainability layers to reduce variance and make the system more robust.
AI-Powered Pipeline for Explainable Resume Scoring — AIToolly	Covers the launch and highlights the transparency argument — making scoring logic public allows scrutiny that proprietary ATS systems never face.
Hacker News discussion	200+ comment thread covering LLM determinism, GDPR Article 22 implications, and the broader ethics of automated resume filtering.

Video coverage

• HackerRank Open-Sourced Their ATS? — YouTube Short
• HackerRank Open-Sourced ATS Tool for selecting Resume — YouTube Short
• HackerRank Custom ATS Released! Get Your Resume Score & Beat ATS Filters — full walkthrough video

Community tools built on this repo

• Resume Reality Check — hosted tool that lets candidates score their own resume against the same rubric

---

Overview

Hiring Agent parses a resume PDF to Markdown, extracts sectioned JSON using a local or hosted LLM, augments the data with GitHub profile and repository signals, then produces an objective evaluation with category scores, evidence, bonus points, and deductions. You can run fully local with Ollama or use Google Gemini.

---

Architecture

Flow pymupdf_rag.py converts PDF pages to Markdown-like text. pdf.py calls the LLM per section using Jinja templates under prompts/templates. github.py fetches profile and repos, classifies projects, and asks the LLM to select the top 7. evaluator.py runs a strict-scored evaluation with fairness constraints. score.py orchestrates everything end to end and writes CSV when development mode is on.

Key modules models.py Pydantic schemas and LLM provider interfaces. llm_utils.py Provider initialization and response cleanup. transform.py Normalization from loose LLM JSON to JSON Resume style. prompts/ All Jinja templates for extraction and scoring.

---

Installation and Setup

Prerequisites

• Python 3.11+

  The repository pins .python-version to 3.11.13.

• One LLM backend (either of them)

  • Ollama for local models Install from the official site, then run ollama serve.
  • Google Gemini if you have an API key, get it from here.

Quick setup with pip

$ git clone https://github.com/interviewstreet/hiring-agent
$ cd hiring-agent

$ python -m venv .venv
# Linux or macOS
$ source .venv/bin/activate
# Windows
# .venv\Scripts\activate

$ pip install -r requirements.txt

Ollama Models

Pull the model you want to use. For example:

$ ollama pull gemma4:latest

If you want different results, you can pull other models such as:

# For higher system configuration
$ ollama pull gemma3:12b

# For lower system configuration
$ ollama pull gemma3:1b

---

Configuration

Copy the template and set your environment variables.

$ cp .env.example .env

Environment variables

Variable	Values	Description
DEFAULT_MODEL	for example gemma4:latest, gemini-2.5-pro, or gpt-6-luna	Model to use; must exist in providers.json — the provider is inferred from which provider lists it. Defaults to default_model in providers.json.
GEMINI_API_KEY	string	Required when using a Gemini model.
OPENAI_API_KEY	string	Required when using an OpenAI model.
GITHUB_TOKEN	optional	Inherits from your shell environment, improves GitHub API rate limits.

Provider mapping lives in providers.json — each provider declares its base_url, an optional API-key env var, and per-model parameters; config.py loads it and resolves the provider for a model. config.py also has a flag:

# config.py
DEVELOPMENT_MODE = True  # enables caching and CSV export

You can leave it on during iteration. See the next section for details.

---

How it works

1) PDF extraction

• pymupdf_rag.py and pdf.py read the PDF using PyMuPDF and convert pages to Markdown-like text.
• The to_markdown routine handles headings, links, tables, and basic formatting.

2) Section parsing with templates

• prompts/templates/*.jinja define strict instructions for each section Basics, Work, Education, Skills, Projects, Awards.
• pdf.PDFHandler calls the LLM per section and assembles a JSONResume object (see models.py).

3) GitHub enrichment

• github.py extracts a username from the resume profiles, fetches profile and repos, and classifies each project.
• It asks the LLM to select exactly 7 unique projects with a minimum author commit threshold, favoring meaningful contributions.

4) Evaluation

• evaluator.py scores the resume against the role selected on the command line.
• Each role lives in roles/<role_name>/ and defines its own scoring categories and weights in role.json, plus its own criteria.jinja and system_message.jinja prompts (encoding fairness and scoring rules).
• The shipped software_engineering_intern role scores open_source, self_projects, production, technical_skills, and ai_fluency, plus bonus and deductions, with evidence for each. Other roles can define entirely different categories.
• The intern role defines bonus_rules in role.json. The LLM returns evidence and allowed point values for each named bonus; Python calculates the total and itemized explanation. For HackerRank Orchestrate, the model supplies the best rank from awards and its points; Python validates +4 for ranks 1–100, +3 for 101–200, +1 for 201–500, and 0 otherwise, once per candidate. All bonuses share the 20-point cap. Roles without bonus_rules keep the original total-and-breakdown response format. Evidence interpretation still depends on the LLM.

5) Output and CSV export

• score.py prints a readable summary to stdout.
• When DEVELOPMENT_MODE=True it creates or appends a per-role resume_evaluations_<role>.csv with key fields (columns follow the role’s categories), and caches intermediate JSON under cache/.

---

CLI usage

End to end scoring

Provide a path to a resume PDF and the role to score against. --role is the name of a directory under roles/ and is required.

$ python score.py ./resume/sample.pdf --role software_engineering_intern

What happens:

1. If development mode is on, the PDF extraction result is cached to cache/resumecache_<basename>.json.
2. If a GitHub profile is found in the resume, repositories are fetched and cached to cache/githubcache_<basename>.json.
3. The evaluator scores the resume against the selected role, prints a report and, in development mode, appends a CSV row to resume_evaluations_<role>.csv.

Roles

A role bundles its rubric in roles/<role_name>/:

roles/software_engineering_intern/
├── role.json           # categories, weights (max), bonus_max, score bounds, position_title
├── criteria.jinja      # evaluation criteria prompt (receives {{ text_content }})
└── system_message.jinja

role.json drives the scoring schema, the printed report, the CSV columns, and the score caps — so each role can score against its own categories and weights.

To add a role, scaffold one with basic template files and then edit them:

$ python score.py --init-role backend_engineer
# edit roles/backend_engineer/{role.json,criteria.jinja,system_message.jinja}
$ python score.py ./resume/sample.pdf --role backend_engineer

--init-role creates the role directory with placeholder categories and prompts (it only scaffolds; it does not score a resume). You can also copy an existing role directory instead.

---

Directory layout

.
├── .env.example
├── .python-version
├── config.py
├── evaluator.py
├── github.py
├── llm_utils.py
├── models.py
├── pdf.py
├── prompt.py
├── prompts/
│   ├── template_manager.py
│   └── templates/
│       ├── awards.jinja
│       ├── basics.jinja
│       ├── education.jinja
│       ├── github_project_selection.jinja
│       ├── projects.jinja
│       ├── skills.jinja
│       ├── system_message.jinja
│       └── work.jinja
├── providers.json
├── pymupdf_rag.py
├── requirements.txt
├── roles.py
├── roles/
│   └── software_engineering_intern/
│       ├── role.json
│       ├── criteria.jinja
│       └── system_message.jinja
├── score.py
└── transform.py

---

Provider details

Ollama

• Set DEFAULT_MODEL to any pulled model listed in providers.json, for example gemma4:latest
• Requests go through models.OpenAICompatibleProvider against Ollama’s OpenAI-compatible endpoint (http://localhost:11434/v1)

Gemini

• Set DEFAULT_MODEL to a Gemini model listed in providers.json, for example gemini-3.8-flash
• Provide GEMINI_API_KEY
• The same models.OpenAICompatibleProvider wrapper is used, pointed at Gemini’s OpenAI-compatible endpoint

OpenAI

• Set DEFAULT_MODEL to an OpenAI model listed in providers.json, for example gpt-6-luna
• Provide OPENAI_API_KEY
• Requests use OpenAI Chat Completions with JSON Schema structured output

---

Contributing

Please read the CONTRIBUTING.md for detailed guidelines on filing issues, proposing changes, and submitting pull requests. Key principles include:

• Keep prompts declarative and provider-agnostic.
• Validate changes with a couple of real resumes under different providers.
• Add or adjust unit-free smoke tests that call each stage with minimal inputs.

---

License

MIT © HackerRank