Software Engineering Evaluator (contract)
Overview
MarketerHire is seeking a Software Engineering Evaluator for a remote, hourly contract supporting the assessment of AI systems that write, debug, and review production code. The role requires 2–3 years of professional software engineering experience, strong command of at least one listed programming language, sound software engineering fundamentals, code review experience, excellent written English, and the ability to evaluate complex code with independent, evidence-based judgment.
Responsibilities
<h2>Key Responsibilities</h2><ul><li>Design task-specific grading criteria for feature implementations, bug fixes, refactors, tests, and code reviews.</li><li>Evaluate AI-generated code for correctness, including failures that pass the code's own tests.</li><li>Assess edge-case handling, security, performance, and readability in coding deliverables.</li><li>Write clear, evidence-based justifications for every evaluation score.</li><li>Apply consistent judgment across software engineering evaluation tasks.</li><li>Review code that may appear correct and run without failing while identifying subtle defects.</li><li>Reason through ambiguity and distinguish clearly between known facts and inferences during evaluations.</li><li>Iterate on evaluation work based on feedback from senior reviewers.</li></ul>
Requirements
<h2>Required Qualifications</h2><ul><li>2–3 years of professional software engineering experience shipping production code.</li><li>Strong command of at least one of Python, JavaScript/TypeScript, Java, Go, C++, C#, or a comparable language.</li><li>Solid grounding in data structures, algorithms, testing, debugging, and version control.</li><li>Experience reviewing other engineers' code.</li><li>Excellent written English.</li><li>Ability to follow complex instructions, reason through ambiguity, and spot subtle errors in code that looks correct and runs without failing.</li><li>Ability to clearly separate what you know from what you are inferring.</li><li>Ability to work independently, learn new tools quickly, and take feedback well.</li><li>Based in and working from the US, Canada, or UK.</li><li>Available for 30–40+ hours per week for the full 12 weeks.</li></ul>
Benefits
<h2>Benefits and Perks</h2><ul><li>Remote contract engagement for professionals in the US, Canada, and UK.</li><li>Flexible scheduling within the required weekly commitment.</li><li>Defined 12-week engagement with the possibility of extension based on needs and performance.</li><li>Opportunity to shape how AI systems are evaluated in your field.</li></ul>