ActiveJobs
24-MAG

Remote | Senior Software Engineer – LLM Evaluation

24-MAG

Full-timeRemotePosted 20 November 2026
Continue to 24-MAG's application →

Applications are made on 24-MAG's own site. We never sit in the middle.

Job description

About the job Remote | Senior Software Engineer – LLM Evaluation We are sharing a specialised part-time consulting opportunity for experienced Senior Software Engineers to contribute to advanced large language model evaluation, coding benchmark development, and AI-assisted software-engineering research. Selected professionals will curate and evaluate code, develop verification mechanisms, assess AI-generated software across multiple programming languages, and help researchers understand where advanced models succeed or fail throughout the software-development lifecycle. The work combines senior-level software engineering judgement with structured evaluation and benchmark design. Key Responsibilities Code Curation & Solution Development • Curate high-quality code examples for model training and benchmarking • Develop precise reference solutions to software-engineering tasks • Correct and improve code across multiple programming languages • Work with Python, JavaScript, ReactJS, C/C++, Java, Rust, and Go • Maintain strong standards for correctness, clarity, and maintainability AI-Generated Code Evaluation • Evaluate AI-generated code for technical correctness • Assess solutions for efficiency, scalability, and reliability • Identify implementation weaknesses and recurring error patterns • Review code quality against professional engineering standards • Provide structured rationales supporting evaluation decisions Verification & Automated Assessment • Design mechanisms that automatically verify software-engineering solutions • Build agents capable of assessing code quality • Develop approaches for identifying common model failure patterns • Create deterministic or structured checks where appropriate • Support reliable evaluation across repeated coding tasks Software Engineering Lifecycle Evaluation • Evaluate model capabilities across the software-development lifecycle • Assess reasoning around prototyping and architecture design • Review API design and production implementation decisions • Evaluate launch, experimentation, monitoring, and operational scenarios • Identify where models struggle with real-world engineering workflows Research & Benchmark Collaboration • Collaborate with researchers and cross-functional technical teams • Contribute to datasets used for training and benchmarking • Help define engineering evaluation strategies and quality standards • Compare model performance against professional engineering expectations • Support iterative improvements to coding-focused evaluation systems Ideal Profile • Several years of professional software-engineering experience • At least 2 years of continuous full-time experience at a top-tier product company • Strong full-stack development capabilities • Experience building scalable, production-grade software • Deep understanding of software architecture and system design • Strong knowledge of debugging and code-quality assessment • Experience reviewing and improving complex production code • Proficiency in one or more of Python, JavaScript, C/C++, Java, Rust, or Go • ReactJS experience is valuable for frontend-oriented assignments • Strong understanding of API design and production implementation • Familiarity with software monitoring and operational maintenance • Ability to reason across the complete software-engineering lifecycle • Strong analytical and problem-solving capabilities • Excellent written and verbal communication skills • Ability to produce clear and structured evaluation rationales • Comfortable collaborating with research and technical teams in a remote environment Engagement Details • Part-time independent contractor engagement • Fully remote • Minimum commitment: 10 hours per week • Flexible workload of up to 40 hours per week • Partial overlap with PST working hours is required • Initial project duration is approximately 1 month • Project is expected to begin the following week • Extension may be available depending on performance and project fit • No medical or paid-leave benefits are included under the contractor arrangement • Work will involve coding evaluation, benchmark development, verification systems, and software-engineering research • Assignments may span Python, JavaScript, ReactJS, C/C++, Java, Rust, Go, and related technologies • Work must be completed without using confidential, proprietary, unreleased, employer-restricted, client-restricted, or otherwise protected code, datasets, architecture materials, or technical information belonging to any employer, client, institution, or other third party About the Platform This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams. By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy  Or refer someone • • • • • • •

Other jobs you might want

Browse all jobs on ActiveJobs →