Evaluations - Member of Technical Staff
Simile · San Francisco · $200,000–$400,000 (stated in the posting)
About the Company
Pilots don't train with real passengers. Actors don't rehearse with real audiences. Yet, the most consequential decisions in society are often pushed straight to production.
Simile is changing that. We have built the first AI simulation of society, populated by generative agents based on real humans. Our research pioneered the field of AI-based simulation, proving it is possible to model human behavior with high accuracy. Today, we are developing a Foundation Model to predict human behavior in any situation, at any scale.
We have raised $300M in total funding, with a $2B post-money valuation in our latest round. Our investors include Greenoaks, Index Ventures, Hanabi, A*, Bain Capital Ventures, Factory, CVS Health Ventures, and Definition.
About the Role
As a Member of Technical Staff, Model Evaluations at Simile, you will help define how simulations of human behavior should be tested and when their predictions are useful enough to guide real decisions. You will develop measurement methods, challenge existing assumptions, and turn evidence into priorities for modeling and product. This is an opportunity to help establish evaluation practices for an emerging field while building systems that people use.
Evaluation at Simile brings together model evals, statistics, behavioral science, research methodology, product quality, and human judgment. Because our models simulate people, populations, markets, and groups, our evals must account for distributions, noisy human ground truth, uncertainty, qualitative outputs, and customer decisions. You will work with rich human data, including surveys, long-form interviews, customer studies, qualitative research, transactions, product interactions, and other real-world behavioral traces.
Questions you could investigate include: How can we tell whether an interview elicits faithful information? When do simulated responses to interventions match observed behavior? Can we predict an individual’s future choices better than population baselines? You will turn questions like these into studies, working tools, and evidence that guides what we build.
The core question for this role is simple:
How do we know when a simulation of human behavior is good enough to trust?
Areas we are especially hiring for
We are building a team with complementary strengths. Depth in one of the four areas below can be a strong fit; we do not expect expertise in all four. We also welcome specialists and generalists in broader model and product evaluation, research engineering, and related fields. Your workstream will reflect your strengths and the team’s priorities.
Human Data and Measurement: Build reliable systems for collecting and evaluating human data. Assess whether interviews and other collection methods elicit faithful, sufficiently rich accounts of people’s experiences and behavior, and whether leading questions or unsupported detail distort the resulting data. Design rubrics, expert review, and annotation workflows; measure data quality, validity, and rater agreement; and turn these into repeatable programs that can run at scale.
Behavioral Experimentalist / Builder: Design and build creative experiments across diverse populations, settings, and behavioral tasks. Develop tools and collection workflows to run studies at scale, test whether simulations reproduce human responses to interventions, and assess how findings generalize beyond each study.
Behavioral Measurement and Causal Inference: Develop scalable strategies to learn causal relationships from large, diverse behavioral datasets. Make identification assumptions explicit and test robustness through sensitivity analyses and validation. Use this evidence to evaluate whether simulations reproduce intervention effects and support defensible counterfactual predictions across populations and settings.
Individual and Longitudinal Evaluation: Evaluate whether simulations capture stable differences between individuals and changes in their behavior over time, beyond what population averages or simple baselines predict. Build benchmarks using repeated observations, temporal holdouts, and individual trajectories. Ideally, extend this work to interactions between people, validating multi-agent behavior against observed human behavior.
What you will work on
You will own a defined evaluation problem or workstream, with scope and independence matched to your experience. Depending on the workstream, you may build on existing datasets and benchmarks or develop new studies, data resources, and evaluation infrastructure. Your work may include:
Build the measurement layer for behavioral simulation: Design evals, metrics, rubrics, datasets, dashboards, and workflows that measure whether Simile’s models are accurately predicting human behavior across customer use cases, populations, question types, and decision contexts.
Partner with modeling to improve models: Evaluate new model versions, diagnose regressions, identify priority areas for model-improvement cycles, and maintain stable eval suites that represent capabilities customers actually care about.
Contribute to product and applied evals: Build evals for qualitative responses, retrieval, survey generation, AI-generated research reports, customer-facing outputs, and other product surfaces where model quality directly shapes customer trust. Turn subjective quality concerns into concrete rubrics, labeled data, automated graders, release criteria, and model-improvement signals.
Make ground truth and uncertainty legible: Develop rigorous ways to compare simulated responses against human data, customer studies, Simile-collected ground truth, and behavioral datasets. Help the company reason about sampling error, uncertainty, calibration, margin of error, representativeness, and what “ground truth” means when human behavior is inherently noisy.
Automate evaluation workflows: Use modern agentic coding tools to rapidly build internal tools, inspect model outputs, create labeling workflows, validate evals, and turn fuzzy evaluation questions into working systems. We value people who can compress long, ambiguous projects into fast, useful prototypes without losing sight of rigor or reliability.
Expand what we can evaluate: Develop repeatable benchmarks for behavioral predictions across transaction and product-interaction data, intervention effects, individual trajectories, and emerging multi-agent settings.
Requirements
Must Haves
Evaluation and Measurement Judgment: You can show how you turned an ambiguous quality question into a validated measurement that informed a decision. You investigate surprising results by inspecting individual examples, checking data and assumptions, and testing whether the evaluation itself is misleading. You compare against meaningful baselines, explain what your measurement misses, and revise your methods or conclusions when the evidence warrants it. Examples can come from research, industry, or independent projects.
LLM and Model Fluency: You can work with LLM outputs, understand modeling team needs, and reason about whether a model change improved the behavior we care about. You can learn unfamiliar model concepts quickly and collaborate effectively on evaluation and model improvement. Prior specialization in LLM training or post-training is not required.
Statistical Judgment: You are comfortable reasoning about noisy data, uncertainty, sampling, distributions, calibration, confidence intervals, measurement validity, bias, variance, and the difference between an observed result and the underlying population quantity it estimates.
Technical and Agentic Execution: You can build internal tools, scripts, dashboards, labeling workflows, analyses, or automated eval pipelines quickly. You are comfortable working with data and automation tools such as Python, SQL, R, notebooks, LLM APIs, and agentic coding tools such as Codex, Claude Code, Cursor, or equivalent systems. You know how to move quickly while still validating outputs, catching errors, and planning for the long term.
Hands-On Ownership and Collaboration: You drive workstreams while doing the work yourself: building prototypes, inspecting data, debugging workflows, and improving methods. When data or infrastructure are incomplete, you make progress with a defensible first test while working with partners to resolve dependencies. You communicate findings and limitations clearly, make pragmatic scope decisions, and follow through until the result is useful.
Additional Useful Experience
Alongside the priority areas above, we also welcome complementary strengths in:
Model and Product Evaluation: Experience building regression suites, benchmark datasets, model-comparison dashboards, automated graders, release criteria, or evaluations for customer-facing research products.
Survey Methodology and Statistics: Experience with sampling, weighting, power analysis, uncertainty quantification, Bayesian modeling, psychometrics, polling, or measurement theory.
We welcome specialists and generalists from LLM evaluation, research engineering, statistics, human data, market research, UXR, behavioral science, computational social science, and related fields. For experienced candidates, we look for depth that expands the team’s capabilities. For earlier-career candidates, we look for strong fundamentals, rapid learning, and concrete examples of building and validating analyses, studies, or tools.
We welcome applicants who are new to behavioral simulation or have done rigorous measurement work under other job titles. If you bring strong measurement judgment and the ability to build, we would love to hear from you.
Compensation & Benefits
At Simile, we provide competitive compensation packages that include base salary, equity, and comprehensive benefits.
Salary Range: $200,000 – $400,000 USD
Note: Final offers are based on experience, specialized skills, interview performance, and relevant training.
Equity: Grants are available for eligible roles, subject to board approval.
Health & Wellness: Comprehensive medical, dental, and vision coverage.
Time Off: Flexible time off policies to support work-life balance.
Our Process
We prioritize thoughtful conversations and clear examples of past work. Our hiring journey is designed to help both sides align on fit, working style, and expectations.
Reapplication Policy: To ensure a fair and thorough evaluation for all applicants, Simile observes a 90-day waiting period before reconsidering candidates for the same role.
Commitment to Diversity & Inclusion
Equal Opportunity: Simile is an equal opportunity workplace. We welcome applicants of all backgrounds and identities, valuing an environment where everyone can contribute authentically.
Accommodations: If you require support or reasonable accommodations during the application process due to a disability, please let us know. We are happy to assist.