Data Scientist
Black Canyon Consulting · Bethesda
Overview
We are seeking a Data Scientist to support applied machine learning, natural language processing, AI evaluation, data quality, and analytical product development for a federal permitting modernization initiative.
The Data Scientist will work across structured and unstructured data, including public comments, permitting records, compliance information, documents, and geospatially linked datasets. A key focus of the role will be developing representative evaluation datasets and quantitative methods to measure the performance, reliability, and limitations of AI-enabled and agentic solutions.
This position is remote.
Duties & Responsibilities
- Develop analytical and machine-learning approaches supporting permitting and natural-resource use cases.
- Build and maintain labeled datasets, evaluation corpora, and benchmark datasets for AI-enabled services.
- Define quantitative evaluation methodologies for classification, extraction, retrieval, summarization, entity matching, and other AI/NLP capabilities.
- Evaluate LLM and agentic workflows, including output quality, retrieval performance, task completion, failure modes, and reliability.
- Analyze public-comment datasets and develop reusable features and evaluation datasets for comment-processing services.
- Apply NLP and machine-learning techniques to structured and unstructured data.
- Evaluate embedding models, semantic retrieval strategies, document-ranking techniques, and AI-generated outputs.
- Develop statistical approaches for detecting anomalies, reconciliation issues, missing data, and inconsistent records.
- Partner with Content Pipeline Engineers to measure OCR, extraction, and document-retrieval performance.
- Partner with Data Engineers to ensure analytical datasets and evaluation pipelines are reproducible and production-ready.
- Support entity matching and harmonization across data originating in multiple source systems.
- Design experiments and validation approaches with clearly defined success metrics.
- Contribute to certified analytical data products, metrics, and governed datasets.
- Develop reproducible Python analyses, models, evaluation pipelines, and technical documentation.
- Communicate model performance, limitations, uncertainty, and failure modes to technical and non-technical stakeholders.
Educational Requirements
- Bachelor's degree in Data Science, Statistics, Computer Science, Mathematics, Engineering, or a related quantitative discipline.
Requirements
- 5+ years of relevant data science, applied machine learning, or quantitative analytics experience.
- Strong Python and SQL skills.
- Experience with statistical analysis, machine learning, NLP, and unstructured text data.
- Experience designing labeled datasets, benchmark datasets, and train/test evaluation frameworks.
- Experience evaluating LLM, RAG, or agentic AI solutions, including retrieval quality, output accuracy, and failure modes.
- Strong understanding of model-evaluation techniques, including precision, recall, F1, ranking metrics, confusion matrices, and related measures.
- Experience with embeddings, semantic retrieval, or vector-based search.
- Experience developing reproducible analytical and evaluation workflows.
- Ability to distinguish data-quality issues from model-performance issues.
- Strong ability to communicate analytical results, limitations, and uncertainty.
Bonus Skills
- Experience with agent evaluation frameworks or LLM observability tools.
- Experience with agentic frameworks such as LangChain, LangGraph, or similar orchestration frameworks.
- Experience with OpenSearch, vector databases, or enterprise search platforms.
- Experience with document AI or OCR pipelines.
- Experience with geospatial analysis.
- Experience developing entity-resolution, record-linkage, or data-quality models.
- Experience working in the U.S. Federal Government space.
- Experience with permitting, regulatory, environmental, energy, or natural-resource datasets.
Benefits and Salary
We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement.
If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!
More jobs at Black Canyon Consulting · Data & Analytics jobs in Bethesda