Principal Member Technical Staff - Data engineer
Nielsen · Bengaluru
Company Description
At Nielsen, we are passionate about our work to power a better media future for all people by providing powerful insights that drive client decisions and deliver extraordinary results. Our talented, global workforce is dedicated to capturing audience engagement with content - wherever and whenever it’s consumed. Together, we are proudly rooted in our deep legacy as we stand at the forefront of the media revolution. When you join Nielsen, you will join a dynamic team committed to excellence, perseverance, and the ambition to make an impact together. We champion you, because when you succeed, we do too. We enable your best to power our future.
Job Description
Nielsen’s Digital Content AI team builds and operates data-processing pipelines that turn digital audience and interaction signals into reliable, analytics-ready datasets. As a principal-level individual contributor, you will bring a Solution first mindset and strong data understanding to the full lifecycle: understand new data sources, build reusable ingestion and transformation workflows, validate and enrich data, publish it to downstream data platforms, and keep production processing dependable. You will lead multiple cross-functional projects through measurable business deliverables, providing technical direction through expertise and influence rather than formal people management. You will work across mobile, Connected TV (CTV), desktop/laptop and web use cases, partnering with product, data collection, platform, quality and downstream analytics teams.
- Lead the technical design and delivery of batch and near-daily data pipelines, from raw inputs through curated outputs and publication to the Master Data Layer (MDL).
- Profile and interpret incoming telemetry from diverse data sources; identify data gaps and translate source formats into dependable processing logic.
- Apply an AI-first approach to engineering and operations: identify appropriate opportunities for AI-assisted data exploration, code and test generation, classification, anomaly detection and workflow automation; validate outputs rigorously and use AI responsibly with appropriate privacy and security controls.
- Design and operate real-time and streaming event workflows alongside batch pipelines, managing event validation, schema evolution, ordering and duplication, late or replayed events, processing latency, backpressure and failure recovery.
- Build and maintain parsers, configuration-driven workflows, event/session logic, business rules and schema mappings for application usage and in-app activity across mobile, CTV, desktop/laptop and web products.
- Develop data-stitching and transformation logic that correlates relevant sources, derives sessions and engagement measures, and produces consistent intermediate and curated datasets.
- Deliver data processing for streaming/video, e-commerce and advertising use cases, including event classification, purchase and product/category/search activity, app and web usage, and incremental dataset updates.
- Integrate reference and enrichment data—including content metadata and classification inputs—where required; collaborate with machine-learning and platform teams to operationalize model-based classification and enrichment.
- Build reusable frameworks for onboarding new applications, platforms, data sources and markets; support configuration, historical backfills, reprocessing and safe daily refreshes.
- Orchestrate and operate workflows using tools such as Airflow and AWS workflow services; monitor schedules, dependencies, retries, failures and data availability, and troubleshoot production issues through root-cause analysis.
- Implement and evolve automated data-quality controls, including schema and completeness checks, freshness and volume monitoring, null/range/duplicate checks, event consistency, stitching validation and reconciliation between processing layers and MDL.
- Create and use operational dashboards, alerts and job-level telemetry to make pipeline health, data quality, performance and processing costs visible; improve reliability and efficiency based on those signals.
- Optimize distributed processing and cloud data layouts for correctness, scalability, runtime and cost, working with technologies such as Spark, Parquet, Amazon S3, AWS Glue, Athena and EMR.
- Contribute to design reviews, code reviews, testing strategy, release readiness, runbooks and technical documentation; maintain clear data contracts, schemas and operational ownership boundaries.
- Provide technical direction and mentoring to engineers, lead moderately complex initiatives through influence, and communicate risks, dependencies, estimates and delivery status to cross-functional partners.
- Drive multiple projects or workstreams end to end, translating business needs into scope, milestones and technical plans; coordinate dependencies and stakeholders; and remain accountable for quality, adoption, timelines and measurable business outcomes—not people-management responsibilities.
- Handle sensitive data responsibly and follow Nielsen’s security, privacy, access-control and engineering practices.
Qualifications
- Typically 10–15 years of relevant data engineering or data processing experience, or equivalent depth of expertise delivering complex production data systems.
- Bachelor’s degree in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience; a relevant advanced degree or specialized certification is a plus.
- Expert-level SQL and strong programming skills in Python and/or Scala/Java, with substantial hands-on experience building distributed data-processing workloads using Apache Spark or a comparable framework.
- Proven experience designing, deploying and supporting production data pipelines and data-lake/lakehouse workflows on AWS or a comparable cloud platform.
- Practical experience with object storage and columnar formats such as S3 and Parquet, workflow orchestration, metadata/catalog services, and query engines such as Athena or equivalent.
- Strong understanding of data semantics and modeling, source-to-target lineage, schema evolution, data contracts, event-driven and batch-processing patterns, partitioning, incremental loads, backfills and idempotent/recoverable processing.
- Hands-on knowledge of real-time event processing concepts, including delivery guarantees, event-time handling, ordering, deduplication, replay, latency and backpressure.
- An AI-first engineering mindset and practical judgment about where AI/ML can improve data understanding, processing, quality or operational efficiency, with the ability to verify AI-generated outputs.
- Demonstrated ability to investigate unfamiliar or inconsistent source data, convert business rules into testable processing logic, and diagnose issues across multiple pipeline stages.
- A track record of independently driving multiple complex, cross-functional projects from problem definition through production delivery, with clear ownership of business outcomes and no reliance on formal people-management authority.
- Experience building data-quality checks, reconciliation, monitoring and alerting for production data products.
- Strong analytical and problem-solving skills, sound technical judgment, and a track record of independently owning complex deliverables from discovery through production support.
- Excellent written and verbal communication skills, including the ability to explain technical trade-offs and coordinate delivery across teams.
Additional Information
- Familiarity with stitching diverse telemetry, user activity, and digital interaction signals to derive session- or content-level metrics.
- Experience with content metadata/reference-data enrichment, entity matching, e-commerce event classification, advertising datasets or ML-assisted data processing.
- Experience with AWS services such as Kinesis, Firehose, Lambda, Step Functions, Glue, Athena and EMR, and with Grafana or Datadog for operational observability.
- Experience improving reusable onboarding frameworks, pipeline resilience, data lineage, operational cost visibility or large-scale backfill processes.
Please be aware that job-seekers may be at risk of targeting by scammers seeking personal data or money. Nielsen recruiters will only contact you through official job boards, LinkedIn, or email with a nielsen.com domain. Be cautious of any outreach claiming to be from Nielsen via other messaging platforms or personal email addresses. Always verify that email communications come from an @nielsen.com address. If you're unsure about the authenticity of a job offer or communication, please contact Nielsen directly through our official website or verified social media channels.