Senior AI Inference Engineer (LLM & Speech Models)

Senior AI Inference Engineer (LLM & Speech Models)

仕事概要

In short

You'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.

• *\Team language:\* English (Japanese not required)
• *\Salary:\* ¥10M–¥20M per year, plus stock options
• *\Location:\* Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.

You might be a fit if you are…

• An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)
• A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems
• An engineer who has optimized throughput and cost for inference workloads on large GPU clusters

About the role

At Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.

This isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.

On the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.

What this role is \not\

• Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.
• GPU cluster operations and general DevOps are handled by a separate team.
• Writing CUDA kernels or building an inference engine from scratch is not the main job.
• RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.

What you'll do

1. Build a common inference framework for speech models

• Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models
• Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models
• Build the inference APIs, SDKs and tooling our model teams use every day

2. Optimize large-scale batch inference for audio

• Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes
• Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O
• Measure and remove bottlenecks based on end-to-end processing time and cost

3. Adapt and optimize LLM inference

• Stand up LLM inference on mature engines such as vLLM and SGLang
• Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache
• Design scheduling that accounts for prefill/decode characteristics and task priority

4. Plan GPU usage for inference workloads

• Handle GPU allocation and capacity planning for batch inference and evaluation
• Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray
• Track and reduce GPU usage and inference cost per model and per task

We're happy to go into more detail during the intro call and interviews.

Why this role

• *\Set the playbook in an area that doesn't have one yet.\* Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.
• *\Own inference end to end.\* You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.
• *\Directly move the business.\* Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.

Tech stack

• *\Languages & frameworks:\* Python, PyTorch
• *\LLM inference:\* vLLM, SGLang, TensorRT-LLM, Triton Inference Server
• *\Speech models:\* ASR, TTS, speech-to-speech, audio codecs, vocoders
• *\Orchestration:\* Kubernetes, Slurm, Ray
• *\Low-level optimization (nice to have):\* C++, CUDA, Triton, NCCL

Our product

We build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.

Our edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.

Hiring process

Intro call → Case assignment & first interview → Paid work trial* → Final interview → Offer meeting

\*The work trial is 1–2 consecutive days at our Tokyo office, working on real tasks alongside the team, and it's paid. It's a chance for both sides to check fit and get a feel for the work.

• *\If you're based outside Japan,\* we'll agree on the format together. We won't ask you to travel to Japan just for this step.
• If your schedule or your current employer's side-work policy makes the trial difficult, just let us know.
• We may also ask for references.

About Recho

Recho develops foundation models for customer-support AI agents and also deploys them for clients, all within one organization. Our core strength is voice AI technology we built entirely in-house, which resolves the industry's long-standing trade-off between quality and cost.

We've closed a Series A of approximately ¥700M and are rapidly rolling out to demanding production environments at major companies such as SBI Shinsei Bank and KDDI.

Learn more

• Corporate website
• Team interviews (note, in Japanese)
• Company deck
• Case study: Partnership with KDDI and Altius Link (Japanese)
• Case study: SBI Shinsei Bank (Japanese)
• 2-minute voice demo — give it a listen!

必須スキル

In short

You'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.

• *\Team language:\* English (Japanese not required)
• *\Salary:\* ¥10M–¥20M per year, plus stock options
• *\Location:\* Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.

You might be a fit if you are…

• An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)
• A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems
• An engineer who has optimized throughput and cost for inference workloads on large GPU clusters

About the role

At Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.

This isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.

On the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.

What this role is \not\

• Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.
• GPU cluster operations and general DevOps are handled by a separate team.
• Writing CUDA kernels or building an inference engine from scratch is not the main job.
• RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.

What you'll do

1. Build a common inference framework for speech models

• Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models
• Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models
• Build the inference APIs, SDKs and tooling our model teams use every day

2. Optimize large-scale batch inference for audio

• Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes
• Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O
• Measure and remove bottlenecks based on end-to-end processing time and cost

3. Adapt and optimize LLM inference

• Stand up LLM inference on mature engines such as vLLM and SGLang
• Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache
• Design scheduling that accounts for prefill/decode characteristics and task priority

4. Plan GPU usage for inference workloads

• Handle GPU allocation and capacity planning for batch inference and evaluation
• Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray
• Track and reduce GPU usage and inference cost per model and per task

We're happy to go into more detail during the intro call and interviews.

Why this role

• *\Set the playbook in an area that doesn't have one yet.\* Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.
• *\Own inference end to end.\* You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.
• *\Directly move the business.\* Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.

Tech stack

• *\Languages & frameworks:\* Python, PyTorch
• *\LLM inference:\* vLLM, SGLang, TensorRT-LLM, Triton Inference Server
• *\Speech models:\* ASR, TTS, speech-to-speech, audio codecs, vocoders
• *\Orchestration:\* Kubernetes, Slurm, Ray
• *\Low-level optimization (nice to have):\* C++, CUDA, Triton, NCCL

Our product

We build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.

Our edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.

Hiring process

Intro call → Case assignment & first interview → Paid work trial* → Final interview → Offer meeting

\*The work trial is 1–2 consecutive days at our Tokyo office, working on real tasks alongside the team, and it's paid. It's a chance for both sides to check fit and get a feel for the work.

• *\If you're based outside Japan,\* we'll agree on the format together. We won't ask you to travel to Japan just for this step.
• If your schedule or your current employer's side-work policy makes the trial difficult, just let us know.
• We may also ask for references.

About Recho

Recho develops foundation models for customer-support AI agents and also deploys them for clients, all within one organization. Our core strength is voice AI technology we built entirely in-house, which resolves the industry's long-standing trade-off between quality and cost.

We've closed a Series A of approximately ¥700M and are rapidly rolling out to demanding production environments at major companies such as SBI Shinsei Bank and KDDI.

Learn more

• Corporate website
• Team interviews (note, in Japanese)
• Company deck
• Case study: Partnership with KDDI and Altius Link (Japanese)
• Case study: SBI Shinsei Bank (Japanese)
• 2-minute voice demo — give it a listen!

歓迎スキル

Nice-to-haves

*\Advanced models & speech\*

• Inference experience with audio generation models (diffusion, flow matching) or audio codecs
• Hands-on experience across multiple model types, such as ASR, TTS and speech-to-speech
• Interest or experience in building and optimizing RL rollout inference environments

*\Performance & low-level optimization\*

• Designing dynamic batching for variable-length audio, or distributed/parallel inference across multiple GPUs and nodes
• Working knowledge of C++, CUDA, Triton and NCCL, and using profilers to pinpoint bottlenecks
• Managing and optimizing GPU cost and compute budgets

*\Platform design & technical leadership\*

• Designing internal AI frameworks, inference libraries or model SDKs
• Open-source contributions to vLLM, SGLang, PyTorch, Ray or similar
• Setting technical roadmaps, design standards or code review guidelines

*\Languages\*

• Japanese or Chinese. Our team works in English, but Japanese (the company's working language) or Chinese (spoken by many of our colleagues) makes cross-team collaboration even smoother.

求める人物像

What we look for

• *\Hands-on:\* You don't stop at architecture. You're on the front line writing the code and care about getting the implementation right.
• *\Data-driven:\* You tackle performance and cost problems with profilers and benchmarks, not gut feel, and you optimize for the whole system.
• *\Pragmatic:\* You don't reinvent the wheel. You pick the right open-source tools and frameworks and extend only what we actually need.
• *\Platform-minded:\* You enjoy building tools and libraries that make other teams, especially model developers, more productive. That's Team-First in practice.

Our values

These are the values our team relies on to deliver Recho's value as fast as possible.

• *\Exceed Expectations:\* We keep exceeding the expectations of customers and teammates by delivering both speed and quality.
• *\Team-First:\* Each of us takes professional ownership, and we maximize the team's results, not individual ones.
• *\Continuous Feedback:\* We engage seriously and speak candidly for the sake of results. The harder something is to say, the more we say it with honesty, respect and care.
• *\Earn Trust:\* Trust is the foundation of every collaboration and every challenge. We act ethically and keep personal and company matters separate. Trust takes time to build and a moment to lose.

応募概要

給与

*\Salary\*
Compensation is based on your current compensation, skills and experience.

• Annual salary: ¥10M–¥20M
• Salary review: Once a year, based on your contribution and company performance
• Stock option plan

勤務地

WeWork Hanzomon (Hanzomon PREX North, 2-3-2 Kojimachi, Chiyoda-ku, Tokyo)
Hybrid: mostly in the office, with some remote work. For candidates based outside Japan, remote arrangements are open to discussion.

雇用形態

Employment type: Full-time, permanent. Starting with a paid contract engagement first is also an option.

勤務体系

*\Working hours\*

• Discretionary work system (professional type under Japanese labor law)
• Deemed working hours: 9 hours/day
• Start and end times are up to you. As a general guideline, the team's core working window is 10:00–19:00, adjusted as work requires.

*\Scope of duties and work location\*
This disclosure is required under Japanese labor law.

• Duties: AI inference infrastructure development at hire; subject to change as determined by the company
• Work location: Hanzomon WeWork at hire (some remote work possible); subject to change as determined by the company

*\Holidays & leave\*
Weekends, Japanese public holidays, year-end/New Year holidays (Dec 29–Jan 3), special leave, and paid annual leave (10 days in the first year)

試用期間

Probation period: 3 months, with no change in terms

福利厚生

*\Insurance\*

• Full Japanese social insurance (health, pension, employment)
• Commuting allowance (up to ¥25,000/month)

*\Office\*

• WeWork Hanzomon, 2 minutes from Hanzomon Station, with a dedicated floor, focus booths and meeting rooms
• Free drinks, including coffee, soft drinks and alcohol
• High-quality office chairs
• MacBook Air and external monitor provided

*\Productivity\*

We build AI for our customers, so we work AI-native ourselves.

• *\AI tools support\*
• Engineering roles: Top-tier plans for Claude Code and Codex
• Business & PM roles: Team plans that let you share context across your team
• Amazon Business account for work-related books and equipment

*\Team & communication\*

Beyond project meetings and day-to-day conversations on Slack, we regularly set aside time for the whole company to share progress, discuss it together and get to know each other.

• *\Monday Kickoff:\* Every Monday, we share everything from work updates to little things going on in our lives.
• *\Monthly All-Team Meeting:\* On the first Friday of each month, each team shares its results from the previous month. Afterward, we get together over food.
• *\All Hands:\* Twice a year, we share company direction and our mid- to long-term vision, bringing the whole organization together.

*\Smoking\*

• Indoor smoking prohibited; outdoor smoking area available