VoxLock keeps the voice of one speaker (the caller) and removes the other
voices. It runs in the offline and online speech denoisers next to GTCRN
and DPDFNet, with two ONNX models: an enroll model that runs once on a
voice sample of the caller and a chunk model that runs on each chunk of
the audio and carries the LSTM state.
Without a voice sample, the offline denoiser uses the loudest 3 seconds
in the first 15 seconds of the input, and the online denoiser passes
audio through until it has heard 3 seconds of speech. A voice sample can
be given with --speech-denoiser-voxlock-voice-sample.
The voice sample is optional, so instead of exiting, log a warning and
take the voice sample from the input audio as if none was given.
The prompt is fed at positions 0..n-1, but generated tokens were fed at
1, 2, ... instead of n, n+1, ..., overlapping the prompt. Feed them at
decoder_input.size() + i - 1, as NeMo's GreedySequenceGenerator does.
Fixes#4096
Fixes#3635
It is for v2.0.0 and breaks the ABI.
CircularBuffer::head_ and tail_ are absolute sample counters that grow
forever. As int32_t they overflow after 2^31 samples (~37 h at 16 kHz);
tail_ % capacity then goes negative and std::copy() writes to an invalid
iterator, crashing long-running VAD streams.
Widen every absolute stream position to int64_t. Counts and capacities
(Push/Pop/Get's n, Size(), capacity) stay int32_t — they are bounded by
buffer memory, not stream duration.
NeMo's AudioToMelSpectrogramPrefab computes log(x + 2**-24), while the
shared fbank extractor applies a hard floor, log(max(x, FLT_EPSILON)).
On quiet recordings a large share of the mel bins sit at that floor and
per-feature normalization turns them into constant columns the model
never saw in training, which degrades accuracy a lot (issue #3997).
For NeMo models (nemo_normalize_type is set), disable use_log_fbank and
apply log(x + 2**-24) to the raw mel energies in GetFrames() before the
per-feature normalization. Other models keep the existing behavior.
Bump the dependency and select the additive guard with eps = FLT_EPSILON / 2 for NeMo fbank models. The manual log pass in GetFrames() is no longer needed, which also keeps MFCC models such as TeleSpeech untouched.
The wav2vec2-style convolutional front end of omnilingual ASR models needs at
least 400 samples. For shorter input onnxruntime throws an Ort::Exception from
DecodeStream that nothing catches, so sherpa-onnx-offline and C API users abort,
Python users get a RuntimeError, and a batch passed to DecodeStreams loses the
results of every stream. Empty input also reached f.size() / feat_dim with
feat_dim == 0. Return an empty result for such input instead.
Both examples passed each 0.1 s read straight to accept_waveform(), which expects
window-sized input. Buffer the reads and feed window_size samples per call, as the
other VAD examples do.
A lead byte announcing a 2-4 byte character near the end of the input was
copied as a whole character even when fewer bytes remained, reading past
the end of the string. This happens for TTS text that ends in the middle
of a character, which reaches SplitUtf8 through the lexicon front ends.
Treat such a lead byte like any other invalid byte and skip it.
The timestamps were built as str(timedelta(seconds=t))[:-3]. That only
works when t has a fractional part: for a whole number of seconds
str(timedelta) has no ".ffffff" suffix, so the slice cuts off the
seconds instead, e.g. a segment starting at 0 s was written as
"0:00 --> 0:00:01,500". Hours were also written with a single digit,
while SRT expects HH:MM:SS,mmm.
Format the timestamps explicitly as HH:MM:SS,mmm for both the .srt
and the .txt output.
- profile_output_dirs(): under a custom Cargo profile (e.g. dx's android-dev), PROFILE is
debug/release but the directory is named after the profile. Fall back to Cargo's fixed
layout, OUT_DIR.ancestors().nth(3).
- download_prebuilt_libs(): only consult the cached jniLibs/<abi> directory for Android; iOS
aarch64 maps to the same arm64-v8a name and picked up Android libraries.
- iOS: reuse an already extracted xcframework instead of re-extracting, which deleted the
other target's lib dir (device lib/ vs simulator lib-sim/).
ToFloat() de-interleaves the captured frames, so samples1_ holds one
float per frame (the first channel), while samples_ still holds
actual_channel_count_ int16 samples per frame. Passing samples_.size()
as input_dim to LinearResample::Resample() made it read twice as many
floats as samples1_ contains whenever the device runs with 2 channels
and a sample rate that needs resampling, reading past the end of the
buffer and producing a too long output with garbage tail.
With keywords that cannot be encoded, e.g. a token missing from tokens.txt, the C++
CreateStream logs the error and returns nullptr, which reached Python as None. The
caller then failed later with "'NoneType' object has no attribute 'accept_waveform'".
Raise a ValueError naming the keywords instead.
Fixes#3731
espeak-ng and piper-phonemize are no longer built into sherpa-onnx.
Text frontends are now:
- **lexicon** — built in, no extra dependency
- **phonemize** — phonemize outside sherpa-onnx (e.g. [piper-phonemize](https://github.com/csukuangfj/piper-phonemize)) and pass the result in. That library is GPL, so it stays optional: sherpa-onnx is GPL-free after this PR.
## API
- `SherpaOnnxGenerationConfig` gains `phoneme_codepoints` and `tokens` (mutually exclusive alternatives to `text`)
- `data_dir` on the TTS model configs is deprecated and ignored; the ABI slot is kept
- all bindings updated: C/C++, Python, Java, Kotlin, Go, Rust, C#, Swift, Pascal, Dart/Flutter, node-addon, WASM, HarmonyOS
The three RUSTSEC advisories against zip 2.x (RUSTSEC-2026-0367 SUID/SGID bits, RUSTSEC-2026-0368 symlink escape, RUSTSEC-2026-0369 permissions through symlinks) are fixed in zip >= 9 only. The zip usage in build.rs is confined to the iOS .xcframework.zip branch and its ZipArchive::new/extract API is unchanged in 9.x, so the bump is source-compatible: verified with `DOCS_RS=1 cargo check` against zip 9.0.4.
Fixes#4058
The C struct SherpaOnnxOfflineRecognizerConfig gained
ctc_fst_decoder_config, but the Swift helper constructing it was not
updated, so the swift CI workflow fails to compile (missing argument
for parameter 'ctc_fst_decoder_config'). Mirror the online wrapper:
add sherpaOnnxOfflineCtcFstDecoderConfig() and pass the new field
(with a default value so existing callers are unaffected).
An exception escaping a JNI boundary calls std::terminate and aborts the
process. In production we observed an Ort::Exception raised while loading
a corrupt model file inside OfflineRecognizer::newFromFile take down the
whole app, uncatchable from Kotlin.
Wrap every model-loading JNI constructor (offline/online recognizer,
offline/online punctuation, both the file and the asset variants) in a
try/catch that converts the C++ exception into a Java exception via
ThrowNew, following the existing pattern in jni/wave-reader.cc. The
helper RethrowCxxExceptionAsJava lives in jni/common.h.
Fixes#3987
* Add subtitle generation Python API example for Doubao ASR.
generate-subtitles-doubao.py uses silero-vad/ten-vad for segmentation and
the Doubao bigmodel recording-file recognition API for ASR. Audio segments
are sent as base64 encoded wav data, so no public URL is needed. Results
are written to <input>.srt and <input>.txt.
Also for generate-subtitles.py:
- Add --max-duration-seconds to process only the first N seconds of the
input file
- Refuse to run if <input>.srt or <input>.txt already exists
Arbaz Siddiqui
Hassan Kazi
Fangjun Kuang
DR1N0
sunerpy
Kyozzz
Emmanuel BORGES
Stef
losewayy
Sergio Fernández
Hulk
dxhome
Anthony
olerast
Mohammad Hijjawi
Pratik Gandhi
James Winegar
Dime
beasaltfish
Cassian433
Elian Doran
Luís Felipe Mesquita Granja
Jasper
Paolo Antinori