[index-tts]更新了代码,启动会 hold 住,Failed to load custom CUDA kernel for BigVGAN. Falling back to torch.

2025-10-28 643 views
5
> uv run webui.py
>> GPT weights restored from: checkpoints/gpt.pth
[2025-05-22 10:39:21,070] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-22 10:39:22,762] [INFO] [logging.py:107:log_dist] [Rank -1] DeepSpeed info: version=0.16.8, git-hash=unknown, git-branch=unknown
[2025-05-22 10:39:22,762] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2025-05-22 10:39:22,762] [INFO] [logging.py:107:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1

需要 ctrl +C 取消bigvan 回退 torch 才会继续执行

^C>> Failed to load custom CUDA kernel for BigVGAN. Falling back to torch.
Removing weight norm...
>> bigvgan weights restored from: checkpoints/bigvgan_generator.pth
2025-05-22 10:49:38,546 WETEXT INFO found existing fst: /mnt/data/workspace/ai/tts/index-tts/indextts/utils/tagger_cache/zh_tn_tagger.fst
2025-05-22 10:49:38,546 WETEXT INFO                     /mnt/data/workspace/ai/tts/index-tts/indextts/utils/tagger_cache/zh_tn_verbalizer.fst
2025-05-22 10:49:38,546 WETEXT INFO skip building fst for zh_normalizer ...
2025-05-22 10:49:38,677 WETEXT INFO found existing fst: /mnt/data/workspace/ai/tts/index-tts/.venv/lib/python3.10/site-packages/tn/en_tn_tagger.fst
2025-05-22 10:49:38,677 WETEXT INFO                     /mnt/data/workspace/ai/tts/index-tts/.venv/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
2025-05-22 10:49:38,677 WETEXT INFO skip building fst for en_normalizer ...
>> TextNormalizer loaded
>> bpe model loaded from: checkpoints/bpe.model
* Running on local URL:  http://127.0.0.1:7860
* To create a public link, set `share=True` in `launch()`.

Image

回答

2

NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4

1

并没有

> uv pip install -e .  --no-build-isolation
Resolved 76 packages in 1.45s
      Built indextts @ file:///mnt/data/workspace/ai/tts/index-tts
Prepared 1 package in 173ms
░░░░░░░░░░░░░░░░░░░░ [0/1] Installing wheels...                                                                                                                                                               warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 1 package in 12ms
 + indextts==0.1.1 (from file:///mnt/data/workspace/ai/tts/index-tts)
> ll
总计 180K
drwxrwxr-x 2 ptw ptw 4.0K  4月  1 09:21 assets
lrwxrwxrwx 1 ptw ptw   31  5月 22 09:43 checkpoints -> /mnt/models/IndexTeam/index-TTS
-rw-rw-r-- 1 ptw ptw 2.2K  4月  1 09:21 DISCLAIMER
-rw-rw-r-- 1 ptw ptw  12K  4月  1 09:21 INDEX_MODEL_LICENSE
drwxrwxr-x 7 ptw ptw 4.0K  5月 24 00:10 indextts
-rw-rw-r-- 1 ptw ptw  12K  4月  1 09:21 LICENSE
-rw-rw-r-- 1 ptw ptw   65  5月  6 17:49 MANIFEST.in
drwxrwxr-x 3 ptw ptw 4.0K  5月 23 16:53 outputs
drwxrwxr-x 2 ptw ptw 4.0K  4月 25 16:19 prompts
-rw-rw-r-- 1 ptw ptw  12K  5月 19 11:45 README.md
-rw-rw-r-- 1 ptw ptw  423  5月 22 10:39 requirements.txt
-rw-rw-r-- 1 ptw ptw  68K  5月 23 16:53 run.log
-rwxrwxr-x 1 ptw ptw   96  5月 22 10:37 run.sh
-rw-rw-r-- 1 ptw ptw 1.5K  5月  6 17:49 setup.py
drwxrwxr-x 2 ptw ptw 4.0K  4月  1 09:29 test
-rw-rw-r-- 1 ptw ptw  602  4月  1 09:33 test.py
drwxrwxr-x 2 ptw ptw 4.0K  5月 19 11:45 tests
drwxrwxr-x 3 ptw ptw 4.0K  4月  1 09:21 tools
-rw-rw-r-- 1 ptw ptw 2.9K  5月 22 11:50 webui.py
-rw-rw-r-- 1 ptw ptw  11K  5月 22 10:18 webui.py.new
-rw-rw-r-- 1 ptw ptw 2.9K  5月 22 10:28 webui.py.old
> vim webui.py
> cp webui.py.new webui.py
> uv run webui.py
>> GPT weights restored from: checkpoints/gpt.pth
[2025-05-24 00:12:02,904] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-24 00:12:03,599] [INFO] [logging.py:107:log_dist] [Rank -1] DeepSpeed info: version=0.16.8, git-hash=unknown, git-branch=unknown
[2025-05-24 00:12:03,599] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2025-05-24 00:12:03,600] [INFO] [logging.py:107:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1

我现在是通过修改 infer.py 来临时绕开

class IndexTTS:
     def __init__(
-        self, cfg_path="checkpoints/config.yaml", model_dir="checkpoints", is_fp16=True, device=None, use_cuda_kernel=None,
+        self, cfg_path="checkpoints/config.yaml", model_dir="checkpoints", is_fp16=True, device=None, use_cuda_kernel=False,
     ):
         """
         Args:
@@ -272,7 +272,7 @@ class IndexTTS:
         if self.gr_progress is not None:
             self.gr_progress(value, desc=desc)
6

你的版本并不是pr里的0.1.4,拉取pr的分支后再试试

git fetch origin pull/180/head:test
git switch test
pip install -e .  --no-build-isolation
4

@yrom 我是直接clone的那个fork,然后执行的pip install -e . --no-build-isolation,还是会在打印WETEXT INFO skip building fst for en_normalizer后卡住,需要按ctrl+c之后,才会打印出来后面的东西。不知道是不是我哪里操作的不对? windows cuda12.8,python 3.10

$ python webui.py W0527 11:57:10.710000 19732 site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs. Detected CUDA files, patching ldflags Emitting ninja build file C:\work\github.com\index-tts-fix\indextts\BigVGAN\alias_free_activation\cuda\build\build.ninja... INFO: Could not find files for the given pattern(s). 2025-05-27 11:57:20,389 WETEXT INFO building fst for zh_normalizer ... 2025-05-27 11:57:46,471 WETEXT INFO done 2025-05-27 11:57:46,471 WETEXT INFO fst path: C:\work\github.com\index-tts-fix\indextts\utils\tagger_cache\zh_tn_tagger.fst 2025-05-27 11:57:46,471 WETEXT INFO C:\work\github.com\index-tts-fix\indextts\utils\tagger_cache\zh_tn_verbalizer.fst 2025-05-27 11:57:46,483 WETEXT INFO found existing fst: C:\ProgramData\anaconda3\envs\index-tts1.5\lib\site-packages\tn\en_tn_tagger.fst 2025-05-27 11:57:46,483 WETEXT INFO C:\ProgramData\anaconda3\envs\index-tts1.5\lib\site-packages\tn\en_tn_verbalizer.fst 2025-05-27 11:57:46,483 WETEXT INFO skip building fst for en_normalizer ...

GPT weights restored from: checkpoints\gpt.pth DeepSpeed▒▒▒▒ʧ▒ܣ▒▒▒▒˵▒▒▒׼▒▒▒▒: No module named 'deepspeed' Failed to load custom CUDA kernel for BigVGAN. Falling back to torch. Removing weight norm... bigvgan weights restored from: checkpoints\bigvgan_generator.pth TextNormalizer loaded bpe model loaded from: checkpoints\bpe.model

  • Running on local URL: http://127.0.0.1:7860
  • To create a public link, set share=True in launch(). Keyboard interruption in main thread... closing server. (index-tts1.5)
4

@itltf512116 pip install -e . --no-build-isolation 要在conda activate index-tts环境后执行,确保pip执行的时候有cuda 版torch。可以加--verbose调试一下输出:pip install -e . --no-build-isolation --verbose 执行正确的话,会在indextts\BigVGAN\alias_free_activation\cuda目录下有个alias_free_activation_cuda.cpxxx.xxx.pyd文件

3

我现在是直接在原来的代码位置执行切换到pull 180这个分支,然后conda activate index1-tts1.5之后,再执行pip install -e . --no-build-isolation --verbose,附件截图是最后的日志,前面有一大票日志,但是太多了,复制不全,就只截图了一部分。然后在目录indextts\BigVGAN\alias_free_activation\cuda 下,产生了一个新文件 anti_alias_activation_cuda.cp310-win_amd64.pyd,然后执行python webui.py后,还是卡住了 Image

你的版本并不是pr里的0.1.4,拉取pr的分支后再试试

git fetch origin pull/180/head:test
git switch test
pip install -e .  --no-build-isolation
3

产生了一个新文件 anti_alias_activation_cuda.cp310-win_amd64.pyd,然后执行python webui.py后,还是卡住了

那你试试写一个test.py:

import torch
print(f"PyTorch 版本: {torch.__version__}")
print(f"CUDA 是否可用: {torch.cuda.is_available()}")
print(f"CUDA 版本: {torch.version.cuda}")

try:
    from indextts.BigVGAN.alias_free_activation.cuda import anti_alias_activation_cuda
    print("Loaded anti_alias_activation_cuda from indextts.BigVGAN.alias_free_activation.cuda", anti_alias_activation_cuda)
except ImportError:
    import importlib
    import os
    try:
        anti_alias_activation_cuda = importlib.import_module("anti_alias_activation_cuda", "indextts.BigVGAN.alias_free_activation.cuda")
        print("Loaded anti_alias_activation_cuda from importlib.import_module", anti_alias_activation_cuda)
    except ImportError as e:
        for path in [
            'indextts/BigVGAN/alias_free_activation/cuda',
            os.path.abspath("indextts/BigVGAN/alias_free_activation/cuda"),
        ]:
            for f in os.listdir(path):
                if f.startswith("anti_alias_activation_cuda") and f.endswith(".pyd"):
                    pyd_file = os.path.join(path, f)
                    break
            print(f"Found anti_alias_activation_cuda at {pyd_file}")
            spec = importlib.util.spec_from_file_location("anti_alias_activation_cuda", pyd_file)
            module = importlib.util.module_from_spec(spec)
            spec.loader.exec_module(module)
            print(f"Loaded anti_alias_activation_cuda from {pyd_file}")        
            break

直接执行看看会不会卡住 @itltf512116

如果没有打印:Loaded anti_alias_activation_cuda xxx 则代表失败了

参考输出:

python.exe d:/Projects/index-tts/test3.py PyTorch 版本: 2.5.1 CUDA 是否可用: True CUDA 版本: 12.4 Loaded anti_alias_activation_cuda from indextts.BigVGAN.alias_free_activation.cuda <module 'indextts.BigVGAN.alias_free_activation.cuda.anti_alias_activation_cuda' from 'd:\Projects\index-tts\indextts\BigVGAN\alias_free_activation\cuda\anti_alias_activation_cuda.cp310-win_amd64.pyd'>

4

这个问题我在linux系统上也碰到了,我通过重新编译解决了

rm -rf /tmp/BigVGAN/cuda/

// ... existing code ...
def load():
    # Check CUDA version and set appropriate compute capabilities
    _, bare_metal_major, bare_metal_minor = _get_cuda_bare_metal_version(cpp_extension.CUDA_HOME)
    cuda_version = float(f"{bare_metal_major}.{bare_metal_minor}")

    # Set compute capabilities based on CUDA version
    cc_flags = []
    if cuda_version >= 11.0:
        cc_flags.extend([
            "-gencode", "arch=compute_70,code=sm_70",
            "-gencode", "arch=compute_75,code=sm_75",
            "-gencode", "arch=compute_80,code=sm_80"
        ])
    else:
        cc_flags.extend([
            "-gencode", "arch=compute_70,code=sm_70",
            "-gencode", "arch=compute_75,code=sm_75"
        ])

    # Build path
    srcpath = pathlib.Path(__file__).parent.absolute()
    buildpath = srcpath / "build"
    _create_build_dir(buildpath)

    # Helper function to build the kernels with optimized flags
    def _cpp_extention_load_helper(name, sources, extra_cuda_flags):
        return cpp_extension.load(
            name=name,
            sources=sources,
            build_directory=buildpath,
            extra_cflags=["-O3"],
            extra_cuda_cflags=[
                "-O3",
                "--use_fast_math",
                "-lineinfo"  # Add debug info for better error messages
            ] + extra_cuda_flags + cc_flags,
            verbose=True,
        )

    extra_cuda_flags = [
        "-U__CUDA_NO_HALF_OPERATORS__",
        "-U__CUDA_NO_HALF_CONVERSIONS__",
        "--expt-relaxed-constexpr",
        "--expt-extended-lambda",
    ]

    sources = [
        srcpath / "anti_alias_activation.cpp",
        srcpath / "anti_alias_activation_cuda.cu",
    ]

    # Handle special characters in path
    buildpath = chinese_path_compile_support(sources, buildpath)

    try:
        anti_alias_activation_cuda = _cpp_extention_load_helper(
            "anti_alias_activation_cuda", sources, extra_cuda_flags
        )
        return anti_alias_activation_cuda
    except Exception as e:
        print(f"CUDA compilation failed with error: {str(e)}")
        raise
// ... existing code ...
#!/usr/bin/env python3
# 自动修复BigVGAN的CUDA扩展导入路径问题

import os
import sys
import re
from pathlib import Path

def patch_import_paths():
    """修复BigVGAN中的CUDA扩展导入路径问题"""
    print("正在修复BigVGAN CUDA扩展导入路径...")

    # 获取项目根目录
    project_root = Path('/workspace')
    bigvgan_path = project_root / 'index-tts' / 'indextts' / 'BigVGAN' / 'bigvgan.py'

    if not bigvgan_path.exists():
        print(f"错误: 找不到文件 {bigvgan_path}")
        return False

    # 读取文件内容
    with open(bigvgan_path, 'r', encoding='utf-8') as f:
        content = f.read()

    # 替换导入路径
    pattern = r'from alias_free_activation\.cuda\.activation1d import'
    replacement = r'from indextts.BigVGAN.alias_free_activation.cuda.activation1d import'

    new_content = re.sub(pattern, replacement, content)

    if new_content == content:
        print("无需修复: 导入路径已经正确")
        return True

    # 备份原文件
    backup_path = str(bigvgan_path) + '.bak'
    with open(backup_path, 'w', encoding='utf-8') as f:
        f.write(content)

    # 写入修复后的内容
    with open(bigvgan_path, 'w', encoding='utf-8') as f:
        f.write(new_content)

    print(f"已修复导入路径并备份原文件到 {backup_path}")
    return True

def compile_cuda_extension():
    """编译CUDA扩展"""
    print("正在编译BigVGAN CUDA扩展...")

    try:
        # 获取项目根目录
        project_root = Path('/workspace')
        cuda_dir = project_root / 'index-tts' / 'indextts' / 'BigVGAN' / 'alias_free_activation' / 'cuda'

        # 切换到CUDA目录
        os.chdir(cuda_dir)

        # 确保环境变量设置正确
        os.environ['PYTHONPATH'] = f"{project_root}/index-tts:{os.environ.get('PYTHONPATH', '')}"

        # 导入并运行编译函数
        sys.path.insert(0, str(project_root / 'index-tts'))

        from indextts.BigVGAN.alias_free_activation.cuda.load import load

        # 编译CUDA扩展
        result = load()

        print("CUDA扩展编译成功!")
        return True
    except Exception as e:
        print(f"编译CUDA扩展时出错: {e}")
        import traceback
        traceback.print_exc()
        return False

if __name__ == "__main__":
    print("开始修复BigVGAN CUDA扩展...")

    # 修复导入路径
    if patch_import_paths():
        # 编译CUDA扩展
        if compile_cuda_extension():
            print("所有修复已完成!")
        else:
            print("CUDA扩展编译失败。")
    else:
        print("导入路径修复失败。") 
6

@yrom 感谢支持! 这个是执行python test.py的输出:

PyTorch 版本: 2.7.0+cu128 CUDA 是否可用: True CUDA 版本: 12.8 Loaded anti_alias_activation_cuda from indextts.BigVGAN.alias_free_activation.cuda <module 'indextts.BigVGAN.alias_free_activation.cuda.anti_alias_activation_cuda' from 'C:\work\github.com\index-tts1.5\indextts\BigVGAN\alias_free_activation\cuda\anti_alias_activation_cuda.cp310-win_amd64.pyd'>

2

好像是用cmd可以,用bash不行

0

@itltf512116 通过pip install -e . --no-build-isolation的方式安装,启动webui.py 不会再卡住了是不?

5

是的,现在是在cmd里面启动正常,在bash里面启动还是卡住 @yrom ,应该是代码生效了,我看他第一次发的截图应该是cmd环境会卡住,我是一直都在bash里面,今天是要给你发截图发现有乱码,才去的cmd环境是执行,结果发现cmd是可以的

9

bash是git-bash吗?可能存在dll兼容性问题。Windows上最好用cmd或powershell启动

5

bash是git-bash吗?可能存在dll兼容性问题。Windows上最好用cmd或powershell启动

是的git-bash,原来的代码(直到1.5)是可以的,就拉取更新到最近的版本之后,突然就卡住了

6

好的,谢谢各位大佬。拉了最新的代码,直接运行 ok 了