“cpu inference”的搜索结果

Find why MNN CPU inference is slow by examining threads, scheduling, memory layout, and the processor instruction path actually used. Operator-level measurements and comparisons with other inference frameworks help identify which layer needs improvement across ARM, x86_64, and RISC-V.

中文简介正在准备中。当前显示英语简介。

• Investigate slow CPU operators or LLM inference
• Diagnose thread count and memory usage anomalies
• Locate the cause of a performance regression

Run language models locally with llama.cpp on CPUs, Apple Silicon, or AMD and Intel GPUs. The skill guides model setup, memory-saving GGUF format choices, interactive chat, and serving models through an OpenAI-compatible API.

中文简介正在准备中。当前显示英语简介。

• Running a language model on a CPU-only computer
• Setting up local chat on an Apple Silicon Mac
• Choosing model quantization to fit available memory

Guides you through running large language models on CPUs, Apple Silicon, and AMD or Intel GPUs. Covers choosing GGUF model files, reducing memory through quantization, and starting an OpenAI-compatible chat server.

中文简介正在准备中。当前显示英语简介。

• Running a local LLM on an Apple Silicon Mac
• Trying LLM inference on a CPU-only machine
• Choosing a model format that fits available memory

Convert LLMs to GGUF and reduce model size through quantization, which stores weights at lower numerical precision. Prepare models for CPU or Apple Silicon inference, compare compression levels, and integrate them with local AI tools.

中文简介正在准备中。当前显示英语简介。

• Preparing an LLM for CPU-only use
• Running a local model on Apple Silicon
• Comparing model size and quality across quantization levels

Convert language models to GGUF and reduce their size through quantization, which stores model weights at lower precision. Guides cover compression choices, local inference on CPUs and Apple Silicon, and integration with tools such as Ollama.

中文简介正在准备中。当前显示英语简介。

• Running a language model on a CPU-only computer
• Choosing model compression for available memory
• Configuring Metal acceleration on Apple Silicon

Diagnose errors and incorrect inference results in MNN's QNN backend for Qualcomm HTP processors. Compare intermediate calculation data against CPU results to locate the first faulty operator, distinguish bugs from precision limits, and guide operator support and performance fixes.

中文简介正在准备中。当前显示英语简介。

• Investigate QNN runtime and model conversion errors
• Find the first operator producing incorrect results
• Distinguish implementation bugs from numerical precision limits

Improve MNN's Vulkan backend by separating CPU scheduling bottlenecks from GPU computation costs. It guides operator and shader changes, new feature integration with compatibility fallbacks, and device testing to verify both correctness and end-to-end performance.

中文简介正在准备中。当前显示英语简介。

• Finding CPU or GPU bottlenecks in Vulkan inference
• Optimizing a specific operator or shader
• Integrating cooperative matrix or subgroup features

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

中文简介正在准备中。当前显示英语简介。

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

中文简介正在准备中。当前显示英语简介。

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

中文简介正在准备中。当前显示原文说明。

Create or update Kubernetes YAML configuration files for GKE Autopilot and Standard. Translates application requirements into manifests covering container permissions, CPU and memory allocation, health checks, storage, external routing, and GPU resources for AI model serving.

中文简介正在准备中。当前显示英语简介。

• Creating GKE application deployment manifests
• Reviewing container security and resource settings
• Configuring external routes with GKE Gateway API

Guide MNN Metal backend development and optimization, from finding performance bottlenecks to changing GPU computation, combining operators, and adjusting execution scheduling. A task-based document index directs work toward relevant implementation rules, benchmarks, and correctness checks.

中文简介正在准备中。当前显示英语简介。

• Finding bottlenecks in Metal inference
• Adding or modifying Metal kernels
• Combining operators into one GPU dispatch

Executa inferência de LLM em CPU, Apple Silicon e GPUs consumer sem hardware NVIDIA. Use para edge deployment, Macs M1/M2/M3, GPUs AMD/Intel ou quando CUDA não está disponível. Suporta quantização GGUF (1,5-8 bits) para redução de memória e aceleração de 4-10× vs PyTorch em CPU.

中文简介正在准备中。当前显示原文说明。

Integrate Flash Attention into Transformer models to speed up attention, the calculation that relates input elements, while reducing GPU memory use. Covers PyTorch integration, performance measurement, output comparison, and a separate H100 optimization path.

中文简介正在准备中。当前显示英语简介。

• Optimize training with long input sequences
• Reduce memory use during long-context inference
• Measure speed and compare model outputs

Set up machine learning training and inference on NVIDIA DGX Spark. Get guidance on choosing containers and packages, resolving CUDA compatibility errors, and checking that PyTorch can access the GPU before starting a workload.

中文简介正在准备中。当前显示英语简介。

• Setting up a new DGX Spark for training or inference
• Diagnosing CUDA compatibility and import errors
• Choosing between containers and direct pip installation

Guides you through setting up RWKV, generating text, processing long documents in chunks, and fine-tuning models. Explains how to carry context between steps and compare memory use with Transformer models.

中文简介正在准备中。当前显示英语简介。

• Build streaming text generation
• Process long documents in chunks
• Plan RWKV fine-tuning

GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.

中文简介正在准备中。当前显示原文说明。

Set up and optimize large language model inference, or response generation, on NVIDIA GPUs. Use quantization, batch processing, and multiple GPUs to improve serving performance, and expose models through a chat completion API.

中文简介正在准备中。当前显示英语简介。

• Serving a language model through a chat completion API
• Processing many prompts in batches
• Configuring quantization to reduce memory use

Plan memory use and monitor long training jobs on NVIDIA DGX Spark. It guides troubleshooting for out-of-memory failures, tracks temperature and power alongside training logs, and helps assess competition between training and inference workloads.

中文简介正在准备中。当前显示英语简介。

• Estimating available memory before training
• Troubleshooting out-of-memory failures
• Investigating slowdowns during long training jobs

Investigate SGLang runs that freeze across multiple GPUs. It uses stack traces, communication logs, GPU memory dumps, and comparisons between processes to locate state differences, trace their causes, and guide fixes.

中文简介正在准备中。当前显示英语简介。

• Locate where a multi-GPU inference run freezes
• Compare logs from distributed processes
• Trace the first difference in process state

Guides the use of RWKV, a language model architecture that carries information forward in a compact state. Covers streaming text generation, long document processing, fine-tuning, and memory comparisons with Transformers.

中文简介正在准备中。当前显示英语简介。

• Build streaming text generation
• Process long documents in chunks
• Compare memory needs with Transformers

Helps reduce GPU memory use by loading large language models in 4-bit or 8-bit formats, a technique called quantization. Covers bitsandbytes configuration, memory checks, QLoRA fine-tuning, and training with memory-saving optimizers.

中文简介正在准备中。当前显示英语简介。

• Load a larger model within available GPU memory
• Compare 4-bit and 8-bit quantization settings
• Fine-tune a quantized model with QLoRA

Compress large language models to 4-bit weights to reduce GPU memory needs and accelerate text generation. Covers loading compressed models, converting your own models, choosing settings, and preparing memory-efficient fine-tuning with transformers and PEFT.

中文简介正在准备中。当前显示英语简介。

• Run language models with limited GPU memory
• Convert your own model to 4-bit weights
• Compare compression settings for quality and speed

Optimize transformer attention, the calculation that relates tokens to one another, using Flash Attention. The skill guides PyTorch integration, GPU memory reduction, performance measurement, and output comparisons for training or inference with long sequences.

中文简介正在准备中。当前显示英语简介。

• Reducing GPU memory use with long sequences
• Integrating Flash Attention into PyTorch models
• Comparing runtime and outputs before and after changes