通过llama.cpp使用Qwen3.5对话 + Qwen3-Embedding 向量化 ，再通过Elasticsearch 存储向量

两个llama.cpp的加载脚本：

# Qwen3.5-4B-Q4_0 — Multimodal (VL) model, Q4_0 quant (~2.7GB) + mmproj-F16
# Context: 64K tokens | Port: 7837 (unchanged)
# Endpoint: https://127.0.0.1:7837 (OpenAI-compatible, /v1/chat/completions)
#
# Usage:
#   .\Qwen3.5-4B-Q4_0.ps1                              # start server
#   .\Qwen3.5-4B-Q4_0.ps1 -Port 8080                   # custom port
#   .\Qwen3.5-4B-Q4_0.ps1 -CtxSize 32768              # shorter context
#
# Test with curl (vision):
#   curl http://127.0.0.1:7837/v1/chat/completions -Method POST `
#     -ContentType "application/json" `
#     -Body '{"model":"Qwen3.5-4B","messages":[{"role":"user","content":[{"type":"text","text":"What is in this image?"},{"type":"image_url","image_url":{"url":"data:image/png;base64,<base64>"}}]}]}'

param(
    [int]$Port = 7837,
    [int]$CtxSize = 65536,
    [int]$NGPU = 99,
    [int]$Threads = 16,
    [int]$ThreadsBatch = 20
)

& "llama-server.exe" `
    -m "Qwen3.5-4B-Q4_0.gguf" `
    --mmproj ".\qwen3.5\mmproj-F16.gguf" `
    -c $CtxSize `
    -ngl $NGPU `
    --cache-type-k q4_0 `
    --cache-type-v q4_0 `
    --host 127.0.0.1 `
    --port $Port `
    --flash-attn on `
    --chat-template-file ".\qwen3.5\chat_template.jinja" `
    -ub 1024 `
    -np 2 `
    -t $Threads `
    --threads-batch $ThreadsBatch `
    --temp 0.6 `
    --top-p 0.95 `
    --top-k 20 `
    --min-p 0.0 `
    --presence-penalty 0.0 `
    --repeat-penalty 1.0

# Qwen3-Embedding-0.6B-Q8_0 — Text Embedding Model, Q8_0 quant (~0.6GB)
# Context: 32K tokens | Embedding dim: 1024 (MRL 32~1024) | Pooling: last-token
# Endpoint: POST /v1/embeddings (OpenAI-compatible)
#
# Usage:
#   .\Qwen3-Embedding-0.6B.ps1                              # start server
#   .\Qwen3-Embedding-0.6B.ps1 -Port 8080                    # custom port
#   .\Qwen3-Embedding-0.6B.ps1 -CtxSize 8192                 # shorter context
#
# Output dimension: server returns full 1024d.
# Qwen3-Embedding supports MRL — take first N dims client-side for any N∈[32,1024].
#
# Test with curl:
#   curl http://127.0.0.1:7838/v1/embeddings -Method POST `
#     -ContentType "application/json" `
#     -Body '{"input":"Hello world","model":"Qwen3-Embedding-0.6B"}'

param(
    [int]$Port = 7838,
    [int]$CtxSize = 2048,
    [int]$Batch = 32,
    [int]$UBatch = 32,
    [int]$NGPU = 99
)

& "llama-server.exe" `
    -m "Qwen3-Embedding-0.6B-Q8_0.gguf" `
    --port $Port `
    --host 127.0.0.1 `
    --ctx-size $CtxSize `
    --batch-size $Batch `
    --ubatch-size $UBatch `
    -ngl $NGPU `
    --embeddings `
    --pooling last `
    --embd-normalize 2 `
    --flash-attn on `
    --alias "Qwen3-Embedding-0.6B"

 

在本地启动好elasticsearch 9.5.1

{
  "name": "WO-PC",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "fk-aMKvSTai0hNyHRzayxg",
  "version": {
    "number": "9.5.1",
    "build_flavor": "default",
    "build_type": "zip",
    "build_hash": "0e1cdd500cb446f51b847d131747da3cc500eee1",
    "build_date": "2026-08-05T23:10:54.760245929Z",
    "build_snapshot": false,
    "lucene_version": "10.5.0",
    "minimum_wire_compatibility_version": "8.19.0",
    "minimum_index_compatibility_version": "8.0.0"
  },
  "tagline": "You Know, for Search"
}

 

Java依赖：

查看代码

 mvn dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------------< org:rag-cli >-----------------------------
[INFO] Building rag-cli 1.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- dependency:3.7.0:tree (default-cli) @ rag-cli ---
[INFO] org:rag-cli:jar:1.0-SNAPSHOT
[INFO] +- dev.langchain4j:langchain4j:jar:1.19.0:compile
[INFO] |  +- dev.langchain4j:langchain4j-core:jar:1.19.0:compile
[INFO] |  |  \- org.jspecify:jspecify:jar:1.0.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.22:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.22.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.22.1:compile
[INFO] |  +- org.apache.opennlp:opennlp-tools:jar:2.5.11:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:2.0.18:compile
[INFO] +- dev.langchain4j:langchain4j-open-ai:jar:1.19.0:compile
[INFO] |  +- dev.langchain4j:langchain4j-http-client:jar:1.19.0:compile
[INFO] |  +- dev.langchain4j:langchain4j-http-client-jdk:jar:1.19.0:runtime
[INFO] |  \- com.knuddels:jtokkit:jar:1.1.0:compile
[INFO] +- dev.langchain4j:langchain4j-elasticsearch:jar:1.19.0-beta29:compile
[INFO] |  +- co.elastic.clients:elasticsearch-java:jar:9.3.1:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- io.opentelemetry:opentelemetry-api:jar:1.37.0:runtime
[INFO] |  |  |  \- io.opentelemetry:opentelemetry-context:jar:1.37.0:runtime
[INFO] |  |  +- io.opentelemetry.semconv:opentelemetry-semconv:jar:1.37.0:runtime
[INFO] |  |  +- tools.jackson.core:jackson-databind:jar:3.0.0:runtime
[INFO] |  |  \- tools.jackson.core:jackson-core:jar:3.0.0:runtime
[INFO] |  +- org.elasticsearch.client:elasticsearch-rest-client:jar:9.3.1:compile
[INFO] |  |  \- org.apache.httpcomponents:httpasyncclient:jar:4.1.5:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.16:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.14:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.22.0:compile
[INFO] |  +- org.apache.httpcomponents:httpcore-nio:jar:4.4.16:compile
[INFO] |  \- jakarta.json:jakarta.json-api:jar:2.0.2:compile
[INFO] +- co.elastic.clients:elasticsearch-rest5-client:jar:9.3.1:compile
[INFO] |  +- org.apache.httpcomponents.client5:httpclient5:jar:5.6:compile
[INFO] |  |  +- org.apache.httpcomponents.core5:httpcore5:jar:5.4:compile
[INFO] |  |  \- org.apache.httpcomponents.core5:httpcore5-h2:jar:5.4:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.3.5:compile
[INFO] |  \- org.eclipse.parsson:parsson:jar:1.1.7:compile
[INFO] +- info.picocli:picocli:jar:4.7.7:compile
[INFO] +- org.jline:jline:jar:3.30.16:compile
[INFO] +- org.jsoup:jsoup:jar:1.23.1:compile
[INFO] \- org.slf4j:slf4j-simple:jar:2.0.18:runtime
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.342 s
[INFO] Finished at: 2026-08-15T12:31:51+08:00
[INFO] ------------------------------------------------------------------------

 

代码创建向量索引，这里我使用的是1024维：

    private static void createIndex(AppConfig cfg, ElasticsearchClient client) {
        try {
            client.indices()
                    .create(c -> c.index(cfg.esIndex())
                            .mappings(m -> m.properties("vector", p -> p.denseVector(dv -> dv
                                            .dims(VECTOR_DIMENSIONS)
                                            .similarity(DenseVectorSimilarity.Cosine)
                                            .index(true)))
                                    .properties("text", p -> p.text(t -> t))
                                    .properties("metadata", p -> p.object(o -> o.dynamic(DynamicMapping.True)))));
            System.out.println("已创建 Elasticsearch 索引 " + cfg.esIndex() + "（dense_vector 维度 " + VECTOR_DIMENSIONS + "，cosine 相似度）");
        } catch (Exception e) {
            throw new IllegalStateException("创建 Elasticsearch 索引 " + cfg.esIndex() + " 失败: " + Errors.rootMessage(e), e);
        }
    }

 

文本chunking使用langchain4j自带的方法，自动按段落→行→句→词→字符切分

dev.langchain4j.data.document.splitter.DocumentSplitter.recursive(int maxSegmentSizeInChars, int maxOverlapSizeInChars)

 

上下文的注入：

import dev.langchain4j.data.message.ChatMessage;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.rag.content.Content;
import dev.langchain4j.rag.content.injector.ContentInjector;
import java.util.List;

/**
 * 将检索到的资料（含来源 metadata）注入用户消息：模型基于资料回答并标注来源。
 * 无资料时原样返回（模型退化为无上下文回答）。
 */
public class SourceInjector implements ContentInjector {

    @Override
    public ChatMessage inject(List<Content> contents, ChatMessage chatMessage) {
        if (contents.isEmpty()) {
            return chatMessage;
        }
        StringBuilder sb = new StringBuilder(
                "以下是检索到的资料，请优先基于这些资料回答，并在回答末尾用「来源：文件名」标注每条依据对应的来源：\n\n");
        for (Content content : contents) {
            sb.append("【资料");
            if (content.textSegment() != null) {
                String source = content.textSegment().metadata().getString("source");
                if (source != null && !source.isBlank()) {
                    sb.append("｜来源：").append(source);
                }
            }
            sb.append("】\n").append(content.textSegment().text()).append("\n\n");
        }
        sb.append("用户问题：").append(((UserMessage) chatMessage).singleText());
        return UserMessage.from(sb.toString());
    }
}

 

import dev.langchain4j.memory.chat.MessageWindowChatMemory;
import dev.langchain4j.model.chat.ChatModel;
import dev.langchain4j.rag.DefaultRetrievalAugmentor;
import dev.langchain4j.service.AiServices;
import org.ragcli.es.EsComponents;

/** AiServices 装配：多轮记忆 + 可选 RAG 检索增强。 */
public final class ChatEngine {
    private ChatEngine() {}

    public static ChatAssistant build(ChatModel llm, EsComponents es) {
        AiServices<ChatAssistant> services = AiServices.builder(ChatAssistant.class)
                .chatModel(llm)
                .chatMemory(MessageWindowChatMemory.withMaxMessages(20));
        if (es != null) {
            services.retrievalAugmentor(DefaultRetrievalAugmentor.builder()
                    .contentRetriever(es.retriever())
                    .contentInjector(new SourceInjector())
                    .build());
        }
        return services.build();
    }
}

 

封装为CLI后的效果：

wo  java -jar target/rag-cli-1.0-SNAPSHOT.jar --index samples/RX100M3说明书.html
已连接 LLM http://127.0.0.1:7837/v1（模型 Qwen3.5-4B-Q4_0.gguf）
已连接 Embedding http://127.0.0.1:7838/v1（模型 Qwen3-Embedding-0.6B）
已连接 Elasticsearch https://127.0.0.1:9200（本地开发模式，信任自签证书）
已索引: samples\RX100M3说明书.html（133 块）
索引完成：文件 1（成功 1 / 跳过 0 / 失败 0），共 133 块，耗时 10910 ms
WARNING: A restricted method in java.lang.foreign.MemorySegment has been called
WARNING: java.lang.foreign.MemorySegment::reinterpret has been called by org.jline.terminal.impl.ffm.FfmTerminalProvider in an unnamed module (file:/E:/work/working/java/rag-cli/target/rag-cli-1.0-SNAPSHOT.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

输入 /help 查看命令，/exit 退出。
you> rx100m3的功能菜单设置里有哪些功能？
assistant> 根据您提供的《DSC-RX100M3 说明书》资料，在“改变（功能）按钮的设置功能”章节中，列出了可以分配给功能按钮（Fn 按钮）和改变控制环设置功能的以下功能：

**可分配的功能列表：**

*   **拍摄模式**
*   **闪光模式**
*   **闪光补偿**
*   **对焦模式**
*   **对焦区域**
*   **曝光补偿**
*   **过滤器**
*   **测光模式**
*   **白平衡模式**
*   **ISO/ND**
*   **自动 DRO/**
*   **创意风格**
*   **照相模式**
*   **照片效果**
*   **HDR/**
*   **拍中央锁定**
*   **笑脸**
*   **人脸检测**
*   **美肤效果**
*   **自动构图**
*   **AF/**
*   **影像尺寸**
*   **纵横比**
*   **影像质量**（静止影像）
*   **SteadyShot**（动态影像）
*   **斑马线**
*   **网格线**
*   **峰值水平**
*   **峰值色彩**
*   **未设定**

*注：资料中也提到“各照相模式无法设定的功能”，例如在“阶段”、“自动”、“连拍”等特定模式下，部分功能（如 HDR、创意风格等）可能无法选择或设置为"—"，这取决于当前所选的照相模式。*

来源：E:\work\working\java\rag-cli\samples\RX100M3 说明书.html
you> 在电池充电期间本相机的充电指示灯闪烁。
assistant> 根据您提供的资料，在电池充电期间，如果本相机的充电指示灯闪烁，可能的原因及解决方法如下：

**可能原因：**
1.  **电池类型错误**：当前插入的电池不是型号为 **NP-BX1** 的电池。
2.  **温度超出充电范围**：由于环境温度过高或过低（不在建议的 **10℃至 30℃** 范围内），导致暂时进入充电待机状态。

**解决方法：**
*   **针对电池类型错误**：请取出电池，然后重新将同一电池装入本相机以确认电池型号。
*   **针对温度问题**：请确保在正常充电温度范围内（10℃至 30℃）给电池充电。

**来源：E:\work\working\java\rag-cli\samples\RX100M3 说明书.html**
you> /help
可用命令：
  /exit  退出会话（同 /quit）
  /quit  退出会话
  /help  显示本帮助
其他输入作为消息发送给模型。
you> /exit

 

文章摘自：https://www.cnblogs.com/JackieJK/p/22491193