Skip to content

L1w-Y/Dynamic-Sampling-LLm-Image-Opt

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
cmd
cmd
 
 
configs
configs
 
 
data
data
 
 
docs
docs
 
 
pkg
pkg
 
 
scripts
scripts
 
 
.gitignore
.gitignore
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
go.mod
go.mod
 
 
go.sum
go.sum
 
 

Repository files navigation

基于动态采样和大模型预测的镜像按需加载优化

目录

一、使用方法

1.1 环境要求

  • Go 1.19+
  • Python 3.8+
  • Linux内核支持eBPF
  • containerd和ctr工具
  • 大模型API密钥（OpenAI或Google Gemini）

1.2 编译和运行

  1. 准备环境

    # 安装依赖
sudo apt-get update
sudo apt-get install containerd

# 启动containerd服务
sudo systemctl start containerd

  2. 编译项目

    make build

  3. 设置API密钥

    export LLM_KEY=your-openai-or-gemini-api-key

  4. 运行优化器

    # 对指定容器进行优化分析（尝试从仓库获取Dockerfile）
make run-sudo CONTAINER=your-container-id

# 使用本地Dockerfile进行分析
make run-sudo CONTAINER=your-container-id DOCKERFILE=./your-dockerfile

  5. 生成stargz优化镜像

    # 生成stargz-snapshotter兼容的优先级文件
python3 scripts/generate_clean_priority.py data/final_priority.json data/stargz_priority.txt

# 使用优先级文件优化镜像
bash scripts/test_stargz_optimization.sh

二、执行情况

运行示例：

lyw@lyw-virtual-machine:~/project/oc_contributor_luoyiwei$ make run-sudo CONTAINER=7f2c270f34b11384dd4c011d0b18dbedcf8b8ffd2db7783ab77db65f7796846b
sudo -E ./bin/optimizer -container=7f2c270f34b11384dd4c011d0b18dbedcf8b8ffd2db7783ab77db65f7796846b -llm-key=$LLM_KEY
[sudo] lyw 的密码： 
=== 容器镜像按需加载优化系统 ===
容器ID: 7f2c270f34b11384dd4c011d0b18dbedcf8b8ffd2db7783ab77db65f7796846b
LLM提供商: gemini
采样持续时间: 30s
输出文件: data/priority_result.json
stargz文件: data/stargz_priority.txt

步骤1: 收集镜像信息...
✓ 镜像信息收集完成
  - 镜像: localai-base
  - 标签: latest
  - Dockerfile: 未获取

步骤2: LLM预测关键文件...
解析得到 10 个预测文件
✓ LLM预测完成，预测了 10 个关键文件

步骤3: eBPF动态采样...
等待eBPF采样完成（超时: 30s）...
采样进行中...
采样进行中...
采样进行中...
采样进行中...
采样进行中...
2025/09/10 19:25:31 收到启动完成信号，停止采样
采样完成
✓ eBPF采样完成，收集了 306 个文件访问事件

步骤4: 转换数据格式...
✓ 原始数据已保存
  - LLM数据: data/priority_result_llm.json
  - eBPF数据: data/priority_result_ebpf.json
=== 数据格式转换 ===
转换LLM数据: data/priority_result_llm.json
转换完成: 10 个LLM预测
LLM数据已保存到: converted_llm.json
转换eBPF数据: data/priority_result_ebpf.json
转换完成: 30 个eBPF数据条目
eBPF数据已保存到: converted_ebpf.json

=== 转换统计 ===
LLM预测文件: 10 个
eBPF采样文件: 30 个

LLM数据样例:
  1. /entrypoint.sh (优先级: 1.000, 置信度: 0.900)
  2. /workspace/tinyllama-1.1b-chat-v1.0.Q2_K.gguf (优先级: 1.000, 置信度: 0.800)
  3. /etc/resolv.conf (优先级: 0.800, 置信度: 0.700)

eBPF数据样例:
  1. /var/lib/docker/overlay2/af3879b102867d2619a23051c5d98feaa297ab22c780164607701f97c6162520/merged (访问次数: 32, 大小: 9480)
  2. /var/lib/docker/overlay2/af3879b102867d2619a23051c5d98feaa297ab22c780164607701f97c6162520/merged/proc (访问次数: 2, 大小: 0)
  3. /var/lib/docker/overlay2/af3879b102867d2619a23051c5d98feaa297ab22c780164607701f97c6162520/merged/dev (访问次数: 2, 大小: 0)
✓ 数据格式转换完成
步骤5: 融合LLM和eBPF结果...
✓ 融合完成，生成了 304 个优先级文件

步骤6: 生成输出文件...
✓ 输出文件生成完成

=== 使用说明 ===
1. 使用stargz-snapshotter优化镜像:
   ctr-remote image optimize --prefetch-list data/stargz_priority.txt \
2025/09/10 19:25:35 检测到ring buffer已关闭，退出读取循环

三、执行结果

3.1 生成的关键文件

项目执行后会生成以下关键文件：

eBPF动态采样结果

  • 文件位置: data/priority_result_ebpf.json
  • 内容: 容器运行时的实际文件访问记录
  • 格式: JSON格式，包含文件路径、访问时间、访问频率等信息

示例展示:

{
  "container_id": "7db300b3085acd47e1f7ed70e183304373a61f29de76c4e6472ad23d53f664ff",
  "start_time": "2025-09-08T21:26:24.681960621+08:00",
  "end_time": "2025-09-08T21:26:54.690343884+08:00",
  "duration": 30008383251,
  "events": [
    {
      "path": "/sys/kernel/tracing/events/syscalls/sys_exit_openat/id",
      "timestamp": 191253428900601,
      "operation": "open",
      "size": 0,
      "pid": 600650,
      "tid": 600688,
      "container_id": "7db300b3085acd47e1f7ed70e183304373a61f29de76c4e6472ad23d53f664ff",
      "access_mode": "read",
      "cache_hit": false,
      "latency": 0
    }
  ]
}

LLM静态预测结果

  • 文件位置: data/priority_result_llm.json
  • 内容: 基于大模型分析的静态文件优先级预测
  • 格式: JSON格式，包含文件路径、重要性评分、框架类型等信息

示例展示:

[
  {
    "path": "/bin/ollama",
    "priority": 1,
    "reason": "The main executable of the application, specified in the ENTRYPOINT and CMD instructions.",
    "category": "binary",
    "confidence": 1,
    "framework": "",
    "file_type": "",
    "startup_phase": "execute",
    "metadata": null
  },
  {
    "path": "/usr/lib/ollama/lib*.so*",
    "priority": 0.9,
    "reason": "Shared libraries are essential for the main executable to function. The Dockerfile copies these from various build stages.",
    "category": "library",
    "confidence": 0.8,
    "framework": "",
    "file_type": "",
    "startup_phase": "load",
    "metadata": null
  }
]

融合优先级文件

  • 文件位置: data/priority_result.json
  • 内容: eBPF和LLM结果的智能融合
  • 格式: JSON格式，包含最终的文件优先级排序

示例展示:

{
  "version": "1.0",
  "image": "",
  "tag": "",
  "files": [
    {
      "path": "/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys",
      "priority": 10,
      "confidence": 0.64,
      "category": "library",
      "startup_phase": "load",
      "reason": "基于eBPF动态采样预测",
      "source": "ebpf",
      "metadata": {
        "access_count": 2,
        "file_size": 0,
        "first_access": "1970-01-03T13:07:34.553361372+08:00",
        "last_access": "1970-01-03T13:07:34.553388785+08:00"
      },
      "llm_priority": 0,
      "ebpf_priority": 1,
      "llm_confidence": 0,
      "ebpf_confidence": 0.64
    }
  ]
}

stargz-snapshotter兼容文件

  • 文件位置: data/stargz_priority.txt
  • 内容: 纯文本格式的优先级文件列表
  • 格式: 每行一个文件路径，按优先级排序

示例展示:

/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys
/proc/thread-self/attr/apparmor/exec
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys/fs/cgroup/net_cls,net_prio
/sys/kernel/mm/transparent_hugepage/hpage_pmd_size
/etc/nsswitch.conf
/usr/lib/ollama/libcudart.so.12
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys/fs/cgroup/memory
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys/fs/cgroup/misc
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys/fs/cgroup/systemd
/proc/thread-self/mountinfo
/etc/passwd
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/dev/shm
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys/fs/cgroup/cpuset
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys/fs/cgroup/pids
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/etc/resolv.conf
/etc/hosts
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/sys/fs/cgroup/devices
/usr/lib/ollama/libggml-cpu-haswell.so
/var/lib/docker/overlay2/7b5f79a3f613c8482e95ff8aee7bd904b44f397b09ab7732e31df863e66b9203/merged/proc

四、镜像优化方法

4.1 与stargz-snapshotter集成

生成的优先级文件可以与stargz-snapshotter配合使用，实现容器镜像的按需加载优化：

步骤1: 生成优先级文件

# 运行优化器生成优先级文件
make run-sudo CONTAINER=your-container-id

步骤2: 优化镜像

# 使用ctr-remote优化镜像
sudo ctr-remote image optimize --prefetch-list data/stargz_priority.txt source-image:latest optimized-image:latest

4.2 优化原理

  1. 文件优先级排序: 根据eBPF动态采样和LLM静态预测的综合结果，对镜像中的文件进行优先级排序
  2. 预取优化: 高优先级文件被放置在镜像层的前部，紧跟在.prefetch.landmark文件之后
  3. 按需加载: 容器启动时，stargz-snapshotter通过单次HTTP Range请求拉取优先文件，实现快速访问
  4. 延迟加载: 低优先级文件在需要时才被加载，减少启动时间

4.3 优化示例

自带的优化方式：

sudo ctr-remote -n localai image optimize \
>   --oci \
>   docker.io/library/localai-base:latest \
>   docker.io/library/localai-base:stargz-native
INFO[0013] converting...                                 digest="sha256:9c6b76b881d92c4c296e0340abd3ffc5d740355e9994e3a9c1a805203fdb14b3"
INFO[0013] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0013] converting...                                 digest="sha256:dc6eb6dad5f9e332f00af553440e857b1467db1be43dd910cdb6830ba0898d50"
INFO[0013] converting...                                 digest="sha256:48ef4563dfde08de13729580dfb6b92bac4ddefcf6b9f6a7f912a49c7b4cc51a"
INFO[0013] converting...                                 digest="sha256:179413cb1c2635de514e4174728c7d3ef82e2990684604e2e4bc953516c6f734"
INFO[0013] converting...                                 digest="sha256:a07447822923fc377593712ba92408e2d82cee1dfc3e9cd8502e9e5476bb3072"
INFO[0013] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0013] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0013] converting...                                 digest="sha256:e84f55e0610d8852013b408e0771b62b98007982a61e05ca93fa6dd5af9051be"
INFO[0013] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0013] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0013] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0013] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0013] converting...                                 digest="sha256:f01968c79528a363659b867d67ba952d36b67b13e32767b19734ec6742058341"
INFO[0013] converting...                                 digest="sha256:702e48f1c0add66221f3bb4f7f69f822f73dbc73bddfe6fbf66683b534210bc4"
INFO[0013] converting...                                 digest="sha256:9c6b76b881d92c4c296e0340abd3ffc5d740355e9994e3a9c1a805203fdb14b3"
INFO[0013] converting...                                 digest="sha256:d0de5f366919a7e18c6c4c7fb04a61da4df99625be151cbf3ba909bbb549118f"
sha256:3e3c57aebbbffb13dabfc8e8d96d4a372cfdf4a3164bdd27797c2e2220cc2a16

自定义优先级文件

lyw@lyw-virtual-machine:~/桌面$ sudo ctr-remote -n localai image optimize \
>   --oci \
>   --prefetch-list /home/lyw/project/oc_contributor_luoyiwei/data/localai_stargz_priority.txt \
>   docker.io/library/localai-base:latest \
>   docker.io/library/localai-base:stargz-optimized
WARN[0000] DEPRECATION: The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use `config_path` instead. 
WARN[0000] DEPRECATION: The `configs` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use `config_path` instead. 
INFO[0000] using prefetch list from /home/lyw/project/oc_contributor_luoyiwei/data/localai_stargz_priority.txt 
INFO[0000] analyzing blob "sha256:dc6eb6dad5f9e332f00af553440e857b1467db1be43dd910cdb6830ba0898d50" 
INFO[0000] analyzing blob "sha256:48ef4563dfde08de13729580dfb6b92bac4ddefcf6b9f6a7f912a49c7b4cc51a" 
INFO[0000] analyzing blob "sha256:179413cb1c2635de514e4174728c7d3ef82e2990684604e2e4bc953516c6f734" 
INFO[0000] analyzing blob "sha256:a07447822923fc377593712ba92408e2d82cee1dfc3e9cd8502e9e5476bb3072" 
INFO[0000] analyzing blob "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 
INFO[0000] analyzing blob "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 
INFO[0000] analyzing blob "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 
INFO[0000] analyzing blob "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 
INFO[0000] analyzing blob "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 
INFO[0000] analyzing blob "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 
INFO[0000] analyzing blob "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 
INFO[0000] analyzing blob "sha256:e84f55e0610d8852013b408e0771b62b98007982a61e05ca93fa6dd5af9051be" 
INFO[0000] analyzing blob "sha256:702e48f1c0add66221f3bb4f7f69f822f73dbc73bddfe6fbf66683b534210bc4" 
INFO[0000] analyzing blob "sha256:f01968c79528a363659b867d67ba952d36b67b13e32767b19734ec6742058341" 
INFO[0000] analyzing blob "sha256:9c6b76b881d92c4c296e0340abd3ffc5d740355e9994e3a9c1a805203fdb14b3" 
INFO[0000] analyzing blob "sha256:d0de5f366919a7e18c6c4c7fb04a61da4df99625be151cbf3ba909bbb549118f" 
INFO[0000] analyzing blob "sha256:9c6b76b881d92c4c296e0340abd3ffc5d740355e9994e3a9c1a805203fdb14b3" 
INFO[0003] converting...                                 digest="sha256:9c6b76b881d92c4c296e0340abd3ffc5d740355e9994e3a9c1a805203fdb14b3"
INFO[0003] converting...                                 digest="sha256:dc6eb6dad5f9e332f00af553440e857b1467db1be43dd910cdb6830ba0898d50"
INFO[0003] converting...                                 digest="sha256:48ef4563dfde08de13729580dfb6b92bac4ddefcf6b9f6a7f912a49c7b4cc51a"
INFO[0003] converting...                                 digest="sha256:179413cb1c2635de514e4174728c7d3ef82e2990684604e2e4bc953516c6f734"
INFO[0003] converting...                                 digest="sha256:a07447822923fc377593712ba92408e2d82cee1dfc3e9cd8502e9e5476bb3072"
INFO[0003] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0003] converting...                                 digest="sha256:f01968c79528a363659b867d67ba952d36b67b13e32767b19734ec6742058341"
INFO[0003] converting...                                 digest="sha256:e84f55e0610d8852013b408e0771b62b98007982a61e05ca93fa6dd5af9051be"
INFO[0003] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0003] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0003] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0003] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0003] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0003] converting...                                 digest="sha256:d0de5f366919a7e18c6c4c7fb04a61da4df99625be151cbf3ba909bbb549118f"
INFO[0003] converting...                                 digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
INFO[0003] converting...                                 digest="sha256:9c6b76b881d92c4c296e0340abd3ffc5d740355e9994e3a9c1a805203fdb14b3"
INFO[0003] converting...                                 digest="sha256:702e48f1c0add66221f3bb4f7f69f822f73dbc73bddfe6fbf66683b534210bc4"
sha256:690f39586d4bce9acb0559cac2aed0e88f0b3f2bfd851be0d0650f562eec2965

五、性能分析

5.1 测试环境

  • 网络环境: 5 MiB/s (Ubuntu 20.04)
  • 测试镜像:
    • ollama:latest (Qwen2.5-7B-Instruct-gguf, 6.94GB)
    • localai-base:latest (tinyllama-1.1b-chat-v1.0.Q2_K.gguf, 734MB)
    • huggingface/transformers-gpu:latest (7.98GB)

5.2 Pull时间对比

数据表格

镜像类型 优化方式 Pull时间 相对差异
ollama stargz-optimized 16m 18s +0.5分钟
ollama stargz-native 15m 49s 基准
localai-base stargz-optimized 0m 56s -0.2分钟
localai-base stargz-native 1m 10s 基准
transformers-gpu stargz-optimized 14m 38s +1.5分钟
transformers-gpu stargz-native 13m 8s 基准

可视化条形图

Pull时间对比 (分钟)

       │
       ┤ ████ ollama-optimized (16.3分钟)
       │ ████ ollama-native (15.8分钟)
       ┤ 
       │ ████ transformers-optimized (14.6分钟)
       │ ████ transformers-native (13.1分钟)
       ┤ 
       │ █ localai-optimized (0.93分钟)
       │ ██ localai-native (1.17分钟)
       └────────────────────────────────────

Pull时间分析:

  • ollama镜像: 优化版本略慢0.5分钟
  • localai-base镜像: 优化版本快0.24分钟
  • transformers-gpu镜像: 优化版本慢1.5分钟
  • 优化效果: 小镜像(localai-base)优化相对明显，大镜像(ollama、transformers-gpu)优化效果略差

5.3 Create+Start时间对比

数据表格

镜像类型 优化方式 Create时间 Start时间 总时间 相对差异
ollama stargz-optimized 0.035s 8.773s 8.808s +1.97s
ollama stargz-native 0.032s 6.805s 6.837s 基准
localai-base stargz-optimized 0.054s 3.217s 3.271s +0.02s
localai-base stargz-native 0.043s 3.206s 3.249s 基准
transformers-gpu stargz-optimized 0.056s 3.247s 3.303s -0.03s
transformers-gpu stargz-native 0.060s 3.330s 3.390s 基准

可视化条形图

Create+Start时间对比 (秒)
       │
       │ ████ ollama-optimized (8.81秒)
       │ ████ ollama-native (6.84秒)
       │
       │ ████ localai-optimized (3.27秒)
       │ ████ localai-native (3.25秒)
       │
       │
       │ ████ transformers-optimized (3.30秒)
       │ ████ transformers-native (3.39秒)
       └────────────────────────────────────

Create+Start时间分析:

  • ollama镜像: 优化版本慢2秒
  • localai-base镜像: 两者基本相当
  • transformers-gpu镜像: 优化版本略快0.09秒
  • 性能影响: 优化前后差异不大

六、项目架构

├── cmd/                    # 主程序入口
├── pkg/
│   ├── ebpf/              # eBPF采样模块
│   ├── llm/               # LLM预测模块
│   ├── fusion/            # 融合引擎
│   └── image/             # 镜像分析模块
├── scripts/               # 辅助脚本
├── data/                  # 数据文件目录
├── configs/               # 配置文件
└── docs/                  # 文档

6.1 工作流程

容器启动 → eBPF动态采样 → LLM静态预测 → 智能融合 → 优先级文件生成 → stargz优化

6.2 各模块详解

  • eBPF动态采样模块 - 实时监控容器文件访问行为 → 查看详情
  • LLM预测模块 - 基于大模型预测文件重要性 → 查看详情
  • fusion融合模块 - 智能融合动态和静态分析结果 → 查看详情
  • image镜像分析模块 - 容器镜像静态特征分析 → 查看详情

About

An optimization framework for on-demand container image loading, based on dynamic sampling and Large Model (LLM) prediction

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages