背景

主要是自定义流量治理的逻辑, 如加header等

扩展的开发

方式

提供了名为 lua 的特殊扩展，允许控制面通过 xDS 协议动态下发 Lua 脚本并由 Envoy 解释执行。
也是本节的主题，Envoy 引入了 WASM 技术用于开发 Envoy 扩展。

wasm的优缺点

优点：

WASM 字节码具备与机器码相似的性能，保证了 WASM 扩展性能；
WASM 扩展在沙箱中执行，更安全，单个功能扩展不会影响到 Envoy 主体功能，可靠性和安全性更高；
WASM 扩展可以以 .wasm 文件动态分发、共享以及装载运行，且不受平台限制；
WASM 扩展无语言限制，理论上所有支持 WASM 的语言都可以用于开发 Envoy 扩展，开发效率更高。

缺点：

性能约为C++编写的原生静态编译的Filter的70％。
由于需要启动一个或多个WASM虚拟机，因此会消耗一定的内存使用量。
开发生态也还不够完善，坑多，成本高

开发

基于solo

rust

初始化

[root@master wasme]# ./wasme init ./wasm-demo
✔ rust
✔ istio:1.5.x, istio:1.6.x, istio:1.7.x, gloo:1.6.x, istio:1.8.x, istio:1.9.x

demo
build

../wasme build rust -t webassemblyhub.io/deeponder/hello:v0.1 .

[root@master wasm-demo]# ../wasme build rust -t webassemblyhub.io/deeponder/hello:v0.1 .
Unable to find image 'quay.io/solo-io/ee-builder:0.0.33' locally
0.0.33: Pulling from solo-io/ee-builder
df27e1f7c31e: Pull complete 
0a8813a60e2e: Pull complete 
3c2cba919283: Pull complete 
26f4837a47c0: Pull complete 
dd7b292cf068: Extracting [============>                                      ]   79.1MB/322.9MB
4a4d78f042bc: Download complete 
9108a736d6a0: Download complete 
70ac09daaa76: Download complete 
809bdff17a4d: Download complete 
31fc029d676e: Download complete 
85533903f7c2: Download complete 
f87e543b124a: Download complete 
93d78f561264: Download complete 
8ba3d0f61e41: Downloading [==================================================>]    404MB/404MB
d511201136be: Download complete 
docker: write /data/docker/tmp/GetImageBlob182227699: no space left on device.

// 清理空间
docker system prune --volumes

// 编译报错，对rust不熟悉，先放弃
[19 / 20] [Prepa] Compiling Rust cdylib filter (1 files)
INFO: From Compiling Rust cdylib filter (1 files):
error[E0407]: method `get_type` is not a member of trait `RootContext`
  --> filter.rs:30:5
   |
30 | /     fn get_type(&self) -> Option<ContextType> {
31 | |         Some(ContextType::HttpContext)
32 | |     }
   | |_____^ not a member of trait `RootContext`
....

cpp

build

../wasme build cpp -t webassemblyhub.io/deeponder/add-header:v0.1 .

RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function
https://stackoverflow.com/questions/6842687/the-remote-end-hung-up-unexpectedly-while-git-cloning
git config --global http.postBuffer 20000000

// 因为太慢了... 尝试改为基于预编译构建, 且配置github访问加速， 然继续报错
INFO: Call stack for the definition of repository 'emscripten_toolchain' which is a http_archive (rule definition at /root/.cache/bazel/_bazel_root/274fa09d7f040167f7809fe120d70954/external/bazel_tools/tools/build_defs/repo/http.bzl:292:16):
 - /root/wasme/new-filter/WORKSPACE:7:1
ERROR: An error occurred during the fetch of repository 'emscripten_toolchain':
   Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/274fa09d7f040167f7809fe120d70954/external/bazel_tools/tools/build_defs/repo/http.bzl", line 83
		patch(ctx)
	File "/root/.cache/bazel/_bazel_root/274fa09d7f040167f7809fe120d70954/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 165, in patch
		fail(<1 more arguments>)
Error applying patch command ./emsdk install 1.39.0-upstream:
Installing SDK 'sdk-releases-upstream-d57bfdd6d43181501bbd3fab502d57c9073ceb49-64bit'..
Installing tool 'releases-upstream-d57bfdd6d43181501bbd3fab502d57c9073ceb49-64bit'..
Downloading: /root/.cache/bazel/_bazel_root/274fa09d7f040167f7809fe120d70954/external/emscripten_toolchain/zips/d57bfdd6d43181501bbd3fab502d57c9073ceb49-wasm-binaries.tbz2 from https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/d57bfdd6d43181501bbd3fab502d57c9073ceb49/wasm-binaries.tbz2, 163342540 Bytes
 [----------------------------------------------------------------------------]
Unpacking '/root/.cache/bazel/_bazel_root/274fa09d7f040167f7809fe120d70954/external/emscripten_toolchain/zips/d57bfdd6d43181501bbd3fab502d57c9073ceb49-wasm-binaries.tbz2' to '/root/.cache/bazel/_bazel_root/274fa09d7f040167f7809fe120d70954/external/emscripten_toolchain/upstream'
['tar', '-xf', '/root/.cache/bazel/_bazel_root/274fa09d7f040167f7809fe120d70954/external/emscripten_toolchain/zips/d57bfdd6d43181501bbd3fab502d57c9073ceb49-wasm-binaries.tbz2', '--strip', '1'] failed with error code 2!
tar (child): lbzip2：无法 exec: 没有那个文件或目录
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Traceback (most recent call last):
  File "./emsdk.py", line 2991, in <module>
    sys.exit(main())
  File "./emsdk.py", line 2971, in main
    success = tool.install()
  File "./emsdk.py", line 1706, in install
    success = tool.install()
  File "./emsdk.py", line 1756, in install
    open(emscripten_version_file_path, 'w').write('"%s"' % version)
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/bazel/_bazel_root/274fa09d7f040167f7809fe120d70954/external/emscripten_toolchain/upstream/emscripten/emscripten-version.txt'



// 尝试版本回退到0.0.24， 继续报错，没有头绪，先放弃
[root@master cpp-filter]# bazel build :filter.wasm
Killed non-responsive server process (pid=25724)
Starting local Bazel server and connecting to it...
... still trying to connect to local Bazel server after 10 seconds ...
INFO: Call stack for the definition of repository 'envoy_wasm_api' which is a new_git_repository (rule definition at /root/.cache/bazel/_bazel_root/cc1de220b5dd5f6b3dd069b0bc6ee9dc/external/bazel_tools/tools/build_defs/repo/git.bzl:182:22):
 - /root/wasme/cpp-filter/WORKSPACE:48:1
ERROR: An error occurred during the fetch of repository 'envoy_wasm_api':
   Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/cc1de220b5dd5f6b3dd069b0bc6ee9dc/external/bazel_tools/tools/build_defs/repo/git.bzl", line 170
		_clone_or_update(ctx)
	File "/root/.cache/bazel/_bazel_root/cc1de220b5dd5f6b3dd069b0bc6ee9dc/external/bazel_tools/tools/build_defs/repo/git.bzl", line 36, in _clone_or_update
		git_repo(ctx, directory)
	File "/root/.cache/bazel/_bazel_root/cc1de220b5dd5f6b3dd069b0bc6ee9dc/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 91, in git_repo
		_update(ctx, git_repo)
	File "/root/.cache/bazel/_bazel_root/cc1de220b5dd5f6b3dd069b0bc6ee9dc/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 103, in _update
		fetch(ctx, git_repo)
	File "/root/.cache/bazel/_bazel_root/cc1de220b5dd5f6b3dd069b0bc6ee9dc/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 129, in fetch
		_git_maybe_shallow(ctx, <5 more arguments>)
	File "/root/.cache/bazel/_bazel_root/cc1de220b5dd5f6b3dd069b0bc6ee9dc/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 171, in _git_maybe_shallow
		_error(ctx.name, <2 more arguments>)
	File "/root/.cache/bazel/_bazel_root/cc1de220b5dd5f6b3dd069b0bc6ee9dc/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 181, in _error
		fail(<1 more arguments>)
error running 'git fetch origin refs/heads/*:refs/remotes/origin/* refs/tags/*:refs/tags/*' while working with @envoy_wasm_api:
fatal: unable to access 'https://github.com/istio/envoy/': Encountered end of file

tiny-go

通过国内源拉取镜像，提升速度

docker pull quay.mirrors.ustc.edu.cn/solo-io/ee-builder:0.0.33
docker tag quay.mirrors.ustc.edu.cn/solo-io/ee-builder:0.0.33 quay.io/solo-io/ee-builder:0.0.33

build成功

[root@master go-filter]# ../wasme build tinygo -t webassemblyhub.io/deeponder/add-header:v0.1 .
Building with tinygo...go: downloading github.com/tetratelabs/proxy-wasm-go-sdk v0.1.1
INFO[0022] adding image to cache...                      filter file=/tmp/wasme917093270/filter.wasm tag="webassemblyhub.io/deeponder/add-header:v0.1"
INFO[0022] tagged image                                  digest="sha256:af48e9f940b4058ca7e5203ab6771ae07029dccf2eceaebc6540c4eb6e9ca901" image="webassemblyhub.io/deeponder/add-header:v0.1"

deploy

// refuse
[root@master go-filter]# ../wasme deploy istio webassemblyhub.io/deeponder/add-header:v0.1 --id=myfilter
Error: ensuring namespace: Post "https://lb.kubesphere.local:6443/api/v1/namespaces": dial tcp 192.168.50.68:6443: connect: connection refused

// 加sudo  提示版本不对
[root@master go-filter]# sudo ../wasme deploy istio webassemblyhub.io/deeponder/add-header:v0.1 --id=myfilter
INFO[0003] cache namespace created                       cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0004] cache configmap created                       cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0004] cache service account created                 cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0004] cache role created                            cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0004] cache rolebinding created                     cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0004] cache daemonset created                       cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
Error: image webassemblyhub.io/deeponder/add-header:v0.1 not supported by istio version 

// 先忽略版本， 提示evaluat超时
[root@master go-filter]# ../wasme deploy istio webassemblyhub.io/deeponder/add-header:v0.1 --id=myfilter2 --istiod-name=istiod-1-9-5
Error: ensuring namespace: Internal error occurred: resource quota evaluation timed out

// 集群恢复后，继续执行
[root@master go-filter]# ../wasme deploy istio webassemblyhub.io/deeponder/add-header:v0.1 --id=myfilter2 --istiod-name=istiod-1-9-5
INFO[0000] cache namespace already exists                cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache configmap already exists                cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache service account already exists          cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache role updated                            cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache rolebinding updated                     cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache daemonset updated                       cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0006] added image to cache config...                cache="{wasme-cache wasme}" image="webassemblyhub.io/deeponder/add-header:v0.1"
INFO[0006] waiting for event with timeout 1m0s          
WARN[0008] event err: expected 3 image-ready events for image webassemblyhub.io/deeponder/add-header:v0.1, only found map[] 

// 重试，执行成功
[root@master go-filter]# ../wasme deploy istio webassemblyhub.io/deeponder/add-header:v0.1 --id=myfilter2 --istiod-name=istiod-1-9-5 --namespace test
INFO[0000] cache namespace already exists                cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache configmap already exists                cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache service account already exists          cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache role updated                            cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache rolebinding updated                     cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0000] cache daemonset updated                       cache=wasme-cache.wasme image="quay.io/solo-io/wasme:0.0.33"
INFO[0007] image is already cached                       cache="{wasme-cache wasme}" image="webassemblyhub.io/deeponder/add-header:v0.1"
INFO[0007] updated workload sidecar annotations          filter="id:\"myfilter2\" image:\"webassemblyhub.io/deeponder/add-header:v0.1\" rootID:\"root_id\" patchContext:\"inbound\" " workload=appauth-rpc-v1
INFO[0007] created Istio EnvoyFilter resource            envoy_filter_resource=appauth-rpc-v1-myfilter2.test filter="id:\"myfilter2\" image:\"webassemblyhub.io/deeponder/add-header:v0.1\" rootID:\"root_id\" patchContext:\"inbound\" " workload=appauth-rpc-v1
INFO[0008] updated workload sidecar annotations          filter="id:\"myfilter2\" image:\"webassemblyhub.io/deeponder/add-header:v0.1\" rootID:\"root_id\" patchContext:\"inbound\" " workload=appmng-v1
INFO[0008] created Istio EnvoyFilter resource            envoy_filter_resource=appmng-v1-myfilter2.test filter="id:\"myfilter2\" image:\"webassemblyhub.io/deeponder/add-header:v0.1\" rootID:\"root_id\" patchContext:\"inbound\" " workload=appmng-v1    // 这里可以看到已经注入
...

效果

Response Headers: 
date: Wed, 16 Feb 2022 02:34:15 GMT
content-type: application/json
content-length: 193
x-envoy-upstream-service-time: 1310
hello: peng   //这里为注入的逻辑
server: istio-envoy
x-envoy-decorator-operation: kubesphere-router-test.kubesphere-controls-system.svc.cluster.local:80/*

wasm工作原理

https://istio.io/latest/zh/blog/2020/deploy-wasm-declarative/

编译好.wasme的扩展
通过configmap把扩展挂载到istio-proxy中
istio-proxy加载扩展，执行定制化的逻辑（istio内置了对wasm的支持）

其它的不足

solo-io/wasm对istio版本要求比较高，目前最高支持1.9
对于不熟悉对应rust/c++语言的开发者，成本比较高，且社区可提供参考的问题解决方案比较少
国内GW的原因，相关资源的拉取比较慢，且可用国内的源比较少

refer

https://istio.io/latest/docs/reference/config/networking/envoy-filter/
https://github.com/solo-io/wasm
https://developer.aliyun.com/article/782181
https://banzaicloud.com/blog/envoy-wasm-filter/
https://blog.red-badger.com/extending-istio-with-rust-and-webassembly
https://cloudnative.to/blog/envoy-wasm/
https://developer.51cto.com/article/692817.html
https://developer.aliyun.com/article/752771
github加速： https://juejin.cn/post/6956887926055632932

istio扩展开发

背景

扩展的开发

方式

wasm的优缺点

开发

rust

cpp

tiny-go

wasm工作原理

其它的不足

refer

CATALOG

FEATURED TAGS