Skip to content

Commit 53cd7e7

Browse files
committed
feat: new plugin = q dev
1 parent 14e9103 commit 53cd7e7

20 files changed

Lines changed: 1114 additions & 0 deletions

backend/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ module github.com/apache/incubator-devlake
33
go 1.20
44

55
require (
6+
github.com/aws/aws-sdk-go v1.55.6
67
github.com/cockroachdb/errors v1.11.1
78
github.com/gin-contrib/cors v1.6.0
89
github.com/gin-gonic/gin v1.9.1

backend/go.sum

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hC
5454
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
5555
github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
5656
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
57+
github.com/aws/aws-sdk-go v1.55.6/go.mod h1:eRwEWoyTWFMVYVQzKMNHWP5/RV4xIUGMQfXQHfHkpNU=
5758
github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
5859
github.com/bketelsen/crypt v0.0.4/go.mod h1:aI6NrJ0pMGgvZKL1iVgXLnfIFJtfV+bKCoqOes/6LfM=
5960
github.com/bwesterb/go-ristretto v1.2.3/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
> **Think:**
2+
> 我需要将这个DevLake开发环境部署文档翻译成中文。我将保持原文档的结构,但翻译所有内容到中文。
3+
>
4+
> 文档主要包括以下部分:
5+
> 1. 环境要求
6+
> 2. 如何设置开发环境
7+
> 3. 克隆仓库
8+
> 4. 安装插件依赖
9+
> 5. 配置环境变量
10+
> 6. 启动MySQL和Grafana容器
11+
> 7. 运行devlake和config-ui
12+
>
13+
> 我会逐部分翻译,保持专有名词如Docker、Golang、MySQL、Grafana等不变,同时保持命令行和代码部分不变。
14+
15+
# DevLake开发环境部署指南
16+
17+
## 环境要求
18+
- Docker v19.03.10+
19+
- Golang v1.19+
20+
- GNU Make
21+
- Mac (预装)
22+
- Windows: [下载](http://gnuwin32.sourceforge.net/packages/make.htm)
23+
- Ubuntu: `sudo apt-get install build-essential libssl-dev`
24+
25+
## 如何设置开发环境
26+
以下指南将介绍如何在开发模式下运行DevLake的前端(config-ui)和后端。
27+
28+
### 克隆代码仓库
29+
导航到您想要安装此项目的位置并克隆仓库:
30+
31+
```bash
32+
git clone https://github.com/apache/incubator-devlake.git
33+
cd incubator-devlake
34+
```
35+
36+
### 安装插件依赖
37+
38+
RefDiff 插件:
39+
安装Go包
40+
```bash
41+
cd backend
42+
go get
43+
cd ..
44+
```
45+
46+
### 配置环境文件
47+
复制示例配置文件到本地新文件:
48+
49+
```bash
50+
cp env.example .env
51+
```
52+
53+
更新`.env`文件中的以下变量:
54+
55+
- `DB_URL`: 将`mysql:3306`替换为`127.0.0.1:3306`
56+
- `DISABLED_REMOTE_PLUGINS`: 设置为`True`
57+
58+
### 启动MySQL和Grafana容器
59+
60+
确保在此步骤之前Docker守护进程正在运行。
61+
62+
> Grafana需要重新build镜像,然后在docker-compose.datasources.yml中更改image为`image: grafana:latest`
63+
64+
```bash
65+
docker-compose -f docker-compose-dev.yml up -d mysql grafana
66+
```
67+
68+
### 运行开发模式
69+
在两个单独的终端中以开发模式运行devlake和config-ui:
70+
71+
```bash
72+
# 安装poetry,按照指南操作:https://python-poetry.org/docs/#installation
73+
# 运行devlake,这里只用了q dev插件
74+
DEVLAKE_PLUGINS=q_dev nohup make dev &
75+
# 运行config-ui
76+
make configure-dev
77+
```
78+
79+
常见错误请参见故障排除文档。
80+
81+
Config UI 运行在 localhost:4000

backend/plugins/q_dev/README.md

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
-->
17+
18+
# Q Developer 插件
19+
20+
本插件用于从AWS S3获取AWS Q Developer的使用数据,并进行处理和分析。
21+
22+
## 功能
23+
24+
- 从AWS S3指定前缀下获取CSV文件
25+
- 解析CSV文件中的用户使用数据
26+
- 按用户聚合数据,计算各项指标
27+
28+
## 配置
29+
30+
配置项包括:
31+
32+
1. AWS访问密钥ID
33+
2. AWS秘钥
34+
3. AWS区域
35+
4. S3桶名称
36+
5. 速率限制(每小时)
37+
38+
通过下面的curl可以创建connection
39+
```bash
40+
curl 'http://localhost:8080/plugins/q_dev/connections' \
41+
--header 'Content-Type: application/json' \
42+
--data-raw '{
43+
"name": "q_dev_connection",
44+
"accessKeyId": "<YOUR_ACCESS_KEY_ID>",
45+
"secretAccessKey": "<YOUR_SECRET_ACCESS_KEY>",
46+
"region": "<AWS_REGION>",
47+
"bucket": "<YOUR_S3_BUCKET_NAME>",
48+
"rateLimitPerHour": 20000
49+
}'
50+
```
51+
请将以下占位符替换为实际值:
52+
<YOUR_ACCESS_KEY_ID>:您的 AWS 访问密钥 ID
53+
<YOUR_SECRET_ACCESS_KEY>:您的 AWS 访问密钥
54+
<YOUR_S3_BUCKET_NAME>:您要使用的 S3 存储桶名称
55+
<AWS_REGION>: 您要使用的 S3 所在的区域
56+
57+
通过下面的curl可以获取所有的connection
58+
```bash
59+
curl Get 'http://localhost:8080/plugins/q_dev/connections'
60+
```
61+
62+
## 数据流程
63+
64+
插件包含以下任务:
65+
66+
1. `collectQDevS3Files`: 从S3收集文件元数据信息,不下载文件内容
67+
2. `extractQDevS3Data`: 使用S3文件元数据,下载CSV数据并解析存储到数据库
68+
3. `convertQDevUserMetrics`: 将用户数据转换为聚合指标,计算平均值和总值
69+
70+
## 数据表
71+
72+
- `_tool_q_dev_connections`: 存储AWS S3连接信息
73+
- `_tool_q_dev_s3_file_meta`: 存储S3文件元数据
74+
- `_tool_q_dev_user_data`: 存储从CSV文件中解析的用户数据
75+
- `_tool_q_dev_user_metrics`: 存储聚合后的用户指标
76+
77+
## 数据收集配置
78+
收集数据步骤
79+
1. 在Config UI页面上选择左侧的`Advanced Mode`,点击`Blueprints`
80+
2. 新建Blueprint
81+
3. ![img.png](img.png) 点击右侧齿轮
82+
4.`JSON Configuration`粘贴如下JSON配置:
83+
84+
```json
85+
[
86+
[
87+
{
88+
"plugin": "q_dev",
89+
"subtasks": null,
90+
"options": {
91+
"connectionId": 5,
92+
"s3Prefix": ""
93+
}
94+
}
95+
]
96+
]
97+
```
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
/*
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package api
19+
20+
import (
21+
"net/http"
22+
23+
"github.com/apache/incubator-devlake/core/errors"
24+
"github.com/apache/incubator-devlake/core/plugin"
25+
"github.com/apache/incubator-devlake/helpers/pluginhelper/api"
26+
"github.com/apache/incubator-devlake/plugins/q_dev/models"
27+
)
28+
29+
// 连接项目的CRUD API
30+
31+
// PostConnections 创建新连接
32+
func PostConnections(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
33+
// 创建连接
34+
connection := &models.QDevConnection{}
35+
err := api.Decode(input.Body, connection, vld)
36+
if err != nil {
37+
return nil, err
38+
}
39+
// 验证
40+
// 保存到数据库
41+
err = connectionHelper.Create(connection, input)
42+
if err != nil {
43+
return nil, err
44+
}
45+
return &plugin.ApiResourceOutput{Body: connection.Sanitize(), Status: http.StatusOK}, nil
46+
}
47+
48+
// PatchConnection 更新现有连接
49+
func PatchConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
50+
connection := &models.QDevConnection{}
51+
if err := connectionHelper.First(&connection, input.Params); err != nil {
52+
return nil, err
53+
}
54+
if err := (&models.QDevConnection{}).MergeFromRequest(connection, input.Body); err != nil {
55+
return nil, errors.Convert(err)
56+
}
57+
if err := connectionHelper.SaveWithCreateOrUpdate(connection); err != nil {
58+
return nil, err
59+
}
60+
return &plugin.ApiResourceOutput{Body: connection.Sanitize(), Status: http.StatusOK}, nil
61+
}
62+
63+
// DeleteConnection 删除连接
64+
func DeleteConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
65+
conn := &models.QDevConnection{}
66+
output, err := connectionHelper.Delete(conn, input)
67+
if err != nil {
68+
return output, err
69+
}
70+
output.Body = conn.Sanitize()
71+
return output, nil
72+
}
73+
74+
// ListConnections 列出所有连接
75+
func ListConnections(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
76+
var connections []models.QDevConnection
77+
err := connectionHelper.List(&connections)
78+
if err != nil {
79+
return nil, err
80+
}
81+
// 敏感信息脱敏
82+
for i := 0; i < len(connections); i++ {
83+
connections[i] = connections[i].Sanitize()
84+
}
85+
return &plugin.ApiResourceOutput{Body: connections}, nil
86+
}
87+
88+
// GetConnection 获取单个连接详情
89+
func GetConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
90+
connection := &models.QDevConnection{}
91+
err := connectionHelper.First(connection, input.Params)
92+
if err != nil {
93+
return nil, err
94+
}
95+
return &plugin.ApiResourceOutput{Body: connection.Sanitize()}, err
96+
}

backend/plugins/q_dev/api/init.go

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
/*
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package api
19+
20+
import (
21+
"github.com/apache/incubator-devlake/core/context"
22+
"github.com/apache/incubator-devlake/core/plugin"
23+
"github.com/apache/incubator-devlake/helpers/pluginhelper/api"
24+
"github.com/go-playground/validator/v10"
25+
)
26+
27+
var vld *validator.Validate
28+
var connectionHelper *api.ConnectionApiHelper
29+
var basicRes context.BasicRes
30+
31+
func Init(br context.BasicRes, p plugin.PluginMeta) {
32+
basicRes = br
33+
vld = validator.New()
34+
connectionHelper = api.NewConnectionHelper(
35+
basicRes,
36+
vld,
37+
p.Name(),
38+
)
39+
}

0 commit comments

Comments
 (0)