pffxowybs
/
educoder_knowledge_base
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
educoder_knowledge_base/README.md

9.9 KiB
Raw Permalink Blame History Escape

Educoder Knowledge Base

Get Started

Requirements: Python 3.11.9 and above.

1. Environment Configuration

1.1 Create a .env file in the root directory with content similar to the following:

PROJECT_NAME=educoder-knowledge-base
# Flask Server Config
SERVER_HOST=0.0.0.0
SERVER_PORT=8730
SERVER_DEBUG=False

# Gunicorn Config
GUNICORN_BIND=0.0.0.0:8730
GUNICORN_WORKERS=2
# Unit: seconds
GUNICORN_TIMEOUT=300
# worker 类型
GUNICORN_WORKER_CLASS=gthread
# 仅适用于使用 gthread worker 的场景
GUNICORN_THREADS=4

# Celery Config
REDIS_URL=redis://:Redis_hbx20230714@r-bp1ttq2kkpxuic9uhwpd.redis.rds.aliyuncs.com:6379/9
# 设置时区（这里使用中国标准时间，可根据需求调整）
TIMEZONE="Asia/Shanghai"
# 禁用 UTC，使用本地时间
ENABLE_UTC=False
# Worker 并发设置：建议根据 CPU 核心数调整，例如 CPU 数量 * 2
CELERY_WORKER_CONCURRENCY=4
# 任务最长执行 300 秒（硬超时）
TASK_TIME_LIMIT=300
# 软超时 290 秒，任务将在此时收到中断信号
TASK_SOFT_TIME_LIMIT=290
# 任务结果缓存时间，单位秒
RESULT_EXPIRES=3600

# Redis config
REDIS_HOST=r-bp1ttq2kkpxuic9uhwpd.redis.rds.aliyuncs.com
REDIS_PORT=6379
REDIS_PASSWORD=Redis_hbx20230714
REDIS_DB=9
REDIS_MAX_CONNECTIONS=5

# LLM Config
LLM_CONFIG_FILE_PATH=/Users/bluesky/work/workspace/zhiqing/educoder-knowledge-base/llm_config.json
# QWen API Key
QWEN_API_KEY=sk-fc37000803a94f3ab0218b81f1de3f62
QWEN_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1

# islide API Key
ISLIDE_HOST_URL=https://openai-api.islide.cc
ISLIDE_API_KEY=7f4df16f-2987-4e5f-a78d-c84e6e1e365f

# kimi API Key
KIMI_API_KEY=sk-9B7mIwfbi4oyW9Whvr1rjycCt4i0hjJcDJItGIokUuea429a
KIMI_BASE_URL=https://api.moonshot.cn/v1

HTTPX_LIMITS_MAX_CONNECTIONS=30
HTTPX_LIMITS_MAX_KEEPALIVE_CONNECTIONS=5
HTTPX_TIMEOUT=600

# 文件上传路径
UPLOAD_FOLDER=uploads

# 日志配置参数
LOG_DIR=logs
LOG_FILE_NAME=educoder-knowledge-base.log
# 1GB 为最大日志文件大小，1024*1024*1024=1GB
LOG_MAX_BYTES=1073741824
# 保留旧日志文件个数
LOG_BACKUP_COUNT=20
LOG_OPENAI_LEVEL=DEBUG

# ruby接口域名
WEB_DOMAIN="https://test4-data.educoder.net"
# 识题、识卷回调
ASYNC_RECOGNIZE_CALLBACK="/api/intelligent_imports/callback.json"

# 头歌平台文件公网访问权限 secret_key
FILE_ACCESS_SECRET_KEY="806ac9e5a8b40233c99bb513a315ed66"

# 本地文件保存路径
FILE_UPLOAD_SAVE_WORKSPACE=/data/workspace
# 文件提取服务地址
FILE_EXTRACT_SERVICE_DOMAIN=http://127.0.0.1:44380

# 回调任务相关配置
CALLBACK_MAX_RETRIES=3
CALLBACK_INITIAL_DELAY=30
CALLBACK_TASK_QUEUE=low

1.2 Create a llm_config.json file with content similar to the following:

{
  "models": {
    "qwen-long": {
      "type": "document",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
      "file_rate_limiter": {
        "limit": 20,
        "period": 1,
        "timeout": 10.0
      }
    },
    "qwen-max": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-max-latest": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-turbo": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-plus": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-max-v2": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-plus-latest": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-plus-v2": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-turbo-v2": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-vl-max-latest": {
      "type": "image",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "deepseek-r1": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "deepseek-v3": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "deepseek-coder": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen-max-2025-01-25": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "deepseek-r1-distill-qwen-32b": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "deepseek-r1-distill-llama-70b": {
      "type": "text",
      "provider": "online_openai",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1"
    },
    "qwen2.5:72b": {
      "type": "text",
      "provider": "local_openai",
      "api_key": "ollama",
      "base_url": "http://8.147.118.14:48080/v1"
    },
    "dashscope.qwen-plus": {
      "type": "text",
      "provider": "online_dashscope",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "empty"
    },
    "dashscope.deepseek-r1": {
      "type": "text",
      "provider": "online_dashscope",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "empty"
    },
    "dashscope.deepseek-v3": {
      "type": "text",
      "provider": "online_dashscope",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "empty"
    },
    "dashscope.wanx2.1-t2i-turbo": {
      "type": "t2i",
      "provider": "online_dashscope",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "empty"
    },
    "text-embedding-v3": {
      "type": "embedding",
      "provider": "bisheng_embedding",
      "api_key": "sk-fc37000803a94f3ab0218b81f1de3f62",
      "base_url": "143.64.38.7:49530",
      "collection_name": "partition_2_knowledge_1"
    }
  }
}

2. Installation

2.1 Create and Activate Virtual Environment

# Your current directory should be the root directory of the project
cd educoder-knowledge-base
python -m venv venv 
source venv/bin/activate

2.2 Install Dependencies

# Your current directory should be the root directory of the project
cd educoder-knowledge-base
pip install -r requirements.txt

3. Running the Application

3.1 Development Environment

Running the Flask Application To run the Flask app in development mode:

cd educoder-knowledge-base
python main.py

Running the Celery Worker To run the Celery worker for asynchronous tasks:

cd educoder-knowledge-base
celery -A main_celery worker --loglevel=info -Q high,celery,low

Note:

  • main_celery.py is your Celery entry point file.
  • Ensure your Celery configuration (e.g., in celery_config.py) includes the proper include settings (e.g., include=["app.tasks"]) so all tasks are registered.

3.2 Production Environment

Running the Flask Application with Gunicorn Deploy the Flask app using Gunicorn:

cd educoder-knowledge-base
nohup gunicorn -c app/config/gunicorn_config.py main:app > /dev/null 2>&1 &
  • main:app assumes your Flask app instance is named app in main.py.

Running the Celery Worker In production, run Celery workers as independent processes managed by Supervisor or systemd.

Using Supervisor Create a Supervisor configuration file (e.g., /etc/supervisor/conf.d/main_celery.conf):

[program:main_celery]
command=/path/to/your/venv/bin/celery -A main_celery worker --loglevel=info -Q high,celery,low 
directory=/path/to/your/educoder-knowledge-base
autostart=true
autorestart=true
stderr_logfile=/var/log/main_celery.err.log
stdout_logfile=/var/log/main_celery.out.log
user=your_username

Then reload Supervisor and start the worker:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start main_celery
sudo supervisorctl restart main_celery

4. Project Structure

A recommended project structure:

educoder-knowledge-base/
├── app/
│   ├── api/           
│   ├── config/       
│   ├── libs/          
│   ├── manager/       
│   ├── models/        
│   ├── service/
│   └── tasks/
├── docs/
|── logs/               # Local log files
├── tests/
├── uploads/            # Local file uploads
├── venv/               # Virtual environment
└── .env
├── .gitignore
├── main.py             # Flask application entry point
├── main_celery.py      # Celery application entry point
├── README.md
├── requirements.txt