Improve Docker build workflow with automated multi-arch script and docs
This commit is contained in:
yangdx
@@ -1,28 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Configuration
|
||||
IMAGE_NAME="ghcr.io/hkuds/lightrag"
|
||||
DOCKERFILE="Dockerfile"
|
||||
TAG="latest"
|
||||
|
||||
# Get version
|
||||
VERSION=$(git describe --tags --abbrev=0 2>/dev/null || echo "dev")
|
||||
|
||||
echo "Building ${IMAGE_NAME}:${TAG} (version: ${VERSION})"
|
||||
|
||||
# Build and push
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file ${DOCKERFILE} \
|
||||
--tag ${IMAGE_NAME}:${TAG} \
|
||||
--tag ${IMAGE_NAME}:${VERSION} \
|
||||
--load \
|
||||
.
|
||||
|
||||
echo "✓ Build complete!"
|
||||
echo "Image pushed: ${IMAGE_NAME}:${TAG}"
|
||||
echo "Version tag: ${IMAGE_NAME}:${VERSION}"
|
||||
|
||||
# Verify
|
||||
docker buildx imagetools inspect ${IMAGE_NAME}:${TAG}
|
||||
77
docker-build-push.sh
Executable file
77
docker-build-push.sh
Executable file
@@ -0,0 +1,77 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Configuration
|
||||
IMAGE_NAME="ghcr.io/hkuds/lightrag"
|
||||
DOCKERFILE="Dockerfile"
|
||||
TAG="latest"
|
||||
|
||||
# Get version from git tags
|
||||
VERSION=$(git describe --tags --abbrev=0 2>/dev/null || echo "dev")
|
||||
|
||||
echo "=================================="
|
||||
echo " Multi-Architecture Docker Build"
|
||||
echo "=================================="
|
||||
echo "Image: ${IMAGE_NAME}:${TAG}"
|
||||
echo "Version: ${VERSION}"
|
||||
echo "Platforms: linux/amd64, linux/arm64"
|
||||
echo "=================================="
|
||||
echo ""
|
||||
|
||||
# Check Docker login status (skip if CR_PAT is set for CI/CD)
|
||||
if [ -z "$CR_PAT" ]; then
|
||||
if ! docker info 2>/dev/null | grep -q "Username"; then
|
||||
echo "⚠️ Warning: Not logged in to Docker registry"
|
||||
echo "Please login first: docker login ghcr.io"
|
||||
echo "Or set CR_PAT environment variable for automated login"
|
||||
echo ""
|
||||
read -p "Continue anyway? (y/n) " -n 1 -r
|
||||
echo
|
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
else
|
||||
echo "Using CR_PAT environment variable for authentication"
|
||||
fi
|
||||
|
||||
# Check if buildx builder exists, create if not
|
||||
if ! docker buildx ls | grep -q "desktop-linux"; then
|
||||
echo "Creating buildx builder..."
|
||||
docker buildx create --name desktop-linux --use
|
||||
docker buildx inspect --bootstrap
|
||||
else
|
||||
echo "Using existing buildx builder: desktop-linux"
|
||||
docker buildx use desktop-linux
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Building and pushing multi-architecture image..."
|
||||
echo ""
|
||||
|
||||
# Build and push
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file ${DOCKERFILE} \
|
||||
--tag ${IMAGE_NAME}:${TAG} \
|
||||
--tag ${IMAGE_NAME}:${VERSION} \
|
||||
--push \
|
||||
.
|
||||
|
||||
echo ""
|
||||
echo "✓ Build and push complete!"
|
||||
echo ""
|
||||
echo "Images pushed:"
|
||||
echo " - ${IMAGE_NAME}:${TAG}"
|
||||
echo " - ${IMAGE_NAME}:${VERSION}"
|
||||
echo ""
|
||||
echo "Verifying multi-architecture manifest..."
|
||||
echo ""
|
||||
|
||||
# Verify
|
||||
docker buildx imagetools inspect ${IMAGE_NAME}:${TAG}
|
||||
|
||||
echo ""
|
||||
echo "✓ Verification complete!"
|
||||
echo ""
|
||||
echo "Pull with: docker pull ${IMAGE_NAME}:${TAG}"
|
||||
@@ -86,9 +86,33 @@ docker-compose up
|
||||
|
||||
Software packages requiring `transformers`, `torch`, or `cuda` will is not preinstalled in the dokcer images. Consequently, document extraction tools such as Docling, as well as local LLM models like Hugging Face and LMDeploy, can not be used in an off line enviroment. These high-compute-resource-demanding services should not be integrated into LightRAG. Docling will be decoupled and deployed as a standalone service.
|
||||
|
||||
## 📦 Build Multi-Architecture Docker Images
|
||||
## 📦 Build Docker Images
|
||||
|
||||
### Prerequisites
|
||||
### For local development and testing
|
||||
|
||||
```bash
|
||||
# Build and run with docker-compose
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
### For production release
|
||||
|
||||
**multi-architecture build and push**:
|
||||
|
||||
```bash
|
||||
# Use the provided build script
|
||||
./docker-build-push.sh
|
||||
```
|
||||
|
||||
**The build script will**:
|
||||
|
||||
- Check Docker registry login status
|
||||
- Create/use buildx builder automatically
|
||||
- Build for both AMD64 and ARM64 architectures
|
||||
- Push to GitHub Container Registry (ghcr.io)
|
||||
- Verify the multi-architecture manifest
|
||||
|
||||
**Prerequisites**:
|
||||
|
||||
Before building multi-architecture images, ensure you have:
|
||||
|
||||
@@ -96,217 +120,3 @@ Before building multi-architecture images, ensure you have:
|
||||
- Sufficient disk space (20GB+ recommended for offline image)
|
||||
- Registry access credentials (if pushing images)
|
||||
|
||||
### 1. Setup Buildx Builder
|
||||
|
||||
Create and configure a multi-architecture builder:
|
||||
|
||||
```bash
|
||||
# Create a new buildx builder instance
|
||||
docker buildx create --name multiarch-builder --use
|
||||
|
||||
# Start and verify the builder
|
||||
docker buildx inspect --bootstrap
|
||||
|
||||
# Verify supported platforms
|
||||
docker buildx inspect multiarch-builder
|
||||
```
|
||||
|
||||
You should see support for `linux/amd64` and `linux/arm64` in the output.
|
||||
|
||||
### 2. Registry Authentication
|
||||
|
||||
#### For GitHub Container Registry (ghcr.io)
|
||||
|
||||
**Option 1: Using Personal Access Token**
|
||||
|
||||
1. Create a GitHub Personal Access Token:
|
||||
- Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
|
||||
- Generate new token with `write:packages` and `read:packages` permissions
|
||||
- Copy the token
|
||||
|
||||
2. Login to registry:
|
||||
```bash
|
||||
echo "YOUR_GITHUB_TOKEN" | docker login ghcr.io -u YOUR_GITHUB_USERNAME --password-stdin
|
||||
```
|
||||
|
||||
**Option 2: Using GitHub CLI**
|
||||
|
||||
```bash
|
||||
gh auth token | docker login ghcr.io -u YOUR_GITHUB_USERNAME --password-stdin
|
||||
```
|
||||
|
||||
#### For Docker Hub
|
||||
|
||||
```bash
|
||||
docker login
|
||||
# Enter your Docker Hub username and password
|
||||
```
|
||||
|
||||
#### For Other Registries
|
||||
|
||||
```bash
|
||||
docker login your-registry.example.com
|
||||
# Enter your credentials
|
||||
```
|
||||
|
||||
### 3. Build Commands
|
||||
|
||||
#### A. Local Build (No Push)
|
||||
|
||||
Build multi-architecture images locally without pushing to registry:
|
||||
|
||||
**Normal image:**
|
||||
```bash
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file Dockerfile \
|
||||
--tag ghcr.io/hkuds/lightrag:latest \
|
||||
--load \
|
||||
.
|
||||
```
|
||||
|
||||
> **Note**: `--load` loads the image to local Docker, but only supports single platform. For multi-platform, use `--push` instead.
|
||||
|
||||
**Lite image:**
|
||||
|
||||
```bash
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file Dockerfile.lite \
|
||||
--tag ghcr.io/hkuds/lightrag:lite \
|
||||
--load \
|
||||
.
|
||||
```
|
||||
|
||||
> The lite version Docker image includes only the default storage and LLM drivers, minimizing image size.
|
||||
|
||||
#### B. Build and Push to Registry
|
||||
|
||||
Build and directly push to container registry:
|
||||
|
||||
**Normal image:**
|
||||
```bash
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file Dockerfile \
|
||||
--tag ghcr.io/hkuds/lightrag:latest \
|
||||
--push \
|
||||
.
|
||||
```
|
||||
|
||||
**Lite image:**
|
||||
```bash
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file Dockerfile.lite \
|
||||
--tag ghcr.io/hkuds/lightrag:lite \
|
||||
--push \
|
||||
.
|
||||
```
|
||||
|
||||
#### C. Build with Multiple Tags
|
||||
|
||||
Add version tags alongside latest:
|
||||
|
||||
```bash
|
||||
# Get version from git tag
|
||||
VERSION=$(git describe --tags --abbrev=0 2>/dev/null || echo "v1.0.0")
|
||||
|
||||
# Build with multiple tags
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file Dockerfile \
|
||||
--tag ghcr.io/hkuds/lightrag:latest \
|
||||
--tag ghcr.io/hkuds/lightrag:${VERSION} \
|
||||
--push \
|
||||
.
|
||||
```
|
||||
|
||||
### 4. Verify Built Images
|
||||
|
||||
After building, verify the multi-architecture manifest:
|
||||
|
||||
```bash
|
||||
# Inspect image manifest
|
||||
docker buildx imagetools inspect ghcr.io/hkuds/lightrag:latest
|
||||
|
||||
# Expected output shows multiple platforms:
|
||||
# Name: ghcr.io/hkuds/lightrag:offline
|
||||
# MediaType: application/vnd.docker.distribution.manifest.list.v2+json
|
||||
# Platforms: linux/amd64, linux/arm64
|
||||
```
|
||||
|
||||
### 5. Troubleshooting
|
||||
|
||||
#### Build Time is Very Slow
|
||||
|
||||
**Cause**: Building ARM64 on AMD64 (or vice versa) requires QEMU emulation, which is slower.
|
||||
|
||||
**Solutions**:
|
||||
- Use remote cache (`--cache-from/--cache-to`) for faster subsequent builds
|
||||
- Build on native architecture when possible
|
||||
- Be patient - initial multi-arch builds take 30-60 minutes
|
||||
|
||||
#### "No space left on device" Error
|
||||
|
||||
**Cause**: Insufficient disk space for build layers and cache.
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# Clean up Docker system
|
||||
docker system prune -a
|
||||
|
||||
# Clean up buildx cache
|
||||
docker buildx prune
|
||||
|
||||
# Check disk space
|
||||
df -h
|
||||
```
|
||||
|
||||
#### "failed to solve: failed to push" Error
|
||||
|
||||
**Cause**: Not logged into the registry or insufficient permissions.
|
||||
|
||||
**Solutions**:
|
||||
1. Verify you're logged in: `docker login ghcr.io`
|
||||
2. Check you have push permissions to the repository
|
||||
3. Verify the image name matches your repository path
|
||||
|
||||
#### Builder Not Found
|
||||
|
||||
**Cause**: Buildx builder not created or not set as current.
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# List builders
|
||||
docker buildx ls
|
||||
|
||||
# Create and use new builder
|
||||
docker buildx create --name multiarch-builder --use
|
||||
|
||||
# Or switch to existing builder
|
||||
docker buildx use multiarch-builder
|
||||
```
|
||||
|
||||
### 6. Cleanup
|
||||
|
||||
Remove builder when done:
|
||||
|
||||
```bash
|
||||
# Switch back to default builder
|
||||
docker buildx use default
|
||||
|
||||
# Remove multiarch builder
|
||||
docker buildx rm multiarch-builder
|
||||
|
||||
# Prune build cache
|
||||
docker buildx prune
|
||||
```
|
||||
|
||||
### 7. Best Practices
|
||||
|
||||
1. **Use specific tags**: Avoid only using `latest`, include version tags
|
||||
2. **Verify platforms**: Always check the manifest after pushing
|
||||
4. **Monitor resources**: Ensure sufficient disk space before building
|
||||
5. **Test both architectures**: Pull and test each platform variant
|
||||
6. **Use .dockerignore**: Exclude unnecessary files to speed up build context transfer
|
||||
|
||||
Reference in New Issue
Block a user