Gradle Build System in Apache Beam

Overview

Apache Beam is a mono-repo using Gradle as its build system. The entire project (Java, Python, Go, website) is managed as a single Gradle project.

Key Files

• build.gradle.kts - Root build configuration
• settings.gradle.kts - Project structure and module definitions
• gradle.properties - Global properties and versions
• buildSrc/ - Custom Gradle plugins including BeamModulePlugin

BeamModulePlugin

Located at buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Purpose

• Manages Java dependencies centrally
• Configures project types (Java, Python, Go, Proto, Docker, etc.)
• Defines common custom tasks

Java Project Configuration

apply plugin: 'org.apache.beam.module'
applyJavaNature(
    automaticModuleName: 'org.apache.beam.sdk.io.kafka'
)

Common Commands

Build

# Build entire project
./gradlew build

# Build specific project
./gradlew :sdks:java:core:build
./gradlew -p sdks/java/core build

# Compile only (no tests)
./gradlew :sdks:java:core:compileJava

Test

# Run tests
./gradlew :sdks:java:core:test

# Run specific test
./gradlew :sdks:java:core:test --tests *MyTest

# Skip tests
./gradlew build -x test

Clean

# Clean specific project
./gradlew :sdks:java:core:clean

# Clean everything
./gradlew clean

Formatting

# Java formatting (Spotless)
./gradlew spotlessApply

# Check formatting
./gradlew spotlessCheck

# Format CHANGES.md
./gradlew formatChanges

Publishing

# Publish to Maven Local
./gradlew -Ppublishing :sdks:java:core:publishToMavenLocal

# Publish all Java artifacts
./gradlew -Ppublishing publishToMavenLocal

Pre-commit Tasks

Java

./gradlew javaPreCommit

Python

./gradlew pythonPreCommit

Combined

./gradlew :checkSetup  # Validates Go, Java, Python environments

Useful Flags

Flag	Description
-p <path>	Run task in specific project directory
-x <task>	Exclude task
--tests <pattern>	Filter tests
-Ppublishing	Enable publishing tasks
-PdisableSpotlessCheck=true	Disable formatting check
-PdisableCheckStyle=true	Disable checkstyle
-PskipCheckerFramework	Skip Checker Framework
--continue	Continue after failures
--info	Verbose output
--debug	Debug output
--scan	Generate build scan
--parallel	Parallel execution

GCP-related Properties

-PgcpProject=my-project
-PgcpRegion=us-central1
-PgcpTempRoot=gs://bucket/temp
-PgcsTempRoot=gs://bucket/temp

Docker Tasks

# Build Java SDK container
./gradlew :sdks:java:container:java11:docker

# Build Python SDK container
./gradlew :sdks:python:container:py312:docker

# Build and push a container into a custom repository
./gradlew :sdks:java:container:java11:docker \
  -Pdocker-repository-root=gcr.io/project \
  -Pdocker-tag=custom \
  -Ppush-containers

If a :docker task produces logs that contain the following:

WARNING: No output specified with docker-container driver.
Build result will only remain in the build cache.

then you must use -PuseDockerBuildx when running :docker tasks in this environment. For example:

# Build and push a go container into a custom repository
./gradlew :sdks:go:container:docker \
  -Pdocker-repository-root=gcr.io/project \
  -Pdocker-tag=custom \
  -Ppush-containers \
  -PuseDockerBuildx

Dependency Management

View Dependencies

./gradlew :sdks:java:core:dependencies
./gradlew :sdks:java:core:dependencies --configuration runtimeClasspath

Force Dependency Version

In build.gradle:

configurations.all {
    resolutionStrategy.force 'com.google.guava:guava:32.0.0-jre'
}

Troubleshooting

Clean Gradle Cache

rm -rf ~/.gradle/caches
rm -rf .gradle
rm -rf build

Common Errors

NoClassDefFoundError

• Run ./gradlew clean
• Delete gradle cache

Proto-related Errors

• Regenerate protos: ./gradlew generateProtos

Dependency Conflicts

• Check dependencies: ./gradlew dependencies
• Use --scan for detailed analysis

Useful Tasks

# List all tasks
./gradlew tasks

# List tasks for a project
./gradlew :sdks:java:core:tasks

# Show project structure
./gradlew projects

IDE Integration

IntelliJ

1. Open repository root as Gradle project
2. Wait for indexing
3. Gradle tool window shows all tasks

VS Code

Install Gradle extension for task discovery