Life of a Prow Job

NOTE: This document uses 5df7636b83cab54e248e550a31dbf1e4731197a6 (July 21, 2021) as a reference point for all code links.

Let’s pretend a user comments /test all on a Pull Request (PR). In response, GitHub posts this comment to Prow via a webhook. See examples for webhook payloads.

Prow’s Kubernetes cluster uses an ingress resource for terminating TLS, and routes traffic to the hook service resource, finally sending the traffic to the hook application, which is defined as a deployment:

• This document describes the configuration for the ingress resource.
• This document describes the configuration for the hook service.
• This document defines the pods for the hook application.

The pods for hook run the hook executable. hook listens for incoming HTTP requests and translates them to “GitHub event objects”. For example, in the case of the /test all comment from above, hook builds an GenericCommentEvent. Afterwards, hook broadcasts these events to Prow Plugins.

Prow Plugins receive 2 objects:

1. a GitHub event object, and
2. a ClientAgent object.

The ClientAgent object contains the following clients:

• GitHub client
• Prow job client
• Kubernetes client
• BuildClusterCoreV1 clients
• Git client
• Slack client
• Owners client
• Bugzilla client
• Jira client

These clients are initialized by hook, during start-up.

hook handles events by looking at X-GitHub-Event, a custom HTTP header. Afterwards, a ConfigAgent object, initialized during hook’s startup, selects plugins to handle events. See githubeventserver.go for more details, and check plugins.yaml for a list of plugins per repo.

Going back to the example, hook delivers an event that represents the /test all comment to the Trigger plugin. The Trigger plugin validates the PR before running tests. One such validation is, for instance, that the author is a member of the organization or that the PR is labeled ok-to-test. The function called handleGenericComment describes Trigger’s logic.

If all conditions are met (ok-to-test, the comment is not a bot comment, etc.), handleGenericComment determines which presubmit jobs to run. The initial list of presubmit jobs to run (before being filtered down to those that qualify for this particular comment), is retrieved with getPresubmits.

Next, for each presubmit we want to run, the trigger plugin talks to the Kubernetes API server and creates a ProwJob with the information from the PR comment. The ProwJob is primarily composed of the Spec and Status objects.

Pod details aside, a sample ProwJob might look like this:

apiVersion: prow.k8s.io/v1
kind: ProwJob
metadata:
  name: 32456927-35d9-11e7-8d95-0a580a6c1504
spec:
  job: pull-test-infra-bazel
  decorate: true
  pod_spec:
    containers:
    - image: gcr.io/k8s-staging-test-infra/bazelbuild:latest-test-infra
  refs:
    base_ref: master
    base_sha: 064678510782db5b382df478bb374aaa32e577ea
    org: kubernetes
    pulls:
    - author: ixdy
      number: 2716
      sha: dc32ccc9ea3672ccc523b7cbaa8b00360b4183cd
    repo: test-infra
  type: presubmit
status:
  startTime: 2017-05-10T23:34:22.567457715Z
  state: triggered

prow-controller-manager runs ProwJobs by launching them by creating a new Kubernetes pod. It knows how to schedule new ProwJobs onto the cluster, responding to changes in the ProwJob or cluster health.

When the ProwJob finishes (the containers in the pod have finished running), prow-controller-manager updates the ProwJob. crier reports back the status of the ProwJob back to the various external services like GitHub (e.g., as a green check-mark on the PR where the original /test all comment was made).

A day later, sinker notices that the job and pod are a day old and deletes them from the Kubernetes API server.

Here is a summary of the above:

1. User types in /test all as a comment into a GitHub PR.
2. GitHub sends a webhook (HTTP request) to Prow, to the prow.k8s.io/hook endpoint.
3. The request gets intercepted by the ingress.
4. The ingress routes the request to the hook service.
5. The hook service in turn routes traffic to the hook application, defined as a deployment.
6. The container routes traffic to the hook binary inside it.
7. hook binary parses and validates the HTTP request and creates a GitHub event object.
8. hook binary sends the GitHub event object (in this case GenericCommentEvent) to handleGenericCommentEvent.
9. handleGenericCommentEvent sends the data to be handled by the handleEvent.
10. The data in the comment gets sent from hook to one of its many plugins, one of which is trigger. (The pattern is that hook constructs objects to be consumed by various plugins.)
11. trigger determines which presubmit jobs to run (because it sees the /test command in /test all).
12. trigger creates a ProwJob object!
13. prow-controller-manager creates a pod to start the ProwJob.
14. When the ProwJob’s pod finishes, prow-controller-manager updates the ProwJob.
15. crier sees the updated ProwJob status and reports back to the GitHub PR (creating a new comment).
16. sinker cleans up the old pod from above and deletes it from the Kubernetes API server.