I have configured my gitlab runner to work with the kubernetes pod. I see that my angular build job gets stuck at below and it keeps going on Warning: Worker #10 exited unexpectedly (code: null | signal: SIGKILL). Current task: {entryPoint: @angular/forms, formatProperty: es2015, processDts: Yes} Current phase: compiling I believe it may be because of a memory/cpu issue. But I am not able to debug as to what would be the issue. Below are my values.yaml file I used for setting up runners. But below mentioned cpu/memory are not picked up while runners are running. ## GitLab Runner Image ## ## By default it's using gitlab/gitlab-runner:alpine-v{VERSION} ## where {VERSION} is taken from Chart.yaml from appVersion field ## ## ref: https://hub.docker.com/r/gitlab/gitlab-runner/tags/ ## image: registry:<artifactory>/docker-images/gitlab image: gitlab-runner tag: alpine-v13.1.0 ## Specify a imagePullPolicy ## 'Always' if imageTag is 'latest', else set to 'IfNotPresent' ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## imagePullPolicy: Always ## The GitLab Server URL (with protocol) that want to register the runner against ## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register ## gitlabUrl: <gitlab-url> ## The Registration Token for adding new Runners to the GitLab Server. This must ## be retrieved from your GitLab Instance. ## ref: https://docs.gitlab.com/ce/ci/runners/README.html ## runnerRegistrationToken: "sdsdsdsdsfsfsfsd" ## The Runner Token for adding new Runners to the GitLab Server. This must ## be retrieved from your GitLab Instance. It is token of already registered runner. ## ref: (we don't yet have docs for that, but we want to use existing token) ## # runnerToken: "" # ## Unregister all runners before termination ## ## Updating the runner's chart version or configuration will cause the runner container ## to be terminated and created again. This may cause your Gitlab instance to reference ## non-existant runners. Un-registering the runner before termination mitigates this issue. ## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-unregister ## unregisterRunners: true ## When stopping the runner, give it time to wait for its jobs to terminate. ## ## Updating the runner's chart version or configuration will cause the runner container ## to be terminated with a graceful stop request. terminationGracePeriodSeconds ## instructs Kubernetes to wait long enough for the runner pod to terminate gracefully. ## ref: https://docs.gitlab.com/runner/commands/#signals terminationGracePeriodSeconds: 9000 ## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use ## Provide resource name for a Kubernetes Secret Object in the same namespace, ## this is used to populate the /home/gitlab-runner/.gitlab-runner/certs/ directory ## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates ## # certsSecretName: ## Configure the maximum number of concurrent jobs ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section ## concurrent: 10 ## Defines in seconds how often to check GitLab for a new builds ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section ## checkInterval: 30 ## Configure GitLab Runner's logging level. Available values are: debug, info, warn, error, fatal, panic ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section ## # logLevel: ## Configure GitLab Runner's logging format. Available values are: runner, text, json ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section ## # logFormat: ## For RBAC support: rbac: create: false ## Define specific rbac permissions. # resources: ["pods", "pods/exec", "secrets"] # verbs: ["get", "list", "watch", "create", "patch", "delete"] ## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs ## cluster-wide or only within namespace clusterWideAccess: false ## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create) ## serviceAccountName: xyz-prod-admin ## Specify annotations for Service Accounts, useful for annotations such as eks.amazonaws.com/role-arn ## ## ref: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html ## # serviceAccountAnnotations: {} ## Configure integrated Prometheus metrics exporter ## ref: https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server metrics: enabled: false ## Configuration for the Pods that that the runner launches for each new job ## runners: ## Default container image to use for builds when none is specified ## image: name: gcr.io/kaniko-project/executor:debug entrypoint: [""] ## Specify one or more imagePullSecrets ## ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ## # imagePullSecrets: [] ## Specify the image pull policy: never, if-not-present, always. The cluster default will be used if not set. ## imagePullPolicy: "always" ## Defines number of concurrent requests for new job from GitLab ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section ## # requestConcurrency: 1 ## Specify whether the runner should be locked to a specific project: true, false. Defaults to true. ## # locked: true ## Specify the tags associated with the runner. Comma-separated list of tags. ## ## ref: https://docs.gitlab.com/ce/ci/runners/#using-tags ## # tags: "" ## Specify if jobs without tags should be run. ## If not specified, Runner will default to true if no tags were specified. In other case it will ## default to false. ## ## ref: https://docs.gitlab.com/ce/ci/runners/#allowing-runners-with-tags-to-pick-jobs-without-tags ## runUntagged: true ## Specify whether the runner should only run protected branches. ## Defaults to False. ## ## ref: https://docs.gitlab.com/ee/ci/runners/#protected-runners ## # protected: true ## The name of the secret containing runner-token and runner-registration-token # secret: gitlab-runner ## Namespace to run Kubernetes jobs in (defaults to the same namespace of this release) ## # namespace: ## The amount of time, in seconds, that needs to pass before the runner will ## timeout attempting to connect to the container it has just created. ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html pollTimeout: 180 ## Set maximum build log size in kilobytes, by default set to 4096 (4MB) ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section outputLimit: 4096 ## Distributed runners caching ## ref: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/configuration/autoscale.md#distributed-runners-caching ## ## If you want to use s3 based distributing caching: ## First of all you need to uncomment General settings and S3 settings sections. ## ## Create a secret 's3access' containing 'accesskey' & 'secretkey' ## ref: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/ ## ## $ kubectl create secret generic s3access \ ## --from-literal=accesskey="YourAccessKey" \ ## --from-literal=secretkey="YourSecretKey" ## ref: https://kubernetes.io/docs/concepts/configuration/secret/ ## ## If you want to use gcs based distributing caching: ## First of all you need to uncomment General settings and GCS settings sections. ## ## Access using credentials file: ## Create a secret 'google-application-credentials' containing your application credentials file. ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscachegcs-section ## You could configure ## $ kubectl create secret generic google-application-credentials \ ## --from-file=gcs-application-credentials-file=./path-to-your-google-application-credentials-file.json ## ref: https://kubernetes.io/docs/concepts/configuration/secret/ ## ## Access using access-id and private-key: ## Create a secret 'gcsaccess' containing 'gcs-access-id' & 'gcs-private-key'. ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-cache-gcs-section ## You could configure ## $ kubectl create secret generic gcsaccess \ ## --from-literal=gcs-access-id="YourAccessID" \ ## --from-literal=gcs-private-key="YourPrivateKey" ## ref: https://kubernetes.io/docs/concepts/configuration/secret/ cache: {} ## General settings # cacheType: s3 # cachePath: "gitlab_runner" # cacheShared: true ## S3 settings # s3ServerAddress: s3.amazonaws.com # s3BucketName: # s3BucketLocation: # s3CacheInsecure: false # secretName: s3access ## GCS settings # gcsBucketName: ## Use this line for access using access-id and private-key # secretName: gcsaccess ## Use this line for access using google-application-credentials file # secretName: google-application-credentials ## Build Container specific configuration ## builds: cpuLimit: 3 cpuLimitOverwriteMaxAllowed: 3 memoryLimit: 2Gi memoryLimitOverwriteMaxAllowed: 5Gi cpuRequests: 2 cpuRequestsOverwriteMaxAllowed: 2 memoryRequests: 1Gi memoryRequestsOverwriteMaxAllowed: 2Gi ## Service Container specific configuration ## services: cpuLimit: 3 memoryLimit: 4Gi cpuRequests: 2 memoryRequests: 2Gi ## Helper Container specific configuration ## helpers: cpuLimit: 512m memoryLimit: 512Mi cpuRequests: 512m memoryRequests: 512Mi image: "<artifactory>/docker-images/gitlab/gitlab-runner-helper:x86_64-6214287e" ## Helper container security context configuration ## Refer to https://docs.gitlab.com/runner/executors/kubernetes.html#using-security-context # pod_security_context: # run_as_non_root: true # run_as_user: 100 # run_as_group: 100 # fs_group: 65533 # supplemental_groups: [101, 102] ## Service Account to be used for runners ## # serviceAccountName: ## If Gitlab is not reachable through $CI_SERVER_URL ## # cloneUrl: ## Specify node labels for CI job pods assignment ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ ## # nodeSelector: {} ## Specify node tolerations for CI job pods assignment ## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ ## # nodeTolerations: {} ## Specify pod labels for CI job pods ## # podLabels: {} ## Specify annotations for job pods, useful for annotations such as iam.amazonaws.com/role # podAnnotations: {} ## Configure environment variables that will be injected to the pods that are created while ## the build is running. These variables are passed as parameters, i.e. `--env "NAME=VALUE"`, ## to `gitlab-runner register` command. ## ## Note that `envVars` (see below) are only present in the runner pod, not the pods that are ## created for each build. ## ## ref: https://docs.gitlab.com/runner/commands/#gitlab-runner-register ## env: DOCKER_TLS_CERTDIR: "" DOCKER_DRIVER: overlay2 ## Configure securitycontext ## ref: http://kubernetes.io/docs/user-guide/security-context/ ## securityContext: # fsGroup: 65533 runAsUser: 100 ## Configure resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ ## resources: limits: memory: 5Gi cpu: 2 requests: memory: 2Gi cpu: 1 ## Affinity for pod assignment ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity ## affinity: {} ## Node labels for pod assignment ## Ref: https://kubernetes.io/docs/user-guide/node-selection/ ## nodeSelector: {} # Example: The gitlab runner manager should not run on spot instances so you can assign # them to the regular worker nodes only. # node-role.kubernetes.io/worker: "true" ## List of node taints to tolerate (requires Kubernetes >= 1.6) ## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ ## tolerations: [] # Example: Regular worker nodes may have a taint, thus you need to tolerate the taint # when you assign the gitlab runner manager with nodeSelector or affinity to the nodes. # - key: "node-role.kubernetes.io/worker" # operator: "Exists" ## Configure environment variables that will be present when the registration command runs ## This provides further control over the registration process and the config.toml file ## ref: `gitlab-runner register --help` ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html ## # envVars: # - name: RUNNER_EXECUTOR # value: kubernetes ## list of hosts and IPs that will be injected into the pod's hosts file hostAliases: [] # Example: # - ip: "127.0.0.1" # hostnames: # - "foo.local" # - "bar.local" # - ip: "10.1.2.3" # hostnames: # - "foo.remote" # - "bar.remote" ## Annotations to be added to manager pod ## podAnnotations: {} # Example: # iam.amazonaws.com/role: <my_role_arn> ## Labels to be added to manager pod ## podLabels: {} # Example: # owner.team: <my_cool_team> ## HPA support for custom metrics: ## This section enables runners to autoscale based on defined custom metrics. ## In order to use this functionality, Need to enable a custom metrics API server by ## implementing "custom.metrics.k8s.io" using supported third party adapter ## Example: https://github.com/directxman12/k8s-prometheus-adapter ## #hpa: {} # minReplicas: 1 # maxReplicas: 10 # metrics: # - type: Pods # pods: # metricName: gitlab_runner_jobs # targetAverageValue: 400m Continue reading...