Sync fails due to "Forbidden: pod updates may not change fields other than `spec.containers[].image`, `spec.initContainers[].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)"

taskrabbit · October 20, 2022, 5:41am

Is this your first time deploying Airbyte?: No
OS Version / Instance: AWS Linux
Memory / Disk: 2 M6 2XLarge
Deployment: Kubernetes
Airbyte Version: 0.40.4
Source name/version: Salesforce
Destination name/version: 1.0.15
Step: Start a sync for a newly connected large salesforce account.
Description: The sync was moving along for the last 12 hours and managed to sync almost 40gbs of data. However, all of a sudden it fails due to “Failed to create pod orchestrator-repl-job-138756-attempt-0, pre-existing pod exists which didn’t advance out of the NOT_STARTED”. I’ve attached the logs below for troubleshooting.

2022-10-20 03:08:52 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):162 - Completing future exceptionally...
java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:341) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:91) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
	at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:184) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:336) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	... 3 more
Caused by: io.airbyte.workers.exception.WorkerException: Failed to create pod orchestrator-repl-job-138756-attempt-0, pre-existing pod exists which didn't advance out of the NOT_STARTED state.
	at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:155) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:336) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	... 3 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.20.0.1/api/v1/namespaces/default/pods/orchestrator-repl-job-138756-attempt-0. Message: Pod "orchestrator-repl-job-138756-attempt-0" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
  core.PodSpec{
  	Volumes: []core.Volume{
  		{Name: "airbyte-config", VolumeSource: {EmptyDir: &{Medium: "Memory"}}},
+ 		{
+ 			Name: "kube-api-access-2h6wc",
+ 			VolumeSource: core.VolumeSource{
+ 				Projected: &core.ProjectedVolumeSource{Sources: []core.VolumeProjection{...}, DefaultMode: &420},
+ 			},
+ 		},
  	},
  	InitContainers: nil,
  	Containers: []core.Container{
  		{
  			... // 3 identical fields
  			Args:       nil,
  			WorkingDir: "",
  			Ports: []core.ContainerPort{
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9880,
+ 					ContainerPort: 9877,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9000,
+ 					ContainerPort: 9878,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{ContainerPort: 9879, Protocol: "TCP"},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9878,
+ 					ContainerPort: 9000,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9877,
+ 					ContainerPort: 9880,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{ContainerPort: 9000, Protocol: "TCP"},
  			},
  			EnvFrom: nil,
  			Env:     {{Name: "METRIC_CLIENT"}, {Name: "DD_AGENT_HOST"}, {Name: "DD_DOGSTATSD_PORT"}, {Name: "PUBLISH_METRICS", Value: "false"}, ...},
  			Resources: core.ResourceRequirements{
- 				Limits:   core.ResourceList{},
+ 				Limits:   nil,
- 				Requests: core.ResourceList{},
+ 				Requests: nil,
  			},
  			VolumeMounts: []core.VolumeMount{
  				{Name: "airbyte-config", MountPath: "/config"},
+ 				{
+ 					Name:      "kube-api-access-2h6wc",
+ 					ReadOnly:  true,
+ 					MountPath: "/var/run/secrets/kubernetes.io/serviceaccount",
+ 				},
  			},
  			VolumeDevices: nil,
  			LivenessProbe: nil,
  			... // 10 identical fields
  		},
  	},
  	EphemeralContainers: nil,
  	RestartPolicy:       "Never",
  	... // 4 identical fields
  	ServiceAccountName:           "airbyte-admin",
  	AutomountServiceAccountToken: &true,
- 	NodeName:                     "",
+ 	NodeName:                     "ip-10-0-99-102.us-west-2.compute.internal",
  	SecurityContext:              &{},
  	ImagePullSecrets:             nil,
  	... // 16 identical fields
  }
. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
  core.PodSpec{
  	Volumes: []core.Volume{
  		{Name: "airbyte-config", VolumeSource: {EmptyDir: &{Medium: "Memory"}}},
+ 		{
+ 			Name: "kube-api-access-2h6wc",
+ 			VolumeSource: core.VolumeSource{
+ 				Projected: &core.ProjectedVolumeSource{Sources: []core.VolumeProjection{...}, DefaultMode: &420},
+ 			},
+ 		},
  	},
  	InitContainers: nil,
  	Containers: []core.Container{
  		{
  			... // 3 identical fields
  			Args:       nil,
  			WorkingDir: "",
  			Ports: []core.ContainerPort{
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9880,
+ 					ContainerPort: 9877,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9000,
+ 					ContainerPort: 9878,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{ContainerPort: 9879, Protocol: "TCP"},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9878,
+ 					ContainerPort: 9000,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9877,
+ 					ContainerPort: 9880,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{ContainerPort: 9000, Protocol: "TCP"},
  			},
  			EnvFrom: nil,
  			Env:     {{Name: "METRIC_CLIENT"}, {Name: "DD_AGENT_HOST"}, {Name: "DD_DOGSTATSD_PORT"}, {Name: "PUBLISH_METRICS", Value: "false"}, ...},
  			Resources: core.ResourceRequirements{
- 				Limits:   core.ResourceList{},
+ 				Limits:   nil,
- 				Requests: core.ResourceList{},
+ 				Requests: nil,
  			},
  			VolumeMounts: []core.VolumeMount{
  				{Name: "airbyte-config", MountPath: "/config"},
+ 				{
+ 					Name:      "kube-api-access-2h6wc",
+ 					ReadOnly:  true,
+ 					MountPath: "/var/run/secrets/kubernetes.io/serviceaccount",
+ 				},
  			},
  			VolumeDevices: nil,
  			LivenessProbe: nil,
  			... // 10 identical fields
  		},
  	},
  	EphemeralContainers: nil,
  	RestartPolicy:       "Never",
  	... // 4 identical fields
  	ServiceAccountName:           "airbyte-admin",
  	AutomountServiceAccountToken: &true,
- 	NodeName:                     "",
+ 	NodeName:                     "ip-10-0-99-102.us-west-2.compute.internal",
  	SecurityContext:              &{},
  	ImagePullSecrets:             nil,
  	... // 16 identical fields
  }
, reason=FieldValueForbidden, additionalProperties={})], group=null, kind=Pod, name=orchestrator-repl-job-138756-attempt-0, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "orchestrator-repl-job-138756-attempt-0" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
  core.PodSpec{
  	Volumes: []core.Volume{
  		{Name: "airbyte-config", VolumeSource: {EmptyDir: &{Medium: "Memory"}}},
+ 		{
+ 			Name: "kube-api-access-2h6wc",
+ 			VolumeSource: core.VolumeSource{
+ 				Projected: &core.ProjectedVolumeSource{Sources: []core.VolumeProjection{...}, DefaultMode: &420},
+ 			},
+ 		},
  	},
  	InitContainers: nil,
  	Containers: []core.Container{
  		{
  			... // 3 identical fields
  			Args:       nil,
  			WorkingDir: "",
  			Ports: []core.ContainerPort{
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9880,
+ 					ContainerPort: 9877,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9000,
+ 					ContainerPort: 9878,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{ContainerPort: 9879, Protocol: "TCP"},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9878,
+ 					ContainerPort: 9000,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{
  					Name:          "",
  					HostPort:      0,
- 					ContainerPort: 9877,
+ 					ContainerPort: 9880,
  					Protocol:      "TCP",
  					HostIP:        "",
  				},
  				{ContainerPort: 9000, Protocol: "TCP"},
  			},
  			EnvFrom: nil,
  			Env:     {{Name: "METRIC_CLIENT"}, {Name: "DD_AGENT_HOST"}, {Name: "DD_DOGSTATSD_PORT"}, {Name: "PUBLISH_METRICS", Value: "false"}, ...},
  			Resources: core.ResourceRequirements{
- 				Limits:   core.ResourceList{},
+ 				Limits:   nil,
- 				Requests: core.ResourceList{},
+ 				Requests: nil,
  			},
  			VolumeMounts: []core.VolumeMount{
  				{Name: "airbyte-config", MountPath: "/config"},
+ 				{
+ 					Name:      "kube-api-access-2h6wc",
+ 					ReadOnly:  true,
+ 					MountPath: "/var/run/secrets/kubernetes.io/serviceaccount",
+ 				},
  			},
  			VolumeDevices: nil,
  			LivenessProbe: nil,
  			... // 10 identical fields
  		},
  	},
  	EphemeralContainers: nil,
  	RestartPolicy:       "Never",
  	... // 4 identical fields
  	ServiceAccountName:           "airbyte-admin",
  	AutomountServiceAccountToken: &true,
- 	NodeName:                     "",
+ 	NodeName:                     "ip-10-0-99-102.us-west-2.compute.internal",
  	SecurityContext:              &{},
  	ImagePullSecrets:             nil,
  	... // 16 identical fields
  }
, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:555) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:342) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:322) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:649) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:195) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:200) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:141) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.replace(CreateOrReplaceHelper.java:69) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:61) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:318) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:308) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
	at io.airbyte.workers.process.AsyncOrchestratorPodProcess.create(AsyncOrchestratorPodProcess.java:320) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:149) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:336) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	... 3 more

harshith · October 20, 2022, 6:22am

Hey so the data was pulled even after the error or there is a new attempt created?

taskrabbit · October 20, 2022, 8:02am

It triggered a new attempt and progressed along. I did notice that for the streams we had finished syncing in the first attempt, it ended up resyncing those streams from the start as well. Seems like the connector state is updated at the end of the sync as opposed to when the stream finished syncing? (this is a tangential question.)

Core question I’m trying to figure out is why it errored in the first place. Our sources get pretty big (E.g. 50+ gbs of data ~ 10 hours of sync), any time it errors we end up having to do another 10 hours of syncing again. Let me know if there’s anything I can check! Appreciate your help.

harshith · October 20, 2022, 2:35pm

Ideally the error in itself looks new. My guess, this is something related to the resource. But yeah incase you face this issue again feel free to create a github issue.

taskrabbit · October 20, 2022, 3:12pm

Ran another sync again and ran into the issue across attempt 1 and 2. Will create a github issue thanks!

If there’s anything I can check or tweak, let me know as well. Appreciate it

Topic		Replies	Views
Running the launcher replication-orchestrator failed Platform, Deploy & Infra Issues connectors , deploy	7	2769	March 3, 2023
Issue with Orchestrator Pod in Airbyte on Kubernetes Platform Questions kubernetes , platform , airbyte , sync-issue , container-orchestrator	0	10	November 28, 2024
Issue with orchestrator pod not starting during sync in Airbyte v0.50.43 on k8s Platform Questions platform , question , sync-issue , container-orchestrator , orchestrator-pod	0	105	May 14, 2024
Syncs failing with 'Failed to create pod orchestrator' error Platform Questions platform , question , syncs-failing , pod-orchestrator-error , settings-enable	7	48	June 24, 2024
Error running syncs with version 0.52.0 Platform Questions platform , syncs , error , bug , version-0520	5	35	July 25, 2024

Sync fails due to "Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)"

Related topics

Sync fails due to "Forbidden: pod updates may not change fields other than `spec.containers[].image`, `spec.initContainers[].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)"