RBAC DENY: user # what it was? Pod ended with: too old resource version containerd: unable to save an empty namespace may not be set when a resource name timeout expired waiting for volumes to attach Fast watcher, slow processing # Temporary (current evaluation) # --------- apiserver.*http: Handler timeout # apiserver was not restarted. Why? etcdserver: request timed out Error syncing job: Timeout: request did not complete within allowed duration Timeout: request did not complete within allowed duration the server cannot complete the requested operation at this time, try again later unable to create pods: the server cannot complete the requested operation at this time Server.processUnaryRPC failed to write status # Whats what???? Why it was triggered? route ip\+net: no such network interface # And what? dockerd-current:.*Error getting v2 registry # OpenShift docker registry died by that moment forbidden: no providers available to validate pod request # This when I tried to old supplementalGroup with coma. Unclear why this was wrong, but ok. unable to validate against any security context constrain # This was supplementalGroup missing in namespace definition ImagePullBackOff: "Back-off pulling image # This was due to registry diying, making it inaccessible ImagePullBackOff: Back-off pulling image PullImage.*getsockopt: connection refused Error syncing pod.*getsockopt: connection refused secret.*not registered # Unclear but was related to development of davmail Couldn't get secret.*not registered Chown failed on Chmod failed on # Too many parallel mounts cause malfunction (transient scope) https://github.com/kubernetes/kubernetes/issues/79194 # ----------------------- Mount failed: exit status 1 mount failed: exit status 1 Failed to start transient scope unit Mounting command: systemd-run # potentially we can use different mounter? Mounting arguments: origin-node: Output: umount.*target is busy # this is probably a very different problem origin-node: Unmounting arguments: Error: UnmountVolume.TearDown failed for volume # docker cgroup-driver related stuff (needs inverstigation) https://github.com/kubernetes/kubernetes/issues/76531 # -------------------- Error while processing event /sys/fs/cgroup/memory RecentStats: unable to find data for container Failed to get RecentStats Failed to read "/sys/fs/cgroup Failed to update stats for container "/user.slice Failed to create summary reader for Failed to detect containers: readdirent: no such file or directory unable to find data for container /system origin-node: container /kubepods.slice # Known errors which we tolerate # ------------------------------ has become an orphan, killing it # left-over resources, etc. is already stopped but volume paths are still present on disk Same superblock, different security settings required revision has been compacted # etcd Could not get instant cpu stats: different number of cpus # performance (may be worth looking into) # ----------- du and find on following dirs took # Set of problems with sandboxes, cleaning resources (CronJobs). It was there before and probably not critical # ------------------------------ Failed to stop sandbox killPodWithSyncResult failed Cannot find network namespace for the terminated container rpc error: code = 4 desc = context deadline exceeded rpc error: code = 2 desc = no such process cannot delete builds.build.openshift.io Unmount skipped because path does not exist stderr: du: cannot access CNI request failed with status 400 # https://github.com/kubernetes/kubernetes/issues/72044 (not a problem) network: failed to Statfs Error while adding to cni lo network ns/net: No such file or directory # this is still the same ns/net": no such file or directory garbagecollector.go.*not found # containers stopped/dying simmultaneously # ------------------- not found in pod's containers # Unclear, containers died in the pod (before cgroups killed one container, but many more dissapperead for some reason shortly) failed to exit within.*trying direct SIGKILL # Unclear, for some reason many containers (pods) were stopped simulatenously.... failed to exit within.*using the force No ref for container # Related. Then, many of them seems exited before or what happened? Can't make a ref to pod Error: No such container Error deleting network when building cni runtime rpc error: code = 2 desc = container not running # Related. The same rpc error: code = 2 desc = Error: No such container rpc error: code = 2 desc = containerd: process not found for container returned error: No such container containerd: container not found json returned error: No such container devmapper: Unknown device # also emptyDir volumes seems missing Both sandbox container and checkpoint for id.*could not be found NetworkPlugin cni failed to teardown pod Failed to retrieve checkpoint for sandbox cannot exec a container that has run and stopped cannot join network of a non running container Container.*is not running # Filtering some known and repeated stuff # -------------------- adei # adei cron jobs mounting/unmounting adei-* volumes # Normal errors, due to current development efforts # ------------- restarting failed container error processing PVC Could not construct volume information: no volume plugin matched Readiness probe for Liveness probe for Failing rollout for is dead, but RestartPolicy says that we should restart it the object has been modified; please apply your changes to the latest version and try again container start failed: ErrImagePull starting container process caused returned error: No such image Error setting up exec command in container endpoints "gfs" not found # I forgot to create it while creating new ns Unable to authenticate the request due to an error volume-subpaths: no such file or directory # Normal errors, like communication failures, etc. # ------------- Error proxying data from client to backend Error proxying data from backend to client # Suppress multi-line output what we can't interpret after filtering # -------------------------- In some cases useful info about processes that use # suppress multi-line output the device is found by with error: exit status origin-node: - exit status origin-node: ' the following error information was pulled from the glusterfs log to help diagnose this issue # Non errors from openshift # ------------------------- origin-master-controllers: I0923 # informational origin-master-controllers: I0925 origin-master-controllers: I0926 # DS: May worth investigating, scheduling failures origin-node: I0923 origin-node: I0924 origin-node: I0925 origin-node: I0926 tomic-openshift-master-api: I0926 atomic-openshift-master-api:.*Trace origin-master-controllers: Trace Starting watch for updating pod condition for Using node IP Delete endpoint is released and reclaim policy found previous inflight deployment already in flight, abandoning entered phase "Released" Checking for prebound volumes with node affinity Discovered runtime cgroups name is not a mountpoint, deleting List /apis LoadBalancerRR: Setting endpoints for dockerd-current:.*cleaned up dockerd-current:.*Attempting next endpoint for pull after error # This is a problem, but this is secondary messages... dockerd-current:.*Failed to cancel upload: unauthorized: authentication required # System informational Messages # ----------------------------- kernel: XFS systemd: Starting systemd: Created systemd: Stopping systemd: Removed systemd: Started systemd: Stopped systemd-logind: New session systemd-logind: Removed session etcd: store.index etcd: segmented wal file etcd: finished scheduled compaction etcd: compacted rev etcd: purged etcd: start etcd: saved etcd: compacted etcd: avoid queries with large range device-mapper: ioctl: remove_all left.*device dnsmasq.*setting upstream servers from DBus dnsmasq.*using nameserver kernel: nf_conntrack: falling back to vmalloc IPv6: ADDRCONF NetworkManager.*veth ntpd.*veth kernel: device veth # infamouse veth devices veth.*link connected Listen normally on Creating netns rotating log files