在上一篇博客《K8S – python client获取POD日志》说过了如何利用python client获取POD的标准输出/标准错误输出。
但是,能够得到log的前提是容器曾经拉起,那么程序输出到终端的内容才可以被获取到。
但是容器被拉起之前还有很多步骤,比如下载镜像失败、分配资源失败、PVC无法找到等等,这些错误则不是log可以解决的,而需要events事件。
POD拉起过程中的会通过产生event来记录发生的事情,我们只需要获取一下做展示即可排查启动故障,不过event不会一直存在,过一阵就会被清理了,所以仅仅当POD启动失败的时候观察错误原因是没有问题的。
实现
要调用的方法是list_namespaced_event,属于corev1,其API地址:https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CoreV1Api.md#list_namespaced_event。
我调用的方式是这样的:
1 |
list_namespaced_event(namespace, limit=10, pretty=True, include_uninitialized=True, field_selector='involvedObject.name={}'.format(pod_name)) |
默认是返回的是该namespace最近发生过的所有event,但是我们只关心某个POD的event,所以需要筛选。
这里利用field_selector进行字段筛选,注意它并不是对POD对象的属性做筛选,而是对Event对象做筛选。
Event就是Event,它可能是其他资源部署中创建的,Event就是一种独立的K8S资源类型,那么Event是个啥东西呢?请点击文档:https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Event.md
我们可以先不传field_selector得到namespace下面的所有的event,返回值是一个EventList,其中items字段是一个数组,里面每一个是一个Event对象,而我们的field_selecotr就是作用在Event上面的。
我们观察一下返回值:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 |
{'api_version': 'v1', 'items': [{'action': None, 'api_version': None, 'count': 1, 'event_time': None, 'first_timestamp': datetime.datetime(2019, 11, 29, 8, 46, 45, tzinfo=tzutc()), 'involved_object': {'api_version': 'v1', 'field_path': None, 'kind': 'Pod', 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4', 'namespace': 'zhongce-v2-0', 'resource_version': '2538304', 'uid': 'c3ee08ea-157e-439a-a26c-d3e59a0047a0'}, 'kind': None, 'last_timestamp': datetime.datetime(2019, 11, 29, 8, 46, 45, tzinfo=tzutc()), 'message': 'Successfully assigned ' 'zhongce-v2-0/qd-php-k8s-zhongce-service-cc6ddf9bc-596h4 ' 'to 10.42.87.6', 'metadata': {'annotations': None, 'cluster_name': None, 'creation_timestamp': datetime.datetime(2019, 11, 29, 8, 46, 45, tzinfo=tzutc()), 'deletion_grace_period_seconds': None, 'deletion_timestamp': None, 'finalizers': None, 'generate_name': None, 'generation': None, 'initializers': None, 'labels': None, 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95cdb1c74298', 'namespace': 'zhongce-v2-0', 'owner_references': None, 'resource_version': '2538309', 'self_link': '/api/v1/namespaces/zhongce-v2-0/events/qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95cdb1c74298', 'uid': '684e04be-f6f4-44bf-a799-949bf3b6dc96'}, 'reason': 'Scheduled', 'related': None, 'reporting_component': '', 'reporting_instance': '', 'series': None, 'source': {'component': 'default-scheduler', 'host': None}, 'type': 'Normal'}, {'action': None, 'api_version': None, 'count': 1, 'event_time': None, 'first_timestamp': datetime.datetime(2019, 11, 29, 8, 46, 46, tzinfo=tzutc()), 'involved_object': {'api_version': 'v1', 'field_path': 'spec.containers{zdm-app}', 'kind': 'Pod', 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4', 'namespace': 'zhongce-v2-0', 'resource_version': '2538307', 'uid': 'c3ee08ea-157e-439a-a26c-d3e59a0047a0'}, 'kind': None, 'last_timestamp': datetime.datetime(2019, 11, 29, 8, 46, 46, tzinfo=tzutc()), 'message': 'Pulling image ' '"registry-team.smzdm.com/php/qd-php-k8s-zhongce-service:master-1574841492331"', 'metadata': {'annotations': None, 'cluster_name': None, 'creation_timestamp': datetime.datetime(2019, 11, 29, 8, 46, 46, tzinfo=tzutc()), 'deletion_grace_period_seconds': None, 'deletion_timestamp': None, 'finalizers': None, 'generate_name': None, 'generation': None, 'initializers': None, 'labels': None, 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95cddfe64801', 'namespace': 'zhongce-v2-0', 'owner_references': None, 'resource_version': '2538326', 'self_link': '/api/v1/namespaces/zhongce-v2-0/events/qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95cddfe64801', 'uid': '9a4cac85-c1ea-48e8-8208-de55c0a3c744'}, 'reason': 'Pulling', 'related': None, 'reporting_component': '', 'reporting_instance': '', 'series': None, 'source': {'component': 'kubelet', 'host': '10.42.87.6'}, 'type': 'Normal'}, {'action': None, 'api_version': None, 'count': 1, 'event_time': None, 'first_timestamp': datetime.datetime(2019, 11, 29, 8, 47, tzinfo=tzutc()), 'involved_object': {'api_version': 'v1', 'field_path': 'spec.containers{zdm-app}', 'kind': 'Pod', 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4', 'namespace': 'zhongce-v2-0', 'resource_version': '2538307', 'uid': 'c3ee08ea-157e-439a-a26c-d3e59a0047a0'}, 'kind': None, 'last_timestamp': datetime.datetime(2019, 11, 29, 8, 47, tzinfo=tzutc()), 'message': 'Successfully pulled image ' '"registry-team.smzdm.com/php/qd-php-k8s-zhongce-service:master-1574841492331"', 'metadata': {'annotations': None, 'cluster_name': None, 'creation_timestamp': datetime.datetime(2019, 11, 29, 8, 47, tzinfo=tzutc()), 'deletion_grace_period_seconds': None, 'deletion_timestamp': None, 'finalizers': None, 'generate_name': None, 'generation': None, 'initializers': None, 'labels': None, 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95d13745f9d0', 'namespace': 'zhongce-v2-0', 'owner_references': None, 'resource_version': '2538375', 'self_link': '/api/v1/namespaces/zhongce-v2-0/events/qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95d13745f9d0', 'uid': '3db4233b-a9ff-4a4b-bada-69cb9b0bfa31'}, 'reason': 'Pulled', 'related': None, 'reporting_component': '', 'reporting_instance': '', 'series': None, 'source': {'component': 'kubelet', 'host': '10.42.87.6'}, 'type': 'Normal'}, {'action': None, 'api_version': None, 'count': 1, 'event_time': None, 'first_timestamp': datetime.datetime(2019, 11, 29, 8, 47, 3, tzinfo=tzutc()), 'involved_object': {'api_version': 'v1', 'field_path': 'spec.containers{zdm-app}', 'kind': 'Pod', 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4', 'namespace': 'zhongce-v2-0', 'resource_version': '2538307', 'uid': 'c3ee08ea-157e-439a-a26c-d3e59a0047a0'}, 'kind': None, 'last_timestamp': datetime.datetime(2019, 11, 29, 8, 47, 3, tzinfo=tzutc()), 'message': 'Created container zdm-app', 'metadata': {'annotations': None, 'cluster_name': None, 'creation_timestamp': datetime.datetime(2019, 11, 29, 8, 47, 3, tzinfo=tzutc()), 'deletion_grace_period_seconds': None, 'deletion_timestamp': None, 'finalizers': None, 'generate_name': None, 'generation': None, 'initializers': None, 'labels': None, 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95d1ed082c8b', 'namespace': 'zhongce-v2-0', 'owner_references': None, 'resource_version': '2538402', 'self_link': '/api/v1/namespaces/zhongce-v2-0/events/qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95d1ed082c8b', 'uid': 'a52d03a6-b044-428f-901a-33ee9cc22a6f'}, 'reason': 'Created', 'related': None, 'reporting_component': '', 'reporting_instance': '', 'series': None, 'source': {'component': 'kubelet', 'host': '10.42.87.6'}, 'type': 'Normal'}, {'action': None, 'api_version': None, 'count': 1, 'event_time': None, 'first_timestamp': datetime.datetime(2019, 11, 29, 8, 47, 3, tzinfo=tzutc()), 'involved_object': {'api_version': 'v1', 'field_path': 'spec.containers{zdm-app}', 'kind': 'Pod', 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4', 'namespace': 'zhongce-v2-0', 'resource_version': '2538307', 'uid': 'c3ee08ea-157e-439a-a26c-d3e59a0047a0'}, 'kind': None, 'last_timestamp': datetime.datetime(2019, 11, 29, 8, 47, 3, tzinfo=tzutc()), 'message': 'Started container zdm-app', 'metadata': {'annotations': None, 'cluster_name': None, 'creation_timestamp': datetime.datetime(2019, 11, 29, 8, 47, 3, tzinfo=tzutc()), 'deletion_grace_period_seconds': None, 'deletion_timestamp': None, 'finalizers': None, 'generate_name': None, 'generation': None, 'initializers': None, 'labels': None, 'name': 'qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95d1f5709844', 'namespace': 'zhongce-v2-0', 'owner_references': None, 'resource_version': '2538405', 'self_link': '/api/v1/namespaces/zhongce-v2-0/events/qd-php-k8s-zhongce-service-cc6ddf9bc-596h4.15db95d1f5709844', 'uid': '386ea8c5-f999-4df8-9f87-ef3d8be59169'}, 'reason': 'Started', 'related': None, 'reporting_component': '', 'reporting_instance': '', 'series': None, 'source': {'component': 'kubelet', 'host': '10.42.87.6'}, 'type': 'Normal'}], 'kind': 'EventList', 'metadata': {'_continue': None, 'resource_version': '2538701', 'self_link': '/api/v1/namespaces/zhongce-v2-0/events'}} |
从中可知:
- involved_object:事件的生产者是谁,借助这个字段过滤POD名字即可(如果对deployment感兴趣就过滤deployment的名字,自己试试即可)。
- message:事件的可读描述信息,我们只需要展示这个即可。
本文完
如果文章帮助您解决了工作难题,您可以帮我点击屏幕上的任意广告,或者赞助少量费用来支持我的持续创作,谢谢~
