How to capture pod to pod http traffic communication using sidecar container
Here in this article we will try to see how we capture the pod to pod communication using a sidecar container feature provided by kubernetes cluster.
Test Environment
Kubernetes Cluster – v1.28.5
Sidecar Containers
Sidecar containers are the secondary containers that run along with the main application container within the same Pod. These containers are used to enhance or to extend the functionality of the primary app container by providing additional services, or functionality such as logging, monitoring, security, or data synchronization, without directly altering the primary application code.
Kubernetes implements sidecar containers as a special case of init containers; sidecar containers remain running after Pod startup.
If you are interested in watching the video. Here is the YouTube video on the same step by step procedure outlined below.
Procedure
Step1: Ensure Kubernetes Cluster Running
As a first step ensure you have working kubernetes cluster setup.
admin@k8master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8master.stack.com Ready control-plane 215d v1.28.5
k8node.stack.com Ready <none> 215d v1.28.5Step2: Enable sidecar container feature gate for the following components
Starting with Kubernetes 1.28, a feature gate named SidecarContainers allows you to specify a restartPolicy for containers listed in a Pod’s initContainers field. But this feature is not enabled by default as stated in the “Feature Gates” documentation.
We need to enable this using the feature gate on all the required components as shown below. Update your API server, Controller, Scheduler manifest files to enable to “SidecarContainers” feature.
root@k8master:/etc/kubernetes/manifests# cat kube-apiserver.yaml | grep -i feature
- --feature-gates=SidecarContainers=true
root@k8master:/etc/kubernetes/manifests# cat kube-controller-manager.yaml | grep -i feature
- --feature-gates=SidecarContainers=true
root@k8master:/etc/kubernetes/manifests# cat kube-scheduler.yaml | grep -i feature
- --feature-gates=SidecarContainers=trueAlso update the kubelet configuration file to enable this feature on both master and worker nodes and restart the kubelet service.
root@k8master:~# cat /var/lib/kubelet/config.yaml
...
kind: KubeletConfiguration
logging:
flushFrequency: 0
options:
json:
infoBufferSize: "0"
verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
featureGates:
SidecarContainers: true
root@k8node:~# cat /var/lib/kubelet/config.yaml
...
kind: KubeletConfiguration
logging:
flushFrequency: 0
options:
json:
infoBufferSize: "0"
verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
featureGates:
SidecarContainers: trueStep3: Create a deployment with a sidecar container
Here in this step we will be using our nginx image to launch in our primary container which will act as our application container. In our sidecar container we are going to use an “sadeghrz/kubernetes-pod2pod-log” for pod to pod http traffic capture.
Here is the configmap that we will use to pass the configuration to our image. As per the documentation i am using “SEND_LOGS_TO” with “stdout” to capture the HTTP traffic and send it to the terminal output. We have other options also to send the logs to search engine which you can explore more.
admin@k8master:~$ cat sidecarloggingconfigmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: k8s-pod2pod-log-configmap
data:
SEND_LOGS_TO: stdoutHere is the deployment definition with the sidecar container for capturing the HTTP traffic for both incoming and outgoing traffic. Please note that we are launching the initcontainer in privileged mode as it requires to capture the pod interface traffic which is an operation not permitted in non privileged containers.
admin@k8master:~$ cat sidecarloggingnginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
initContainers:
- name: k8s-pod2pod-log
image: sadeghrz/kubernetes-pod2pod-log
restartPolicy: Always
envFrom:
- configMapRef:
name: k8s-pod2pod-log-configmap
securityContext:
privileged: trueStep4: Deploy application
Now let’s apply our yaml definition files to our kubernetes cluster.
admin@k8master:~$ kubectl apply -f sidecarloggingconfigmap.yaml
admin@k8master:~$ kubeclt apply -f sidecarloggingnginx.yamlStep5: Expose application
Here we will expose our deployment as a service on nodeport as shown below.
admin@k8master:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 215d
nginx-deployment NodePort 10.96.242.95 <none> 80:31568/TCP 4sStep6: Generate Incoming and Outgoing Traffic
Here we will try to generate some incoming and outgoing traffic from node and from within the pod respectively as shown below.
Traffic from node to pod
admin@k8master:~$ curl http://k8master.stack.com:31568Traffic from pod to internet
root@nginx-deployment-6db4f9dbdb-h2tdb:/# curl www.google.comStep7: Validate logs
If the privilege mode is not enabled in the initcontainer and you try to check the logs for the initcontainer you will see the following error.
admin@k8master:~$ kubectl logs -c k8s-pod2pod-log nginx-deployment-598cdb85ff-cg4v6
log to stdout
/usr/src/app/src/cap.js:46
const linkType = cap.open(IFACENAME, _CAPFILTER, bufSize, buffer);
^
Error: socket: Operation not permitted
at Object.<anonymous> (/usr/src/app/src/cap.js:46:22)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)If privilege mode is enabled, you will be able to see the incoming and outgoing http traffic logged in json format to the stdout as shown below.
admin@k8master:~$ kubectl logs -f -c k8s-pod2pod-log nginx-deployment-6db4f9dbdb-h2tdb
log to stdout
start capture http traffic on iface: eth0 with serverAddr: 10.0.1.94
check timeouted requests every 2000ms. timeout request after 10000ms
{ resTime: 0.508,
type: 'outgo',
srcIP: '10.0.0.x',
srcPort: 46986,
dstIP: '142.250.70.68',
dstPort: 80,
host: 'www.google.com',
url: '/',
method: 'GET',
resCode: '200',
HOSTNAME: 'nginx-deployment-6db4f9dbdb-h2tdb' }
{ resTime: 0.014,
type: 'outgo',
srcIP: '10.0.0.x',
srcPort: 57846,
dstIP: '23.62.41.85',
dstPort: 80,
host: 'www.ndtv.com',
url: '/',
method: 'GET',
resCode: '301',
HOSTNAME: 'nginx-deployment-6db4f9dbdb-h2tdb' }
{ resTime: 0.001,
type: 'income',
srcIP: '10.0.0.x',
srcPort: 58522,
dstIP: '10.0.0.x',
dstPort: 80,
host: 'k8master.stack.com:31568',
url: '/',
method: 'GET',
resCode: '200',
HOSTNAME: 'nginx-deployment-6db4f9dbdb-h2tdb' }Hope you enjoyed reading this article. Thank you..
Leave a Reply
You must be logged in to post a comment.