Documentation
HomePortalStatus

Kubernetes(k8s) Plugin

Suggest Edits

Kubernetes(k8s) Plugin

Going deeper into container realm we continue the same methodology from EC2 and Docker into K8s

(Optional)Setup kubernetes

If you don’t have a kubernetes cluster available you can use the following steps to create one using kind.

 brew install kind

After install kind you can create a cluster with kind create cluster

$ kind create cluster
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.24.0) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

And set the context to the new cluster so you can use it with kubectl.

$ kubectl cluster-info --context kind-kind
Kubernetes control plane is running at https://127.0.0.1:51141
CoreDNS is running at https://127.0.0.1:51141/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Deploy connector

Now we're ready to deploy the Border0 connector into our cluster.

We need a couple of resources in the cluster to spin up a connector.

  • ServiceAccount, the service account is used to bind the needed cluster role to the deployment
  • ClusterRole, a cluster role is used to give the deployment the permission to read the services in the cluster
  • ClusterRoleBinding, this is used to bind the role to the service account.
  • Configmap, this is used to store the the connector configuration file.
  • Deployment, the actual connector deployment

Apply the following yaml:
NOTE: you need to specify the namespace in the ClusteRoleBinding!
NOTE: you need to specify the token you want to use for the connector!

We probably want to move the token to an ENV var or something, this way we can store it as a secret in kubernetes.

If the namespace where you want to deploy the connector does not yet exists you can use the following command to create the namespace: kubectl create ns connector-test

$ kubectl create ns <namespace name>
namespace/connector-test created

Create a the following border0.yaml file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: border0-connector
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: border0-connector
rules:
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: border0-connector
subjects:
  - kind: ServiceAccount
    name: border0-connector
    namespace: <namespace> # replace this with the right namespace !!!
roleRef:
  kind: ClusterRole
  name: border0-connector
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: connector-config
data:
  config.yaml: |
    connector:
       name: "connectortest"
    credentials:
       token: <TOKEN>  # replace this with the actual token
    k8_plugin:
        - group: testgroup
          policies: [my-k8s-policy]

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: border0-connector
  name: border0-connector
spec:
  selector:
    matchLabels:
      app: border0-connector
  template:
    metadata:
      labels:
        app: border0-connector
    spec:
      serviceAccount: border0-connector
      containers:
        - image: ghcr.io/borderzero/border0:latest
          name: border0-connector
          args: ["connector", "start", "--config", "/etc/border0-connector/config.yaml"]
          env: # Optional, for logging
            - name: BORDER0_LOG_LEVEL
              value: "info"
          volumeMounts:
            - name: config
              mountPath: /etc/border0-connector
      volumes:
        - name: config
          configMap:
            name: connector-config

We can deploy this with kubectl, in this example we use namespace connector-test to deploy in. Make sure this namespace is specified in the ClusterRoleBinding in the yaml file and that the token is replaced with the actual token.

$ kubectl apply -f border0.yaml -n connector-test
serviceaccount/border0-connector created
clusterrole.rbac.authorization.k8s.io/border0-connector created
clusterrolebinding.rbac.authorization.k8s.io/border0-connector created
configmap/connector-config created
deployment.apps/border0-connector created

Now we should have a pod running the connector

$ kubectl -n connector-test get pods
NAME                                    READY   STATUS    RESTARTS   AGE
border0-connector-655c6cdccd-82fzs   1/1     Running   0          104s

And you can view the logs with:

$ kubectl -n connector-test logs -f border0-connector-655c6cdccd-82fzs
2022/07/22 11:40:13 reading the config /root/.border0_connector_config
2022/07/22 11:40:13 loading the ssm with:
2022/07/22 11:40:13 starting the connector service
2022/07/22 11:40:13 using token defined in config file
2022/07/22 11:40:16 receiving sockets to update
2022/07/22 11:40:16 statics sockets found 0
2022/07/22 11:40:16 api sockets found 3
2022/07/22 11:40:16 number of sockets to connect:  0
2022/07/22 11:40:23 receiving sockets to update
2022/07/22 11:40:23 receiving sockets to update
2022/07/22 11:40:23 statics sockets found 0
2022/07/22 11:40:23 api sockets found 3
2022/07/22 11:40:23 number of sockets to connect:  0
2022/07/22 11:40:23 statics sockets found 0
2022/07/22 11:40:23 api sockets found 3
2022/07/22 11:40:23 number of sockets to connect:  0

Setup test env, let's create a service and make it available through Border0

If you don’t have any services running you want to expose, you can use the following steps to deploy a sample service.

Create a new namespace for our test

$ kubectl create ns testsvc
namespace/testsvc created

Deploy a helloworld web deployment:
kubectl create deployment hello-server --image=us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0 -n testsvc

$ kubectl create deployment hello-server --image=us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0  -n testsvc
deployment.apps/hello-server created

Create a sample service:

$ kubectl create service clusterip -n testsvc hello-server --tcp 8080:8080
service/hello-server created

We should now have a new pod:

$ kubectl -n testsvc get pods
NAME                           READY   STATUS    RESTARTS   AGE
hello-server-bdb64769d-kdkpc   1/1     Running   0          2m33s

Test the new service:

$ kubectl -n testsvc exec -it hello-server-bdb64769d-kdkpc -- wget -O- hello-server:8080
Connecting to hello-server:8080 (10.96.38.5:8080)
writing to stdout
Hello, world!
Version: 2.0.0
Hostname: hello-server-bdb64769d-kdkpc
-                    100% |*********************************************************************************************************************************************************|    68  0:00:00 ETA
written to stdout

Expose svc as socket

After the connector is deployed you can modify any service to be exposed as a border0.io socket.

If any network policies are in place, you need to make sure the connector pod is allowed to access the services you want to expose.

We are using the annotation border0.com/group to specify the group value we set in the config file (configmap), in this case that is “testgroup”. We'll also use this annotation to instruct the border0 connector on what type of socket to create, in this case HTTP.

kubectl annotate service -n testsvc hello-server border0.com/group=testgroup border0.com/socketType=http

Optionally, for HTTP sockets, you can also set the following annotations to configure the upstream type to either http or https. border0.com/upstreamType=https

if your HTTP origin expects a specific hostname or SNI header useborder0.com/upstreamHttpHostname=www.mywebserver.com

You can validate the annotation like this kubectl -n testsvc describe service hello-server

$ kubectl -n testsvc describe service hello-server

Name:              hello-server
Namespace:         testsvc
Labels:            app=hello-server
Annotations:       border0.com/group: testgroup
Selector:          app=hello-server
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.80.39
IPs:               10.96.80.39
Port:              8080-8080  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.0.12:8080
Session Affinity:  None
Events:            <none>

In the logs from the connector you can see it will create a new socket:

2022/07/22 12:07:13 statics sockets found 1
2022/07/22 12:07:13 api sockets found 3
2022/07/22 12:07:13 creating a socket: tls-hello-server-connectortest
Connecting to Server: tunnel.border0.com

Welcome to border0.io
tls-hello-server-connectortest - tls://tls-hello-server-connectortest-bas.border0.io

=======================================================
Logs
=======================================================

By default it will create an tls socket, but with the annotation border0.com/socketType you can specify the socket type. For example in this case border0.com/socketType: http

This will create a new socket that will available as https://http-hello-server-connectortest-.border0.io

(you can also use kubectl -n testsvc edit service hello-server to modify the annotations).

Updated 14 days ago