Playing with OCP 4.9

 Thursday 24 February 2022  16:46:16 +0000 (0:00:20.765)       0:05:44.496 *****

===============================================================================

openshift_node : Install openshift packages ----------------------------------------------------------------- 242.41s

openshift_node : Reboot the host and wait for it to come back ------------------------------------------------ 34.46s

openshift_node : Wait for node to report ready --------------------------------------------------------------- 20.77s

openshift_node : Approve node CSRs ---------------------------------------------------------------------------- 6.79s

openshift_node : Pull release image --------------------------------------------------------------------------- 6.63s

openshift_node : Get available cri-o RPM versions ------------------------------------------------------------- 4.68s

openshift_node : Pull MCD image ------------------------------------------------------------------------------- 3.39s

openshift_node : Get machine controller daemon image from release image --------------------------------------- 3.18s

openshift_node : Apply ignition manifest ---------------------------------------------------------------------- 2.49s

openshift_node : Fetch bootstrap ignition file locally -------------------------------------------------------- 2.10s

openshift_node : Setting sebool container_use_cephfs ---------------------------------------------------------- 1.73s

openshift_node : Setting sebool container_manage_cgroup ------------------------------------------------------- 1.64s

openshift_node : Setting sebool virt_use_samba ---------------------------------------------------------------- 1.54s

Gathering Facts ----------------------------------------------------------------------------------------------- 1.43s

openshift_node : Write /etc/containers/registries.conf -------------------------------------------------------- 1.25s

openshift_node : Enable the CRI-O service --------------------------------------------------------------------- 0.75s

openshift_node : Restart the CRI-O service -------------------------------------------------------------------- 0.64s

openshift_node : Check for cluster no proxy ------------------------------------------------------------------- 0.62s

openshift_node : Check for cluster http proxy ----------------------------------------------------------------- 0.62s

openshift_node : Check for cluster https proxy ---------------------------------------------------------------- 0.61s

(venv) [krishna@bastion openshift-ansible]$

(venv) [krishna@bastion openshift-ansible]$ oc get nodes

NAME                                        STATUS   ROLES           AGE     VERSION

ip-10-0-185-24.us-east-2.compute.internal   Ready    master,worker   5h55m   v1.22.3+fdba464

ip-10-0-30-244.us-east-2.compute.internal   Ready    worker          3m31s   v1.22.0-rc.0+a44d0f0

(venv) [krishna@bastion openshift-ansible]$ export RHEL_NODE=$(oc get nodes -l node.openshift.io/os_id=rhel -o jsonpath="{.items[0].metadata.name}")

(venv) [krishna@bastion openshift-ansible]$ echo "export RHEL_NODE=${RHEL_NODE}" >> ~/.bashrc

(venv) [krishna@bastion openshift-ansible]$ echo $RHEL_NODE

ip-10-0-30-244.us-east-2.compute.internal

(venv) [krishna@bastion openshift-ansible]$ ^C

(venv) [krishna@bastion openshift-ansible]$ history

    1  sudo -i

    2  id

    3  ssh $(cat $HOME/RHEL_workers.txt) sudo yum repolist

    4  pwd

    5  ll

    6  INV_FILE=$HOME/openshift-ansible/inventory/hosts

    7  cat << EOF > $INV_FILE

[all:vars]

# SSH user, this user should allow ssh based auth without requiring a

# password. If using ssh key based auth, then the key should be managed by an

# ssh agent.

ansible_user=ec2-user

# If ansible_user is not root, ansible_become must be set to true and the

# user must be configured for passwordless sudo

ansible_become=True

###############################################################################

# Required configuration variables                                            #

###############################################################################

openshift_kubeconfig_path="~/.kube/config"

# For running RHEL worker upgrades

[new_workers]

$(<$HOME/RHEL_workers.txt)

EOF

    8  cd $HOME/openshift-ansible/

    9  virtualenv venv

   10  source venv/bin/activate

   11  pip install -r requirements.txt

   12  which ansible

   13  oc login -u admin -p 0o3nYjrL6EZ9vICa

   14  ansible-playbook -v -i inventory/hosts playbooks/scaleup.yml

   15  oc get nodes

   16  export RHEL_NODE=$(oc get nodes -l node.openshift.io/os_id=rhel -o jsonpath="{.items[0].metadata.name}")

   17  echo "export RHEL_NODE=${RHEL_NODE}" >> ~/.bashrc

   18  echo $RHEL_NODE

   19  history

(venv) [krishna@bastion openshift-ansible]$ oc new-project needs-rhel

Now using project "needs-rhel" on server "https://api.cluster-r7h7v.r7h7v.sandbox212.krishna.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

(venv) [krishna@bastion openshift-ansible]$ oc annotate namespace needs-rhel openshift.io/node-selector="node.openshift.io/os_id=rhel"

namespace/needs-rhel annotated

(venv) [krishna@bastion openshift-ansible]$ kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

deployment.apps/hello-node created

(venv) [krishna@bastion openshift-ansible]$ oc get pods -n needs-rhel --field-selector=spec.host=$RHEL_NODE -o wide

NAME                          READY   STATUS    RESTARTS   AGE   IP           NODE                                        NOMINATED NODE   READINESS GATES

hello-node-5dd7685477-vqtkc   1/1     Running   0          16s   10.129.0.6   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

(venv) [krishna@bastion openshift-ansible]$ cat playbooks/scaleup.yml

---

- name: Pre-scaleup checks

  hosts: localhost

  connection: local

  gather_facts: no

  tasks:

  - import_role:

      name: openshift_node

      tasks_from: scaleup_checks.yml

- name: install nodes

  hosts: new_workers

  roles:

  - openshift_node

(venv) [krishna@bastion openshift-ansible]$ oc get pods -A --field-selector=spec.host=$RHEL_NODE -o wide

NAMESPACE                                NAME                                  READY   STATUS    RESTARTS   AGE     IP            NODE                                        NOMINATED NODE   READINESS GATES

needs-rhel                               hello-node-5dd7685477-vqtkc           1/1     Running   0          68s     10.129.0.6    ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-cluster-csi-drivers            aws-ebs-csi-driver-node-dngbh         3/3     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-cluster-node-tuning-operator   tuned-pgqhv                           1/1     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-dns                            dns-default-5j5tq                     2/2     Running   0          6m59s   10.129.0.4    ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-dns                            node-resolver-7pb22                   1/1     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-image-registry                 node-ca-hjv9n                         1/1     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-ingress-canary                 ingress-canary-jrbrr                  1/1     Running   0          6m59s   10.129.0.5    ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-machine-config-operator        machine-config-daemon-4sglx           2/2     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-monitoring                     node-exporter-6ktb5                   2/2     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-multus                         multus-additional-cni-plugins-r4zv9   1/1     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-multus                         multus-gw2p2                          1/1     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-multus                         network-metrics-daemon-snzvp          2/2     Running   0          7m19s   10.129.0.3    ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-network-diagnostics            network-check-target-qk8bh            1/1     Running   0          7m19s   10.129.0.2    ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

openshift-sdn                            sdn-6lxjr                             2/2     Running   0          7m19s   10.0.30.244   ip-10-0-30-244.us-east-2.compute.internal   <none>           <none>

(venv) [krishna@bastion openshift-ansible]$

 Issue:-

oc login -u kubeadmin -p blah

error: x509: certificate is not valid for any names, but wanted to match oauth-openshift.apps.cluster-psk

Sol:- now we have to troubleshoot

1) To explore this environment, export KUBECONFIG using the configuration created by the installer:

export KUBECONFIG=~/.kube/config

2) Check to see whether the default ingresscontroller has a defaultCertificate configured. Pay special attention to what is defined for spec: (if that value is present):

oc get -n openshift-ingress-operator ingresscontroller/default -o json

 "spec": {

        "defaultCertificate": {

            "name": "cee-cf-110"

        }

or run cmd to get directly

oc get -n openshift-ingress-operator ingresscontroller/default -o jsonpath='{.spec.defaultCertificate.name}'

cee-cf-110

3) If the spec value is not present, you might want to check what certificate is being used. To do this, you would use oc describe as shown here. Run this command and compare your output to what's shown here:

oc describe -n openshift-ingress-operator ingresscontroller/default

Events:

  Type    Reason                     Age                  From                    Message

  ----    ------                     ----                 ----                    -------

  Normal  Admitted                   57m (x2 over 5h49m)  ingress_controller      ingresscontroller passed validation

  Normal  DeletedDefaultCertificate  57m                  certificate_controller  Deleted default wildcard certificate "router-certs-default"

  Normal  Admitted                   11m                  ingress_controller      ingresscontroller passed validation

  Normal  CreatedDefaultCertificate  11m                  certificate_controller  Created default wildcard certificate "router-certs-default"

  Normal  DeletedDefaultCertificate  10m                  certificate_controller  Deleted default wildcard certificate "router-certs-default"

  Normal  Admitted                   10m                  ingress_controller      ingresscontroller passed validation

[root@bastion ~]# oc describe -n openshift-ingress-operator ingresscontroller/default

4)

Certificates in RHOCP 4 are stored as secrets. Once you know the name of the certificate, you can use oc describe secret as shown here to get additional details. Your key focus will be the Data section of that output. Run this command and compare your output to what's shown here:

oc describe secret router-certs-default -n openshift-ingress

oc describe secret router-certs-default -n openshift-ingress

Name:         router-certs-default

Namespace:    openshift-ingress

Labels:       <none>

Annotations:  <none>

Type:  kubernetes.io/tls

Data

====

tls.crt:  2404 bytes

tls.key:  1675 bytes

Notice how the secret is stored in openshift-ingress, but the ingress controller is in the openshift-ingress-operator namespace.