Container Service: Developing an Application - General Information

What about time zones?

Docker containers may default to UTC so the time seen in a container may be surprising if there is a need to export a time (say in a zip archive) or compare file times on mounted volumes.  The local container time zone may be changed by specifying an explicit environment variable. E.g. You can set east coast time from the Docker command line by adding " -e TZ=America/New_YORK ". Alpine based images will probably need more work than that. Please link to examples if you find some. In OpenShift environment variables can be set from Environment tab or in the deployment configuration or template json/yaml.

Java - A Maven Build Needs A Jar Not In A Public Repository

Build and install that dependency from source right in the Dockerfile. It just adds a couple of lines to file. If this is a frequent problem other approaches can be discussed. This is likely not to work for a S2I build that bypasses the Dockerfile. Other methods such as linked builds and a UMich internal Maven repository may be possible.

Examining A Persistent Volume 

OpenShift persistent volumes are not visible outside of the project they are in.That makes it difficult to examine or extract files directly. If you have persistent pods then it's possible to ssh into the pod and examine the volume from the command line or to use the oc rsync command. If you have a job that only runs periodically (e.g. a CronJob) then you can create a temporary application in the project and attach the storage to that. E.g. use the bash image from Dockerhub. Make sure to override the image arguments so that it runs an enduring process. Otherwise OpenShift will continuously kill and restart it. Adding ["sleep","60000"] as image args works.

Running Scheduled Jobs

For an example of running cron in a container, go here. However this may be difficult if you can't run as root.

OpenShift has an explicit concept of Jobs and CronJobs to run scheduled batch processing. However, as of OS 3.5, the cron job can not be created in the UI, will not show up as an application in the UI, and will not automatically clean up the Jobs and pods created during executions. 

Viewing Logs In OpenShift

If viewing the logs on OpenShift via kibana you can get to the logs without going through the application by using the "Monitoring" tab and looking at "Events" and "View details".  You can then click on a specific pod and the "View Archive". This will allow looking at pods created from a cron job. There may be an easier way to do this.

The logs can then be searched for text with a query similar to that below. This will search logs over a namespace, not just a specific log.

kubernetes.namespace_name:"spanish-placement-exam-dev" AND "users added"

Log Search Syntax

A elasticsearch (Kibana) syntax cheat sheet is at: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax

Below is a sample query to find log lines that contain a entry and count for "users added:" while ignoring those where the count is zero. There other forms of queries that seem like they ought to work but I could not get them to parse.

"users added:" AND NOT "added: 0"

Troubles

In a crash loop deploying a pod and need it to stop. The page for this deployment attempt has a "cancel" button right next to "Status:". Or you can scale the number of pods to zero. Or one of the actions on the deployment config page is to "pause rollouts".

Pod Is Failing

At the bottom of the deployment page there is a list of the pod(s) being created. On the pod page, under Status ... Restart Count: is a button to "Debug in Terminal". This starts the container and temporarily overrides the command to give you a window with a shell prompt. Great for looking around to see if things are setup as expected.

It builds fine on my laptop but fails to build in OpenShift

Container Service customers occasionally find that the underlying distribution of the base image being used by their application is different when building on their laptop than when they build in OpenShift. The Container Service provides base images from both Docker Hub (typically debian-based) and Red Hat in the openshift namespace within the integrated registry. Unless the tag has a suffix at the end of its name, you can assume that tag is debian-based. Red Hat images have a rhel suffix affixed to the end of the tag name. For instance:

python:3.6-rhel

Shibboleth

Several Container Service applications have utilized shibboleth as its authentication provider. This write-up documents how a python developer accomplished this.

Last Updated: 
Wednesday, June 3, 2020