1. Starting.
The best way to start all-in-one bundle is 'oc cluster up' command
But by default OpenShift uses /var directory for etcd and other data, so using custom directories is more usable (as well allows to run different versions/configs of OpenShift), something like cluster_up.sh
#!/bin/sh
oc cluster up --host-data-dir=$(pwd)/my-cluster/etcd \
--host-volumes-dir=$(pwd)/my-cluster/volumes \
--host-config-dir=$(pwd)/my-cluster/openshift.local.config \
--use-existing-config
allows to keep things permanent as well more or less manageable
2. Memory
OpenShift Origin is designed to run on top hardware with many nodes (10k) and it pre-allocates a lot of resources by default (~2Gb of RAM) and very unhappy with swapping.
To reduce memory additional editing of master-config.yaml is needed (version 1.4.1 and above), adding next values will reduce RAM use down to 400-500Mb
kubernetesMasterConfig:
apiServerArguments:
deserialization-cache-size:
- "500"
event-ttl:
- "48h"
target-ram-mb:
- "300"
please note, event-ttl is here just for convenience (to keep events for longer period, instead of default 1 hour)
The settings above were tested for few weeks with dozen of docker containers running without any issues
3. Metrics
Unfortunately OpenShift Origin replaced original Kubernetes metrics (cAdvisor) with own integration based on Hawkular and Cassandra . Even with all tuning it still uses over 1Gb of RAM and relatively unusable for local use (though it disabled by default).
As alternative it is possible to use cAdvisor (resource monitoring for docker) , when running via
docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=false --name=cadvisor google/cadvisor:latest
it provides the same data, though it targets docker containers and doesn't provide easy way to sort containers (by example to search one that is using the most CPU or RAM)
4. Network
OpenShift has inbuilt DNS server (SkyDNS) which can be used locally as well, just add it to /etc/resolv.conf as first server (nameserver MASTER_IP_OF_OPENSHIFT) . Then you can use something like mysql.myproject.svc.cluster.local instead of raw IPs
Please note, '.local' domain might not work well with dns interceptors , like avahi-daemon (by default installed on desktop linuxes), you probably should unlink .local from it or use other domain , i.e. edit /etc/avahi/avahi-daemon.conf and add
[server]
domain-name=.alocal
Additionally, you probably should ensure the sysctl net.ipv4.ip_forward is set to 1, without forwarding OpenShift will fail with weird errors (it uses iptables to communicate with docker containers)
5. File system
To have the best performance you probably should enable overlay2 in docker as well use hostmount friendly templates (i.e. with ability to use local fs instead of network based - glusterfs/ceph etc), like mysql/percona one.
also, OpenShift internally uses repeated mounts for providing fresh configs for containers. i.e. something like 'volumes/pods/0d657a7b-eb63-11e6-8cb6-74d02b8fa488/volumes/kubernetes.io~secret/router-token-xpf7m' will be mounted every few minutes.
while this isn't a problem on non-gui installations of Linux it made crazy automounters and other gui tools that monitors mount points, like gvfs-udisks2-volume-monitor
Though OpenShift uses this technique for reason and there no way to disable it, simple cron will remove mount points (while it may result in restart of some container)
10 * * * * umount $(mount | grep my-cluster | grep secret | cut -d' ' -f3)
6.haproxy issues.
By default OpenShift Origin haproxy (used as load balancer ) binds to all interfaces/IPs on ports 80/443. If you have web server running locally it's convenient in many cases.
While fixing is still in progress you may create custom haproxy.template and replace bind :port with bind ip:port rules
