How to Install Oracle OpenStack 2.1.1 Step by Step(I)

Oracle Linux OpenStack借助Oracle Linux与Oracle VM软硬件的兼容优势，集成Oracle Clusterware和MySQL企业版来确保OpenStack服务的高可用性，打造了一个企业级的开放解决方案。使用了Oracle实施和支持全球最复杂企业负载的经验，其设计侧重于OpenStack的部署、升级、稳定性和可支持性。保持了OpenStack的灵活性，还允许客户部署不同的配置以及与不同的软件和硬件供应商集成。

Oracle OpenStack部署包括计算节点和控制节点：

控制节点安装在 Oracle Linux 7.1 或更高版本上

对于计算节点，用户可以使用安装在 Oracle Linux 7.1 或更高版本上的 KVM

Oracle OpenStack软件包包括：

openstack-kolla-preinstall：该软件包将安装在部署中包括的主机（即控制器节点、计算节点、数据库节点、网络节点、存储节点）上。这些节点被称为目标节点

openstack-kollacli：该软件包将安装在控制器节点上，需要时将安装在单独的 Oracle Linux 主机上。该软件包包括面向 Oracle Linux 的 Oracle OpenStack 工具包，用于将 Docker 容器部署到目标节点。安装了该工具包的节点被称为主节点

openstack-kolla-utils：该软件包包含用于访问 Docker容器的OpenStack CLI

注意：Oracle Linux OpenStack版部署所需的Oracle Linux软件包可从Oracle Public Yum(http://public-yum.oracle.com) 和Oracle Unbreakable Linux Network (ULN) (https://linux.oracle.com) 获取

本文通过Oracle Virtualbox建了5台虚拟机，配置情况如下。一台主机用于网络，两台主机用于控制节点(其中ctrl1.ohsdba.cn配置了registry)，两台为计算节点。Oracle默认安装MySQL数据库，本文配置了两个数据库节点，会以MySQL HA的方式安装。

注意：如果资源有限，所有的服务也可以安装在一台服务器。当然我们还可以使用Oracle VM Server。

在读本文之前，你需要了解下面的知识

A.了解Docker，安装配置过Docker和registry

B.了解过Openstack

本次安装的配置信息表

节点类型	IP地址	服务器名称	Oracle Linux 7.2安装选项
Network	172.16.1.4	net1.ohsdba.cn	Minimal + UEK4
Master/Registry/Controller/Storage/Database Node1	172.16.1.10	ctrl1.ohsdba.cn	Minimal + UEK4
Controller/Storage/ Database Node2	172.16.1.11	ctrl2.oracle.com	Minimal + UEK4
Compute Node1	172.16.1.16	compute1.oracle.com	Virtualization + UEK4
Compute Node2	172.16.1.17	compute2.oracle.com	Virtualization + UEK4

Note：配置了3块网卡：第一块为NAT类型，连接Internet，用来更新和安装Kernel、Docker。第二块为172网段，用于Openstack业务。第三块为'Internal Network'类型。在网络节点(net1.ohsdba.cn)还配置了第四块网卡，用于浮动地址，为在计算节点上创建的虚拟服务器使用。

主机角色配置表

+--------------------+------------------------------------+
| Host               | Groups                             |
+--------------------+------------------------------------+
| compute1.ohsdba.cn | ['compute']                        |
| compute2.ohsdba.cn | ['compute']                        |
| ctrl1.ohsdba.cn    | ['control', 'storage', 'database'] |
| ctrl2.ohsdba.cn    | ['control', 'storage', 'database'] |
| net1.ohsdba.cn     | ['network']                        |
+--------------------+------------------------------------+
[ohsdba@ctrl1 ~]$

注意：安装前，必须满足一下条件

         Oracle Enterprise Linux 7.2 64bit

         SELinux已被禁用，已关闭和禁用防火墙
         配置DNS或使用/etc/hosts文件

         使用self-signed CA certificate

OpenStack节点介绍

There are a number of node types used in OpenStack. Nodes are a physical host computer, with an operating system installed, with Oracle Linux using KVM (Kernel-based Virtual Machine), or Oracle VM Server. The main node types we discuss in this guide are:

A controller node is a system running Oracle Linux, and is where most of the OpenStack services are installed. The term controller node is used to describe nodes that do not run virtual machine instances. The controller nodes may have all the non-compute services or only some of them. A controller node may also include the Oracle OpenStack for Oracle Linux CLI (kollacli), which is used to perform the deployment of OpenStack services to other nodes.

A compute node is a system running Oracle Linux using KVM, or Oracle VM Server. A compute node runs the bare minimum of services to manage virtual machine instances.

A database node is a system running Oracle Linux, and the services required to manage databases for images and instances.

A network node is a system running Oracle Linux, and runs the neutron network worker daemon. The neutron worker daemon provides services such as providing an IP address to a booting Nova instance.

A storage node is a system running Oracle Linux and the services required to manage storage for images and instances.
Some storage is not directly managed by the OpenStack services, but is instead managed by the storage appliance. On the storage node, Cinder communicates with the storage appliance's API, and it is the storage appliance that performs the storage management. For example, when using the Oracle ZFS Storage Appliance, the Cinder driver on the storage node communicates with the Oracle ZFS Storage Appliance NFS driver, and it is the ZFS driver which performs the storage management.

A master node is a system running Oracle Linux and kollacli, used to deploy the OpenStack services to the nodes. A master node is not an OpenStack node, although kollacli may be installed on a controller node. 

OpenStack安装最低要求

http://docs.oracle.com/cd/E73172_01/E73174/html/os-ol-system-requirements.html
OpenStack Node Minimum System Requirements

In addition to the OpenStack nodes, Oracle OpenStack for Oracle Linux requires a node to host a Docker registry and a node (known as a master node) from which you deploy OpenStack services using the kollacli command. Typically these are hosted on a controller node, but you can host these on separate nodes if you prefer. The following are the minimum requirements for separate nodes:

配置hosts文件(在所有节点上执行)

[root@ctrl1 ~]# cat /etc/hosts
127.0.0.1  .ohsdba.cnhost.ohsdba.cnhost.ohsdba.cndomain.ohsdba.cnhost4.ohsdba.cnhost4.ohsdba.cndomain4
172.16.1.1      rhan.ohsdba.cn rhan
172.16.1.4      net1.ohsdba.cn net1
172.16.1.8      registry.ohsdba.cn registry
172.16.1.9      ctrl-vip.ohsdba.cn ctrl-vip
172.16.1.10     ctrl1.ohsdba.cn ctrl1
172.16.1.11     ctrl2.ohsdba.cn ctrl2
172.16.1.16     compute1.ohsdba.cn compute1
172.16.1.17     compute2.ohsdba.cn compute2

10.10.10.4      net1-tun.ohsdba.cn net1-tun
10.10.10.9      ctrl1-tun.ohsdba.cn ctrl1-tun
10.10.10.10     ctrl2-tun.ohsdba.cn ctrl2-tun
10.10.10.16     compute1-tun.ohsdba.cn compute1-tun
10.10.10.17     compute2-tun.ohsdba.cn compute2-tun

升级Kernel并重启(在所有节点上都安装)

[root@ctrl1 ~]# uname -a
Linux ctrl1.ohsdba.cn 4.1.12-61.1.14.el7uek.x86_64 #2 SMP Wed Oct 12 17:37:07 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@ctrl1 ~]#
# yum update -y
[root@ctrl1 ~]# uname -a
Linux ctrl1.ohsdba.cn 4.1.12-32.el7uek.x86_64 #2 SMP Thu Dec 17 19:31:20 PST 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@ctrl1 ~]#
[root@ctrl1 ~]#init 6

安装docker(在所有节点上都安装)

yum install docker-engine
[root@ctrl1 ~]# yum install docker-engine -y
Loaded plugins: langpacks
Resolving Dependencies
--> Running transaction check
---> Package docker-engine.x86_64 0:1.12.0-1.0.2.el7 will be installed
--> Processing Dependency: docker-engine-selinux >= 1.12.0-1.0.2.el7 for package: docker-engine-1.12.0-1.0.2.el7.x86_64
--> Processing Dependency: libseccomp.so.2()(64bit) for package: docker-engine-1.12.0-1.0.2.el7.x86_64
--> Processing Dependency: libltdl.so.7()(64bit) for package: docker-engine-1.12.0-1.0.2.el7.x86_64
--> Running transaction check
---> Package docker-engine-selinux.noarch 0:1.12.0-1.0.2.el7 will be installed
---> Package libseccomp.x86_64 0:2.2.1-1.el7 will be installed
---> Package libtool-ltdl.x86_64 0:2.4.2-21.el7_2 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================
 Package                      Arch          Version                  Repository         Size
=============================================================================================
Installing:
 docker-engine                x86_64        1.12.0-1.0.2.el7         ol7_addons         19 M
Installing for dependencies:
 docker-engine-selinux        noarch        1.12.0-1.0.2.el7         ol7_addons         29 k
 libseccomp                   x86_64        2.2.1-1.el7              ol7_latest         49 k
 libtool-ltdl                 x86_64        2.4.2-21.el7_2           ol7_latest         48 k

Transaction Summary
=============================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 19 M
Installed size: 79 M
Downloading packages:
(1/4): docker-engine-selinux-1.12.0-1.0.2.el7.noarch.rpm  |  29 kB  00:00:00     
(2/4): libseccomp-2.2.1-1.el7.x86_64.rpm                  |  49 kB  00:00:00     
(3/4): libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm             |  48 kB  00:00:00     
(4/4): docker-engine-1.12.0-1.0.2.el7.x86_64.rpm          |  19 MB  00:00:41     
---------------------------------------------------------------------------------------------
Total                                                        469 kB/s |  19 MB  00:00:41     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libseccomp-2.2.1-1.el7.x86_64                    1/4
  Installing : docker-engine-selinux-1.12.0-1.0.2.el7.noarch    2/4
setsebool:  SELinux is disabled.
  Installing : libtool-ltdl-2.4.2-21.el7_2.x86_64               3/4
  Installing : docker-engine-1.12.0-1.0.2.el7.x86_64            4/4
  Verifying  : libtool-ltdl-2.4.2-21.el7_2.x86_64               1/4
  Verifying  : docker-engine-1.12.0-1.0.2.el7.x86_64            2/4
  Verifying  : docker-engine-selinux-1.12.0-1.0.2.el7.noarch    3/4
  Verifying  : libseccomp-2.2.1-1.el7.x86_64                    4/4

Installed:
  docker-engine.x86_64 0:1.12.0-1.0.2.el7                                                    
Dependency Installed:
  docker-engine-selinux.noarch 0:1.12.0-1.0.2.el7       libseccomp.x86_64 0:2.2.1-1.el7      
  libtool-ltdl.x86_64 0:2.4.2-21.el7_2                 
Complete!
[root@ctrl1 ~]#

更改docker配置文件内容/etc/sysconfig/docker (在所有节点上都安装)

DOCKER_CERT_PATH=/etc/docker
OPTIONS='--storage-driver btrfs --selinux-enabled=false'
INSECURE_REGISTRY='--insecure-registry=o3l1.oracle.com:5443'
setsebool -P docker_transition_unconfined
GOTRACEBACK=crash

安装openstack-kolla-preinstall包(在所有节点上都安装)

# yum install openstack-kolla-preinstall -y

安装部署包openstack-kollacli(在Master节点上安装，本文中是ctrl1和ctrl2)

# yum install openstack-kollacli -y

重启Docker(在所有节点上都安装)

[root@ctrl1 sysconfig]# systemctl daemon-reload
[root@ctrl1 sysconfig]# systemctl restart docker
[root@ctrl1 sysconfig]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@ctrl1 sysconfig]# 

建立self-signed CA certificate(在Master节点上建立，本文中是ctrl1)

# mkdir -p /var/lib/registry/conf.d
# cd /var/lib/registry/conf.d
# openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt
# chmod 600 domain.key
[root@ctrl1 ~]# mkdir -p /var/lib/registry/conf.d
[root@ctrl1 ~]# cd /var/lib/registry/conf.d
[root@ctrl1 conf.d]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt
Generating a 4096 bit RSA private key
..................................................................................................................++
..............++
writing new private key to 'domain.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:Massachusetts
Locality Name (eg, city) [Default City]:Boston
Organization Name (eg, company) [Default Company Ltd]:Oracle Com
Organizational Unit Name (eg, section) []:DevOps
Common Name (eg, your name or your server's hostname) []:ctrl1.ohsdba.cn
Email Address []:ohsdba@qq.com
[root@ctrl1 conf.d]# chmod 600 domain.key
[root@ctrl1 conf.d]# 

复制认证文件到其他节点(在Master节点上执行，本文是ctrl1)

[root@ctrl1 conf.d]# mkdir -p /etc/docker/certs.d/ctrl1.ohsdba.cn:5443
[root@ctrl1 conf.d]# cp /var/lib/registry/conf.d/domain.crt /etc/docker/certs.d/ctrl1.ohsdba.cn:5443/ca.crt
[root@ctrl1 conf.d]#
root@ctrl1 conf.d]# ssh ctrl2 mkdir -p /etc/docker/certs.d/ctrl1.ohsdba.cn:5443
root@ctrl2's password:
[root@ctrl1 conf.d]# ssh net1 mkdir -p /etc/docker/certs.d/ctrl1.ohsdba.cn:5443
root@net1's password:
[root@ctrl1 conf.d]# ssh compute1 mkdir -p /etc/docker/certs.d/ctrl1.ohsdba.cn:5443
root@compute1's password:
[root@ctrl1 conf.d]# ssh compute2 mkdir -p /etc/docker/certs.d/ctrl1.ohsdba.cn:5443
root@compute2's password:
[root@ctrl1 conf.d]# cd /etc/docker/certs.d/ctrl1.ohsdba.cn:5443
[root@ctrl1 ctrl1.ohsdba.cn:5443]# scp ca.crt ctrl2:`pwd`
root@ctrl2's password:
ca.crt                                                     100% 2139     2.1KB/s   00:00    
[root@ctrl1 ctrl1.ohsdba.cn:5443]# scp ca.crt net1:`pwd`
root@net1's password:
ca.crt                                                     100% 2139     2.1KB/s   00:00    
[root@ctrl1 ctrl1.ohsdba.cn:5443]# scp ca.crt compute1:`pwd`
root@compute1's password:
ca.crt                                                     100% 2139     2.1KB/s   00:00    
[root@ctrl1 ctrl1.ohsdba.cn:5443]# scp ca.crt compute2:`pwd`
root@compute2's password:
ca.crt                                                     100% 2139     2.1KB/s   00:00    
[root@ctrl1 ctrl1.ohsdba.cn:5443]#
[root@ctrl1 ctrl1.ohsdba.cn:5443]# cd
[root@ctrl1 ~]#

创建本地Docker registry(在Master节点上执行)

注意：Docker使用的文件系统必须为btrfs

[root@ctrl1 ~]# mount|grep btrfs
/dev/mapper/vg00-var_lib_docker on /var/lib/docker type btrfs (rw,relatime,space_cache)
/dev/sdb on /var/lib/registry type btrfs (rw,relatime,space_cache)
/dev/mapper/vg00-var_lib_docker on /var/lib/docker/btrfs type btrfs (rw,relatime,space_cache)
[root@ctrl1 ~]# 
# docker run -d -p 5443:5000 --name registry --restart=always -v /var/lib/registry:/registry_data -e REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry_data -e REGISTRY_HTTP_TLS_KEY=/registry_data/conf.d/domain.key -e REGISTRY_HTTP_TLS_CERTIFICATE=/registry_data/conf.d/domain.crt registry:2

[root@ctrl1 ~]# docker run -d -p 5443:5000 --name registry --restart=always -v /var/lib/registry:/registry_data -e REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry_data -e REGISTRY_HTTP_TLS_KEY=/registry_data/conf.d/domain.key -e REGISTRY_HTTP_TLS_CERTIFICATE=/registry_data/conf.d/domain.crt registry:2
Unable to find image 'registry:2' locally
2: Pulling from library/registry
c0cb142e4345: Pull complete
a5002dfce871: Pull complete
df53ce740974: Pull complete
9ce080a7bfae: Pull complete
517dc3530502: Pull complete
Digest: sha256:1cfcd718fd8a49fec9ef16496940b962e30e3927012e851f99905db55f1f4199
Status: Downloaded newer image for registry:2
ab9c177d83e3fa068df111f645baff76059dd2524b31e6154bb0fb0183053716
[root@ctrl1 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
ab9c177d83e3        registry:2          "/entrypoint.sh /etc/"   6 seconds ago       Up 4 seconds        0.0.0.0:5443->5000/tcp   registry
[root@ctrl1 ~]#

创建用户并修改权限(主节点ctrl1上执行，之后部署要用到)

[root@ctrl1 sysconfig]# useradd ohsdba
[root@ctrl1 sysconfig]# usermod -aG kolla,docker ohsdba
[root@ctrl1 sysconfig]# id ohsdba
uid=1001(ohsdba) gid=1001(ohsdba) groups=1001(ohsdba),988(docker),600(kolla)
[root@ctrl1 sysconfig]# 

Reference

http://docs.oracle.com/cd/E73172_01/E73174/html/os-ol-system-requirements.html

http://docs.oracle.com/cd/E73172_01/E73174/html/setup-example.html

http://www.oracle.com/technetwork/server-storage/openstack/linux/downloads/index.html

http://www.oracle.com/technetwork/cn/server-storage/openstack/linux/downloads/index.html

http://www.oracle.com/technetwork/community/developer-vm/openstack-linux-2931133.html(2.0.2)

https://blogs.oracle.com/openstack/

https://wiki.openstack.org/wiki/ReleaseNotes/Kilo