Recently I was building a multi-node OpenShift 3.4 environment in Softlayer - AKA - Bluemix Infrastructure. I ran into an issue with the installer (running via Ansible) that was not very easy to troubleshoot and get to the the root cause. This is a ~quick~ blog to ideally save someone else in a similar position a lot of time, google searching and chasing their tail.
This issue could easily come up in on-prem installations depending on the vm or server configuration. A multi-master setup utilizes etcd and this is where the core issue came up, basically during the install etcd tries to communicate with the other etcd nodes via the API and it times out and fails the install (connection refused). Jumping to the root cause was the fact that my vm’s were configured with 2 interfaces as follows:
**1 for internet access (inbound & out) - 169.55.187.**/28**
**1 for internal access - 10.166.21.128/26**
Most Cloud providers I have used don’t use this type of configuration, they simple give you an internal vlan and have a NAT service to allow internet connectivity inbound and outbound. Int his case we need both as we need to get packages from the internet also and allow access inbound on a few ports.
If you end up with a similar configuration and run into the issue, you will likely see something like this in your ansible log upon failure:
*Sorry for this length of this blog up-front but I am hoping this helps someone who is googleing error messages they find - like I was…
Now next step will be dig deeper by getting on one of the masters and looking at journalctl -xe you will see some additional information (maybe something like this):
connection refused was the tip…
So some more google searching will prob lead you to one of these bugzilla’s but they don’t provide a solution or a work around - remember all you want to do in have your OCP install go well. (I will add a link to this blog also)
So how https://bugzilla.redhat.com/show_bug.cgi?id=1375111 did I troubleshoot this? As you can see the error logs is showing internal 10. ip’s. When you are installing OpenShift you have this nice handy facts playbook, it basically gathers all the facts about your setup. This is what Ansible will use when actually doing the install so the information here is very important.
Here it the output below that directed me to the root cause - See the 169.55.191. IP ADDRESS?** Well this is the external (public ip), I don’t want it using that IP to communicate across etcd nodes. It will fail as I don’t open all those ports on the internet side of things. This odd part is that I use an internal BIND server for all my communication and the FQDN names use all the internal 10. blocks. My entire Ansible host file is based on FQDN names. In the end I don’t want anything to do with the public address - but how do I get around this? I could not find a solution even with all my google searching. Lots of attempts failed…
Small Segway… One great thing about having an amazing team at Arctiq is we can collaborate on things that stump us. So I send a Slack message to Shea. Quickly we figure out we can’t solve this over chat, so I call him… explain the issues, show him what I am seeing… and in the end we found a ‘what I call’ an ‘undocumented’ work around.
**Yah collaboration / team!!! **
So in the end we used a working around in the Ansible host file, if you are installing OpenShift you will know this file well. It generally lives in etc/ansible/hosts (we recommend not running it from here but that is another blog).
SO HERE IS THE WORKAROUND -> Its a super simple workaround but again it took me a long time to figure it out and the errors can send you chasing other “non” issues… Drum roll …
Just add openshift_ip=<ip address> to all references to hostnames in your host file. When Ansible gathers facts it will force it to use the IP you define rather that it grabbing the public IP you want nothing to do with… After I made this change, I ran the clean-up / uninstall scripts and ran my install again and all went fine.
Lesson learned and I hope it helps someone a lot of time, effort and stress some day.
Now remember to “Pay it Forward” and write a blog when you solve a weird technical issue.
Kyle has been at the forefront of open source and transformative solutions throughout his IT career spanning a number of large North American organiztions. From designing HPC solutions, to archecting complex Container solutions, Kyle provides deep consulting on today’s greatest IT challanges.
As a partner at Arctiq, Kyle focusses on the following core competencies:
Architecture and design leadership
Open source solution focused on modern platforms and disruptive technologies
DevOps consulting and culture conditioning
Virtualization, containers, and microservices
Strategic business planning and process execution
Mike’s business and IT background with over 20 years experience led to the formation of Arctiq’s BPM focused business consulting practice. Business case development, technical process improvement, and consulting on DevOps culture are the foundations of this practice.
As a partner at Arctiq, Mike is prime on:
Developing and delivering Arctiq’s business-focused consulting services
Managing Arctiq’s business operations, ensuring we practice what we preach
Automating and integrating back end processes using a DevOps approach to operations management and visibility
Building an open ecosystem of technology and delivery partners
Delivery Project Management services for complex Arctiq engagements
With design and solution architecture being a primary focus of his career, Aly has spent a great deal of time problem solving complex issues, creating designs and implementing infrastructure solutions. Having worked with multinational companies, he has spent a great deal of time working on globe spanning projects, including classified and bleeding edge technologies. Having a deep background working with security focused clients Aly has a strong knowledge base with regards to system compliance, audits and security hardening. He focuses on core technologies that run deep into the OS including kernel tuning, performance driven solutions and highly secure systems as well as a passion for development and writing code to help automate and streamline tasks.
Outside of Arctiq, Aly can be found spending time with this family, developing code, learning the next great technology and spending time outdoors.
Aly’s focus at Arctiq is in the following key areas:
Container and Orchestration - Docker, Kubernetes and Openshift
DevOps Consulting - Team and culture improvements, development and R&D
With the bulk of his career focused on infrastructure solution architecture and implementation, Shea now brings that knowledge into the development space. Working with software development and infrastructure teams, Shea focuses on improving technology processes through automation and collaboration tooling and culture changes. Being a technologist, Shea is happiest when found exploring emerging technologies and applying them to real-world customer challenges.
Outside of the office Shea can be found out with his friends enjoying a Jays game, being active, or hunting down a new craft brew.
Along with an eye on security processes and partnerships, Shea’s focus at Arctiq is in the following key areas:
Automation & self-service of infrastructure systems
Container platform deployment and training
CI/CD pipeline development with integrated security
DevOps culture improvements through process optimization and collaboration tooling
Tim has spent the bulk of his career in the solution architecture space, problem solving and designing solutions to meet very specific needs. Having worked across many vertical and technology areas, Tim applies this expertise to his work with Arctiq’s clients.
Along with marketing and website responsibilities, Tim’s focus at Arctiq is in the following key areas:
Automation - Ansible, Puppet
Foundations - Red Hat Enterprise Linux, VMware, Cloud (AWS, GCE)
DevOps Consulting - Team and culture improvements, workflow and process improvements
Daniyal has served his time at all levels of development, working his way up to Arctiq. He has a varied knowledge of programming languages and a wealth of problem solving experience. Daniyal is intrigued by Big Data and Mobile Development but Daniyal’s focus is now on his newfound love - DevOps.
Daniyal enjoys helping customers by automating processes with Ansible, building CI/CD Jenkins pipelines on OpenShift and establishing infrastructure management with Satellite.
When Daniyal is not at Arctiq, he enjoys soccer, travelling, and cars (in that order). Daniyal also holds a Honours degree in Computer Science from the Lassonde School of Engineering in Toronto.
Throughout his early career, Jacob has spent time in development and system administration where he gained knowledge in production programming and basic network architecture. Transitioning to DevOps, Jacob is taking his skills from his past work and school experience and adapting quickly.
Jacob has taken a focus on Satellite for infrastructure management, Ansible for IT automation, and Red Hat Enterprise Linux for server foundation.
Outside of Arctiq, Jacob spends time learning new technologies and frameworks to continue making his IT knowledge stronger. One of his favourite things to do is to build projects with his multiple Raspberry Pis.
Through nearly all sizes of enterprise, Aaren has experience from end-user support, application deployment, through to system administration. Combined over 6 years, he is versed in enterprise technology architectures, data management, open-source methodology, systems compliance and general infrastructure technology. While neither for or against ‘DevOps’, he believes in the benefits that such methodologies provide, should any enterprise be willing to embrace change.
Aside from Arctiq, Aaren enjoys time with his wife and two daughters, deploying tech in his home-lab, admiring electric cars, and enjoying the outdoors.
Aaren tends to focus in the following key areas:
Essentials - RHEL, RHV, GlusterFS, openZFS
Automation & Languages - Ansible, Bash, Python
Infra & Storage - practices and training, data integrity
DevOps Consulting - team and culture improvements, legacy modernization