Recently I was building a multi-node OpenShift 3.4 environment in Softlayer - AKA - Bluemix Infrastructure. I ran into an issue with the installer (running via Ansible) that was not very easy to troubleshoot and get to the the root cause. This is a ~quick~ blog to ideally save someone else in a similar position a lot of time, google searching and chasing their tail.
This issue could easily come up in on-prem installations depending on the vm or server configuration. A multi-master setup utilizes etcd and this is where the core issue came up, basically during the install etcd tries to communicate with the other etcd nodes via the API and it times out and fails the install (connection refused). Jumping to the root cause was the fact that my vm’s were configured with 2 interfaces as follows:
**1 for internet access (inbound & out) - 169.55.187.**/28**
**1 for internal access - 10.166.21.128/26**
Most Cloud providers I have used don’t use this type of configuration, they simple give you an internal vlan and have a NAT service to allow internet connectivity inbound and outbound. Int his case we need both as we need to get packages from the internet also and allow access inbound on a few ports.
If you end up with a similar configuration and run into the issue, you will likely see something like this in your ansible log upon failure:
*Sorry for this length of this blog up-front but I am hoping this helps someone who is googleing error messages they find - like I was…
Now next step will be dig deeper by getting on one of the masters and looking at journalctl -xe you will see some additional information (maybe something like this):
connection refused was the tip…
So some more google searching will prob lead you to one of these bugzilla’s but they don’t provide a solution or a work around - remember all you want to do in have your OCP install go well. (I will add a link to this blog also)
So how https://bugzilla.redhat.com/show_bug.cgi?id=1375111 did I troubleshoot this? As you can see the error logs is showing internal 10. ip’s. When you are installing OpenShift you have this nice handy facts playbook, it basically gathers all the facts about your setup. This is what Ansible will use when actually doing the install so the information here is very important.
Here it the output below that directed me to the root cause - See the 169.55.191. IP ADDRESS?** Well this is the external (public ip), I don’t want it using that IP to communicate across etcd nodes. It will fail as I don’t open all those ports on the internet side of things. This odd part is that I use an internal BIND server for all my communication and the FQDN names use all the internal 10. blocks. My entire Ansible host file is based on FQDN names. In the end I don’t want anything to do with the public address - but how do I get around this? I could not find a solution even with all my google searching. Lots of attempts failed…
Small Segway… One great thing about having an amazing team at Arctiq is we can collaborate on things that stump us. So I send a Slack message to Shea. Quickly we figure out we can’t solve this over chat, so I call him… explain the issues, show him what I am seeing… and in the end we found a ‘what I call’ an ‘undocumented’ work around.
**Yah collaboration / team!!! **
So in the end we used a working around in the Ansible host file, if you are installing OpenShift you will know this file well. It generally lives in etc/ansible/hosts (we recommend not running it from here but that is another blog).
SO HERE IS THE WORKAROUND -> Its a super simple workaround but again it took me a long time to figure it out and the errors can send you chasing other “non” issues… Drum roll …
Just add openshift_ip=<ip address> to all references to hostnames in your host file. When Ansible gathers facts it will force it to use the IP you define rather that it grabbing the public IP you want nothing to do with… After I made this change, I ran the clean-up / uninstall scripts and ran my install again and all went fine.
Lesson learned and I hope it helps someone a lot of time, effort and stress some day.
Now remember to “Pay it Forward” and write a blog when you solve a weird technical issue.
Kyle has been at the forefront of open source and transformative solutions throughout his IT career spanning a number of large North American organiztions. From designing HPC solutions, to archecting complex Container solutions, Kyle provides deep consulting on today’s greatest IT challanges.
As a partner at Arctiq, Kyle focusses on the following core competencies:
Architecture and design leadership
Open source solution focused on modern platforms and disruptive technologies
DevOps consulting and culture conditioning
Virtualization, containers, and microservices
Strategic business planning and process execution
Mike’s business and IT background with over 20 years experience led to the formation of Arctiq’s BPM focused business consulting practice. Business case development, technical process improvement, and consulting on DevOps culture are the foundations of this practice.
As a partner at Arctiq, Mike is prime on:
Developing and delivering Arctiq’s business-focused consulting services
Managing Arctiq’s business operations, ensuring we practice what we preach
Automating and integrating back end processes using a DevOps approach to operations management and visibility
Building an open ecosystem of technology and delivery partners
Delivery Project Management services for complex Arctiq engagements
With design and solution architecture being a primary focus of his career, Aly has spent a great deal of time problem solving complex issues, creating designs and implementing infrastructure solutions. Having worked with multinational companies, he has spent a great deal of time working on globe spanning projects, including classified and bleeding edge technologies. Having a deep background working with security focused clients Aly has a strong knowledge base with regards to system compliance, audits and security hardening. He focuses on core technologies that run deep into the OS including kernel tuning, performance driven solutions and highly secure systems as well as a passion for development and writing code to help automate and streamline tasks.
Outside of Arctiq, Aly can be found spending time with this family, developing code, learning the next great technology and spending time outdoors.
Aly’s focus at Arctiq is in the following key areas:
Container and Orchestration - Docker, Kubernetes and Openshift
DevOps Consulting - Team and culture improvements, development and R&D
With the bulk of his career focused on infrastructure solution architecture and implementation, Shea now brings that knowledge into the development space. Working with software development and infrastructure teams, Shea focuses on improving technology processes through automation and collaboration tooling and culture changes. Being a technologist, Shea is happiest when found exploring emerging technologies and applying them to real-world customer challenges.
Outside of the office Shea can be found out with his friends enjoying a Jays game, being active, or hunting down a new craft brew.
Along with an eye on security processes and partnerships, Shea’s focus at Arctiq is in the following key areas:
Automation & self-service of infrastructure systems
Container platform deployment and training
CI/CD pipeline development with integrated security
DevOps culture improvements through process optimization and collaboration tooling
Tim has spent the bulk of his career in the solution architecture space, problem solving and designing solutions to meet very specific needs. Having worked across many vertical and technology areas, Tim applies this expertise to his work with Arctiq’s clients.
Along with marketing and website responsibilities, Tim’s focus at Arctiq is in the following key areas:
Automation - Ansible, Puppet
Foundations - Red Hat Enterprise Linux, VMware, Cloud (AWS, GCE)
DevOps Consulting - Team and culture improvements, workflow and process improvements
Daniyal has served his time at all levels of development, working his way up to Arctiq. He has a varied knowledge of programming languages and a wealth of problem solving experience. Daniyal is intrigued by Big Data and Mobile Development but Daniyal’s focus is now on his newfound love - DevOps.
Daniyal enjoys helping customers by automating processes with Ansible, building CI/CD Jenkins pipelines on OpenShift and establishing infrastructure management with Satellite.
When Daniyal is not at Arctiq, he enjoys soccer, travelling, and cars (in that order). Daniyal also holds a Honours degree in Computer Science from the Lassonde School of Engineering in Toronto.
Throughout his early career, Jacob has spent time in development and system administration where he gained knowledge in production programming and basic network architecture. Transitioning to DevOps, Jacob is taking his skills from his past work and school experience and adapting quickly.
Jacob has taken a focus on Satellite for infrastructure management, Ansible for IT automation, and Red Hat Enterprise Linux for server foundation.
Outside of Arctiq, Jacob spends time learning new technologies and frameworks to continue making his IT knowledge stronger. One of his favourite things to do is to build projects with his multiple Raspberry Pis.
Through nearly all sizes of enterprise, Aaren has experience from end-user support, application deployment, through to system administration. Combined over 6 years, he is versed in enterprise technology architectures, data management, open-source methodology, systems compliance and general infrastructure technology. While neither for or against ‘DevOps’, he believes in the benefits that such methodologies provide, should any enterprise be willing to embrace change.
Aside from Arctiq, Aaren enjoys time with his wife and two daughters, deploying tech in his home-lab, admiring electric cars, and enjoying the outdoors.
Aaren tends to focus in the following key areas:
Essentials - RHEL, RHV, GlusterFS, openZFS
Automation & Languages - Ansible, Bash, Python
Infra & Storage - practices and training, data integrity
DevOps Consulting - team and culture improvements, legacy modernization
Phil has spent the majority of his career in the infrastructure space focused on systems, storage, and virtualization. Phil has taken on Canadian and international roles working with developers and users to get their code running on systems, making sure those systems are monitored and redundant, and automating processes along the way, including underground mine haul trucks. Phil is super excited about the software defined DevOps world and the open and collaborative work environment that it brings.
Outside of Arctiq, Phil enjoys climbing mountains on bikes, SUP’ing on Okanagan Lake with his wife, tinkering with new tech in his home-lab, and dialing in his espresso machine.
Phil has a strong background in and focuses in on the following key areas:
Essentials - Linux, VMware, and Windows
Automation & Languages - Powershell and Bash
Container and Orchestration - Docker and OpenShift
DevOps Consulting - Team communication and collaboration
Kayla brings years of operations experience to Arctiq, with client support, inside sales, and real-world expertise across a wide range of companies. Kayla is building upon her exciting experiences in the IT industry, and deep background in the arts, through her role as Arctiq’s Operations Lead. With Kayla’s leadership, Arctiq is developing and refining modern processes in finance, partnerships, technical marketing, social media, and client support.
Kayla holds a degree in Creative Writing and English Literature from York University.
Outside of Arctiq, Kayla can be found attending theatrical events, trying out new foods, digging new tunes, and talking more about her cat than you were prepared for (or willing to hear)!
Kayla helps Arctiq with a focus on the following key areas:
Business Development - Sales support, new opportunity management
Partner Management - Ecosystem liason
Hart has over 10 years experience in architecting enterprise environments, delivering, and managing global network and security solutions. Working in unique environments has provided Hart with a strong competence in multi-vendor interoperability and invaluable experience, which serves well to help clients in the most complex of environments. Hart has focused on building strong network foundational knowledge with an emphasis on security, environment hardening, and implementing best practices.
Outside of Arctiq, Hart enjoys spending time with his family, all things baseball, and live music.
Hart’s focus at Arctiq is in the following key areas:
Network Automation and self-healing of enterprise systems
Automating configuration consistency and network security hardening
Integrations with on-premise or cloud-based network architectures using an automated and repeatable approach
Victor is an infrastructure engineer with extensive hands-on network, systems, security and programming experience in the design, development and maintenance of high-volume networks. He leverages ecosystem partner tools to build highly available, fast, secure web applications. Victor also develops and manages tools for monitoring, logging, reporting and alerting of critical applications. His network automation and application performance analysis skill supports real time metrics for continuous code and configuration improvements. Like the other members of Arctiq, Victor is also an advocate for using open source tools in product and platform development, testing, and deployments through continuous integration and delivery strategies.
In his spare time Victor plays Minecraft with his kids, continuously strives to do one more pull up, and dreams of living on a sailboat anchored off an uninhabited island.
Victor’s focus at Arctiq is in the following key areas:
Development workflow automation and CI/CD pipelines
DevOps culture and process enablement
Security analysis, protecting secrets, system hardening, vulnerability testing
More than 20 years ago Roman started his IT career as a Microsoft Certified Systems Engineer. With the emergence of the Internet his focus shifted to programming, development and ecommerce. Later leaving the corporate setting, Roman spent over a decade working as an independent consultant.
A broad scope of infrastructure and development skills allow Roman to deliver holistic advice, insightful troubleshooting and efficient solutions. He is able to utilize his skills to automate processes, slice and dice data and drive efficiencies to deliver business value.
Apart from work, Roman likes to keep on top of new technologies and travel. He also enjoys photography, philosophy and strives to practice minimalism and mindfulness.
At Arctiq, Roman is primarily focused on all things MongoDB and making data more effective for Arctiq’s customers.
Throughout his career, Mohan has worked on System Administration and Network Operations, gaining experience in network architecture and software development. Mohan has troubleshooting expertise in resolving critical infrastructure events, building targeted monitoring tools, and solution validation. He is excited to utilize his technical acumen in his DevOps Consultant role at Arctiq. Mohan currently focuses on Satellite, Ansible, Red Hat Enterprise Linux, and other open source tools to help customers automate their processes.
Outside of work, he can be found spending time with his family, playing cricket, and enjoying the outdoors. Mohan also holds an Honours degree in Mathematics and Computer Science from McMaster University.