Oct
23
2018
--

Reclaiming space on your Docker PMM server deployment

reclaiming space Docker PMM

reclaiming space Docker PMMRecently we had a customer that had issues with a filled disk on the server hosting their Docker pmm-server environment. They were not able to access the web UI, or even stop the pmm-server container because they had filled the /var/ mount point.

Setting correct expectations

The best way to avoid these kinds of issues in the first place is to plan ahead, and to know exactly with what you are dealing with in terms of disk space requirements. Michael Coburn has written a great blogpost on this matter:

https://www.percona.com/blog/2017/05/04/how-much-disk-space-should-i-allocate-for-percona-monitoring-and-management/

We are now using Prometheus version 2 inside PMM server, so you should take it with a pinch of salt. On the other hand, it will show how you should plan ahead, and think about the “steady state” disk usage, so it’s a good read.

That’s the first step to make sure you won’t get into trouble down the line. But, what happens if you are already in trouble? We’ll see two quick ways that may help reclaiming space.

Before anything else, you should stop any and all PMM clients running, so that you don’t have a race condition after recovering some space, in which metrics coming from the running clients will fill up whatever disk you had freed.

If

pmm-admin stop --all

  won’t work, you can stop the services manually, or even manually kill the running processes as a last resort:

shell> systemctl list-unit-files | grep enabled | grep pmm | awk '{print $1}' | xargs -n 1 systemctl stop
shell> ps ax | egrep "exporter|qan-agent|pmm" | grep -v "ssh" | awk '{print $1}' | xargs kill

Removing unused containers

In order for the next steps to be as effective as possible, make sure there are no unused containers running, or stopped:

shell> docker ps -a

If you see any container that you know you don’t need anymore:

shell> docker stop <container_name>
shell> docker rm -v <container_name>

WARNING! Do not remove the pmm-data container!

Reclaiming space from unused Docker images

After you are done cleaning unused containers, we can move forward with removing unused images. Unless you are manually building your own Docker images, it’s really easy to get them again if needed, so you shouldn’t be afraid of deleting the ones that are not being used. In fact, you don’t need to explicitly download the images. By simply running

docker run … image_name

  Docker will automatically do it for you if it’s not found locally.

shell> docker image prune -a
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
...
Total reclaimed space: 3.97GB

Not too bad, we just reclaimed 4Gb of disk space. This alone should be enough to restart the Docker service and have the pmm-server container back up. But we want more, just because we can ?

Reclaiming space from orphaned Docker volumes

By default, when removing a container (with

docker rm

 ) Docker will not delete the associated volumes, unless you use the -v switch as we did above. This will mean that, unless you were aware of this fact, you will probably have some other gigabytes worth of data occupying disk space. We can easily do this with the volume prune command:

shell> docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
...
Total reclaimed space: 115GB

Yeah… that’s some significant amount of disk space we just reclaimed back! Again, make sure you don’t care about any of the volumes from your past containers to be able to do this safely, since there is no turning back from this, obviously.

For earlier versions of Docker where this command is not available, you can check this link.

Planning ahead

As mentioned before, you should now revisit Michael’s blogpost, and set the metrics retention and queries retention variables to whatever makes sense for your environment. Even if you plan ahead, you may not be counting on the additional variable overhead of images and orphaned volumes, so you may want to (warning: shameless plug for my own blogpost ahead) use different mount points for your PMM deployment, and avoid using the shared /var/lib/docker/ mount point for it.

PMM also includes a Disk Space usage dashboard, that you can use to monitor this.

Don’t forget to start back up your PMM clients, and continue to monitor them 24×7!

Photo by Andrew Wulf on Unsplash

Sep
04
2018
--

Atlassian acquires OpsGenie, launches Jira Ops for managing incidents

Atlassian today announced the first beta of a new edition of its flagship Jira project and issue tracking tool that is meant to help ops teams handle incidents faster and more efficiently.

Jira Ops integrates with tools like OpsGenie, PagerDuty, xMatters, Statuspage, Slack and others. Many teams already use these tools when their services go down, but Atlassian argues that most companies currently use a rather ad hoc approach to working with them. Jira Ops aims to be the glue that keeps everybody on the same page and provides visibility into ongoing incidents.

Update: after Atlassian announced Jira Ops, it also announced that it has acquired OpsGenie for $295 million.

This is obviously not the first time Atlassian is using Jira to branch out from its core developer audience. Jira Service Desk and Jira Core, for example, aim at a far broader audience. Ops, however, goes after a very specific vertical.

“Service Desk was the first step,” Jens Schumacher, Head of Software Teams at Atlassian, told me. And we were looking at what are the other verticals that we can attack with Jira.” Schumacher also noted that Atlassian built a lot of tools for its internal ops teams over the years to glue together all the different pieces that are necessary to track and manage incidents. With Jira Ops, the company is essentially turning its own playbook into a product.

In a way, though, using Jira Ops adds yet another piece to the puzzle. Schumacher, however, argues that the idea here is to have a single place to manage the process. “The is that when an incident happens, you have a central place where you can go, where you can find out everything about the incident,” he said. “You can see who has been paged and alerted; you can alert more people if you need to right from there; you know what Slack channel the incident is being discussed in.”

Unlike some of Atlassian’s other products, the company doesn’t currently have any plans to launch a self-hosted version of Jira Ops. The argument here is pretty straightforward: if your infrastructure goes down, then Jira Opes could also go do down — and then you don’t have a tool for managing that downtime.

Jira Ops is now available for free for early access beta users. The company expects to launch version 1.0 in early 2019. By then Atlassian will surely also have figured out a pricing plan, something it didn’t announce today.

Jun
13
2018
--

Docker aims to federate container management across clouds

When Docker burst on the scene in 2013, it brought the idea of containers to a broad audience. Since then Kubernetes has emerged as a way to orchestrate the delivery of those containerized apps, but Docker saw a gap that wasn’t being addressed beyond pure container deployment that they are trying to address with the next release of Docker Enterprise Edition. Docker made the announcement today at DockerCon in San Francisco.

Scott Johnston, chief product officer at Docker says that Docker Enterprise Edition’s new federated application management feature helps operations manage multiple clusters, whether those clusters are on premise, in the cloud or across different public cloud providers. This allows federated management of application wherever they live and supports managed Kubernetes tools from the big three public cloud providers including Azure AKS, AWS EKS and Google GKE.

Johnston says that deploying the containers is just the first part of the problem. There is a whole set of issues to deal with outside of Kubernetes (and other orchestration tools) once your application begins being deployed. “So, you know, you get portability of containers with the Docker format and the Kubernetes or Compose description files, but once you land on an environment, that environment has deployment scripts, security models, user management and [so forth]. So while the app is portable, the management of these applications is not,” he explained.

He says that can lead to a set of separate deployment tools creating a new level of complexity that using containers was supposed to eliminate. This is especially true when deploying across multiple clouds (and on prem sometimes too). If you need load balancing, security, testing and so forth — the kinds of tasks the operations team has to undertake — and you want to apply these in a consistent way regardless of the environment, Johnston says that Docker EE should help by creating a single place to manage across environments and achieve that cloud native goal of managing all your applications and data and infrastructure in a unified way.

In addition to the federated management component, Docker also announced Windows Server containers on Kubernetes for Docker Enterprise Edition. It had previously announced support for Linux containers last year.

Finally, the company is introducing a template-based approach to Docker deployment to enable people in the organization with a bit less technical sophistication to deploy from a guided graphical process instead of a command line interface.

The federated application management is available in Beta starting the second half of this year, support for Windows Server Containers will be included in the next release of Docker Enterprise Edition later this year and Templates will be available in Docker Desktop in Beta later this year.

May
30
2018
--

Vulcan Cyber raises $4M for its vulnerability remediation platform

Vulcan Cyber, a Tel Aviv-based security startup that helps enterprises quickly detect and fix vulnerabilities in their software stack and code, is coming out of stealth today and announcing a $4 million seed round led by YL Ventures with participation from a number of other cybersecurity investors.

The general idea behind Vulcan Cyber is that as businesses continue to increase the pace at which they build and adopt new software, the risk of introducing vulnerabilities only increases. But at the same time, most companies don’t have the tools in place to automatically detect and mitigate these issues, meaning that it can often take weeks before a patch rolls out.

The company argues that its position in the cybersecurity space is somewhat unique because it doesn’t just focus on detecting vulnerabilities but also helps businesses remediate them. All users have to do is give Vulcan access to the APIs of their existing vulnerability, DevOps and IT tools and the service will simply take over from there. It then watches over both the infrastructure as well as the code that runs on it.

“It might sound more glamorous to talk about zero-day and next-generation threats, but vulnerability remediation is truly where the rubber meets the road,” said Yaniv Bar-Dayan, Vulcan Cyber’s CEO and co-founder. “The only way to deal with this continuous risk exposure is through continuous remediation, achieved with robust data collection, advanced analytics, automation, and closed-loop remediation planning, orchestration and validation. This is exactly what we are delivering to IT security teams with Vulcan Cyber.”

Vulcan cyber plays nicely with all o the major cloud platforms, as well as tools like Puppet, Chef and Ansible, as well as GitHub and Bitbucket. It also integrates with a number of major security testing tools and vulnerability scanners, including Black Duck, Nessus, Fortify, Tripwire, Checkmarx, Rapid7 and Veracode.

Apr
21
2018
--

Through luck and grit, Datadog is fusing the culture of developers and operations

There used to be two cultures in the enterprise around technology. On one side were software engineers, who built out the applications needed by employees to conduct the business of their companies. On the other side were sysadmins, who were territorially protective of their hardware domain — the servers, switches, and storage boxes needed to power all of that software. Many a great comedy routine has been made at the interface of those two cultures, but they remained divergent.

That is, until the cloud changed everything. Suddenly, there was increasing overlap in the skills required for software engineering and operations, as well as a greater need for collaboration between the two sides to effectively deploy applications. Yet, while these two halves eventually became one whole, the software monitoring tools used by them were often entirely separate.

New York City-based Datadog was designed to bring these two cultures together to create a more nimble and collaborative software and operations culture. Founded in 2010 by Olivier Pomel and Alexis Lê-Quôc, the product offers monitoring and analytics for cloud-based workflows, allowing ops team to track and analyze deployments and developers to instrument their applications. Pomel said that “the root of all of this collaboration is to make sure that everyone has the same understanding of the problem.”

The company has had dizzying success. Pomel declined to disclose precise numbers, but says the company had “north of $100 million” of recurring revenue in the past twelve months, and “we have been doubling that every year so far.” The company, headquartered in the New York Times Building in Times Square, employs more than 600 people across its various worldwide offices. The company has raised nearly $150 million of venture capital according to Crunchbase, and is perennially on banker’s short lists for strong IPO prospects.

The real story though is just how much luck and happenstance can help put wind in the sails of a company.

Pomel first met Lê-Quôc while an undergraduate in France. He was working on running the campus network, and helped to discover that Lê-Quôc had hacked the network. Lê-Quôc was eventually disconnected, and Pomel would migrate to IBM’s upstate New York offices after graduation. After IBM, he led technology at Wireless Generation, a K-12 startup, where he ran into Lê-Quôc again, who was heading up ops for the company. The two cultures of develops and ops was glaring at the startup, where “we had developers who hated operations” and there was much “finger-pointing.”

Putting aside any lingering grievances from their undergrad days, the two began to explore how they could ameliorate the cultural differences they witnessed between their respective teams. “Bringing dev and ops together is not a feature, it is core,” Pomel explained. At the same time, they noticed that companies were increasingly talking about building on Amazon Web Services, which in 2009, was still a relatively new concept. They incorporated Datadog in 2010 as a cloud-first monitoring solution, and launched general availability for the product in 2012.

Luck didn’t just bring the founders together twice, it also defined the currents of their market. Datadog was among the first cloud-native monitoring solutions, and the superlative success of cloud infrastructure in penetrating the enterprise the past few years has benefitted the company enormously. We had “exactly the right product at the right time,” Pomel said, and “a lot of it was luck.” He continued, “It’s healthy to recognize that not everything comes from your genius, because what works once doesn’t always work a second time.”

While startups have been a feature in New York for decades, enterprise infrastructure was in many ways in a dark age when the company launched, which made early fundraising difficult. “None of the West Coast investors were listening,” Pomel said, and “East Coast investors didn’t understand the infrastructure space well enough to take risks.” Even when he could get a West Coast VC to chat with him, they “thought it was a form of mental impairment to start an infrastructure startup in New York.”

Those fundraising difficulties ended up proving a boon for Datadog, because it forced the company to connect with customers much earlier and more often than it might have otherwise. Pomel said, “it forced us to spend all of our time with customers and people who were related to the problem” and ultimately, “it grounded us in the customer problem.” Pomel believes that the company’s early DNA of deeply listening to customers has allowed it to continue to outcompete its rivals on the West Coast.

More success is likely to come as companies continue to move their infrastructure onto the cloud. Datadog used to have a roughly even mix of private and public cloud business, and now the balance is moving increasingly toward the public side. Even large financial institutions, which have been reticent in transitioning their infrastructures, have now started to aggressively embrace cloud as the future of computing in the industry, according to Pomel.

Datadog intends to continue to add new modules to its core monitoring toolkit and expand its team. As the company has grown, so has the need to put in place more processes as parts of the company break. Quoting his co-founder, Pomel said the message to employees is “don’t mind the rattling sound — it is a spaceship, not an airliner” and “things are going to break and change, and it is normal.”

Much as Datadog has bridged the gap between developers and ops, Pomel hopes to continue to give back to the New York startup ecosystem by bridging the gap between technical startups and venture capital. He has made a series of angel investments into local emerging enterprise and data startups, including Generable, Seva, and Windmill. Hard work and a lot of luck is propelling Datadog into the top echelon of enterprise startups, pulling New York along with it.

Feb
12
2018
--

XebiaLabs lands $100 million Series B led by Susquehanna Growth Equity and Accel

 XebiaLabs, the Boston-based software startup that helps companies automate DevOps functions, announced a healthy $100 million Series B investment led by Susquehanna Growth Equity and Accel. Today’s investment brings the total raised to $121.5 million. Derek Langone, Xebia’s CEO says they raised the money out of a desire to expand more rapidly. “You always want to raise money… Read More

Aug
03
2017
--

Heptio launches two new open source projects that make using Kubernetes easier

 Heptio, the Seattle-based company recently launched by Kubernetes co-founders Craig McLuckie and Joe Beda, wants to make it easier for businesses to use Kubernetes in production. Since its launch in late 2016, the well-funded company has remained pretty quiet about its products, but today, the team released two open source projects into the wild: Ark and Sonobuoy.
While Kubernetes&#8217… Read More

Jun
13
2017
--

Atlassian launches a new subscription bundle that includes all of its developer tools

This photo taken on December 8, 2015 shows flags adorning the head office of Australian tech start-up Atlassian . Atlassian today announced the launch of Atlassian Stack, a new subscription service that bundles virtually all of the company’s self-hosted developer tools into a single offering. Starting at $186,875 per year for 1,000 licenses, this new bundle is meant to make the procurement process for enterprises easier and cheaper (despite what looks like an eye watering price at first). Instead… Read More

Dec
01
2016
--

AWS Personal Health Dashboard helps developers monitor the state of their cloud apps

img_20161201_091825 DevOps teams will be happy to hear that Amazon is launching its own dashboard for Amazon Web Services. Personal Health Dashboard, as the company calls it, is its latest release from the stage of re:Invent 2016 to support more advanced cloud apps monitoring. The tool puts critical infrastructure data in one place. The dashboard will automatically notify teams of failures and allow them… Read More

Jun
21
2016
--

Microsoft expands its support for Docker containers

docker_whale_dockerconeu Docker is hosting its developer conference in Seattle this week and showcasing the quickly growing ecosystem that has grown up around its container tools. One of the companies on stage today was Microsoft, which announced that it is great expanding its support for Docker containers by more deeply integrating it into a number of its enterprise and DevOps tools. Microsoft’s interest in… Read More

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com