By adopting time series data, users can discover the underlying causes of trends and monitor any changing patterns over time. With these tools, it is also possible to compare data over time and detect the cause of changing patterns. Besides this, these tools can help discover any outliers in the data while making important predictions.
For this reason, the use of monitoring tools must be embraced by businesses to provide a good user experience while measuring systemic patterns over a specific period. With this in mind, it is important to talk about one of the most common monitoring tools that can gather and evaluate application and infrastructure metrics: Graphite. With these tools, users can determine data over consistent intervals. This comprehensive guide will discuss, in detail, the benefits and features of Graphite monitoring tools.
Graphite Monitoring: What Is It?
Graphite is an open-source monitoring tool that many organizations widely adopt to monitor and graph time-series data. With this tool, team members can plot data from various sources on a massive scale. This makes it a free, next-generation tool that simultaneously holds, visualizes, and queries time-series data gathered from different sources – devices, systems, or servers.
Graphite monitoring solutions also offer full visibility while allowing users to track the behavior of systems and applications. Besides this, it helps detect different errors while facilitating continuous improvement. It also assists businesses in storing, monitoring, and visualizing data without much stress. In addition, it helps in generating alerts for specific events.
Using Carbon and Whisper techniques, Graphite can better understand any underlying time-series data and its components. This tool offers the GitHub Engineering team the necessary tools to gather and compare huge amounts of time-series data.
The monitoring tool also offers an API, ensuring system administrators, app developers, and network engineers can make the most of time-series data. Graphite has enjoyed popularity as one of the most trusted open-source monitoring tools, which can be adopted for time-series visualization and data storage. Graphite boasts several excellent features, which make it unique among other related tools.
Graphite is a simple tool that organizations commonly adopt for recording and plotting data. This tool helps them monitor the performance of their websites, servers, and apps. Besides monitoring, Graphite has greater use cases, which companies make the most of to evaluate huge amounts of time-series data and trend analysis. Some of the top companies that have adopted the use of Graphite include GitHub, Reddit, Etsy, Hive Streaming, and Booking.com.
Graphite Features
Graphite is designed with many exciting features, contributing to its popularity among users:
- Its simple interface allows for a unique option.
- It comes with a simple metrics format, which makes it a top monitoring tool. This metrics format requires just an epoch timestamp, the dot-delimited metric name, and a numeric value for data submission to the Carbon listener. It takes little time for a user to send telemetry by using simple shortcodes.
- Graphite is designed with an extensive statistical library. This makes use of API to carry out computations on time-series data. Also, this feature provides primitives that can facilitate the classification of the maximum, minimum, and mean of a series. Furthermore, it is also possible for users to estimate the percentiles for each series. Graphite allows Holt-Winters forecasting algorithms while enabling users to sort and filter each criterion virtually.
- The composer and amazing web interface are nothing short of exceptional. Another unique feature that makes Graphite amazing is Rapid Prototyping. With this, users can easily navigate metrics while taking snapshots of graphs. Furthermore, users can add and remove metrics from the graphs by adopting this tool. In addition, Graphite helps expose rendering APIs and apply statistical transformations.
- On the other hand, the composer ensures that users can visualize and correlate the data. The graph can be refreshed automatically after each step, which can help display new arrangements. The Graphite monitoring tool also helps identify causal relationships while rapidly isolating anomalous behavior.
- Another top feature of Graphite is Graphing API. The graph does not rely on static configurations to create a new graph. Also, It adopts comprehensive APIs to create and compose dashboards and information graphics. Besides this, users can share the composed graphs with other users while embedding the URL in websites or dashboards.
About Graphite
Graphite monitoring solution offers complete visibility and contributes to eliminating irrelevant downtime. With Graphite, users can be kept posted about every part of the stack that is about to fail. Besides, it ensures that IT administrators can collect metrics while exposing them to in-depth knowledge about the servers, network components, and applications. As a result, this helps users to understand the causes of major problems in performance as they try to make informed business decisions.
With Graphite, teams can also save valuable time and energy. This is done by defining which system requires more attention and the metrics that need to be measured. Administrators can use this feature to create intuitive dashboards and report the best fit, which can be used for business objectives. Graphite can be used easily
Graphite is a ready-to-use infrastructure monitoring solution for enterprises that can be integrated with existing infrastructures and address issues with time-series data warehousing, performance measurements, and data visualization. It powers both the technical and the business metrics tracking in GitHub.
Pros of Using Graphite
- Helps in achieving accurate results
- Easy to find people as part of the community.
- Graphite is a free, open-source monitoring tool. As a result, users have full access to download their source code from the GitHub repository.
- The software is easy to use.
- With Graphite, users can automate the display and data collection process.
- Graphite can support different formats
Installing Graphite
Before installing graphite, some prerequisites must be considered:
- A server running Ubuntu 22.04.
- A non-root user with sudo privileges.
- A Fully Qualified domain name (FQDN) pointing to the server
- The uncomplicated Firewall(UFW) is enabled and running.
- Once you ensure they are all updated, install basic utility packages.
$ sudo apt install wget curl nano unzip -y
Configure Firewall
The first step involves allowing HTTPS and HTTP connections. Check the firewall’s status.
$ sudo ufw status
Then, allow HTTP and HTTPS ports.
$ sudo ufw allow http
$ sudo ufw allow https
After this, check the status again to confirm.
$ sudo ufw status
Install Necessary Packages
For this guide, Graphite will be installed using the PIP Python package manager. First, install the required packages.
$ sudo apt install vim python3-dev python3-pip libcairo2-dev libffi-dev build-essential
Install Graphite and Graphite Web
Graphite is installed in the /opt/graphite directory.
$ export PYTHONPATH=”/opt/graphite/lib/:/opt/graphite/webapp/”
$ sudo pip install –no-binary=:all: https://github.com/graphite-project/whisper/tarball/master
$ sudo pip install –no-binary=:all: https://github.com/graphite-project/carbon/tarball/master
$ sudo pip install –no-binary=:all: https://github.com/graphite-project/graphite-web/tarball/master
Install and Configure PostgreSQL
The installation will be done by using PostgreSQL’s official APT repository. The following command should be run:
$ curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg –dearmor | sudo tee /usr/share/keyrings/postgresql-key.gpg >/dev/null
Add the APT repository to your sources list.
$ sudo sh -c ‘echo “deb [signed-by=/usr/share/keyrings/postgresql-key.gpg arch=amd64] http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main” > /etc/apt/sources.list.d/pgdg.list’
Run the following command to update the system repository.
$ sudo apt update
From here, you can install PostgreSQL using this command:
$ sudo apt install postgresql postgresql-contrib libpq-dev
Confirm the status of the PostgreSQL service.
$ sudo systemctl status postgresql
? postgresql.service – PostgreSQL RDBMS
Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
Active: active (exited) since Tue 2022-09-27 10:09:35 UTC; 4s ago
Process: 4456 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 4456 (code=exited, status=0/SUCCESS)
CPU: 1ms
Sep 27 10:09:35 matrix systemd[1]: Starting PostgreSQL RDBMS…
Sep 27 10:09:35 matrix systemd[1]: Finished PostgreSQL RDBMS.
The service has been enabled and runs by default.
Log in to the PostgreSQL shell.
$ sudo -su postgres psql
Then, create a database user for Graphite.
postgres=# CREATE USER graphite WITH PASSWORD ‘your_password’;
After this, create a Graphite database and give the graphite user ownership.
postgres=# CREATE DATABASE graphitedb WITH OWNER graphite;
Exit the PostgreSQL shell.
postgres=# \q
Configure Graphite Carbon and Web
Configure carbon
There are three services in Carbon. The first is carbon-cache, which can accept metrics as it writes them to the disk. The second is carbon-relay, and this replicates the data. The third one is the carbon-aggregator. This runs in front of the carbon-cache service to buffer metrics over time before they are forwarded into Whisper.
While the configuration of carbon-aggregator and carbon-relay is an option, carbon-cache is very important.
Create the carbon.conf file by using this example file.
$ sudo cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
Then, create the storage schema configuration.
$ sudo cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
Open the storage schema configuration file.
$ sudo nano /opt/graphite/conf/storage-schemas.conf
This contains entries like
[carbon]
pattern = ^carbon\.
retentions = 60:90d
This says that there is a pattern that matches a regular expression ^carbon\. should retain the data with retention policy 60:90d. This explains:
- how often a metric is recorded: 60 seconds
- length of time to store those values: 90 days
Configure the Graphite Web
Then, configure the Graphite web app.
Generate a secret key for the Graphite application.
$ python3 -c ‘from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())’
sp%71)6b$%^bc(7xpz1d!)x3(azog01&k^8l02*!y0#)72p07y
Create the web app settings file.
$ sudo cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
You need to configure the Graphite web application with the database settings. Next, open the local_settings.py for editing.
$ sudo nano /opt/graphite/webapp/graphite/local_settings.py
Uncomment the SECRET_KEY variable and enter a random value for it.
SECRET_KEY = ‘your-secret-key’
Uncomment the ALLOWED_HOSTS variable.
ALLOWED_HOSTS = [ ‘*’ ]
Uncomment the TIME_ZONE variable and set it to the appropriate value.
TIME_ZONE = ‘Asia/Kolkata’
Uncomment the USE_REMOTE_USER_AUTHENTICATION variable and set it to TRUE. This will ensure that the remote user becomes authenticated before any database changes can be made.
USE_REMOTE_USER_AUTHENTICATION = True
Next, the database settings must be changed.
DATABASES = {
‘default’: {
‘NAME’: ‘graphitedb’,
‘ENGINE’: ‘django.db.backends.postgresql_psycopg2’,
‘USER’: ‘graphite’,
‘PASSWORD’: ‘your_password’,
‘HOST’: ‘127.0.0.1’,
‘PORT’: ”
}
}
Save the file by clicking Ctrl + X and entering Y when prompted.
Install certain prerequisites for Python’s PostgreSQL wrapper.
$ sudo pip install psycopg2-binary
To import the database schema, enter the following command:
$ sudo PYTHONPATH=/opt/graphite/webapp/ django-admin.py migrate –settings=graphite.settings
The following output will be generated:
Operations to perform:
Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener
Running migrations:
Applying contenttypes.0001_initial… OK
Applying auth.0001_initial… OK
Applying account.0001_initial… OK
Applying admin.0001_initial… OK
Applying admin.0002_logentry_remove_auto_add… OK
Applying admin.0003_logentry_add_action_flag_choices… OK
Applying contenttypes.0002_remove_content_type_name… OK
Applying auth.0002_alter_permission_name_max_length… OK
Applying auth.0003_alter_user_email_max_length… OK
Applying auth.0004_alter_user_username_opts… OK
Applying auth.0005_alter_user_last_login_null… OK
Applying auth.0006_require_contenttypes_0002… OK
Applying auth.0007_alter_validators_add_error_messages… OK
Applying auth.0008_alter_user_username_max_length… OK
Applying auth.0009_alter_user_last_name_max_length… OK
Applying auth.0010_alter_group_name_max_length… OK
Applying auth.0011_update_proxy_permissions… OK
Applying auth.0012_alter_user_first_name_max_length… OK
Applying dashboard.0001_initial… OK
Applying events.0001_initial… OK
Applying sessions.0001_initial… OK
Applying tagging.0001_initial… OK
Applying tagging.0002_on_delete… OK
Applying tags.0001_initial… OK
Applying url_shortener.0001_initial… OK
Next, collect the static files.
$ sudo PYTHONPATH=/opt/graphite/webapp/ django-admin.py collectstatic –settings=graphite.settings
The correct ownership settings should be set
$ sudo chown -R www-data:www-data /opt/graphite/storage/
$ sudo chown -R www-data:www-data /opt/graphite/static/
$ sudo chown -R www-data:www-data /opt/graphite/webapp/
Create a root user for login.
$ sudo PYTHONPATH=/opt/graphite/webapp/ django-admin.py createsuperuser –settings=graphite.settings
Username (leave blank to use ‘root’): navjot
Email address: navjot@example.com
Password:
Password (again):
Superuser created successfully.
Create a superuser. This user will be used later to connect to the Graphite application.
Configure Apache
Next, install the Apache server.
$ sudo apt install apache2 libapache2-mod-wsgi-py3
Create the mod_wsgi file.
$ sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
Copy the graphite example configuration file to the Apache location.
$ sudo cp /opt/graphite/examples/example-graphite-vhost.conf /etc/apache2/sites-available/graphite.conf
Next, edit by opening the Graphite configuration file.
$ sudo nano /etc/apache2/sites-available/graphite.conf
Change the port number in the first line from 80 to 127.0.0.1:8080. Putting 127.0.0.1 in front of it restricts it from being accessible over the web.
<VirtualHost 127.0.0.1:8080>
Add your domain name.
ServerName graphite.example.com #Replace with your domain
The following lines below the line Alias /static/ /opt/graphite/static/ should be added
#Add below lines
<Directory /opt/graphite/static/>
Require all granted
</Directory>
Save the file by pressing Ctrl + X and entering Y when prompted.
Then, disable the default virtual host while enabling the Graphite virtual host file.
$ sudo a2dissite 000-default
$ sudo a2ensite graphite
Then, it is important to command Apache to stop listening to port 80 and start listening to port 8080. After all, Nginx will be used as a proxy server.
Open the file /etc/apache2/ports.conf for editing.
$ sudo nano /etc/apache2/ports.conf
Look for the line Listen 80 and substitute it with the following:
Listen 127.0.0.1:8080
Pressing Ctrl + X to save the file and entering Y when prompted.
Now, restart the Apache server.
$ sudo systemctl restart apache2
You should verify that Graphite is accessible and runs properly. Do so with this command:
$ curl 127.0.0.1:8080
The following output will be generated:
<!DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01 Frameset//EN” “http://www.w3.org/TR/html4/frameset.dtd”>
<!– Copyright 2008 Orbitz WorldWide
Licensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and limitations under the License. –>
<html>
<head>
<title>Graphite Browser</title>
</head>
<frameset rows=”80,*” frameborder=”1″ border=”1″>
<frame src=”/browser/header” name=”Header” id=’header’ scrolling=”no” noresize=”true” />
<frame src=”/composer?” name=”content” id=”composerFrame”/>
</frameset>
</html>
With this output, it works well
Install Nginx
Nginx will be used as a proxy server for Apache. The benefits include security and obscurity.
Installing the latest version involves downloading the official Nginx repository.
Import Nginx’s signing key.
$ curl https://nginx.org/keys/nginx_signing.key | gpg –dearmor \
| sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null
Add the repository for Nginx’s stable version.
$ echo “deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg arch=amd64] \
http://nginx.org/packages/ubuntu `lsb_release -cs` nginx” \
| sudo tee /etc/apt/sources.list.d/nginx.list
Next, update the system repositories.
$ sudo apt update
After this, install Nginx.
$ sudo apt install nginx
Then, verify the installation.
$ nginx -v
nginx version: nginx/1.22.0
Start the Nginx server.
$ sudo systemctl start nginx
Install SSL
Certbot must be installed to generate the SSL certificate. You can install the Certbot by using Ubuntu’s repository or by grabbing the new version using Snapdtool. This guide will make use of the Snapd version.
$ sudo snap install core
$ sudo snap refresh core
Install Certbot.
$ sudo snap install –classic certbot
Then, input this command:
$ sudo ln -s /snap/bin/certbot /usr/bin/certbot
Generate an SSL certificate with this command:
$ sudo certbot certonly –nginx –agree-tos –no-eff-email –staple-ocsp –preferred-challenges http -m name@example.com -d graphite.example.com
This aforementioned command will download a certificate to the /etc/letsencrypt/live/graphite.example.com directory on your server.
Generate a Diffie-Hellman group certificate.
$ sudo openssl dhparam -dsaparam -out /etc/ssl/certs/dhparam.pem 4096
Check the Certbot renewal scheduler service.
$ sudo systemctl list-timers
There is snap.certbot.renew.service, one of the services scheduled to run.
NEXT LEFT LAST PASSED UNIT ACTIVATES
…………………………………………………………………………………………………………………
Wed 2022-09-28 00:00:00 UTC 7h left Tue 2022-09-27 00:00:01 UTC 16h ago logrotate.timer logrotate.service
Wed 2022-09-28 02:39:09 UTC 10h left Tue 2022-09-27 09:42:42 UTC 6h ago apt-daily.timer apt-daily.service
Wed 2022-09-28 06:02:00 UTC 13h left n/a n/a snap.certbot.renew.timer snap.certbot.renew.service
If you wish to confirm if the SSL renewal runs fine, carry out a dry run
$ sudo certbot renew –dry-run
You are good to go if no error shows up. The certificate will automatically renew.
Configure Nginx
To edit, open the file /etc/nginx/nginx.conf.
$ sudo nano /etc/nginx/nginx.conf
Add the following line before the line include /etc/nginx/conf.d/*.conf;
server_names_hash_bucket_size 64;
Save the file by pressing Ctrl + X and entering Y when prompted.
Create and open the file /etc/nginx/conf.d/uvdesk.conf for editing.
$ sudo nano /etc/nginx/conf.d/graphite.conf
Paste the following code in it.
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name graphite.example.com;
access_log /var/log/nginx/graphite.access.log;
error_log /var/log/nginx/graphite.error.log;
# SSL
ssl_certificate /etc/letsencrypt/live/graphite.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/graphite.example.com/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/graphite.example.com/chain.pem;
ssl_session_timeout 5m;
ssl_session_cache shared:MozSSL:10m;
ssl_session_tickets off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_ecdh_curve X25519:prime256v1:secp384r1:secp521r1;
ssl_stapling on;
ssl_stapling_verify on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
resolver 8.8.8.8;
location / {
proxy_set_header Connection “upgrade”;
proxy_set_header Upgrade $http_upgrade;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://127.0.0.1:8080;
proxy_redirect off;
}
}
# enforce HTTPS
server {
listen 80;
listen [::]:80;
server_name graphite.example.com;
return 301 https://$host$request_uri;
}
Save the file by pressing Ctrl + X and entering Y when prompted once finished.
Verify the Nginx configuration file syntax.
$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Restart the Nginx service.
$ sudo systemctl restart nginx
Then, you are set to access and use Graphite. First, visit https://graphite.example.com. Then, hit the log-in link and input the superuser details created above. Then add any random data with this command:
$ echo “test.count 9 `date +%s`” | nc -q0 127.0.0.1 2003;
You can also add more values by looping through values.
$ for i in 4 6 8 16 2; do echo “test.count $i `date +%s`” | nc -q0 127.0.0.1 2003; sleep 6; done
Integration With Other Tools
Graphite is capable of supporting integrations with various third-party tools. These include Jira, Oracle, NetSuite, SAP, Zendesk, and more. We will provide more details about some other tools Graphite can be integrated with.
- Collectd A daemon that can periodically collect metrics and statistics related to the performance of systems and applications. This tool is compatible with macOS, Linux, OpenBSD, and FreeBSD operating systems. With Graphite, users can rely on some features, making Graphite read directly from RRD files that Collectd stores.
- Sensu Go This is another monitoring solution that facilities the transfer of metrics to Graphite. Users can use a centralized management interface, which can help them easily manage infrastructure-related complexities.
- Grafana This is an open-source monitoring solution that adopts various graph editing features, as well as a dashboard creation interface. Grafana is also designed with a Graphite target parser, which makes it easier to edit metrics and functions.
Conclusion
As the use cases for sensors, applications, and system rise, organizations have found it difficult to track data manually.
For this reason, monitoring systems have become more important for businesses. Users rely on the best features of Graphite to help move their organizations forward. It is easy to install – as discussed above in detail – and can work well with other tools.