Skip to main content

Fossils, Canines, Lookouts, Oh My

Growing Up...

Running even a small Cloud Infrastructure can definitely be a lot of fun.. Whats running, whats slow, whats eating up memory, why is load average at 10, but processors are barely touched.... Always interesting...

Then you scale up, grow up, and move to Containers... Docker is the best thing that ever happened to me (ok, marriage, birth of son, etc., excluded). But then keeping an eye on everything gets even more interesting.

New Relic starts to look like and Old Fossil


I love(d) New Relic, insight into all levels of the stack, all from one place, good alters, graphs, everything to keep us happy.

Then Docker and MicroServices architecture.... 

To get things really monitored, need to install agents all over the place, break a lot of MicroService principles that we have just recently committed to... Ohhh my...

And what about log processing? We don't really pay attention to them enough now, and only use them reactively....

Plus Docker adds a whole new level of headache there....

So the quest begins again

So now we need to find something a bit more MicroService friendly, and since we are a start-up, can't spend tons on it... 


Here are the candidates....



  • New Relic (maybe i just need to relearn it)
  • DataDog
  • Ruxit
  • Rolling our own again.

What we are monitoring..

  • GCE based virtual machines
    • Docker Hosts
    • Ancillary VMs (FTP gateway, NFS server)
  • MongoDb Cluster
    • Sharded Cluster
    • GridFS
  • Core application Docker Instances
    • Node.js application servers
    • NGINX accellerator
    • Apache Web Application Servers
    • Wordpress Instances
  • Web sites and applications
    • Are they up
    • Response times


Next up KPIs for decision...

Labels:

Comments

Post a Comment

Popular posts from this blog

A more sane approach to elastix call recordings

We put together this solution a while ago (thanks Brother Miles!), but never documented it, thought it would a good idea to put it here for anyone else who might need this (and anyone who uses Elastix for high call volume, probably does. The challenge. Elastix/Freepbx has good call recording abilities, but very strangely just dumps everything into a single directory. This can get REALLY bad if you have tens of thousands of calls per day, I have run accross a few scripts to convert and mode call recordings after they are saved, but that is a waste of resources, and makes it hard to integrate call recording UI's. So we figured out the 2 places that need to be changed so that all recorded calls are saved in a Year/Month/day file structure. Just 2 files For regular calls, we need to alter the included and symlinked extensions.conf, this is found at /etc/asterisk/extensions.conf we just need to add the macro-dialout-trunk-predial-hook, which already exists as a secti...
3 comments
Read more

Cloud Enterprise Project/Program Management Tools

In our ongoing quest for an integrated, cloud/Open Source hybrid Business Operating System, we need to pick and implement a tool for project/Program management. Environment Our organization has 8 separate companies, which we need to track project progress across all 8.  For some companies, end users (project resources) will enter information themselves. For other companies and projects, a project co-ordinator will track progress manually from resource input. Task tracking can be on many levels, some projects will be quite detailed, other will be tracked at the milestone level only. Basic Requirements In order for this component to be successfull, it needs to satisfy the following requirements. Integrate with GApps for authentication. Integrate with GApps for document associations Support a fairly large number of project contributers/users (about 100). Handle multiple projects. Enable reporting on multiple levels, from detailed to executive overview. ...
Post a Comment
Read more

Installing dependancies for Sun Ray on Centos 5.8

Ooo the beauty of Linux based proprietary thin clients... One of the best parts is that operating systems get updated, and SRSS doesn't. So re-installing/updating SRSS can be a bit of a moving target. So here are notes from my latest install. Dependencies and prerequisites yum install glib dhcp openldap-clients openldap tftp-server libXp openmotif22 openssl compat-libstdc++-33 libusb-devel compat-openldap gdbm.i386 openldap-devel pdksh libXfont yum groupinstall "Development Tools" wget http://pkgs.repoforge.org/htop/htop-1.0.1-1.el5.rf.x86_64.rpm yum install htop-1.0.1-1.el5.rf.x86_64.rpm (ok, not really required, but helpful on gauging load easily) wget ftp://mirror.switch.ch/pool/3/mirror/centos/5.8/os/i386/CentOS/glib-1.2.10-20.el5.i386.rpm yum install  glib-1.2.10-20.el5.i386.rpm --nogpgcheck -y cd /usr/lib ln -sf libldap-2.3.so.0.2.31 libldap.so.199 ln -sf libgdbm.so.2.0.0 libgdbm.so.3 ln -sf liblber-2.3.so.0.2.31 liblber.so.199 extract the include...
2 comments
Read more