by

Interesting Thoughts on Cloud Server Performance

Apache load testing on a Cloud Server – Jason – 7/31/2009

I recently created a cloud server for a wordpress blog, and configured it to the point that the blog was working OK.  Then I decided to check the performance aspects of the server, as it was a small 256 MB + 10GB machine.
Using apachebench (ab), I ran some load tests on the blog home page.  The server choked to death. It was swapping so bad, that RackSpace Cloud sent me this email:

This is an automatic notification to let you know that your Cloud Server, city.appcove.com, is showing a considerable amount of consistent swapping activity. Quite often this is an indicator that your application or database are not as efficient as they could be. It also may indicate that you need to upgrade your Cloud Server for more RAM.

That’s strange…
I found that the response rate was:

4 requests per second, 10 concurrent connections

When the concurrency was raised to 50, the server died.  It took 10 minutes for it to calm down enough that I could LOG IN and KILL apache.
—
So upon further investingation, I found that the default httpd.conf configuration was WAY TOO LARGE:
We’re only working with 256 MB ram here, so if each apache process takes up any amount of memory at all, we have a low limit.

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

Only after drastically reducing the configuration to the following, did we get reasonable performance:

<IfModule prefork.c>
StartServers       4
MinSpareServers    2
MaxSpareServers   4
ServerLimit      4
MaxClients       4
MaxRequestsPerChild  4000
</IfModule>

As it turns out, the performance went up considerably:

16 requests per second, 50 concurrent connections

Still, I thought that it could get better.  So I looked into installing some PHP opcode caching software.

http://www.php.net/manual/en/intro.apc.php

The Alternative PHP Cache (APC) is a free and open opcode cache for PHP. Its goal is to provide a free, open, and robust framework for caching and optimizing PHP intermediate code.

As it turns out, it was easy to install.

# yum install php-pecl-apc

And after restarting apache:

47 requests per second, 50 concurrent connections

Even during this load test, the site was still responsive from a web browser.
Not bad for a cheap little Cloud Server, eh?
by

Starting to use SolidWorks

We recently purchased a copy of SolidWorks to use with our various engineering projects.  In about 3 hours, I was able to learn how to create parts, assemblies, and drawings.  

The software (to this point) is nothing short of incredible.  And I’m just using the most basic features.  

Here is a graphic of Rubber Band Gun #6 internals…

55254

 

I’ll have more updates as we progress deeper into the software.

-Jason

by

Great post on PHP’s preg_match regular expressions

Over at smashing magazine, they have a great post on PHP’s preg_match() regular expressions.  They cover a number of advanced topics in light detail, including:

1. Greediness/Laziness

2. Back Referencing

3. Named Groups

4. Word Boundaries

5. Atomic Groups

6. Recursion

7. Callbacks

8. Commenting

See it here!

http://www.smashingmagazine.com/2009/05/06/introduction-to-advanced-regular-expressions/

by

Perspective on keeping your computer secure.

I’d like to start this off by saying if you want better security…

…switch to a good BSD or Linux distribution!

What follows is an email that I typed up to someone who was being aggressively sold anti virus software by a PC security vendor. It was not intended to be 100% technically complete, but rather just an overview and recommendations for that specific person.

Your mileage may vary.

Hello,

There are only a couple ways of getting a virus:

  • Your computer has a security hole, and you visit a malicious web site.
  • Your computer has a security hole, and another computer “pushes” a virus into your computer (other computers in the same office).
  • Your email client has a security hole and you read a malicious email.
  • You open a malicious attachment
  • You download stuff or install stuff that other people gave you.

The last two are bold, because they are the most common.  And the most easy to avoid.

Here is how to not get viruses:

  1. I DO NOT open attachments from people unless I know WHAT THEY ARE and WHY IT WAS SENT to me.
  2. I DO NOT download stuff from the internet, except from the most reputable sites (Microsoft, Google, Sun, etc…)
  3. I KEEP my computers up to date all the time by running windows update.
  4. I DO NOT browse the internet using Internet Explorer.   I only use it for specific sites that require it.

What if you get one anyway, through something that is beyond your control?

The thing to remember here is that computers can DIE AT ANY TIME WITHOUT NOTICE.  They are fragile machines that have 1,000,000 things that can go wrong.  If you are keeping data on your PC and planning on it being there tomorrow, you will eventually be in for a nasty surprise.

Your computer will stop working at some point due to (a) malfunction, (b) virus, (c) hardware failure, (d) software failure.

So what is the answer?  Backup, backup, backup…

—
To illustrate…  I could lose any of my computers right now, and not be much more than inconvenienced.  In fact, I just erased my whole PC at home and had 100% confidence that I could put everything back.

That is the attitude that NEEDS to be taken — PC’s are totally unreliable.  Keep anything of any value always backed up.

How do you keep it backed up?

I did a good bit of research on that.  Here is what my circumstance looks like:

—
At work, we keep everything on the server.  And we backup the server every day.  However, this is impractical for some users who just naturally use “My Documents” and forget to use the server drive.

—
At home, I use http://sugarsync.com.  That is a great service that ALWAYS keeps your computers backed up ALL the time.  You just tell it what to keep track of, and it notices any time there is a change and backs it up.  I think this is an excellent solution (it starts at $5.00 / month for 30 GB or $10.00 / month for 60 GB).

I recommend that you guys take a look at sugar sync.  It’s a snap to use, and gives you the confidence that your PC’s are unreliable, but you are covered.  It also keeps the latest 5 versions of each file, incase you need to look back at a previous version.  I did a lot of research before picking one that I was confident in.

Actually, my laptop just died.  Really — it won’t turn on.  I cannot get into it to do anything at all.  But everything I had on it is at my fingertips.  I can pull it down to my PC and keep moving.

But what about viruses, etc…?

Well, users are not always the most thoughtful about what files they download, or install, or interact with.  So it is an advisable idea to run a good UP TO DATE anti virus solution.

I use Norton on some computers (notably, not on any of my PC’s for years, and never had a virus on any of those computers).  But I do prefer to have it on most computers just because it does scan incoming files, and tells you if you were infected.

However, the anti virus software MUST be kept up to date.

Anti virus won’t do any good if Windows has a security hole.  Windows update is not optional, it is a MUST.

Lastly, DO NOT use internet explorer for browsing the internet.  Use Fire Fox.  Sometimes you need to use IE for a conference call, or to check compatibility, but it has far fewer and less severe security holes than IE.

In Summary:

  1. Keep your computer updated via windows update (automatic).
  2. Keep your computer backed up (via sugar sync (automatic).
  3. Keep your browser secure and updated (firefox, automatic)
  4. Don’t download anything except the most reputable software
  5. Don’t install anything from anyone — directly download it if you need it
  6. Don’t open attachments unless you know WHAT they are and WHY you have them.
  7. Run a good up to date anti virus package (norton, automatic updates)

Did I mention keep your computer backed up?  https://www.sugarsync.com/

by

Updating a cert on the Cisco 11500 Series Content Services Switches (CSS)

Having recently moved some of our hosting infrastructure to the excellent Rackspace Platform group, we inherited the management of the Cisco 11500 Series Content Services Switches (CSS), which we use for general load balancing + ssl termination.

As a side note, it’s really powerful, fast, and well, plain nice.  Not having to manage SSL certs on each apache instance is really nice, and all the LAN communication is done over plain old HTTP.

This blog post is a regurgitation of some notes I took internally.  Perhaps someone who finds themselves managing this device will benefit…

The task at hand was re-issuing and updating one of our primary wildcard certificates that powers a lot of subdomains.

The first step is to generate the key, csr, and crt…

All these files should be:

  • Named the same as the domain that SSL is being generated for.
  • use WILD for a wildcard subdomain
  • Use this format “www.domain.com-0810.key”, where 08 is the from year and 10 is the to year
  • (the short version is because of name length limits on the CSS)

Start by generating the key and csr

This should be done in the ciscoftp role under the ~/load directory

# openssl genrsa -out WILD.vosecure.com-0810.key 1024
# openssl req -new -key WILD.vosecure.com-0810.key -out WILD.vosecure.com-0810.csr

Then get the certificate issued by (global sign)

Put the certificate into the the ~/load directory.  When done, it should look like:

-rw-rw-r-- 1 ciscoftp ciscoftp  3139 Apr  6 15:59 WILD.vosecure.com-0810.crt
-rw-rw-r-- 1 ciscoftp ciscoftp   773 Apr  6 15:49 WILD.vosecure.com-0810.csr
-rw-rw-r-- 1 ciscoftp ciscoftp   883 Apr  6 15:47 WILD.vosecure.com-0810.key

Put the crt and key onto the load balancer

To do this, use the “copy command” on the load balancer

20132-201292# copy ssl ftp base import WILD.vosecure.com-0810.crt PEM "rack"
20132-201292# copy ssl ftp base import WILD.vosecure.com-0810.key PEM "rack"

Then make the associations...

20132-201292# config
20132-201292(config)# ssl associate cert WILD.vosecure.com-0810.crt WILD.vosecure.com-0810.crt 
20132-201292(config)# ssl associate cert WILD.vosecure.com-0810.key WILD.vosecure.com-0810.key

Now, it’s time to install it.  Requires SSL downtime!

  1. Suspend the SSL content rule
  2. Suspend the SSL service
  3. Suspend the SSL proxy list
  4. Run the updates
  5. Activate the SSL proxy list
  6. Activate the SSL service
  7. Activate the SSL content rule

Here are the exact commands:

20132-201292# config
20132-201292(config)# owner vosecure.com
20132-201292(config-owner[vosecure.com])# content 74.205.111.161-ssl
20132-201292(config-owner-content[vosecure.com-74.205.111.161-ssl])# suspend

20132-201292# config
20132-201292(config)# service ssl-service
20132-201292(config-service[ssl-service])# suspend

20132-201292# config
20132-201292(config)# ssl-proxy-list ssl-proxy

In the following commands, we remove the whole ssl-server so that it shows up at the bottom in one concise unit. Otherwise, the startup-config and running-config become fragmented.

20132-201292(config-ssl-proxy-list[ssl-proxy])# suspend
20132-201292(config-ssl-proxy-list[ssl-proxy])# no ssl-server 6
20132-201292(config-ssl-proxy-list[ssl-proxy])# ssl-server 6
20132-201292(config-ssl-proxy-list[ssl-proxy])# ssl-server 6 rsakey WILD.vosecure.com-0810.key
20132-201292(config-ssl-proxy-list[ssl-proxy])# ssl-server 6 rsacert WILD.vosecure.com-0810.crt
20132-201292(config-ssl-proxy-list[ssl-proxy])# ssl-server 6 vip address 192.168.1.161
20132-201292(config-ssl-proxy-list[ssl-proxy])# ssl-server 6 cipher rsa-with-rc4-128-sha 192.168.1.161 81
20132-201292(config-ssl-proxy-list[ssl-proxy])# active

20132-201292# config
20132-201292(config)# service ssl-service
20132-201292(config-service[ssl-service])# active

20132-201292# config
20132-201292(config)# owner vosecure.com
20132-201292(config-owner[vosecure.com])# content 74.205.111.161-ssl
20132-201292(config-owner-content[vosecure.com-74.205.111.161-ssl])# active

Test test test.  Firefox, IE, Chrome...

20132-201292# copy running-config ftp base running-config

Review changes with git diff

20132-201292# write memory

20132-201292# copy startup-config ftp base startup-config

And… Here is the git diff

diff --git a/load/startup-config b/load/startup-config
index 7042490..36fbbaa 100644
--- a/load/startup-config
+++ b/load/startup-config
@@ -1,4 +1,4 @@
-!Generated on 04/06/2009 16:05:48
+!Generated on 04/06/2009 21:51:02
!Active version: sg0810205

@@ -64,6 +64,8 @@ configure
+  ssl associate rsakey WILD.vosecure.com-0810.key WILD.vosecure.com-0810.key
+  ssl associate cert WILD.vosecure.com-0810.crt WILD.vosecure.com-0810.crt

!*********************** SSL PROXY LIST ***********************
ssl-proxy-list ssl-proxy
-  ssl-server 6
-  ssl-server 6 rsakey vosecure.com(080421-04300)-key
-  ssl-server 6 rsacert vosecure.com(080421-04300)-cert
-  ssl-server 6 vip address 192.168.1.161
-  ssl-server 6 cipher rsa-with-rc4-128-sha 192.168.1.161 81
@@ -146,6 +141,11 @@ ssl-proxy-list ssl-proxy
+  ssl-server 6
+  ssl-server 6 rsakey WILD.vosecure.com-0810.key
+  ssl-server 6 rsacert WILD.vosecure.com-0810.crt
+  ssl-server 6 vip address 192.168.1.161
+  ssl-server 6 cipher rsa-with-rc4-128-sha 192.168.1.161 81
active
by

xhprof php profilier

Worth noting:

XHProf is a hierarchical profiler for PHP. It reports function-level call counts and inclusive and exclusive metrics such as wall (elapsed) time, CPU time and memory usage. A function’s profile can be broken down by callers or callees. The raw data collection component is implemented in C as a PHP Zend extension called xhprof. XHProf has a simple HTML based user interface (written in PHP). The browser based UI for viewing profiler results makes it easy to view results or to share results with peers. A callgraph image view is also supported.

Read more at http://mirror.facebook.com/facebook/xhprof/doc.html

by

I highly recommend yum + createrepo + rpmbuild

As I was discussing lightly before, I have recently been involved in building quite a few RPMs for our server clusters at AppCove.

Where we have arrived:

Our (new) primary production cluster consists of multiple RedHat Enterprise Linux 5 boxes in different capacities (webserver, appserver, database master, database slave, etc…).

Each machine is registered with 3 yum repositories:

  1. RHEL (RedHat Enterprise Linux)
  2. EPEL (Extra Packages for Enterprise Linux)
  3. ACN (AppCove Network)

All of our custom software packages and custom builds of open source software are placed into individual RPMs, and entered into our ACN repository.

From there, it is a snap to update any given server with the correct version of the software that server needs.

We have a dedicated build area, versioned with git, that is used to build and package all of the custom software that is needed.

(note, RPMs are not used for web application deployment — rsync via ssh is used for that)

Recommendation:

Having worked through the process from start to finish, I must say that I would highly recommend the following tools to anyone who is responsible for RedHat Enterprise, Centos, or Fedora system administration.

  • git – to keep your .spec files versioned
  • rpmbuild – to build the rpms
  • createrepo – to create your very own yum repository
  • apache – to serve the yum repository
  • yum – to obtain, install, and upgrade your rpms

Additionally, if you are using RedHat Enterprise or Centos, I would highly recommend using Extra Packages for Enterprise Linux (EPEL) to get a few of those “other” packages that don’t come with your OS (git, for example).

—
Learning how to build RPMs was a fairly steep curve.  But it wasn’t long.  It is one of those things that if you know it you say “that’s easy” and if you don’t you say “what the ???”

yum+rpm was invented (I assume) to make life easier for countless system administrators and software publishers.  So it’s not the kind of thing that everyone is involved in.

I was a bit tough to figure out the caveats of how to correctly build RPM’s that work.  The documentation is a bit sparse.  A bit here and a bit there.

What are the benefits?

Many.  Let me list a few.

Your system stays really clean. With RPMs, you can uninstall everything you installed without leaving extra files laying around.

Upgrades are a snap. Once you have registered your own yum repository on a system, you can upgrade a given package by running:

yum upgrade your-package

All your systems can be on the same “page”. It is very easy, using yum, to ensure that all of your systems are using the exact same version of software.

Custom builds are super easy to maintain. We custom-compile php, python, and various other software.  Once the .spec files are in place, all of your software can be re-packaged with a single command.

In our specific case, we wanted to have the memcached client statically compiled into PHP.  With a few extra commands in the .spec file, it was a snap to pull in the source from pecl, and update `configure` to take it into account.

All builds can take place in one place. With one set of documentation, one consistent set of development tools, etc…  We have a user called `build` on one of the hosts that is specifically used for building all of the RPMs.

Where to learn?

The best way to learn, as usual, is to jump in and figure it out.   There is some really good documentation buried in the rpm.org site.   It is a book called Maximum RPM, origninally published by redhat.  The current snapshot of the book is available online.

http://www.rpm.org/max-rpm-snapshot/

Google is another good resource, depending on what it is you are looking for.